Category: Glossary

A JavaScript challenge is a browser-side check that asks a client to run code before the site serves the requested content. It may be part of a security or abuse-prevention system, and its presence can indicate that a site’s access policy requires verification.

How to interpret the response

If a crawler receives a challenge page instead of the expected content, record the status, response headers, and a small sanitized sample. Do not parse the challenge markup as if it were the target page; that can create plausible-looking but invalid records. Repeated challenges should prompt a pause and review of the site’s published access rules.

For a site owner or authorized tester, check that legitimate users and supported crawlers can complete the intended flow. Browser automation may be appropriate for testing the site, but automating a challenge to bypass a third-party restriction is not an appropriate collection strategy.

Challenge behavior can vary by browser and network. Treat it as an access decision, not a parser error, and use an official API or request permission when automated access is needed.

Sources

• Cloudflare: Challenges
• Google: reCAPTCHA overview

“Generic AI” is an informal label rather than a specific model, product category, or technical standard. People often use it to mean general-purpose AI tools that can handle several kinds of tasks, such as summarizing text, drafting, answering questions, or helping with code.

How the phrase differs from a specific system

The label does not tell you which model is used, what data it can access, whether it runs in the cloud, or how its outputs are checked. Those details matter more than the broad phrase when choosing a tool for a workflow. Ask what the system actually does, what inputs it receives, and what happens to the information afterward.

In a data collection pipeline, an AI model might help classify or summarize records after retrieval. That is different from the crawler itself: the crawler fetches pages, while the model interprets some of the collected content. Keep a record of source data and review model-generated fields for errors, especially when the result affects decisions.

Because the term is used loosely, define it in context instead of treating it as a formal technical term.

Sources

• NIST: Artificial Intelligence

Font fingerprinting uses the fonts available to a browser as one possible signal for distinguishing devices or browser environments. A page script may test whether particular fonts render differently from a fallback, then combine the result with other characteristics. Font availability alone is not a reliable identifier of a person.

Privacy and diagnostics

Font lists can reveal hints about an operating system, installed software, or language setup. In anti-fraud or compatibility work, those hints may help explain why a page renders differently. They are also part of the broader browser-fingerprinting discussion because multiple seemingly ordinary signals can become more distinctive when combined.

If a page looks wrong, check the actual font files, CSS fallbacks, language coverage, and browser console before assuming the browser is being fingerprinted. For privacy reviews, inventory scripts that inspect local fonts and assess whether the collection is necessary and disclosed.

Browsers and privacy tools may limit or standardize some fingerprinting surfaces, but behavior changes over time. Avoid making strong identity claims from a single font check.

Sources

• W3C: Fingerprinting Guidance

A fetch queue holds requests waiting for a crawler to retrieve them. It is usually part of a larger frontier that also tracks discovered URLs, priorities, duplicate checks, and retry state. The queue’s policy determines what gets fetched next and how work is shared across workers.

Queue state that helps

For each request, keep a stable URL key, priority, host, attempt count, and state such as pending, in progress, completed, or failed. A delayed queue can hold retries until their backoff period ends. Per-host limits prevent a busy domain from filling all available worker slots.

Monitor queue depth together with the age of the oldest request and completion rate. A growing queue may mean the crawler discovered too much, workers are failing, or a particular host is slow. Increasing worker count without checking host-level limits can simply increase pressure on the same site.

For a short, small job, a basic in-memory queue is often enough. Durable queues matter when jobs must survive restarts or distribute work across machines.

Sources

• Scrapy: Scheduler

“Elite proxy” is a common industry label for a proxy that does not expose the usual proxy-related request headers to the destination. The term is not a formal protocol standard, and providers may define it differently, so it is better to check the specific behavior than rely on the label alone.

What the label can and cannot tell you

A proxy forwards traffic through another network endpoint. Some services describe proxies as transparent, anonymous, or elite based on how much identifying information the destination can infer from the connection and headers. Those labels do not guarantee anonymity: the destination may still observe the proxy IP, browser characteristics, cookies, or patterns of use.

When evaluating a proxy for a legitimate collection workflow, check the provider’s documentation, geographic routing, authentication method, logging terms, and whether the service supports the protocols you need. Verify the response path with a test endpoint you control or are permitted to use. Do not send credentials or sensitive data through an untrusted proxy.

Proxy choice does not replace permission to collect from a site. Review its terms, published access rules, and rate limits before running requests.

Sources

• RFC 9110: HTTP Semantics

The Domain Name System (DNS) maps names such as example.com to network records used to connect to a service. A crawler usually resolves a hostname before opening a connection; the resolver’s answer, cache, and network path can therefore affect where a request goes and how long it takes to start.

DNS in a crawl

When a request fails before an HTTP response arrives, check whether the hostname resolved, which record type was returned, and whether the failure is repeatable from the same network. DNS errors are different from HTTP errors: a 404 came from an HTTP server, while a name-resolution failure means the client did not reach that stage.

Caching matters. A resolver may reuse an answer for the record’s time to live, so a changed DNS record may not appear immediately everywhere. Conversely, resolving the same host for every URL can add needless latency. Most HTTP clients and operating systems already have caching behavior, so diagnose actual resolution patterns before adding another cache.

For reproducible debugging, record the hostname and error class rather than assuming a particular IP address is permanent. DNS answers can legitimately vary by resolver, location, and service configuration.

Sources

• ICANN: DNS basics
• RFC 1034: Domain Names

A content delivery network (CDN) is a group of servers that delivers web content from locations closer to visitors. A CDN can cache eligible files at its edge, so repeated requests may be served without going back to the origin server. Many CDNs also act as reverse proxies and offer routing or security services.

What a CDN Changes for a Crawler

A crawl request may reach a CDN edge first. The response can be a cached copy, a fresh response from the origin, or a challenge or error generated at the edge. The response headers can help explain what happened: cache status and Age may indicate whether a stored copy was served, while a vendor-specific header can identify a challenge.

This matters when checking freshness or comparing pages from different regions. A successful request does not always mean the origin was contacted, and an edge cache can serve a version that has not yet been revalidated. Record the response time, headers, and location context when those details affect the collection.

CDN Does Not Mean Every Page Is Cached

Static assets such as images, CSS, and JavaScript are often cacheable. Dynamic HTML may be bypassed unless the site owner configures caching rules. The origin’s cache headers, CDN rules, cookies, and query strings can all affect behavior. Avoid assuming every response from a CDN is stale—or that every page is cached.

Sources

• Cloudflare: CDN reference architecture
• Cloudflare Cache documentation

Browser fingerprinting identifies or recognizes a browser by combining details it exposes, such as its version, language, time zone, display size, fonts, and available media features. A website can collect several of these signals through browser APIs; together, they may distinguish one browser configuration from many others.

Fingerprints Are Signals, Not Login Credentials

A fingerprint can help a site assess whether visits may come from the same browser, but it is not a stable account identifier. Browser updates, settings, extensions, shared devices, and privacy protections can change the signals. Two people can also share similar configurations. Treat a fingerprint as one probabilistic signal, not proof of identity.

In web crawling and testing, fingerprinting may come up when a site evaluates client behavior or when teams investigate why two browser sessions receive different responses. A changed fingerprint can affect the result, but it does not by itself explain a block or verify that a browser belongs to a particular person.

Privacy Considerations

Fingerprinting can support fraud prevention and session risk checks, but it can also contribute to tracking across visits. Browser vendors limit some exposed information or add variation to make identification harder. Those protections reduce the usefulness of fingerprints; they do not make every browser indistinguishable.

For site owners, collect only signals that serve a clear purpose and account for false positives. For users, browser privacy settings can reduce some fingerprinting, though behavior varies by browser and may affect site compatibility.

Sources

• W3C: Fingerprinting Guidance

Asynchronous scraping lets a crawler make progress on other work while a request is waiting for a network response. It is useful when a task spends much of its time waiting on websites, APIs, or storage. It does not make each request finish faster, and it does not automatically make CPU-heavy parsing faster.

How Async I/O Overlaps Network Waits

In a synchronous workflow, code commonly waits for one request to finish before starting the next. With asynchronous I/O, a request can yield control while it waits. The event loop can then resume another ready task. Several requests may therefore be in flight at once without assigning a dedicated thread to each wait.

“Concurrent” describes overlapping work; it does not necessarily mean the CPU is executing several tasks at the exact same instant. Async I/O helps hide network waiting time. Heavy image processing or other CPU-bound work may need a separate approach, such as worker processes.

Where Async Helps and Where It Does Not

It can help when many independent pages or API records need to be fetched and the destination allows a measured level of concurrency. It offers little benefit if each URL depends on the previous response, the site is already the bottleneck, or local parsing and storage consume most of the time.

Frameworks can manage concurrency without requiring every part of a spider to use async and await. For example, Scrapy uses non-blocking I/O for concurrent downloads. In Python applications built around asyncio, coroutines can be scheduled together and awaited as a group.

Concurrency Limits and Failure Handling

Set a maximum number of in-flight requests and a per-host rate or delay. More concurrency is not always faster: it can increase memory use and trigger throttling, errors, or access restrictions. Handle timeouts, cancellation, retries, and partial failures, and follow the site’s published access rules. Start conservatively and adjust from observed response behavior.

Async scraping is a way to organize I/O-bound work, not a permission bypass or a guarantee of higher throughput. Choose it when network waits dominate, then measure the result under appropriate request limits.

Sources

• Python: asyncio
• Scrapy: Architecture overview
• Scrapy: Optimization

X-Forwarded-For (XFF) is an HTTP header commonly used by proxies to pass along the IP address of a client that connected through them. A chain of proxies may append multiple addresses, usually separated by commas.

Trust the proxy chain, not the header by itself

Clients can send arbitrary headers. A server should use XFF for security or logging only when the request came through a trusted proxy that overwrites or validates the value. Otherwise an attacker can supply a misleading address. The trusted proxy configuration determines which entries in a chain are meaningful.

For crawler diagnostics, XFF may explain how an application identifies the original client behind a load balancer. It should not be assumed to exist or be accurate on a direct request. Avoid adding it yourself unless you operate the trusted proxy that sets it.

Do not log full IP chains longer than necessary; define retention and access rules for network identifiers.

Sources

• RFC 7239: Forwarded HTTP Extension
• Cloudflare: http.x_forwarded_for