clawoxy Website Logo
English
English简体中文
Start free

glossary

Category: Glossary

  • Proxy Pool

    A proxy pool is a set of proxy endpoints that an application can select from for outbound connections. Organizations may use one to manage egress, test regional behavior, or separate workloads. Pool size alone says little about reliability, privacy, or whether a use is permitted.

    Evaluate the pool for the task

    Check supported protocols, authentication, geographic routing, availability, logging terms, and how the provider handles failures. Keep credentials out of code and logs. If a proxy fails, retry cautiously and record the endpoint and error class so recurring issues can be diagnosed.

    Apply request limits by destination host across the entire pool. Switching endpoints should not reset a site’s rate limit or be used to evade a block. Proxies can also alter cookies, TLS behavior, and location-dependent responses, so keep the route stable when reproducibility matters.

    Use a trusted provider for sensitive data and verify the network path with an endpoint you control or are allowed to test.

    Sources

  • Protocol

    A protocol is an agreed set of rules for exchanging information between systems. Web collection relies on several protocols: DNS resolves names, TCP or QUIC carries connections, TLS protects many connections, and HTTP defines request and response semantics.

    Diagnose one layer at a time

    When a fetch fails, identify the stage. A DNS lookup error happens before a connection; a TLS certificate failure occurs during secure connection setup; an HTTP 429 means the server responded and is rate-limiting the client. Treating all of these as “the site is down” makes troubleshooting harder.

    Tools may negotiate different protocol versions or use different proxies. Capture the relevant error, status, and negotiated protocol when available. Avoid changing several layers at once, because that can hide the cause of a problem.

    Understanding the protocol stack also helps choose the right client: a plain HTTP library is enough for many pages, while browser automation is needed only when browser execution or interaction is part of the task.

    Sources

  • Playwright

    Playwright is a browser automation framework for controlling Chromium, Firefox, and WebKit. Developers use it for end-to-end tests and for workflows that need a real browser engine, such as waiting for client-rendered content or interacting with a page.

    A stable browser workflow

    Use locators tied to accessible names or meaningful page structure, and wait for a specific condition rather than sleeping for a fixed duration. Reuse a browser context when the workflow needs shared cookies, but isolate unrelated accounts and jobs. Close pages and contexts when finished so workers do not accumulate resources.

    For data collection, check whether the response already contains the needed fields before opening a browser. Browser automation costs more than a direct request and does not grant permission to access a site. Keep concurrency modest and respect the site’s published limits.

    Playwright APIs and browser versions change. Use the official documentation for the language binding and version installed in the project.

    Sources

  • NAT (Network Address Translation)

    Network Address Translation (NAT) changes network address information as packets pass through a router or gateway. It commonly lets devices on a private network share one or more public IPv4 addresses when connecting to the internet.

    Why it appears in crawler diagnostics

    Several workers behind one NAT gateway may appear to a destination as traffic from the same public IP address. That can affect rate limits, logs, and how a site groups requests. A crawler operator should monitor request pace at the site level rather than assuming each worker has a separate identity.

    NAT is different from an application-layer proxy, although both can affect the address a destination observes. Network topology may also involve carrier-grade NAT upstream, which is outside the local router’s control. If an IP-based limit is hit, do not evade it by changing addresses; review the service’s rules and lower or stop traffic.

    For debugging, compare local routing information with the public egress address using an endpoint you control or trust, and avoid publishing internal addressing details unnecessarily.

    Sources

  • Metadata

    Metadata is information that describes other data. A scraped record’s metadata might include its source URL, collection time, language, response status, or parser version. These details help people understand where a value came from and whether it is suitable for a particular use.

    Keep metadata useful

    Choose fields that answer real questions: when was this collected, from which source, under what locale, and how was it transformed? Give each field a stable definition and avoid mixing metadata with source content. A timestamp generated by the crawler should not be confused with a publication date shown on the page.

    Metadata supports filtering, debugging, and reproducibility, but more is not always better. Request headers, cookies, and account identifiers can contain sensitive information. Retain only what the task needs and define who can access it and how long it is kept.

    If the schema changes, document the new field or meaning and preserve version information where older records remain. This prevents downstream users from interpreting a field inconsistently across runs.

    Sources

  • JSON-LD

    JSON-LD is a way to express structured data using JSON and linked-data concepts. Web pages often include it in a <script type="application/ld+json"> element to describe entities such as articles, products, events, or organizations.

    Extracting JSON-LD

    Locate the script element, parse its text as JSON, and inspect the @type and fields before mapping it to your schema. A page can contain multiple JSON-LD blocks or use an @graph array, so do not assume there is one object with one predictable shape.

    Structured data is supplied by the page author and may be missing, stale, or inconsistent with visible content. Validate important values against the page and preserve the source URL. Search engines may use structured data under their own rules; its presence does not guarantee a rich result.

    When parsing fails, save the exact script text for diagnosis. HTML entities, trailing commas, or a site-specific format can explain why a generic JSON parser rejects it.

    Sources

  • IP Rotation

    IP rotation changes the network address used for successive requests, often through a proxy service or a pool of outbound connections. It can support legitimate testing across regions or help distribute traffic among approved egress points, but it does not make a request anonymous or automatically compliant with a site’s rules.

    Operational considerations

    If different addresses are required for a permitted task, record which egress address was used for each request and keep rate limits at the service or domain level. A global limit should not reset just because the outbound IP changed. Otherwise a crawler may unintentionally multiply load.

    Rotating addresses can also complicate sessions, caching, and debugging. A site may bind session state to an address, and a failure may be harder to reproduce if the route changes every time. Use stable egress when diagnosing problems and rotate only when there is a clear operational reason.

    Do not use rotation to evade an access restriction or rate limit. If requests are denied, stop and review the site’s terms or use an authorized API or collection method.

    Sources

  • Data Pipeline

    A data pipeline is the sequence of steps that moves information from a source to a usable destination. For web data, that sequence may include URL discovery, request scheduling, fetching, parsing, validation, deduplication, and storage. Thinking of it as a pipeline makes failures easier to locate than treating “the scraper” as one opaque program.

    A practical layout

    Keep each stage’s input and output clear. A fetcher returns a response plus status and timing details; a parser turns that response into candidate records; validation checks required fields and types; storage writes accepted records while preserving source metadata. Failed requests and invalid records should have their own paths so they can be reviewed or retried safely.

    The stages do not all need separate services. A small job can run them in one process. Separation becomes useful when one stage has a different workload, needs independent retries, or is causing delays elsewhere. Measure where time and errors occur before adding queues or distributed workers.

    Idempotency matters when a task retries. A stable record key or URL fingerprint can prevent a repeated run from silently creating duplicate rows. Keep enough provenance to tell a fresh result from a retry or an update.

    Sources

  • Data Extraction

    Data extraction is the step that turns a page, API response, or document into fields a task can use. A crawler may fetch HTML successfully and still extract nothing useful if it selects the wrong element, reads a placeholder, or misses content rendered after the initial response.

    From response to records

    Start by identifying the source format. JSON responses can often be mapped directly to fields; HTML requires a parser and selectors; a browser workflow may be needed when relevant content only appears after interaction. For each field, define its expected meaning and type before writing extraction logic.

    A reliable extractor should handle missing values, repeated elements, and small layout changes deliberately. For example, a product page may show a price in more than one place. The extraction rule should identify the primary price and preserve the currency instead of returning whichever matching number appears first.

    Keep extraction separate from cleaning and validation. Extraction answers “what did the source contain?”; later steps can normalize formats and decide whether the values meet the dataset’s requirements. Saving the source URL and capture time makes later corrections much easier.

    Sources

  • CGNAT (Carrier-Grade NAT)

    Carrier-Grade NAT (CGNAT) lets an internet provider share public IPv4 addresses among multiple customers. A customer’s router may perform one layer of network address translation, while the provider performs another between its network and the public internet. As a result, many subscribers can appear to use the same public IP address.

    Why It Matters for Crawling and Rate Limits

    IP-based systems may see a shared address for traffic from unrelated users. A rate limit, block, or reputation score attached only to that address can therefore affect more than one subscriber. Conversely, an address change does not necessarily mean the underlying user or device changed.

    The range 100.64.0.0/10 is reserved as shared address space for this purpose; it is not the same as the private IPv4 ranges in RFC 1918. Seeing an address in that range on the provider side can be a clue, but it does not reveal every detail of a network path.

    CGNAT is a network translation arrangement, not a proxy service. It can complicate inbound connections and attribution, but it does not itself provide a different country or a dedicated public IP. When diagnosing a crawler, compare application logs with network information instead of treating one IP as a user identity.

    Sources