clawoxy Website Logo
English
English简体中文
Start free

glossary

Category: Glossary

  • WebRTC (Web Real-Time Communication)

    WebRTC is a set of browser technologies for real-time audio, video, and data communication. It enables peer connections and media exchange, typically with signaling and network traversal mechanisms configured by the application.

    Why it can matter in browser diagnostics

    WebRTC may use network paths that differ from ordinary HTTP page requests. When diagnosing connectivity or privacy behavior, distinguish media traffic from the browser’s web requests and review the browser’s current privacy controls. A WebRTC observation is not a general measure of a user’s identity or location.

    For a crawler, WebRTC is rarely needed to retrieve page text. If a site uses it for a legitimate interactive feature, browser automation may be required to test that feature, but it adds complexity and should be scoped to the task.

    WebRTC behavior depends on browser, network, and application configuration. Use the relevant browser documentation and avoid treating one test result as universal.

    Sources

  • URL Canonicalization

    URL canonicalization converts addresses that should be treated as equivalent into one consistent representation. A crawler may use it to reduce duplicate requests and keep records from being split across harmless URL variations.

    Normalize without losing meaning

    Common steps include lowercasing the hostname, removing a fragment that is not sent to the server, and normalizing default ports. Query parameters need more care: tracking tags may be safe to remove, while language, pagination, filters, or product variants may change the content.

    Keep the original URL alongside the canonical key. This preserves provenance and makes it possible to revisit the exact source. Test normalization rules against pairs that should merge and pairs that must remain distinct.

    A page’s canonical link can be a useful hint for search indexing, but it may be absent or incorrect and does not necessarily define identity for your dataset. Use the task’s requirements and observed content to decide.

    Sources

  • TTFB (Time to First Byte)

    Time to First Byte (TTFB) measures the interval from starting a request until the first response byte arrives. It includes multiple parts of the request path, such as connection setup, network travel, and server processing, depending on how the measurement is taken.

    Use TTFB to narrow a delay

    Compare TTFB across repeated requests and note whether the connection was reused. A slow first request may reflect DNS, TLS, or connection setup; a slow response on a reused connection may point more toward server processing or network latency. TTFB alone does not identify the cause.

    Measure with a consistent client and environment, and separate it from total download time. A response can have a low TTFB but a large body that takes much longer to transfer. For crawl scheduling, high latency may justify reducing concurrency to a host, but first confirm that the delay is persistent rather than a one-off.

    Sources

  • TLS Fingerprint

    A TLS fingerprint summarizes selected characteristics of a client’s TLS handshake, such as supported versions, extensions, or cipher suites. A server can use it as one signal when classifying traffic, but it is not a stable identity and can vary by client library or version.

    Use it for diagnostics, not certainty

    TLS fingerprints can help an operator understand why two clients reach different routes or receive different treatment. They can also change after software updates, so log the client version and network path when comparing results. A match does not prove two requests came from the same person, and a difference does not necessarily mean they did not.

    For crawler developers, use supported TLS implementations and identify the client honestly. Trying to mimic another browser to bypass a site’s controls is not a substitute for authorization. For privacy reviews, consider whether collecting or retaining handshake details is necessary for the operational purpose.

    Sources

  • SSL/TLS

    TLS (Transport Layer Security) protects data exchanged over a network and is used by HTTPS. SSL is an older predecessor that is obsolete; people still write “SSL/TLS” when referring broadly to secure web connections, but modern deployments should use supported TLS versions.

    Diagnosing a TLS failure

    Certificate errors can indicate a hostname mismatch, an expired or untrusted certificate, a wrong system clock, or an intermediary changing the connection. Keep certificate verification enabled. Disabling it removes an important security check and can expose credentials or data to interception.

    When debugging, capture the error message, hostname, client version, and network path. A proxy may tunnel TLS or terminate it, which changes what certificate the client sees. Do not assume an HTTP status exists if the handshake failed before HTTP began.

    Protocol and cipher support can change over time. Follow current browser and library guidance rather than enabling old SSL versions for compatibility.

    Sources

  • Scrapy

    Scrapy is a Python framework for building web crawlers and scraping workflows. It provides spiders, request scheduling, downloader middleware, item pipelines, and tools for following links and extracting structured data.

    When Scrapy is a good fit

    Use it when a task has a repeatable set of URLs, multiple pages to follow, or processing steps that benefit from a scheduler and pipeline. A small one-off fetch may be simpler with Python’s standard library or an HTTP client. Browser automation is a separate choice for pages that require JavaScript execution.

    Define allowed domains and URL patterns, set conservative concurrency and download delays, and handle retries deliberately. Keep item fields and validation rules explicit so a page-layout change does not quietly produce empty records. Scrapy’s components can be extended, but custom middleware should not hide failures or bypass site restrictions.

    The framework evolves; use the documentation for the installed version when configuring settings or APIs.

    Sources

  • Robots.txt

    robots.txt is a plain-text file at a site’s root that publishes crawler instructions, such as which paths a user-agent group may or may not fetch. A crawler typically checks a URL like https://example.com/robots.txt before requesting pages on that host.

    Reading the file correctly

    Rules apply to user-agent groups and URL paths according to the Robots Exclusion Protocol. A Disallow line is not a password-protected boundary; the URL may still be accessible to a browser or another client. Conversely, a path omitted from the file is not automatically authorized for every use. Review the site’s terms and access policy as well.

    For site owners, keep the file syntactically valid and test how intended crawlers interpret the rules. Do not list private URLs as a way to hide them; use authentication for restricted content.

    Directives such as Crawl-delay are not standardized by RFC 9309, so crawler support differs. Consult the documentation for the crawler you use.

    Sources

  • Request Fingerprinting

    Request fingerprinting groups or classifies HTTP requests using a combination of their attributes. These may include headers, URL patterns, timing, connection properties, or repeated behavior. Operators use such signals for traffic analysis and abuse prevention, but classification is often probabilistic.

    Limits and responsible use

    Different clients can share common request patterns, while the same browser can produce different requests across versions or settings. A fingerprint should be treated as one signal, not proof of a particular person or intent. False positives can affect legitimate automation, monitoring, and accessibility tools.

    For site operators, document the signals used and review how they affect users. For crawler developers, send accurate headers, respect published access rules, and avoid trying to disguise a client to evade controls. If a request is challenged or denied, pause and seek an authorized route.

    Collect only the request data needed for a clear operational purpose and define retention. Headers can contain identifiers or credentials that should not be logged in plain text.

    Sources

  • Puppeteer

    Puppeteer is a JavaScript library for controlling Chrome or Chromium through browser automation. It can navigate pages, interact with elements, inspect the DOM, and capture screenshots, making it useful for testing and browser-dependent workflows.

    Use a browser only when needed

    Check whether the required content is present in the initial HTML or a documented data endpoint first. Puppeteer is useful when JavaScript rendering or interaction is essential, but it consumes more resources than a direct HTTP request. Wait for a specific selector or network response rather than relying on a fixed delay.

    Keep browser sessions isolated when tasks use different accounts, and avoid logging cookies or authorization data. Limit parallel pages and handle timeouts so a slow site does not exhaust the worker. Browser automation does not bypass terms or access restrictions.

    Puppeteer is maintained by the Chrome team; consult its current API docs because browser and library behavior evolves.

    Sources

  • Public Suffix List (PSL)

    The Public Suffix List (PSL) is a community-maintained list of domain suffixes that helps software determine where registrable domain boundaries fall. For example, the public suffix may be more than one label long, so simply taking the last two parts of a hostname is not always correct.

    Why crawlers and data tools use it

    The PSL can help group URLs by registrable domain, apply per-site limits, or reason about cookie scope. Use a maintained library and current list rather than writing a simplistic string rule. The list changes as domain registrations and policies evolve.

    The PSL is not a DNS database and does not tell you whether a host exists or who owns it. It describes suffix boundaries for software use. If domain grouping affects request pacing or data joins, record the library and list version used so results can be reproduced.

    Avoid treating subdomains as interchangeable by default; they can be operated by different services or have different access policies even when they share a registrable domain.

    Sources