clawoxy Website Logo
English
English简体中文
Start free

glossary

Category: Glossary

  • cheerio

    Cheerio is a Node.js library for loading and querying HTML or XML with a jQuery-like API. Give it markup, then use familiar selectors such as $('h1') or $('.price') to read or traverse elements. It is a parser, not a browser.

    A Small Example

    import * as cheerio from 'cheerio';

const $ = cheerio.load('<h1 class="title">Books</h1>');
const title = $('.title').text();

    The $ function searches the parsed document. Cheerio does not fetch the HTML for you in every setup; commonly, a separate HTTP client gets the response and passes its body to cheerio.load().

    When Cheerio Is a Good Fit

    It works well when the response already contains the content you need and CSS selectors are a convenient way to extract it. It is lighter than launching a full browser for every page. If a site inserts the target content with JavaScript, Cheerio will not run those scripts; inspect the received HTML first, then use a browser-based tool only if rendering is actually required.

    One easy trap: an empty selection often returns an empty string rather than throwing an error. Check the number of matches before assuming your selector found the right field. Also trim or normalize whitespace yourself when the source markup contains line breaks and indentation.

    Sources

  • Coverage

    Coverage describes how much of the intended collection a crawler actually captured. It is not simply the number of rows in an export: a large result can still miss important sections, dates, or page types.

    Define the denominator

    Before measuring coverage, state what counts as the target set. It might be URLs from a sitemap, products in a catalog, pages within a date range, or records returned by a defined query. Then count successful, valid results against that scope. If the total is unknown, report a proxy—such as discovered URLs processed—and label it clearly.

    Break results down by useful dimensions: section, page type, status code, or crawl date. This helps distinguish a parser failure from pages the crawler never discovered. Track missing required fields separately from pages that were never fetched; they represent different gaps.

    Coverage is a measurement, not a guarantee of completeness. Sites change, links may be inaccessible, and the target set may be ambiguous. Preserve the scope and exclusions with each run so future comparisons mean the same thing.

    Sources

  • Crawlee

    Crawlee is an open-source JavaScript and TypeScript library for building web crawlers and scraping tasks. It provides crawler classes, request handling, and a queue for URLs that a crawler still needs to process. The class you choose determines whether the task fetches HTML directly or uses a browser.

    Choosing a Crawlee Crawler

    CheerioCrawler makes HTTP requests and parses the returned HTML with Cheerio. It is a sensible starting point when the content is already in the response. If a page needs JavaScript to render the data or requires browser interaction, Crawlee also provides browser-based options such as PlaywrightCrawler and PuppeteerCrawler.

    The request queue can start with a known list of URLs and grow as the crawler discovers links. A request handler then processes each response, extracts fields, and can add more requests. That is the useful distinction from a parser library alone: Crawlee can coordinate the crawl as well as parse pages.

    What It Does Not Decide for You

    You still need to define the fields to collect, validate the output, and choose limits that fit the target site. More workers or retries are not automatically better; they can increase load or lead to throttling. Follow the site’s access rules and use an approved API when one fits the task.

    Sources

  • Git Bash

    Git Bash is a command-line environment for Windows that provides Git plus a collection of Unix-style shell tools. It lets developers use familiar commands such as ls, grep, and shell scripts alongside Git without switching to a Linux machine.

    Where it fits

    Git Bash is useful when a project documents commands for Bash or when a small script needs common Unix utilities. It is not the same thing as Git itself: Git is the version control program, while Git Bash provides a shell environment in which Git and other commands can run.

    Windows paths and shell syntax can differ from PowerShell or Command Prompt. A command that uses environment variable syntax, quoting, or path separators in Bash may need adjustment in another shell. If a team shares setup instructions, state which shell they assume.

    For crawler work, Git Bash can run repository commands and simple scripts. For longer or platform-specific automation, choose the shell and runtime that match the project and document the expected environment.

    Sources

  • Googlebot

    Googlebot is the name used for Google’s crawlers that discover and fetch pages for Google Search and related services. Site owners can review crawl activity in server logs and Search Console, but a request that merely claims to be Googlebot should not automatically be trusted.

    Verifying crawler requests

    Google documents a verification process for requests claiming to come from Google crawlers. Depending on the method, this can involve checking the source IP against published ranges or performing a reverse DNS lookup followed by a forward lookup. Follow Google’s current instructions rather than relying only on the user-agent string, which any client can set.

    Googlebot’s behavior is separate from the schedule of a crawler you operate. A site can influence crawling through availability, internal links, and supported controls, but it does not directly set Google’s internal crawl schedule. If Google’s requests cause a critical load problem, use Google’s documented crawl-rate guidance.

    Do not use the Googlebot name for unrelated automated requests. Accurate identification helps site operators distinguish legitimate crawler traffic from other clients.

    Sources

  • Web Scraping

    Web scraping is the process of collecting selected information from web pages or related interfaces and turning it into structured data. It can involve direct HTTP requests, HTML parsing, browser automation, or an API, depending on how the source exposes its content.

    A reliable collection workflow

    Define the fields and scope first. Inspect a representative response, choose selectors or a documented endpoint, validate the extracted records, and preserve the source URL and collection time. Separate extraction from normalization so the original value remains available when a transformation needs to be checked.

    Before running a collection, review the site’s terms and published access rules. Use conservative request pacing, handle 429 responses by backing off, and avoid retry loops. Public accessibility does not automatically settle whether a particular use is permitted.

    Scraping quality depends on more than parsing. Coverage, freshness, duplicates, and source changes all affect whether the resulting dataset is useful. Measure them explicitly and mark partial results honestly.

    Sources

  • Zero-Result Page

    A zero-result page is a search, filter, or category page that contains no matching records. In a crawl, it can be a valid outcome, a sign that a filter is too narrow, or evidence that the request landed on an error or challenge page.

    Distinguish an empty result from a failed fetch

    Check the HTTP status, page title, expected result container, and any explicit “no results” message. A 200 status alone does not prove that the page represents a normal empty search; some sites return error or verification pages with the same status.

    Store zero-result outcomes separately from fetch failures and parser failures. This keeps a legitimate empty query from being retried endlessly and makes coverage reports more accurate. If filters are user-controlled, validate them and record the exact query state.

    For site owners, make empty states clear and provide a useful way to adjust filters. For crawlers, set a stopping rule so a sequence of empty pages does not continue indefinitely.

    Sources

  • Hydration

    Hydration is the process by which client-side JavaScript attaches behavior to HTML that was already rendered, often on the server. A page may therefore contain readable HTML before the browser finishes initializing its interactive components.

    Why it matters when collecting pages

    If the first response includes the target text, a parser may not need to wait for hydration. If the page renders only a shell or fills fields after JavaScript runs, a browser may be necessary. Inspect the raw HTML and network requests first; this reveals whether the data is already present or arrives later from an API.

    Hydration can cause timing issues in automation. Waiting for the document’s load event may be too early for a particular component, while waiting for all network activity to stop may never finish on pages with analytics or long-polling. Wait for a specific selector or state that corresponds to the content you need.

    Framework behavior varies, so avoid relying on internal hydration scripts as a stable data interface. Prefer documented endpoints or semantic page content where available.

    Sources

  • XPath Selector

    An XPath selector is an XPath expression used to choose nodes from an XML or HTML document. A crawler can use it to extract text, attributes, or groups of elements such as product cards or article links.

    Make the selector match the page’s structure

    Use stable attributes and relationships instead of counting nested div elements. For example, select a card by a known class and then find its title within that card, rather than querying every title on the page and hoping order stays fixed.

    Check whether the selector returns zero, one, or many results, and handle each case deliberately. A zero result may mean the page changed, the wrong response was parsed, or the content is loaded later with JavaScript. Inspect the raw response before rewriting the selector.

    Different libraries expose XPath results in different forms. Read the framework’s documentation and test a saved response sample before running at scale.

    Sources

  • Anti-Bot Detection

    Anti-bot detection is the process of identifying automated requests and deciding how a website should handle them. A bot is not automatically harmful: search crawlers, uptime monitors, and user-authorized automation can be legitimate, while credential attacks, spam, and excessive scraping can create risk or load. Good detection evaluates the request’s context and purpose instead of treating every automated visit the same way.

    Signals Used to Assess Automated Traffic

    Websites may combine several signals rather than rely on one test:

    • Request patterns: unusually high rates, repeated paths, or synchronized activity across many accounts or addresses.
    • Network context: source IP, network provider, reputation, and geographic patterns.
    • Protocol and client details: headers, TLS characteristics, and whether the claimed client is consistent with the request.
    • Session and browser behavior: cookie continuity, JavaScript capability, and interaction patterns.
    • Known-bot verification: checks that establish whether a crawler claiming to be a search engine actually belongs to that service.

    No single signal proves that a request is malicious. A shared network, privacy setting, browser extension, or monitoring job can resemble automation. Detection should therefore produce a risk assessment that can be reviewed, not an unquestionable identity label.

    Possible Responses to Detected Traffic

    Depending on the risk and resource, a site can allow the request, log it, apply a rate limit, request additional verification, or block it. Logging or monitoring first helps reveal false positives before stricter rules affect real users or useful crawlers. For sensitive actions such as login, account creation, and checkout, use controls appropriate to the action rather than applying the same challenge to every page.

    Verifying Legitimate Search Crawlers

    Do not trust a user-agent string by itself; it can be copied. Google recommends verifying Googlebot requests with a reverse DNS lookup or by matching the source IP against Google’s published crawler IP ranges. Keep verification procedures current and apply them only to the crawler identity they are designed to confirm.

    Investigating a False Positive

    1. Capture the request ID, timestamp, path, status code, and the detection reason available in your logs or security dashboard.
    2. Check which layer acted: application, CDN, web application firewall, or bot-management service.
    3. Compare affected requests with normal traffic and confirm whether the client is a legitimate crawler or an authorized integration.
    4. Narrow the rule or add a verified exception only when evidence supports it; keep rate limits and monitoring where appropriate.
    5. Recheck the affected workflow and continue watching for abuse or accidental blocking.

    For a crawler operator who receives a block or challenge, treat it as an access decision. Review the site’s published policies and contact the site owner for permission or an approved interface. Do not attempt to disguise automation or bypass a challenge.

    In short: Anti-bot detection combines traffic and client signals to manage automated access. Classify behavior, verify trusted crawlers, and tune actions carefully to reduce both abuse and false positives.

    Sources