A fetch queue holds requests waiting for a crawler to retrieve them. It is usually part of a larger frontier that also tracks discovered URLs, priorities, duplicate checks, and retry state. The queue’s policy determines what gets fetched next and how work is shared across workers.
Queue state that helps
For each request, keep a stable URL key, priority, host, attempt count, and state such as pending, in progress, completed, or failed. A delayed queue can hold retries until their backoff period ends. Per-host limits prevent a busy domain from filling all available worker slots.
Monitor queue depth together with the age of the oldest request and completion rate. A growing queue may mean the crawler discovered too much, workers are failing, or a particular host is slow. Increasing worker count without checking host-level limits can simply increase pressure on the same site.
For a short, small job, a basic in-memory queue is often enough. Durable queues matter when jobs must survive restarts or distribute work across machines.
Leave a Reply