A web crawler sounds simple in theory: visit a page, grab the links, follow them, repeat. But the internet has billions of pages, and naive crawling will get you into trouble fast. You will hit the same URLs over and over, overwhelm web servers with too many requests, fall into infinite URL traps, and burn through bandwidth fetching duplicate content. The key insight is that a crawler is really a **data pipeline** -- a series of processing stages connected by queues, where each stage can be scaled independently. Once you see it that way, the design becomes a matter of choosing the right data structures for each stage and connecting them cleanly. ## The Crawler Loop At its core, a crawler is a loop with four stages. Seed URLs (starting points like top-level domain pages or a sitemap) enter the **URL frontier**, which is a specialized queue that decides what to fetch next. The **fetcher** downloads the page via HTTP. The **parser** extracts structured data (text, metadata, images) and pulls out all hyperlinks. Those new URLs feed back into the frontier, and the loop continues. Each stage is connected by a message queue (Kafka, SQS, or similar). This decoupling is what makes the whole thing scalable. If fetching is the bottleneck, you add more fetcher workers without touching anything else. If parsing is slow because you are doing heavy NLP, you scale that stage independently. This is the same pattern you see in any well-designed data pipeline: independent stages, connected by queues, each with its own scaling knob. **See it in practice:** The [CI/CD](/learn/system-design/questions/cicd) question uses a nearly identical pipeline architecture -- build, test, and deploy stages chained through message queues with independent scaling and failure handling at each stage. The [LeetCode](/learn/system-design/questions/leetcode) question applies the same multi-stage approach: submissions flow through compilation, execution, and result comparison, each decoupled by queues. ## URL Frontier: Priority Meets Politeness The frontier is more than a simple FIFO queue. It needs to balance two competing concerns: **what** to fetch (priority) and **when** to fetch it (politeness). ### Priority Queue Not all URLs are equally valuable. A news site's homepage changes hourly and has high PageRank. A deep archive page from 2008 rarely changes. The frontier uses multiple priority levels so high-value URLs get fetched first. Redis Sorted Sets work well here -- the score is the priority, and ZPOPMIN gives you the next URL to fetch. How do you assign priority? A few signals: PageRank (how many other pages link to it), freshness (when was it last crawled), and domain importance (major news sites rank higher than personal blogs). You do not need a perfect ranking -- a rough tiering into 3-5 priority levels is enough. ### Per-Domain Politeness Fetching 1,000 pages from the same domain in one second will get you IP-banned and might take down a small site. The frontier maintains separate queues per domain and enforces a minimum delay between requests -- typically one to two seconds per domain. This means even if example.com has 10,000 URLs queued, you only fetch one per second from that domain. Meanwhile, you are fetching from thousands of other domains in parallel. The per-domain queues ensure you are a good citizen of the web. Before crawling any domain, always fetch and parse its robots.txt file. Respect Disallow directives (paths the site does not want crawled) and Crawl-Delay directives (minimum seconds between requests). Use a descriptive User-Agent string that identifies your crawler and provides contact information. **See it in practice:** The [Job Scheduler](/learn/system-design/questions/job-scheduler) question deals with a similar challenge -- prioritizing which jobs to execute next while respecting resource constraints and rate limits across tenants. The [Rate Limiting](/learn/system-design/design-patterns/rate-limiting) pattern covers the sliding-window and token-bucket algorithms that power per-domain throttling. ## Deduplication: Avoiding Wasted Work The same content often lives at multiple URLs. Mirrors, URL variations (/page vs /page/ vs /page?ref=home), syndicated content, and query parameter permutations can generate millions of duplicate URLs. Fetching and processing all of them is a massive waste of bandwidth and compute. ### URL-Level Deduplication Before adding a URL to the frontier, check whether you have seen it before. A **Bloom filter** is the standard tool here. It is a probabilistic data structure that can tell you "definitely not seen" or "probably seen" using very little memory. One billion URLs can be tracked in roughly 1.2 GB of memory with a 1% false-positive rate. The trade-off is worth it: you might occasionally skip a URL you have not actually seen (false positive), but you will never crawl a duplicate (no false negatives). For web crawling, occasionally missing one page out of a hundred is perfectly acceptable. ### Content-Level Deduplication Even after URL dedup, different URLs can serve identical content. To catch these, compute a content hash (MD5 or SHA-256) of each fetched page. Before processing, check whether you have already seen that hash. If you have, skip the page. This catches mirrors, syndicated articles, and URL variations that serve the same content. For near-duplicate detection (pages that are 95% identical but have minor differences like ads or timestamps), **SimHash** or **MinHash** provide fuzzy fingerprinting -- they produce similar hashes for similar content, letting you set a similarity threshold. **See it in practice:** The [Distributed Counters](/learn/system-design/questions/distributed-counters) question also faces a deduplication challenge -- ensuring the same event is not counted twice across distributed workers. The [Idempotency and Exactly-Once](/learn/system-design/design-patterns/idempotency-exactly-once) pattern covers deduplication techniques using idempotency keys and exactly-once semantics. ## The ETL Pipeline: Processing Crawled Data Crawling the web is only half the job. The raw HTML you fetch is messy -- full of ads, navigation chrome, JavaScript, and broken markup. You need an **ETL pipeline** (Extract, Transform, Load) to turn raw pages into clean, structured data that is actually useful. **Extract** parses the raw HTML and pulls out the fields you care about: title, body text, author, date, links, images. This is where you run an HTML parser (like BeautifulSoup or Cheerio) and apply CSS selectors or XPath expressions to find structured data. **Transform** cleans and normalizes the extracted data. This includes stripping HTML tags from text, normalizing dates to a standard format, deduplicating content (using the hashes from the previous section), running language detection, and potentially applying NLP for entity extraction or classification. **Load** writes the cleaned data to its final destination -- a search index (Elasticsearch), a data warehouse (BigQuery, Redshift), or a relational database. This stage handles batching writes for efficiency and managing schema migrations. Each stage has its own error handling. A page with broken HTML should not crash the entire pipeline -- it should go to a **dead letter queue** (DLQ) where you can inspect and retry it later. The DLQ pattern is critical in any data pipeline because you will always encounter malformed input at scale. **See it in practice:** The [Ads Click Aggregator](/learn/system-design/questions/adds-click-aggregator) question designs a full ETL pipeline for processing billions of ad click events -- extracting click data, transforming it into aggregated metrics, and loading results into a queryable store. The [Stream Processing and Real-Time Aggregation](/learn/system-design/design-patterns/stream-processing-realtime-aggregation) pattern covers how to build similar pipelines using Kafka Streams and Flink for real-time processing. ## Crawler Traps and Defenses Some websites, intentionally or not, generate infinite URL spaces. A calendar page that links to every future date. Session IDs in URLs that create a unique URL for every visit. Query parameters that produce endless combinations. These traps can consume your entire crawling budget on a single domain. A practical approach is to combine all of these. Set a default depth limit of 15-20 levels, cap each domain at 10,000-50,000 pages per crawl cycle, normalize all URLs before they enter the frontier, and monitor for domains that are generating unusually large numbers of new URLs. ## Scaling to Billions: Distributed Crawling A single crawler machine, no matter how well-optimized, cannot crawl the entire web. You need multiple crawler workers running in parallel, coordinating through a shared frontier. The key design decision is how to partition work across workers. **Domain-based partitioning** is the most common approach: each worker owns a set of domains (e.g., worker 1 handles *.com domains starting with A-G, worker 2 handles H-N, and so on). This has a crucial advantage -- since only one worker crawls each domain, you automatically enforce politeness without any cross-worker coordination. The shared URL frontier can be implemented as Kafka partitions (one partition per domain group) or as a Redis cluster. The shared Bloom filter needs to be accessible by all workers -- either centralized in Redis or implemented as a distributed Bloom filter with periodic synchronization. For fault tolerance, each worker periodically checkpoints its progress (last crawled URL, frontier state). If a worker crashes, the coordinator reassigns its domain range to another worker, which resumes from the last checkpoint. **See it in practice:** The [Yelp](/learn/system-design/questions/yelp) question involves crawling and indexing restaurant data from multiple sources, requiring deduplication and normalization across different data formats. The [Google Maps](/learn/system-design/questions/google-map) question processes massive geospatial data pipelines with similar partitioning and coordination challenges. The [Scaling Writes](/learn/system-design/design-patterns/scaling-writes) pattern covers the sharding and partitioning strategies that underpin distributed crawling, and the [Pub/Sub Messaging](/learn/system-design/design-patterns/pub-sub-messaging) pattern explains how Kafka partitions enable parallel processing across workers. ## Putting It All Together A production web crawler combines all of these pieces into a cohesive system: 1. **Seed URLs** initialize the frontier with starting points. 2. The **URL frontier** prioritizes what to fetch and enforces per-domain politeness. 3. A **Bloom filter** deduplicates URLs before they enter the frontier. 4. **Distributed fetcher workers** download pages in parallel, partitioned by domain. 5. The **parser** extracts structured data and new URLs from each page. 6. An **ETL pipeline** cleans, transforms, and loads the data into its final store. 7. **Dead letter queues** capture failures for retry and investigation. The beauty of this architecture is that each component can be developed, tested, and scaled independently. When you need to crawl faster, add more fetcher workers. When storage fills up, upgrade the storage tier. When you need to handle new content formats, update the parser without touching the fetcher. This is the power of pipeline-oriented design.