System Requirements
- Frontier URL manager ensuring host-level politeness delays
- Content deduplication using 64-bit SimHash / MinHash fingerprints
- DNS caching layer to eliminate domain resolution bottlenecks
- Raw HTML blob storage + extracted link graph pipeline
Graded Constraints
01Budget: $25,000/month
02Distributed priority queue for URL frontier management
03Worker cluster for fetching and parsing HTML
04DNS cache server cluster
05Object storage for raw HTML snapshots
06NoSQL database for crawled metadata and document checksums
Ready to design?
Open this challenge in the interactive system design playground and satisfy all architecture constraints.
Explore More
All ProblemsRelated Architecture Challenges
AmazonMedium
Distributed Cache Cluster
Design a distributed in-memory key-value caching system supporting consistent hashing, replication, write-through/write-back policies, and LRU eviction.
1M operations/sec →
YouTubeHard
YouTube Video Recommendation Engine
Design YouTube's two-stage recommendation pipeline: candidate generation (filtering millions to hundreds) followed by ranking & scoring for personalized user home feeds.
500M Daily Active Users →
NetflixHard
Netflix Home Page Recommendation
Design Netflix's personalized row generation system that curates customized titles, tailored artwork thumbnails, and dynamic row rankings for every member profile.
250M paid subscribers →