#18Distributed Centralized Logging (ELK/Datadog)
18 / 20
Datadog logoDatadog System
MediumInfrastructure
30–45 min

Distributed Centralized Logging (ELK/Datadog)

Design a high-volume log aggregation pipeline that collects application stdout/syslogs across thousands of microservices, parses JSON structures, and supports real-time search.

Scale & Traffic10 TB logs/day · 100,000 events/sec peak
Monthly Cloud Budget$8,000/month
Latency Target (P99)< 50ms Realtime SLA
Availability SLA99.99% Multi-AZ Active

System Requirements

  • Reliable backpressure-safe log ingestion agents
  • Log indexing with configurable retention tiers (7 days hot, 90 days warm, 1 yr cold)
  • Real-time alerting triggers on error spikes
  • Full-text log search with filtering by service, trace ID, and log level

Graded Constraints

01Cost under $8,000/month
02Message queue buffer (Kafka) to decouple producers from indexing speed
03Log processing / enrichment worker nodes
04Search Engine cluster for fast querying
05Object storage for cold log archival

Ready to design?

Open this challenge in the interactive system design playground and satisfy all architecture constraints.

Explore More

Related Architecture Challenges

All Problems
Distributed Centralized Logging (ELK/Datadog) — System Design Problem | Mimir Nest