Customers ask us every week: “My app crashed last Tuesday, but I can’t find the logs. How do I set up centralized logging?” The answer depends on your scale, your budget, and how many servers you’re wrangling. We’ve deployed all the major options — here’s what actually works.
Why Centralized Logging Matters
When logs live on individual servers, troubleshooting means SSH-ing into each box, grepping through gigabytes of text, and hoping the log wasn’t already rotated away. Centralized logging puts everything in one place with search, filtering, and alerting.
We covered syslog-ng as a transport layer in a previous post. This comparison focuses on the full stack: collection, storage, search, and visualization.
The Comparison: 10 Logging Stacks
| Tool | Best For | Min RAM | Query Language |
|---|---|---|---|
| Grafana Loki | Lightweight log aggregation | 1 GB | LogQL |
| Elastic Stack (ELK) | Full-text search at scale | 8 GB | KQL / Lucene |
| OpenSearch | ELK alternative (AWS fork) | 8 GB | DQL / SQL |
| Graylog | Log management with alerting | 4 GB | Graylog query |
| Promtail | Log shipper for Loki | 64 MB | — |
| Fluent Bit | Lightweight log forwarder | 32 MB | — |
| Fluentd | Flexible log routing | 256 MB | — |
| Vector | High-performance log pipeline | 64 MB | VRL |
| Filebeat | Elastic log shipper | 128 MB | — |
| rsyslog | System log forwarding | 32 MB | — |
Grafana Loki: Logs Without the Cost of Elasticsearch
Loki is to logs what Prometheus is to metrics. It indexes only labels (not full text), which makes it dramatically cheaper to run than Elasticsearch. If you’re already using Grafana for dashboards, Loki slots in naturally.
Why we recommend it for most teams:
- Uses the same label system as Prometheus — correlate metrics and logs in one Grafana dashboard
- Stores logs in object storage (S3, MinIO) or local filesystem — no JVM heap tuning
- LogQL is intuitive:
{job="nginx"} |= "500" | json | response_time > 2 - Fraction of the RAM cost of Elasticsearch
Where it falls short:
- No full-text indexing — searches by label are fast, grep-style searches on content are slower
- Less mature than ELK for complex aggregations and analytics
- Fewer pre-built integrations than Elasticsearch
Quick setup with Docker Compose:
version: "3.8"
services:
loki:
image: grafana/loki:latest
ports:
- "3100:3100"
volumes:
- loki_data:/loki
command: -config.file=/etc/loki/local-config.yaml
promtail:
image: grafana/promtail:latest
volumes:
- /var/log:/var/log:ro
- ./promtail-config.yml:/etc/promtail/config.yml
command: -config.file=/etc/promtail/config.yml
volumes:
loki_data:Elastic Stack (ELK): Full-Text Search Power
Elasticsearch + Logstash + Kibana is the industry standard for log analytics. It indexes every word in every log line, enabling complex queries, aggregations, and visualizations that Loki can’t match.
When you need it:
- Security/compliance — need to search arbitrary fields across millions of log lines
- Complex analytics — aggregations, anomaly detection, machine learning on log data
- Existing Elastic ecosystem (APM, SIEM, Uptime)
The cost:
- Memory-hungry — Elasticsearch alone wants 8-16 GB RAM in production
- JVM tuning required — heap size, garbage collection, shard management
- Operational complexity — rolling upgrades, shard rebalancing, index lifecycle
- Licensing changed — newer versions use SSPL (not open source). OpenSearch is the open fork.
OpenSearch: The Open-Source ELK
After Elastic’s license change, AWS forked Elasticsearch into OpenSearch. It’s API-compatible with Elasticsearch 7.x, fully open source (Apache 2.0), and includes OpenSearch Dashboards (the Kibana fork).
If you need ELK’s capabilities but want true open source, OpenSearch is the move. Same resource requirements, same operational overhead, but no licensing surprises.
Graylog: The Middle Ground
Graylog sits between Loki’s simplicity and ELK’s power. It uses Elasticsearch (or OpenSearch) as a backend but adds its own web UI, alerting, and pipeline processing. Easier to operate than raw ELK, more powerful than Loki for search.
- Built-in alerting and correlation
- Pipeline rules for parsing and enriching logs
- Role-based access control out of the box
- Needs MongoDB + Elasticsearch — three services minimum
Log Shippers: Getting Logs from A to B
Your logging stack needs an agent on each server to collect and forward logs. Here’s how the shippers compare:
| Shipper | Pairs With | RAM | Strength |
|---|---|---|---|
| Promtail | Loki | 64 MB | Native Loki labels, service discovery |
| Fluent Bit | Any (Loki, ES, Splunk) | 32 MB | Smallest footprint, C-based |
| Vector | Any | 64 MB | Fastest throughput, Rust-based, VRL transforms |
| Filebeat | Elasticsearch | 128 MB | Best Elastic integration, modules |
| Fluentd | Any | 256 MB | 1,000+ plugins, flexible routing |
| rsyslog | Any syslog receiver | 32 MB | Already installed on most Linux systems |
Our recommendation: Promtail for Loki stacks, Vector for everything else. Vector’s VRL (Vector Remap Language) lets you parse, filter, and transform logs before they hit storage — reducing storage costs and query noise.
How to Choose: Decision Tree
| Scenario | Recommended | Why |
|---|---|---|
| Already using Grafana + Prometheus | Loki + Promtail | Same ecosystem, correlated dashboards |
| Need full-text search + analytics | Elastic Stack or OpenSearch | Index everything, aggregate anything |
| Security/SIEM use case | OpenSearch or Graylog | Built-in alerting, correlation |
| Minimal resources available | Loki + Fluent Bit | Under 2 GB total RAM |
| Mixed destinations (multiple backends) | Vector or Fluentd | Route logs to multiple outputs |
| Simple syslog forwarding | syslog-ng or rsyslog | Already installed, proven |
Hosting Requirements
| Stack | CPU | RAM | Storage |
|---|---|---|---|
| Loki + Grafana (small) | 2 cores | 2 GB | 50 GB SSD |
| Loki + Grafana (production) | 4 cores | 4 GB | 200 GB SSD |
| Elastic Stack (small) | 4 cores | 8 GB | 100 GB SSD |
| Elastic Stack (production) | 8 cores | 16 GB | 500 GB+ SSD |
| Graylog | 4 cores | 8 GB | 100 GB SSD |
Loki runs comfortably on a Cloud VPS with 4 GB RAM. For Elastic Stack or Graylog at production scale, a dedicated server avoids the I/O contention that kills Elasticsearch performance on shared storage.
Storing logs for compliance? Our infrastructure is SOC 2 Type II certified with Canadian data residency — your logs stay in Canada.
Retention and Storage Tips
- Set retention from day one — logs grow fast. 30-90 days is typical; compliance may require longer.
- Compress and tier — Loki supports filesystem and S3 backends. Move old logs to cheaper storage.
- Filter at the edge — use Vector or Fluent Bit to drop debug/info logs before they hit storage. Most teams only need warn+ in long-term storage.
- Separate hot/warm/cold — Elasticsearch Index Lifecycle Management (ILM) automates this.
Need help designing a logging architecture? Our Managed Support team can plan retention policies, set up alerting, and ensure your logging stack meets compliance requirements.
Pair with Monitoring
Logs tell you what happened. Metrics tell you how bad. The most effective setup pairs both: Prometheus alerts on a CPU spike, and you pivot to Loki to find the runaway query that caused it. If you’re running Docker containers, centralized logging is especially critical — container logs vanish when the container restarts.
Be First to Comment