How To

Building Your First Observability Stack with Open‑Source Tools

Jigar Bhatt

Jigar Bhatt

— 4 min read
Building Your First Observability Stack with Open‑Source Tools
Goal: Spin up Prometheus+Alertmanager, Loki+Promtail, Jaeger, and Grafana with a single docker‑compose.yml, then watch a tiny Java HTTP service emit metrics, logs, and traces—all in less than an hour.

Why this post?

You keep hearing that “observability ≠ monitoring” and that you need metrics, logs, and traces working together. But SaaS bills and steep learning curves can be buzz‑kills. This guide is a hands‑on shortcut: everything runs locally in Docker, so you can poke around, break things, and learn risk‑free.

What is Observability? – Metrics, Logs, and Traces Demystified
Observability is the ability to understand the internal state of a system by analyzing its outputs, such as logs, metrics, and traces.
Observability.Jigar Bhatt

Prerequisites

  • Docker & docker‑compose installed
  • Basic docker compose up / docker compose down familiarity
  • JDK 21+ and Maven

1 – Scaffold a Tiny Java Service

Create a folder tiny-app/

mkdir tiny-app && cd tiny-app

1.1 pom.xml

1.2 TinyApp.java

1.3 Dockerfile

Build it:

docker build -t tiny-app:latest .

2 – Prometheus (+ Alertmanager) Config

2.1 prometheus.yaml

2.2 alert-rules.yaml

2.3 alertmanager.yaml

We’ll use a throw‑away “alert‑logger” container that simply prints POST bodies to stdout so you can observe alerts without email/SMS.

3 – Loki & Promtail Config

3.1 loki.yaml

3.2 promtail.yaml

4 – Docker Compose Config

5 – Fire It Up!

docker compose up -d

Give it ~30 seconds. Hit the demo app:

curl http://localhost:8080/hello
ab -n 200 -c 20 http://localhost:8080/hello   # generate some load

You should see “Hello” responses and, in the container logs, Handled /hello … log lines (Promtail picks them up).

6 – Exploring the Data

6.1 Grafana Login

  • Browser → http://localhost:3000
    User: admin Password: admin

Hit Connections → Data Sources → Add new data source and select each

  • Prometheus url → http://prometheus:9090
  • Loki url → http://loki:3100
  • Jaeger url → http://jaeger:16686

6.2 Quick Dashboards

Metrics panel: Choose the Prometheus data source with below query.

rate(tiny_hits_total[1m])

Logs panel: Choose the Loki data source with below query. 

{service_name="tiny-observability-stack-tiny-app-1"}

Traces panel: Choose the Jaeger data source and click “Search” pick tiny-app.

Three pillars, one pane of glass.

6.3 Alert in Action

Keep hammering /hello (e.g., ab above). When rate(demo_hits_total) exceeds 5 RPS for > 1 minute, Prometheus fires HighRequestRate. Check:

docker compose logs alert-logger

You should see a POST with JSON alert payload 👏.

Grafana also lists it under Alerting → Alert rules. (Feel free to wire Alertmanager to Slack/email later.)

7 – Under the Hood: What Just Happened?

  • OpenTelemetry Java agent auto‑instrumented Javalin, captured HTTP spans, and exported via OTLP → Jaeger.
  • Micrometer registers /metrics, Prometheus pulls every 15 s.
  • Promtail tails container stdout, ships to Loki.
  • Grafana unifies everything, and Prometheus Alertmanager paged us.

8 – Where to Go from Here

Next Step Why it’s Cool Link
Create a service map in Grafana with Tempo traces See cross‑service flow Tempo docs
Add Prometheus remote‑write to a cloud TSDB for long‑term storage Keep metrics for >30 days Prometheus docs
Enable Loki ruler for log‑based alerts Catch error patterns Loki alerts
Try Grafana k6 to load‑test and visualize in same stack Unified perf testing k6 docs
Swap Docker for Kubernetes  Closer to prod reality e.g. kube-prometheus-stack

9 – Wrap‑Up

You just:

  1. Built a tiny Java service
  2. Containerized it with auto‑instrumentation
  3. Launched Prometheus + Alertmanager, Loki + Promtail, Jaeger, and Grafana in one shot
  4. Watched live metrics, logs, traces, and an alert—all open source, no credit card, ~200 lines of YAML.

Pat yourself on the back 👏. From here, you can iterate: add more apps, define SLOs, integrate CI/CD. Remember: observability isn’t a destination; it’s a practice. But now you’ve got a solid starter toolbox—happy hacking!

Source code: https://github.com/ObservabilityHow/tiny-observability-stack

Read more