Building Observability and What I Learnt

Prologue

I had always heard people talking about their observability stacks: Loki, Prometheus, Grafana and whatnot. It intrigued me, but I had never set one up myself, nor did I know much about it. In this blog, I share how I set up logging for a production backend and what I learnt along the way.

This isn’t a guide to setting up observability for your applications. But I’m pretty sure there’s something to learn from it nevertheless. So, here it goes.

What does logging actually mean?

Centuries ago, sailors estimated their speed using a wooden chip attached to a knotted rope. Counting the knots passing over a measured interval gave them their speed, hence nautical “knots.” NOAA explains the history (opens in a new tab). Measurements and events went into ship’s logs so people could understand the voyage afterward. That’s literally what log means in software, a record of what happened so we can investigate later.

Now back to our application, a typical log stream for our backend might look like this:

01  11:08:21.104  INFO   Warehouse product sync daemon started
02  11:08:21.286  INFO   HTTP request completed
03  11:08:21.519  ERROR  Redis client error
04  11:08:21.744  WARN   Skipping product sync; previous run in progress
05  11:08:22.038  ERROR  Scheduled warehouse sales order sync crashed
06  11:08:22.315  INFO   Scheduled warehouse product sync finished

Something failed. Okay, but now what?

• What exactly failed, beyond the job’s name?
• Why did it fail?
• Which users or clients were affected?
• What should I check or fix, and is retrying safe?

Structured logging is the answer to this. It puts that context into named fields. Take line 05: if the database connection closed while loading sync sources, a useful error record could look like this:

ERROR Scheduled warehouse
      sales order sync crashed
|
+-- time: 11:08:22.038
+-- service: SERVICE_NAME
+-- component: SYNC_WORKER
+-- version: RELEASE_TAG
+-- durationMs: DURATION_MS
+-- err:
|   type: Error
|   message:
|     Connection terminated
|     unexpectedly
|   stack: STACK_TRACE
+-- why:
|   Database connection closed
|   while loading sync sources.
`-- fix:
    Inspect the original error.
    Check database and ERP health.
    Reconcile partial work before
    retrying the job.

What I wanted

• The whole stack should be self-hostable
• Can track backend API errors and performance
• Can track backend server health and data stores (Postgres and Redis)
• Can identify slow queries

So, I picked out the following to implement all of that:

Tool	What it does here
Terraform	IaC tool to define, build, change, and version resources using declarative configuration files.
Pino	Produces structured application logs.
Alloy	Collects and forwards logs and measurements.
Loki	Stores and searches log history.
Prometheus	Stores measurements over time.
Grafana	Displays dashboards, explores logs, and evaluates alerts.
Exporters	Expose host, PostgreSQL, and Redis statistics.
Blackbox Exporter	Checks whether endpoints respond.
Discord	Receives alert notifications.

A brief overview of the observability data flow:

Logs -> Alloy -> Loki
                  |
                  v
               Grafana
                  ^
                  |
Metrics -> Prometheus

A separate monitoring server

Imagine the production server is down. So is all the monitoring that could have helped diagnose the issue. A separate server keeps logs off the primary one, so they don’t pile up there over time and they’re still available when it’s down.

But now I need to manage two servers, and the cost just went up. However cost aside, managing more than one server is easy with Terraform. It lets me manage the monitoring server without going to AWS Lightsail and clicking around to make things work.

Manual changes
  Click -> change -> remember?

Terraform
  Describe -> plan -> review
                 |
                 v
              apply -> verify

Grafana, behind Traefik, is the only thing on that monitoring server which is reachable from outside. Loki, Prometheus and the rest sit on a private network inside it.

Making logs useful

Every incoming request gets a request ID. Logs within that request pick it up through request-scoped context, so I don’t have to pass it around everywhere. When the request finishes, a log records its status and duration. Successful health checks are skipped, otherwise the logs would be full of “still alive.”

Logs also carry who made the request, workers record how long they took, and errors come with a short why and fix. This borrows from wide events (similar to Stripe’s canonical log lines (opens in a new tab)): fewer, richer records. Errors take a cue from Problem Details for HTTP APIs (opens in a new tab), where an error explains what went wrong and how to correct it.

Getting logs to Loki

Now those logs had to get to the monitoring server.

APPLICATION SERVER
  Pino -> stdout -> Docker logs
                         |
                         v
                       Alloy
                         |
                     HTTP push
                  private network
                         |
                         v
MONITORING SERVER
                        Loki

Pino only writes to stdout stdout Short for standard output, the default channel a program writes its normal output to. In a terminal it shows up on screen, but it can just as easily be redirected to a file or piped into another program. . Docker keeps that output, and Alloy, running next to the backend, reads it through the Docker socket Docker socket A Unix socket file, usually /var/run/docker.sock, that the Docker daemon listens on for API requests. The docker command talks to Docker through it, and anything with access to it can manage every container on the machine. . It keeps just the API and worker logs, batches and compresses them, and sends them as plain HTTP POSTs to Loki’s push API (opens in a new tab). So the backend has no idea monitoring exists, and if Loki goes down, requests still work.

Data only goes one way: the application server pushes, so nothing on it has to listen for the monitoring server. The two talk over Lightsail’s private network, where Loki and Prometheus listen only on the monitoring server’s private address, and its firewall lets in only the application server. A good analogy would be two locks on the same door, so if I get one wrong, the other still holds.

Alloy saves where it stopped reading in a volume, so a restart doesn’t resend or skip logs. If Loki is down, it retries for a while, not forever, and Docker only keeps a few small log files per container (more on rotation below), so a long outage leaves a gap.

Tracking metrics with Prometheus

Logs are good for what happened, but if memory has been creeping up for the last hour, I want a number tracked over time, and that’s where Prometheus comes in.

Node Exporter exposes host numbers: CPU, memory, disk, network, and uptime. Alloy scrapes it, which just means asking for the current numbers every few seconds, and pushes them to Prometheus over the same private connection, with the same kind of batched HTTP push, called remote write remote write Prometheus's protocol for sending metrics from one system to another. Instead of waiting to be scraped, the sender batches samples, compresses them, and POSTs them over HTTP to a receiver that stores them. Read the docs (opens in a new tab) . Prometheus normally does the asking itself, so it had to be told to accept pushes too.

APPLICATION SERVER
  Docker logs ----+
                  |
  Node Exporter --+--> Alloy
                         |
                   private, push
                         |
MONITORING SERVER        v
          +--------------+
          |              |
        logs          numbers
          v              v
        Loki         Prometheus
          |              |
        LogQL          PromQL
          |              |
          +-> Grafana <--+

Logs vs Metrics

Okay, so this is where the difference finally clicked for me. Loki keeps the logs and Prometheus keeps numbers over time, while Grafana doesn’t store any of that data itself. It just queries both to show me the results, and runs the alert rules on top. One is built for searching through text and the other for doing math on numbers, which is why they’re separate tools.

LogQL (opens in a new tab) is for querying logs: find failed operations, pull out durations, count matching events. PromQL (opens in a new tab) is for metrics: is memory going up, how fast is a counter growing?

Both of them can draw graphs though, and that tripped me up for a while. You can build a graph of error counts straight from logs without it ever touching Prometheus, and a few of the performance panels here are actually built from log fields.

Then I had to make sure the new disk didn’t fill up. Loki has time-based retention, and Prometheus has time and size limits. Docker also needs log rotation on the application server, since forwarding logs doesn’t delete Docker’s local copy. Keep in mind that just changing the config doesn’t apply to existing containers. Those containers need to be restarted. (Docker’s rotation documentation (opens in a new tab))

Probing the backend

Everything so far comes from the application server itself. If the backend is running but nobody can reach it, it can happily keep logging that everything is fine. A wrong DNS record, a blocked port, an expired certificate or a broken proxy all look like this: the requests never arrive, so there’s nothing in the logs to notice.

MONITORING SERVER
       Blackbox Exporter
               |
       DNS -> public IP
               |
               v
APPLICATION SERVER
   firewall -> Traefik (TLS)
               |
               v
       Backend endpoints
               |
       up? status? time?
               |
               v
MONITORING SERVER
     Prometheus -> Grafana
               |
               v
            Discord

Blackbox Exporter on the monitoring server probes the backend’s public endpoints. On Prometheus’s behalf, it sends a real HTTPS request and reports whether it worked, the status code, and how long it took. It has to run on the other machine, so the request takes the same public route a user’s would. From the application server it could just go through localhost and skip every part that tends to break, which is just the app checking itself. It only probes once a minute, so the backend’s own logs don’t fill up with probe requests.

Monitoring the monitoring server

So the monitoring server needed the same treatment, since its disk can fill up just as easily as the main one. It runs its own Node Exporter, which Prometheus pulls from directly, and its own Alloy for its container logs.

Application host -> Node Exporter
                         |
                    Alloy (push)
                         |
                         v
                     Prometheus
                         ^
                         |
                       (pull)
                         |
Monitoring host -> Node Exporter

Here Prometheus does two jobs: it receives what the application server pushes, and it also pulls from its own server.

The bottleneck isn’t always the backend

A server can have plenty of free memory while users are still waiting. Sometimes the slow part isn’t the backend at all, it’s something the backend is calling.

So every outgoing call, to the ERP, the SMS service, and Google Directions, logs its own completion record: which integration, which operation, status, outcome, and duration. No URLs or payloads. These sit next to the incoming request logs on one dashboard, over the same time range, so a slow request can be compared with whatever it was waiting on.

Client -> Backend -> Dependency
             |
             v
     Backend records both:
     request + dependency timing
             |
             v
        Loki -> Grafana

Incoming request timing stops when the server is done sending the response, even if a computer is still receiving it. Outgoing timing runs until the response has been read and validated, otherwise a dependency could look fast just because its headers came back quickly.

Watching the databases

The server can look perfectly healthy while the database struggles, like PostgreSQL running out of connections or getting stuck on locks, or Redis running low on memory, evicting keys or having trouble persisting.

PostgreSQL --+
             |
Redis -------+
             |  localhost
             v
Alloy (built-in exporters)
             |  remote write
             v
Prometheus -> Grafana

Alloy, already running next to the backend, has PostgreSQL and Redis exporters exporters A component that reads a system's internal statistics and serves them as metrics in a format Prometheus understands. Most databases and services have one. Read the docs (opens in a new tab) built in, so there are no extra containers to run. Every minute it connects to each database over localhost, asks for its current statistics, turns the answers into metrics, and remote-writes them to Prometheus with the host metrics. Nothing new is exposed, and neither database is changed or restarted.

Each connection uses its own restricted monitoring account. On PostgreSQL that’s a few read-only queries against system views like pg_stat_database pg_stat_database A PostgreSQL system view with one row per database, counting commits, rollbacks, cache hits and disk reads, deadlocks, and temporary file usage since the statistics were last reset. Read the docs (opens in a new tab) and pg_stat_activity pg_stat_activity A PostgreSQL system view with one row per connection, showing what each session is doing right now: its state, what it's waiting on, and when its current query started. Read the docs (opens in a new tab) , with short timeouts: connections against the limit, transactions, cache hits, deadlocks deadlocks Two transactions each holding a lock the other one needs, so neither can finish. PostgreSQL detects this and cancels one of them with an error. Read the docs (opens in a new tab) , lock waits lock waits Time a query spends paused because another transaction holds a lock on a row or table it needs. , and the oldest running query. Redis’s account can only run commands like INFO, SLOWLOG and LATENCY, so it never touches a key, but that covers memory, evictions evictions Redis deleting keys to free memory once it reaches its configured memory limit. Which keys go depends on its eviction policy. Read the docs (opens in a new tab) , blocked and rejected clients, cache hits, AOF AOF Append-only file. Redis appends every write command to a file on disk, so the data can be rebuilt by replaying them after a restart. Read the docs (opens in a new tab) and RDB RDB Redis Database file. A point-in-time snapshot of the whole dataset, written to disk at intervals. Read the docs (opens in a new tab) persistence, and the slow log slow log Redis's built-in, in-memory list of commands that took longer than a configured time to run, along with how long each took. Read the docs (opens in a new tab) . Evictions, failed persistence and PostgreSQL connections above 80% each have an alert.

Slow queries

Setting log_min_duration_statement log_min_duration_statement A PostgreSQL setting that logs every completed statement taking at least this long, along with its duration. -1 turns it off and 0 logs every statement. Read the docs (opens in a new tab) to one second makes PostgreSQL log every statement that takes that long, with its duration and SQL, and log_parameter_max_length = 0 leaves bound values bound values Values sent separately from the SQL text in a parameterized query, filling placeholders like $1 and $2 instead of being written into the SQL itself. out. Alloy picks those lines up from the Docker logs, keeps only completed slow queries, and masks values still written into the SQL before they leave the server:

before: WHERE status = 'paid'
after:  WHERE status = '[REDACTED]'

If a query is too tricky to mask safely, Alloy swaps the SQL for a “withheld” note and keeps just the duration. Duration and SQL are stored inside the record, not as labels labels Key-value pairs that Loki indexes to group logs into streams, like service or environment. Every unique combination creates a separate stream, so labels should have only a few possible values. Read the docs (opens in a new tab) , so each query doesn’t get its own Loki stream. This only shows time spent in the database, not the whole request. And masking isn’t perfect, so access to these logs stays restricted.

Alerts on Discord

A good alert tells user what went wrong and where to look. It shouldn’t dump raw request data into chat. I started with availability, missing metrics, disk usage and application errors. Picking thresholds took some tweaking, because if every short spike pinged me I’d stop reading the channel pretty quickly.

Loki / Prometheus
        |
        v
Grafana evaluates rules
        |
        v
Group and route by category
        |
  +-----------+-----------+
  v           v           v
availability  infra       application
        (Discord channels)

Grafana runs the alert rules against Loki and Prometheus, groups related alerts together and sends each one to an availability, infrastructure or application channel.

Backups

Database backups were already running every night to off-server storage. What was missing was knowing when they didn’t work. So I wanted two checks: did the last backup fail, and is the last good one getting too old?

Backup job
  -> status file
     (last run ok? last success)
  -> Node Exporter
  -> Alloy
  -> Prometheus
  -> Grafana alert rules
  -> Discord

When a run finishes, the backup job writes a tiny status file. Node Exporter’s textfile collector already reads files like this, so backup status travels the same path as host metrics with nothing new to run. The file is written to a temporary copy and renamed into place, so it’s never read half-written, and a failed run keeps the old success time, so yesterday’s backup can’t pass as today’s.

The full picture

Put together, it looks like this:

APPLICATION SERVER
  Backend + workers -> Pino logs
  PostgreSQL / Redis -> stats
  Host -> Node Exporter
  Existing backups -> status
             |
             v
           Alloy
             |
     private, push only
             |
             v
MONITORING SERVER
  Logs -----------> Loki
  Measurements ---> Prometheus

  Own logs -> local Alloy -> Loki
  Own host -> Node Exporter
                  |
                  v
              Prometheus

  Blackbox probes -> endpoints
       results ---> Prometheus

  Loki + Prometheus
          |
          v
       Grafana -> Discord
          ^
          |
  Login <- HTTPS <- Browser

On the security side, Grafana behind a login is the only thing exposed, ingestion stays private, and the collectors only get the access they need and only send what’s needed. Traefik and Let’s Encrypt handle HTTPS with certificates kept in persistent storage, and secrets stay out of the repo.

What didn’t go according to the plan

The static IP is still stuck in a support ticket. Lightsail has a default static IP limit per region, so I requested an increase since upon reaching the max limit 5 on Sep 23. Last response was Oct 12 and till then we are waiting for the increment. Hopefully, someone from AWS may see this and could fast forward this :)

Getting the Discord alerts right took four tries. The first version linked to Grafana’s settings page, the next one to the alert’s details (better, but still no logs), then application alerts started opening the matching logs with the right filters and time range, and now the message itself shows the actual error and how many times it happened. Finally!

“Log everything” sounds nice until the logs eat your storage and drown the useful lines in noise. There’s no rule book. It’s mostly trial and error, aimed at keeping what the team needs to debug, so nobody is hunting for a needle in a haystack on a Friday evening.

Epilogue

So, is observability done? No. There’s still distributed tracing, business-level checks, detecting when workers go quiet or restart, uptime checks from outside, checking that retention actually works, and tuning alerts around what affects users. Plus the boring parts, like testing log delivery during outages and reviewing access regularly.

Looking back, my “all best practices followed” plan was written before I even knew what to ask. But I did end up with a setup I can actually navigate, and with a better way of working with the agent, where I ask what each piece is for and follow the data through it, then review the change and check it works before moving on. The agent was always going to be faster than me, so making sure I actually learned along the way was the goal here.

Next, I want to work on disaster recovery: what happens after the alert, when something actually has to be rebuilt. Restoring backups, what order to bring things back in, which config and secrets I need, and how much downtime or data loss is okay. Till then, this is it.