Prometheus Pull vs Push: A Tradeoff, Not a Moral Position
Scrapes, service discovery, Pushgateway, and OpenTelemetry collectors: when pull is the operational win, and when short-lived jobs need push.
Tag
20 articles · 68 min of reading
Scrapes, service discovery, Pushgateway, and OpenTelemetry collectors: when pull is the operational win, and when short-lived jobs need push.
bpftrace, BCC, and production agents: kprobes versus tracepoints, map memory, and why CO-RE beat copying kernel headers into the cluster.
How Elasticsearch ingest pipelines transform documents on the way in, and how Index Lifecycle Management rolls data from hot to delete without custom jobs.
How to interpret width versus height, distinguish on-CPU from off-CPU, and use allocation flames to catch the copy you cannot see in the source.
Always-on JFR: which event templates belong in prod, how to dump recordings on a stall, and what JFR will not tell you that async-profiler will.
Sigelman et al. on Dapper: trace trees, sampling, and the production constraints that still shape OpenTelemetry and Zipkin.
What real-user monitoring actually samples, why synthetics still catch the outage in a region with no traffic, and how to budget both without two sources of truth.
traceparent, baggage, SDK propagators, and the one missing middleware that turns a mesh into a pile of orphan spans.
What to actually instrument in an LLM application beyond latency and error rate, and how to close the loop from production traffic to eval sets.
How to build an eval suite that catches LLM application regressions before deploy, from labeled datasets to LLM-as-judge pitfalls.
Why average response time hides the experience of your worst-served users, and how to measure, diagnose, and reduce tail latency in production systems.
How Facebook built Scuba, an in-memory analytics engine that trades storage cost and precision for sub-second answers engineers need while debugging live incidents.
How to set service level objectives and error budgets that drive real engineering decisions, instead of becoming a number nobody looks at after launch.
Logs, metrics, and traces are usually taught as separate pillars, but the real value comes from how they connect during an actual incident investigation.
How Uber built M3, its open-sourced metrics platform, when off-the-shelf time-series tooling couldn't handle its monitoring scale.
How Micrometer and OpenTelemetry fit together in a Spring Boot service, and which metrics and traces are worth instrumenting by hand versus for free.
How Netflix's Atlas telemetry platform handles dimensional time-series data at a scale where most off-the-shelf monitoring systems fall over.
Why Stripe engineers emit a single structured log line per request instead of many scattered ones, and how it changes debugging at scale.
How GitHub built the Scientist library to run new and old code paths side by side in production before ever trusting the new one.
Which Spring Boot Actuator endpoints earn their place in production, and the security configuration that keeps them from becoming an attack surface.