Software Development

Every developer has lived through this scene: the adrenaline spike when a production incident is announced. That mix of dread about what you are going to find and frenzy to collect any piece of information that will let you understand and then fix the problem. It happened to me again a few days ago. I can still picture myself rushing to the logs, and I still remember the frustration of finding nothing but basic information and an unhelpful error message: “Unable to load cache”.

Sometimes an investigation tells a different story than the one you expected.

That’s what happened to me recently while investigating why a pod was ending up OOMKilled two to three times a day.

A quick look at the memory of the incriminated pod doesn’t show the typical rising curve of a memory leak. I’m missing data just before the OOM (because it’s always when your metrics system is migrating that this kind of incident happens), but with the day’s data, the cause seems to lie elsewhere.