I came up with some gnarly Prometheus queries that would pull data over a time range and would be more sensitive to stale data. When testing the query, a host was showing as down. I went over to one of the senior admins and asked “Hey, is host X down?”. He tried to SSH to the box: connection timed out. We had a short term fix for our metrics stack. My first win. We could catch errors earlier, our on-call people had something more responsive. My lead was happy, and I set out to figure out migrating to node_exporter and deploying it with Puppet. I was excited to have solved a problem, I had tackled the technical side of it.

But I had not thought about the structural side.

I was told I’d come back to the metrics work once we’d handled other priorities. I never would.

00: Introduction

So it goes, days turn to weeks turn to months. You’re told you’re appreciated, how you’re crucial, you’re the go-to person for the infrastructure, or for your IaC, or for the observability stack, or for system troubleshooting (or maybe multiple of these)… It feels good. You feel valuable. Then more work is piled on. You become the person that’s trusted to handle it all; or if you’re lucky, there’s a few others that take on the huge workload with you, but you’re all spread too thin.

I dealt with that in my first DevOps role — one I took on last year and finally managed to leave. The work was modernizing a legacy system’s infrastructure and transitioning it from VMWare to AWS. AWS was new to nearly the entire team, including myself. Part of this was my personality: I hate saying ‘No’, I want to help my team. Good traits to have in a teammate, I think. But also the traits that burnout-prone environments hunt for.