A low quality rap done one night on Zoom after a brutal on-call shift. Rapping is always the best way to burn off energy, even a little LQ one over zoom like this. 😉
[Intro]
Take the graphs, low, low, low, low-er…
(You don't know what you're doin'— doin'— doin'—)
Scary ain’t it?
[Verse 1]
I took availability numbers and brought the thunder
You claim eleven nines is it any wonder
All the confusion, your uptime is an illusion
InGraphs are bruisin’, leakin’ red like a contusion
Business and service alerts come together in a fusion
The impact to our revenue is going to be huge and
NOCs calling on-call, keep hitting a ten-ton wall
Monitors lit up through the hall, watching every host fall
Logs overflow with stack traces, times with no zones, all numbers are baselеss
No unit tests over our edgе cases, but ramp plans feel like track races
Hitting LiX with no concern like I already hit up licks
Operational Excellence that’s the way this gets fixed
Add the missing metrics, attaching runbooks to alerts
While we keep on earning like I’m living next to Burt
I’m not a warden, but there’s nothing more important
Flip the script for our devs and then properly reward ‘em…
[Interlude]
(You don't know what you're doin'— doin'— doin'—)
Let’s do the math
[Verse 2]
We’re here to run a business that generates billions
Every second we’re not serving ads we’re losing the Ms
One million, two million, three million, four
Mitigate the issue before we lose any more
A thousand CPUs maxed right out to the core
So many screams from Meeseeks it’s like I’m screaming “fore”...
[Interlude]
(You don't know what you're doin'— doin'— doin'—)
Undetected
[Verse 3]
I came into this on-call with alerts out of control
Graphs shooting up like GameStop, for no reason at all
I call them “leaves in Autumn” and like to watch them fall
I got an eye on Ion to prevent the IRIS calls
They tell me, “rebuild the engine while the plane is still in flight”
As I sit here red-eyed fighting fires, up all night
My latency budget is measured down to the milli
And when the charts jump up and wail, I call them Free Willy
One million, two million, three million, four
QPS starts stacking, here comes a million more
Why do you think I’m the one they’re calling for?
The dashboard’s bleeding red, every node a pool of gore
Grab my keyboard, do my magic, then the graphs hit the floor
The red goes straight down like the wine I’m 'bout to pour
[Interlude]
(You don't know what you're doin'— doin'— doin'—)
That's where you're wrong
[Verse 4]
Train the on-calls, primary and secondary
Kill the backlog even the P2s get buried
Alerts calm down so there's no further warnings
Metrics so important like if they all came with Mortys
Now we've cleared the thunder, it's time for Summer
Turning bricks to coins like Nintendo's #1 plumber
Operational Excellence (The Partial Health Song) was written by Adam Wolkov.
AWOL (Nerd) released Operational Excellence (The Partial Health Song) on Mon Oct 17 2022.