Running a Game Day When You Only Have One Observability Tool

Last November, I watched a team of six engineers spend $18,000 on Gremlin licenses. Three months later, they'd run exactly one chaos experiment — a CPU stress test on a staging pod that nobody was monitoring. The experiment ran. Nobody noticed. The Gremlin dashboard showed a green checkmark. Great success, apparently.

The next month, their Redis cluster went down in production. Average time to detection: 23 minutes. Time to root cause: another 41 minutes. The $18K chaos platform didn't help because nobody had practiced using their actual observability tools under stress. They had chaos automation. What they needed was a game day.

Here's the take that gets me side-eyes at SRE meetups: most teams don't need a chaos platform. They need to run a half-day game day with whatever observability tool they already have. One afternoon of deliberate practice beats six months of unused chaos automation.

Why Game Days Beat Chaos Platforms for Small Teams

Chaos engineering platforms solve a real problem — at a certain scale. Netflix invented Chaos Monkey because they had thousands of services and needed automated, continuous fault injection. Gremlin and LitmusChaos productized that approach for enterprises with hundreds of engineers and complex service meshes.

But if you're a team of 4-30 engineers running 5-20 services? The bottleneck isn't automating chaos. It's building incident response muscle memory. And muscle memory comes from practice, not tooling.

A game day forces your team to do the things they'll need to do at 3am:

Find the right dashboard
Interpret the signals
Communicate what they're seeing
Coordinate a response
Verify the fix worked

No chaos platform teaches that. You have to do it. Repeatedly. Under pressure (simulated, but still pressure).

The unpopular opinion: I'd rather have an engineer who's run 10 game days with basic observability than one who's configured 50 automated chaos experiments they've never watched unfold. The first engineer knows where to look. The second one knows how to write YAML.

What You Actually Need to Run a Game Day

Here's the minimal setup:

One observability tool that shows multiple signals. You need errors, latency, and some form of infrastructure health in the same place. Bouncing between Sentry for errors, Datadog for metrics, and PagerDuty for alerts during a game day teaches bad habits — you'll do the same thing during a real incident. (This is why we built JustAnalytics as a single platform. Errors, APM, uptime, session replay — one dashboard, one mental model. But honestly, even a cobbled-together Grafana setup works if everything's on the same screen.) For teams tracking sales pipeline alongside infrastructure, VeloCalls applies similar "one dashboard" thinking to call center observability.

A failure injection method. This doesn't need to be fancy:

kill -9 on a process
tc for network latency injection on Linux
A feature flag that returns errors for 10% of requests
Literally pulling a network cable (I've done this)
AWS Fault Injection Simulator if you're on AWS and want something click-button

A facilitator. Someone who's not debugging — they're injecting failures, timing the response, taking notes, and calling "stop" when things go sideways. This person needs to stay out of the weeds.

A timebox. 2-4 hours is the sweet spot. Shorter and you don't get enough reps. Longer and people burn out.

A staging environment or low-traffic production window. Start in staging. Seriously. Your first game day will reveal that your staging environment doesn't match production in embarrassing ways. That's a finding, not a failure.

The Game Day Script That Actually Works

I've facilitated probably 30 game days across four companies. Here's the structure that works:

Hour 0-0.5: Setup and Context (30 minutes)

Gather the team. Remind everyone this is practice, not a test. Failures are good — we're here to find gaps.

Review:

What services are we testing?
What observability do we have? (Pull up the dashboards)
What's our hypothesis about what will break?
What's the rollback plan if we break something we can't fix?

Assign roles:

Facilitator (injects failures, takes notes)
Incident commander (coordinates response)
Investigators (debug the issue)
Scribe (documents timeline in a shared doc — honestly, this role often gets forgotten mid-chaos, and then you're reconstructing the timeline from memory during the retro, which is... less useful). If your team uses JustEmails for transactional notifications, you can test email delivery failures as a game day scenario too.

Hour 0.5-1.5: First Failure Scenario (60 minutes)

Start simple. Kill a service instance and see how long it takes to:

Detect something is wrong
Identify which service is affected
Understand the user impact
Remediate

The facilitator injects the failure without announcing what they broke. The team uses only their observability tools to figure out what happened.

Time everything. Write down what worked and what didn't.

After 15-20 minutes of investigation (or resolution, whichever comes first), do a hot debrief:

What signal told you something was wrong?
What signal was missing or hard to find?
What dashboard did you wish you had?

Hour 1.5-2.5: Second Failure Scenario (60 minutes)

Escalate the complexity. Good options:

Database connection pool exhaustion (harder to diagnose than crashes)
Latency injection on a downstream dependency
Memory leak simulation (let it build for 10 minutes before investigating)
Partial failure (50% error rate, not 100% — these are sneakier)

Same structure: inject, investigate, debrief.

Hour 2.5-3: Cascading Failure (30 minutes, optional)

If your team handled the first two well, try a cascading failure. Inject latency on a dependency, then while they're investigating, kill a separate service. This simulates real incidents where multiple things go wrong at once.

Most teams aren't ready for this on their first game day. That's fine. (I've been that facilitator who escalated too fast. Watching five engineers stare at dashboards in panicked silence for eight minutes is... educational, but not in the way you want.)

Hour 3-4: Retro and Action Items (60 minutes)

This is the most important part and teams always want to skip it. Don't.

Review the timeline. For each failure:

Time to detect
Time to identify root cause
Time to remediate
What signals were useful?
What signals were missing?
What dashboard changes would help?
What runbook gaps did we find?

Assign action items with owners and deadlines. Common outputs:

"Add an alert for X threshold" (due: next sprint)
"Create a dashboard for Y service" (due: before next game day)
"Write runbook for Z scenario" (due: 2 weeks)
"Fix staging environment parity issue" (due: next quarter) — this one always comes up, be realistic about prioritization

Schedule the next game day before you leave. Quarterly at minimum.

What Your Observability Tool Needs to Show You

During a game day, you'll quickly learn whether your observability setup is actually useful under pressure. Here's what you need to see in under 30 seconds:

Is something wrong right now? An overview that shows service health at a glance. Red/yellow/green is fine. Error rates, latency percentiles, and success rates for your critical paths.

What changed recently? Deployments, config changes, traffic spikes. The first question in any incident is "what changed?" If your observability tool doesn't correlate errors with deploys, you're debugging blind.

Who's affected? Not just "errors are up" but "errors on the checkout flow for users in EU." If you're running uptime monitoring, you should be able to see which endpoints are degraded and which are fine.

What's the dependency chain? When service A is slow, is it because service B is slow, or because the database is slow, or because the CDN is having issues? Distributed tracing or at least a service map makes this visible. Otherwise you're guessing.

What did the user experience? This is where session replay pays off. Seeing the actual user session — the rage clicks, the error messages they saw, the page that froze — turns "500 errors increased" into a story you can debug. JustAnalytics includes this in the same platform as APM and error tracking, which matters during game days because you're not tab-switching.

If your current setup doesn't show all five within a single interface, your game day will reveal that gap. Add it to the action items.

The Failures Worth Practicing

Not all failures teach the same lessons. Here's a prioritized list based on what actually happens in production:

High priority (run these first):

Service crash (immediate, obvious impact — good warmup)
Database connection exhaustion (common, non-obvious symptoms)
Downstream dependency latency (third-party APIs being slow)
Memory leak (slow-building, easy to miss until too late)

These four will teach your team 80% of what they need to know. Start here.

Medium priority (game days 2-4):

Disk full (surprisingly common, often missed by monitoring)
DNS resolution failures (rare but catastrophic)
Certificate expiration (should be caught by uptime monitoring, but test it)
Feature flag misconfiguration (if you use flags heavily)
Payment processing failures — if you're using VeloCards for card issuing, test what happens when authorization fails

Advanced (once the basics are solid):

Cascading failures across multiple services
Data corruption scenarios (read replicas out of sync)
Cache poisoning
Rate limiting under load
Browser automation failures — teams using JustBrowser for headless automation should test scraper resilience too

Skip the exotic stuff until you've nailed the basics. I've seen teams practice "Byzantine fault tolerance scenarios" when they couldn't even detect a crashed process in under 5 minutes. Walk before you run.

(Confession: I once spent a week designing an elaborate multi-region failover test, complete with automated traffic shifting and DNS cutover timing. The actual game day? We discovered nobody knew how to interpret the basic latency dashboard. The elaborate scenario never even ran.)

For teams also monitoring ad spend, ClickzProtect handles a different kind of failure detection — but the principle is the same. You need to see what's breaking before you can fix it.

What This Won't Give You

Honest section, because I don't want to oversell this.

Game days don't find every bug. They find the observability gaps and the incident response gaps. They don't find the subtle race condition that only manifests under specific traffic patterns. For that, you need actual chaos engineering running continuously in production. But that's a tool for teams who've already mastered the basics.

Game days don't scale to hundreds of services. If you have 200 microservices, you can't manually inject failures in each one. At that point, automated chaos makes sense. But most teams reading this don't have 200 services. They have 8-15, and they're not confidently responding to incidents in any of them.

Game days don't replace monitoring. If your observability is fundamentally broken — no error tracking, no latency metrics, no uptime checks — a game day will just confirm that you can't see anything. Fix the basics first. JustAnalytics bundles analytics, errors, APM, session replay, uptime, and logs in one under-5KB script, which is the minimum viable observability setup. Whatever you're using, make sure it's actually collecting data before you try to interpret it under stress.

Game days don't build culture on their own. If your organization penalizes people for outages, game days will feel like tests, not learning. The facilitator matters — their job is to make failures feel safe. If that's not possible in your org, you have a bigger problem than observability.

And look — this is frustrating to write because I know some of you are nodding along thinking "yes, blameless culture, we have that" while your last incident retro included the phrase "who approved this deployment." But I digress.

The ROI Argument for Your Manager

If you need to justify the time:

A 4-hour game day with 5 engineers costs 20 engineer-hours. Call it $2,000-$4,000 in loaded cost, depending on your location.

The median cost of a production incident at a mid-stage SaaS company — accounting for MTTR, customer impact, and engineering follow-up — is somewhere between $5,000 and $50,000 depending on severity. (I've seen ranges from $1,400 to $400,000 in published postmortems, but median is probably $8K-15K.)

If a quarterly game day shaves 10 minutes off your average MTTR — and it will, because your team will know which dashboard to open — that's worth it on the first incident it affects.

Plus: the action items from game days (better dashboards, missing alerts, runbook gaps) prevent incidents entirely. Hard to measure, but real.

The pitch to your manager: "I want to run a quarterly 4-hour team exercise that will reduce our incident response time and generate a prioritized list of observability improvements. It costs one afternoon."

If they say no to that, you have a different problem. (And maybe start updating your LinkedIn.)

My Prediction

By mid-2027, the chaos engineering platform market will contract. Not because chaos is a bad idea — it's a great idea — but because the platforms overbuilt for a narrow slice of the market. Most teams don't need automated fault injection across 500 services. They need to run a game day once a quarter with the observability they already have.

The winners will be observability platforms that make game days easy: pre-built failure injection for common scenarios, game day mode that enhances dashboards for practice sessions, automatic timeline generation for retros. Some of that's coming to JustAnalytics later this year (the timeline generation bit, at least). DevOS handles similar coordination challenges for distributed development teams — different problem, same principle of making practice easier than improvisation.

But you don't need to wait for product features. The game day script above works with whatever you're running today. Pick a Friday afternoon. Kill a process. Watch your dashboards. Learn what you don't know.

Your 3am self will thank you.

Frequently Asked Questions

Do I need a chaos engineering platform to run game days?

No. Chaos platforms like Gremlin and LitmusChaos add convenience for large-scale, automated fault injection across hundreds of services. But for teams under 50 engineers, manual fault injection during a controlled game day teaches the same lessons. Kill a process, throttle a database connection, return 500s from a dependency. Your observability tool shows you what breaks. That's the learning.

How often should we run game days?

Quarterly is the minimum that builds muscle memory. Monthly is better if you can staff it. The first game day will be chaotic and reveal gaps. By the fourth, your team will have runbooks that actually work and dashboards they trust. Some teams run mini game days (30 minutes, one failure mode) weekly during slower periods.

What's the difference between a game day and chaos engineering?

Chaos engineering is the practice of injecting failures to discover weaknesses. A game day is a scheduled event where the team practices responding to failures together. Game days are a form of chaos engineering, but they're also incident response training. The goal isn't just to find bugs — it's to build the muscle memory so your 3am self knows exactly which dashboard to open first.

Can we run game days in production?

Yes, but start in staging. Once your team has run 3-4 game days and built confidence, production game days reveal issues staging won't. Real traffic patterns, real data volumes, real third-party dependencies. Use feature flags to limit blast radius. Have a rollback plan. And schedule during low-traffic windows — Friday 4pm EST, not Monday 9am.

Try JustAnalytics

All-in-one observability in one under-5KB script: cookieless analytics + error tracking + APM + session replay + uptime + structured logs. Replaces GA4 + Sentry + Datadog + Pingdom + LogRocket. Free tier (100K events/mo), Pro $49/month ($39 annual).

Start free → · AI Command Center MCP