On-call engineers know the drill: an alert fires, and you're suddenly scrambling to figure out which service is broken, what changed, and where to even start looking. The first twenty minutes are often pure information gathering—checking dashboards, reading logs, searching docs, and trying to remember if you've seen this before.
Instacart decided that process was ripe for an AI assist. They built Blueberry, an AI-powered incident response system that sits inside Slack and does a lot of that legwork for you. It's not about replacing engineers—it's about giving them a head start.
Why on-call is so painful
The core problem in large-scale operations is context. When something breaks, you need to know which team owns the service, what deployments just went out, what the logs are saying, and whether this matches a previous incident. That's a lot of digging before you even start diagnosing.
Blueberry automates that digging. It pulls together service ownership, deployment history, logs, metrics, and past incident reports—then hands the engineer a summary of what's happening and a few likely root causes. The engineer still makes the call, but they're not starting from zero.
How Blueberry works
According to Instacart, Blueberry is built on a multi-agent architecture. When an alert triggers, about ten sub-agents fire off in parallel. They each go hunting for information—checking deployment status, scanning logs, comparing symptoms to historical incidents—and then report back into the same Slack thread where the engineer is already working.
The system typically produces an initial diagnostic summary in about three minutes. That's a huge improvement over the old way, where an engineer might spend ten or fifteen minutes just figuring out where to look.
Slack-native by design
Blueberry lives inside Slack, which is where most incident response already happens. Engineers don't need to jump to a separate tool or dashboard. They just see a thread with Blueberry's findings, right alongside their teammates' chatter.
That might sound like a small thing, but it's actually a big deal. When you're in the middle of a firefight, the last thing you want is another tab to manage. Keeping everything in the conversation thread reduces friction and helps the team stay coordinated.
Accuracy gains from 14 years of history
Instacart says Blueberry's diagnostic accuracy jumped from 60% to over 90% after they connected it to their incident history—more than 14 years' worth of past outages and root causes. That's the kind of institutional knowledge that usually lives in people's heads or in scattered postmortems. Now it's feeding the AI's recommendations.
In April alone, Blueberry completed about 25,000 diagnoses across more than 270 Slack channels. The workflow success rate was 99.9%, and the system made over 58,000 tool calls using MCP (Model Context Protocol). It's also adaptable—Blueberry can be tuned to fit the working patterns of roughly 60 different teams.
Engineers stay in control
One of the key design principles is that Blueberry doesn't touch production. It only reads data and generates hypotheses. The actual decision to deploy a fix or roll back a change is still made by a human. That's a sensible boundary, especially in a system that's still learning.
As Instacart's CTO Anirban Kundu put it, Blueberry is part of a broader push into AI agents. The system is meant to be a force multiplier, not a replacement. Engineers get a head start, but they're still responsible for the final diagnosis and action.
Lessons for the wider community
What Instacart is doing with Blueberry offers some useful lessons for anyone running large systems—or for online communities that rely on shared operational knowledge.
- Context is everything. An AI assistant is only as good as the data it can pull. Instacart's accuracy jump came from connecting to real incident history, not from a generic language model.
- Meet people where they are. By integrating with Slack, Blueberry fits into the existing workflow instead of forcing engineers to adopt a new tool.
- Keep humans in the loop. Blueberry suggests, but doesn't act. That builds trust and avoids automation gone wrong.
- Learn from the past. The system gets smarter over time by feeding on each incident's resolution, creating a feedback loop that benefits future on-call shifts.
What's next?
Instacart is still developing Blueberry, but the early results are promising. For on-call engineers, having an AI that can sift through years of incident history in seconds could mean fewer late-night rabbit holes and faster resolutions.
It's a reminder that the biggest wins in AI often come not from flashy new models, but from connecting them to the right data and putting them in the right place. Whether you're running a grocery delivery service or an online community, the same principles apply: give your people context, keep them in control, and let the tools do the heavy lifting.
Comments (0)
Please sign in to post a comment.
Don't have an account? Create one
No comments yet. Be the first to comment!