Last week, Hemant and I walked away with first prize at the Infraspec hackathon. Our project was the RCA Agent: a bot that automates incident reports so engineers can focus on fixing problems instead of writing docs.
The idea is simple. Once an incident is resolved, tag @RCA Agent in Google Chat with brief summary. Minutes later, a full root cause analysis appears in Outline, with every fact cited back to its source.
Getting there was not simple at all. This is the story of how we built it, how a handful of AI agents ended up working together like a small investigation team, and the things I learned along the way.
Understanding the problem first
Our mentors gave us the problem statement: automate the RCA. Before writing any code, we wanted to understand why RCAs go unwritten in the first place.
Picture this. A service goes down on a Friday evening. Someone spends two hours finding the cause, pushes a fix, and everything is green again. They make a note to write the RCA on Monday.
On Monday there's a sprint to plan and a release to ship. By Wednesday, they're trying to remember whether it started at 6:10 or 6:40, and which of the three commits that evening actually fixed it.
When we broke it down, the problem was never that engineers don't care. Writing a good RCA is just a lot of work. You have to dig back through logs, Kubernetes events and commits to piece together what really happened. By the time things calm down, there's always something more urgent, so the RCA waits. A week later, nobody remembers the exact times, which commit it was, or what was tried first.
Then, months later, the same thing breaks again, and there's no record of how we fixed it last time. The fix was done, but the learning was lost.
That gave us a clear picture of what the agent had to do. It had to do the digging, it had to do it right after the fix while the data is still fresh, and it had to produce something people would actually trust.
The one thing we didn't want to build
The easy version of this project is obvious. Paste the incident into any AI tool, like ChatGPT or Claude, and ask it to write an RCA. You'd get something that reads beautifully, and you'd have no idea which parts are true.
An RCA you can't trust is worse than no RCA. So before writing any code, we agreed on a few rules:
- Every fact needs proof. If the RCA says a pod restarted at 09:58, there must be real output from a real tool that shows it.
- Look, don't touch. Nothing in the system is allowed to change the cluster. It can only read.
- Numbers come from code, not from the AI. Durations, times and the confidence level are calculated, never written by a model.
- A human signs off. Every RCA is published as Pending review. The agent writes the first draft, and an engineer owns the final one.
There was one more rule, and it turned out to be my favourite. When we ask the investigator to look into an incident, we don't tell it how the incident was fixed. If it still lands on the same cause and fix on its own, that's a strong sign the story is right.
Meet the team (of agents)
The way I like to explain it: think of a small investigation team. There's someone who takes the case, a detective who goes out and collects clues, and a writer who turns it all into a report. Each of them is an agent, or a tool used by one.
The RCA Agent is the lead on the case, not the detective. It takes the request, decides what to ask, checks every answer that comes back, and writes the final report. The actual digging is done by another agent.
Here's who does what:
- The AI Debugger is a separate agent that helps fix incidents. It isn't a part of the RCA Agent, but once the fix is in, it can pass the case along for an RCA. People can also start a case themselves, from Google Chat or a simple API call.
- The RCA Agent is the lead. It's the only one that sees the whole picture: what was reported, what was found, and whether the two agree.
- HolmesGPT is the detective. It's an open-source AI investigator running inside our cluster. Given the incident, it decides which of our systems to check: pods, events and deployments in Kubernetes, error logs in SigNoz, and recent commits our GitOps repo. It brings back not just its conclusions but the raw output of everything it checked. That part comes out of the box. What our teammates set up is which tools it can use: Kubernetes support is built in, and they added SigNoz and GitHub through MCP servers.
- The MCP servers are the detective's toolkit. One lets Holmes search our logs in SigNoz, another lets it read commits on GitHub through the GitHub MCP server. The RCA Agent uses one more to publish the final report in Outline.
- The LLM (Claude Haiku) is the writer. It reads messy human reports, compares the story with the evidence, and drafts the RCA. It never gets the last word, though. The agent's built-in rules checks everything it writes and drop anything the evidence doesn't back up.
Following one incident through
Let's say an engineer drops this in the chat:
@RCA Agent ollama in namespace llm returned 404 on every call from 10:00 UTC on 25 Sep. Fixed at 10:40 by recreating the volume.
Here's what happens next:
- Read the report. The LLM pulls out the basics: which service, when it started, what people saw, what they did to fix it, and any questions the report asks.
- Hide the answer. The report is rewritten for Holmes, minus the fix. Holmes only hears that the problem has been fixed, not how.
- Investigate. Holmes goes through Kubernetes, the logs and the Git history on its own, and comes back with findings plus the raw output of every check it ran.
- Compare stories. The agent puts the reported cause and fix next to what Holmes found. If something is still unclear, Holmes goes into one more round of specific questions.
- Check every claim. Code checks the report line by line. Did it really start at 10:00? Did the fix really happen before it recovered? Did the failures stay gone?
- Write it up. The LLM drafts the RCA in our team's template, with a citation on every sentence.
- Publish. It lands in Outline as the next numbered incident, INC-007 or whatever is next, with a full run trace underneath. The agent replies in the same chat thread with the link.
The whole thing takes a few minutes. And if a step fails, the run doesn't give up. It finishes and tells you what it couldn't check.
"But how do you know the AI didn’t make it up?"
It's the first question everyone asks about an AI-written report, and honestly, it's the right one.
Our answer is a step we call grounding. Remember that Holmes brings back the raw output of every check it ran? Before anything goes into the RCA, the agent takes every name, time and commit Holmes reported and looks for it in that raw output. If it's there, it becomes evidence. If it isn't, it's thrown out, and the run trace lists what was thrown out.
I love how simple the idea is. An AI can misread a log or invent a timestamp. What it can't do is make that timestamp appear in the actual output of kubectl.
A few more guardrails sit on top of that:
- Every sentence cites its evidence. A sentence with nothing behind it simply gets dropped.
- Confidence is a rule, not a feeling. The agent runs five checks: Did Holmes find the same cause on its own? Does the evidence support it? Did the fix actually work? Do the times match? Does anything contradict the story? Four or more passes means High, two means Medium, anything less is Low. The RCA shows each check and why it passed or failed.
- "We don’t know" is allowed. If the evidence can’t answer something, the RCA says so, and says what data would answer it next time. If that data isn’t recorded anywhere today, the RCA adds an action item to start collecting it. If the cause or fix is still open, it adds one for the team to review. It never fills the gap with a guess.
Confidence checks from a real RCA
Every piece of evidence also comes with a "how to see it": the exact command or query that shows the same thing. So a reviewer can check any claim in a few seconds.
How we split the work
We split it in a way that let us both move fast. I took the overall implementation of the agent: the pipeline, how it talks to HolmesGPT and the other systems, and how people ask for an RCA. Hemant connected the agent to Outline and owned the refinement of the RCA document itself. The agent's workflow, how each step hands over to the next, we refined together. Teammates set up HolmesGPT and the MCP servers.
A few moments from my side stand out.
One door into our systems
The first decision I made was to keep the agent on a short leash. It talks to exactly three things: HolmesGPT, Outline and the LLM. Everything about the cluster, the logs and Git goes through Holmes, and Holmes can only read. The agent itself doesn't even have cluster credentials.
It felt restrictive at first. It turned out to make everything simpler. There was one place to check permissions and one place to look when something went wrong.
Testing on a real incident
We didn't want to demo on a made-up scenario, so we replayed a real one: an Ollama service that returned 404 on every call because its model went missing after a storage change.
It broke things immediately, which was exactly the point. On our first run, Holmes was on a small local model, and its instructions alone were about twice what the model could take in. Holmes was quietly losing half the question. Moving to Claude Haiku, with a much bigger context window, fixed it.
Asking from Google Chat
I wanted people to ask for an RCA where they already talk, in Google Chat, so I built a Chat app for the agent.
Our cluster doesn’t accept calls from the internet, so Chat can’t push messages to the agent. Instead, the agent polls the Chat space every few seconds through the Chat API and picks up any message that tags it. A small Apps Script replies "Got it" instantly, so people know they were heard, and the agent replies in the same thread with the link once the RCA is ready. No extra infrastructure, and no billing.
How Hemant made the RCAs actually good
Once everything worked, the RCAs were... fine. Correct-ish. Hemant wasn't happy with fine.
He read our own RCAs the way a tough reviewer would, line by line. Every time something was wrong, he didn't just fix that sentence. He found the step that let the mistake through and changed it so the agent would catch it next time. One RCA disagreed with itself about whether the fix worked. Another blamed an error that shows up every hour of every day. Each of those became a rule the agent now follows.
What I'm taking away
A few things I'll carry into whatever I build next:
- Let the AI choose, let code decide. Models are great at deciding where to look and at writing clearly. Anything people will rely on, like a time, a duration or a confidence level, should come from code.
- Keep the receipts. Grounding only works because Holmes brings back the raw output. Without it, every claim is just "the AI said so".
- Small permissions make simple systems. Giving the agent less access made it easier to build, easier to debug and easier to trust.
- "We don’t know" is a valid answer. An RCA that admits what it couldn’t find, and says what data would help next time, beats one that guesses.
- Test on real incidents and read the output like a critic. Almost every real improvement came from a real RCA that was wrong in some specific way.
A big thank you to Hemant, for making the RCAs so much better with that relentless review loop. To the teammates who set up HolmesGPT and the MCP servers.
References
If you want to dig into the pieces we used:
- HolmesGPT: the open-source AI investigator (GitHub)
- Model Context Protocol (MCP): the standard that lets AI agents use tools
- GitHub MCP server: how Holmes reads commits and diffs
- SigNoz: where our logs and events live
- Outline: where the RCAs are published
- Claude Haiku: the model behind the agent and Holmes
- Google Chat apps and Apps Script: how people tag @RCA Agent