TL;DR: We ran a 48-hour hackathon on AI agents in observability, with four randomly paired teams each taking on one problem from the incident lifecycle. By the end, every team had a working agent running in our cluster: one that diagnoses failed CI runs and deployments, one that investigates alerts as they fire, one that writes evidence-backed RCAs, and one that finds noisy alerts and suggests fixes.
Production incidents don't come one at a time, and they rarely end when the fix ships. A bad deploy fails in CI. A service falls over at 2 AM. Someone has to find the cause, someone has to write it up, and somewhere a dozen alerts keep firing that nobody trusts any more.
Every SRE and platform team we work with knows this loop. Most of it is still done by hand.
So we ran a hackathon to see how much of it AI agents could take on. We gave our engineers 48 hours, four problems from that loop and a shared platform to build on. By Wednesday afternoon, four working agents were running in our own cluster.
This post covers why we ran it, how we set it up, and what came out of it.
Why we ran it
Two things came together.
Shipping had become cheap. We had recently brought up FluxCD on our bare metal machines and moved several services onto it. Getting a piece of work deployed and running had gone from a project to a pull request. That made us want to show people how quickly an idea can become a running system.
We wanted to push on AI in observability. Our internal platform team had been exploring it for a few sprints. Observability is where SRE and dev teams lose the most time to repetitive work: reading logs, correlating events, chasing noisy alerts. It is also a place where getting things wrong is expensive. That made it a good test of whether AI agents can do real work, not just demos.
A hackathon fit both goals. We wanted engineers to:
- Get hands-on with AI agents and the tools around them (MCP servers, agent frameworks, LLM APIs).
- Apply them to real bottlenecks, on our real cluster, with real data.
- See how quickly a proof of concept can go from idea to running service.
Building observability skills is a priority for us as a team. The best way to build them is to work on problems that matter.
Choosing the problems
Vijay, Arihant and I picked the problems. We didn't want a list of AI ideas. We wanted the places where teams actually get stuck.
We followed an incident through its lifecycle and picked one bottleneck at each stage:
| Stage | Problem | The question |
|---|---|---|
| Before and right after a deploy | CI/CD debugging | Can we catch a bad change before it ships, or recover as soon as it does? |
| During an incident | AI debugging | Once something breaks, can we mitigate and resolve it faster? |
| After the fix | RCA investigation | Can we understand the real root cause and validate it? |
| Ongoing | Alert noise reduction | Can we cut the alerts that fire without meaning anything? |
Each team got one of these. The problem was fixed, but the solution wasn't. Teams could design and build it however they wanted. We offered suggestions, not rules.
How it ran
The hackathon ran for 48 hours, from 2 PM on Monday, 28 September to 2 PM on Wednesday, 30 September.
Pre-prep
We announced the theme, AI in observability, so people could read up and prepare.
Kickoff
On Monday, we announced the four problems and paired people up at random, making four teams of two. Random pairing was deliberate. It put people together who don't usually work together.
Design first
Each team's first job was to brainstorm and present an architecture design, before writing any code. Arihant and I acted as judges and mentors. We answered questions, pushed back on designs and gave feedback after each team presented.
Shared platform
Several teams needed the same building blocks, including HolmesGPT, the SigNoz MCP server and the GitHub MCP server. The teams noticed this early and split the setup between them. One team deployed HolmesGPT in the cluster, another connected it to SigNoz and GitHub, and everyone built on top. Nobody asked them to do this. It was one of the best parts of the week.
Check-ins
Throughout the build, we met every team at 9:30 AM and 4 PM each day to see where they were, unblock them and give feedback.
Development
Teams worked through the night. There were plenty of commits at 3 AM. Nobody asked for that. People stayed up because they were hooked on the problems and wanted to see their solutions work.
Final demos
On Wednesday, each team demoed a working agent, running in our cluster against real alerts, real CI runs and real incidents.
How we judged
We judged on four things:
- Collaboration: within the team and across teams.
- Demo quality: does it work, and does it show the problem being solved?
- Responsiveness to feedback: what did the team do with the feedback from each check-in?
- Communication: can the team explain the problem, the design and the tradeoffs?
We left out "most features" on purpose. We cared more about how teams worked and thought than about how much they built.
What came out of it
Four agents, all running on our FluxCD-managed Kubernetes cluster and built on a shared set of tools: HolmesGPT for investigation, SigNoz for logs, metrics and alerts, and MCP servers that give agents access to SigNoz, GitHub and Outline.
CI/CD debugging: cicd-agent
An agent that investigates CI failures on pull requests and deploy failures after merge.
- When a GitHub Actions run fails, it reads the failed job's log and the PR diff, then comments with the cited log lines and a fix the author can commit in one click.
- When a Flux deploy fails, it traces the failure to the workload, finds the PR that actually caused it (not just the newest merge), and opens a draft fix PR.
Code collects the evidence and decides who caused what. The model only explains the evidence, and code checks its citations before anything is posted.
AI debugging: alert debugger
An agent that picks up firing SigNoz alerts, asks HolmesGPT to investigate each one, and posts the findings to the team's Google Chat space.
It dedupes repeated notifications, so one incident gets one investigation. When something breaks, the on-call engineer starts with an investigation already done instead of a blank screen.
Alert noise reduction: alert-agent
An agent that runs once a week to find noisy SigNoz alerts and suggest how to fix them.
- It replays each alert's week of history under real rule changes, such as a longer wait or a higher threshold, and measures how many firings each change would remove.
- It flags changes that would hide long firings, since those are probably real problems.
- A model labels each alert (flapping, no-data, duplicate, always-on or real) and picks the safest fix. Code checks every answer against the numbers.
It only suggests. Engineers apply the fixes.
RCA investigation: RCA Agent (Winner 🏆)
An agent that writes the root cause analysis after an incident is fixed.
An engineer tags the agent in Google Chat with a short summary. A few minutes later, a full RCA appears in Outline, with every fact cited back to the tool output that shows it.
- HolmesGPT investigates without being told the fix. If it reaches the same cause on its own, that is strong evidence the story is right.
- Any value HolmesGPT reports is kept only if it appears in the raw output of a tool it actually ran.
- Confidence is worked out by code from five checks, not picked by the model.
Venkidesh and Hemant built it and took first place. Venkidesh tells the full story in Building an AI Agent That Writes RCAs You Can Trust.
What surprised us
The pace and the coordination. Teams moved fast, both within each team and across teams. The shared HolmesGPT and MCP setup happened without anyone planning it.
The depth of the conversations. One discussion stood out. Akshay and Nischit, working on alert noise, debated when to let the AI decide and when to write plain code, especially for anything numeric. Their answer, which every team reached in its own way: let the model judge and explain, and let code do the counting. You can see it in all four agents. No duration, count or confidence score comes from a model.
The care in the details. Every team built in guardrails without being asked: read-only access, citations, dry-run modes, and answers that say "we don't know" instead of guessing.
Overall, the hackathon went better than we expected.
What's next
All four projects are continuing after the hackathon. They are still running in our cluster, and teams are still improving them.
We are taking them from proof of concept to production ready. They are in active use, and real usage is shaping what we fix and build next. Once they are ready, we plan to open source all four.
This post is the first in a series. Each team will write about their own project: the problem, the design, the tradeoffs and what they learned. The first one, on the RCA Agent, is already out.
Beyond the results
What stayed with us wasn't the agents. It was how people worked: the curiosity, the attention to detail, and the attitude that any of these problems could be solved in 48 hours.
It also taught us something about running AI agents, not just building them. Getting an agent to work in a demo is the easy part. Keeping it useful in production is harder, and it raises new questions. Is it still right? Is it getting better or worse? Would an engineer act on what it says?
That is what we are exploring now: feedback pipelines for agents, and what it takes to have confidence in one. How do we capture whether an agent's answer was right? How do we know a change made it better and not worse? And when has an agent earned enough trust to act on its own?
Watch this space to read more about our approach.
More from the hackathon