AI Automation: How I Built an End-to-End AI Agent for Production Fixes a Zero-Touch Incident Response

AI DevOps Automation Banner

Orchestrating a Zero-Touch Incident Response with AI Agents

Zero-Touch Incident Response: How I Built an End-to-End AI Agent for Production Rollbar Fixes

The Problem:

As developers, we’ve all been there: a Rollbar alert hits. You stop coding, open the browser, find the stack trace, search your local repo for the file, checkout a branch, fix it, write a Jira ticket, draft an RCA in Confluence, push to GitHub, and then manually paste the PR link into Slack.

This process isn't "engineering"—it's administrative toil. I decided to automate the entire lifecycle using the Cursor AI Agent.

The Architecture: A Closed-Loop AI Workflow

The goal was to create a "Self-Healing" loop where the AI acts as the Project Manager, Developer, and Technical Writer simultaneously.

Step 1: Tools: The Integration Foundation (Token Generation)

To make this work, you need to grant the Cursor Agent "API keys" for your ecosystem. Here is the exact path to generate each one:

Service Token Type Navigation Path
Rollbar Project Access Token Settings > Project Access Tokens > Generate (Need read & write)
GitHub Personal Access Token Settings > Developer Settings > Tokens (Classic) > Select repo and workflow scopes
Atlassian API Token id.atlassian.com > Security > Create API Token (Works for Jira & Confluence)
Slack Webhook URL api.slack.com > Create App > Incoming Webhooks > On > Add to Channel

Step 2: Securing the Environment

I stored these credentials in a .env.automation file. Crucially, I added this file to .gitignore to ensure these sensitive keys are never pushed to the cloud.

ROLLBAR_ACCESS_TOKEN="your_token"
GITHUB_TOKEN="your_token"
ATLASSIAN_API_TOKEN="your_token"
SLACK_WEBHOOK_URL="https://hooks.slack.com/services/..."

Step 3: The "Agentic" Instructions

The secret sauce is the Master Instruction File I feed to the Cursor Agent. This file defines the behavior for the five phases of the response.

Workflow Blueprint:

  • Phase 1: Discovery & Environment
    1. Fetch All the recent 'active' error or warning from Rollbar API.
    2. Identify the file, line number, and stack trace.
    3. Checkout a new branch `fix/rollbar-{id}` from `dev` branch.
  • Phase 2: Analysis & Fix
    1. Read the local file identified in the error.
    2. Propose a fix that prevents the error (e.g., null checks, try-catch, or logic correction).
    3. Apply the fix only, no need to run anything.
  • Phase 3: Documentation (Jira & Confluence) Confluence: Create an RCA page using this template:
    1. - Title: RCA - {Error Name} ({Date})
    2. - Summary: {Brief explanation of why it crashed}
    3. - Resolution: {Description of the code change}
    4. - Links: [Rollbar Link]
  • Phase 4: GitHub PR
    1. Commit changes: `fix: resolved rollbar error {id}`.
    2. Push and create a PR against `dev`.
    3. PR Body must include the Confluence RCA link.
  • Phase 5: Slack Notification

    Send a POST request to the Slack Webhook with this JSON structure:

    {
  "text": "🔍 *New PR Raised: Rollbar Fix for dev*",
  "attachments": [{
    "color": "#3AA3E3", 
    "fields": [
      {"title": "Error Name", "value": "{Error_Name}", "short": true},
      {"title": "Pull Request", "value": "<{GH_PR_URL}|View PR on GitHub>", "short": true},
      {"title": "RCA Document", "value": "<{Confluence_URL}|View Confluence Doc>", "short": false},
      {"title": "Status", "value": "Pending Review", "short": true}
    ]
  }]
}

Note: Make sure you only push the branch and create the Pull Request. Do not attempt to merge it or trigger a deployment. The Slack notification should strictly say 'PR Raised'.

Step 4: Execute the Agent

Now open the cursor composer and add below prompt with both the files tagged, file that has secret and above prompt instruction saved file,

      
Follow the instructions in the attached markdown file to resolve the next active Rollbar error.
@.env.automation @automation_fix_rollbars.md

Congratulations Done 🎉 🎉 🤟 🤌

Now check the slack message, Confluence doc and PR

The Execution Flow

[ PRODUCTION ERROR ] 
       │
       ▼
[ ROLLBAR API ] <───────┐
       │                │
       ▼                │ (1) Fetch Metadata
[ CURSOR AI AGENT ] ────┘
       │
       ├─► (2) LOCAL REPO: Creates Fix Branch & Commits Code
       │
       ├─► (3) ATLASSIAN: Creates Jira Ticket & Confluence RCA
       │
       ├─► (4) GITHUB: Pushes Branch & Raises PR to Staging
       │
       └─► (5) SLACK: Sends Final Notification with Links

Outcomes:

(Slack + Confleunce RCA Doc + Github PR changes)

Confluence RCA:

Github PR changes:

Slack Message:

The Results & Impact

By implementing this, I’ve transformed our incident response:

  • MTTR (Mean Time to Recovery): Reduced from many hours of manual work to under 5 minutes of AI execution.
  • Documentation Compliance: We now have 100% RCA coverage in Confluence for every production fix.
  • Focus: I no longer have to leave my IDE to manage the "paperwork" of a bug fix.

Conclusion

The future of software engineering isn't just writing code; it's orchestrating agents. By connecting your IDE to your monitoring and communication tools, you move from being a "coder" to being an "architect of automation."

Comments

Post a Comment

Popular posts from this blog

Use ChatGPT for improve your coding quality

Image
Hi, This is  Shubham Mishra  from India,   In this post we will discuss about how to Use ChatGPT for improve your coding quality and I am sure you will get excited to make your hands dirty with those code, so why to wait let's get started... When first time i asked ChatGpt to write some script for me, I asked "Write a shell script to open multiple chrome windows with some url's", it was to my surprise that ChatGPT not only wrote the script it also tried to explain every line of code, you can check the script here : Shell script to open multiple chrome windows , so then onwards not only for writing script but also for reviewing some of my work related code I ask ChatGPT to review and rewrite me optimised code too. How can Chat GPT provide some general tips and best practices that developers can follow to write good code: 1. Follow coding standards and best practices: Consistency is key when it comes to writing good code. Follow established coding standards and bes...
Read more

Divide and Conquer an Algorithm design technique

Image
Hey, this is  Shubham Mishra . In our previous post of  Algorithm and its analysis , we have discussed all the algorithm, its designing and many another part. In the same post, In the algorithm designing section, I have mentioned a step that is 'Algorithm Strategy' for selecting a technique to design an algorithm. Out of all the, one very most popular among all is Divide and Conquer. Now, let's start with divide and conquer technique for algorithm designing. In this technique, a problem is, Broken down into smaller sub-problems. The sub-problems are solved independently. Merge all the solutions of all sub-problems to get the solution to the original problem. Basically, If a problem seems too big but if the problem needs repetitive steps then it's better to divide the big problem and solve sub-problems. The sub-problems once solved then combine the solutions of sub-problems to get the result for the original problem. Now let's understand it with the ...
Read more

Top interview questions for iOS developer Page 2

Image
Hi, This is  Shubham Mishra  a ROR, iOS, Python developer. In this blog we will see top iOS interview questions. In the dynamic world we should be updated so as to compete with others hence I have listed below top interview questions based on my and others interview experience. This is the another sets of questions and answers, you can check my previous post,  iOS interview questions Page 1 Let's Begin.... Designed by slidesgo / Freepik Q : What do you mean by Dynamic Dispatch? A : The dynamic dispatch method is the selection mechanism for which implementation technique or function of a polymorphic operation to call at runtime. Which means invoking our methods like object method when we want to. But Swift is not dynamic dispatch by default. Q : What do you mean by Code Coverage? A : Code coverage is a measure which helps us calculate unit test values. Q : Explain Completion Handler? A : When our app is making an API call, completion handlers are super handy...
Read more