When Automation Outruns Oversight: Keeping Humans in the Incident Response Loop
Recent major outages have shown how quickly a disruption can spread across services that share the same infrastructure. For teams using AI to write code and prepare releases, that raises a practical question. If a bad change reaches one server, how quickly will someone know enough to stop it from reaching the rest?
AI can shorten the time it takes to get a change ready to ship. Some failures still take time and real traffic to show up. If the next stage of a rollout starts before the first one reveals a problem, the same defect can reach more customers. When several releases land in that window, responders also have more changes to investigate. The 2024 DORA research found an association between higher AI adoption and lower delivery stability. Its authors pointed to small batches and robust testing as practices worth protecting as teams produce code faster.
Catch a bad release while it’s still on one server
Imagine a team rolling out an AI-assisted update to an application that uses a shared database. The deployment pipeline sends the new build to one server first. It passes its health check and serves requests successfully. But after each request, the application leaves a database connection open. Customers can still complete transactions while the server quietly consumes capacity that other services need.
This is an opportunity to prevent a major incident. The team can configure its pipeline to check database connection usage before expanding the rollout. If connections on the updated server keep climbing while traffic stays flat and its peers look healthy, the pipeline holds the release. Monitoring triggers an xMatters workflow that brings the build number, change record, and connection trend to the on-call application owner. If that person is unavailable, xMatters escalates while the rollout stays on hold. An authorized responder can review the evidence and approve a rollback through the connected deployment tool.
Catching the leak at that point can spare every other service that depends on the database. The pipeline controls where the code runs. xMatters closes the gap between spotting the warning and reaching someone who can act on it.
What if the database starts failing first?
Suppose the rollout checks only application errors. The new build passes, and the leak continues until other services start competing for connections. By the time the database alert fires, customers are already seeing timeouts. A restart might look like a quick way to clear the connections. If one application server is leaking them, though, it will keep opening new connections. Taking a database node offline could also shift more traffic to the remaining capacity just as it’s under the most pressure.
The team needs to see the connection leak, understand what changed, and weigh the risk of a restart against the alternatives. Here’s how that can work in xMatters.
Gather the evidence while responders mobilize
Monitoring sends the database alert to xMatters. On-call schedules and escalation rules get the database team involved while a configured Flow Designer workflow looks up service dependencies and runs read-only checks through connected diagnostic tools. Those checks point to one application server holding an unusually high number of open connections. The other servers in the pool look healthy and are running an older build.
The workflow checks change records. A deployment updated that one server 30 minutes before the errors began, and the record shows no other changes in the deployment. xMatters brings in the on-call application owner and gives both teams the same evidence to work from. The database team can see which application server is driving the load, and the application team can see when the database started to feel it.
Use AI to weigh the options, put the decision in human hands
In the xMatters Incident Console, a responder can ask the Incident Agent to compare a database restart with rolling back the application deployment, using the diagnostics and change details available to it. Given that evidence, the agent might recommend rolling back the affected server first. The connection leak appears isolated to that server, and the older build is still serving traffic elsewhere in the pool. A database restart would leave the suspected leak in place and could put more strain on the remaining nodes.
The database and application responders verify the evidence together. They check that the healthy servers can carry traffic while the affected server is out of rotation and confirm that the previous build is safe to restore. An authorized responder then approves the rollback. If approval doesn’t come through, the workflow can hold the change and escalate to the next authorized responder.
After approval, the configured xMatters workflow calls the connected deployment tool to roll back the affected server. Health checks determine when it can rejoin the pool. Responders watch database connection counts, request timeouts, and customer-facing error rates to confirm that service is recovering. If those signals stay elevated, the workflow can route the issue back to the responders for another decision.
Put the lesson back into the next release
Once service is stable, the most useful question in the incident review is also the simplest. What could we see while the problem was still confined to one server? In this case, database connections began to climb before customer requests started failing. The team can use that signal in its next rollout, give it enough time to reveal a slow leak, and test that a warning reaches someone with authority to hold or reverse the change.
That gives xMatters two opportunities to help. During a release, it can bring an early warning to the people who can contain it. If the problem reaches customers, those same teams can use the change record, diagnostic evidence, and approval workflow to choose a recovery action and see whether it worked.
Want to try incident workflows with your team? Sign up for xMatters Free.