Saving Valuable Time During a Major Incident

NBN reduces its time to notify and engage resolution teams with xMatters to get them investigating the issues and restore IT services faster.

NBN is the National Broadband Network enterprise established by the Australian federal government in 2009 to upgrade and deliver broadband throughout the country. The company is fulfilling its promise to upgrade the current legacy copper network with best-fit broadband technology to all 23 million Australians.

NBN is challenged to deliver an effective rollout of a multi-technology mix as part of the National Broadband Network. It’s a complicated undertaking where people, process, and technology play a critical role in NBN’s success. As with many large enterprise organizations, a major IT incident can cause unexpected interruptions, and limiting business impact depends on the time to react and restore services.

But with xMatters, NBN has significantly reduced the time to find the right people during a major incident and pulling them into the process faster to start working on the issue. They can also keep the business and key stakeholders updated in near real time, for a collaborative focus on corrective actions. The net results are faster incident response times, shorter mean times to recover, and a reduced impact to the business, all through targeted intelligent communications.

Challenges

NBN was relying on spreadsheets and other disparate systems for managing on-call schedules, escalations, and manual call trees. This manual call-out process for incidents took 25 minutes or more before assigning ownership of an incident to a resolver. They also had no way to track, audit, or log who was contacted or whether there were any responses.

A major incident often required a conference bridge. In a war room style approach, the NBN Major Incident Management team manually corralled the right teams to work the issue, frequently sending out bridge numbers via SMS and email, requesting participants to join the bridge.

If they were lucky, the Major Incident bridge reached a quorum in 30-40 minutes after the incident was escalated. But usually, some parties joined late and interrupted the call, adding 10 more minutes before they could make real progress.

The business frequently became frustrated with the delays and would continually call the IT Service Desk and the Major Incident Management team to get status updates which created even more interruptions. With long resolution times and only reactive communication, NBN’s IT Operations felt like it was always on the back foot.

Unenforced SLAs

NBN outsources many of its IT support functions to providers like IBM, Accenture, and Fujitsu. When incidents occur, these providers often charge NBN for any specialized staff they engage for resolution.

These providers have SLAs to meet and have to pay NBN a penalty for any SLA breaches. With such manual communications, NBN Co had no way to alert the providers that the clock was ticking on an incident or to verify whether the SLA was breached. This made it virtually impossible to hold their service providers accountable for the service they deliver, let alone collect any eligible credits for service level breaches.

Solution

NBN leveraged xMatters’ certified integration with BMC’s Remedy Incident Management Service Desk to automate calls to the right people on their preferred devices (email, SMS, app push or voice call) instantly. Using xMatters’ bi-directional, closed-loop communication capability, NBN can solicit a response from target individuals, asking whether they can take ownership of the issue.

If the first person cannot, the xMatters advanced scheduling capability escalates to the next person on call until someone takes ownership. Incident assigned times plummeted from 25 minutes to 2-3 minutes.

The xMatters system targets alerts to the major incident resolution team and other parties that need to participate so that assembling a conference bridge is painless and efficient.

“You never have to worry about thumbing for the dial-in members,” Bowler says. “xMatters calls you on the phone, authenticates you, then reads out a brief on why you’re being called. Then with a touch of a button, xMatters pushes you into the conference bridge instantly. Amazing!”
NBN has reduced its time to assemble a major incident conference bridge from more than 30 minutes to less than two minutes, with all active participants accountable and informed.

Proactive Service Impact Communications

Now, the Major Incident Management team communicates proactively to the business and key stakeholders about service interruptions or outages, so they know IT is on the job. As a result, follow-up calls into the Service Desk for status updates have been reduced by 75% or more, and the incident responders can stay focused on resolving issues faster.

Benefits

Targeted closed-loop communications have resulted in fewer but more actionable notifications. Any time someone receives a notification and does not respond, system auditing tracks it and logs it. Greater accountability has reduced incident assignment times by 85% and reduced wasted time throughout the incident life cycle.

Local providers can get started on work faster, creating more customer satisfaction. Now NBN can start the clock and accurately verify if local providers’ crews breached SLAs, so NBN can recoup on service penalties.

A Proactive Reputation

Informing the business and key stakeholders proactively and restoring services faster has helped people see NBN’s IT Operations as a proactive department. With a centralized system of record, IT can also help other NBN departments, such as Media Relations and Customer Services, craft the message to NBN customers, strategic partners, the media, and the public to improve NBN’s external reputation too.

“It reduced our mean time to restore and improved our overall service levels to the business,” Bowler explains. “xMatters provides a level of visibility and auditability into the communications that occurs during an incident that we have never had before.”

The Best Is Yet to Come

Now that NBN leaders see how xMatters can streamline Major Incident Management processes, they are looking to optimize other IT processes throughout the organization. They are targeting Change Management, Event Management, and Batch Job Scheduling now. NBN corporate communications is also interested in leveraging xMatters for internal communications so they can stop managing painful distribution lists. Business Continuity, Disaster Management, and Field Workforce Management processes are on the horizon.