Every Mobile Developer’s Biggest Nightmare
If you develop a successful iOS app you’re certain to find yourself in this situation: end users report that the app is crashing or behaving strangely. You’re struggling to help because you don’t have any iOS crash reporting information about what’s happening in the app on their device.

If they’re encountering a problem you can’t reproduce, debugging and fixing user-reported issues can be a major headache — more so if your users rely on your app to help them manage critical business processes. In this post, I’ll outline how the iOS development team at xMatters has approached this problem.

Mobile Apps at xMatters
Mobile apps for iOS and Android are an important part of our product. Rich functionality and push notifications instead of SMS messages have resulted in a strong mobile user base.

If you’re encountering a problem you can’t reproduce, debugging and fixing user-reported issues can be a major headache.

Since users rely on our apps for notifications about events that are potentially disruptive to their business operations, it’s important for us to solve problems quickly. At minimum we need crash reports, but ideally we also want access to device logs to see what was happening leading up to the crash.

To accomplish our goal of being less reactive and more proactive, we have to detect when problems in the app are affecting users — before they begin to report issues. And we want to know as quickly as possible if a new app release has introduced an issue that affects users.

Why we don’t rely on Apple crash reporting tools
Like most iOS developers, we started with the troubleshooting solutions that Apple provides, since we’re already making daily use of tools like Xcode and App Store Connect.

While Xcode shows crash reports in its Organizer window, it reports crashes only for users who have chosen to share analytics with app developers, which requires the user to know where this setting is. (BTW, your users are a lot more likely to share analytics if you take security seriously, which we do.)

With some digging and upfront investment, we’ve been able to implement the troubleshooting tools we need to quickly hunt down and resolve iOS app issues.

Additionally, Xcode doesn’t provide an easy way to track the rate of crashes per app release. You can’t customize or enrich the iOS crash reporting information with identifiers that indicate which users are experiencing crashes. And while you do get a ‘symbolicated’ (human readable) backtrace for each crash, you don’t get any logs or event history to indicate what was happening prior to the app crashing.

App Store Connect, under App Analytics, presents trends for crashes over time, per release. However, there’s no way to get from the crash counts to the crash reports.

(It’s worth mentioning that we often make use of the Phased Releases for Automatic Updates feature of App Store Connect. This updates users in stages over a period of 7 days, starting with small groups of users and gradually increasing the proportion of users upgraded.)

Why we chose Firebase
Given these limitations, our iOS development team adopted Google’s Firebase product for crash reporting and mobile analytics. The core Firebase SDK is simple to add to an iOS app, and starts forwarding crash reports to its Crashlytics component within an hour of the crash occurring.

Firebase has some great features that behave like a black box on an airplane:

It associates a user ID with the crashes and analytics events that originate from a given device. We populate this with a UUID (Universally Unique Identifier) that we persist on the iOS device in a way that survives the app being reinstalled, providing us visibility of a given user’s experience over time.

It enables you to set custom keys in Crashlytics that are included in crash reports to provide information about the state of the app at the time of the crash.

Another Firebase feature we use a lot is the Latest Release screen:

The Firebase Latest Release screen feature allows us to see how quickly users are adopting our latest app release.

This allows us to see how quickly users are adopting our latest app release, as the metrics per release show crash-free users and crashes per 10K sessions.

Even better, Firebase monitors crash reports and reports emerging stability issues if a release starts to report crashes at higher-than-normal levels:

Extending crash reports with device logs
Often the stack trace from the iOS crash reporting is all you need to pinpoint a bug in the app. However, sometimes the app has crashed because of an error that occurred sometime earlier and that isn’t evident from the stack at the time of the crash. In situations like this it would be ideal to have console logs similar to what Xcode displays when you run an app as a developer.

While full-fledged remote logging solutions exist for iOS apps, we didn’t want to add another third-party product into the mix. And Firebase Crashlytics supports custom logs which we have used in the past.

Hopefully, our experience will save you a time and toil and inspire you to evaluate tools that can accelerate your iOS development.

The solution that we chose was influenced by our wider use of Google BigQuery and the capability that Google provides to integrate your Firebase project with BigQuery. You can enable the Firebase / BigQuery integration in seconds, although you may have to upgrade your Firebase project to the Blaze plan, which has pay-as-you-go pricing. Once they’re integrated, you can search your analytics events using the BigQuery Query Editor.

So we have created a custom “Log” analytics event in Firebase that registers significant app events. These Log events are visible along with other Firebase events in Crashlytics crash reports on the Log tab, but we usually view and filter device logs using BigQuery so you can view the records directly or save them to a CSV file.

In summary, our app uses the Firebase SDK to send analytics events to Firebase, which in turn forwards them to BigQuery. In BigQuery we can construct queries that track events from a single user over time, or analyze the behavior of all users of the app.

From Theory to Practice
The screenshots above actually show this system in action. Last year we released version 3.24.1 of our app, and later that same day we started to get notifications from Firebase about an issue with the stability of the new release. The Crashlytics and Latest Release screens in Firebase reported increased crash rates.

The stack traces in the crash reports showed that the app was crashing when retrieving account settings from the keychain. We hadn’t changed that code in the new release and couldn’t see anything unsafe going on at the point of the crash. The crash report also showed that our app was running code on another thread at the time of the crash, but it wasn’t clear how that situation was arising.

Fortunately, the crash report includes the user’s ID:

The Firebase crash report includes the user IOD.

We headed off to BigQuery and got the logs from that device for the period leading up to the crash. They showed that the app had just been launched and was starting in the background in response to a push notification from xMatters.

That information led us to examine the code that handles background notification processing and ultimately to discover that a code change was incorrectly using a background thread to process the notification. We were able to reproduce the problem and publish a new release containing a fix to the App Store.

Look beyond standard tools for leverage
With some digging and upfront investment, we’ve been able to implement the troubleshooting tools we need to quickly hunt down and resolve iOS app issues. We’ve learned that the basic reports built into Xcode are a good start but they’re simply not robust enough for serious development. Hopefully, our experience will save you a bunch of time and toil and inspire you to evaluate Firebase, BigQuery and other tools that can accelerate your iOS development — and save you from trying to track down crash nightmares in the ‘dark.’

Isn’t this a great time for automation?
iOS crash reporting is a critical element of providing support. Find out for yourself! Try xMatters for free and get a 14-day trial of our advanced features!