Visit the xMatters Trust site

What are the keys to building software development security into the early stages of product development? And what are the costs of ignoring security? In this article, xMatters Product Manager Kit Brown-Watts provides his insights on the matter.

Every investment decision comes with trade-offs, usually in the form of cost, quality, or speed. The CQS Matrix, as I like to call it, captures the dilemma most product people face. If you’re a perfectionist, you may find the temptation to build 100% of a feature set irresistible. On the other side of the perfection spectrum, the temptation to “just ship it” can be equally strong.

When it comes to software development security and compliance, regardless of your product instincts and tendencies, most top-tier organizations tend to take quality very seriously and are often willing to invest lots of resources and time to meet accepted standards. This has the unintended consequence of product leaders often “playing it safe” and erring on the side of security. However, the idea that security takes extra time, costs more, or reduces innovation is often a symptom of not properly building security into the Software Development Life Cycle (SDLC) in the first place.

Speed, quality, and cost in product development. Image source: https://m.subbu.org/cloud-optimization-circus-65f3b47d0c79

Building software development security right into the product delivery process through automated build processes is frequently referred to as DevSecOps. This portmanteau of DevOps and Security plays off the DevOps mindset of continuous improvement and continuous delivery using containers. It also includes testing security compliance targets with each code commit, in addition to testing against performance and deployability metrics.

The benefits of testing early and often are well documented. According to the Consortium for Information and Software Quality, there is often an exponential increase in cost associated with fixing issues in production instead of at the coding or integration testing phase of the delivery cycle. In fact, poor software quality cost US companies over $2.08 trillion in recent years.

Total CPSQ in 2020 in the United States of America. Image source: https://www.it-cisq.org/cisq-files/pdf/CPSQ-2020-report.pdf

Further evidence of material gains can be found in Google’s 2019 State of DevOps report. It identifies top-tier performers as companies who are able to recover from incidents 2,604x faster, deploy code 208x more frequently, reduce their lead times from commit to deploy by 106x, reduce their change failure rate by 7x, and meet their organizational goals 2x as often as their lower-tier counterparts.

Of course, creating efficient automation within a team’s software delivery toolchain doesn’t happen on its own. It takes investment upfront, support from management, and dedication. It also takes tools, culture, and some common sense.

Enabling tools

Enabling tools
Scanning code for vulnerabilities as part of the build-test-deploy pipeline is a critical enabler for CI/CD processes. Continuous, automated security scanning is so important because our development teams could never comb through every line of code or every change set. Here’s why:

Modern applications typically rely on open source libraries, which themselves rely on other third-party dependencies. Taken together, these chains of dependencies constitute a complex web of code, owned and maintained by disparate teams distributed all over the world who are often unaware of each other’s existence or how they are using code.

Neglect security scanning at your own risk. Popular package repositories like NPM have been subject to attacks, leaving developers at the mercy of repository owners (either hijacked or otherwise).

Fortunately, we have tools to help with this, including open source tools like UpGreat, which make it easy to ensure complex chains of dependencies are up-to-date.

Company culture

Company culture
But tools alone do not make a top-performing product development team — it’s also about company culture. According to the Software Security Culture in Development Teams: An Empirical Study, a culture of standards compliance in combination with structured guidelines and established values is necessary for securing success.

An effective culture can make the best possible decisions early enough in the software delivery process. No scan is perfect. We apply the scientific method to identify real threats from the red herrings, but there is still an art to it.

This is especially the case for native mobile apps. Native apps typically follow a different release cadence than web apps because they need to be updated on the device instead of simply refreshing a webpage, and they go through a more involved review process with Apple and Google. The code is much harder for potential hackers to inspect, and apps can be designed with much greater control over their runtime operation. At xMatters, we integrate continuous Static Application Security Testing with a variety of tools and services. We build vulnerability scanning into every commit through automated processes, which saves us a lot of time, most of the time.

Common sense

Common sense
Automation is not a silver bullet either. Automated scanning rulesets don’t know your use case, but they still generate findings that need to be addressed.

In the late 20th century, MIT professor Ronald Rivest authored several well-known cryptographic hashing algorithms. One of the most popular, MD-5 (the 5th iteration of the Message Digest algorithm) continues to be widely used today, despite having been shown to be “cryptographically broken and unsuitable for further use.” This was part of a wider realization that the standards defined in SHA-2 (Secure Hashing Algorithm) were inadequate, giving rise to the current standard, SHA-3. Clearly, anyone still using SHA-2 algorithms must be disregarding security best practices, right?

Not necessarily.

Hashing algorithms can be used for other purposes beyond generating unbreakable cryptographic strings. They are also good at generating unique values, such as unique device identifiers — this is how it was being used. However, it would always get flagged (rightfully so) by clients’ scans showing the usage of a “cryptographically insecure” algorithm.

Lesson learned: Don’t use security algorithms if you’re not using them for security. Use fit-for-purpose algorithms rather than general case ones to avoid the appearance of non-compliance. In our case, we’ve now switched to a hashing algorithm that is designed explicitly for performance and unique ID generation.

Striking the right balance between software development security and user experience requires plenty of critical thinking to make the best decision. Take the following example – imagine you have no other information other than the following:

What might you take away from this? Observe your initial thoughts.

When you focus only on the risk in the worst-case scenario, it is easy to forget the feature’s value. Copy-Paste is a familiar, universal function that users expect to use everywhere in their digital lives. It saves time under stressful and chaotic conditions.

In our case, the recommendation was to clear the device’s clipboard when our users switch apps. Since the xMatters app helps DevOps teams get notified and handle incidents as quickly as possible, it wasn’t long before we heard from customers who were prevented from copying and sharing complex technical logs with their dev teams to get help. We quickly re-instituted Copy-Paste, to our customers’ gratitude.

Looking to leverage integrated tools to drive software development security?

Request a demo