In Michael Feathers’s talk “The Deep Synergy Between Testability and Good Design,”he discusses how design and testability are related. Good design results in code that’s fairly easy to write tests for. However, when an application has many design issues, writing tests becomes a painful process. Areas of an application that are difficult to write tests for are the areas that can benefit from refactoring. It’s these same areas that are also brittle, making it hard to fix bugs or add features to them.
If you’re wondering what testing and code quality have to do with application security, read on. They have everything to do with it. These brittle areas are a veritable petri dish of security vulnerabilities. When code is brittle, it’s due to a lack of quality in the design and implementation, and this same brittleness also leads to more security vulnerabilities.
Before diving into the security aspects of testing, let’s review how common testing strategies and code quality are related.
Common Testing Strategies
Testing is a critical stage in the delivery of any product, and not all testing has the same needs in mind. There are three general areas of testing, and each plays an important role in software quality.
Traditional Quality Assurance (QA)
This is the status quo in most organizations. Write some code, build an artifact, toss it over the wall for manual human testing. This testing relies on business requirements and the experience of the person performing the testing. This form of testing is ideal for issues with usability, workflow, and business logic.
Manual testing, while great for some things, doesn’t scale very well. No one can possibly test all the possible inputs and code paths manually and effectively. This is especially true when new releases are going out daily.
In order to compensate for faster release cycles (and to save us from carpal tunnel), testing strategies involving automation are the way to go.
People often conflate unit tests with integration tests. Unit tests should only test small sections of code, such as a single function or class method. Unit tests shouldn’t access files, networks, or databases. These tests run when developers are writing code as well as any time the application runs through the build cycle. Most integrated development environments (IDEs) also have built-in support for unit testing. This provides immediate feedback to developers to fix whatever is wrong before committing code.
Integration tests generally depend on external components such as a database or web service. These tests don’t run as quickly as unit tests and may only run at specific intervals in the build lifecycle.
Integration tests are typically what a lot of application test suites look like. It’s generally easier to write integration tests for applications that don’t have tests than to jump straight into unit tests. It can be time consuming to refactor parts of an application to create effective unit tests.
Measuring Code Quality
Applications have a multitude of components, all with varying levels of quality and complexity. Intuition can often tell us the complexity of a given piece of code. However, having data to back up that intuition is always helpful.
Measuring code quality is not an exact science. No single measurement should serve as a barometer of the quality of an application’s codebase. Not all measurements are useful either. If there were only three measurements to keep an eye on, they would be code complexity, churn, and test coverage.
Measuring code complexity effectively tells us how difficult the code is to understand. Complex code is harder to understand and to maintain. One way of measuring complexity is by counting the number of nested control structures in a piece of code. A single if statement or for loop may not contribute much, but deep nesting of conditional or control flow statements can result in code that’s difficult to understand.
Code churn is a measurement of how much code is changing by looking at the number of lines changing between source code revisions and the locations of the lines. Churn identifies areas where bugs are most likely to occur and areas that are in need of refactoring.
If the same files (or classes) are continually changing between releases, the corresponding code most likely isn’t adhering to the single responsibility principle (SRP). These pieces of code can be broken into smaller, and thus more maintainable, pieces.
Unit Test Coverage
This is the percentage of code covered by unit tests. Knowing which areas of the codebase have unit tests helps prioritize areas that need additional testing. It also points to ineffective tests or tests that are missing certain code paths.
Security Vulnerabilities Aren’t Special
So how does this all relate back to security testing? Well, here’s a little secret: s*ecurity vulnerabilities are just bugs. *It doesn’t matter if the vulnerability is cross-site scripting (XSS), SQL injection, or resource exhaustion. They’re all just coding mistakes that are classified as a danger to the confidentiality, integrity, or availability of the application.
Security bugs in applications generally fall into two broad categories: defects in the code or flaws in business logic. A code defect might be forgetting to release a resource, resulting in a memory leak. A customer bypassing a check that allows for additional discounts on an order would be a flaw in business logic. This distinction is important.
Following best practices and utilizing the right tools can prevent many common vulnerabilities. Flaws in this category, such as injection flaws, typically manifest the same way across different applications. As such, they can be identified without understanding the specific business logic of the application.
Off-the-shelf tools are unable to find flaws in business logic with a lot of success. Since each application has different logic, custom tests need to be written. This is ideally where development and quality assurance teams work together to create custom tests.
In order to fix bugs, code has to change. This code then has to undergo another round of testing before its release. How long does this process usually take? A few hours? Days? Weeks? Wait … months?
What if a bug results in the exposure of critical information, such as credit card numbers or health records?
Improving Security by Focusing on Quality
“It has been shown that investments in software quality will reduce the incidence of computer security problems, regardless of whether security was a target of the quality program or not.” —Ross Anderson, Security Engineering
Improving the quality of code is directly related to improving the security posture of the application. The higher the quality of the codebase, the easier it is to test and maintain. The smart people at MITRE came to the realization that a lot of issues in source code don’t quite fit into the existing Common Vulnerability Enumeration (CVE) or Common Weakness Enumeration (CWE), but still left the reviewer with a bad taste. These are issues that, with a slight tweak or mishandling of data, would result in a security vulnerability. Since these issues don’t quite fit into existing enumerations, they are working on the Common Quality Enumeration (CQE). This enumeration enables a common language for discussing these particular issues. In the CQE are issues related to quality, such as excessive complexity and class inheritance issues, as well as performance issues.
While no off-the-shelf tool is going to be adequate for testing business logic issues, there are tools that can aid in identifying code-level flaws. These tools can be run manually or, preferably, as part of a continuous integration pipeline.
Static Application Security Testing (SAST)
Security testing using static analysis is done at the language level, through either parsing the language or looking at the produced machine code to identify common problems. These tools are built to work with specific languages with most commercial tools supporting multiple languages. Some of these tools will also report on code quality issues along with security issues.
Dynamic Application Security Testing (DAST)
Most application security assessments fall into the dynamic analysis, or black box, testing strategy. This is from the point of view of an attacker without any knowledge or access to the application source code.
Open-source and commercial tools in this category generally are looking for architectural or integration issues. For example, testing for cross-site scripting or SQL injection flaws is done with tests that are known to exploit certain conditions. DAST tools are also excellent at identifying common issues between the application and the rest of the environment—issues such as TLS settings, HTTP header configurations, or information exposure.
Runtime Application Self Protection (RASP)
These tools are not as common as SAST and DAST tools. RASP tools are designed to instrument the application code to try and identify unusual activity or an active attack. Once identified, the tool can be configured to carry out actions, such as immediately blocking the requests or performing some other action. Since they are closely integrated with the application, they can be difficult to set up and work with. In some cases, the RASP tool will modify the code being run, which can make troubleshooting difficult or, in the case of a false positive, block legitimate traffic.
Application testability and security are intertwined. Well-written applications are all-around easier to work with and result in fewer bugs and therefore fewer vulnerabilities. The more testable an application is, the easier it is to be proactive and fix issues before they become security vulnerabilities (and land you on the news). Before investing time and money in expensive tools, it’s a better idea to invest that time into streamlining the application release cycle. This ensures that fixes make it to production quickly and, most importantly, safely.
This is just scratching the surface on security testing. It’s a long process to take an application with little security testing to a place that instills confidence. Plan for a marathon and not a sprint.