Managing Application Security Complexity in Large Organizations

In 1999, I was a young Private First Class fresh out of a programming school that was based out of Twentynine Palms, California. I had just spent the last few months in a crash course on the ADA programming language, which was, at the time, the most widely used language for the most critical US military defense systems. It was a hard-hitting daily eight-hour course where we learned how to build flexible and modular code for increasingly complex tasks. Our final project was to make a fully functional ATM system, and our class became so good at coding that many of us had points taken away for being a bit too ambitious. One classmate had the ATM display a complex 3-minute ASCII art cartoon before the user reached the main menu. Our instructor was a massive and imposing staff sergeant who did not appreciate our shenanigans. Still, it’s safe to say our entire class was pretty smug about it, and we all thought we were some hot-shot developers. Despite the hot weather, hard training, and our reckless junior military lifestyle, the months we spent in that school were a programmer’s dream. However, one thing we never had to worry about was application security. The ADA language allowed us to control what went in and out of memory carefully, and many systems were completely closed or connected to mainframe systems in batch jobs to share updates.

Let’s fast forward 25 years. Up to 2024, software development has advanced at breakneck speeds, with massive changes in the security perimeter and an unyielding and frantic adoption of AI. Application security has taken a back seat to GRC and cloud security, and even organizations that have adopted modern AppSec processes and tools are woefully overwhelmed. Having worked with large organizations, I’ve seen some do application security very well, some do it poorly despite having a large budget, and some ignore it completely. I want to talk about these challenges and ways organizations can simplify their security program for application development.

Challenge 1: An Overwhelming Number of Security Alerts

I recently worked at a large healthcare organization trying to figure out how to handle tens of thousands of security vulnerability alerts per application. Developer groups were clearly aware of the vulnerabilities, but reducing them was too massive of a project to consider. Deadline impacts made everyone run for the hills whenever the subject was brought up. I would joke that it would take more developers than the company even had to fix the security vulnerabilities in our backlogs. Their solution, which I disagreed with, was to create a massive dashboard of all vulnerabilities with automated Jira ticketing integrated into every team. If you are a developer, you probably feel some pity for the poor guys with a permanent multi-thousand-ticket backlog for the rest of their career at that firm.

So, what is the right solution? You can’t get rid of software vulnerability scanning software, or you would have zero visibility into the organization’s application security posture. You also absolutely need to know if there are any critical vulnerabilities so your team can fix them immediately, hopefully with the support of an experienced security developer. Well-run security organizations rely on multiple layers of security review before the first SAST or SCA tool is even run. This includes a structured Security Architecture Review process for all development lifecycles. The best way to do this is through Threat Modeling. A proper threat model will fully map the software architecture and create a visible picture of how the applications communicates, which protocols and dependencies are used to do so, and how that affects the application's security. It will also be your first line of defense against major software vulnerabilities, and a proper threat model will report on a list of vulnerabilities your application has and the exact steps to take to resolve them in the context of the application.

This shifts most of your code vulnerability remediation work to a shared specialist instead of wasting the precious time of your dedicated development professionals. The threat modeling process should be done alongside other security reviews, such as GRC and cloud security, to create a comprehensive first-line defense against attack vectors. By the time you get to a vulnerability scan, the white noise will be drastically reduced.

Challenge 2: Core Developer Team Disruption

Most developers did not sign up to be security professionals and are too busy solving complex challenges to get their products out the door on time. If you’ve read this far, you can probably already see that these security processes greatly strain product development teams. Not only are developers being overloaded with unforeseen vulnerability tickets forced onto their backlog, but they also have to participate in time-consuming security review and approval processes. Simply assigning a developer as a security champion doubles their workload, and at the end of the day, security will never be their higher priority.

What do well-run companies do to resolve this? They provide dedicated security professionals whose only job is to remediate software vulnerabilities. These team members don’t throw work over the wall to core developers and instead take the time to understand the applications they are assigned to as semi-persistent team members. They are coders who love security and know how to dive into the code or fully support developers to remediate vulnerabilities. Ultimately, they work for the development teams they are assigned to, not the other way around. I’ve spent quite a lot of time and effort building these kinds of teams, so I know it’s not easy, but it beats the alternative and ultimately saves money.

Challenge 3: Too Many Scanning Tools

In application security, there are three general vulnerability scanning tool types: Static Application Security Testing (SAST), Dynamic Application Security Testing (DAST), and Software Composition Analysis (SCA). There are more specialized tools, but that falls outside this article's scope. You probably know everything about these tools and the various options if you are reading this, but I encourage you to research them if you don’t. They have a valuable place in the tooling ecosystem of any large development organization. Problems start to occur when the same tools are used for overlapping purposes. The tools are not always used through an established process, so sometimes, there will be multiple scans on the same applications with numerous tools. I’ve seen organizations with various instances of Coverity, SonarQube, and Black Duck scans in numerous branches on the same code base. This not only created an unrealistic vulnerability picture but hid the most critical vulnerabilities that needed to be addressed immediately behind a massive wall of white noise.

The solution is to streamline, reduce, and standardize vulnerability scanning tools across the organization. Big enough organizations will have multiple development teams using their tools, primarily because some security organizations have traditionally only felt responsible for dictating policy, not how it is adhered to. Many organizations must address this because it divides product and security teams. This segways beautifully into my final major challenge in application security.

Challenge 4: Lack of Security Process Uniformity Across the Organization

A side effect of giving development teams much-needed autonomy in their work is that tooling use grows out of control. Multiple licenses of the same tool are purchased across the organization in different development organizations and individual teams. Some developers run manual scans on their code after major releases, some integrate those scans into their pipeline, some automate them outside the pipeline using another third-party tool, and some don’t even bother running scans at all. In the same organization, some have threat models and architecture reviews, some don’t, some have a budget for security, and some don’t have any security budget at all. Some listen to and work well with their security team, while others tune security out. The breakneck speed of everything shift-left took away almost all of the well-established cybersecurity processes and left a mess for those left to clean up.

This is more of an organizational process problem than a technical problem. The solution needs to fit in with your unique culture, but I can give you an idea of how to solve it. Consider creating a core DevSecOps team responsible end-to-end for all things application security. Put your security champions on it and allocate a training budget to make them a world-class cybersecurity team. Avoid lofty titles such as “AppSec Center of Excellence” to avoid the trap of new silos, management kingdoms, and bureaucracy. Embed this team in every development group you have with a service mindset. Once again, this team should work to make the core developers’ lives easier, not harder. This will also attract world-class security developers who want to be part of a real cybersecurity team. This work stream would fit nicely with the vendor model, which may be more inclined to treat the developer groups as valued clients. This model doesn’t fit every organization, but it should give you a starting point to brainstorm your own organizational processes.

In Conclusion…

I hope this article was helpful for cybersecurity professionals and managers. These days, I am not the hot-shot developer I thought I was back in ’99, and now I almost exclusively work with people and processes. I was there when the web went from static pages to fully dynamic functional applications and got to be a part of the security process every step of the way. Now that GenAI is here, I see a new set of Chatbot monitoring and data protection processes. Data teams, I’m sure your life has gotten a lot more interesting in the past year as GenAI is trained and stores metadata in a way that isn’t quite well understood just yet. This is a good topic for another time, so for now, good luck out there, Cybersecurity professionals!