Managing Application Security Complexity in Large Organizations

In 1999, I was a young Private First Class fresh out of a programming school that was based out of Twentynine Palms, California. I had just spent the last few months in a crash course on the ADA programming language, which was, at the time, the most widely used language for the most critical US military defense systems. It was a hard-hitting daily eight-hour course where we learned how to build flexible and modular code for increasingly complex tasks. Our final project was to make a fully functional ATM system, and our class became so good at coding that many of us had points taken away for being a bit too ambitious. One classmate had the ATM display a complex 3-minute ASCII art cartoon before the user reached the main menu. Our instructor was a massive and imposing staff sergeant who did not appreciate our shenanigans. Still, it’s safe to say our entire class was pretty smug about it, and we all thought we were some hot-shot developers. Despite the hot weather, hard training, and our reckless junior military lifestyle, the months we spent in that school were a programmer’s dream. However, one thing we never had to worry about was application security. The ADA language allowed us to control what went in and out of memory carefully, and many systems were completely closed or connected to mainframe systems in batch jobs to share updates.

Let’s fast forward 25 years. Up to 2024, software development has advanced at breakneck speeds, with massive changes in the security perimeter and an unyielding and frantic adoption of AI. Application security has taken a back seat to GRC and cloud security, and even organizations that have adopted modern AppSec processes and tools are woefully overwhelmed. Having worked with large organizations, I’ve seen some do application security very well, some do it poorly despite having a large budget, and some ignore it completely. I want to talk about these challenges and ways organizations can simplify their security program for application development.

Challenge 1: An Overwhelming Number of Security Alerts

I recently worked at a large healthcare organization trying to figure out how to handle tens of thousands of security vulnerability alerts per application. Developer groups were clearly aware of the vulnerabilities, but reducing them was too massive of a project to consider. Deadline impacts made everyone run for the hills whenever the subject was brought up. I would joke that it would take more developers than the company even had to fix the security vulnerabilities in our backlogs. Their solution, which I disagreed with, was to create a massive dashboard of all vulnerabilities with automated Jira ticketing integrated into every team. If you are a developer, you probably feel some pity for the poor guys with a permanent multi-thousand-ticket backlog for the rest of their career at that firm.

So, what is the right solution? You can’t get rid of software vulnerability scanning software, or you would have zero visibility into the organization’s application security posture. You also absolutely need to know if there are any critical vulnerabilities so your team can fix them immediately, hopefully with the support of an experienced security developer. Well-run security organizations rely on multiple layers of security review before the first SAST or SCA tool is even run. This includes a structured Security Architecture Review process for all development lifecycles. The best way to do this is through Threat Modeling. A proper threat model will fully map the software architecture and create a visible picture of how the applications communicates, which protocols and dependencies are used to do so, and how that affects the application's security. It will also be your first line of defense against major software vulnerabilities, and a proper threat model will report on a list of vulnerabilities your application has and the exact steps to take to resolve them in the context of the application.

This shifts most of your code vulnerability remediation work to a shared specialist instead of wasting the precious time of your dedicated development professionals. The threat modeling process should be done alongside other security reviews, such as GRC and cloud security, to create a comprehensive first-line defense against attack vectors. By the time you get to a vulnerability scan, the white noise will be drastically reduced.

Challenge 2: Core Developer Team Disruption

Most developers did not sign up to be security professionals and are too busy solving complex challenges to get their products out the door on time. If you’ve read this far, you can probably already see that these security processes greatly strain product development teams. Not only are developers being overloaded with unforeseen vulnerability tickets forced onto their backlog, but they also have to participate in time-consuming security review and approval processes. Simply assigning a developer as a security champion doubles their workload, and at the end of the day, security will never be their higher priority.

What do well-run companies do to resolve this? They provide dedicated security professionals whose only job is to remediate software vulnerabilities. These team members don’t throw work over the wall to core developers and instead take the time to understand the applications they are assigned to as semi-persistent team members. They are coders who love security and know how to dive into the code or fully support developers to remediate vulnerabilities. Ultimately, they work for the development teams they are assigned to, not the other way around. I’ve spent quite a lot of time and effort building these kinds of teams, so I know it’s not easy, but it beats the alternative and ultimately saves money.

Challenge 3: Too Many Scanning Tools

In application security, there are three general vulnerability scanning tool types: Static Application Security Testing (SAST), Dynamic Application Security Testing (DAST), and Software Composition Analysis (SCA). There are more specialized tools, but that falls outside this article's scope. You probably know everything about these tools and the various options if you are reading this, but I encourage you to research them if you don’t. They have a valuable place in the tooling ecosystem of any large development organization. Problems start to occur when the same tools are used for overlapping purposes. The tools are not always used through an established process, so sometimes, there will be multiple scans on the same applications with numerous tools. I’ve seen organizations with various instances of Coverity, SonarQube, and Black Duck scans in numerous branches on the same code base. This not only created an unrealistic vulnerability picture but hid the most critical vulnerabilities that needed to be addressed immediately behind a massive wall of white noise.

The solution is to streamline, reduce, and standardize vulnerability scanning tools across the organization. Big enough organizations will have multiple development teams using their tools, primarily because some security organizations have traditionally only felt responsible for dictating policy, not how it is adhered to. Many organizations must address this because it divides product and security teams. This segways beautifully into my final major challenge in application security.

Challenge 4: Lack of Security Process Uniformity Across the Organization

A side effect of giving development teams much-needed autonomy in their work is that tooling use grows out of control. Multiple licenses of the same tool are purchased across the organization in different development organizations and individual teams. Some developers run manual scans on their code after major releases, some integrate those scans into their pipeline, some automate them outside the pipeline using another third-party tool, and some don’t even bother running scans at all. In the same organization, some have threat models and architecture reviews, some don’t, some have a budget for security, and some don’t have any security budget at all. Some listen to and work well with their security team, while others tune security out. The breakneck speed of everything shift-left took away almost all of the well-established cybersecurity processes and left a mess for those left to clean up.

This is more of an organizational process problem than a technical problem. The solution needs to fit in with your unique culture, but I can give you an idea of how to solve it. Consider creating a core DevSecOps team responsible end-to-end for all things application security. Put your security champions on it and allocate a training budget to make them a world-class cybersecurity team. Avoid lofty titles such as “AppSec Center of Excellence” to avoid the trap of new silos, management kingdoms, and bureaucracy. Embed this team in every development group you have with a service mindset. Once again, this team should work to make the core developers’ lives easier, not harder. This will also attract world-class security developers who want to be part of a real cybersecurity team. This work stream would fit nicely with the vendor model, which may be more inclined to treat the developer groups as valued clients. This model doesn’t fit every organization, but it should give you a starting point to brainstorm your own organizational processes.

In Conclusion…

I hope this article was helpful for cybersecurity professionals and managers. These days, I am not the hot-shot developer I thought I was back in ’99, and now I almost exclusively work with people and processes. I was there when the web went from static pages to fully dynamic functional applications and got to be a part of the security process every step of the way. Now that GenAI is here, I see a new set of Chatbot monitoring and data protection processes. Data teams, I’m sure your life has gotten a lot more interesting in the past year as GenAI is trained and stores metadata in a way that isn’t quite well understood just yet. This is a good topic for another time, so for now, good luck out there, Cybersecurity professionals!

An Unfortunate Truth: Your Cloud Security Team is Not Ready

I'm going to come right out and say it. Your cloud environment is not secure. Career-wise, this may be a risky subject for me to bring up in public, but I think it is time we all came to terms with it. This hard truth has come from a 20-year career of working with many fantastic Fortune 500 companies as a software developer, DevOps engineer, security architect, and engineering technical-lead. You may have a great security team, and you may have a great cloud enablement team, but they are not prepared to properly secure their cloud in the era of distributed Infrastructure-as-Code deployments. This is a bold statement, but I can back it up after spending last decade assessing and trying to secure cloud environments for Fortune 50 clients, cloud providers as a third-party auditor, and working for various consulting groups such as Deloitte.

I have been on the ground floor of security; working with management teams, CISOs, and frustrated and shackled security engineers to try to overcome major cloud security hurdles. The truth is that no one has cloud security figured out, and most organizations are not even remotely ready to tackle the huge security risk that is their enterprise-level cloud environment. This sounds pretty alarmist, but it is a reality that we all need to come to terms with. In this article I want to try to give you a good 10,000-foot overview, and hopefully you will see not only the problem, but the solution by the end.

Security is undergoing a massive shift as organizations move their workloads over to the cloud. One of the biggest challenges that security teams have is the removal of organizational silos for the different domains they work with. In traditional on-premises or co-located environments, Identity, VM infrastructure, and networking security standards are generally managed and enforced through respective teams. This has provided security with a single point of contact for each security domain. Now however, cloud environments are distributed with VMs, networks, services, and other infrastructure deployed and decommissioned by individual developers and business units.

This decentralization of responsibilities is cause for an amount of concern for security executives as it creates an unmanageable amount of coordination between security teams and a multitude of developer teams. Traditional IT teams have lost control over their security process as infrastructure already has been “Shifted to the Left” to developers through Infrastructure-as-Code deployments. A new paradigm of security management has begun to emerge through this in the past couple of years, and you have probably heard about it in various forms. SecOps, DevSecOps, and a combination of both are being adopted throughout the industry as you read this.

None of these methodologies work yet because real modern-day distributed cloud-security challenges have not yet been properly recognized in the industry. Every organization has smart engineers on the ground, but they are overloaded with too many organizational units and challenges to effectively integrate these solutions into their cloud environments. The hard truth is that if you are a CISO, you need to know that your team isn’t ready for this, nor are they equipped to handle it. It is all well and good to have security requirements at the Development level, but how do you enforce them? Your security team probably put in a lot of effort coming up with cloud security controls, but there is a vanishing chance that they have any way to make them a reality.

The challenges and questions that come up in doing so are almost overwhelming. How are security requirements enforced across environments spanning multiple cloud providers? How is access to these platforms and environments restricted in a way that does not slow down the development lifecycle? How can the security team reduce exceptions and enable teams to deploy infrastructure and services in a way that is guaranteed to satisfy their controls?

The answers to these questions come through a thoughtful and enterprise-wide application of tools and processes. Multi-cloud environments need multi-cloud tools that are flexible enough for developers to use, but rigid enough to give security back the control they need. These tools include DevOps pipelines, multi-cloud Identity Access Management, IT Service Management, deployment orchestrators such as Terraform, and the integration and automation of security event detection and response through SIEM and SOAR tools. Above all, your team needs to have the courage to enforce security guardrails, and be ready to deal with pushback from all over your organization.

Only when these tools are established and adopted across the organization can security protocols be reliably enforced. Security configuration of cloud services can be templatized and pushed at the deployment level through the orchestrator. Networking and routing controls can be put in place through service requests. Identity controls can be established using the principle of least privilege through pipeline deployments. Security events can be immediately detected and resolved through automated response playbooks.

What is exciting about this all is that we are truly on the cusp of a new era where security is becoming both easier and more transparent to enforce without causing any organizational friction to do so. When done right, your cloud workloads will be secure, monitored, and automatically protected from malicious attackers or integrated into response ticketing systems. Most likely, you aren’t there yet, and to get there your organization needs to dramatically rethink the way it does security.