From Developer to Security Experience in a Cloud Native World

We often talk about the disparate experience in the security ecosystem versus the dev-tooling world. Where developer experience has begun taking center stage in the world of dev-first and cloud native, security experience is still quite lacking across the board in our ecosystem. (I would try to coin the term DevSecEx similar to DevSecOps with a focus on DevEx, but it just doesn’t have the same ring. The idea is the same, though.)

On our journey to build an open DevSecOps platform that is inherently pluggable and dev-first, there are a number of challenges that we encountered that have been repeated and reinforced by many engineering organizations. These challenges include:

• Cost
• Isolation, fragmentation and dashboard fatigue
• Time to coverage
• External to developer workflows
• Noise and vulnerability alert fatigue

There are many more pains but these come up in product discussions all the time. So why are we still here with so many excellent tools and so much progress having been made in the DevSecOps world?

What DevSecOps Can Learn from DevOps

The problems the security ecosystem is undergoing are very similar to other disciplines, and there are plenty of lessons that we can learn from other practices. When DevOps started to become widely adopted we could see an ecosystem with a jungle of tools, frameworks, clouds and much more. Just beginning to understand which tools you needed to power the growing complexity of cloud operations was a job unto itself.

Enter the DevOps engineer. They built the tooling suite and consolidated everything into well-defined processes and workflows, powered by orchestration platforms to deliver CI/CD and automation at scale.

Then managing and maintaining these operations in the long term became a pain point as well. Once the strategy, process and workflow were well-defined, the post-deployment management and maintenance needed to be covered too. Enter the site reliability engineer (SRE) that came in to ensure production operations continue as expected, provide the monitoring and observability into running services and ensure business continuity and uptime.

Once the workflows and orchestration as well as post-deployment operations were robust enough, a growing need presented itself in the form of repeatable infrastructure and self-service tools for developers. Enter platform engineering. These teams now serve to make everything we have achieved in the DevOps domain simple for all developers to adopt, with the safety and guardrails baked in.

This evolution took approximately 10 years, and today the DevOps practice is well-defined into subsets of domain expertise for the various areas of CloudOps. The security industry requires a similar evolution.

DevSecOps Consolidation First

We will start with the consolidation of tooling and orchestration platforms. Alert and dashboard fatigue became a huge issue in the world of cloud native operations, and that is why tooling to manage and orchestrate tools and platforms for large cloud fleets quickly became widely adopted.

If we take a similar look at the security industry, where CVE alert fatigue is getting out of hand and dashboard fatigue with the numerous isolated tooling dashboards, it is simply impossible to qualify, research, PoC, integrate and configure every security tool required today. Just this alone is weeks and months of work before achieving any level of security coverage.

The problem is that even the single-vendor promise of a consolidated platform comes with two distinct downsides, particularly when built from acquisitions and not internal tools built from the ground up to work together.

First off, platforms that promise to consolidate tooling and provide an end-to-end security experience suffer from the same fragmentation and isolation issues when their suite of tools isn’t well integrated, nor come configured out of the box.

So the ultimate experience for the end-user developer is the same as adopting individual tools, just through a single vendor. This means that even if you have consolidated to one vendor, the management and maintenance issue remains a pain point — and this is even before touching on the cost.

The second downside is ultimately the low quality. The reason craftsmanship commands a higher price tag — from premium watches to engines — is that those makers excel in one domain and do it flawlessly. While many large vendors have understood the issue with isolated tools and too many vendors, some have added on to their offerings, though the reality is that being domain experts in multiple areas remains difficult.

They are often tempted to acquire other companies to complement their offering, hurting the overall experience through half-baked integration, bringing us back to our previous pain point.

When each of these add-ons comes with its own price tag and deepening costs, the trade-offs start to surface. This means that even if one of the tools in the stack is best of breed, by taking a suite of tools from a single vendor, you often will find that the rest of the tooling is not up to industry standards.

We need to be able to adopt best-of-breed tools in individual domains while consolidating all of the actionable information and remediation into a single master platform.

This is because, like all of our favorite technologies, “best of breed” still tends to be subjective. Therefore, the best bet is to provide users the flexibility to pick their own tools, while preserving the overall consistency of experience and to enable users to switch to a more performant option if there is a new shiny player in the field.

This not only reduces costs, but also ensures top-tier coverage for each area of security domain expertise required today.

DevEx, Automation, Remediation

Improving developer experience is another important thing DevSecOps can learn from the DevOps world. Nobody wants eight different isolated and fragmented tools that report to different dashboards and require you to then hunt down the issue with no explicit understanding of how to do so.

Rather than different dashboards for monitoring different stacks and platforms, in DevOps we have learned to funnel all of our metrics, logging and observability into single dashboards, to receive alerts in our day-to-day tools, and to try our best to provide actionable logging errors to help reproduce and troubleshoot issues in real-time.

We need much the same thing in the security world. Along with the push to “shift left,” we need to make security easier for developers.

Developers want to stay focused on their main objective, which is to write and deploy their code to production. Anything distracting them from this task is often perceived as a time and productivity sink. No developer wants to go into some external dashboard, even more so if it ultimately sends them on a goose chase of critical vulnerabilities that need to be fixed. This just adds cognitive load, redundant work and loss of focus.

If just mitigating the high-severity issues consumes too many resources, it’s safe to say that, in most cases, low-severity issues can be ignored and only serve to create noise and CVE fatigue.

This is the primary difference between having a security-first mentality that is biased toward being exhaustive, versus a more practical approach built for the way developers want to work, and only makes visible what really matters — practical security vs. theatrical security.

Developers want to receive real-time alerts in either Slack or GitHub and resolve issues immediately when high-risk vulnerabilities are introduced. This allows them to fix those vulnerabilities in the context of their work, reduce the security overhead and handle them much like any other software bug.

Once they move on to their next task after committing or deploying their code, the context switch of coming back to a previous task or project becomes very expensive for the organization.

This approach is what we call a fix-first mindset, derived from lessons learned around MTTR — mean time to restore (and remediate). The faster we fix an issue, the less cognitive load it creates and the less risk to the system.

By automating remediation through dev-native tooling and channels, you can keep developers in the flow and ensure that issues are addressed, not pushed to the backlog for hunting down later. This improves overall security posture and makes long-term vulnerability management much more feasible and actionable, as well as sustainable in the long-term. (It’s like the zero-inbox equivalent for security.)

From Code to Cloud

To gain a comprehensive understanding of your security posture, much like the consolidation of Dev and Ops, you cannot have isolated views of the different parts of your product stack. This includes code, CI/CD, third parties, infrastructure and runtime. A truly viable cloud native security platform will provide visibility end to end of your security posture from the first line of code through your cloud configurations, deployment and ultimately runtime.

This also isn’t just limited to the aspects of code, which are important, but also policies, identity and access management, and many other aspects. Just like you can’t decouple your software engineering from your business goals, you cannot decouple pieces of your security puzzle from each other if you really want to move the security needle. Your code needs to be as secure as your cloud, alongside your governance and policy.

How to Level Up the DevSec Experience?

For starters, explore tools that will enable you to consolidate your security stack and dashboarding to the minimal viable requirements (and add more as tolerance and maturity increase), but do so without compromising security. Orchestration platforms will be key to achieving security with best-of-breed tools while consolidating and centralizing the management.

Next, don’t compromise the developer experience. If remediating and managing security overloads your engineers and defocuses them from the end goal of software delivery, then you simply aren’t doing it right. This also means you will likely fail in implementing your security programs; you either won’t be able to secure your systems or your developers will leave out of frustration.

Your security tooling needs to play to your developers’ strengths. Leverage the places where they spend most of their time and ensure they receive the alerts and suggested fixes where they can actually handle them.

Last, just like building a holistic product, your security needs to be viewed in much the same way. Isolating different parts and decoupling them will ultimately lead to ineffective security strategies and programs. You need to have a comprehensive end-to-end view of all the layers of your product stack to be able to understand your security posture and close the gaps when necessary.

Once we think about DevSecOps the same way we think about DevOps, we will be able to unlock next-level security experience for our developers.

‍

Originally posted on The New Stack, Aug 22nd, 2023.

‍

‍