The Security Tooling Faceoff — Open Source Security vs. Commercial

The shift-left movement has done wonders with advancing many engineering disciplines over the past decade, and none have seen more progress than the security discipline, with regards to shifting actions left of production. One of the first and biggest proponents of shift-left security was Snyk, which came to market with a novel approach to opening pull requests (PRs) inside the developer workflow to remediate CVEs found in open source packages, and we’ve taken this further and spoken about born-left security.

Since first launching its SCA scanner for open source, which was their claim to fame, Snyk has added quite a few tools to its suite to provide more extensive security. In this post, we’ll take a look at how the industry has evolved from a security perspective, and where we still need to improve and level up our developer experience.

What Developers Really Need

Auto-remediation through opening PRs taught the industry a lot about what security experience needs to look like for developers to truly adopt it — we also wrote about this security experience and what developers really want. We have evolved as an industry with the way we expect developers to receive security alerts, and respond to them. We have come a long way in understanding what developers need in order for the tools to be useful. They don’t want fragmented tooling, too many alerts or dashboards, they need to have immense trust in the quality of the results of their security tools, and have the tasks they need to address keep them in context and flow at all times.

Developers have also evolved over the years. There is almost no security-minded engineer today that doesn’t know that for their products and platforms to be secure, they need more than just third-party OSS package scanners. They now know that they need to scan their code, their IaC and cloud environment, their containers and Kubernetes clusters, and even go as far as doing some dynamic web app security testing in runtime. All this, just to achieve the most baseline security when shipping code and products to customers.

Yet, it’s interesting to note that one of the biggest problems in our industry is still fragmentation. That’s because each and every single one of these security capabilities is a company and domain expertise unto itself. It’s extremely difficult to excel in all of these domains simultaneously, and also provide this with a seamless developer experience.

This is where we’re starting to see the world of single-vendor security platforms break down. Today’s security suites from the largest vendors in the industry now seemingly provide this end-to-end suite of capabilities in a single shop, but do they really stack up?

Benchmarking Security Platforms against Open Source Alternatives

When building Jit from the ground up, it was clear to us that this consolidation approach is indeed the right approach to security, but the execution is ultimately what determines if we’ve succeeded as an industry. We had the privilege of doing a lot of research — yes that critical piece in R&D that is often overlooked — when building a truly viable security orchestration platform. We wanted to explore the world of commercial tools vs. open source tools and understand what to provide out of the box for our users to level us up as an industry.

The results were astounding. While we found that companies with a traditional core offering benchmarked well against open source tools, the compelling data shows that additional tools added to the suite to provide greater end-to-end coverage, rarely stacked up (if at all) against best-of-breed open source tools.

Examples of this include tools like Kubescape and KICS for Kubernetes manifest file scanning — Kubesape was superior alongside KICS in second place, ahead of other commercially available tools in the market. This was compounded when adding dedicated rules curated by the Jit team with AI, to enhance detection (which is not possible with closed-source commercial products).

This was also true for code scanning, and even with languages promoted as well-supported like Python. Open source Semgrep performed better across a wide range of programming languages, and particularly the less supported ones in the commercial tooling — despite being widely adopted ones — like Scala.

This just strengthened our working hypothesis that developers should have the freedom to choose the tools in their stack, and not have this dictated by an engagement with a single supplier. Our security cannot be compromised because of the vendor we choose to buy from.

True Shift Left for Developers

As we came to this conclusion, we understood that the best platform we could provide for our users (the developers) would have this QUADfecta of capabilities:

• The breadth of coverage;
• Orchestration, unification, and extensibility in a single platform;
• Auto-remediation in context for developers;
• Developer experience — DevEx all the things!

When we talk about the breadth of coverage, it’s clear that we can’t compromise or prioritize one part of the stack over the other. We know the code is equally as important as the open source packages, as the cloud configurations to which the software is deployed. This can only be achieved by enabling developers to choose any tool they want — open source or commercial — and also curate and extend a really great set of open source controls out of the box. Which brings us to the next capability.

It’s not enough to just orchestrate a predefined set of tools and leave it at that. There are several aspects at play to truly deliver the experience developers need and expect. First, you need to unify the output and schema — each tool has a different way of reporting and alerting about issues, oftentimes through exhaustive lists of findings (that need to be filtered and prioritized and later assigned as well). Next, you have to be able to support the addition of new tools and coverage all the time in a seamless way and make it possible to bring your own homegrown tools.

Even if you provide a really sleek UI with all the data unified in a human-readable format, with the data prioritized, most developers don’t work this way, in any case. They want the critical alerts to arrive in context and upon other gating and compelling events — like when a PR is created. Our fix-first mindset is built to point you to the exact problematic line of code and offer code fixes and fix guidelines inside PRs, so you don’t have to go hunt findings down later in some UI somewhere.

On top of this, a lot has been invested in making alerting more humane (no more alert or dashboard fatigue!), and enabling developers to only use the UI when they want or need to — and certainly not have to log in to several fragmented UIs to get the full picture of their security posture.  Instead, developers can receive alerts directly in Slack, Github or JIRA, or even send tasks to JIRA to handle later — so security integrates with their existing tooling and workflows and doesn’t require them to leave context to address security.

Swiss Army Knife Security

We’ve come a very very long way in the security industry, shift-left security is well understood, and excellent tooling is now available to cover the many layers of a modern software stack. This makes it possible to now take our security capabilities to the next level, and unlock much-needed developer experience for the security world — security experience. Developers are the new tooling decision-makers, and if we don’t optimize for the humans in the system, we will be setting our security programs up for failure.  By providing a similar experience and approach to tooling that has been available for the DevOps and engineering world for some time, we will enable security to move at the same pace and velocity of high-scale engineering teams.