Building a more robust security focussed deployment pipeline
I bored myself writing that title. If you’re still here, we’re following on from the previous post about why WAFs still exist, and whether they add any value to your infrastructure.
If you know the enemy and know yourself, you need not fear the result of a hundred battles. If you know yourself but not the enemy, for every victory gained you will also suffer a defeat. If you know neither the enemy nor yourself, you will succumb in every battle.”
Sun Tzu, The Art of War
After recently deploying and testing the new AWS WAFv2, I decided to dig deeper to see if other alternative security layers add any value to applications that have public APIs.
Bringing the stakeholders inline
We’ve all worked with large teams where risks are put on backlogs, and the technical debt pile grows as you see features prioritised and maintenance work pushed further down your Jira board.
If you are part of the development process or manage the platform, it is your responsibility to make sure the stakeholders understand why security and operability matter. They need to understand the importance of data, what the risks are and decide what resources are required to keep on top of these risks.
Aside from your maintenance and technical debt pile, the cost of an unpatched security vulnerability or an unattended API scraping attack isn’t immediately obvious. This needs to be visualised in a way that makes sense, not only to your stakeholders but to you and your team so that you can prioritise your backlog.
Start assessing the risk, giving it a rating by comparing the best and worst case scenarios, and providing context.
- Does the risk have any public precedent? Are there any documented cases or scenarios you can refer to?
- Is there a way to quantify or T-shirt size the effort involved in putting a short term fix in place.
- Is there an obvious long term fix and is your team equipped to deploy it?
Document and visualise the risks and patches where possible. Having your stakeholders on your side makes operability and maintenance a much easier, and dare I say, enjoyable task.
IAST and RASP
IAST stands for Interactive Application Security Testing and IAST tooling allows security vulnerability checks to be run with automated tests during and after your application deployments.
Gartner defines RASP as “a security technology that is built or linked into an application or application runtime environment and is capable of controlling application execution and detecting and preventing real-time attacks”.
Unlike a WAF, which lives at the edge of your network, RASP tools are designed to run in the application security management layer and uses mechanisms like dynamic binary analysis, JVM replacement and virtualisation for better decision making.
In simple terms, it is a security layer that lives closer to the application, which provides it with better visibility to the calls being made to the application and therefore, more information to block malicious calls.
Convincing your developers
You can’t introduce a new runtime without convincing your developers and getting them onboard. You’ll need to consider if the RASP implementations offered by the various vendors actually are as seamless as they’re made out to be. Your developers need to understand if a change like this affects the development cycle.
There is also almost always a case that needs to be made about changing something close to the application runtime instead of adding one of the next generation WAFs.
WAF vendors have been working on addressing automated tuning, monitoring and API protection too so making a case for a RASP needs to be balanced with your team’s risk tolerance.
Performance Degradation
Performance degradation concerns are legitimate in some cases. Additional latency and compute requirements for this runtime change need to be quantified.
In recent studies using the the Contrast RASP agent with a Node app deployment. Turning on blocking or monitoring mode can add up to 3 times the overhead in response time per request. This is due to the nature of the RASP deployment.
The smallest application response time increase was 55%. This could potentially be a deal breaker. I would however suggest setting up a test bed and following the procedure highlighted below.
- Test your application performance in request time per second with a standard set of user journeys without a WAF or any network hops.
- Deploy your application with your current network stack, and measure this again.
- Run the application with your automated “malicious” user journeys and make a note of this too.
- Now deploy your RASP and run the same set of tests again.
For a lot of teams, this is should provide enough feedback to go ahead with a more refined proof of concept.
Language support
Some languages and platforms do not have mature offerings. This could be an immediate deal breaker. Gartner provide a list of twenty vendors with RASP capabilities and this should be the starting point for your proof of concept.
Next Generation WAFs
We previously discussed the four generations of bots. These next generation WAFs provide better intelligence when dealing with crafty bots and malicious actors that know what they want from your infrastructure.
For any application that is getting scraped or experiencing credential stuffing, you want some level of anti-automation defense.
There are multiple vendors that provide additional bot detection capabilities. Here is the recommended list of vendors that I would start with.
- Shape Security
- Imperva - Distil Networks
- Signal Sciences
- Reblaze
- Fortinet
- F5
These WAFs need to be at the edge of your network. Testing them requires getting a trial and having a testbed ready to test the effectiveness of your WAF.
Avoid SOC Fatigue
Your security operations centre team are short on staff, resources and visibility in a lot of cases. The fact that most security “bundles” add to their pain instead of reducing it should be a concern for most application development teams.
When deploying any solution, the acceptance criteria for completion needs to include a list of achievable goals.
- A clear rollout plan which includes additional visibility for all first responders on your team.
- Additional runbooks for out of hours support. This should be put together with the help of your SOC. This needs to be adjusted based on the skillset of the first responders.
- A set of feedback loops that the SOC has the ability to participate in. This includes any logging, tuning, training or performance measurement tweaks.
Securing your platform and addressing your risks might require additional human interaction. This is something you should always keep in mind.
Agile Security Operations
When addressing the security infrastructure for any team, I start the process by making sure I have a few prerequisites in place.
- A mechanism to test the current state of your application.
- A testbed or sandboxed environment that is like for like for your current infrastructure, but that can be torn down and built back up. A lot of teams are uncomfortable for their preprod environments to be use for such tests so make sure you have the ability to create these resources in a separate environment or account.
- The scope for your automated tests.
Here’s what your pipeline might look like.
What’s next?
Setting up a functional pipeline is hard work. This requires the love, care and attention of a skilled team of developers, platform engineers and security engineers.
It’s exciting to build. You now need to focus on maintaining it. Focus on the technical debt left behind everytime there is a feature added to your estate.
Keep an eye out for the next blog post where we’ll talk about creating these pipelines and keeping them in shape.
Further reading
Dynamic Binary Instrumentation Technology - Kunping Du, Fei Kang, Hui Shu, Li Dai
Insertion, Evasion, and Denial of Service - Ptacek, Newsham
ITSP Brighttalk Webinar - Sean Martin
Investigating the Effectiveness of RASP - Alexander Fry
Jumpstarting Your DevSecOps Pipeline with IAST and RASP
Weaving Application Security into the Software Development Life Cycle
Continuous Application Security at Scale with IAST and RASP — Transf…