DevSecOps Building Security int Yur DevOps Prcesses
The DevOps mvement has pushed fr and succeeded in breaking dwn barriers and sils within rganizatins dividing teams int specialized functins f Develpment and Operatins. DevOps enables rganizatins that embrace the mvement and culture t be mre cmpetitive by enabling faster, mre reliable sftware releases by leveraging autmatin t replace manual prcesses invlved in shipping sftware. A side effect f this speed is that security tls and prcesses need t mve at the same pace t keep up. The idea driving DevSecOps r Rugged DevOps is t bake the security testing f the applicatin under develpment int the prcess used t ship it. Autmatin f these prcesses takes peple ut f the chain and puts them in a different capacity. Instead f peple being the prcess, tls and autmatin are the prcess and peple mnitr and respnd t prcess failures. Thus, cmbining the strengths f bth cmputers and peple. The image belw lists sme f the tls leveraged t autmate the security testing and auditing f a DevOps pipeline. Sme types f tls that exist already fr security testing are: Clud infrastructure Tls t scan yur clud infrastructure and cnfiguratin f resurces against best practices fr that clud. Examples are Azure Advisr and evident.i Autmated security tests There are nw framewrks that let yu write security tests fr applicatins just like the traditinal unit and integratin tests. Gauntit is a framewrk that is gaining ppularity t build and autmate tests like these. Static cde analysis This set f tls can scan yur cdebase and pen surce libraries and find ptential vulnerabilities. VeraCde is a ppular tl t perfrm these kinds f analyses. 1
Runtime security analysis This set f tls runs alngside/within yur applicatin in prductin and can help identify and prevent security issues in real time. Cntrast Security is ne such tl that prvides these features. Vulnerability scanning fr Cntainers Tls like Clair frm CreOS help yu t scan yur cntainer images fr vulnerabilities. The API interface fr Clair makes it easy t add this t custm build pipelines. Yu can have a lk at- https://github.cm/devsecps/awesmedevsecps#tls fr a mre cmprehensive list f DevSecOps tls fr each stage f the SDLC. Best Practices fr Implementing DevSecOps in Yur Organizatin Autmated and Prgrammable Security Cntrls Just like DevOps relies n fast shrt feedback lps, DevSecOps als uses shrt feedback lps with the difference f having cntinuus mnitring and analytics at the cre f bth the develpment and peratins ends. Infrmatin security architects must try t incrprate security cntrls with a minimum t n manual cnfiguratin at every stage f the cycle in such a way that they are transparent and dn t impede the hard-wn agility DevOps brings while managing risk and regulatry cmpliance needs. This means that any security cntrls added need t be capable f autmatin t fit int DevOps tlchains. Autmatin brings in tw benefits immediately. Autmatin reduces the risk f misadministratin and mistakes which are the tw leading causes f security breaches and unexpected peratinal dwntime. Secndly, autmatin remves peple frm the prcess and makes them mnitrs f the prcess respnding t prcess failures allwing a speed bst fr security activities and making sure that the agility f DevOps envirnments isn t impacted. Surce: Gartner (September 2016) 2
When a security platfrm s capabilities like IAM, firewalling, vulnerability scanning, AST (applicatin security testing) are available thrugh a prgrammable interface, the integratin and autmatin f these cntrls are easy within an autmated DevOps pipeline. Security teams within an rganizatin nw can set plicies which can be standardized and applied rganizatin wide. Chse security and management vendrs wh: Fully API-enable their platfrm services and expse 100% f functinality via APIs Prvide explicit supprt fr cmmn DevOps tlchain envirnments, such as Chef, Puppet and similar autmatin tls Prvide explicit supprt fr cntainers and cntainer rchestratin and management systems (which are nt necessary fr DevSecOps, but help streamline service delivery frm develpment int prductin). Separatin f Duties and Enfrcement via RBAC and IAM A majr jb fr auditrs and security architects is making sure there is a clear separatin f wh can d what in additin t where and when especially in terms f service deplyment and develpment. Even within a single team respnsible fr a service there are clearly defined rles peple take. The idea is nt t lck dwn r unnecessarily hamper the individuals frm ding their jbs but t give them the minimum set f permissins they need t achieve their gals. This will mean that they will be highly empwered within the areas they are respnsible fr. The scpe and grant f capabilities can be managed by linking existing IAM systems and defining rles fr each unique stage (develpment, staging, and prductin). Cnnect tling t existing IAM systems t pull dwn identities and permissins, e.g., LDAP and Active Directry. Allw enfrcement f security plicies in tls and mnitr all access t tling and activities. Define unique rles fr envirnments, e.g., develpment vs. prductin. An ideal situatin is where all changes within prductin happen via audited and verified scripts with n peple invlved in making direct changes themselves. Make teams respnsible and allw audits fr changes t their service via a trust and verify basis. Verificatin can be achieved thrugh audit lgs and develpment changes lgged int smething, such as SCM, e.g., Git, Subversin, ClearCase. Simplify Risk and Threat Mdelling fr Applicatins A standard best practice fr DevSecOps is having at least a basic risk based threat mdel. An easy way t start is having a simple questinnaire s develpment teams can assess the risk f the service they are develping at a high level. A sample questinnaire can include questins like: Is sensitive data being handled? What type f sensitive data? Are cmmunicatins being encrypted? Is data at rest being encrypted? 3
Develper training shuld be cnducted fr basic security best practices and peridic cmmunicatin arund changes t these plicies need t be circulated. A list f basic security plicies can include things like input sanitizatin, encryptin f cmmunicatin and data. heavyweight and need a qualified security prfessinal t run them. There is a new breed f lightweight tls that integrate directly with a develper's IDE and allw fr a quick check f security as a cde is written. In additin t tls like these, autmated scanning and security testing sftware shuld be a part f the cntinuus testing pipeline. Train develpers in secure cding best practices and t write resilient cde that sanitizes input and blcks cmmn attack patterns, such as buffer verflws, SQL injectin and crss-site scripting. Develp a simple threat-and-risk mdel assessment tl and implement it as a part f the planning and design prcess. Base the level f threat mdelling n the risk f the applicatin. Applicatins handling sensitive data r directly accessing the internet shuld require deeper threat mdelling and cllabratively invlve infrmatin security. Plan t mask, de-identify r synthesize data used in develpment fr testing. D nt use raw sensitive prductin data in develpment. Scan Custm Cde, Applicatins, and APIs Any custm cde being written shuld be scanned fr pssible security vulnerabilities during develpment. Current tling fr traditinal static applicatin security testing (SAST) and dynamic applicatin security testing (DAST) are nt suited fr the scale DevSecOps needs. They are t Interactive applicatin security testing (IAST) is a great ptin if the platfrm the applicatin is being develped n supprts instrumentatin (Java,.Net, PHP). IAST is a great fit fr the highly-autmated testing needed fr DevSecOps. A substitute fr IAST is applicatin security testing (AST) tls. Tls that can be driven via autmatin shuld be preferred. Evaluate and adpt IAST fr applicatins that supprt it, and favur slutins using selfinducers fr autmated testing. Plan t fully autmate any traditinal static r dynamic tls r services that are used. Fr example, DevOps tlchain scripting tls can invke autmated testing. D nt make develpers leave their native envirnment and tlchains. If SAST and DAST slutins are used, require vendrs t supprt differential scans that test nly the mdified cde and dwnstreamimpacted mdules. 4
Acknwledge and accept that having zer vulnerabilities isn't pssible. Reduce false psitives (albeit with a risk f higher false negatives) and trim the utput f AST tls and services t fcus develpers first n the highest severity and highest cnfidence vulnerabilities. Favur AST scanning tls and services that use machine learning and cllective intelligence t trim results t nly the highest cnfidence results. By plicy, dn't allw custm cde with knwn critical vulnerabilities t enter prductin. Accept that vulnerabilities that represent lwer levels f risk may r may nt be addressed in future iteratins. Appraches that identify and accept manageable risk are necessary. Wrk with DevOps managers t measure and mtivate develpment teams t prduce cde with fewer vulnerabilities. Make security metrics a part f cde quality metrics and hld develpment teams accuntable. Scan Open Surce Dependencies fr Issues Mst mdern applicatins can be described as assembled vs. being built frm scratch. Develpers rely n a multitude f Open Surce libraries and framewrks t accelerate the building f their applicatin. This is a prblem when there are vulnerabilities with these pen surce libraries and framewrks. All dependencies shuld be scanned and vetted fr vulnerabilities during the build prcess and flagged fr review and remediatin. Scan all applicatins, system images, virtual machines and cntainers in develpment fr unknwn, embedded r vulnerable OSS cmpnents in the perating system, applicatin platfrm and in the applicatin itself. Implement an "OSS firewall" t practively prevent develpers frm dwnlading knwn vulnerable cde frm Maven, GitHub and ther OSS cde repsitries by plicy. Scanning fr Vulnerabilities and Cnfiguratin at the Surce Develpment The previus sectins scan fr discvered and knwn vulnerabilities in custm cde and Open Surce dependencies. As develpment cntinues and packages are built and integrated, it becmes imprtant t scan the entire cntent f images (vms, amis r cntainers). This scanning shuld be built int the build pipeline and shuld be autmated. These scans shuld als target the scanning f the cnfiguratin f the OS and applicatin platfrms and cmpare them t the best practices fr that platfrm s secure cnfiguratin and hardening. An estimate frm Gartner ntes that thrugh 2020, 99% f vulnerabilities that are explited will cntinue t be knwn fr at least ne year. Finding these issues at the surce during build time eliminates them frm reaching prductin. Priritize OSS sftware mdule identificatin and vulnerability scanning in develpment. Architect DevOps prcesses t autmatically scan the cntents f all system images, including the base OS, applicatin platfrm and all cntainers fr knwn vulnerabilities 5
and cnfiguratin issues as part f the cntinuus integratin prcess. By plicy, dn't allw systems t leave develpment with knwn critical vulnerabilities. Require develpers t remve unnecessary mdules and harden all systems t industry standard best practices. Integrate with anti-malware scanners (such as VirusTtal), netwrk sandbxing and algrithmic malware detectin (such as Cylance) t scan systems t ensure the malicius cde hasn't been intrduced t the image during the develpment prcess. Expand the Definitin f Sensitive Cde t Include Scripts, Recipes, Templates, and Layers DevOps prmtes a prcess f Infrastructure as a cde. This allws yu t versin audit and autmate the deplyment and cnfiguratin f infrastructure, essentially making yur infrastructure prgrammable. The first strategy talks abut making security tls prgrammable. Since infrastructure is being treated as a cde, these artefacts need t have security cding principles applied t them. This means that the templates, scripts, recipes, and blueprints need t be secured and audited. We ve discussed hw autmatin can reduce the chance f a mistake, but a prly written script can magnify a mistake if released int prductin. An example f this is a recent S3 utage that was caused by an engineer mistakenly remving prductin servers instead f replicas. This actin thrugh the script caused Amazn s S3 service t grind t a halt. The script didn t have checks r rate limiting arund hw many servers it culd affect. Incidents like these drive hme the need t make sure that cnfiguratin files and scripts, like surce cde, need t be scanned fr mistakes, pssible vulnerabilities, and excessive risk. Earlier we discussed hw high levels f autmatin can reduce the chance f a mistake. All cnfiguratin that can be expressed in text files shuld be held in a central repsitry like Git that allws changes t these cnfiguratins t be recrded. Git can recrd nt nly what change was made but als when and by whm. Eventually, all infrastructure shuld be treated like surce cde with versin cntrl rllback audit, lgging and alerting based n usage. Making this the rganizatinal standard means that n changes t infrastructure can be made withut being recrded and audited. This prcess will be valuable t auditrs and get their buy-in in implementing a DevOps expansin t nt just applicatin but als infrastructure. Ensure that DevOps teams have implemented gd versin cntrl practices and tls t maintain clear accuntability and traceability fr all the applicatin sftware that is deplyed int the live envirnment. Extend the scpe f the versin cntrl and autmated deplyment tls t the cnfiguratin, infrastructure setup and mnitring cnfiguratin. 6
Use autmatin scripts t deply t the staging envirnment fr final tests (may be an autmated test in advanced DevOps envirnments). Scan scripts fr errrs and embedded risk, such as embedded credentials, encryptin keys, API keys and s n, that represent a significant and avidable risk. System Integrity and Cnfiguratin Cmpliance in Prductin Let s mve the fcus f this paper t the best practices in prductin. The pririty fr any prductin system is that the infrastructure and services installed and running are what we need and are cnfigured crrectly. Any tampering with the images against standards intrducing vulnerabilities shuld be detected and islated. We shuld measure as many system elements as we can (H/W, virtualizatin, images VMs, Cntainers). This measurement shuld extend t the validatin f any cntainer assembly layers used in cntainer management slutins. Use a cntainer management system (if cntainers are used) that supprts hashing r ther techniques t measure and verify system integrity when laded. Use Whitelisting n Prductin Systems, Including Cntainer-Based Implementatins Leveraging Whitelisting in Prductin One f the mst pwerful security cntrls fr a wrklad is whitelisting and subsequent mnitring and enfrcement f all its interactins. The use f whitelisting t prevent any thing nt explicitly listed as safe prevents the running f malicius files and wrklads. Whitelisting can extend well beynd just what is allwed t run n a system. This technique can be used t whitelist netwrk cnnectivity, user access, administrative access, and file system access. Histrically this has been difficult t achieve but with the autmatin f infrastructure thrugh DevOps cnfiguratin tls this is nw straightfrward. The declarative nature f DevOps templates lend themselves really well t whitelisting. Implement system integrity measurement n systems as they are bted, including the hardware-based rt f trust measurements f the basic input/utput system, btlader, hypervisr and OS n systems yu wn. Disable runtime-signature-based anti-malware scanning and implement a whitelisting mdel n server wrklads. Antivirus scanning prvides little r n value n well-managed servers and is a waste f resurces in a DevSecOps envirnment. Stre VMs at rest encrypted and hashed, if VMs Autmatically cnfigure whitelists frm the are used in the DevSecOps wrkflw. Verify declarative surces f DevOps tlchains and against tampering at bt. cntainers. 7
Require vendrs t supprt whitelisting appraches fr cntainers if cntainers are used. Dn t Assume Perfect Prtectin Organizatins in this day and age face advanced and targeted attacks and preventing them all isn t pssible. It s better t assume they will be cmprmised and engineer t minimize the impact f such a cmprmise. This means everything in yur envirnment needs t be mnitred cntinuusly and any unusual behaviur that is indicative f a breach shuld trigger alerts and autmated respnses. Machine learning and advanced analytic techniques t identify patterns and deviatins can be leveraged here. Design fr pervasive mnitring f critical applicatins, user lgins/lguts, transactins, interactins, netwrk activity and system activity. Use the mnitring data t establish baselines f "nrmal" fr the applicatin in rder t detect meaningful deviatins. Share mnitring data acrss DevOps r prduct teams, platfrm teams and security peratins center teams, as unusual activity may be caused by a hardware failure, sftware failure, bug, insider threat r attack. Deply deceptin and decy services autmatically t mre easily identify attackers as these technlgies mature ver the next several years. Restrict and Lck Dwn Access t Prductin Infrastructure and Services Making sure that autmated tls are the nly way t make changes in prductin allws the standardizatin f remediatin and makes all actins auditable. The exclusive use f autmated tls als lets yu perfrm rllback f any changes attempted. Rapid iteratin and immediate feedback fr prblems and vulnerabilities at the Develpment end means yur security psture is imprved. Infrmatin security architects shuld cllabrate with DevOps teams t: Restrict changes t nly being made via autmated tls and scripts. Disable remte administratin via Secure Shell (SSH) and Remte Desktp Prtcl (RDP) t frce access via APIs and scripts. Adpt an immutable infrastructure mindset (where pssible) and autmate all changes t the envirnment using DevSecOps-style wrkflws. Out-f-date wrklads shuld simply be replaced with newer images in an autmated, systematic way. Require privileged access management systems t manage credentialed access in the rare cases when direct administrative access is needed. Cntainers Security Limitatins While cntainers aren t necessary fr a DevOps 8
transfrmatin, they are extremely ppular because f the cnsistency and streamlining they prvide fr develpers as their cde mves frm develpment t prductin. Cntainers d intrduce several security issues that need t be acknwledged. Cntainers share a cmmn OS; this means that islatin is prvided by the OS and nt the hypervisr. Netwrk traffic is visible t all cntainers n the same hst OS withut the additin f any additinal tls. This means that any attack that is successful n the OS kernel will expse all cntainers n that hst. These are the reasns why it s recmmended t use cntainers n wrklads f similar trust levels and using hypervisrs as islatin by running the cntainers within VMs. The use f lean stripped dwn speciality OSes develped t run cntainers is als a recmmended measure. Cnclusin DevSecOps aims fr a gal f having security checks and cntrls applied transparently and autmatically within a rapid-develpment autmated DevOps pipeline. Shifting security left t start at develpment makes sure that DevSecOps is effective and it fllws the jurney f service thrughut its lifecycle. Abve all, successful DevSecOps initiatives must remain true t the riginal DevOps philsphy: teamwrk and transparency, and cntinual imprvement thrugh cntinual learning. 9