SESSION ID: CSV-R03 Orchestration Ownage: Exploiting Container-Centric Datacenter Platforms Bryce Kunz Senior Threat Specialist Adobe Mike Mellor Director, Information Security Adobe
Intro Mike Mellor Director, Information Security @ Adobe Bryce Kunz Senior Threat Specialist @ Adobe 2
Containers - The Future is Now! 2016 Surveys: 15-16% of all organizations are already using containers in production 35% organizations have done a proof-of-concept The Future is Now! Containers are in production now Containers are continuing to grow in popularity 3
Containers appear more secure The biggest drivers: 39% to increase developer efficiency and 36% to support microservices Organizations want to avoid cloud platform lock-in 2016 Surveys: Many (42%) organizations gain value in the secure/isolated capabilities that containers provide 4
But managing Containers feels complex 2016 Survey: The more exposure an organization has to containers, The more complexities are exposed. Respondents said they found containers too complex to integrate into existing environments, and require too many skilled resources to manage. 5
And are very challenging to manage at scale 2016 Survey: The #1 challenge of containers, according to the 53% of respondents who are either using or evaluating containers, is Container Management. 6
Probable Security Nightmare Too Complex + Challenging to Manage = Probable Security Nightmare Complexity the Worst Enemy of Security - Bruce Schneier 7
Container and Cluster Management Options Technology Design Pros Cons Public Cloud Container Services Container Centric Easy, Scalable Vendor Lock-in; Proprietary Docker Swarm Docker Centric Native Clustering Limited by API Kubernetes Mesos & DC/OS Clusters of Containerized s Cluster Management Works w/ Docker; Mounts persistent volumes Works w/ Docker, Kubernetes, & Native s; Very Flexible Custom overlay requires more specialization Additional layers adds more complexities 8
Cluster Management CoreOS Linux OS Many servers in DataCenter AWS Azure etc Datacenter, Azure, AWS, GCE, etc 9 How do we effectively use all of these resources?
Mesos Master & s Mesos Master 5050/TCP by default Distributes Tasks Mesos Master 5051/TCP by default Executes Tasks Datacenter, Azure, AWS, GCE, etc 10 CoreOS Linux OS
Mesos is the Kernel of DC/OS Mesos is the kernel of the distributed operating system known as DC/OS Master Kernel: Datacenter, Azure, AWS, GCE, etc 11
Frameworks Frameworks provide the logic Frameworks: Init Jobs Marathon Master Kernel: Cron Jobs Datacenter, Azure, AWS, GCE, etc 12 Chronos Metronome
Supporting: Configuration Stores Configuration Stores Supporting: keep everyone on the same page Frameworks: Kernel: Master Zoo Keeper Etcd Datacenter, Azure, AWS, GCE, etc 13
Supporting: Discovery Discovery Supporting: Enables the finding of other services within the cluster Frameworks: Master Mesos DNS Kernel: Datacenter, Azure, AWS, GCE, etc 14
DC/OS Design s: Supporting: Containers w/ s Docker Containers Frameworks: Kernel: Master Web s etc Datacenter, Azure, AWS, GCE, etc 15
Internet Accessible Containers Internet s: Supporting: Frameworks: Containers w/ s Public Internet Accessible Master Private Kernel: Internal Datacenter, Azure, AWS, GCE, etc 16
Scenario Internet s: Supporting: Frameworks: RCE Master Initial Access (RCE) Via a vulnerable web application Into a container As limited user (e.g. www-data) Kernel: Datacenter, Azure, AWS, GCE, etc 17
Scenario: RCE via web app within a container e.g. JBoss, Tomcat, OSGi Console, Axis2, etc 18
Recon via Mesos DNS Internet s: Supporting: RCE Query via pivot: Mesos DNS 53/UDP & TCP DNS service Frameworks: Master Kernel: Datacenter, Azure, AWS, GCE, etc 19
.mesos TLD The easy way to find services within the cluster 20
Recon via Mesos DNS Internet s: RCE Query via pivot: Mesos DNS Supporting: Frameworks: 8123/TCP by default DNS via REST API Kernel: Master Service Discover within the Cluster Datacenter, Azure, AWS, GCE, etc 21
Undocumented? /v1/enumerate -> all mesos dns information 22
Enumerate Mesos DNS using REST API /v1/enumerate -> all mesos dns information 23
Find IP & RHP TCP ports of all services /v1/enumerate -> all mesos dns information 24
Secure: Disable Risky Mesos DNS Features Internet s: Supporting: Frameworks: Kernel: RCE Master Datacenter, Azure, AWS, GCE, etc 25 Disable the AXFR Enumerate API Calls Harder for attacker to discover all services lications shouldn t commonly be using these API calls
Recon via Mesos Master Internet s: Supporting: RCE Query via pivot: Mesos Master 5050/TCP by default Distributes Tasks Frameworks: Master Kernel: Datacenter, Azure, AWS, GCE, etc 26
Enumerate Mesos Master Request via the REST API 27
Enumerate Mesos Master Response: json w/ all Mesos s IP addresses within the cluster 28
Recon via Mesos DNS Internet s: Supporting: RCE Query via pivot: Mesos 5051/TCP by default Executes Tasks Frameworks: Master Kernel: Datacenter, Azure, AWS, GCE, etc 29
Enumerate Mesos Request via the REST API 30
Enumerate Mesos Response: json w/ what containers are currently running on the server (i.e. basic0012) 31
Secure: Logical Internal Network Segmentation s: Supporting: Separates out the network into zones: s w/ Data Management Frameworks: Kernel: Master Commonly with Calico, Datacenter, Azure, AWS, GCE, etc 32 Canal, or Flannel
Secrets via Configuration Store Internet s: Supporting: Frameworks: Kernel: RCE Master Datacenter, Azure, AWS, GCE, etc 33 Etcd RHP/TCP by default 2379/TCP client/server 2380/TCP peers Configuration Store Core OS Fleets Units lications ZooKeeper 2181/TCP by default Binary Protocol
Enumerate Etc Request via the REST API recursively 34
Enumerate Etc Response: json frequently containing secrets including credentials 35
Secure: Separate Configuration Stores Internet s: Supporting: Frameworks: Kernel: RCE Master Datacenter, Azure, AWS, GCE, etc 36 Separate out the configuration stores into zones: s w/ Data Management Enforce separation via Authentication Credentials and Logical Network Segmentation
Frameworks Internet s: Supporting: Frameworks: Kernel: RCE Master Datacenter, Azure, AWS, GCE, etc 37 Marathon Long Running Services e.g. Containers Ensures always running Chronos Cron for the Cluster Batch Jobs
RCE via Marathon Jobs Request via the REST API 38
RCE via Marathon Jobs Internet s: Supporting: Frameworks: Kernel: RCE Master Datacenter, Azure, AWS, GCE, etc 39 RCE Marathon Long Running Services e.g. Containers Ensures always running Chronos Cron for the Cluster Batch Jobs
RCE via Marathon Jobs Response: json with the malicious job status 40
RCE via Chronos Jobs Internet s: Supporting: Frameworks: Kernel: RCE Master Datacenter, Azure, AWS, GCE, etc 41 RCE Marathon Long Running Services e.g. Containers Ensures always running Chronos Cron for the Cluster Batch Jobs
Secure: Enforce Authentication Internet s: Supporting: Frameworks: Kernel: RCE Master Datacenter, Azure, AWS, GCE, etc 42 lications must support and be configured to use authentication as well securely store and use credentials be deployed securely and/or retrieve credentials securely Alert on brute force attempts
Creds via MitM with ARP Spoofing Internet s: RCE Another Container has the Creds for Marathon Supporting: Frameworks: Master Kernel: Datacenter, Azure, AWS, GCE, etc 43
Creds via MitM with ARP Spoofing Internet s: Supporting: Frameworks: RCE ARP Attacker uses ARP spoofing to redirect that containers traffic to the compromised container Kernel: Master Attacker collect the credentials Datacenter, Azure, AWS, GCE, etc 44
Creds via MitM with ARP Spoofing Internet s: Supporting: Frameworks: RCE ARP Attacker can now create malicious Marathon jobs Negating authentication security controls Master Kernel: RCE Datacenter, Azure, AWS, GCE, etc 45
Secure: TLS for Internal Communications Internet s: Supporting: RCE ARP Enable TLS w/ valid certificates for strong HTTPS communications Anything using credentials needs TLS! Frameworks: Kernel: Master Validate Certificates Fail closed on bad certificates Datacenter, Azure, AWS, GCE, etc 46 Alert on certificates errors
Strategic Actions Next week: Assess which services you can enable Authentication & TLS on w/o breaking your existing applications within the cluster Three months from now: Implement Authentication & TLS on safe services and frameworks Focusing on services responsible for orchestration within the cluster Deploy separate services where possible for s that do not support TLS & Auth Six months from now: Retrofit all lications within the cluster to use TLS & Authentication Enforce the use of TLS & Authentication internal everywhere (disable clear-text) 47
Big Picture Container Adoption Is Maturing, especially in Enterprises Enterprises are using containers in production. 48
Big Picture Pivoting from a compromised service within the cluster No container breakout / 0day / exploit needed J May enable an attacker to completely compromise the cluster 49
Big Picture Looking Beyond the Border with a Defense in Depth strategy Secures the Future & the cluster 50
Thank you! Thank you! 51
Future Research Testing MitM from compromised container NCC Group s report states this is possible for co-hosted containers Test downgrade HTTPS communications Can we downgrade from HTTPS to HTTP and capture creds from another container? Test Certs (e.g. can cert pinning be enabled?) to REST APIs Can we MitM and impersonate the API service? Test Authentication Brute force attacks Fairly certain there are no lockouts, can we enable better authentication security? Write module to brute-force and guess creds Test Logical Network Segmentation Tools Calico, Canal, Flannel Note: these should work as advertised but probably we should independently verify 52
References https://www.cloudfoundry.org/wp-content/uploads/2016/06/cloud- Foundry-2016-Container-Report.pdf https://clusterhq.com/assets/pdfs/state-of-container-usage-june- 2016.pdf http://www.rightscale.com/blog/cloud-industry-insights/new-devopstrends-2016-state-cloud-survey 53