Orchestration Ownage: Exploiting Container-Centric Datacenter Platforms

Transcription

1 SESSION ID: CSV-R03 Orchestration Ownage: Exploiting Container-Centric Datacenter Platforms Bryce Kunz Senior Threat Specialist Adobe Mike Mellor Director, Information Security Adobe

2 Intro Mike Mellor Director, Information Adobe Bryce Kunz Senior Threat Adobe 2

3 Containers - The Future is Now! 2016 Surveys: 15-16% of all organizations are already using containers in production 35% organizations have done a proof-of-concept The Future is Now! Containers are in production now Containers are continuing to grow in popularity 3

4 Containers appear more secure The biggest drivers: 39% to increase developer efficiency and 36% to support microservices Organizations want to avoid cloud platform lock-in 2016 Surveys: Many (42%) organizations gain value in the secure/isolated capabilities that containers provide 4

5 But managing Containers feels complex 2016 Survey: The more exposure an organization has to containers, The more complexities are exposed. Respondents said they found containers too complex to integrate into existing environments, and require too many skilled resources to manage. 5

6 And are very challenging to manage at scale 2016 Survey: The #1 challenge of containers, according to the 53% of respondents who are either using or evaluating containers, is Container Management. 6

7 Probable Security Nightmare Too Complex + Challenging to Manage = Probable Security Nightmare Complexity the Worst Enemy of Security - Bruce Schneier 7

8 Container and Cluster Management Options Technology Design Pros Cons Public Cloud Container Services Container Centric Easy, Scalable Vendor Lock-in; Proprietary Docker Swarm Docker Centric Native Clustering Limited by API Kubernetes Mesos & DC/OS Clusters of Containerized s Cluster Management Works w/ Docker; Mounts persistent volumes Works w/ Docker, Kubernetes, & Native s; Very Flexible Custom overlay requires more specialization Additional layers adds more complexities 8

9 Cluster Management CoreOS Linux OS Many servers in DataCenter AWS Azure etc Datacenter, Azure, AWS, GCE, etc 9 How do we effectively use all of these resources?

10 Mesos Master & s Mesos Master 5050/TCP by default Distributes Tasks Mesos Master 5051/TCP by default Executes Tasks Datacenter, Azure, AWS, GCE, etc 10 CoreOS Linux OS

11 Mesos is the Kernel of DC/OS Mesos is the kernel of the distributed operating system known as DC/OS Master Kernel: Datacenter, Azure, AWS, GCE, etc 11

12 Frameworks Frameworks provide the logic Frameworks: Init Jobs Marathon Master Kernel: Cron Jobs Datacenter, Azure, AWS, GCE, etc 12 Chronos Metronome

13 Supporting: Configuration Stores Configuration Stores Supporting: keep everyone on the same page Frameworks: Kernel: Master Zoo Keeper Etcd Datacenter, Azure, AWS, GCE, etc 13

14 Supporting: Discovery Discovery Supporting: Enables the finding of other services within the cluster Frameworks: Master Mesos DNS Kernel: Datacenter, Azure, AWS, GCE, etc 14

15 DC/OS Design s: Supporting: Containers w/ s Docker Containers Frameworks: Kernel: Master Web s etc Datacenter, Azure, AWS, GCE, etc 15

16 Internet Accessible Containers Internet s: Supporting: Frameworks: Containers w/ s Public Internet Accessible Master Private Kernel: Internal Datacenter, Azure, AWS, GCE, etc 16

17 Scenario Internet s: Supporting: Frameworks: RCE Master Initial Access (RCE) Via a vulnerable web application Into a container As limited user (e.g. www-data) Kernel: Datacenter, Azure, AWS, GCE, etc 17

18 Scenario: RCE via web app within a container e.g. JBoss, Tomcat, OSGi Console, Axis2, etc 18

19 Recon via Mesos DNS Internet s: Supporting: RCE Query via pivot: Mesos DNS 53/UDP & TCP DNS service Frameworks: Master Kernel: Datacenter, Azure, AWS, GCE, etc 19

20 .mesos TLD The easy way to find services within the cluster 20

21 Recon via Mesos DNS Internet s: RCE Query via pivot: Mesos DNS Supporting: Frameworks: 8123/TCP by default DNS via REST API Kernel: Master Service Discover within the Cluster Datacenter, Azure, AWS, GCE, etc 21

22 Undocumented? /v1/enumerate -> all mesos dns information 22

23 Enumerate Mesos DNS using REST API /v1/enumerate -> all mesos dns information 23

24 Find IP & RHP TCP ports of all services /v1/enumerate -> all mesos dns information 24

25 Secure: Disable Risky Mesos DNS Features Internet s: Supporting: Frameworks: Kernel: RCE Master Datacenter, Azure, AWS, GCE, etc 25 Disable the AXFR Enumerate API Calls Harder for attacker to discover all services lications shouldn t commonly be using these API calls

26 Recon via Mesos Master Internet s: Supporting: RCE Query via pivot: Mesos Master 5050/TCP by default Distributes Tasks Frameworks: Master Kernel: Datacenter, Azure, AWS, GCE, etc 26

27 Enumerate Mesos Master Request via the REST API 27

28 Enumerate Mesos Master Response: json w/ all Mesos s IP addresses within the cluster 28

29 Recon via Mesos DNS Internet s: Supporting: RCE Query via pivot: Mesos 5051/TCP by default Executes Tasks Frameworks: Master Kernel: Datacenter, Azure, AWS, GCE, etc 29

30 Enumerate Mesos Request via the REST API 30

31 Enumerate Mesos Response: json w/ what containers are currently running on the server (i.e. basic0012) 31

32 Secure: Logical Internal Network Segmentation s: Supporting: Separates out the network into zones: s w/ Data Management Frameworks: Kernel: Master Commonly with Calico, Datacenter, Azure, AWS, GCE, etc 32 Canal, or Flannel

33 Secrets via Configuration Store Internet s: Supporting: Frameworks: Kernel: RCE Master Datacenter, Azure, AWS, GCE, etc 33 Etcd RHP/TCP by default 2379/TCP client/server 2380/TCP peers Configuration Store Core OS Fleets Units lications ZooKeeper 2181/TCP by default Binary Protocol

34 Enumerate Etc Request via the REST API recursively 34

35 Enumerate Etc Response: json frequently containing secrets including credentials 35

36 Secure: Separate Configuration Stores Internet s: Supporting: Frameworks: Kernel: RCE Master Datacenter, Azure, AWS, GCE, etc 36 Separate out the configuration stores into zones: s w/ Data Management Enforce separation via Authentication Credentials and Logical Network Segmentation

37 Frameworks Internet s: Supporting: Frameworks: Kernel: RCE Master Datacenter, Azure, AWS, GCE, etc 37 Marathon Long Running Services e.g. Containers Ensures always running Chronos Cron for the Cluster Batch Jobs

38 RCE via Marathon Jobs Request via the REST API 38

39 RCE via Marathon Jobs Internet s: Supporting: Frameworks: Kernel: RCE Master Datacenter, Azure, AWS, GCE, etc 39 RCE Marathon Long Running Services e.g. Containers Ensures always running Chronos Cron for the Cluster Batch Jobs

40 RCE via Marathon Jobs Response: json with the malicious job status 40

41 RCE via Chronos Jobs Internet s: Supporting: Frameworks: Kernel: RCE Master Datacenter, Azure, AWS, GCE, etc 41 RCE Marathon Long Running Services e.g. Containers Ensures always running Chronos Cron for the Cluster Batch Jobs

42 Secure: Enforce Authentication Internet s: Supporting: Frameworks: Kernel: RCE Master Datacenter, Azure, AWS, GCE, etc 42 lications must support and be configured to use authentication as well securely store and use credentials be deployed securely and/or retrieve credentials securely Alert on brute force attempts

43 Creds via MitM with ARP Spoofing Internet s: RCE Another Container has the Creds for Marathon Supporting: Frameworks: Master Kernel: Datacenter, Azure, AWS, GCE, etc 43

44 Creds via MitM with ARP Spoofing Internet s: Supporting: Frameworks: RCE ARP Attacker uses ARP spoofing to redirect that containers traffic to the compromised container Kernel: Master Attacker collect the credentials Datacenter, Azure, AWS, GCE, etc 44

45 Creds via MitM with ARP Spoofing Internet s: Supporting: Frameworks: RCE ARP Attacker can now create malicious Marathon jobs Negating authentication security controls Master Kernel: RCE Datacenter, Azure, AWS, GCE, etc 45

46 Secure: TLS for Internal Communications Internet s: Supporting: RCE ARP Enable TLS w/ valid certificates for strong HTTPS communications Anything using credentials needs TLS! Frameworks: Kernel: Master Validate Certificates Fail closed on bad certificates Datacenter, Azure, AWS, GCE, etc 46 Alert on certificates errors

47 Strategic Actions Next week: Assess which services you can enable Authentication & TLS on w/o breaking your existing applications within the cluster Three months from now: Implement Authentication & TLS on safe services and frameworks Focusing on services responsible for orchestration within the cluster Deploy separate services where possible for s that do not support TLS & Auth Six months from now: Retrofit all lications within the cluster to use TLS & Authentication Enforce the use of TLS & Authentication internal everywhere (disable clear-text) 47

48 Big Picture Container Adoption Is Maturing, especially in Enterprises Enterprises are using containers in production. 48

49 Big Picture Pivoting from a compromised service within the cluster No container breakout / 0day / exploit needed J May enable an attacker to completely compromise the cluster 49

50 Big Picture Looking Beyond the Border with a Defense in Depth strategy Secures the Future & the cluster 50

51 Thank you! Thank you! 51

52 Future Research Testing MitM from compromised container NCC Group s report states this is possible for co-hosted containers Test downgrade HTTPS communications Can we downgrade from HTTPS to HTTP and capture creds from another container? Test Certs (e.g. can cert pinning be enabled?) to REST APIs Can we MitM and impersonate the API service? Test Authentication Brute force attacks Fairly certain there are no lockouts, can we enable better authentication security? Write module to brute-force and guess creds Test Logical Network Segmentation Tools Calico, Canal, Flannel Note: these should work as advertised but probably we should independently verify 52

53 References Foundry-2016-Container-Report.pdf pdf 53