Case Study: Seagate s OpenStack Swift Security

Size: px

Start display at page:

Download "Case Study: Seagate s OpenStack Swift Security"

Cecilia O’Neal’
6 years ago
Views:

Case Study: Seagate s OpenStack Swift Security Presentation for InnoTech Oklahoma Rodney Beede, Seagate Technology November 3, 2015 Background image care of

1 Case Study: Seagate s OpenStack Swift Security Presentation for InnoTech Oklahoma Rodney Beede, Seagate Technology November 3, 2015 Background image care of Perspecsys Photos, 2013, "Cloud Security - abstract 3" from flickr.com, LIcensed under Creative Commons Attribution-ShareAlike 2.0 Generic License.; Resized to slide.

2 Introduction Rodney Beede Cloud Security Engineer Seagate Technology M.S. in Computer Science University of Colorado A Framework for Benevolent Computer Worms 2012 Doing computer security since 2001 Primary interests are web and cloud security Tech blog The views expressed in this blog are my personal view and have not been reviewed or approved by Seagate.

3 Agenda Cloud Asset Risk Evaluation What Is OpenStack Swift? What Do We Use It For? The Short Version Overview Of Swift Architecture Q&A: Real World Mistakes Securing Storage Services Object Storage "Account" Terminology Securing Proxy Services Identity (Keystone) Security Testing Interesting Problem & The Solution from IT Security The Road Ahead

4 Cloud Asset Risk Evaluation The Seven Key Questions: 1. How would we be harmed if the asset became public and widely distributed? 2. How would we be harmed if an employee of our cloud provider accessed the asset? 3. How would we be harmed if the process or function was manipulated by an outsider? 4. How would we be harmed if the process or function failed to provide expected results? 5. How would we be harmed if the information/data was unexpectedly changed? 6. How would we be harmed if the asset was unavailable for a period of time? 7. How does this affect our compliance obligations? Cloud Risk Thoughts: Deciding What, When, and How to Move to the Cloud - Rich Mogull, CEO of Securosis See slide note or ref slide for detailed citation.

What Is OpenStack Swift? Object Storage HTTP REST API web service Stores objects Object Storage API example PUT /v1/my_account/container/example_obj.txt HTTP/1.1 User-Agent: curl/7.32.

5 What Is OpenStack Swift? Object Storage HTTP REST API web service Stores objects Object Storage API example PUT /v1/my_account/container/example_obj.txt HTTP/1.1 User-Agent: curl/ Host: localhost:8080 Accept: */* X-Auth-Token: authtokenhere Content-Length: 38 Expect: 100-continue Image from OpenStack Installation Guide for Red Hat Enterprise Linux, CentOS, and Fedora and licensed under the Apache License, Version 2.0 HTTP/ Created Content-Length: 38 Content-Type: text/html; charset=utf-8 Etag: f7d40eceffdd9c2ecab b2a6

6 What Do We Use It For? Testing hard drives Benchmarking Measuring drive temperature Measuring drive vibration IT uses Backups Petabytes of storage space EVault Long term storage (tape replacement)

The Short Version - Swift Security Lots of components to secure OS Web server Authentication store Network traffic Remote console interface Lack of guidelines

7 The Short Version - Swift Security Lots of components to secure OS Web server Authentication store Network traffic Remote console interface Lack of guidelines (until 2013) we do not have specific guidance related to configuration of the storage projects So I wrote my own Published 12/2013

10 Question: Network Protocol Which services have encryption over the wire?

Remote Consoles - Overlooked Commonly known as IPMI BMC ilo DRAC Security problem Default password IPMI protocol insecure A Penetration Tester's Guide to IPMI and BMCs - Rapid 7 HD Moore, Metasploit,

12 Remote Consoles - Overlooked Commonly known as IPMI BMC ilo DRAC Security problem Default password IPMI protocol insecure A Penetration Tester's Guide to IPMI and BMCs - Rapid 7 HD Moore, Metasploit, July 2, :22:49 PM Unpatched firmware - admin access Admin hash vulnerability - unpatchable design flaw Image care of Licensed under CC BY-SA 2.0. Modified with red highlight. Original by Cloned Milkmen and titled T2000 USB and LED Close-up 2006.

13 Remote Consoles - Remediation 1. Network segregation 2. Patch to latest firmware 3. Rotate your passwords often a. Typically manual process b. IPMI has standard protocol for user management i. Hint: Linux ipmitool command c. We use CyberArk with custom plugin

14 Securing Services - General Standard OS configuration file permissions # chown -R root:swift /etc/swift/* # find /etc/swift/ -type f -exec chmod 640 {} \; # find /etc/swift/ -type d -exec chmod 750 {} \; Run services as swift Not root You won t need TCP ports < 1024 anyway Public facing service is API API URLs are typically not typed in web browsers Hence not critical user experience to use default ports 80 & 443 Proxy ports: 8443, 40443, anything > 1024

15 Real World Mistake Question: What are these files? $ ls /etc/certs/ ca.crt intermediate.crt server.crt server.csr server.pem

16 Real World Mistake - Problem Question: What is wrong? [user@host] $ ls -la /etc/certs/ drwxrwxr-x -rw-r--r--rw-r--r--rw-r--r-- www www 4096 Jul 22 12:15. www www 2110 Jan server.crt www www 1813 Jan server.csr www www 3243 Jan server.pem

17 Real World Mistake - Correct Security Question: Which is correct? A. -r--r www www server.pem B. -rw-r root www server.pem C. -rw-r root root server.pem D root root server.pem

Real World Mistake - Answer Answer: TRICK QUESTION! ALL ARE CORRECT! A. -r--r----- 1 www www server.pem B.

18 Real World Mistake - Answer Answer: TRICK QUESTION! ALL ARE CORRECT! A. -r--r www www server.pem B. -rw-r root www server.pem C. -rw-r root root server.pem D root root server.pem

19 Securing Storage Services The following are the default listening ports for the various storage services: Service name Port Type Account service 6002 TCP Container service 6001 TCP Object service 6000 TCP Rsync 873 TCP

20 Object Storage "Account" Terminology An object storage "account" isn t what you think it means OpenStack Object Storage Account Collection of containers; not user accounts. OpenStack Object Storage Containers Collection of objects. OpenStack Object Storage Objects The actual data objects. Supports ACLs to associate users with the account. Supports ACLs. Supports ACLs.

21 Object Storage "Account" Terminology Another way of thinking about it: A single shelf (Account) holds zero or more Buckets (Containers) Buckets (Containers) each hold zero or more Objects A garage (Object Storage cloud environment) may have Multiple shelves (Accounts) Each shelf may belong to zero or more users.

22 Securing Proxy Services This is the consumer facing service

swift system account is one option b. Listen on port > 1024 i.

23 Securing Proxy Services 1. Use SSL/TLS a. Not the built-in web server b. Apache, Nginx, etc. with mod-wsgi 2. Run web server as non-root a. swift system account is one option b. Listen on port > 1024 i. URL is typically not typed anyway 3. SSL Load Balancer a. Common pitfall - early termination

24 Load Balancer with SSL offload

25 Identity (Keystone) Seagate formerly used SWAuth Stored all info as Swift objects Now using Identity Service API ver 2.0 Custom implementation in one environment Keystone used in another environment Supports LDAP integration for authn authz is still performed via mappings in database Special Note: Keystone is NOT an identity provider Built-in credential store does NOT support password complexity rules Design choice as IdP is not goal of Keystone Basic principles Manage credentials Password length, complexity Account termination Audit your logs for policy conformity

Distributed clustered system causes auth delays Check for exposed storage node

26 Security Testing CVE list Cvedetails.com - Openstack Authentication token validation Lots of CVEs for invalid tokens Distributed clustered system causes auth delays Check for exposed storage node ports TCP ports (default) Also think about packet sniffing internally Network switch!= secure Think spoofing or sniffing

Security Testing It s a web service Similar approach as other

in URIs Ex: Third party web applications tend to trust the

object with a name like test<script>alert( xss )</script>me

27 Security Testing It s a web service Similar approach as other web services WSDL is optional, Swift is REST Encoding attacks in URIs Ex: Third party web applications tend to trust the web service data without proper output encoding Create an object with a name like test<script>alert( xss )</script>me Valid name to Swift Can trip up third party web application consumers

28 Interesting Problem & The Solution from IT Security

29 Interesting Problem & The Solution from IT Security These nodes have no Internet or corporate network access

30 Options NAT Requires more resources Additional security rules to manage HTTP Proxy Must configure each node to use it Puppet module can do this Possibility of caching is a bonus Local package repo Provides local mirror Faster Available even if Internet is not

31 Trial and Error NAT Worked for small scale Security misconfiguration HTTP Proxy Wrote Puppet module Provided auto-configuration Not helpful if Internet goes down

32 Solution Local mirror repository Controlled package versions No constant Internet connection required Use Puppet for node configuration automation

33 Puppet Automation is the key Account management Ex: swift OS acct. Cloud installation Public (not Seagate) puppetlabs/swift puppetlabs/apache Private (Seagate) HP OpenView Security hardening SSH key management Configuration Hiera driven (Hierarchical Database) Allows separate environments Con: Mistake in Puppet config can take down entire environment Although usually can recover via Puppet too

34 Puppet - Example user { swift : ensure comment expiry forcelocal password shell system } =>present, => OpenStack, =>absent, =>true, => *, # no logins via password => /sbin/nologin, =>true,

35 The Road Ahead Background image care of RainbowLady77, 2014, Lenticular cloud at Saddleback Mountain, Orange County, California on Sept 7, 2014 from Wikimedia Commons, LIcensed under CC BY-SA 4.0; Resized to slide.

36 Future Swift Projects At-rest encryption Community OpenStack draft project Offers option for cloud servers to do encryption Still trusting cloud provider Offloads the computation burden Allows quicker erasure of data Currently in design stage Discussion of user controlled keys Security bug fixes Most around tokens/private material appearing in verbose debug logs Enhanced auth functionality with Keystone Swift service tokens (service only has write privilege) Ensures data integrity for service (i.e. Glance)

38 References OpenStack Security Guide Cloud Risk Thoughts: Deciding What, When, and How to Move to the Cloud Rich Mogull, CEO of Securosis Tuesday 1st December Licensed under Creative Commons Attribution-NonCommercial-ShareAlike 3.0 United States License. ( Source for The Seven Key Questions from Cloud Asset Risk Evaluation slide Some clipart images care of openclipart.org wikimedia.org Source citations found in original presentation slide notes

OWASP Top 10 Risks. Many thanks to Dave Wichers & OWASP

OWASP Top 10 Risks. Many thanks to Dave Wichers & OWASP OWASP Top 10 Risks Dean.Bushmiller@ExpandingSecurity.com Many thanks to Dave Wichers & OWASP My Mom I got on the email and did a google on my boy My boy works in this Internet thing He makes cyber cafes