Created'by'Cody'Herriges

Transcription

1 GETTING DATA OUT OF YOUR CLOUD(S) WITH OPEN SOURCE OBTAINING SUPERIOR INSIGHT Created'by'Cody'Herriges

2 INTRODUCTION Puppet'Labs Opera4ons'Engineer Systems'Team'lead

3 OUR PROBLEMS No'consumable'historic'data Incomplete'real'4me'data Data'collec4on'wasn't'eBec4vely'scalable Time'wasted'correla4ng'mul4ple'interfaces'by'hand Short'lived'issues'almost'impossible'to'understand Metrics'gathering'can'require'secondary'purchases

4 OUR PRIVATE CLOUDS Aging'VMware'stack'(vCenter'graph'failures) Budding'OpenStack'expansion'(Incomplete'built'in'tooling)

5

6

7

8

9 GETTING MORE INFORMATION VMware'KB:' A'4me'box'for'adequate'collec4on'is'not'reasonable

10

11 CEILOMETER NOT TARGETED AT ME No'func4ons'to'help'analyze'code Not'able'to'be'broken'down'by'hypervisor Originally'developed'with'a'slant'towards'billing'and'charge back Lacking'4me'granularity

12 EMC'S MONITORING & REPORTING Mostly'worked Never'had'a'4me'it'failed'to'provide'a'graph Its'downfall'was'the'inability'export'data'progroma4cally Automated'reported'geared'towards'managers

13 CLOSEST I GOT TO AUTOMATING DATA EXPORT A REPLAY OF A FIREBUG SESSION IN RUBY uri0 = URI.parse(" response0 = http.get(uri0.request_uri) cookie = response0.get_fields('set-cookie') uri1 = URI.parse(" response1 = http.get(uri1.request_uri, {'Cookie' => cookie.first}) cookie = response1.get_fields('set-cookie') uri2 = URI.parse(" response2 = http.get(uri2.request_uri, { 'Cookie' => cookie.first }) all_cookies = response2.get_fields('set-cookie') cookies_array = Array.new all_cookies.each { cookie cookies_array.push(cookie.split('; ')[0]) } cookies = cookies_array.join('; ')

14 MY SINGLE PANE OF GLASS NOT YOUR'S! Vendor'single'panes'of'glass'are'seldom'single Might'"work"'if'you're'the'tradi4onal'enterprise virtualiza4on'or'storage'team Only'achievable'if'I'can'provide'my'own'inputs'and business'logic

15 THE TOOLS THAT MADE IT POSSIBLE Elas4cSearch Logstash Kibana Grafana Graphite Diamond StatsFeeder CloudTrail

16 TACKLING MOST OF OUR VMWARE PROBLEM STATSFEEDER+GRAPHITE+GRAFANA Wasn't'able'to'force'everyone'to'a'diBerent'plaZorm StatsFeeder'grabs'real'4me'stats'for'all'hypervisors'and VMs Dumps'to'CSV'and'we'batch'convert'to'graphite Pulls'all'discrete'instances'of'vCenter'into'a'single dashboard puppet[graphite puppet[statsfeeder Basic'con\g

17 LESS IMPORTANT VMWARE INFORMATION ELASTICSEARCH+LOGSTASH+KIBANA (ELK) Really'noisy Hard'to'si^'through Searchable'for'events'like'everything'involving'a'speci\c LUN vcenter'log'forwarding ESXi'log'forwarding

18

19 THEN WE STARTED LEARNING THINGS

20

21

22 WE ARE AND MUST PURCHASE MULTIPLE TIMES MORE HARDWARE THAN WE USE ON AVERAGE!

23 MAKING A PUBLIC CLOUD DECISION ACTUALLY WE HAVE FOUR ALREADY...

24 MAKING A BETTER PUBLIC CLOUD DECISION I'think'I'am'changing... Insight'and'manageability'over'performance

25 ELK+CLOUDTRAIL UNDERSTANDING WHO IS DOING WHAT IN AWS Track'every'API'call Only'way'to'reliably'know'who'launched'a'speci\c'instance id Used'to'track'down'rogue'comprimised'instances puppet[elas4csearch puppet[logstash Tip:' S3'has'a'bat'namespace Results'in'a'simple'logstash'con\g

26

27 DIAMOND+GRAPHITE "SOLVED" OUR STORGE METRICS ISSUES AND AUGMENTS CEILOMETER That'EMC'we'couldn't'automate...replaced'by'Linux Diamond'kvm,'libvirtkvm,'and'iostat'collectors /sys/kernel/debug'and'/sys/fs/cgroup'must'be'mounted for'kvm'stats garethr[diamond'(we'forked'it)

28

29 NEW CHALLENGES Have'to'scale'Elas4cSearch'&'Graphite Data'normaliza4on More'and'more'dashboards'to'generate

30 END Created'by'Cody'Herriges ' PuppetConf''14