StackTach. Integration.

Transcription

1 StackTach Ceilometer

2 StackTach was Introduced in the Diablo Timeframe Feb 2012

3

4 A tool for monitoring/debugging OpenStack

5 Nova amqp Worker StackTach Glance Events DB

6 Web and REST Interface

7 And our cmdline tool stacky For dev and op access

8 HowTo video and demo

9 Event Flow & State Transitions compute.instance.update api None->building None-> scheduling scheduler.run_instance.start scheduler -> -> scheduler.run_instance.scheduled scheduler -> -> scheduler.run_instance.end scheduler -> -> compute.instance.update compute building->building scheduling-> None compute.instance.create.start compute ->building -> compute.instance.update compute None->building None-> None compute.instance.update compute building->building None-> networking compute.instance.update compute building->building networking-> block_device_mapping compute.instance.update compute building->building block_device_mapping-> spawning compute.instance.update compute None->building None-> spawning compute.instance.update compute None->building None-> spawning compute.instance.update compute None->building None-> spawning compute.instance.update compute None->building None-> spawning compute.instance.update compute None->building None-> spawning compute.instance.update compute None->building None-> spawning compute.instance.update compute None->building None-> spawning compute.instance.update compute building->active spawning-> None compute.instance.create.end compute ->active ->

10 Evolved into a tool for Tracking performance

11 Build time per flavor by region Build failures by region Build failures by compute node Scheduler performance Amqp In-flight message latency Build times by tenant Failure rate by tenant Resize / migration times by image type

12 Monitoring on a Per-request basis unavailable from statsd/graphite

13 Events & samples

14 Samples are Small, abundant and disposable what, when

15 CPU 70%

16 Events are Rich with context Who, what, when, where, why

17 '_context_auth_token': u'3d8b13de1b7d499587dfc69b77dc09c2', '_context_is_admin': True, '_context_project_id': u'7c150a59fe714e6f af9688f0e', '_context_quota_class': None, '_context_read_deleted': u'no', '_context_remote_address': u' ', '_context_request_id': u'req-d68b36e f-9afb-d81435d64d66', '_context_roles': [u'admin'], '_context_timestamp': u' t20:23: ', '_context_user_id': u'1e3ce f1a61c1996d1a531a2', 'event_type': u'compute.instance.create.end', 'message_id': u'dae6f69c-00e0-41c0-b371-41ec3b7f4451', 'priority': u'info', 'publisher_id': u'compute.vagrant-precise', 'timestamp': u' :23: ', 'payload': { 'created_at': u' :23:41', 'deleted_at': u'', 'disk_gb': 0, 'display_name': u'testme', 'fixed_ips': [{ 'address': u' ', 'floating_ips': [], 'meta': {}, 'type': u'fixed', 'version': 4}], 'image_ref_url': u' 'instance_id': u'9f9d01b9-4a e27-398b21ab20d1', 'instance_type': u'm1.tiny', 'instance_type_id': 2, 'launched_at': u' :23: ', 'memory_mb': 512, 'state': u'active', 'state_description': u'', 'tenant_id': u'7c150a59fe714e6f af9688f0e', 'user_id': u'1e3ce f1a61c1996d1a531a2', 'reservation_id': u'1e3ce f1a61c1996d1a531a3', 'vcpus': 1, 'root_gb': 0, 'ephemeral_gb': 0, 'host': u'compute-host-name', 'availability_zone': u'1e3ce f1a61c1996d1a531a4', 'os_type': u'linux?', 'architecture': u'x86', 'image_ref': u'uuid', 'kernel_id': u'1e3ce f1a61c1996d1a531a5', 'ramdisk_id': u'1e3ce f1a61c1996d1a531a6',

18 Events > samples

19 Evolved into a tool for Auditing & billing

20 Create.start Rebuild.start Resize.prep.start Resize.revert.start Rescue.start Delete.end create.end rebuild.end resize.prep.end resize.revert.end rescue.end Resize.finish.end Compute.instance.exists

21 Nova issues.exists events whenever Launched at changes on an instance (mostly)

22 Generate.exists events on Rebuild, resize, rescue And end of audit period for all instances

23 How would you know if An.exists was dropped?

24 With billing, Events become Mission critical (cannot be dropped)

25 Our focus for the Havana release was reliable, audited & reconciled event collection

26 amqp Events Worker DB Raw data Request tracker Lifecycle usage deletes exists

27 reconciler update deletes Validator exists amqp Compute.instance.exists.verified

28 Things we check... Instance type match Tenant mismatch Rs-specific options field mismatches field validity image size (glance) Os architecture Os version Os distro

29 This all feeds into the downstream billing systems

30 Nova Notifications.* Monitor.* amqp YAGI Atom hopper Verified Glance stacktach Get the messages persisted and out of the production queues ASAP!

31

32

33

34

35 Ceilometer

36 Ceilometer proposed Folsom Timeframe oct 2012

37 Ceilometer started as Billing solution for OpenStack

38 In Grizzly, the mission changed to a Monitoring solution for OpenStack

39 Ceilometer was incubated At the start of Havana

40 StackTach was started Before the incubation process was established

41 During Havana we announced Stacktach would merge with Ceilometer

42 Opportunity to fix the Design problems we faced in StackTach

43 One worker per rabbit queue No idempotency in processing pipeline Database migrations were painful Extracting data from raw data json column not feasible Report batch window narrowing Hard coded event massaging Didn't use oslo

44 Ceilometer is sample-based and we need events

45 Some of our accomplishments during Havana

46 Ack/Requeue semantics in oslo.rpc Event storage in collector (mysql) Event body storage in mongo Events exposed in api Millisecond resolution storage Data-driven notification->event translation Detailed bp's for trigger manager And trigger pipeline

47 In Icehouse, we Finish the job

48 It starts with our notification->event mapping

49 _context_remote_address: u' ', _context_request_id: u'req-d68b36e f-9afb-d81435d64d66', _context_roles: [u'admin'], event_type: u'compute.instance.create.end', message_id: u'dae6f69c-00e0-41c0-b371-41ec3b7f4451', priority: u'info', publisher_id: u'compute.vagrant-precise', timestamp: u' :23: ', payload: { created_at: u' :23:41', deleted_at: u'', disk_gb: 0, display_name: u'testme', fixed_ips: [{address: u' ', floating_ips: [], meta: {}, type: u'fixed', version: 4}], instance_id: u'9f9d01b9-4a e27-398b21ab20d1', instance_type: u'm1.tiny', instance_type_id: 2, launched_at: u' :23: ', memory_mb: 512, state: u'active', state_description: u'', tenant_id: u'7c150a59fe714e6f af9688f0e', user_id: u'1e3ce f1a61c1996d1a531a2', reservation_id: u'1e3ce f1a61c1996d1a531a3', os_type: u'linux?', architecture: u'x86', image_ref: u'uuid', kernel_id: u'1e3ce f1a61c1996d1a531a5', ramdisk_id: u'1e3ce f1a61c1996d1a531a6',

50 event_type: compute.instance.* traits: &instance_traits user_id: fields: payload.user_id instance_id: fields: payload.instance_id host: fields: publisher_id Event plugin: name: split parameters: segment: 1 max_split: 1 os_architecture: fields: payload.image_meta.'org.openstack 1 architecture' event_type: - compute.instance.exists - compute.instance.update traits: <<: *instance_traits audit_period_beginning: type: datetime fields: payload.audit_period_beginning audit_period_ending: type: datetime fields: payload.audit_period_ending Traits

51 Then into the Event manager for routing

52 Request id 0001 event event event event event Instance id 0A7F-ABAB Collector event event Server MyServer Event manager event event event event Tenant BadGuy event event event

53 All unique requests Match * distinquished_by request_id expires last + 1 hour

54 Daily.exists Match Compute.instance.* Timestamp first.bod first.eod Compute.instance.exists audit_period beginning first.bod audit_period ending first.eod distinquished_by instance_id firing_delay 3600 firing_criteria Compute.instance.exists load_criteria Compute.instance.exists audit_period.begin first.bod 1 day audit_period.end first.eod 1 day Verified true

55 Triggering sends a Temporally ordered Set of events into the trigger pipeline

56 .verified.resize.update.end.exists Notifications Samples transform transform transform Events

57 Instead of adding tables to the database Generate new events

58 But there is still work to do Incremental Reporting framework Trigger pipelines for neutron, heat, Etc. Better notification support in other projects

59 And there is great potential Zombie/orphan detection Ip management Capacity planning

60 We need your help Join the fun wiki.openstack.org/wiki/ Ceilometer

61 Questions?