OpenStack Ceilometer Tong Li (IBM) Brad Topol (IBM)
Agenda What is Ceilometer? Brief history of Ceilometer Meters of Ceilometer Workflow Ceilometer Architect & Building Blocks Collecting Process Pipeline Transformer Publisher Collector Data store drivers Alarm Dispatcher Ceilometer ReSTful API Ceilometer Python Client Ceilometer in Devstack Inter-Component Security & High Availability Who are the users of OpenStack Ceilometer Ceilometer Road Map Q & A (Demo maybe)
Who are we? Tong Li IBM senior software engineer Twitter @email4tong Email: litong01@us.ibm.com Brad Topol IBM distingushed engineer Twitter @bradtopol Email: btopol@us.ibm.com
What is Ceilometer? OpenStack measurement project Julien Danjou Ceilometer PTL A system that keeps track of what is happening in OpenStack clusters. It was originally created for billing but it has been evolved into an all purpose measurement repository. It can be extended for purposes such as performance report, troubleshooting, capacity analytics, smart scheduling, optimization, ultimately BIG DATA etc.
Brief History of Ceilometer Before year 2012 billing has been left out of OpenStack core so far as it was not the primary problem and was non-trivial, yet almost every OpenStack deployment needs a way to track usage information Started in May 2012 by developers from Canonical, DreamHost, RedHat, Dell, Intel, AT&T, etc. Targeted to do metering, rating, billing Grizzly objective: The project aimed to become the infrastructure to collect measurements within OpenStack so that no two agents would need to be written to collect the same data. Havana objective: remains the same as Grizzly objective plus become an integrated OpenStack project. Current PTL: Julien Danjou 91 contributors, 7 core reviewers
Meters of Ceilometer (from Nova)
Meters of Ceilometer (from Neutron) Additional measurements: http://docs.openstack.org/developer/ceilometer/measurements.html
Workflow Collect Transform Publish Store Read Collect interesting info from OpenStack components (or any other components), create meters (or samples) Transform meters into other meters if needed Publish meters to any destination (including Ceilometer itself) Store received meters into data store Read data via Ceilometer REST API
Ceilometer Architecture
Collecting Notification bus (AMQP) queue name: notification.info Nova Neutron Cinder Glance Swift API API API API API Polling Notification Handlers, Central Agents Publishing pipeline Publishing
New in Grizzly! Pipeline Ceilometer collector & agent Meter Transformer Transformer Transformer Publisher Publisher Publisher Receiver Receiver Receiver Pipeline: a set of transformers mutating meters into something that publishers know how to send to external systems. Configure pipelines @pipeline.yaml
New in Grizzly! Transformer Meter Name=cpu.time Value=5 Timestamp=T Meter Name=cpu.time Value=20 Timestamp=T+1 Transformer Meter Name=cpu.time Value=80 Timestamp=T+2 Meter Name=cpu.percentage Value=9 Timestamp=T+2 Transform meters into new meters!
New in Grizzly! Publisher Meter Ceilometer RPC Publisher AMQP signed message Ceilometer Message queue Other Publisher External system Billing, monitoring, alarming, statistics, capacity planning
Data Store Meter Message Queue ceilometer.collector.metering Dispatchers Default backend SQL Database MongoDB Database HBase New in Grizzly New in Havana Multiple backend choices IBM DB2
Alarm New in Havana Both alarm evaluator and alarm notifier are Ceilometer services 1. Alarm evaluator uses Ceilomter public APIs to pull meters to produce alarms according to alarm definition, then place the alarm message onto the alarm queue 2. Alarm notifier read the alarm queue then inform external systems (log, rest, test)
Dispatchers New in Havana Multiple dispatchers support allows Ceilometer to be extended easily with many different outlet for all meters coming into Ceilometer. Dispatchers can be easily developed and deployed to add more features. For example, sending email, text messages. 1. Dispatcher only needs to implement one method. 2. By default, database dispatcher is used even if the configuration file has no trace of any dispatchers. 3. To configure multiple dispatchers: [ceilometer.dispatcher] file = ceilometer.collector.dispatcher.file:filedispatcher database = ceilometer.collector.dispatcher.database:databasedispatcher
Ceilometer ReSTful API request Ceilometer API Consumers Ceilometer API Billing, monitoring, alarming, statistics, capacity planning HTTP REST w/ JSON & XML Storage abstraction layer Database Raw events: GET /v2/meters/vcpus Statistics (sum, average, min, max ): GET /v2/meters/vcpus/statistics New in Grizzly! Filter: GET /v2/meters/vcpus?q.field=project_id&q.value=foobar&q.op=eq Group statistics by period: GET /v2/meters/vcpus/statistics?period=3600
Ceilometer ReSTful API response Response can be in JSON or XML Raw events: /v2/meters/storage.object.size Statistics (sum, average, min, max ): /v2/meters/storage.api.request/statistics
Ceilometer Python Client
Ceilometer in Devstack devstack localrc Enable the ceilometer metering services enable_service ceilometer-acompute,ceilometer-acentral,ceilometer-collector Enable the ceilometer alarming services enable_service ceilometer-alarm-evaluator,ceilometer-alarm-notifier Enable the ceilometer api services enable_service ceilometer-api
Inter-Component Security
Ceilometer High Availability
Who uses Ceilometer? https://etherpad.openstack.org/p/ceilometerfeeedback
Roadmap Grizzly Havana Icehouse Incubated Project Integration with Horizon Agents for other components o Swift o Ceph? o Nicira? SQLAlchemy storage driver Multi-Publisher API v2 o User accessible API o More aggregation o Multi-dimension Integrated Project Integration with Horizon Publishing meters to other systems Enhance SQL driver Alarming Integration with Heat Deprecating APIv1 Completing APIv2 Move publishing part to Oslo and other projects Tighter integration with Nova Nova-scheduler integration OpenStack Design Summit in Hong Kong
Questions? http://launchpad.net/ceilometer http://docs.openstack.org/developer/ceilometer http://wiki.openstack.org/ceilometer Freenode: Mailing List: #openstack-metering openstack-dev [ceilometer]