DevOps with SUSE: How SUSE Manager, SUSE Studio and SUSE Cloud APIs Facilitate Continuous Software Delivery Joachim Werner Wolfgang Engel Senior Product Manager SUSE/joe@suse.com Engineer SUSE/wengel@suse.com
2
Why?
4
Traditional Collaboration CODE Development APP OS Operations HW 5
Get Together! 6
The DevOps Way CODE APP Development OS Operations HW 7
Continues Delivery 8
What If You Want To...
Automate repetitive tasks? Generate customized reports from the data in a system? Hook systems together that weren't built to work together? Orchestrate several separate systems or tools? Write my own UI because I have different needs than what the authors of the tool had in mind? 10
11 Development Ease package creation Automate tests and code deployment Control application stack Metrics Operations Automate system installation/creation and configuration Control systems Monitoring
Then You Need APIs! 12
Setting The Stage...
Objectives of This Session: Demonstrate the power of using APIs in SUSE's Cloud and Management products in a real-life example Provide starting points and hints for your own work with the APIs Not: 14 A ready to run solution (just limited-scope proof of concept) Comprehensive API tutorial (just the tip of the iceberg, much more to discover below sea level)
Scenario QA environment for testing new builds of a custom RPM package Package should be tested 15 as part of an appliance image as an RPM maintenance update to a running instance Everything should be fully automated from the time a package is checked in to running the tests
Solution Components Open Build Service Build your RPMs SUSE Studio Build a new appliance image Deploy it to the Cloud SUSE Manager Deploy RPM-based update Run test suite SUSE Cloud 16 Run test instances
Open Build Service (OBS) 17 OBS builds binary packages for many distributions and platforms OBS makes them available for download
SUSE Studio + Streamline the OS + Simplify Deployments + Simplify Maintenance SUSE Studio is a collection of tools designed to improve the efficiency of building, managing and maintaining software virtual and cloud applications 18
SUSE Manager Modular Approach SUSE Manager delivers complete lifecycle management for Linux servers through its management, provisioning, and monitoring modules 19
SUSE Cloud Integrated platform and tools for hybrid clouds SUSE Linux Enterprise in public clouds 20 Private cloud infrastructure solution powered by OpenStack
Optional Components GIT or Subversion (everybody should be using a version control system) 21 Jenkins (to orchestrate) http://jenkins-ci.org/ Jenkins is like cron on steroids :-)
Getting Prepared...
Prerequisites (1) For replicating the full setup you need: 23 Open Build Service available here: www.open-build-service.org/ or on the SUSE Linux Enterprise SDK SUSE Studio www.suse.com/products/susestudio/ Talk to your sales contact for an evaluation version SUSE Manager www.suse.com/products/suse-manager/ Eval available on the website SUSE Cloud www.suse.com/products/suse-cloud/ Eval available on the website
Prerequisites (2) For replicating the full setup you need: 24 On the machine or instance the API automation is run: Python (part of SUSE Linux Enterprise) the requests module for Python: http://docs.python-requests.org/en/latest/index.html http://software.opensuse.org/package/python-requests Hardware: SUSE Studio Onsite needs a dedicated server The rest can be virtualized You need a lot of RAM for Studio, Manager, Cloud The API automation script can be run on any OS that has Python.
Prerequisites for just playing : Some of the components used here are available online... You can use the opensuse instance of the Open Build Service: You can use the free Studio Online: 25 https://build.opensuse.org http://susestudio.com/
The Script...
Workflow 27 Create a project in OBS Create a specfile for an RPM and add it to the project Build a new version of the RPM Update the RPM in SUSE Studio Update the RPM in SUSE Manager Trigger SUSE Studio image rebuild Trigger update of package with SUSE Manager Webhook to trigger test deployment of cloud image from SUSE Studio Use SUSE Manager to trigger test run of test suite
Automating an RPM Package QA Environment: 5 Rebuild image SUSE Studio RPM MyApp 1 6 Check in sources Redeploy image SUSE Cloud Instance 1 4 GIT 2 Instance 2 Sync repository/ upload RPM 8 Check out sources Update RPM 9 Run test SUSE Manager 3 28 Rebuild RPM 7 Sync repository
And One RingScript To Rule Them All: 1 2 3 4 5 6 7 8 9 29
Integration Via 30 Python-based web service process In the presentation we are skipping the web service part Demos are short executable Python scripts OBS, Studio, Manager, and Cloud APIs
We Are Going To Use Python Because... xmlrpc support is built in http and url handling built in OBS client API is usable as a Python module optional: python-requests module The setup can be replicated in other programming languages of your choice, like 31 Perl Ruby Java
Open Build Service
Open Build Service API 33 The OBS API is a RESTful API Authentication is done by sending a Basic HTTP Authorisation header (every time) Documentation: https://api.opensuse.org/apidocs/ (API Reference) http://en.opensuse.org/images/d/df/obs-cheat-sheet.pdf (Cheat Sheet for the command line client)
All code examples are in Python 2 syntax! And remember: In real life you need to do exception handling and other boring stuff! 34
A Hello World! Example... Using the hard way: urllib2 #!/usr/bin/env python import urllib import urllib2 url = 'https://api.opensuse.org' user = 'username' pwd = 'password' pwdmgr = urllib2.httppasswordmgrwithdefaultrealm() pwdmgr.add_password(none, url, user, pwd) authhandler = urllib2.httpbasicauthhandler(pwdmgr) opener = urllib2.build_opener(authhandler) urllib2.install_opener(opener) response = urllib2.urlopen(url + '/about') print response.read() 35
The Response... <about> <title>open Build Service API</title> <description>api to the Open Build Service</description> <revision>2.3.0.git201207101028</revision> </about> 36
Now Let's Introduce The Requests Module... 37 HTTP for Humans (quote from the website) ISC License (permissive BSD-style license) Very easy to use Well documented
Doesn't This Look Much Nicer? #!/usr/bin/env python import requests url = 'https://api.opensuse.org' user = 'username' pwd = 'password' r = requests.get(url + '/about', auth=(user, pwd)) print r.text 38
Or We Can Use The osc Command Line Client As A Library What is osc? 39 A command line client for OBS that you can use with a syntax just like svn to check out, modify, and commit packages to OBS But it can also be used to call the api directly! You can extend osc with your own commands. And: you can load it as a library from a Python script.
Preparing For Launch... zypper in osc osc Your user account / password are not configured yet. You will be asked for them below, and they will be stored in /home/joe/.oscrc for future use. Creating osc configuration file /home/joe/.oscrc... Username: username Password: password 40
Now You Can Use osc Like This: osc api about <about> <title>open Build Service API</title> <description> API to the Open Build Service </description> <revision>2.3.0.git201207101028</revision> </about> 41
Or From A Python Script, As Promised: #!/usr/bin/env python from osc import core, conf 3 This is step 3 in our workflow conf.get_config() url = conf.config['apiurl'] x=core.rebuild(url, 'home:joachimwerner', 'treeline', 'SLE_11_SP2', 'x86_64') print x 42
SUSE Studio
SUSE Studio API (1) Documentation: http://en.opensuse.org/opensuse:suse_studio_api (Background information, examples etc.) http://susestudio.com/help/api/v2 (API Reference) Client libraries: 44 http://susestudio.com/help/api/libraries_and_clients.html
SUSE Studio API (2) 45 The SUSE Studio API is a RESTful API Authentication is done by sending a Basic HTTP Authorisation header (every time) You need to create an API key for that in the web UI (once)
Creating the API Key 46
4 Sync The Updated RPM Repository To SUSE Studio
This Is A New API Call! POST /api/v2/user/repositories/<id>/refresh 48 As of today, this is only in SUSE Studio Online Planned for SUSE Studio Onsite 1.3 Workaround is to use the API calls for uploading single RPMs
5 Rebuild The Image
Trigger The Rebuild: #!/usr/bin/env python import requests url = 'http://susestudio.com/api/v2/' user = 'username' pwd = 'api_key' parameters = { 'appliance_id': '9999999', 'force': 'true', 'image_type': 'vmx'} r = requests.post(url + 'user/running_builds', params=parameters, auth=(user, pwd)) print r.text 50 You can look this up in the web UI!
A build id Is Returned: <build> <id>12345</id> </build> You can use this id to query the build status later... 51
6 Redeploy Image
Webhooks 53
Redeploy Image We use a webhook for that Here is a nice explanation on how to use them: 54 http://susestudio.com/help/api/webhooks.html You need your script to either start a web server (easy with Python) or e.g. run in Apache as a CGI script When the rebuild in Studio is finished, the webhook gets called
Create a New Image In OpenStack's Glance Image Store Example, using the current (soon to be deprecated) glance command line: glance image-create --name=#{data[:name] + " " + data[:build][:version]} --ispublic=true --disk-format=qcow2 --container-format=bare --copy-from #{data[:build][:download_url]} 55
SUSE Manager
SUSE Manager API The SUSE Manager API uses XML-RPC. Authentication is done by requesting a session key with the auth.login method call Documentation: https://www.suse.com/documentation/suse_manager/ 57 API Reference Manager Quick Start Guides and reference documentation
Connecting to SUSE Manager #!/usr/bin/env python import xmlrpclib url = "https://servername/rpc/api" user = "username" password = "password" api = xmlrpclib.server(url, verbose=0) session = api.auth.login(user, password) print api.user.listusers(session) 58
What's So Convenient About The XML-RPC API? 59 In Python, Java and other languages there are XMLRPC modules that allow to transparently communicate over the protocol Methods are just called like local methods and return Python or Java data structures (e.g. a Python list) that can be used directly No need for parsing the response. All XML parsing is done transparently
7 Sync The Updated RPM Repository To SUSE Manager
8 Update RPM On Test Instance
9 Run Validation Tests
SUSE Cloud
SUSE Cloud/OpenStack APIs (1) The OpenStack APIs are RESTful They can use Jason or XML Documentation: http://api.openstack.org/ (API Reference) http://docs.openstack.org/api/quick-start/content/ (API Quick Start Guide) http://docs.openstack.org/api/openstack-compute/programmer/content/ (Programming OpenStack Compute API with Shell and Python) More here: http://wiki.openstack.org/documentation#openstack_apis 64
SUSE Cloud/OpenStack APIs (2) 65 Authentication works by requesting a token from the keystone authentication service and using that during a session. Similar to Kerberos...
You Can Download An RC File To Configure Your Client For A SUSE Cloud 66
Get A Token: Example using curl, for a change. ;-) curl -X 'POST' -v https://servername:5000/v2.0/tokens -d '{"auth":{"passwordcredentials": {"username": "username", "password":"password"}, "tenantid":"453c05b"}}' -H 'Content-type: application/json' 67
Keystone Returns A Token In Jason Format Tip: From Python you can use the jason module to easily parse Jason responses 68
List Available Images curl -H "X-Auth-Token:ed6b342" https://servername:9292/v1/images 69
Wrapping Up...
If you haven't yet, you should really start looking into using our APIs! Thank you. 71
Send an e-mail to joe@suse.com with the subject APIs rock! to request more information and stay in touch on this topic! Thank you. 72
73 Corporate Headquarters +49 911 740 53 0 (Worldwide) Join us on: Maxfeldstrasse 5 90409 Nuremberg Germany www.suse.com www.opensuse.org
Unpublished Work of SUSE. All Rights Reserved. This work is an unpublished work and contains confidential, proprietary and trade secret information of SUSE. Access to this work is restricted to SUSE employees who have a need to know to perform tasks within the scope of their assignments. No part of this work may be practiced, performed, copied, distributed, revised, modified, translated, abridged, condensed, expanded, collected, or adapted without the prior written consent of SUSE. Any use or exploitation of this work without authorization could subject the perpetrator to criminal and civil liability. General Disclaimer This document is not to be construed as a promise by any participating company to develop, deliver, or market a product. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. SUSE makes no representations or warranties with respect to the contents of this document, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. The development, release, and timing of features or functionality described for SUSE products remains at the sole discretion of SUSE. Further, SUSE reserves the right to revise this document and to make changes to its content, at any time, without obligation to notify any person or entity of such revisions or changes. All SUSE marks referenced in this presentation are trademarks or registered trademarks of Novell, Inc. in the United States and other countries. All third-party trademarks are the property of their respective owners.