Get started with the IBM Alert Notification service. Author: Ray Stoner Senior Managing Consultant IBM Cloud Client Technical Engagement Lab Services

Transcription

1 Get started with the IBM Alert Notification service Author: Ray Stoner Senior Managing Consultant IBM Cloud Client Technical Engagement Lab Services

2 Contents 1. Introduction ANS features and functions Alert Notification solution overview Prerequisites Adding the Alert Notification service to your organization Install the Alert Notification sample application Launch the sample application Adding users, groups, and schedules to ANS Create a user More information on the scheduling option Manage groups Notification and escalation policies Managing alerts in the Alert Viewer Swagger API Curl command sample Integrations Network Operations Insight Slack New Relic Runbook Including the Runbook link in a notification policy Mobile app Use cases Simple alert first responder use case Escalation use cases First responder escalates to a Slack channel Service levels are being impacted Operational use case Related links Copyright IBM Page! 2 of! 52

3 1. Introduc;on IBM Alert No8fica8on (ANS) is an IBM Bluemix service that instantly delivers no8fica8ons of problem occurrences in your Bluemix environment using automated , Short Message Service (SMS), and voice messaging. IBM ANS lets you: Define groups so alerts are sent to the appropriate people for a problem or class of problem. Create groups based on administra8ve roles, applica8on names, department names, or other criteria. Customize filters for aler8ng different users based on incident type and severity. IBM posi8ons the Alert No8fica8on Service (ANS) as a component of an overall cloud- based incident management solu8on. ANS is intended to complement services such as IBM Netcool Opera8ons Insight (NOI) and Applica8on Performance Management (APM) and serve as the no8fica8on tool for the solu8ons. This document shows you how to set up ANS in a Public Bluemix account, although they can also be applied in SoOware as a Service (SaaS) and Dedicated Bluemix environments. We verified SaaS and Dedicated Bluemix environments when this document was wriqen. You only need to set up one instance of ANS (SaaS or Bluemix) to deliver alerts for your enterprise ANS features and func;ons IBM Alert No8fica8on lets you: Make sure you do not miss ac8onable, cri8cal alerts. Route alerts to the right people. Accelerate alert response with automated escala8on policies. IBM Alert No8fica8on helps you: Establish no8fica8on policies by filtering alerts and crea8ng groups to focus on cri8cal issues and disregard distrac8ons. Track alerts with an alert viewer feature to priori8ze no8fica8ons, perform ac8ons, and monitor no8fica8ons virtually in real 8me. Establish no8fica8on policies by filtering alerts and crea8ng groups: Classify alert filters based on frequency, incident type, severity, "x- in- y" type scenarios and status. Align experts and groups with alert classifica8ons and send no8fica8ons to several experts at once. Create groups based on administra8ve roles, applica8on names, department names, or other criteria. Copyright IBM Page! 3 of! 52

4 Store contact details of experts to be automa8cally no8fied of problems in their area of responsibility. Escalate unacknowledged alerts aoer a set 8me period to the appropriate contact. Track alerts with an alert viewer feature: Track alerts from their receipt by the system through acknowledgment to resolu8on. Track changes with an alert history. 2. Alert No;fica;on solu;on overview A basic run8me flow related to this architecture might look like this: 1. An alert is raised through NOI or the POST API and sent to the Alert No8fica8on service. 2. The Alert No8fica8on service processes the alerts through the no8fica8on policies and delivers the alert as specified in the policy ( , SMS, Slack or voice). 3. The alert is delivered through one or more op8ons, , SMS, Slack or voice. 4. The first responder, development team, and the incident manager use collabora8on tools for alert resolu8on. Copyright IBM Page! 4 of! 52

5 3. Prerequisites Before using ANS, you need the following: Working Bluemix account Bluemix and CF command line tools Mobile device Browser: If you use MicrosoO Internet Explorer, make the security and privacy se`ngs are set to medium- high or lower. If you use Mozilla Firefox or Google Chrome, make sure third- party cookies are allowed. 4. Adding the Alert No;fica;on service to your organiza;on Log into your Bluemix account and click the plus sign to add the service. Click DevOps. Copyright IBM Page! 5 of! 52

6 Click the IBM Alert No;fica;on service. Leave the service unbound. Scroll down and click Create. Copyright IBM Page! 6 of! 52

7 The service is added to your organiza8on and immediately launched: Copyright IBM Page! 7 of! 52

8 5. Install the Alert No;fica;on sample applica;on Use the Cloud Foundry command line to login to Bluemix and download the sample applica8on from GitHub: git clone Initialized empty Git repository in /home/user/alert-notificationhelloworld/.git/ remote: Counting objects: 53, done. remote: Total 53 (delta 0), reused 0 (delta 0), pack-reused 53 Unpacking objects: 100% (53/53), done. Go to the newly created directory: user@home>~]$ cd alert-notification-helloworld/ Edit the manifest.yml file and assign a unique hostname as shown: user@home>alert-notification-helloworld]$ vi manifest.yml Copyright IBM Page! 8 of! 52

9 applications: - path:. memory: 256M instances: 1 domain: mybluemix.net name: AlertNotificationSampleApp host: CSMOCASEalertnotification disk_quota: 1024M Push the sample applica8on to Bluemix: user@home>alert-notification-helloworld]$ cf push AlertNotificationSampleApp Using manifest file /home/user/alert-notification-helloworld/ manifest.yml Creating app AlertNotificationSampleApp in org cent@us.ibm.com / space ServiceManagement as rstoner@us.ibm.com... OK Creating route CSMOalertnotification.myBluemix.net... OK Binding CSMOalertnotification.myBluemix.net to AlertNotificationSampleApp... OK Uploading AlertNotificationSampleApp... Uploading app files from: /home/user/alert-notification-helloworld Uploading 24.9K, 13 files Done uploading OK Starting app AlertNotificationSampleApp in org cent@us.ibm.com / space ServiceManagement as rstoner@us.ibm.com > Downloaded app package (20K) -----> IBM SDK for Node.js Buildpack v Based on Cloud Foundry Node.js Buildpack v > Creating runtime environment NPM_CONFIG_LOGLEVEL=error NPM_CONFIG_PRODUCTION=true NODE_ENV=production Copyright IBM Page! 9 of! 52

10 NODE_MODULES_CACHE=true -----> Installing binaries > Uploading droplet (15M) 0 of 1 instances running, 1 starting 1 of 1 instances running App started OK App AlertNotificationSampleApp was started using this command `./ vendor/initial_startup.rb` Showing health and status for app AlertNotificationSampleApp in org cent@us.ibm.com / space ServiceManagement as rstoner@us.ibm.com... OK requested state: started instances: 1/1 usage: 256M x 1 instances urls: CSMOalertnotification.myBluemix.net last uploaded: Thu Jul 21 14:37:17 UTC 2016 stack: unknown buildpack: SDK for Node.js(TM) (ibm-node.js , buildpackv ) state since cpu memory disk details #0 running :38:09 AM 0.0% 80.2M of 256M 58.4M of 1G Use the Cloud Foundry command line to list the Alert No8fica8on service. You need the service name for the next command: user@home>alert-notification-helloworld]$ cf services grep Alert IBM Alert Notification-j4 alertnotification authorizedusers Microservices-UI-rstoner-1823 create succeeded Bind the sample applica8on to the Alert No8fica8on service: user@home>alert-notification-helloworld]$ cf bind-service AlertNotificationSampleApp "IBM Alert Notification-j4" Copyright IBM Page! 10 of! 52

11 Binding service IBM Alert Notification-j4 to app AlertNotificationSampleApp in org / space ServiceManagement as OK TIP: Use cf restage AlertNotificationSampleApp to make sure your env variable changes take effect. Restage the sample applica8on to the Alert No8fica8on service: cf restage AlertNotificationSampleApp Restaging app AlertNotificationSampleApp in org / space ServiceManagement as -----> Downloaded app package (20K) -----> Downloaded app buildpack cache (712K) -----> IBM SDK for Node.js Buildpack v Based on Cloud Foundry Node.js Buildpack v > Uploading droplet (15M) 1 of 1 instances running App started OK App AlertNotificationSampleApp was started using this command `./ vendor/initial_startup.rb` Showing health and status for app AlertNotificationSampleApp in org cent@us.ibm.com / space ServiceManagement as rstoner@us.ibm.com... OK requested state: started instances: 1/1 usage: 256M x 1 instances urls: CSMOalertnotification.myBluemix.net last uploaded: Thu Jul 21 14:37:17 UTC 2016 stack: unknown buildpack: SDK for Node.js(TM) (ibm-node.js , buildpackv ) Copyright IBM Page! 11 of! 52

12 state since cpu memory disk details #0 running :40:42 AM 80.3% 74.6M of 256M 58.4M of 1G The sample applica8on is now ready to use Launch the sample applica;on Your organiza8on workspace should now have the sample applica8on installed. Click the arrow buqon in the boqom right corner. The following tab opens. Click Send Sample Alert: Return to your workspace and launch the ANS user interface (UI). Navigate to services and click the Alert No;fica;on service 8le to open the applica8on: Copyright IBM Page! 12 of! 52

13 Click Launch: By default, the UI opens to the Alert Viewer. The test alert created in the prior steps is included. 6. Adding users, groups, and schedules to ANS ANS provides the facili8es to set up users, groups, and schedules, which let you manage who gets what alerts and when. To set up your ini8al user, group, and schedule, follow these steps: Click the Manage Users and Group icon on the upper right of the UI: Copyright IBM Page! 13 of! 52

14 The tabs in the UI change to Manage Users and Manage Groups. Click Manage Users Create a user Click Create a User: Add the user details: Copyright IBM Page! 14 of! 52

15 Click Work Hours. For this user I selected the default hours, which are 8am to 5pm, Monday through Friday: Copyright IBM Page! 15 of! 52

16 You can also specify if the person in on call aoer hours Set up the no8fica8on preferences for the user. Click No;fica;ons: Copyright IBM Page! 16 of! 52

17 Select the no8fica8on method for each op8on by clicking the down arrow on the right. Copyright IBM Page! 17 of! 52

18 Once completed, the no8fica8on op8ons for the user are displayed. Click Save. Copyright IBM Page! 18 of! 52

19 6.1.1.More informa;on on the scheduling op;on To remove a no8fica8on method, click the minus sign (- ) on the right. You can also send a reminder to a user that he or she will be on call: Copyright IBM Page! 19 of! 52

20 By selec8ng days, the user will get a reminder three days before going on call Manage groups Click Manage Groups: Click Create a Group: You have two op8ons: 1) Create a group 2) Create a group schedule Copyright IBM Page! 20 of! 52

21 I created a group named CSMO. Note: Use the down arrows at the right to select the user and the group manager. Use the minus sign to remove users. Set up a no8fica8on schedule for your group. I selected 8am to 5pm from the ShiO PaQerns list: Copyright IBM Page! 21 of! 52

22 Assign a name to the schedule and select a start day and how far in advance to generate the schedule. Click Save. I used the default of Monday, and I selected to have the schedule generated for two weeks: Return to the main window of the ANS UI and click Alert No;fica;on: Copyright IBM Page! 22 of! 52

23 7. No;fica;on and escala;on policies Now that you have user(s) and a group defined, the next step is to create a no8fica8on policy. ANS requires no8fica8on polices in order to deliver the alerts to the correct personnel. Select Manage No;fica;on Policies and click the plus sign to create a no8fica8on policy: Enter the name of your policy and then click Add Rule: Copyright IBM Page! 23 of! 52

24 The Rule editor comes with simple pre- defined rules, or you can create your own: To create your own, click Add Rule, add a descrip8on, select an aqribute and operator, and add a value. For example: Copyright IBM Page! 24 of! 52

25 To use a pre- defined rule and have an sent for the alert, select the rule and click Save: On the main window, click Add recipient: Copyright IBM Page! 25 of! 52

26 You have three op8ons: Users, Groups, or Channels: Use the down arrow on the right to see a list of available groups. I selected the CSMO group created earlier. Copyright IBM Page! 26 of! 52

27 Select the group and click Save. On the main window, you have the op8on to send a no8fica8on every 8me an alert is received or once over an eight hour period. I chose Each ;me the rules are matched: To add an escala8on to this no8fica8on policy, click Add Escala;on: Copyright IBM Page! 27 of! 52

28 If an alert is not acknowledged within 20 minutes, the alert is escalated to opera8ons management. Contacts are added to the escala8on policy the same as described earlier: Click Save to save the no8fica8on policy. To enable the policy, click Disabled: The policy is now ac8ve. Sample Copyright IBM Page! 28 of! 52

29 Note the ability to login and acknowledge or unacknowledged the alert. 8. Managing alerts in the Alert Viewer The Alert Viewer gives you a real- 8me view of the alerts in your IT environment. The Alert Viewer screen is comprised of two parts: the list of alerts and the Alert History pane. In the list of alerts, each row in the list is a unique alert. In Alert History, when you select an alert, it shows changes that were made by you and other operators against the alerts. A 8mestamp indicates when changes were made. Copyright IBM Page! 29 of! 52

30 To manage your alerts in the Alert Viewer: Click Alert Viewer. Choose My Alerts to see only the alerts for which you are defined as a recipient. The default value is All Alerts. Alert No8fica8on provides default filters for the most common monitoring scenarios. Click use the check boxes to apply one or more filters. Filter and Click anywhere in an alert row to see the details for that alert displayed in the Alert History sec8on of the screen. To sort a column, click the arrow in the column header. You can sort the column into ascending or descending order, remove the sort, or hide the column. An arrow next to the column text indicates the sort order. Use the icons in the Ac8ons column to respond to an alert. Each 8me that you change a response to an alert, the Alert History sec8on of the Alert Viewer is updated. 9. Swagger API Use the Swagger API to generate alerts into ANS. Addi8onally, use the API to delete and get alerts. The API is available through a web interface. The URL is found in the creden8als sec8on for the no8fica8on service: Go to the URL: Copyright IBM Page! 30 of! 52

31 Click Show/Hide to show the available API and click Post: Fill in the fields for pos8ng an alert. Click Example Value to have it populate the body window as shown: Copyright IBM Page! 31 of! 52

32 Fill in the following highlighted fields in the body sec8on as shown in this example. Hint: This text can be copied and pasted into the Swagger body window. { "Identifier": "Alert1", "What": "JVM Heap Exhausted", "Where": "PubBMX", "Severity": "Fatal", "Type": "Problem", "Source": "JVM", "ApplicationsOrServices": [ "MyApp" ], "URLs": [ { "Description": "Launch to IBM", "URL": " } Copyright IBM Page! 32 of! 52

33 ], "Details": [ { "Name": "BMX Service", "Value": "Your MicroService" } ] } Click Try IT Out. Authen8ca8on is required for the API: Copy and paste the ANS creden8als as shown earlier: Copyright IBM Page! 33 of! 52

34 Scroll down. Your alert is successfully sent if you see a response code of 200. The alert is now in your Alert Viewer: Copyright IBM Page! 34 of! 52

35 The Delete and Get API func8ons rely on the provided alert ID. Those func8ons have been successfully tested. 10. Curl command sample The following code is a sample curl command that you can use in a program or script. Subs8tute the values in red with the proper creden8als and data you want to send to ANS. When using the API to send alerts, be sure to leverage the URL sec8ons. This command populates the link icon, giving you the ability to launch in context from the ANS UI: #!/bin/bash curl -v -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -u username:password -d'{ "Identifier": "string", "What": " string ", "Where": " string ", "Severity": " string ", "Type": " string ", "Source": " string ", "ApplicationsOrServices": [ " string " ], Copyright IBM Page! 35 of! 52

36 "URLs": [ { "Description": "string", "URL": "string" specify more than one if needed. } ], "Details": [ { "Name": "string", ==== URL to launch. you can ==== Additional optional details, data is visible by the! icon in the ANS UI "Value": "string" } ] }' ' exit 11. Integra;ons ANS has been integrated with both Slack and NOI Network Opera;ons Insight Most of the work for integra8ng ANS with NOI is done in either OMNIbus or Impact. Both of these integra8ons were performed by my Netcool administrator. Netcool requires the API key or service creden8als for ANS. In your Bluemix workspace, launch the Alert No8fica8on service and click Service Creden;als: Copy the creden8als to your clipboard. The NOI integra8on requires the name and password informa8on as highlighted. Provide this informa8on to the NOI administrator. Copyright IBM Page! 36 of! 52

37 See How to Guide for Netcool Opera8ons Insights for Bluemix for details on this integra8on. See Sending Events with Netcool/OMNIbus for OMNIbus integra8on steps. See Sending Events with Netcool/Impact for Impact integra8on steps. The following screens show the ANS configured into NOI: Copyright IBM Page! 37 of! 52

38 Note that if NOI receives a clearing event, NOI archives the event in ANS. With this deployment, Impact was customized to add two URLs using the API to the alert in ANS. 1. Link1 launches the Netcool event viewer 2. Link2 launches to the individual event in the event viewer Slack Slack integra8on is performed using a webhook. When the webhook is created, a URL is generated. This URL is required in ANS. Addi8onally, you must create a no8fica8on policy to allow the alert to arrive in Slack: Copyright IBM Page! 38 of! 52

39 In the ANS UI, click Manage Integra;ons then click Create Channel Collec;on: Accept the warning message, fill in the fields, and save: Copyright IBM Page! 39 of! 52

40 Be sure to enable the Channel Collec8on by clicking the Disable buqon: Next, create a no8fica8on policy to send alerts to Slack. Create a policy as described earlier, this 8me for recipients. Select Channel. Select the name of the created Slack integra8on and click Save. Copyright IBM Page! 40 of! 52

41 Be sure to enable your policy by clicking the Disabled buqon as shown earlier. You can now include the Slack channel as a no8fica8on recipient. Note: This is strictly just a no8fica8on to Slack; you cannot perform acknowledge or unacknowledge in Slack New Relic New Relic integra8on is performed through a webhook. From the UI, select the Manage Integra;ons tab. Click New Relic. Click Create Webhook URL. Copyright IBM Page! 41 of! 52

42 Provide a name for your webhook and click Save. Your Webhook is created. Click OK. Be sure to enable the webhook and provide the URL to your New Relic administrator for configura8on into New Relic. Copyright IBM Page! 42 of! 52

43 11.4. Runbook Runbook integra8on uses an API to integrate with ANS. Your Runbook administrator can provide you with the necessary API key and password. Go to the Manage Integra;ons tabs and click Runbook. Click the + sign to add the API informa8on: Enter the API key, password, and select where the Runbook service is running. In this example, I selected Marketplace: Copyright IBM Page! 43 of! 52

44 Test your connec8on. Note: You cannot save your connec8on un8l the test is successful. Copyright IBM Page! 44 of! 52

45 Including the Runbook link in a no;fica;on policy Once configured, you can include the link to a Runbook in a no8fica8on policy. This link is delivered and only available within the no8fica8on. In the No8fica8on Policy window, select or create the policy you want to work with and the last op8on for the policy: Click the + sign to add the Runbook and select the a Runbook from the drop- down menu. Copyright IBM Page! 45 of! 52

46 The selected Runbook(s) is associated with the Alert No8fica8on policy. Save the policy and make sure the policy is enabled. The link to the Runbook is provided in the no8fica8on. You can also pass parameters to Runbook, for example Bluemix Org informa8on, applica8on URLs, etc: Copyright IBM Page! 46 of! 52

47 12. Mobile app The IBM Alert No8fica8on mobile applica8on is available on itunes and GooglePlay. Once the applica8on is installed on your mobile device, you can follow these steps to get it working: Login to your Bluemix workspace through the browser on your mobile device and launch the ANS UI. ANS will recognize you are on a mobile device and prompt you to add the ANS subscrip8on to the mobile app. AOer selec8ng add subscrip;on you have an opportunity to replace the subscrip8on ID with a meaningful name. You can also edit the name later, which is highly recommended if you need to use mul8ple ANS UIs. Copyright IBM Page! 47 of! 52

48 When ANS is launched on the mobile device it display a list of instances to connect to: Copyright IBM Page! 48 of! 52

49 13. Use cases In addi8on to receiving alerts through the API, the integra8on with NOI also provides alerts from the following sources: Bluemix event feed / JSON objects IBM Performance Management OMNIbus Probes ( , syslogd, ProbeWatch, and Connec8on Watch) New Relic The following scenarios detail the roles that specific members of your team complete in the alert no8fica8on process Simple alert first responder use case The first responder receives an alert on his or her mobile device, acknowledges the alter through the applica8on, and begins the runbook process to resolve the issue. No further ac8on is required in ANS. Alerts received in or the applica8on allow you to acknowledge or unacknowledge the alert using embedded links, as long as they are provided as part of the alert. Copyright IBM Page! 49 of! 52

50 13.2. Escala;on use cases First responder escalates to a Slack channel Should the first responder be unable to resolve the issue, the alert is unacknowledged. The first responder re- sends the alert to the Slack channel that the developer is monitoring. Click the paper airplane in the UI, select the recipient, add addi8onal informa8on, and click Send. The developer receives the message and corrects the issue Service levels are being impacted In this scenario, an alert for a poor response 8me was unacknowledged by the first responder and a no8fica8on sent to the developer. As part of the no8fica8on policy, an escala8on 8mer is running. If the alert goes unresolved for 20 minutes, a no8fica8on is sent to the incident owner and an escala8on message is sent to the Slack channel. Copyright IBM Page! 50 of! 52

51 Here is a sample escala8on and Slack no8fica8on: Copyright IBM Page! 51 of! 52

52 ! Opera;onal use case The developer has automated the overnight applica8on deployment (A,B) process. The code that performs this task leverages the ANS API to provide alerts so that the pass/fail deploy status is easily obtained. These alerts are for informa8onal purposes to the developer regarding the deployment, since a failure would not stop the applica8on from running. In the example below, the first link takes you to the URL of the applica8on, the second link takes you to the Bluemix workspace. The applica8on s Slack channel is also no8fied to make sure the applica8on owner is aware of the status 14.Related links For more informa8on: Bluemix: hqps://ibm.biz/bd45yr IBM Alert No8fica8on hqps://ibm.biz/bdryk5 NOI: hqps://ibm.biz/bd45yr Slack: hqps://ibm.biz/bd45yb Copyright IBM Page! 52 of! 52