Don t Fly Blind. Christoph Eichhorn & Christian Düfel SAP Karlsruhe Germany

Transcription

1 Don t Fly Blind Christoph Eichhorn & Christian Düfel SAP Karlsruhe Germany

2 Router Microservice Architecture Your App UAA Some Service Payment Service Some Other Service Discovery Service Recommendation Service CF logs > "This is a stupid mistake","stacktrace":[ "java.lang.arithmeticexception: / by zero","\at com.sap.hcp.cf.sample.pickgreeting.pick( PickGreeting.java:49)","\tat com.iamjambay.cloudfoundry.stickysession. MainServlet.printGreeting(MainServlet.java:94)","\tat com.iamjambay.cloudfoundry.stickysession.mainservlet.doge t(mainservlet.java:137)","\tat javax.servlet.http.httpservlet.service (HttpServlet.java:635)","\tat javax.servlet.http.httpservlet.service(httpservlet.java:742)", x x

3 How to Investigate Failures on Live Systems? Is there an Issue? Where is it? Does it occur frequently? No debugging in production (debug port is closed) Can I understand it?

4 Logging with CF-Logs (Loggregator) Monitor current logs and retrieve recent logs: cf logs <appname> cf logs <appname> --recent But: No search No structure Very limited log history Shows only logs related to one application

5 Drain Logs to 3rd party log archive Two ways to feed logs to external system: 1. Via Firehose 2. Via Syslog Drain For an application developer, only the second option is possible But where do you send the logs to?

6 A twelve-factor app never concerns itself with routing or storage of its output stream from

7 Logging Service Wish List Easy to use Full Text Searching Rich and Searchable Context Information Display Anomalies Correlation of Logs Level Stack Trace App Log Time Space Org Metrics History of Logs

8 SAP Logging Service for CF Offered as service on Cloud Foundry marketplace Powered by Elastic Stack (Elasticsearch, Logstash, Kibana) Adapted to Cloud Foundry context with tailored parsing rules Predefined dashboards Multitenant stack Complementary open source logging libraries

9 Standard Features

10 SAP Logging Service Ready to Use Get ready to go in 3 simple steps 1. Find the logging service on cf-marketplace cf marketplace 2. Create an instance of the logging service cf create-service application-logs lite <you_name_it!> 3. Bind the service to your application cf bind-service <app_name> <service_instance> Your logs are now being fed into the logging service!

11 Kibana Dashboards Overview Usage Requests & Logs Statistics Performance & Quality Network & Load Prefabricated dashboards for common use cases and cross navigation

12 Overview

13 Kibana Dashboards Overview Usage Requests & Logs Statistics Performance & Quality Network & Load Prefabricated dashboards for common use cases and cross navigation

14 Insights from Router Logs Response-times Response status values Request size Payload size Request Methods Requests per second

15 Usage

16 Performance & Quality

17 Network & Load

18 Is that enough? Is there an Issue? Where is it? Does it occur frequently? Can I understand it??

19 Application Logs Relevant Fields: Log Level Precise Timestamp CorrelationID ContainerID Pure Log Message Stacktrace Custom Fields Logs have to be structured in a language-agnostic json format By Author:Basil Newburn - Bnewburn (talk) (Uploads), CC BY-SA 3.0,

20 Open Source Support Libraries We provide open source support libraries that: provide means to emit structured application log messages Instrument parts of application stack to collect request metrics Java: Node.js:

21 Setup Java Logging Support 1. Add Maven dependency 2. Choose logging framework (logback or log4j2) 3. Configure the framework (Blueprints are provided)

22 Kibana Dashboard for Application Logs Overview Usage Requests & Logs Statistics Performance & Quality Network & Load Prefabricated dashboards for common use cases and cross navigation

23 Requests & Logs

24 CorrelationID Connects related log entries Important for operations spanning multiple microservices Origin Can be provided in header of incoming request If not provided, request-id will be used Can be read from the MDC in order to forward it to another microservice

25 Stack trace in a single log message Too long stack traces are shortened in a sensible way: " STACK TRACE TRUNCATED ", "java.lang.illegalargumentexception: too long", "\tat com.sap.hcp.cf.logging.common.helper.stacktracegenerator.f3isasimplefunction(stacktracegenerator.java:23)", "\tat com.sap.hcp.cf.logging.common.helper.stacktracegenerator.f3isasimplefunction(stacktracegenerator.java:23)", [ ] "\tat com.sap.hcp.cf.logging.common.helper.stacktracegenerator.f3isasimplefunction(stacktracegenerator.java:23)", "\tat com.sap.hcp.cf.logging.common.helper.stacktracegenerator.f3isasimplefunction(stacktracegenerator.java:72)", " OMITTED ", "\tat com.sap.hcp.cf.logging.common.helper.stacktracegenerator.f2isasimplefunction(stacktracegenerator.java:54)", "\tat com.sap.hcp.cf.logging.common.helper.stacktracegenerator.f2isasimplefunction(stacktracegenerator.java:54)", [ ] "\tat com.sap.hcp.cf.logging.common.helper.stacktracegenerator.f2isasimplefunction(stacktracegenerator.java:54)",. "\tat com.sap.hcp.cf.logging.common.helper.stacktracegenerator.f2isasimplefunction(stacktracegenerator.java:78)", "\tat com.sap.hcp.cf.logging.common.converter.convertertest$x.run(defaultstacktraceconvertertest.java:99)", "\tat java.lang.thread.run(thread.java:33)"

26 Router Dynamic Log Level Your App Some Service Some Other Service Recommendation Service UAA Payment Service Discovery Service

27 Kibana Dashboards Overview Usage Requests & Logs Statistics Performance & Quality Network & Load Prefabricated dashboards for common use cases and cross navigation

28 No Free Lunch Resources are limited / cost money Quotas: Log volume per hour Burst limitation (log volume per second) Service plans with different quotas available

29 Statistics

30 Don t Fly Blind Give it a try: 1. Get the right tool cf create-service 2. Open your eyes cf bind-service 3. Switch on the autopilot use open source library 4. Enjoy your CF-journey with visual flight conditions! Thank you!