Filebeat is able to do multiline while collecting logs from the container. you can use autodiscover to configure it in many ways

Transcription

1 elasticguest8086 Since k8 by default is using the json log file driver which will not support call trace (multi-line log).. Any solution for it with ELK? like using logstash as a logging driver? Filebeat is able to do multiline while collecting logs from the container ^ good question. i tried adding the multiline wrapper to the filebeat configmap, but that resulted in it oftentimes grabbing multiplelines into one blob message you can use autodiscover to configure it in many ways i think there's some rate overflow issues with the multiline processor i elasticguest8086 is now known as vikrant considering how much logging a docker engine can generate with a lot of active containers you have some examples in Filebeat docs, and this blog post: mmhmm, thats what i did some events are parsed fine elasticguest5588 Hi It s possible to send filebeat module to a logstash? some are blobbed canti the logs processing is made on elasticsearch or by filebeat? yes, just amend the filebeat configmap with your logstash as the target elasticguest5588 Nice

2 @elasticguest5588 you can migrate the ingest pipeline to logstash with a utility as well to migrate the grok patterns and other What are you running in the container you had a n issue with? filebeat can do some basic stuff (like drop events with a filter), but logstash is the heavylifting processor, and es pipelines can do some lighter stuff jonathan I've had a lot of trouble amending the filebeat configmap with my logstash as the target; it seems to send the same log files repeatedly (and indefinitely), and never pick up new log lines. : it generates very chatty logs, but multiline ended up not able to keep up vikrant what advantages ES provide our TSDB? brandon_elastic Also note that if you are using an ingest node pipeline from the Filebeat module, and you direct the Beat traffic to Logstash instead, you can use these pipelines in place of the ingest node: speaking of kube-state-metrics, anyone have issues with the container constantly being killed due to OOMKilled? i can't track down why its doing that ok, I was wondering if there were multiple streams going to STDOUT, or if it was just one. Sounds like just one. I would pop a question on discuss and work through with engineering vikrant What about the application level metrics which are exposed by statsd? Is metric beat collecting from statsd also? : will do, thanks :)

3 I have not had any issues with kube-state-metrics, just at apply two pods start and one dies off then the one left lasts forever (or I never noticed) yeah, most of the time i see my ksm pod having like 1000 restarts while cluster isnt overloaded i figured it was just the older not updated image, but its same after you might need to modify the deployment.yaml for it and increase the memory limit above the default of 30MiB : did that set it to 200 same problem how large is your cluster? 6 minions, 1 master in dev err, sorry, 4/1 in dev prod seems to be okay, 6 minions,3 masters ive been trying to set that pod as a protected class, but didnt help you can use the graphite module to receive metrics from statsd: vikrant thanks. why should one chose ES over TSDB? for storing the metrics is there a timeframe or plans to get.net (and.net core) as one of the supported languages? ^ for APM that is vikrant, there are several benefits. To me, the most important one is: all your monitoring is in one place, so you can correlate logs, metrics and traces together, without jumping between applications

4 @ The APM team has that on the long term roadmap when I last spoke to them they share tags and metadata, so you can filter logs & metrics by the same terms, and watch them together ha, okay, so i wont be holding my breath short term Our APM is open source so the community or Microsoft might get to it before the APM team yeah i figured that would be the Regarding statsd: You can connect statsd to ES: VincentNgai Do any one exp EKS after install metricbeat then there appear grpc: received message larger than max (XXXXX vs ) error in EKS elasticguest4524 To be able to use the APM, do people need to do changes in their Application? VincentNgai, I would recommend posting to our forum or opening a github issue: & elasticguest4524: normally yes, you have to instrument your application, by adding a few lines, have a look to the docs elasticguest4447 you mentioned that APM is open source extension. Is this free or we need to buy x-pack subscription in order to get access to Elastic but somethings like node.js and python can be pretty well autoinstrumented Jamie-Elastic APM has multiple components vikrant does metric module provide support for non-core system metrics?

5 apm is open source, the specialized apm ui in kibana is x-pack basic so apm agents and the apm-server are open source the default dashboard for k8s in kibana seems to clump all clusters together is there a way to add a variable to it? or something? yes, you can edit any dashboard, or use a custom field in Metricbeat to filter by it in the dashboard you can add a filter or an input control for the cluster name to pivot between yes, I generally add a filter based on namespace to get just my app, or just kube-system Are there specific metrics you want to gather? The full set of metrics gathered are here: oh that namespace selector is here is a list of metricbeat modules, plus there are more on github from the community: i should update my dashboards more often Cool, I did not know Adam had the selector in there. Yeah, that control is good. I have been doing it with the "Add a filter" thing by the search bar. well, the filters can be a little restrictive, because they are all additive It is very hard to see the details of the dashboard the author created without being able to run in full screen mode.

6 :) i cant (or at least couldnt figure out how to) do A or B you can add it yourself, have a look to try the input you can configure it to switch between the values of a field very quickly vikrant Thanks I was trying to understand how metric will monitor the PV thanks guys, thats awesome will do it starts as additive, but in the options you can set it to be a single filter, build it as a visualization and add to the dashboard Is there a Github repository containing a demo application similar to the one that the author is demoing? yes, there is a repo. I will get it for you during the Q/A Great, thanks. There is also this one: github is in the blog post Thanks! sure thing, note: that one does not have APM. Adam's does. It's too bad I can't read any of the details or URL. demo.elastic.co brandon_elastic You can set your resolution higher in the youtube stream. Choose the gear icon. Also, as Dan pointed out, please feel free to use the demo site too!

7 Adam's demo is under elasticguest5160 Can you deploy the ElasticSearch cluster to Kubernetes? Could you compare the APM features to other APM products? elasticguest4848 can we do autodiscovery on newflow based monitoring elasticguest8266 Can you monitor JAVA VIRTUAL MACHINE metrics? elasticguest9790 Are there plans to support other schedulers outside of Kubernetes? Nomad for example? MikeRoy How does ELK compare to an EFK deployment on K8s? What are the advantages/disadvantages? Metricbeat has a JMX module, you can use it to gather metrics from the Java machine using you can run Elasticsearch in Kubernetes but be careful about what persistent volume you back it with. High io workloads will probably not do well on SDS or NFS. APM can do reponse time tracking, call tracing and display as well as error collection. More features are there and coming. Try it out, it's open can you elaborate slightly on how you would want to change processing of things based on netflow data? Will the Infra UI be part of Elastic Stack? elasticguest5426 when will APM be available apm is already available to download Infra UI will be part of Kibana, yes elasticguest3458

8 Has elasticsearch/k8s been used to monitor k8s pods and use it as a basis to autoscaling? (at least to your knowledge) - I am asking mainily in hindsight of XPACK/ML to start scaling before the demand is there, not after. elasticguest8287 Auto discovery support Mesos Marathon based deployments? that's a really interesting use case elasticguest3458, many people use it for monitoring, including us for our internal if you want to have alerting run polling and trigger an action via a webhook or have something poll ES data to make decisions to then autoscale you can certainly do that. I don't know of anyone using these metrics for autoscaling, but it makes sense elasticguest8287: not yet, Autodiscover supports Kubernetes and Docker (so anything running on top of Docker is supported too) that would actually be really cool, to use shipped metrics from pods/nodes into es, to fire off a scaling command against the api you could probably do that right now with if you can build a kibana visualization you can grab the query it makes for your own use as a short cut to an external poller ML can also do forecasting, so you can anticipate the future with Elasticsearch for autoscaling based on prior usage patterns elasticguest3458 that was exactly how I was thinking about. Thanks! elasticguest6870 what is the advantage of metric beat over infulxdb How soon will this Java implementation be available? elasticguest5426 how quickly can you start monitoring a new pod? any delay of it's going to pick up the pod in the scale of seconds-minute and with a module get the actual timestamp from the logs for the java agent for apm is beta, is that what you are asking for?

9 ive seen metrics be a little behind after initial spin, but no more than a minute after pod start, and then they are within 10s mark i think but i think that more depends on how backed up your ES cluster is than the metricbeat or heapster or whatever you're using navn X pack is now open source? If yes, is there any restrictions over the functionalities with respect to paid version? navn: elasticguest3458 Thanks for the insight! g2go. Good Presentation! this lists features of paid vs free versions the code is open, but to run it in production will still require a subscription for non basic -pack features x-pack features How soon will we get the slides? Yes. How soon will we get the slides? thank you for your time! navn any alerting modules in place as part of ELK? elasticguest8022 Thanks for the show! elasticguest9790 Appreciate the session. Thank you Thank you! elasticguest5160 Great session! Thank you! VincentNgai Thanks you~ elasticguest8287 Good session! Thank you! MikeRoy

10 Thanks for a great session... jumpy do you have a slack channel i think elastic co runs on irc channels elasticguest3280 Thank you.. #elasticsearch/#kibana/#beats/#logstash for community endpoints anyway a1exus thank you! but it'd be great if they had a slack shomeyer very informative session. thx a lot and have a nice day :-) jumpy thx very interesting discuss is a good entry point too, with the benefit of having answers recorded for future searches!