Click to edit Master title style

Transcription

1 Click to edit Master title style SCALING NETWORK MONITORING IN A LARGE ENTERPRISE BroCon 2016 Austin, TX

2 Click to edit Master Who title am style I? I work for Amazon s Worldwide Consumer Information Security group

3 What Click are to we edit going Master to talk title about? style How we scaled our network monitoring solution while the network is continuously growing

4 Click Why to edit do Master we even title do this? style Understanding what is occurring on our corporate network is important to us

5 Click to edit In Master the beginning title style

6 Click How to do edit we Master approach title this? style We originally decided on using vendor network sensors to get visibility in to what was occurring on our network

7 Click to edit How Master we started title style off Decided a vendor appliance was an effective way of gathering the data we needed We can buy network sensors, right? So we bought network sensors and plugged them into our network

8 Click to Vendor edit Master network title sensor style Life was much simpler back then... 1Gb/s capable firewalls SPAN sessions from our routers to vendor network sensors Small number of firewalls to monitor We got layer 3 and layer 4 header information from this network sensor

9 Click It looked to edit something Master title like style this Authorized users Vendor appliance Corporate network Netflow export SPAN session Router Firewall The Internet Netflow collector

10 Click to edit What Master is a SPAN title port? style

11 Click Where to edit do we Master go from title here? style Our network traffic volume kept growing Our sensor vendor stopped selling and supporting the platform we were using Increased internal maturity about using this data Vendor Management platform can t scale Driven by API usage by internal customers Started getting close to the limit of network sensors the management platform could handle

12 Click to edit Master Future proofing? title style We have a vendor s system we re starting to push the limits on What features do we need? Do we continue to buy or do we look at building instead?

13 Click to edit Master Build title vs style Buy Build Buy Speed of execution Control Vendor support Logistics Performance

14 Click Pushing to edit for Master the next title style level My co-workers evaluated various options nprobe Snort Suricata Bro

15 Click to edit Bro Master Generation title style One Ran on a single host Connected to our router via a 10G fiber link SPAN session from the router to our Bro host

16 Bro Click Generation to edit Master One looks title like style Authorized users Log store Bro Corporate network Netflow export SPAN session Router Firewall The Internet Netflow collector

17 The Click challenges to edit Master of Generation title style One The Bro host was a single point of failure Individual host installs have high operational costs High traffic volumes on our SPAN sessions caused our router to reboot Will this continue to scale with the growth of our network?

18 Click to edit Master Scorecard title style Vendor solution Generation One Single point of failure? Data collected via SPAN SPAN Control Scalability Logistics / Install effort Cost per Gb/s $$$ $

19 Click to edit And Master we are title done! style Or so we thought.

20 Click to edit Along Master came title Seth style Seth spotted everything in the history field was in upper case Turned out to be a trivial configuration change We started off with 32GB of RAM in our hosts and ended up upgrading to 128GB

21 Scaling Click to to edit infinity Master and title beyond! style Capture loss levels (as reported by Bro) started rising beyond acceptable levels once we were past 3Gb/s of traffic on our existing hardware platform We knew that traffic levels were going to continue to increase so our design needed to evolve as well

22 Introducing Click to edit Bro Master Generation title style 1.5 We migrated to optical taps over SPAN sessions SPAN sessions were good for speed of deployment but not for long term use Introduced a method to allow us load balance traffic among physical hosts Similar outcome to the work done by LBNL Eliminated the SPOF with our Bro host

23 Click to Bro edit horizontal Master title scaling style While we do run Bro in a cluster, it is limited to a single physical host We don t want to share state across hosts The Bro manager process being a single point of failure isn t all that appealing to us Keep the hosts simple and consistent

24 Click to And edit here Master is how title looks style Bro host #1 Bro host #2 Bro host #3 Load balancer Corporate network Router Optical tap The Internet Netflow export Firewall Netflow collector

25 Click to edit Master Scorecard title style Single point of failure? Data collected via Vendor solution Generation One Generation 1.5 SPAN SPAN Optical taps Control Scalability Logistics/ Install effort Cost per Gb/s $$$ $ $

26 Click to Optical edit Master taps title overview style TX RX Optical tap RX TX Router TX 10Gb/s TX 10Gb/s Firewall Load balancer

27 Click to edit Still Master some work title to style do This was a great step forward, but it was only an incremental improvement We can now scale out but it is still time consuming to get individual hosts deployed Migrating to an integrated solution would help solve these challenges

28 Click to edit Bro Master Generation title style 2.0 Combined our hosts, load balancers and optical taps into a cookie cutter rack design We now just order a small, medium or large rack depending expected traffic volumes

29 Bro Generation Click to edit 2.0 Master physical title layout style Network rack Bro rack Load balancer Load balancer Optical tap Router Firewall Bro host #1 Bro host #2 Bro host #n

30 Scaling Click Bro Generation to edit Master 2.0 title footprint style Bro rack #1 Load balancer Network rack Load balancer Bro host #1 Bro host #2 Bro host #n Router Optical tap Firewall Bro rack #2 Load balancer Bro host #1 Bro host #2 Bro host #n

31 Click to edit Master Scorecard title style Vendor solution Generation One Generation 1.5 Single point of failure? Data collected via Control Scalability SPAN SPAN Optical taps Logistics/ Install effort Generation 2 Optical taps Cost per Gb/s $$$ $ $ $

32 What Click do we to edit do with Master all this title data? style We stream the logs to our central log store

33 Click to edit Central Master log title storage style

34 Learn Click from to some edit of Master our mistakes title style Our original ETL jobs were based on the Bro 2.3 field order (output in TSV) Bro 2.4 changed the ordering of some of the fields Use JSON if you re loading this data elsewhere One line configuration change!

35 Click to edit Master Wrapping title style up

36 Click to edit Master Lessons title learnt style Scale horizontally and not vertically Stateless sensors Decouple dependencies Plan up-front Lab testing is never overrated Get experts on-site to validate Document wins Know your customers

37 Click to edit Master Thanks title style to Industry peers Thanks to LBNL, Mozilla and the others who responded to our queries and everyone who has publicly spoke or documented their install

38 Click to edit Master Thank title style you!