Research at PNNL: Powered by AWS NLIT 2018

Size: px

Start display at page:

Download "Research at PNNL: Powered by AWS NLIT 2018"

Loreen Burke
5 years ago
Views:

its endorsement, recommendation, or favoring by the United States Government or any agency thereof, or Battelle Memorial Institute.

1 Research at PNNL: Powered by AWS NLIT 2018 RALPH PERKO AND MIKE GIARDINELLI Pacific Northwest National Laboratory Reference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States Government or any agency thereof, or Battelle Memorial Institute. The views and opinions of authors expressed herein do not necessarily state or reflect those of the United States Government or any agency thereof. June 13,

2 The national laboratory system

3 PNNL at a glance $920.4 M In R&D expenditures 104 U.S. and foreign patents granted 1,058 Peer-reviewed publications 2 FLC Awards 5 R&D 100 Awards 4,400 Scientists, engineers and non-technical staff

4 Software engineering at PNNL Staff focus is research and innovation, not operations Developers work with scientists to enable research Limited space and resources for hardware Big driver for moving to AWS! Agile is difficult

novel concepts Lack of collaboration with engineers Creates

5 Problem: isolated research Who are the researchers Researchers work independently Focus on innovation and novel concepts Lack of collaboration with engineers Creates long delivery times Product usually isn t what the customer has envisioned

Enabling research with AWS Research is the life blood of the organization Researchers should not be troubled with environment configurations, optimizations, etc.

6 Enabling research with AWS Research is the life blood of the organization Researchers should not be troubled with environment configurations, optimizations, etc. Software engineers provide expertise needed to build applied solutions Utilizing AWS has been a turning point. AWS has dramatically helped to improve collaboration. AWS fits better with our Agile software processes As a result, researchers can focus on the problem

7 Moving to the cloud Drivers Lack of resources internally (hardware and people) Customer deliverables and demands / deadlines Concerns Cost Vendor lock-in Initial Approach Fork-lift model Missed out on AWS services Still had operational headaches Current Approach Serverless wherever possible Our progression to AWS

9 Overview Goal Enable novel image classification research on live, streaming data First primarily serverless solution

10 Image retrieval and classification - requirements Requirements Handle static and streaming media Scalable, robust, and flexible Easily deployed and maintained Extensible (add additional models and instantiations) Identify optimal ways to collaborate Research and customer requirements

11 Image retrieval and classification Research and engineering implementation

12 Image retrieval and classification How research and engineering collaborated on the effort

13 Image classification Apache NiFi Why NiFi?

14 NiFi overview Process and distribute data Message / data routing is very flexible and robust ETL is painless Easy to install, scale, configure, and extend Visually see what is going on with your pipelines Backpressure and queueing are baked into the flows excellent for systems that have brittle endpoints Low barrier to entry; broadens user audience Where we find benefit and why we use it

15 Image cache data flow example

16 NiFi tuning on AWS C4 and M4 instance types work well Scaling: we go vertical, then horizontal Keep normal CPU load at 50 60% CPU utilization Set provenance to Volatile General purpose SSDs work well Follow the NiFi Configuration Best Practices in the admin guide

Data flow logic Read data from Kafka topic {"image":"http://somewebsite.com/puppies.jpeg"}, {"image":"http://somewebsite.com/kittens.jpeg"}, {"image":"http://somewebsite.com/koalas.

17 Data flow logic Read data from Kafka topic {"image":" {"image":" {"image":" Filter We only care about Koala bears! {"image":" Create new message and push to Amazon SNS { } "url":" "hash" : "092f6b17d186adb2e121afcdc7e5470b0c6f82a5", "name" : "koalas.jpeg", "type" : "jpeg", "bucket : image-classifier-test

18 Image classification Why SNS?

19 AWS SNS { "Records": [ { "EventVersion": "1.0", "EventSubscriptionArn": "arn:aws:sns:example", "EventSource": "aws:sns", "Sns": { "SignatureVersion": "1", "Timestamp": null, "Signature": "EXAMPLE", "SigningCertUrl": "EXAMPLE", "MessageId": "95df01b4-ee98-5cb c221d41eb5e", "Message": "{\"hash\": \"092f6b17d186adb2e121afcdc7e5470b0c6f82a5\", \"url\": \" \"bucketname\": \"imageclassifier-test\", \"name\" : \"koalas.jpeg\", \"type\" : \"jpeg\",}",

20 Image classification Why Lambda?

21 Why Lambda? Hands down best choice for scaling Performance, cost, maintenance Straightforward code 60+ image requests per second Required 4+ large EC2 instances clustered One month pilot Lambda 161,490,065 requests; 61,490,065 seconds $1,050 EC2: 4 EC2 m4.10xlarge instances $3,329 (reserved) Where we went serverless instead of NiFi

22 AWS Lambda Where we went serverless instead of NiFi

23 Lambda code example public void handlerequest(snsevent snsevent, Context context) throws Exception { //get the JSON payload String message = snsevent.getrecords().get(0).getsns().getmessage(); //parse JSON //after retrieving the URL download the image BufferedImage image = ImageIO.read(imageUrl); //convert ImageIO.write(image, jpeg, bytearrayoutputstream); //save to S3 s3client.putobject(new PutObjectRequest(bucketName, filename, inputstream,metadata)); //write metadata to Dynamo Table table = dynamodb.gettable(dynamodbtable); Item item = new Item().withString("url_hash", request.gethash()).withstring("url", request.geturl()).withstring("s3_bucket", s3bucket); table.putitem(item); } //create and send a notification SendMessageRequest sendnewimagecachedmsg = new SendMessageRequest().withQueueUrl(queueUrl).withMessageBody(newImageJson); amazonsqs.sendmessage(sendnewimagecachedmsg);

24 Collaboration How research and engineering collaborated on the effort

25 Lessons learned Fantastic for scaling Obvious choice Very performant when functions are loaded (warm) API is easy to use Just Java Used for two key situations Low cost development/pilot efforts High volume/throughput Where we find benefit and why we use it

26 Lessons learned (continued) Cold start performance 30 s (cold) as opposed to 400 ms (warm) Legacy code vs new development Limits on jar sizes Message size on Amazon SNS 256 KB limit Combine functionality in a single Lambda function! Easier and cheaper to manage Step functions for our use cases were too expensive Where we find benefit and why we use it

27 EC2 vs Lambda - EC2 based solution

28 EC2 vs Lambda - Lambda based solution

30 Amazon Athena Great for exploring data in Amazon S3 HQL / SQL support Partition support Use AWS Glue crawlers Complements Hadoop cluster Where else can we use serverless?

32 Supporting research Evaluation of capabilities requires infrastructure

34 In Summary More and more we lean on AWS serverless services We don t have the resources for operations and maintenance Government customers we support prefer serverless solutions Makes it easier to provide researchers and engineers with flexible blueprints for their implementations Focus on solving problems not setting up infrastructure What are your technical needs. Do we already have something similar? Leverage AWS environment to provide easy access to data, services, tools, and resources Pleased with performance We can brute force solutions if we have to Most performance tuning is trivial Find most cost-effective use cases for your needs We have been able to strike a balance between serverless and managed Periodically do spot checks on cost. Upfront calculations may have been incorrect

35 In Summary Cont d Go-to tech stack Apache NiFi, Amazon S3, AWS Lambda, Amazon SQS, Amazon SNS, Amazon DynamoDB, Amazon RDS, others as needed Take advantage of built-in events / triggers when you can Most of the time S3 + events are good enough Free capability We have abandoned Kafka in favor of Apache NiFi site to site or Amazon SQS Apache Kafka is great, just don t have the administrative resources to support. Use AWS alternative, when possible. Most streaming requests by our customers don t really require streaming Request that researchers and engineers catalog their data and try to follow basic data lake practices Keep raw and enriched / augmented separate Add metadata to known events and important time frames Enable start / stop and replay to improve evaluation

36 Questions

Containers or Serverless? Mike Gillespie Solutions Architect, AWS Solutions Architecture

Containers or Serverless? Mike Gillespie Solutions Architect, AWS Solutions Architecture A Typical Application with Microservices Client Webapp Webapp Webapp Greeting Greeting Greeting Name Name Name Microservice