HOW TO PLAN & EXECUTE A SUCCESSFUL CLOUD MIGRATION

Size: px

Start display at page:

Download "HOW TO PLAN & EXECUTE A SUCCESSFUL CLOUD MIGRATION"

Martina Carter
5 years ago
Views:

1 HOW TO PLAN & EXECUTE A SUCCESSFUL CLOUD MIGRATION Steve Bertoldi, Solutions Director, MarkLogic

2 Agenda Cloud computing and on premise issues Comparison of traditional vs cloud architecture Review of use cloud use cases SLIDE: 2

3 What is cloud computing? Self service provisioning Elasticity and scale Pay as you use services Efficient and economic Infrastructure as a Service (IaaS) SLIDE: 3

4 Cloud Providers Public clouds Amazon Web Services (AWS) Google Cloud Microsoft Azure Private Clouds GovCloud SLIDE: 4

Issues with on premise infrastructures Infrastructure related software bugs Frequent patching causes down time Patches require reboot forcing

5 Issues with on premise infrastructures Infrastructure related software bugs Frequent patching causes down time Patches require reboot forcing outages Systems unstable after patch Hardware stability issues Can be difficult to troubleshoot Virtualized infrastructures as fixed assets SLIDE: 5

6 Pain points realized by customers Customers want a stable platform Tired of infrastructure headaches Bare metal management costs increasing Procurement taking longer Administration issues Issues with some in-house directives SLIDE: 6

7 Don t need in-house systems No need to be close to data center Stop hugging servers! The best of today will eventually be outdated SLIDE: 7

8 The MarkLogic AWS solution MarkLogic is supported on AWS Linux No other external dependencies Higher network availability Easier to procure and administer Better disaster/backup recovery Different availability zones Simpler backups with Simple Storage Service (S3) SLIDE: 8

9 Gains of MarkLogic and cloud implementations Faster build and delivery times Scale-on-demand and elastic capabilities Easier tools and management Internal cost reduction Asset recovery SLIDE: 9

10 TRADITIONAL VS AWS SLIDE: 10

11 System builds the old way CPU specifications Cores, sockets, speed Memory Dependent on CPU Size Capacity Motherboard Chassis/capacity Network Fabric/topology Storage RAID? Send to procurement SLIDE: 11

12 Building the AWS way MarkLogic AWS cloud optimized instance Amazon EBS Volume CPU/Memory/Networking Elastic Compute Cloud () CPU/Memory Networking Disk/Storage Elastic Block Storage (EBS) GP2 PIOPS Magnetic Amazon Machine Image (AMI) Install MarkLogic SLIDE: 12

13 Migrating a typical MarkLogic Cluster 3 node MarkLogic cluster 2 CPUs, 8 cores each 32 GB RAM 3 TB of local storage SLIDE: 13

14 Select a region 3 us-west-2 Oregon us-west-1 N. California 3 us-east-1 N. Virginia 4 SLIDE: 14

15 Select the AMI Amazon Machine Image (AMI) Bootstrap image for the OS Provides details about the instance Copies image to disk SLIDE: 15

16 instance small instance large Choose an instance type Provides compute power Choose based upon the system needs c4.8xlarge 32 vcpus 60 GiB RAM Enhanced networking SLIDE: 16

17 availability zone Us-east-1c Define instance details Select availability zone (AZ) Do you need a Virtual Private Cloud (VPC) Networking options/subnet Many more options Us-east-1d Us-east-1e SLIDE: 17

18 Add storage Amazon EBS Need to have a root volume Size to your needs Data volume types GP2 PIOPS Magnetic 3TB GP2 SLIDE: 18

19 security group SSH:22 admin:8001 QC:8000 Us-east-1c SSH keys Setup Security Configure Security Group Firewall rules to control traffic Spans availability zones SSH keys for private/public key authentication Launch! SLIDE: 19

20 Building out the cluster security group Elastic Load Balancing First instance in the cluster Repeat for remaining nodes Install MarkLogic Elastic Load Balancer (ELB) Us-east-1c Us-east-1d SLIDE: 20

21 Building MarkLogic clusters in AWS Unmanaged/manual install Launching with a MarkLogic AMI directly AWS Marketplace Bring your own license (BYOL) Metered MarkLogic Cloud Formation templates Managed clusters SLIDE: 21

22 USE CASES SLIDE: 22

23 Use case: AWS for testing New system part of a larger infrastructure Target SLAs defined 4 node cluster Customer hardware in procurement cannot change Testing on internal virtualized systems SLIDE: 23

24 Operations and problems Transformation hub Large amount of record loading Not meeting SLAs Bad performance on local VMs SLIDE: 24

25 Testing infrastructure root data root data Setup Virtual Private Cloud Users connect through VPN 5 m4.4xlarge instances Build MarkLogic cluster Run loading tests Run times < SLA root data root data mlcp SLIDE: 25

26 Transform job Transformation job Runs on a single node Not meeting SLA Elastic Load Balancing Added elastic load balancer Routed traffic to all nodes Performance < SLA trans SLIDE: 26

27 Use case: Utilizing scaling features Search and lookup during the day Millions of documents from legacy system Transform batch job Cannot meet SLAs SLIDE: 27

28 Initial infrastructure Us-east-1c Us-east-1d 3 node architecture Medium sized instances Loading occurs all day SLA in place APP Elastic Load Balancing SLIDE: 28

29 Scaling infrastructure Us-east-1c Us-east-1d Primary made smaller Second cluster added 8 vcpu each Load to secondary APP Elastic Load Balancing Us-east-1c transform Us-east-1d Transform scale-up Loading disabled Scale up instances 32 vcpu each Transform on all 3 nodes Met SLAs SLIDE: 29

30 Scale down and switch APP Us-east-1c Us-east-1d Transform complete 2 clusters exist Scale 2 nd cluster to 8 vcpus Swap cluster at load balancer Clear primary Attach loader to primary Elastic Load Balancing Us-east-1c Us-east-1d SLIDE: 30

31 Use case: Elastic auto-scaling architecture Document based system with searching Provides alerting and highlighting 3 node primary system Need to meet peak demands SLIDE: 31

32 Master cluster Master1 Elastic Load Balancing 3 node cluster Separate availability zone Used for highlighting C3.2xlarge 8vCPU 15GB RAM Master1 Master1 SLIDE: 32

33 Foreign node Master1 Elastic Load Balancing replication Elastic Load Balancing Foreign 1 Single node Uses database replication Search capability Scheduled auto-scaling Base set for normal loads Increase for peak loads Reduce for weekend loads Master1 Master1 SLIDE: 33

34 Scale up Elastic Load Balancing Elastic Load Balancing Snapshot master replication Foreign 1 Create new foreign node Master1 EBS snapshot replication Attach snapshots Create volumes Setup replication Attach foreign cluster Attach replicas Master1 EBS snapshot Foreign 2 Master1 EBS snapshot AWS Cloud Formation Custom tools SLIDE: 34

35 Scale down Master1 Master1 Elastic Load Balancing replication replication Elastic Load Balancing Foreign 1 Foreign 2 Recover resources Disable replication Remove replicas Detach foreign node Cleanup instance Stop instance Remove instance Remove volumes Master1 AWS Cloud Formation Custom tools SLIDE: 35

36 WRAPPING UP SLIDE: 36

37 Takeaways It works! Produce results faster and cheaper Flexibility, scalability and elasticity Easy to test out several configurations No restrictions from fixed infrastructures New enhancements and growing feature sets Cost declining SLIDE: 37

38 Q&A SLIDE: 38

CIT 668: System Architecture. Amazon Web Services

CIT 668: System Architecture. Amazon Web Services CIT 668: System Architecture Amazon Web Services Topics 1. AWS Global Infrastructure 2. Foundation Services 1. Compute 2. Storage 3. Database 4. Network 3. AWS Economics Amazon Services Architecture Regions