Testing Cloud Services: SaaS, PaaS and IaaS. Kees Blokland Jeroen Mengerink

Transcription

1 Testing Cloud Services: SaaS, PaaS and IaaS Kees Blokland Jeroen Mengerink

2 Agenda Introduction Cloud computing Challenges Risks Solutions Test measures

3 Objectives Learn how to Cope with Cloud services Detect the (new) risks Mitigate these risks By using Your skills Heuristics The Book Discussions and exchange experiences

4 In the cloud?

5 What can be done in the cloud?

6 searching, recording, accounting, paying, writing, reviewing, tracking, calculating, developing, listening, analyzing, transmitting, learning, controlling, purchasing, testing, alarming, changing, updating, deleting, accessing, rejecting, correcting, studying, booking, receiving, tracing, protecting, deciding, managing, teaching, facilitating, identifying, copying, removing, demonstrating, checking, showing, selecting, subscribing, unsubscribing, sharing, mailing, communicating, reading, playing, working, meeting, gambling, shopping, storing, cross checking, retrieving, configuring, sketching, saving, accelerating, enhancing, creating, growing, checking in, checking out, finding out, reaching, denying, talking, designing, making, verifying, measuring

7 Develop and Test Surf Transfer Operate and Manage Store

8 Develop and Test redundancy limitations Surf Transfer costs for innovation 80% unused management overhead storage claim environmentally unfriendly Operate and Manage Store

9 Develop and Test SOA Surf Transfer internet technology virtualization standard software bandwidth Operate and Manage Store

10 searching, recording, accounting, paying, writing, reviewing, tracking, calculating, developing, listening, analyzing, Develop and Test transmitting, learning, controlling, Surf purchasing, testing, alarming, changing, Transfer updating, deleting, accessing, rejecting, correcting, studying, booking, receiving, tracing, protecting, deciding, managing, teaching, facilitating, identifying, copying, removing, demonstrating, checking, showing, selecting, subscribing, unsubscribing, sharing, mailing, communicating, reading, playing, working, meeting, gambling, shopping, storing, cross checking, retrieving, configuring, sketching, saving, accelerating, Operate and Manage enhancing, creating, growing, Store checking in, checking out, finding out, reaching, denying, talking, designing, making, verifying, measuring

11

12 Essential characteristics On-demand service Self service provisioning, pay-per-use No human interaction US: National Institute of Standards and Technology

13 Essential characteristics On-demand service Broad network access Standard mechanisms over networks Any client US: National Institute of Standards and Technology

14 Essential characteristics On-demand service Broad network access Resource pooling Multi-tenant Storage, processing, memory, virtual machines, Location independent US: National Institute of Standards and Technology

15 Essential characteristics On-demand service Broad network access Resource pooling Rapid elasticity Rapid scale in and out Any quantity at any time US: National Institute of Standards and Technology

16 Essential characteristics On-demand service Broad network access Resource pooling Rapid elasticity Measured service Controlled resource use Transparency, pay-per-use US: National Institute of Standards and Technology

17 Service models Nocloud Infrastructure as a Service Platform as a Service Software as a Service Internal Application Platform Virtualization Hardware Cloud

18 Deployment models Public Private Community Hybrid

19 Essential characteristics On-demand service Broad network access Resource pooling Rapid elasticity Measured service Deployment models private cloud community cloud public cloud hybrid cloud Service Models Software as a Service Platform as a Service Infrastructure as a Service US: National Institute of Standards and Technology

20 What is done in the cloud? <500 employees Public *aas >500 Private Hybride Community IaaS, PaaS, DaaS, SaaS Taas *aas Consumer Public SaaS Mail Storage Infrastructure CRM Finance Business processes Data Centre Data Management Business processes Surf and mail Apps Social media Dropbox Google services Spotify Picasa Games

21 Theoretical Basis What is in the book? 1. What is cloud computing? 2. The changing role of the test manager 3. What to test? 4. How to test? ISBN Page xx

22 What do you see as the two main risks of involving the cloud?

23 Standards Cyber crime Continuity Privacy Legislation Multi platform Impact organisation 143

24 R i s k s Performance Standards Security Cyber crime Continuity Functionality Privacy Legislation Maintainability Multi platform Impact organisation Suppliers & Outsourcing

25 R i s k s Performance Security Functionality Maintainability Suppliers & Outsourcing

26 R i s k s Performance Security Functionality Maintainability Suppliers & Outsourcing

27 R i s k s Performance Security Functionality Maintainability Suppliers & Outsourcing Page 37

28 R i s k s Performance Security YOUR Operational Profile Functionality Other customers Maintainability Suppliers & Outsourcing YOUR Operational Profile PLUS

29 R i s k s Performance Security Functionality Maintainability Suppliers & Outsourcing

30 R i s k s Performance Security Functionality Maintainability Suppliers & Outsourcing Page 40

31 R i s k s Performance Security The idea: it s safe Functionality Everything over the web Manageability Suppliers & Outsourcing Home ground for hackers

32 R i s k s Performance Security Functionality Maintainability Suppliers & Outsourcing Page 40

33 R i s k s Performance Security No free choice of device. Functionality Bring Your Own Device Maintainability Suppliers & Outsourcing Endless possibilities.

34 R i s k s Performance Security Functionality Maintainability Suppliers & Outsourcing

35 R i s k s Performance Security Functionality Maintainability Suppliers & Outsourcing Page 42

36 R i s k s Performance Security Functionality Maintainability Internet connection other systems Off line does not work Information is lost Suppliers & Outsourcing

37 R i s k s Performance Security Functionality Maintainability Suppliers & Outsourcing

38 R i s k s Performance Security Functionality Maintainability Suppliers & Outsourcing

39 R i s k s Performance Security Functionality Maintainability Suppliers & Outsourcing Page 44

40 R i s k s Performance Security Mismatch service <> business process Functionality is changed Functionality Insufficient usability Maintainability Suppliers & Outsourcing

41 R i s k s Performance Security Functionality Maintainability Suppliers & Outsourcing

42 R i s k s Performance Security Functionality Maintainability Suppliers & Outsourcing Page 46

43 R i s k s Performance Security Taken care of. Functionality Backup and recovery Maintainability Suppliers & Outsourcing Who will support me?

44 R i s k s Performance Security Functionality Maintainability Suppliers & Outsourcing Page 46

45 R i s k s Performance Security Planned and controlled Functionality Updates, patches, fixes, Manageability Suppliers & Outsourcing Do I have a choice?

46 R i s k s Performance Security Functionality Maintainability Legislation & regulations Suppliers & Outsourcing

47 R i s k s Performance Security Functionality Maintainability Legislation & regulations Suppliers & Outsourcing

48 R i s k s Performance Security In house. Functionality Where is my data? And is that OK? Maintainability Legislation & regulations Somewhere Suppliers & Outsourcing

49 R i s k s Performance Security Functionality Maintainability Legislation & regulations Suppliers & outsourcing

50 R i s k s Performance Security Functionality Maintainability Legislation & regulations Suppliers & outsourcing Page 49

51 R i s k s Performance Vendor lock in Security No agreements Supplier of the supplier of the supplier Functionality Maintainability Supplier is taken over Legislation & regulations Suppliers & outsourcing

52 R i s k s Performance Security Functionality Maintainability Suppliers & Outsourcing

53 Testing? Interview Check Monitor Proof of concept Review

54 Testing! Interview Check Monitor Proof of Testen Proef Interview concept Intake Proof of concept Review

55 Testing during Selection Performance Testing Security Testing Manageability Testing Test Measures Testing Proof of Testen Proef Interview concept Intake Functional Testing Migration Testing Testing due to Testing in Production

56 Selection Implementation Testing during Selection Performance Testing Security Testing Manageability Testing Testing Functional Testing Test Measures Production Migration Testing Testing due to Testing in Production

57 Testing during Selection Performance Testing Security Testing Manageability Testing Test Measures Testing Functional Testing Migration Testing Testing due to Testing in Production

58 R i s k s Performance Security Functionality Maintainability Testing during Selection Performance Testing Security Testing Manageability Testing Testing Functional Testing Migration Testing Test Measures Suppliers & Outsourcing Testing due to Testing in Production

59 R i s k s Performance Security Functionality Maintainability Testing during Selection Performance Testing Security Testing Manageability Testing Testing Functional Testing Migration Testing Test Measures Suppliers & Outsourcing Testing due to Testing in Production

60 R i s k s Performance Security Functionality Maintainability Testing during Selection Performance Testing Security Testing Manageability Testing Testing Functional Testing Migration Testing Test Measures Suppliers & Outsourcing Testing due to Testing in Production

61 R i s k s Performance Security Functionality Maintainability Architecture Testing during Selection Performance Testing Security Testing Manageability Testing From individual risks to Testing Functional Testing Migration Testing individual test measures Testing due to Test Measures Suppliers & Outsourcing Testing in Production

62 Page 47

63 R i s k s Performance Security Functionality Maintainability Testing during Selection Performance Testing Security Testing Manageability Testing Testing Functional Testing Migration Testing Test Measures Suppliers & Outsourcing Testing due to Testing in Production

64 Testing during Selection Performance Testing Security Testing Manageability Testing Test Measures Selection Criteria Completeness Controllable For service For supplier Spec s and terms References Testing Functional Testing Migration Testing Testing due to Testing in Production

65 How do you select the service?

66 Page 61-64

67 Testing during Selection Performance Testing Security Testing Manageability Testing Test Measures Proof of Concept Dynamic testing More suppliers Time boxing Representative Testing Functional Testing Migration Testing Testing due to Testing in Production

68 R i s k s Performance Security Functionality Maintainability Testing during Selection Performance Testing Security Testing Manageability Testing Testing Functional Testing Migration Testing Test Measures Suppliers & Outsourcing Testing due to Testing in Production

69 Known measures tuned and tweaked New measures developed Use: Your skills Heuristics The Book Your peers Testing during Selection Performance Testing Security Testing Manageability Testing Testing Functional Testing Migration Testing Testing due to Test Measures Testing in Production

70 Suppose your payroll system moves into the Cloud. What would you do?

71 YOUR Operational Profile Load Testing Testing during Selection Performance Testing Security Testing Manageability Testing Testing Functional Testing Test Measures Migration Testing YOUR Operational Profile PLUS ACTUAL MOMENT Testing due to Testing in Production

72 Operational profile Page 73

73 Yes, you can! Stress Testing Testing during Selection Performance Testing Security Testing Manageability Testing Testing Functional Testing Test Measures Migration Testing Definitely NOT! Testing due to Testing in Production

74 Load and stress. Elasticity Testing during Selection Performance Testing Security Testing Manageability Testing Testing Functional Testing Test Measures Migration Testing Load and elasticity. Testing due to Testing in Production

75 How would you test elasticity?

76 load test up load test down boundary values 99 load path test boundary values extend? yes no up tc 1: use=99, pay 100 tc 2: use=100, pay 100 tc 3: use=101, pay charged 100 charged down tc1: use=101, pay 200 tc2: use=100, pay 100 tc3: use=99, pay 100

77 load test up load test down load boundary values (Automatic) scaling up or down 99 does not perform as required At scaling moments functional problems emerge boundary values path test extend? yes no up tc 1: use=99, pay 100 tc 2: use=100, pay 100 tc 3: use=101, pay 200 Insight in use based costs is not sufficient 200 charged 100 charged down tc1: use=101, pay 200 tc2: use=100, pay 100 tc3: use=99, pay 100

78 Security at: Network Supplier User Encryption Authentication and authorisation IDaaS Testing during Selection Performance Testing Security Testing Manageability Testing Testing Functional Testing Test Measures Test logs and audit trails Security Audits Experts Security patch routines Migration Testing Testing due to Testing in Production

79 Completeness and correctness of specifications and manuals Supplier User Availability of test environments Interface specifications Supported platforms Business process specs User manuals Testing during Selection Performance Testing Security Testing Manageability Testing Testing Functional Testing Test Measures Migration Testing Testing due to Testing in Production

80 How do you setup your environment including the external service?

81 Manageablity of test environments Everything in the cloud

82 Manageablity of test environments Link all current environments to the service

83 Manageablity of test environments Link Production to the real service Link other environments to a MOCK SERVICE (or another instance of the service)

84 Completeness and correctness of specifications and manuals Supplier User Availability of test environments Management of: Defects Changes Testing during Selection Performance Testing Security Testing Manageability Testing Testing Functional Testing Migration Testing Testing due to Test Measures Testing in Production

85 Defect Management Supplier resolves it Test Change work process Test Incident Client resolves it Change configuration Custom solution Test Test Incident not resolved Service not selected Terminate use of service Migrate and test Workaround work instruction Test

86 Completeness and correctness of specifications and manuals Supplier User Availability of test environments Management of: Defects Changes Maintainability of the software Testing during Selection Performance Testing Security Testing Manageability Testing Testing Functional Testing Migration Testing Testing due to Test Measures Testing in Production

87 Role of system architecture Monitoring and Logging Guarantees and SLA s Test fail-over mechanism Testing during Selection Performance Testing Security Testing Manageability Testing Testing Functional Testing Test Measures Test online/offline Migration Testing Testing due to Testing in Production

88 B is disrupted no change disruption in B ended no change Fail-over testing A: active B: inactive A is disrupted B takes over service A: disrupted B: active A: disrupted B: disrupted A is disrupted no change disruption in A ended no change A: active B: disrupted B is disrupted A takes over service A: inactive B: active

89 Testing during Selection Performance Testing Security Testing Manageability Testing Test Measures Online Offline Testing Functional Testing Migration Testing Use case testing. Global testing. Testing due to Testing in Production

90 Testing during Selection Performance Testing Security Testing Manageability Testing Test Measures Testing Functional Testing Migration Testing Testing caused by Testing in Production

91 How does functional testing change with services?

92 Functional test objectives Does the service fit the business processes and vv? Is the service quality sufficient (number of bugs)? Is the service sufficiently user friendly? Is the service configuration done correctly? Does supplier customization function properly? Does customer customization function properly? Do interfaces work properly? Are platforms properly supported? Does everything work after changes (is there no regression)?

93 Multiplatform testing. Any device any platform Testing during Selection Performance Testing Security Testing Manageability Testing Testing Functional Testing Test Measures Migration Testing Multiplatform testing. Testing caused by Testing in Production

94 3997 distinct Android devices

95 Windows XP Internet Explorer 6 Windows Vista Windows 7 Windows 2003 Server Internet Explorer 7 Internet Explorer 8 Firefox 3.5 Firefox 3.6 Windows 8 Browsers Firefox 4 Windows CE Linux Unix Mac OS Lion Mac OS Snow Leopard ios Operating systems Multi-platform testing Computer Safari 4 Safari 5 Chrome11 Opera11 PC SUN Macintosh Android iphone... Windows Mobile Samsung Devices Mobile phones NOKIA Tablet ASUS... Blackberry MOTOROLA

96 Windows XP Internet Explorer 6 Windows Vista Windows 7 Windows 2003 Server Internet Explorer 7 Internet Explorer 8 Firefox 3.5 Firefox 3.6 Windows 8 Browsers Firefox 4 Windows CE Linux Unix Mac OS Lion Mac OS Snow Leopard ios Operating systems Multi-platform testing Computer Safari 4 Safari 5 Chrome11 Opera11 PC SUN Macintosh Android iphone... Windows Mobile Samsung Devices Mobile phones NOKIA Tablet ASUS... Blackberry MOTOROLA

97 Multiplatform testing. Any device any platform Testing during Selection Performance Testing Security Testing Manageability Testing Testing Functional Testing Test Measures Migration Testing Multiplatform testing. Testing caused by Testing in Production

98 Scenarios Transfer into the cloud, applications remain the same data moved to another location Transfer to SaaS data migrated to new service Transfer from one to another SaaS similar Testing during Selection Performance Testing Security Testing Manageability Testing Testing Functional Testing Migration Testing Test Measures Transfer out of the cloud. similar Testing due to Testing in Production

99 Data conversion Testing conversion rules Testing conversion on input data Testing if any data is lost Testing ongoing transactions Rounding (totals incorrect) Field lengths (truncation) Totals (information lost) Date and time conversions what is ? Audit trail, check sums E2E business scenario s Extraction Conversion Import Existing systems Conversion software Service

100 Incidental testing. Sarbanes Oxley Legislation + Regulations = Test basis Testing during Selection Performance Testing Security Testing Manageability Testing Testing Functional Testing Test Measures Migration Testing Compliancy testing. Testing due to Testing in Production

101 Checking for legislation and regulations List where data that is stored in the cloud Find the requirements that are applicable to this data Legal support needed for high risk Check supplier terms with customer s requirements Example. A supplier of a storage service claims to be the owner of the intellectual capital of all data stored at their facilities. It is highly unlikely that this is compatible with the interests of the organization that is the actual owner of the data. Perform (external) audit for high risk Test manager provides advice, management decides

102 Example: Dropbox Compliance with Laws and Law Enforcement Requests; Protection of Dropbox's Rights. We may disclose to parties outside Dropbox files stored in your Dropbox and information about you that we collect when we have a good faith belief that disclosure is reasonably necessary to (a) comply with a law, regulation or compulsory legal request; (b) protect the safety of any person from death or serious bodily injury; (c) prevent fraud or abuse of Dropbox or its users; or (d) to protect Dropbox s property rights. If we provide your Dropbox files to a law enforcement agency as set forth above, we will remove Dropbox s encryption from the files before providing them to law enforcement. However, Dropbox will not be able to decrypt any files that you encrypted prior to storing them on Dropbox.

103 Is testing in production necessary?

104 R i s k s Performance Security Functionality Maintainability Testing during Selection Performance Testing Security Testing Manageability Testing Testing Functional Testing Migration Testing Test Measures Suppliers & Outsourcing Testing due to Testing in Production

105 Continuous Change Testing during Selection Performance Testing Security Testing Manageability Testing Testing Functional Testing Test Measures Migration Testing Continuous End-to-End Testing Testing due to Testing in Production

106 Standards Cyber crime Check Continuity Interview Privacy Legislation Monitor Proof of concept Multi platform Impact organisation Intake

107 R i s k s Performance Security Testing during Selection Performance Testing Security Testing Manageability Testing Test Measures Testing Functionality starts early: in selection Testing Scope of testing is widened Maintainability Functional Testing Testing continues in production Migration Testing Suppliers & Outsourcing Testing due to Testing in Production

108 R i s k s Performance Security Testing during Selection Performance Testing Security Testing Manageability Testing Test Measures Functionality Manageability Questions? Testing Functional Testing Migration Testing martin.pol@polteq.com jeroen.mengerink@polteq.com Testing due to Suppliers & Outsourcing Testing in Production

109 R i s k s Performance Security Testing during Selection Performance Testing Security Testing Manageability Testing Test Measures Functionality Manageability Thank you! Testing Functional Testing Migration Testing kees.blokland@polteq.com jeroen.mengerink@polteq.com Testing due to Suppliers & Outsourcing Testing in Production