Testing PDI solutions

Transcription

1 Testing PDI solutions Slawomir Chodnicki BI Consultant

2 The sample project. -- bin # entry point scripts -- environments # environment configuration -- etl # ETL solution `-- spec # tests and helpers 2 GmbH Helmholtzstr Berlin twineworks.com

3 Test orchestration $ bin/robot test 3 GmbH Helmholtzstr Berlin twineworks.com

4 Test orchestration 4 GmbH Helmholtzstr Berlin

5 Test orchestration 5 GmbH Helmholtzstr Berlin

6 Test orchestration 6 GmbH Helmholtzstr Berlin

7 Jenkins is a continuous integration server. It s basic role is to run the test suite and build any artifacts upon changes in version control. Example server:

8

9

10

11 Jenkins is a continuous integration server. It s basic role is to run the test suite and build any artifacts upon changes in version control. Example server:

12 Testable solutions

13 Configuration management configure all data sources/targets and paths through kettle variables or parameters local environment - not in version control test environment - reference environment production environment - optional 13 GmbH Helmholtzstr Berlin twineworks.com

14 Configuration management environments `-- local # dev environment not in version control -- environment.sh # shell environment variables -- my.cnf # database config file `--.kettle # KETTLE_HOME -- shared.xml # database connections `-- kettle.properties # kettle variables `-- test # test environment in version control `-- production # other environments 14 GmbH Helmholtzstr Berlin twineworks.com

15 Environments are self-contained share nothing reproducible results: you can run it your team can run it ci-server can run it 15 GmbH Helmholtzstr Berlin twineworks.com

16 Configuration management $ bin/robot spoon 16 GmbH Helmholtzstr Berlin twineworks.com

17 Configuration management 17 GmbH Helmholtzstr Berlin twineworks.com

18 Initialize db $ bin/robot db reset clearing database [ OK ] initializing database 2017/10/20 15:32:37 - Kitchen - Start of run. 2017/10/20 15:32:37 - reset_dwh - Start of job execution 2017/10/20 15:32:41 - Kitchen - Finished! 2017/10/20 15:32:41 - Kitchen - Processing ended after 4 seconds. [ OK ] 18 GmbH Helmholtzstr Berlin twineworks.com

19 Sub-systems/Phases Define sub-systems/phases define pre-requisites data expected in certain sources define outcomes data written to certain sinks A sub-system/phase of the ETL process is responsible for a small set of related sideeffects to happen 19 GmbH Helmholtzstr Berlin twineworks.com

20 Sub-systems/Phases 20 GmbH Helmholtzstr Berlin twineworks.com

21 Sub-systems/Phases 21 GmbH Helmholtzstr Berlin twineworks.com

22 Entry points Define entry points with a full functional contract. An entry point implements an application feature. 22 GmbH Helmholtzstr Berlin twineworks.com

23 Entry points 23

24 Testing ETL solutions

25 Kinds of automated tests Computation tests Integration tests Functional tests Non-functional tests 25 GmbH Helmholtzstr Berlin twineworks.com

26 Computation tests Single unit of ETL under test Performs a computation (no side-effects) What is a unit in PDI? Job? Transformation? Sub-transformation (mapping)? 26 GmbH Helmholtzstr Berlin twineworks.com

27 A simple computation test Test job spec/dwh/validate_params/validate_params_spec.kjb 27 GmbH Helmholtzstr Berlin twineworks.com

28 A simple computation test Test job spec/dwh/validate_params/validate_params_spec.kjb The job calls etl/dwh/validate_params.kjb - with DATA_DATE= and expects it to succeed - with DATA_DATE= and expects it to fail The job succeeds if all expectations are met. It fails otherwise. 28 GmbH Helmholtzstr Berlin twineworks.com

29 Test jobs 29 GmbH Helmholtzstr Berlin twineworks.com

30 Test transformation results 30 GmbH Helmholtzstr Berlin twineworks.com

31 Test transformation results 31 GmbH Helmholtzstr Berlin twineworks.com

32 Test sub-transformations 32 GmbH Helmholtzstr Berlin twineworks.com

33 Test sub-transformations 33 GmbH Helmholtzstr Berlin twineworks.com

34 Integration tests

35 Integration tests ETL responsible for a set of related sideeffects under test Most common case in ETL testing Test individual phases of a batch process 35 GmbH Helmholtzstr Berlin twineworks.com

36 Integration tests 36 GmbH Helmholtzstr Berlin twineworks.com

37 Integration tests 37 GmbH Helmholtzstr Berlin twineworks.com

38 Integration tests 38 GmbH Helmholtzstr Berlin twineworks.com

39 Functional testing

40 Functional tests Entry point of ETL solution under test Assertions reflect invocation contract Behavior on happy path Behavior on errors Behavior on incorrect invocation 40 GmbH Helmholtzstr Berlin twineworks.com

41 Test the daily run 41 GmbH Helmholtzstr Berlin twineworks.com

42 Test the daily run 42 GmbH Helmholtzstr Berlin twineworks.com

43 Test the daily run 43 GmbH Helmholtzstr Berlin twineworks.com

44 Non-functional tests

45 Non-functional tests Performance how long does workload x take? Stability what does it take to break it? How much memory is too little? What happens when loading unexpected data? (truncated file, column too long, 50MB XML in string field, badly formatted CSV reads as single field, empty files) 45 GmbH Helmholtzstr Berlin twineworks.com

46 Non-functional tests Security Verify configuration assumptions automatically Compliance We must use version x of library y 46 GmbH Helmholtzstr Berlin twineworks.com

47 Test compliance 47 GmbH Helmholtzstr Berlin twineworks.com

48 Scripting tests

49 JRuby JRuby is the ruby language on the JVM Maintained by Redhat. Runs Rails on JBoss 49 GmbH Helmholtzstr Berlin twineworks.com

50 Rspec Rspec is a testing framework for ruby GmbH Helmholtzstr Berlin twineworks.com

51 Rspec $ bin/robot test Includes helper files in spec/support traverses the spec folder looking for files whose names end in _spec.rb and loads them as tests 51 GmbH Helmholtzstr Berlin twineworks.com

52 describe "dwh clear job" do end end 52 GmbH Helmholtzstr Berlin twineworks.com

53 describe "dwh clear job" do describe "when db is not empty" do end end 53 GmbH Helmholtzstr Berlin twineworks.com

54 describe "dwh clear job" do describe "when db is not empty" do before :all do dwh_db.load_fixture = run_job "etl/dwh/load/load.kjb", {} end end end 54 GmbH Helmholtzstr Berlin twineworks.com

55 Rspec helpers spec/support/spec_helpers.rb def dwh_db... end Returns a JDBC database object. Connects on demand, and closes automatically when testsuite ends. 55 GmbH Helmholtzstr Berlin twineworks.com

56 Rspec helpers spec/support/spec_helpers.rb def dwh_db... end In addition dwh_db.load_fixture(path) allows loading a sql or json fixture file dwh_db.reset() triggers $ bin/robot db reset 56 GmbH Helmholtzstr Berlin twineworks.com

57 describe "dwh clear job" do } describe "when db is not empty" do before :all do dwh_db.load_fixture "spec/fixtures/steelwheels/steelwheels.sql" end end end 57 GmbH Helmholtzstr Berlin twineworks.com

58 describe "dwh clear job" do describe "when db is not empty" do before :all do dwh_db.load_fixture = run_job "etl/dwh/util/clear.kjb" end end end 58 GmbH Helmholtzstr Berlin twineworks.com

59 Rspec helpers spec/support/spec_helpers.rb def run_job file, params... end Runs a kettle job and returns a map { } :successful? => true/false, :log => log text, :result => [row1, row2, row3, ] 59 GmbH Helmholtzstr Berlin twineworks.com

60 describe "dwh clear job" do describe "when db is not empty" do before :all do dwh_db.load_fixture = run_job "etl/dwh/util/clear.kjb" end end end 60 GmbH Helmholtzstr Berlin twineworks.com

61 describe "dwh clear job" do describe "when db is not empty" do before :all do dwh_db.load_fixture = run_job "etl/dwh/util/clear.kjb" end it "completes successfully" do expect(@result[:successful?]).to be true end end end 61 GmbH Helmholtzstr Berlin twineworks.com

62 describe "dwh clear job" do describe "when db is not empty" do before :all do dwh_db.load_fixture = run_job "etl/dwh/util/clear.kjb" end it "completes successfully" do expect(@result[:successful?]).to be true end it "clears the db" do expect(dwh_db.query("show TABLES").to_a.length).to eq 0 end end end 62 GmbH Helmholtzstr Berlin twineworks.com

63 Test orchestration $ bin/robot test 63 GmbH Helmholtzstr Berlin twineworks.com

64 Test orchestration 64 GmbH Helmholtzstr Berlin

65 Test orchestration 65 GmbH Helmholtzstr Berlin

66 Test orchestration 66 GmbH Helmholtzstr Berlin

67 Running jobs as rspec tests spec/etl/etl_spec.rb Recursively traverses etl/spec looking for files whose names end in _spec.kjb, and dynamically generates a describe and it block for it. Hence all such job files are part of the test suite. 67 GmbH Helmholtzstr Berlin twineworks.com

68 describe "ETL" do Dir.glob("./**/*_spec.kjb").each do path describe "#{path} do it "completes successfully" = run_job path.to_s, {} expect(@result[:successful?]).to be true end end end end 68 GmbH Helmholtzstr Berlin twineworks.com

69 Rspec test orchestration Rspec runs in two phases Phase 1: collects tests, recording the structure as given by the describe blocks. Phase 2: filters found tests as per command line parameters and executes them 69 GmbH Helmholtzstr Berlin twineworks.com

70 Rspec test orchestration Run only tests containing the word clear in their name or enclosing describe blocks: $ bin/robot test --example 'clear' Run only tests tagged long_running : $ bin/robot test --tag 'long_running' Run only tests in spec/commands $ bin/robot test spec/commands 70 GmbH Helmholtzstr Berlin twineworks.com

71 Test orchestration $ bin/robot test 71 GmbH Helmholtzstr Berlin twineworks.com

72 Test orchestration 72 GmbH Helmholtzstr Berlin

73 Test orchestration 73 GmbH Helmholtzstr Berlin

74 Test orchestration 74 GmbH Helmholtzstr Berlin

75 Jenkins is a continuous integration server. It s basic role is to run the test suite and build any artifacts upon changes in version control. Example server:

76 Thank you!

77 Backup Slides

78 Testing in practice

79 Test what you run Verify behavior of the entity you run directly 79 GmbH Helmholtzstr Berlin twineworks.com

80 Tools of the trade Helpers utility code/etl of components reused to make tests about the what, not about the how fixture loaders assertion helpers data comparison helpers 80 GmbH Helmholtzstr Berlin twineworks.com

81 Data Fixtures Data Fixtures sets of test data, encoded in a convenient way, easily loaded into data sources and sinks JSON, CSV, SQL, XML, YAML Use whatever is easiest to maintain for the team Generate data fixtures through parameterized scripts if you need to generate datasets with consistent relationships 81 GmbH Helmholtzstr Berlin twineworks.com

82 File Fixtures File Fixtures sets of test files acted upon during a run Maintain file fixtures separate from source location expected by ETL If fixture files are changed as part of the test, copy them to a temporary location before running tests Create a unique source location per test run, if the file location is shared (like sftp) 82 GmbH Helmholtzstr Berlin twineworks.com