Transcription

1 Update on Testing Infrastructure and SVN statistics Sergey Lyskov Jeff GrayLab JHU

2

3 Tests Statistics Score Function Fingerprint SVN Statistics

4 Tests Statistics

5 Quick overview, the types of tests we currently run:

6 Quick overview, the types of tests we currently run: Build tests

7 Quick overview, the types of tests we currently run: Unit Test Build tests 524 Unit tests

8 Quick overview, the types of tests we currently run: Build tests 127 Unit tests 76 Integration tests

9 Quick overview, the types of tests we currently run: Build tests Unit tests Integration tests Profile Tests 9 11 Performance Tests 9 8 Score Function fingerprint Tests - 3 Performance tests Profile test [new this year] Score Function fingerprint

10 Quick overview, the types of tests we currently run: Scientific Tests Build tests 12 Unit tests Integration tests Performance tests Profile test [new this year] Score Function fingerprint Scientific tests

11 Few new scientific tests added. Why?

12 Few new scientific tests added. Why? Scientific tests are added after protocol publication. But by then the developer is about to leave!

13 Few new scientific tests added. Why? Scientific tests are added after protocol publication. But by then the developer is about to leave! Suggestion: Make ST addition a publishing requirement (as we do for doc s)

14 Few new scientific tests added. Why? Scientific tests are added after protocol publication. But by then the developer is about to leave! Suggestion: Make ST addition a publishing requirement (as we do for doc s) Use ST as a developer tool as opposed to a test for finished protocol

15 Few new scientific tests added. Why? Scientific tests are added after protocol publication. But by then the developer is about to leave! Suggestion: Make ST addition a publishing requirement (as we do for doc s) Use ST as a developer tool as opposed to a test for finished protocol In that case, Pass/Fail ST output might not be the best thing:

16 Few new scientific tests added. Why? Scientific tests are added after protocol publication. But by then the developer is about to leave! Suggestion: Make ST addition a publishing requirement (as we do for doc s) Use ST as a developer tool as opposed to a test for finished protocol In that case, Pass/Fail ST output might not be the best thing: Pass/Fail line Time Various SVN revisions

17 Few new scientific tests added. Why? Scientific tests are added after protocol publication. But by then the developer is about to leave! Suggestion: Make ST addition a publishing requirement (as we do for doc s) Use ST as a developer tool as opposed to a test for finished protocol In that case, Pass/Fail ST output might not be the best thing:

18 Few new scientific tests added. Why? Scientific tests are added after protocol publication. But by then the developer is about to leave! Suggestion: Make ST addition a publishing requirement (as we do for doc s) Use ST as a developer tool as opposed to a test for finished protocol In that case, Pass/Fail ST output might not be the best thing:

19 Few new scientific tests added. Why? Scientific tests are added after protocol publication. But by then the developer is about to leave! Suggestion: Make ST addition a publishing requirement (as we do for doc s) Use ST as a developer tool as opposed to a test for finished protocol In that case, Pass/Fail ST output might not be the best thing: Green revision will get blamed for failing the test

20 Few new scientific tests added. Why? Scientific tests are added after protocol publication. But by then the developer is about to leave! Suggestion: Make ST addition a publishing requirement (as we do for doc s) Use ST as a developer tool as opposed to a test for finished protocol In that case, Pass/Fail ST output might not be the best thing: Green revision will get blamed for failing the test But Purple one is the real culprit!

21 Few new scientific tests added. Why? Scientific tests are added after protocol publication. But by then the developer is about to leave! Suggestion: Make ST addition a publishing requirement (as we do for doc s) Use ST as a developer tool as opposed to a test for finished protocol In that case, Pass/Fail ST output might not be the best thing: Green revision will get blamed for failing the test But Purple one is the real culprit! So boolean result for scientific tests might not be our best choice...

22 Tests Statistics 5

23 Score Function Fingerprint 5

24 New test type: Score-Function-Fingerprint

25 New test type: Score-Function-Fingerprint Each test calls version_scorefunction (written by Andrew).

26 New test type: Score-Function-Fingerprint Each test calls version_scorefunction (written by Andrew). Testing server runs these tests for all Mini revisions and keeps track of differences.

27 New test type: Score-Function-Fingerprint Each test calls version_scorefunction (written by Andrew). Testing server runs these tests for all Mini revisions and keeps track of differences. Each score function is fully ID d with unique number - its version [equal to SVN revision which initially introduces it]

28 New test type: Score-Function-Fingerprint Each test calls version_scorefunction (written by Andrew). Testing server runs these tests for all Mini revisions and keeps track of differences. Each score function is fully ID d with unique number - its version [equal to SVN revision which initially introduces it] Comparison: right now literal, in near future with specified numeric precisions.

29 New test type: Score-Function-Fingerprint Each test calls version_scorefunction (written by Andrew). Testing server runs these tests for all Mini revisions and keeps track of differences. Each score function is fully ID d with unique number - its version [equal to SVN revision which initially introduces it] Comparison: right now literal, in near future with specified numeric precisions. Testing server will automatically send notification when score fn changes. User can inspect history of changes using web interface similar to SVN-trac

30 New test type: Score-Function-Fingerprint Each test calls version_scorefunction (written by Andrew). Testing server runs these tests for all Mini revisions and keeps track of differences. Each score function is fully ID d with unique number - its version [equal to SVN revision which initially introduces it] Comparison: right now literal, in near future with specified numeric precisions. Testing server will automatically send notification when score fn changes. User can inspect history of changes using web interface similar to SVN-trac Currently we have 4 of these tests implemented; additions welcome!

31 New test type: Score-Function-Fingerprint Each test calls version_scorefunction (written by Andrew). Testing server runs these tests for all Mini revisions and keeps track of differences. Each score function is fully ID d with unique number - its version [equal to SVN revision which initially introduces it] Comparison: right now literal, in near future with specified numeric precisions. Testing server will automatically send notification when score fn changes. User can inspect history of changes using web interface similar to SVN-trac Currently we have 4 of these tests implemented; additions welcome!

32 Score Function: history view

33 Score Function: history view revision Testing server automatically determines if a freshly committed ScoreFunction equals any previous version, and if so assigns the previous revision number revision 42738

34 Benefits of SF fingerprint test

35 Benefits of SF fingerprint test The burden of keeping track of SF moved from developers to TestingServer.

36 Benefits of SF fingerprint test The burden of keeping track of SF moved from developers to TestingServer. In publications, you can precisely specify SF version used. Releases could do that as well.

37 Benefits of SF fingerprint test The burden of keeping track of SF moved from developers to TestingServer. In publications, you can precisely specify SF version used. Releases could do that as well. History-view allows us to keep track of changes in SF: who/when/ what

38 Score Function Fingerprint

39

40 BuildBot server tracking more builds - a lot of them!

41 BuildBot server tracking more builds - a lot of them! BuildBot Master

42 BuildBot server tracking more builds - a lot of them! BuildDaemon, Linux GCC bit debug/release/release-static BuildBot Master

43 BuildBot server tracking more builds - a lot of them! BuildDaemon, Linux GCC bit debug/release/release-static BuildDaemon, Linux GCC bit release/release-mpi BuildBot Master

44 BuildBot server tracking more builds - a lot of them! BuildDaemon, Linux GCC bit debug/release/release-static BuildDaemon, Linux GCC bit release/release-mpi BuildDaemon, Mac GCC debug/release/release-static BuildBot Master

45 BuildBot server tracking more builds - a lot of them! BuildDaemon, Linux GCC bit debug/release/release-static BuildDaemon, Linux GCC bit release/release-mpi BuildDaemon, Mac GCC debug/release/release-static BuildBot Master BuildDaemon, Mac Clang debug/release

46 BuildBot server tracking more builds - a lot of them! BuildDaemon, Linux GCC bit debug/release/release-static BuildDaemon, Linux GCC bit release/release-mpi BuildDaemon, Mac GCC debug/release/release-static BuildBot Master BuildDaemon, Mac Clang debug/release BuildDaemon, Windows (cygwin) release (I know it is failing, but it is implemented!)

47 BuildBot server tracking more builds - a lot of them! BuildDaemon, Linux GCC bit debug/release/release-static SVN server BuildDaemon, Linux GCC bit release/release-mpi BuildDaemon, Mac GCC debug/release/release-static BuildBot Master BuildDaemon, Mac Clang debug/release BuildDaemon, Windows (cygwin) release (I know it is failing, but it is implemented!)

48 BuildBot server tracking more builds - a lot of them! BuildDaemon, Linux GCC bit debug/release/release-static SVN server BuildDaemon, Linux GCC bit release/release-mpi BuildDaemon, Mac GCC debug/release/release-static BuildBot Master BuildDaemon, Mac Clang debug/release BuildDaemon, Windows (cygwin) release (I know it is failing, but it is implemented!) WEB interface

49 11

50 Thanks: Matthew O'Meara and Samuel DeLuca for setting up and supporting BuildBot daemons on their sites!

51 Benefits of BuildBot

52 Benefits of BuildBot We can track a lot of builds - 11 now! And it s really easy to add others.

53 Benefits of BuildBot We can track a lot of builds - 11 now! And it s really easy to add others. All builds are incremental - so it is super fast even on average machines.

54 Benefits of BuildBot We can track a lot of builds - 11 now! And it s really easy to add others. All builds are incremental - so it is super fast even on average machines. Decentralized system, failure tolerant.

55 Benefits of BuildBot We can track a lot of builds - 11 now! And it s really easy to add others. All builds are incremental - so it is super fast even on average machines. Decentralized system, failure tolerant. We tests builds on exactly the same systems that run your code.

56 Benefits of BuildBot We can track a lot of builds - 11 now! And it s really easy to add others. All builds are incremental - so it is super fast even on average machines. Decentralized system, failure tolerant. We tests builds on exactly the same systems that run your code. Many machines (6 now) are involved, so it s fast!

57 Benefits of BuildBot We can track a lot of builds - 11 now! And it s really easy to add others. All builds are incremental - so it is super fast even on average machines. Decentralized system, failure tolerant. We tests builds on exactly the same systems that run your code. Many machines (6 now) are involved, so it s fast! Builds are non-sequential, we can test revision R and R+1 simultaneously!

58 Benefits of BuildBot We can track a lot of builds - 11 now! And it s really easy to add others. All builds are incremental - so it is super fast even on average machines. Decentralized system, failure tolerant. We tests builds on exactly the same systems that run your code. Many machines (6 now) are involved, so it s fast! Builds are non-sequential, we can test revision R and R+1 simultaneously!

59 Benefits of BuildBot We can track a lot of builds - 11 now! And it s really easy to add others. All builds are incremental - so it is super fast even on average machines. Decentralized system, failure tolerant. We tests builds on exactly the same systems that run your code. Many machines (6 now) are involved, so it s fast! Builds are non-sequential, we can test revision R and R+1 simultaneously! Some crazy features to be aware of:

60 Benefits of BuildBot We can track a lot of builds - 11 now! And it s really easy to add others. All builds are incremental - so it is super fast even on average machines. Decentralized system, failure tolerant. We tests builds on exactly the same systems that run your code. Many machines (6 now) are involved, so it s fast! Builds are non-sequential, we can test revision R and R+1 simultaneously! Some crazy features to be aware of: You can see current and past console outputs of all builds as they go on the web!

61 Benefits of BuildBot We can track a lot of builds - 11 now! And it s really easy to add others. All builds are incremental - so it is super fast even on average machines. Decentralized system, failure tolerant. We tests builds on exactly the same systems that run your code. Many machines (6 now) are involved, so it s fast! Builds are non-sequential, we can test revision R and R+1 simultaneously! Some crazy features to be aware of: You can see current and past console outputs of all builds as they go on the web! You can request (re)build for a particular revision using web interface!

62 Future plans for BuildBot

63 Future plans for BuildBot Get a dedicated hi-end machine that will run numerous virtual machines. This would allow to expand the range of tested architectures.

64 Future plans for BuildBot Get a dedicated hi-end machine that will run numerous virtual machines. This would allow to expand the range of tested architectures. Bleeding-edge builds on latest versions of GCC-4.6 and Clang- 3.0.

65 Future plans for BuildBot Get a dedicated hi-end machine that will run numerous virtual machines. This would allow to expand the range of tested architectures. Bleeding-edge builds on latest versions of GCC-4.6 and Clang Run unit tests on various platforms. Hopefully we can have native Windows build!

66 Future plans for BuildBot Get a dedicated hi-end machine that will run numerous virtual machines. This would allow to expand the range of tested architectures. Bleeding-edge builds on latest versions of GCC-4.6 and Clang Run unit tests on various platforms. Hopefully we can have native Windows build! Valgrind runs.

67 Future plans for BuildBot Get a dedicated hi-end machine that will run numerous virtual machines. This would allow to expand the range of tested architectures. Bleeding-edge builds on latest versions of GCC-4.6 and Clang Run unit tests on various platforms. Hopefully we can have native Windows build! Valgrind runs. PyRosetta builds.

68 S

69 SVN Statistics

70 SVN Statistics!

71 SVN Statistics! Right now mini source code is ~ 1,426,000 lines long

72 SVN Statistics! Right now mini source code is ~ 1,426,000 lines long [Last year was at ~1,163,000 lines, so it grew ~23%]

73 SVN Statistics! Right now mini source code is ~ 1,426,000 lines long [Last year was at ~1,163,000 lines, so it grew ~23%] Since last year we have 2,659 revisions committed to mini trunk, thats ~7.3 rev/day

74 SVN Statistics! Right now mini source code is ~ 1,426,000 lines long [Last year was at ~1,163,000 lines, so it grew ~23%] Since last year we have 2,659 revisions committed to mini trunk, thats ~7.3 rev/day [Last year: 2,187 ~ 6 rev/day]

75 Student/postdoc with highest rate of broken builds: Elizabeth H.Kellogg

76 Student/postdoc with highest rate of broken builds: Elizabeth H.Kellogg Elizabeth H.Kellogg Possu Huang John Karanicolas Nikolas Sgourakis Last year highest rate was: 18%

77 Student/postdoc with highest rate of broken Unit tests: Gordon Lemmon

78 Student/postdoc with highest rate of broken Unit tests: Gordon Lemmon Gordon Lemmon Kenneth Jung Frank Dimaio Steven Combs Last year highest rate was: 11%

79 Student/postdoc with highest rate of Integration tests change: Monica Berrondo

80 Student/postdoc Last year highest with highest rate was rate the of same: Integration 56% tests change: Monica Berrondo Monica Berrondo Ingemar Andre Oliver Lange Gordon Lemmon Colin A. Smith Barak Raveh

81 Student/postdoc who committed most revisions: Mike Tyka

82 Student/postdoc who committed most revisions: Mike Tyka Last year highest number was: 204 Mike Tyka Christopher Miles Sarel Fleishman James Thompson Matthew O'Meara William Sheffler Oliver Lange Andrew Leaver-Fay Frank Dimaio Steven Lewis

83 Next priorities

84 Next priorities 1. More score function tests.

85 Next priorities 1. More score function tests. 2. New dedicated machine for BuildBot builds (?)

86 Next priorities 1. More score function tests. 2. New dedicated machine for BuildBot builds (?) 3. More BuildBot builds on different platforms.

87 Next priorities 1. More score function tests. 2. New dedicated machine for BuildBot builds (?) 3. More BuildBot builds on different platforms. 4. New faster Test server for revision-by-revision daemon.

88 Special Thanks to: Steven Lewis Matthew O'Meara Vladimir Yarov-Yarovoy and to everyone else who helped develop and maintain these tests, feedback, and bug reports.

89 Thank you!

90 24

91 List of currently implemented builds: Scientific Linux-64 GCC 4.1.2: pilot apps debug/release/release-static CentOS Linux-64 GCC pilot apps release/release-mpi Mac GCC 4.2 pilot apps debug/release/release-static Mac Clang debug/release Windows (cygwin) (I know it is failing, but it is implemented!)

92

93 Unit Test

94 Unit Test Integration Tests

95

96 Profile Tests 9 11 Performance Tests 9 8 Score Function fingerprint Tests - 3

97 Profile Tests 9 11 Performance Tests 9 8 Score Function fingerprint Tests - 3 Scientific Tests

98

99

100 [2010] User with highest rate of broken builds: Oliver Lange 30

101 List of currently implemented scientific tests: 1. abinitio 2. dna_interface_design 3. docking 4. ligand_docking 5. loop 6. membrane 7. monomer_ddg 8. multi_residue_ligand_docking 9. relax 10.rna_denovo 11.rna_design 12.sequence_recovery 13.detailed_balance 14.enzdes_benchmark 15.Rotamer Recovery