dcache as open-source project showcase for education Tigran Mkrtchyan for dcache team CHEP2018, Sofia,

Size: px

Start display at page:

Download "dcache as open-source project showcase for education Tigran Mkrtchyan for dcache team CHEP2018, Sofia,"

Ami Warren
5 years ago
Views:

1 dcache as open-source project showcase for education Tigran Mkrtchyan for dcache team CHEP2018, Sofia,

2 76 installations for WLCG ~10 no GRID installations ~50% of LHC data dcache as open-source project showcase for education Tigran Mkrtchyan 2

3 Some numbers ~60 developers/contributors in 18 years DESY FNAL NDGF 25 modules 61 production branches (since ) 926 releases (since ) ~400K lines of code in ~2600 files ~100 external dependencies RPM, DEB, TAR, Solaris and Docker packages dcache as open-source project showcase for education Tigran Mkrtchyan 3

4 Version management Branch 1.0 Branch 1.2 Branch 1.1 v1.2.0 v1.1.0 Master/trunk v1.2.1 v1.1.1 v1.2.2 v1.0.0 v1.0.1 v1.0.2 v1.0.3 v1.0.4 Golden/LTS or Enterprise release dcache as open-source project showcase for education Tigran Mkrtchyan 4

5 Release Policy Yearly long-term branch Time-based feature released 2+ years support ~every 3 mounts supported until next LTS release Weekly maintenance releases release for all supported branches dcache as open-source project showcase for education Tigran Mkrtchyan 5

6 The Release Process gives a common framework to get stuff done with a predictable and reproducible result. dcache as open-source project showcase for education Tigran Mkrtchyan 6

7 Code policy master/dev ALWAYS compilable ALMOST deployable Branch ALWAYS releasable dcache as open-source project showcase for education Tigran Mkrtchyan 7

8 Course goal Data intensive science, Big Data Introduction to Open Source project Use of standard tools Developer team interaction Students should be ready to join any project github Maven Code review Release management dcache as open-source project showcase for education Tigran Mkrtchyan 8

9 Project adjustments Standard project structure Standard build environment Convention vs. Configuration Make project IDE ready Standard workflow Simple instruction how to build BUILDING.md dcache as open-source project showcase for education Tigran Mkrtchyan 9

10 Development effort guideline Impact Effort Easy to do Difficult to do Major 1 3 Minor Implement immediately 2. Implement immediately 3. Long term plans and development 4. Discarded dcache as open-source project showcase for education Tigran Mkrtchyan 10

11 Development effort guideline Impact Effort Easy to do Difficult to do Major 1 3 Minor Implement immediately 2. Implement immediately 3. Long term plans and development 4. Discarded Students work target dcache as open-source project showcase for education Tigran Mkrtchyan 11

12 Student assignments Simple development task (easy with minor impact) Formed as GitHub issues No project prior knowledge is required Collected over the time Mostly in a single file easypick / education Goal: merge of a pull request dcache as open-source project showcase for education Tigran Mkrtchyan 12

13 Commit/PR workflow Full Build/Deploy/Test Code analysis Unit tests Functional tests Fail Fast dcache as open-source project showcase for education Tigran Mkrtchyan 13

14 Good COMMIT Does only one thing As small as possible easy to review easy to move around easy to revert your diff does not fit on my screen Informative log message `git blame` is your friend dcache as open-source project showcase for education Tigran Mkrtchyan 14

15 Contributor agreement External contributors may sneak proprietary code Have clear contribution policy dcache uses linux kernel model `Signed-off-by:` annotation Check with your legal department! dcache as open-source project showcase for education Tigran Mkrtchyan 15

16 Example: dcache: added a decorator for RepositoryChannels to get IO statistics (#3430) Motivation: Getting a possibility to generate statistics and spot misbehavior in IO processes of RepositoryChannel. Modification: Added package statistics, created the class IoStatisticsChannel as decorator for a RepositoryChannel and created the class IoStatistics which provides statistics on number of read and write requests, read and write IO block sizes and different measures about the read and writ e IO speed. Result: Possibility to add a decorator to a RepositoryChannel which provides statistics about IO processes. Signed-off-by: Lotta xxx <xxx@htw-berlin.de> dcache as open-source project showcase for education Tigran Mkrtchyan 16

17 Make my code better! dcache as open-source project showcase for education Tigran Mkrtchyan 17

18 Goals of code review Let others see my code Learn best-practice Forces me to publish better code New libraries used New technique used Learn about code changes (by others) reduce 'bus factor' dcache as open-source project showcase for education Tigran Mkrtchyan 18

19 Conclusions Use standard tools to attract students make them benefit from new skills Make your infrastructure robust for external contributors ensure reliable testing and deployment Clear license and contribution policy Be open for new contributors simple tasks for newcomers Document build process dcache as open-source project showcase for education Tigran Mkrtchyan 19

20 Thank You! dcache as open-source project showcase for education Tigran Mkrtchyan 20

21 Build system setup Build Jobs Testing Jobs Deployment Jobs Build nodes Jobs Jenkins master Test nodes Deploy nodes dcache as open-source project showcase for education Tigran Mkrtchyan 21

22 dcache as open-source project showcase for education Tigran Mkrtchyan 22

23 Commit message module: summary line (up to 60 chars) (empty line here) Motivation: (explain why we need this change) Modification: (explain what was change) Result: (what will be different now) Some extra info: (bug tracker id, branch back-port) dcache as open-source project showcase for education Tigran Mkrtchyan 23

24 Commit message ssh: fix handling of idle connection timeout Motivation: To preserve idle ssh connection one have to adjust IDLE_TIMEOUT, however sshd has yet an other timeout - READ timeout. Modification: Explicitly set read timeout to prevent it to fire before idle timeout. Result: idle ssh session is not closed Ticket: #9175 Target: master, 3.1 dcache as open-source project showcase for education Tigran Mkrtchyan 24

Git. Ľubomír Prda. IT4Innovations.

Git. Ľubomír Prda. IT4Innovations. Git Ľubomír Prda IT4Innovations lubomir.prda@vsb.cz support@it4i.cz VCS Version Control System Versioning - creation and management of multiple releases of a product, all of which have the same general