Package Management and Build Tools Objektumorientált szoftvertervezés Object-oriented software design Dr. Balázs Simon BME, IIT
Outline Ant+Ivy (Apache) Maven (Apache) Gradle Bazel (Google) Buck (Facebook) Dr. Balázs Simon, BME, IIT 2
Problems of Java building JAR file: collection of.class files JAR files may have dependencies on other JAR files, but: there is no explicit dependency descriptor in the JAR there is no version information in the JAR JAR files and all their dependencies have to be collected manually very time consuming error prone: a missing dependency shows up as a ClassNotFoundException at runtime We need a tool for: describing version and dependency information of JAR files automatically collecting transitive dependencies Dr. Balázs Simon, BME, IIT 3
Ant+Ivy (Apache) Dr. Balázs Simon, BME, IIT 4
Ant Ant is a build tool Ant configuration is in XML: build.xml imperative description of the build process Similar to Unix Make: tasks + dependencies between the tasks Ant can be extended with custom task types Build tasks have to be defined manually example: clean compile jar test deploy Dr. Balázs Simon, BME, IIT 5
Ant example: build.xml <?xml version="1.0" encoding="utf-8"?> <project name="hello" default="compile"> <target name="clean" description="remove all artifact files"> <delete dir="classes"/> <delete file="hello.jar"/> </target> <target name="compile" description="compile the Java source code to class files"> <mkdir dir="classes"/> <javac srcdir="." destdir="classes"/> </target> <target name="jar" depends="compile" description="create a Jar file for the application"> <jar destfile="hello.jar"> <fileset dir="classes" includes="**/*.class"/> <manifest> <attribute name="main-class" value="helloprogram"/> </manifest> </jar> </target> <target name="test" depends="jar" description="test the code"> <junit printsummary="yes" haltonfailure="no"> <!--... --> </junit> </target> clean <target name="deploy" depends="jar" description="deploy to the production environment"> <copy file="hello.jar" todir="../production/dir"/> </target> </project> test compile jar deploy 6
Ivy: dependency resolution for Ant Ant cannot resolve JAR dependencies by itself you had to collect all the JARs manually before Ivy Ivy is an extension for Ant which can resolve JAR dependencies Ivy can also resolve transitive dependencies similarly to Maven but Maven is much more than just a dependency resolver ivy.xml ivy-module version="2.0"> <info organisation="com.companyname" module="my-project"/> <dependencies> <dependency org="junit" name="junit" rev="4.11"/> </dependencies> /ivy-module> build.xml <project xmlns:ivy="antlib:org.apache.ivy.ant" name="my-project" default="jar"> <target name="resolve"> <ivy:retrieve /> </target> <!--... --> </project> Dr. Balázs Simon, BME, IIT 7
Properties of Ant+Ivy Advantages: full control of the build process flexible relatively fast great IDE support for all modern IDEs Disadvantages: no common convention for build steps and project structure a new member of the team has a hard time getting familiar with the build process Ant s XML is write-only the build.xml quickly becomes a mess: very hard to maintain Dr. Balázs Simon, BME, IIT 8
Maven (Apache) Dr. Balázs Simon, BME, IIT 9
Maven Build tool and dependency management system Maven configuration is in XML: pom.xml declarative description of version information, dependencies and build steps Maven promotes convention over configuration: consistent project structure consistent build model Maven is extensible with plugins e.g. generating files, running special tests, creating special artifacts, etc. Maven build artifact: usually a JAR file three properties: group id: identifier of the organization (usually the DNS name in reverse order) artifact id: identifier of the artifact within the group version: semantic version number Maven stores build artifacts and their dependency information in repositories Dr. Balázs Simon, BME, IIT 10
Maven repository Repositories hold build artifacts (usually JAR files) and their dependency information Two types of repositories: local: local computer (~/.m2/ folder) it is a cache of remote downloads also contains own build artifacts not yet released remote: remote server provides build artifacts for downloading Remote repository: public: available for everyone internal: internal use for a company to store private build artifacts for sharing between teams and for releases Remote repositories must be configured for maven: this is where it will look for build artifacts Maven central repository: http://central.maven.org/maven2/ special public repository contains the most commonly used jars and all their dependencies all well known open-source jars are in there (Apache, RedHat,...) Dr. Balázs Simon, BME, IIT 11
Maven repositories Developer s computer ~/.m2/ Developer s computer ~/.m2/ Company s intranet Internal maven repository Internet Public maven repository Developer s computer ~/.m2/ Individual developer s computer ~/.m2/ Dr. Balázs Simon, BME, IIT 12
Maven project structure Maven promotes convention over configuration: fix project structure (can be overridden if needed, but it is rarely done) Standalone application: project root src main Java sources java resources Non-Java files test java Test files resources target pom.xml Created during build Web application: project root src main java resources webapp test java resources target pom.xml Dr. Balázs Simon, BME, IIT 13
Maven archetypes Maven can generate projects with default contents Maven archetypes are templates for these projects e.g. standalone application, web application, etc. Standalone application: mvn archetype:create -DgroupId=[your project's group id] -DartifactId=[your project's artifact id] Web application: mvn archetype:create -DgroupId=[your project's group id] -DartifactId=[your project's artifact id] -DarchetypeArtifactId=maven-archetype-webapp And there are a lot of other archetypes You can even create your own. Dr. Balázs Simon, BME, IIT 14
Maven build steps Maven has a predefined set of steps in the build process can be customized, if needed The build process is incremental: only the changed parts are rebuilt Do not call clean everytime, otherwise the build process will take very long! Build lifecycle: validate Clean lifecycle: clean compile test package Example commands: mvn clean mvn package mvn clean deploy verify install deploy Dr. Balázs Simon, BME, IIT 15
Maven example: pom.xml <?xml version="1.0" encoding="utf-8"?> <project xmlns="http://maven.apache.org/pom/4.0.0" xmlns:xsi="http://www.w3.org/2001/xmlschema-instance" xsi:schemalocation="http://maven.apache.org/pom/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelversion>4.0.0</modelversion> <groupid>com.companyname.project-group</groupid> <artifactid>project</artifactid> <version>1.0</version> Group identifier Our project s identifier </project> Version Artifact identifier project-1.0 Dr. Balázs Simon, BME, IIT 16
Maven version numbers Semantic version numbers: X.Y.Z-Q-W X: major (number) incremented on incompatible API changes Y: minor (number) incremented on backwards compatible API changes Z: patch (number, optional) incremented on backwards compatible bug fixes, no API changes Q: qualifier (string, optional) e.g. alpha, beta, snapshot, etc. W: build (number, optional) Examples: 1.2, 2.0, 2.1.1, 3.2.4-SNAPSHOT, 5.1.2-alpha-4 Dr. Balázs Simon, BME, IIT 17
Maven dependencies <?xml version="1.0" encoding="utf-8"?> <project xmlns="http://maven.apache.org/pom/4.0.0" xmlns:xsi="http://www.w3.org/2001/xmlschema-instance" xsi:schemalocation="http://maven.apache.org/pom/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelversion>4.0.0</modelversion> <groupid>com.companyname.project-group</groupid> <artifactid>project</artifactid> <version>1.0</version> junit-4.8.2 <dependencies> <dependency> <groupid>junit</groupid> <artifactid>junit</artifactid> <version>4.8.2</version> <scope>test</scope> </dependency> </dependencies> </project> Scope of the dependency project-1.0 Dependency identifier (transitive dependencies will be collected automatically) Dr. Balázs Simon, BME, IIT 18
Maven dependencies Dependencies define the required build artifacts (JARs) Dependency: group id + artifact id + version number reference version number reference: exact number or range examples: 1.3: generally means 1.3 or later, i.e. [1.3,) [1.0]: exactly 1.0 [1.2,1.3]: 1.2 <= x <= 1.3 [1.0,2.0): 1.0 <= x < 2.0 [1.5,): x >= 1.5 A dependency can have an optional scope (compile is the default): compile: required in all build phases, also propagated to dependent projects provided: required only in the compilation and test phase, at runtime it is provided by the environment (e.g. JDK, web server, etc.) runtime: not required for compilation, but it is for execution test: only required for the test compilation and test execution phases system: similar to provided, but we have to specify the path of the JAR explicitly for the local system (at runtime it will be provided by the environment) import: use dependencies from another pom.xml Dr. Balázs Simon, BME, IIT 19
Maven dependency resolution Maven builds a tree from the dependencies and from all their transitive dependencies But Maven cannot use multiple versions of the same JAR at the same time It has to select a specific version: based on the version number ranges based on compatibility (minor, patch) based on the distance from the root of the tree based on which comes first log4j-1.2.12 commonslogging 1.1 POM servlet-2.3 log4j-1.2.13 The dependency resolution algorithm is complex, but Maven will try its best Maven will report conflicts if it cannot perform the resolution correctly (e.g. version ranges don t overlap) solution: dependencymanagement section in the POM Dr. Balázs Simon, BME, IIT 20
Maven dependency management There is a dependencymanagement section in the POM We can specify the exact artifact version we want to use for the whole project uses exact versions overrides transitive dependencies can be used for dependency conflict resolution It can also be used in a hierarchical Maven project to centralize dependency information for child projects simpler to define them in one common POM than to repeat them in all child POMs Dr. Balázs Simon, BME, IIT 21
Maven dependency management example <?xml version="1.0" encoding="utf-8"?> <project xmlns="http://maven.apache.org/pom/4.0.0" xmlns:xsi="http://www.w3.org/2001/xmlschema-instance" xsi:schemalocation="http://maven.apache.org/pom/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelversion>4.0.0</modelversion> <groupid>com.companyname.project-group</groupid> <artifactid>project</artifactid> <version>1.0</version> <dependencymanagement> <dependencies> <dependency> <groupid>junit</groupid> <artifactid>junit</artifactid> <version>4.8.2</version> <scope>test</scope> </dependency> </dependencies> </dependencymanagement> <dependencies> <!--... --> </dependencies> </project> Dependency management: Exact dependencies Other dependencies junit-4.8.2 project-1.0 22
Maven project hierarchies Maven can also handle project hierarchies The root project references the children as modules The child projects reference the parent project Maven commands executed from the root will be also executed on the children e.g. mvn clean, mvn test, etc. Children will inherit the root s configurations and dependencies pom.xml <project> <modelversion>4.0.0</modelversion> <groupid>com.mycompany.app</groupid> <artifactid>root-project</artifactid> <version>1.0</version> <packaging>pom</packaging> Parent of other Maven projects <modules> <module>project1</module> <module>project2</module> </modules> </project> pom.xml root-project project1 pom.xml project2 pom.xml pom.xml <project> <parent> <groupid>com.mycompany.app</groupid> <artifactid>root-project</artifactid> <version>1.0</version> </parent> <modelversion>4.0.0</modelversion> <artifactid>project1</artifactid> </project> Group id and version number are inherited from the parent 23
Properties of Maven Advantages: consistency new team members know immediately everything clear convention for the project structure clear convention for the build steps well established, well supported easy to find solutions for problems on the net declarative dependencies are easy to write and maintain Maven resolves transitive dependencies automatically great IDE support for Eclipse, IntelliJ Disadvantages: too rigid: hard to deviate from the convention but it s usually not worth it: not following the convention is bad practice, new team members will have a hard time learning things can be slow usually when checking and downloading dependencies: as if it is downloading the whole internet can be resolved: keep a local copy of the dependencies once they are downloaded, and use offline mode use parallel builds Dr. Balázs Simon, BME, IIT 24
Gradle Dr. Balázs Simon, BME, IIT 25
Gradle Build and dependency management tool Combines Ant s power and flexibility with Maven s life-cycle and ease of use Ant does not have a fixed set of build steps and fixed project structures Maven is too strict, customizing builds is often very hard: if there is no plugin for the customization you have to create one Gradle uses a specific DSL based on Groovy shorter and clearer than XML the DSL readable, since it is designed to solve a specific problem: move software through its life-cycle compilation and static analysis packaging and deployment using Groovy code you can customize builds however you want but it can mean that you will deviate from a convention or standard Gradle uses its own dependency resolution engine Dr. Balázs Simon, BME, IIT 26
Gradle project structure Gradle follows Maven s convention in the project structure but it can be customized if necessary To compile Java we have to add the Java plugin Minimal Gradle script to compile a standalone application based solely on the JDK (no 3 rd party JARs): build.gradle apply plugin: 'java' Build command: gradle build Standalone application: project root Dr. Balázs Simon, BME, IIT 27 src main java resources test java resources target build.gradle Java sources Non-Java files Test files Created during build
Gradle dependency management Gradle follows Maven s groupid:artifactid:version (GAV) convention to identify artifacts compact form: 'log4j:log4j:1.2.17' extended form: group: 'log4j', name: 'log4j', version: '1.2.17' Example: dependencies { compile filetree(dir: 'libs', include: ['*.jar']) compile group: 'log4j', name: 'log4j', version: '1.2.17' testcompile 'junit:junit:4.12' } Gradle can use various repositories for resolution: Maven, Ivy, Example: repositories { mavencentral() maven ("http://repo.mycompany.com/maven") ivy { url "http://repo.mycompany.com/ivy" } } Dr. Balázs Simon, BME, IIT 28
Gradle example with 3 rd party dependencies build.gradle apply plugin: 'java' repositories { mavencentral() } dependencies { compile 'log4j:log4j:1.2.17' testcompile 'junit:junit:4.12' } Much shorter and clearer than Ant or Maven Dr. Balázs Simon, BME, IIT 29
Properties of Gradle Advantages: follows Maven s convention easy to use flexible Groovy DSL is nicer than XML IDE support for Eclipse, IntelliJ, Netbeans Disadvantages: very slow builds can be mitigated: daemon mode but it is still slow memory leaks customized builds deviating from the convention are harder to learn and maintain Dr. Balázs Simon, BME, IIT 30
Bazel (Google) Dr. Balázs Simon, BME, IIT 31
Bazel Multi-platform build tool can build software for many target platforms (Linux, Mac OS X, Windows, Android) Multi-language support can build software written in many languages (Java, C++, Objective-C), and can be extended for others Custom, simple high-level build language each library, test and binary must specify its direct dependencies completely Reproducible builds all builds are incremental and will always produce the same result Scalable can handle large builds, even millions of lines of code Dr. Balázs Simon, BME, IIT 32
Bazel example: Java Minimal Bazel script to compile a standalone application based solely on the JDK (no 3 rd party JARs): BUILD java_binary( name = "my-runner", srcs = glob(["**/*.java"]), main_class = "com.example.projectrunner", ) Dr. Balázs Simon, BME, IIT 33
Bazel example: Java As projects get larger it's important to break up the build into self-contained libraries: BUILD java_binary( name = "my-other-runner", srcs = ["src/main/java/com/example/projectrunner.java"], main_class = "com.example.projectrunner", deps = [":greeter"], ) java_library( name = "greeter", srcs = ["src/main/java/com/example/greeting.java"], ) Bazel can parallelize build steps! If greeter doesn t change, it won t be recompiled! Dr. Balázs Simon, BME, IIT 34
Properties of Bazel Advantages: simple build files very fast scalable reproducible builds can resolve external transitive dependencies (e.g. Maven) clear convention for the build steps IDE support for IntelliJ, Xcode partially for Eclipse Disadvantages: no convention for the project structure Dr. Balázs Simon, BME, IIT 35
Buck (Facebook) Dr. Balázs Simon, BME, IIT 36
Buck Similar to Bazel Written by people at Facebook who worked before at Google before Google open-sourced Bazel Multi-platform build tool Multi-language support Custom, simple high-level build language Reproducible builds Scalable Dr. Balázs Simon, BME, IIT 37
Buck example: Java BUCK java_library( name = 'greet_api', srcs = ['Greeter.java', 'Group.java'], ) java_library( name = 'hello_world', srcs = ['HelloGreeter.java', 'WorldGroup.java'], deps = [':greet_api'], ) Dr. Balázs Simon, BME, IIT 38
Properties of Buck Advantages: simple build files very fast scalable reproducible builds can resolve external transitive dependencies (e.g. Maven) clear convention for the build steps IDE support for IntelliJ, Xcode Disadvantages: no convention for the project structure no IDE support for Eclipse Dr. Balázs Simon, BME, IIT 39
Summary Dr. Balázs Simon, BME, IIT 40
Which one to use? Build time is precious Bazel and Buck are really fast! Flexibility Gradle is very flexible Bazel and Buck are relatively flexible but be careful: anything deviating from a convention is hard to learn and maintain Stability, consistency Maven is mature, reliable and consistent Maintainability Maven, Bazel, Buck are simple and easy to maintain IDE support Ant and Maven have great IDE support Dr. Balázs Simon, BME, IIT 41