Composer and Drupal CIDUG Meeting December 13, 2018 John Rearick
* Similar to other dependency managers such as: yum, apt, brew, macports, npm, pip, etc. * Helps manage dependency hell. * Lots of dependencies * Not only project dependencies, but also versions of projects as well * Long chains of dependency - A requires B requires C requires. * Not 100% solution, but makes it managable. Composer "Dependency manager for PHP"
PHP is a much larger community than Drupal. With the basis in Symfony Drupal 8 is really getting off the Drupal Island. In order to more easily use the solutions the rest of the PHP community has come up with, Drupal decided that playing nicely was required. So we now use composer. Circular work. Composer has made Drupal development easier, and Drupal has made composer nicer for the benefit of other PHP projects. Get off the Drupal island
Building a Drupal Website Normally think about building a drupal website so we start with Drupal. Then we add things to that.
Building a PHP Website with Drupal as a dependency However, we re really building a PHP project that uses Drupal. So our website s first dependency is Drupal itself. This allows us to only have our own code in the repository and we don t have to keep track so much with Drupal or any other dependencies
Get and install Composer Globally copy commands from getcomposer.org/download/ mv composer.phar /usr/local/bin/composer Add /Users/username/.composer/vendor/bin to your PATH Update composer with composer self-update
Start a project with drupal-composer/drupal-project Kickstart a plain Drupal project with composer: composer create-project drupal-composer/ drupal-project:8.x-dev some-dir --stability dev --no-interaction https://github.com/drupal-composer/drupal-project composer - the composer command, installed globally create-project - creates a folder with all the files we expect drupal-composer/drupal-project:8.x-dev drupal-composer - org drupal-project - project/repo name :8.x-dev - defined version. (this is not the drupal version, this is the drupalproject version) some-dir - Whatever directory you want this to create stability dev - Just go with it. no-interaction - Please don t bother asking questions, just go with defaults. Thanks
https://github.com/drupal-composer/drupal-project to find the command Replace some-dir with mysite Let s Build
Where does it get code from? Packagist is a Repository or database of PHP projects. Configured by default with composer. Not a git repository (necessarily) Similar to something like the subscription channels in RHEL https://packagist.org/
Where does it get code from? packages.drupal.org Other repositories are available for composer. Usually one per project or look at Satis, an open source repository generator https://github.com/ composer/satis Your own private repositories!
WARNING Projects from public repositories, such as Packagist, may not be relied upon. Project could be pulled off GitHub SCM Repository could get compromised Use trusted repositories and projects Examples: * left-pad in npm was pulled when the owner had a disagreement of philosophy and removed all of their own projects from the repository. NPM had to restore the left-pad project after builds broke. https:// www.theregister.co.uk/2016/03/23/npm_left_pad_chaos/ * event-stream in NPM (again). Commit access was given to someone unvetted and added bitcoin code to the project, which, in turn got deployed to numerous websites https://www.theregister.co.uk/2018/11/26/ npm_repo_bitcoin_stealer/ * drupal.org code is pretty well reviewed and there is a policy for managing projects in the Drupal repository
Check out the build log from composer and what we got in finder Done Building?
click to zoom
notice: we have a.gitignore, but not a.git folder. This isn t a git repo. Yet notice: composer.json notice: composer.lock notice: web folder.
Composer.json * simple json file * Don t usually need to edit this by hand. Please use the composer commands if you can * Starts with basic information * Note the lack of comma on the last time of a list. Always catches me off guard * composer validate to check your composer.json for errors * be in the root of the directory (though it has gotten smarter about that recently)
* Repositories are the repositories we talked about earlier. * Not always git repositories * Can define a git repo for a single private project if needed * here we see the Drupal 8 project repository maintained by the DA
* Here we see the project s dependencies. * format of org/project * Note: cweagans/composer-patches * Semantic Verstioning Major.Minor.Patch. * Major - breaking changes. remove deprecated functions (Drupal 9) * Minor - New features. Generally shouldn t break anything, but you know how that goes * Patch - quick bug fixes that can t wait for a minor release or security updates usually * Noticing the ^ has some meaning, we ll get to that in a bit * Fuzzy version dependency. You can define a range of versions. * add these dependencies with the command line. We ll get to that. * composer has a concept of a dev version so you can add dependencies when in development mode. (I haven t really used this yet) * perhaps can be used with something like devel module or features module
* Extra stuff can be added to enhance composer or configure other plugins. * Here we can see that we re configuring patch level to -p2 so that patches can be applied in the web folder * Patches can be defined here too thanks to the composer-patches requirement we had earlier (we can cover that if you want later) *
* Installer paths define where code should go. * by default code ends up in the the /vendor/org/project folder * Drupal needs modules, themes, profiles, core, and drush commands installed in a specific place * type is defined in a project s composer.json at the top.
* Drupal Scaffold is a plugin written for this composer project so that files like index.php,.htaccess, etc get placed properly * Here we see a configuration to move the.editorconfig and.gitattributes files to the root of the project instead of the web folder * When updating drupal core check the git diff to review any possible changes you may have made to these files. * You can configure drupal-scaffold to ignore certain files by default as well. We have.htaccess ignored because shibboleth config
Composer.lock * Don t edit this file manually! * Is used to build out all the files as defined in composer.json * locks the specific version of dependency code. * sort of like a cache
Composer.lock Merge Conflict It s easy to get conflicts in this file when two people update the lock file. I found http://blog.doh.ms/2016/11/28/solving-conflicts-in-composer-lock/ to be a good strategy to manage those conflicts. http://blog.doh.ms/2016/11/28/solving-conflicts-incomposer-lock/
.gitignore next slide to zoom in
.gitignore * We don t have a git repo yet. But it has suggestions for what to put in.gitignore. * Notice we don t commit core, contrib, or libraries! * These files are all managed by composer and brought in. So, there really isn t any need to track these files.
fiddle stuff mkdir -p config/sync Set up Drupal
Install Modules composer require drupal/token
Define Versions Same as before. composer require drupal/token:^1.5 ^ Latest stable version according to semver ~ Allows only last digit to go up Other constraints such as * < > work too. getcomposer.org/doc/articles/versions.md#nextsignificant-release-operators
Other Examples composer require drupal/token:^1.5' composer require 'drupal/simple_fb_connect:~3.0' composer require 'drupal/ctools:3.0.0-alpha26' * Token 8.x-1.5 to 8.x-1.n * simple_fb_connect 8.x-3.0 to 3.n * ~ and ^ mean almost the same thing in Drupal land for now, because semver translates by adding an extra.0 at the end of the versions. * ctools specifically 8.x-3.0-alpha26 * token 8.x-1.x-dev composer require 'drupal/token:1.x-dev'
* composer require drupal/token * See the change in composer.json * composer.lock also updated Install token
Updating Modules * `composer update` without anything will update everything. Be careful! * --with-dependencies is important because we want to update any dependencies as required usually. composer update drupal/token --with-dependencies
Updating Drupal Core composer update drupal/core webflo/drupal-corerequire-dev symfony/* --with-dependencies * Go to the composer project page and look at the readme for recommended steps: https://github.com/drupal-composer/drupalproject#updating-drupal-core * Git diff any changes to confirm stuff. Especially for scaffold files that you modified like.htaccess *
Git * git init * git add. * git commit -m inital commit * Install another module drupal/redirect * observe changes only in composer.json and composer.lock * take a gander at the diff * clone a new copy * cd.. * mkdir other * cd other/ * git clone../some-dir/. * realize there is no vendor, core, or nothing in modules directory! * Let s bring it all back in. * composer install * reads the composer.lock file and fetches all the stuff and puts them in place * git status shows no changes * see our files are there! * oh hey, where s redirect? (we didn t commit it ) * got back to ~/Sites/some-dir * git add. * git commit -m adding redirect module * cd.. * cd other * git pull * composer install to get us the redirect module YAY
Deployment * composer install as part of your deployment is fine composer install can be a bit of a resource hog, and some may not want that running on production. Use a build server or something to create an artifact to deploy
Artifact Adds complexity to the build/deployment process Adds production code auditing. Reduces resources required for production servers. * There s a lot of moving parts to keep track of. Need a place to make the build and move the files * Code on the production server can be committed to a repo and it s easier to see if files were changed, with git or some such. * composer building requires a lot of memory to run and can be resource intensive. Better to offload that to once in a while on circleci or something ephemeral and cheap than needing to run an Amazon Larger that spends most of its time idle. or trying to figure out how to scale containers.
Artifact 1. Deploy to a build server 2. build server runs composer install 3. rsync or commit/push/pull the code files to a production server * Build server something like circleci or travis, can be configured to work on web hooks, for automatic deployment * Build server does the hard work of building everything * Build server will need to have access to commit or trigger a deployment with Tower or something. Keys can be added to the build server in an encrypted store * Could be rsync, a separate git repo or even a branch on the original repo * Using git, be sure that the.gitignore doesn t ignore the important files like vendor, core, modules, etc
Private Repositories * Remember, not git repo, package repo We can t use a public repository like Packagist Private/proprietary code on private git repo Doesn t make sense to share with the world
Private Repositories Private Packagist Open source project Requires infrastructure to build and host https://github.com/composer/satis * Made by the people who made composer * It s a PHP project that can be installed and managed with composer (of course) * Has a build process that needs to run any time there s an update to any project * Build process is usually done on cron every few minutes * I haven t done this, but to looks pretty easy to set up
Private Repositories Project specific repo A project can also be a repository Easy to setup * A couple lines in the project s composer.json * can be hosted on any git repository * Bringing into project you need to define both a repo and project requirement. * If your org has a lot of private projects, can be hard to manage since there s not central place Cumbersome if you have too many
Set up project repo * See https://github.com/jrearick/drush_example * Name matches github path * description is nice * type is drupal-module to make sure it goes into modules/contrib/ drush_example * Perhaps this can be changed to org-module or something and update the base project composer.json to handle it * repositories. Define itself with type git and url * a composer type is used for other repositories like drupal.org, satis. Added here in case this module wants to define dependencies itself * Go back to our local composer.json * { * "type": "git", * "url": "git@github.com:jrearick/drush_example.git" * }, * composer validate * Ignore warning about lock file for now, see that the file is valid * "jrearick/drush_example": master", * master means master branch. Always get the master. You can probably define tags here with semver? * composer validate * See warning about trying to avoid unbound version constraints. * composer update jrearick/drush_example * composer validate should no longer warn about outdated composer.lock (avoided updating everything else ) * git diff to see the composer.lock changes * cd web/modules/contrib/drush_example to see the code is there * cd back to the project root and commit
Patch Management * The killer app for composer * With the super fast development of Drupal 8 and Drupal 8 modules, it s nice to be able to get patches and manage them composer can download and apply patches for you!
* we have an issue https://www.drupal.org/node/1091852 * Have a patch on comment 80 * Copy raw patch link https://www.drupal.org/files/issues/ Patch Management 2018-11-08/1091852-80.patch * edit composer.json *, * "patches": { * "drupal/core": { * "1091852-80 Patches states for ajax fields": "https:// www.drupal.org/files/issues/2018-11-08/1091852-80.patch * } * } * composer validate * composer update drupal/core * local patches can be applied too, instead of a url give a path patches/ my_custom.patch * when patches don t apply, get you notified in update process.
Composer Resources The quick install: https://getcomposer.org/download/ Using composer to manage a Drupal project: https://www.drupal.org/docs/develop/using-composer/usingcomposer-to-manage-drupal-site-dependencies composer.lock resolving merge conflict: http://blog.doh.ms/2016/11/28/ solving-conflicts-in-composer-lock/ Satis composer repository generator: https://github.com/composer/ satis Tips for Managing Drupal 8 projects with composer: https:// www.jeffgeerling.com/blog/2017/tips-managing-drupal-8-projectscomposer