How to add or update opam dependencies

When a merge request (MR) introduces a new dependency to an opam package, or updates an existing dependency to a different version of an opam package, additional steps must be taken in the development and merge process. This document explains those steps.

If you have already read this guide and only need a refresher, skip to the TL;DR.

Background

The Octez project is built under a system that is somewhat stricter than the default for OCaml projects. The goal is to make sure that users, developers and the CI use the exact same dependencies, with the exact same versions. To this end:

  • the set of opam dependencies and their exact version number is stored in an opam lock file ;

  • the hash of the commit to use from the public opam repository is stored in scripts/version.sh in the variable full_opam_repository_tag;

  • make build-deps and make build-dev-deps use the lock file and the hash to select dependencies;

  • the CI uses Docker images that come with those dependencies pre-compiled.

The Docker images for the CI are built by the CI of another repository, the so-called Tezos opam repository. The set of dependencies that is used to build those images is also defined by a lock file and a commit hash from the public opam repository. Both must be kept synchronized with their counterpart in the tezos/tezos repository.

Note

Docker images contain additional dependencies such as odoc which are needed by the CI but not to build Octez.

Adding, removing or updating dependencies thus requires to work both on the main codebase and on the Tezos opam repository. Moreover, work between those two components must happen in a specific order.

The rest of this document explains the process from the point-of-view of a developer (you). The instructions below assume you have already set up your work environment but that you installed development dependencies (make build-dev-deps instead of make build-deps).

Local work

The simplest way of using a new dependency on the Octez codebase when working locally (i.e., on your own machine) is to install it using opam.

Because you have used make build-dev-deps in order to install the Octez dependencies, you have access to the default opam repository in addition to the Tezos opam repository.

Install your dependency: opam install foo

Add dependencies to build files: both opam files and dune files must be updated. Add the dependency to the relevant declarations in manifest/main.ml. And then use make -C manifest to update the opam and dune files accordingly.

For example, if you are modifying the Shell using the new dependency, you must add an entry in the ~deps list of the let octez_shell = entry of the manifest/main.ml and then run make -C manifest. You should see the changes propagated onto opam/octez-libs.opam and src/lib_shell/dune, as well as opam/virtual/octez-deps.opam.

You can work on your feature, using the types and values provided by your new dependency.

Making an MR

Even though you can compile and run code locally, the CI will likely fail. This is because its set of available dependencies is now different from yours. You must follow the steps below in order to produce the necessary Docker images, allowing your work to eventually be merged.

First, in your local copy of Octez, update the full_opam_repository_tag variable in the scripts/version.sh file. You should set this variable to the commit hash of a recent version of the master branch of the default opam repository. (Note: this is not always necessary, but it is simpler for you to do so than to check whether it is necessary to do so.)

Second, update the opam lock file. The safest way to do that is to execute the scripts/update_opam_lock.sh script. It will ask opam to upgrade all Octez dependencies, making sure that unwanted package versions are not selected for dependencies, and will update the lock file accordingly. Note that the diff may include a few more changes than what you strictly need. Specifically, it might include some updates of some other dependencies. This is not an issue in general but it might explain some changes unrelated to your work.

Note

If you do not wish to upgrade all dependencies, you can also just run opam lock opam/virtual/octez-deps.opam followed by mv octez-deps.opam.locked opam/virtual, or even edit the lock file manually. Neither of these guarantees that packages are available in the commit identified by full_opam_repository_tag of the public opam repository, and even so, you may end up with unwanted versions of dependencies; so you should review the resulting lock file even more carefully. Editing the lock file manually is even less safe than running opam lock as it does not guarantee that the set of dependencies is actually a valid solution that the opam solver could have chosen.

Third, create an MR on the Tezos opam repository. This is the opam repository MR, its role is to prepare the environment for the Octez MR that we will create below.

In order to create the opam repository MR:

  • If you haven’t already done so, clone the Tezos opam repository.

  • Create a branch from the repository’s master and switch to it.

  • Update file scripts/version.sh (in the Tezos opam repository) to set the value of opam_repository_commit_hash to match the value of full_opam_repository_tag that you have set in scripts/version.sh (in the Octez repository).

  • Copy file opam/virtual/octez-deps.opam.locked (from the Octez repository) to the root of the Tezos opam repository.

  • Commit the result. Take note of the commit hash, it will be useful later.

  • Push your branch.

  • Create the opam repository MR from this branch.

You can test the MR locally using the command OPAM_REPOSITORY_TAG=<commit-id> make build-deps. This will rebuild the dependencies locally using the <commit_id> of the opam-repository.

Fourth, back in your local copy of Octez, update the opam_repository_tag variable in the scripts/version.sh file. Specifically, set it to the hash of your commit on the opam repository MR. Afterwards, you will also need to regenerate the GitLab CI configuration by running make -C ci from the root of the repository. Commit the change of scripts/version.sh and the GitLab configuration with a title along the lines of “CI: use dependency foo”.

This commit will point the build scripts and CI to the modified opam-repository and the associated Docker images. Do note that the CI on your branch of Octez will only be able to run after the CI on your branch of opam-repository has completed.

Finally, still in your local copy of Octez, push these changes and open an MR on the tezos/tezos project. Make sure you add links referencing the opam-repository MR from the Octez MR and vice-versa. This gives the reviewers the necessary context to review.

That’s it. You now have two MRs:

  • The opam-repository MR from tezos/opam-repository:<your-branch> onto tezos/opam-repository:master updates the environment in which the Octez libraries and binaries are built.

  • The Octez MR from <your-organisation>/tezos:<your-branch> onto tezos/tezos:master uses this new environment.

Merging the MR

This section is for the Octez merge team. It is the last step in the lifetime of the MRs you have opened. Understanding the basics of this process may help you when communicating with the reviewers and the mergers of your MR. Understanding all the minutiae and details is not necessary. For this reason, this final section is addressed to whichever member of the Octez merge team takes care of this MR (you).

After the iterative review-comment-edit process has reached a satisfying fixpoint, you can merge the two MRs opened by the developer. To avoid interference with other MRs, it is better to perform all the steps described below relatively quickly (the same day).

First, mention the MR on the #opam-repo Slack channel and make sure there isn’t another merge ongoing.

Second, merge the opam-repository MR. Make sure that the commit hash of master is the value of opam_repository_tag in scripts/version.sh. The hash could have changed if a merge commit was introduced, if the branch had to be rebased, if it was squashed, etc. This is important because the name of the Docker images is based on this hash.

Finally, assign the Octez MR to Marge Bot for merging.

TL;DR

As a developer:

  • You have an Octez MR from <your-organisation>/tezos:<your-branch> onto tezos/tezos:master introducing a dependency to foo.

  • You amend the manifest/main.ml file to declare the dependency.

  • You propagate the changes to opam and dune files by running make -C manifest.

  • You update the full_opam_repository_tag to the commit hash of a recent version of the public default opam repository.

  • You update opam/virtual/octez-deps.opam.locked, for instance by executing scripts/update_opam_lock.sh.

  • You open an opam repository MR from tezos/opam-repository:<your-branch> onto tezos/opam-repository:master that updates:

    • variable opam_repository_commit_hash in scripts/version.sh;

    • file octez-deps.opam.locked at the root.

  • You update opam_repository_tag to the hash of the last commit of your opam repository MR and regenerate the CI configuration.

  • You push the changes to your Octez MR.

  • You update the descriptions of your MRs to include links between them.

As a merger:

  • You test, review, etc. the code.

  • You merge the opam repository MR.

  • You make sure the commit hash has been preserved by merging (no squashing, no rebasing, no merge commit…).

  • You assign the Octez MR to Marge Bot.