JakeHillion/nixpkgs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
988677d741
nixpkgs/pkgs/development/haskell-modules/HACKING.md

12 KiB
Raw Blame History

Maintainer Workflow

The goal of the @NixOS/haskell team is to keep the Haskell packages in Nixpkgs up-to-date, while making sure there are no Haskell-related evaluation errors or build errors that get into the Nixpkgs master branch.

We do this by periodically merging an updated set of Haskell packages on the haskell-updates branch into the master branch. Each member of the team takes a two week period where they are in charge of merging the haskell-updates branch into master. This is the documentation for this workflow.

The workflow generally proceeds in three main steps:

  1. create the initial haskell-updates PR, and update Stackage and Hackage snapshots
  2. wait for contributors to fix newly broken Haskell packages
  3. merge haskell-updates into master

Each of these steps is described in a separate section.

Initial haskell-updates PR

In this section we create the PR for merging haskell-updates into master.

  1. Make sure the haskell-updates branch is up-to-date with master.

  2. Update the Stackage Nightly resolver used by Nixpkgs and create a commit:

    $ ./maintainers/scripts/haskell/update-stackage.sh --do-commit

  3. Update the Hackage package set used by Nixpkgs and create a commit:

    $ ./maintainers/scripts/haskell/update-hackage.sh --do-commit

  4. Regenerate the Haskell package set used in Nixpkgs and create a commit:

    $ ./maintainers/scripts/haskell/regenerate-hackage-packages.sh --do-commit

  5. Push these commits to the haskell-updates branch of the NixOS/nixpkgs repository.

  6. Open a PR on Nixpkgs merging haskell-updates into master.

Use the title haskellPackages: update stackage and hackage and the following message body:

### This Merge

This PR is the regular merge of the `haskell-updates` branch into `master`.

This branch is being continually built and tested by hydra at https://hydra.nixos.org/jobset/nixpkgs/haskell-updates.

I will aim to merge this PR **by 2021-TODO-TODO**. If I can merge it earlier, there might be successor PRs in that time window. As part of our rotation @TODO will continue these merges from 2021-TODO-TODO to 2021-TODO-TODO.

### haskellPackages Workflow Summary

Our workflow is currently described in
[`pkgs/development/haskell-modules/HACKING.md`](https://github.com/NixOS/nixpkgs/blob/haskell-updates/pkgs/development/haskell-modules/HACKING.md).

The short version is this:
* We regularly update the Stackage and Hackage pins on `haskell-updates` (normally at the beginning of a merge window).
* The community fixes builds of Haskell packages on that branch.
* We aim at at least one merge of `haskell-updates` into `master` every two weeks.
* We only do the merge if the [`mergeable`](https://hydra.nixos.org/job/nixpkgs/haskell-updates/mergeable) job is succeeding on hydra.
* If a [`maintained`](https://hydra.nixos.org/job/nixpkgs/haskell-updates/maintained) package is still broken at the time of merge, we will only merge if the maintainer has been pinged 7 days in advance. (If you care about a Haskell package, become a maintainer!)

---

This is the follow-up to #TODO.

Make sure to replace all TODO with the actual values.

Notify Maintainers and Fix Broken Packages

After you've done the previous steps, Hydra will start building the new and updated Haskell packages. You can see the progress Hydra is making at https://hydra.nixos.org/jobset/nixpkgs/haskell-updates. This Hydra jobset is defined in the file release-haskell.nix.

Notify Maintainers

When Hydra finishes building all the updated packages for the haskell-updates jobset, you should generate a build report to notify maintainers of their newly broken packages. You can do that with the following commands:

$ ./maintainers/scripts/haskell/hydra-report.hs get-report
$ ./maintainers/scripts/haskell/hydra-report.hs ping-maintainers

The hyda-report.hs ping-maintainers command generates a Markdown document that you can paste in a GitHub comment on the PR opened above. This comment describes which Haskell packages are now failing to build. It also pings the maintainers so that they know to fix up their packages.

It may be helpful to pipe hydra-report.hs ping-maintainers into xclip (XOrg) or wl-copy (Wayland) in order to post on GitHub.

This build report can be fetched and re-generated for new Hydra evaluations. It may help contributors to try to keep the GitHub comment updated with the most recent build report.

Maintainers should be given at least 7 days to fix up their packages when they break. If maintainers don't fix up their packages with 7 days, then they may be marked broken before merging haskell-updates into master.

Fix Broken Packages

After getting the build report, you can see which packages and Hydra jobs are failing to build. The most important jobs are the maintained and mergeable jobs. These are both defined in release-haskell.nix.

mergeable is a set of the most important Haskell packages, including things like Pandoc and XMonad. These packages are widely used. We would like to always keep these building.

maintained is a set of Haskell packages that have maintainers in Nixpkgs. We should be proactive in working with maintainers to keep their packages building.

Steps to fix Haskell packages that are failing to build is out of scope for this document, but it usually requires fixing up dependencies that are now out-of-bounds.

Mark Broken Packages

Packages that do not get fixed can be marked broken with the following commands. First check which packages are broken:

$ ./maintainers/scripts/haskell/hydra-report.hs get-report
$ ./maintainers/scripts/haskell/hydra-report.hs mark-broken-list

This shows a list of packages that reported a build failure on x86_64-linux on Hydra.

Next, run the following command:

$ ./maintainers/scripts/haskell/mark-broken.sh --do-commit

This first opens up an editor with the broken package list. Some of these packages may have a maintainer in Nixpkgs. If these maintainers have not been given 7 days to fix up their package, then make sure to remove those packages from the list before continuing. After saving and exiting the editor, the following will happen:

  • Packages from the list will be added to configuration-hackage2nix/broken.yaml. This is a list of Haskell packages that are known to be broken.

  • hackage-packages.nix will be regenerated. This will mark all Haskell pacakges in configuration-hackage2nix/broken.yaml as broken.

  • The configuration-hackage2nix/transitive-broken.yaml file will be updated. This is a list of Haskell packages that depend on a package in configuration-hackage2nix/broken.yaml or configuration-hackage2nix/transitive-broken.yaml

  • hackage-packages.nix will be regenerated again. This will set hydraPlatforms = none for all the packages in configuration-hackage2nix/transitive-broken.yaml. This makes sure that Hydra does not try to build any of these packages.

  • All updated files will be committed.

Merge master into haskell-updates

You should occasionally merge the master branch into the haskell-updates branch.

In an ideal world, when we merge haskell-updates into master, it would cause few Hydra rebuilds on master. Ideally, the nixos-unstable channel would never be prevented from progressing because of needing to wait for rebuilding Haskell packages.

In order to make sure that there are a minimal number of rebuilds after merging haskell-updates into master, master should occasionally be merged into the haskell-updates branch.

This is especially important after staging-next is merged into master, since there is a high chance that this will cause all the Haskell packages to rebuild.

Merge haskell-updates into master

Now it is time to merge the haskell-updates PR you opened above.

Before doing this, make sure of the following:

  • All Haskell packages that fail to build are correctly marked broken or transitively broken.

  • The maintained and mergeable jobs are passing on Hydra.

  • The maintainers for any maintained Haskell packages that are newly broken have been pinged on GitHub and given at least a week to fix their packages. This is especially important for widely-used packages like cachix.

  • Make sure you first merge the master branch into haskell-updates. Wait for Hydra to evaluate the new haskell-updates jobset. Make sure you only merge haskell-updates into master when there are no evaluation errors.

  • Due to Hydra having only a small number of Darwin build machines, the haskell-updates jobset on Hydra often has many queued Darwin jobs. In order to not have these queued Darwin jobs prevent the haskell-updates branch from being merged to master in a timely manner, we have special rules for Darwin jobs.

    • It is alright to merge the haskell-updates branch to master if there are remaining queued Darwin jobs on Hydra.

    • We would like to keep GHC and the mergeable job building on Darwin. Do not merge the haskell-updates branch to master if GHC is failing to build, or the mergeable job has failing Darwin constituent jobs.

      If GHC and the mergeable job are not failing, but merely queued, it is alright to merge the haskell-updates branch to master.

    • We do not need to keep the maintained job building on Darwin. If maintained packages are failing on Darwin, it is helpful to mark them as broken on that platform.

When you've double-checked these points, go ahead and merge the haskell-updates PR. After merging, make sure not to delete the haskell-updates branch, since it causes all currently open Haskell-related pull-requests to be automatically closed on GitHub.

Additional Info

Here are some additional tips that didn't fit in above.

  • Hydra tries to evaluate the haskell-updates branch (in the nixpkgs:haskell-updates jobset) every 4 hours. It is possible to force a new Hydra evaluation without waiting 4 hours by the following steps:

    1. Log into Hydra with your GitHub or Google account.
    2. Go to the nixpkgs:haskell-updates jobset.
    3. Click the Actions button.
    4. Select Evaluate this jobset.
    5. If you refresh the page, there should be a new Evaluation running since: line.
    6. Evaluations take about 10 minutes to finish.

  • It is sometimes helpful to update the version of cabal2nix / hackage2nix that our maintainer scripts use. This can be done with the maintainers/scripts/haskell/update-cabal2nix-unstable.sh script.

    You might want to do this if a user contributes a fix to cabal2nix that will immediately fix a Haskell package in Nixpkgs. First, merge in the PR to cabal2nix, then run update-cabal2nix-upstable.sh. Finally, run regenerate-hackage-packages.sh to regenerate the Hackage package set with the updated version of hackage2nix.

  • Make sure never to update the Hackage package hashes in pkgs/data/misc/hackage/, or the pinned Stackage Nightly versions on the release branches (like release-21.05).

    This means that the update-hackage.sh and update-stackage.sh scripts should never be used on the release branches.

    However, changing other files in ./. and regenerating the package set is encouraged. This can be done with regenerate-hackage-packages.sh as described above.

  • The Haskell team members generally hang out in the Matrix room #haskell:nixos.org.