JakeHillion/nixpkgs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

gradle: add setup hook

Browse Source
This commit is contained in:
chayleaf 2023-12-06 06:24:17 +07:00
parent 62d13413f4
commit c12b2a0b19
No known key found for this signature in database
GPG Key ID: 78171AD46227E68E
10 changed files with 1080 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

189
doc/languages-frameworks/gradle.section.md Normal file
View File

@ -0,0 +1,189 @@
# Gradle {#gradle}
Gradle is a popular build tool for Java/Kotlin. Gradle itself doesn't
currently provide tools to make dependency resolution reproducible, so
nixpkgs has a proxy designed for intercepting Gradle web requests to
record dependencies so they can be restored in a reproducible fashion.
## Building a Gradle package {#building-a-gradle-package}
Here's how a typical derivation will look like:
```nix
stdenv.mkDerivation (finalAttrs: {
pname = "pdftk";
version = "3.3.3";
src = fetchFromGitLab {
owner = "pdftk-java";
repo = "pdftk";
rev = "v${finalAttrs.version}";
hash = "sha256-ciKotTHSEcITfQYKFZ6sY2LZnXGChBJy0+eno8B3YHY=";
};
nativeBuildInputs = [ gradle ];
# if the package has dependencies, mitmCache must be set
mitmCache = gradle.fetchDeps {
inherit (finalAttrs) pname;
data = ./deps.json;
};
# this is required for using mitm-cache on Darwin
__darwinAllowLocalNetworking = true;
gradleFlags = [ "-Dfile.encoding=utf-8" ];
# defaults to "assemble"
gradleBuildTask = "shadowJar";
# will run the gradleCheckTask (defaults to "test")
doCheck = true;
installPhase = ''
mkdir -p $out/{bin,share/pdftk}
cp build/libs/pdftk-all.jar $out/share/pdftk
makeWrapper ${jre}/bin/java $out/bin/pdftk \
--add-flags "-jar $out/share/pdftk/pdftk-all.jar"
cp ${finalAttrs.src}/pdftk.1 $out/share/man/man1
'';
meta.sourceProvenance = with lib.sourceTypes; [
fromSource
binaryBytecode # mitm cache
];
})
```
To update (or initialize) dependencies, run the update script via
something like `$(nix-build -A <pname>.mitmCache.updateScript)`
(`nix-build` builds the `updateScript`, `$(...)` runs the script at the
path printed by `nix-build`).
If your package can't be evaluated using a simple `pkgs.<pname>`
expression (for example, if your package isn't located in nixpkgs, or if
you want to override some of its attributes), you will usually have to
pass `pkg` instead of `pname` to `gradle.fetchDeps`. There are two ways
of doing it.
The first is to add the derivation arguments required for getting the
package. Using the pdftk example above:
```nix
{ lib
, stdenv
# ...
, pdftk
}:
stdenv.mkDerivation (finalAttrs: {
# ...
mitmCache = gradle.fetchDeps {
pkg = pdftk;
data = ./deps.json;
};
})
```
This allows you to `override` any arguments of the `pkg` used for
the update script (for example, `pkg = pdftk.override { enableSomeFlag =
true };`), so this is the preferred way.
The second is to create a `let` binding for the package, like this:
```nix
let self = stdenv.mkDerivation {
# ...
mitmCache = gradle.fetchDeps {
pkg = self;
data = ./deps.json;
};
}; in self
```
This is useful if you can't easily pass the derivation as its own
argument, or if your `mkDerivation` call is responsible for building
multiple packages.
In the former case, the update script will stay the same even if the
derivation is called with different arguments. In the latter case, the
update script will change depending on the derivation arguments. It's up
to you to decide which one would work best for your derivation.
## Update Script {#gradle-update-script}
The update script does the following:
- Build the derivation's source via `pkgs.srcOnly`
- Enter a `nix-shell` for the derivation in a `bwrap` sandbox (the
sandbox is only used on Linux)
- Set the `IN_GRADLE_UPDATE_DEPS` environment variable to `1`
- Run the derivation's `unpackPhase`, `patchPhase`, `configurePhase`
- Run the derivation's `gradleUpdateScript` (the Gradle setup hook sets
a default value for it, which runs `preBuild`, `preGradleUpdate`
hooks, fetches the dependencies using `gradleUpdateTask`, and finally
runs the `postGradleUpdate` hook)
- Finally, store all of the fetched files' hashes in the lockfile. They
may be `.jar`/`.pom` files from Maven repositories, or they may be
files otherwise used for building the package.
`fetchDeps` takes the following arguments:
- `attrPath` - the path to the package in nixpkgs (for example,
`"javaPackages.openjfx22"`). Used for update script metadata.
- `pname` - an alias for `attrPath` for convenience. This is what you
will generally use instead of `pkg` or `attrPath`.
- `pkg` - the package to be used for fetching the dependencies. Defaults
to `getAttrFromPath (splitString "." attrPath) pkgs`.
- `bwrapFlags` - allows you to override bwrap flags (only relevant for
downstream, non-nixpkgs projects)
- `data` - path to the dependencies lockfile (can be relative to the
package, can be absolute). In nixpkgs, it's discouraged to have the
lockfiles be named anything other `deps.json`, consider creating
subdirectories if your package requires multiple `deps.json` files.
## Environment {#gradle-environment}
The Gradle setup hook accepts the following environment variables:
- `mitmCache` - the MITM proxy cache imported using `gradle.fetchDeps`
- `gradleFlags` - command-line flags to be used for every Gradle
invocation (this simply registers a function that uses the necessary
flags).
- You can't use `gradleFlags` for flags that contain spaces, in that
case you must add `gradleFlagsArray+=("-flag with spaces")` to the
derivation's bash code instead.
- If you want to build the package using a specific Java version, you
can pass `"-Dorg.gradle.java.home=${jdk}"` as one of the flags.
- `gradleBuildTask` - the Gradle task (or tasks) to be used for building
the package. Defaults to `assemble`.
- `gradleCheckTask` - the Gradle task (or tasks) to be used for checking
the package if `doCheck` is set to `true`. Defaults to `test`.
- `gradleUpdateTask` - the Gradle task (or tasks) to be used for
fetching all of the package's dependencies in
`mitmCache.updateScript`. Defaults to `nixDownloadDeps`.
- `gradleUpdateScript` - the code to run for fetching all of the
package's dependencies in `mitmCache.updateScript`. Defaults to
running the `preBuild` and `preGradleUpdate` hooks, running the
`gradleUpdateTask`, and finally running the `postGradleUpdate` hook.
- `gradleInitScript` - path to the `--init-script` to pass to Gradle. By
default, a simple init script that enables reproducible archive
creation is used.
- Note that reproducible archives might break some builds. One example
of an error caused by it is `Could not create task ':jar'. Replacing
an existing task that may have already been used by other plugins is
not supported`. If you get such an error, the easiest "fix" is
disabling reproducible archives altogether by setting
`gradleInitScript` to something like `writeText
"empty-init-script.gradle" ""`
- `enableParallelBuilding` / `enableParallelChecking` /
`enableParallelUpdating` - pass `--parallel` to Gradle in the
build/check phase or in the update script. Defaults to true. If the
build fails for mysterious reasons, consider setting this to false.
- `dontUseGradleConfigure` / `dontUseGradleBuild` / `dontUseGradleCheck`
\- force disable the Gradle setup hook for certain phases.
- Note that if you disable the configure hook, you may face issues
such as `Failed to load native library 'libnative-platform.so'`,
because the configure hook is responsible for initializing Gradle.

245
pkgs/development/tools/build-managers/gradle/README.md Normal file
View File

@ -0,0 +1,245 @@
# Gradle Setup Hook
## Introduction
Gradle build scripts are written in a DSL, computing the list of Gradle
dependencies is a turing-complete task, not just in theory but in
practice. Fetching all of the dependencies often requires building some
native code, running some commands to check the host platform, or just
fetching some files using either JVM code or commands like `curl` or
`wget`.
This practice is widespread and isn't considered a bad practice in the
Java world, so all we can do is run Gradle to check what dependencies
end up being fetched, and allow derivation authors to apply workarounds
so they can run the code necessary for fetching the dependencies our
script doesn't fetch.
"Run Gradle to check what dependencies end up being fetched" isn't a
straightforward task. For example, Gradle usually uses Maven
repositories, which have features such as "snapshots", a way to always
use the latest version of a dependency as opposed to a fixed version.
Obviously, this is horrible for reproducibility. Additionally, Gradle
doesn't offer a way to export the list of dependency URLs and hashes (it
does in a way, but it's far from being complete, and as such is useless
for nixpkgs). Even if did, it would be annoying to use considering
fetching non-Gradle dependendencies in Gradle scripts is commonplace.
That's why the setup hook uses mitm-cache, a program designed for
intercepting all HTTP requests, recording all the files that were
accessed, creating a Nix derivation with all of them, and then allowing
the Gradle derivation to access these files.
## Maven Repositories
(Reference: [Repository
Layout](https://cwiki.apache.org/confluence/display/MAVENOLD/Repository+Layout+-+Final))
Most of Gradle dependencies are fetched from Maven repositories. For
each dependency, Gradle finds the first repo where it can successfully
fetch that dependency, and uses that repo for it. Different repos might
actually return different files for the same artifact because of e.g.
pom normalization. Different repos may be used for the same artifact
even across a single package (for example, if two build scripts define
repositories in a different order).
The artifact metadata is specified in a .pom file, and the artifacts
themselves are typically .jar files. The URL format is as follows:
`<repo>/<group-id>/<artifact-id>/<base-version>/<artifact-id>-<version>[-<classifier>].<ext>`
For example:
- `https://repo.maven.apache.org/maven2/org/slf4j/slf4j-api/2.0.9/slf4j-api-2.0.9.pom`
- `https://oss.sonatype.org/content/groups/public/com/tobiasdiez/easybind/2.2.1-SNAPSHOT/easybind-2.2.1-20230117.075740-16.pom`
Where:
- `<repo>` is the repo base (`https://repo.maven.apache.org/maven2`)
- `<group-id>` is the group ID with dots replaced with slashes
(`org.slf4j` -> `org/slf4j`)
- `<artifact-id>` is the artifact ID (`slf4j-api`)
- `<base-version>` is the artifact version (`2.0.9` for normal
artifacts, `2.2.1-SNAPSHOT` for snapshots)
- `<version>` is the artifact version - can be either `<base-version>`
or `<version-base>-<timestamp>-<build-num>` (`2.0.9` for normal
artifacts, and either `2.2.1-SNAPSHOT` or `2.2.1-20230117.075740-16`
for snapshots)
- `<version-base>` - `<base-version>` without the `-SNAPSHOT` suffix
- `<timestamp>` - artifact build timestamp in the `YYYYMMDD.HHMMSS`
format (UTC)
- `<build-num>` - a counter that's incremented by 1 for each new
snapshot build
- `<classifier>` is an optional classifier for allowing a single .pom to
refer to multiple .jar files. .pom files don't have classifiers, as
they describe metadata.
- `<ext>` is the extension. .pom
Note that the artifact ID can contain `-`, so you can't extract the
artifact ID and version from just the file name.
Additionally, the files in the repository may have associated signature
files, formed by appending `.asc` to the filename, and hashsum files,
formed by appending `.md5` or `.sha1` to the filename. The signatures
are harmless, but the `.md5`/`.sha1` files are rejected.
The reasoning is as follows - consider two files `a.jar` and `b.jar`,
that have the same hash. Gradle will fetch `a.jar.sha1`, find out that
it hasn't yet downloaded a file with this hash, and then fetch `a.jar`,
and finally download `b.jar.sha1`, locate it in its cache, and then
*not* download `b.jar`. This means `b.jar` won't be stored in the MITM
cache. Then, consider that on a later invocation, the fetching order
changed, whether it was because of a running on different system,
changed behavior after a Gradle update, or any other source of
nondeterminism - `b.jar` is fetched before `a.jar`. Gradle will first
fetch `b.jar.sha1`, not find it in its cache, attempt to fetch `b.jar`,
and fail, as the cache doesn't have that file.
For the same reason, the proxy strips all checksum/etag headers. An
alternative would be to make the proxy remember previous checksums and
etags, but that would complicate the implementation - however, such a
feature can be implemented if necessary. Note that checksum/etag header
stripping is hardcoded, but `.md5/.sha1` file rejection is configured
via CLI arguments.
**Caveat**: Gradle .module files also contain file hashes, in md5, sha1,
sha256, sha512 formats. It posed no problem as of yet, but it might in
the future. If it does pose problems, the deps derivation code can be
extended to find all checksums in .module files and copy existing files
there if their hash matches.
## Snapshots
Snapshots are a way to publish the very latest, unstable version of a
dependency that constantly changes. Any project that depends on a
snapshot will depend on this rolling version, rather than a fixed
version. It's easy to understand why this is a bad idea for reproducible
builds. Still, they can be dealt with by the logic in `gradle.fetchDeps`
and `gradle.updateDeps`.
First, as you can see above, while normal artifacts have the same
`base-version` and `version`, for snapshots it usually (but not
necessarily) differs.
Second, for figuring out where to download the snapshot, Gradle consults
`maven-metadata.xml`. With that in mind...
## Maven Metadata
(Reference: [Maven
Metadata](https://maven.apache.org/repositories/metadata.html),
[Metadata](https://maven.apache.org/ref/3.9.8/maven-repository-metadata/repository-metadata.html)
Maven metadata files are called `maven-metadata.xml`.
There are three levels of metadata: "G level", "A level", "V level",
representing group, artifact, or version metadata.
G level metadata is currently unsupported. It's only used for Maven
plugins, which Gradle presumably doesn't use.
A level metadata is used for getting the version list for an artifact.
It's an xml with the following items:
- `<groupId>` - group ID
- `<artifactId>` - artifact ID
- `<versioning>`
- `<latest>` - the very latest base version (e.g. `2.2.1-SNAPSHOT`)
- `<release>` - the latest non-snapshot version
- `<versions>` - the version list, each in a `<version>` tag
- `<lastUpdated>` - the metadata update timestamp (UTC,
`YYYYMMDDHHMMSS`)
V level metadata is used for listing the snapshot versions. It has the
following items:
- `<groupId>` - group ID
- `<artifactId>` - artifact ID
- `<versioning>`
- `<lastUpdated>` - the metadata update timestamp (UTC,
`YYYYMMDDHHMMSS`)
- `<snapshot>` - info about the latest snapshot version
- `<timestamp>` - build timestamp (UTC, `YYYYMMDD.HHMMSS`)
- `<buildNumber>` - build number
- `<snapshotVersions>` - the list of all available snapshot file info,
each info is enclosed in a `<snapshotVersion>`
- `<classifier>` - classifier (optional)
- `<extension>` - file extension
- `<value>` - snapshot version (as opposed to base version)
- `<updated>` - snapshot build timestamp (UTC, `YYYYMMDDHHMMSS`)
## Lockfile Format
The mitm-cache lockfile format is described in the [mitm-cache
README](https://github.com/chayleaf/mitm-cache#readme).
The nixpkgs Gradle lockfile format is more complicated:
```json
{
"!comment": "This is a nixpkgs Gradle dependency lockfile. For more details, refer to the Gradle section in the nixpkgs manual.",
"!version": 1,
"https://oss.sonatype.org/content/repositories/snapshots/com/badlogicgames/gdx-controllers": {
"gdx-controllers#gdx-controllers-core/2.2.4-20231021.200112-6/SNAPSHOT": {
"jar": "sha256-Gdz2J1IvDJFktUD2XeGNS0SIrOyym19X/+dCbbbe3/U=",
"pom": "sha256-90QW/Mtz1jbDUhKjdJ88ekhulZR2a7eCaEJoswmeny4="
},
"gdx-controllers-core/2.2.4-SNAPSHOT/maven-metadata": {
"xml": {
"groupId": "com.badlogicgames.gdx-controllers"
}
}
},
"https://repo.maven.apache.org/maven2": {
"com/badlogicgames/gdx#gdx-backend-lwjgl3/1.12.1": {
"jar": "sha256-B3OwjHfBoHcJPFlyy4u2WJuRe4ZF/+tKh7gKsDg41o0=",
"module": "sha256-9O7d2ip5+E6OiwN47WWxC8XqSX/mT+b0iDioCRTTyqc=",
"pom": "sha256-IRSihaCUPC2d0QzB0MVDoOWM1DXjcisTYtnaaxR9SRo="
}
}
}
```
`!comment` is a human-readable description explaining what the file is,
`!version` is the lockfile version (note that while it shares the name
with mitm-cache's `!version`, they don't actually have to be in sync and
can be bumped separately).
The other keys are parts of a URL. Each URL is split into three parts.
They are joined like this: `<part1>/<part2>.<part3>`.
Some URLs may have a `#` in them. In that case, the part after `#` is
parsed as `#<artifact-id>/<version>[/SNAPSHOT][/<classifier>].<ext>` and
expanded into
`<artifact-id>/<base-version>/<artifact-id>-<version>[-<classifier>].<ext>`.
Each URL has a value associated with it. The value may be:
- an SRI hash (string)
- for `maven-metadata.xml` - an attrset containing the parts of the
metadata that can't be generated in Nix code (e.g. `groupId`, which is
challenging to parse from a URL because it's not always possible to
discern where the repo base ends and the group ID begins).
`compress-deps-json.py` converts the JSON from mitm-cache format into
nixpkgs Gradle lockfile format. `fetch.nix` does the opposite.
## Security Considerations
Lockfiles won't be human-reviewed. They must be tampering-resistant.
That's why it's imperative that nobody can inject their own contents
into the lockfiles.
This is achieved in a very simple way - the `deps.json` only contains
the following:
- `maven-metadata.xml` URLs and small pieces of the contained metadata
(most of it will be generated in Nix, i.e. the area of injection is
minimal, and the parts that aren't generated in Nix are validated).
- artifact/other file URLs and associated hashes (Nix will complain if
the hash doesn't match, and Gradle won't even access the URL if it
doesn't match)
Please be mindful of the above when working on Gradle support for
nixpkgs.

163
pkgs/development/tools/build-managers/gradle/compress-deps-json.py Normal file
View File

@ -0,0 +1,163 @@
import json
import sys
from typing import Dict, Set
# this compresses MITM URL lists with Gradle-specific optimizations
# specifically, it splits each url into up to 3 parts - they will be
# concatenated like part1/part2.part3 or part1.part2
# part3 is simply always the file extension, but part1 and part2 is
# optimized using special heuristics
# additionally, if part2 ends with /a/b/{a}-{b}, the all occurences of
# /{a}/{b}/ are replaced with #
# finally, anything that ends with = is considered SHA256, anything that
# starts with http is considered a redirect URL, anything else is
# considered text
with open(sys.argv[1], "rt") as f:
data: dict = json.load(f)
new_data: Dict[str, Dict[str, Dict[str, dict]]] = {}
for url, info in data.items():
if url == "!version":
continue
ext, base = map(lambda x: x[::-1], url[::-1].split(".", 1))
if base.endswith(".tar"):
base = base[:-4]
ext = "tar." + ext
# special logic for Maven repos
if ext in ["jar", "pom", "module"]:
comps = base.split("/")
if "-" in comps[-1]:
# convert base/name/ver/name-ver into base#name/ver
filename = comps[-1]
name = comps[-3]
basever = comps[-2]
ver = basever
is_snapshot = ver.endswith("-SNAPSHOT")
if is_snapshot:
ver = ver.removesuffix("-SNAPSHOT")
if filename.startswith(f"{name}-{ver}"):
if is_snapshot:
if filename.startswith(f"{name}-{ver}-SNAPSHOT"):
ver += "-SNAPSHOT"
else:
ver += "-".join(
filename.removeprefix(f"{name}-{ver}").split("-")[:3]
)
comp_end = comps[-1].removeprefix(f"{name}-{ver}")
else:
ver, name, comp_end = None, None, None
if name and ver and (not comp_end or comp_end.startswith("-")):
base = "/".join(comps[:-1]) + "/"
base = base.replace(f"/{name}/{basever}/", "#")
base += f"{name}/{ver}"
if is_snapshot:
base += "/SNAPSHOT"
if comp_end:
base += "/" + comp_end[1:]
scheme, rest = base.split("/", 1)
if scheme not in new_data.keys():
new_data[scheme] = {}
if rest not in new_data[scheme].keys():
new_data[scheme][rest] = {}
if "hash" in info.keys():
new_data[scheme][rest][ext] = info["hash"]
elif "text" in info.keys() and ext == "xml":
# nix code in fetch-deps.nix will autogenerate metadata xml files groupId
# is part of the URL, but it can be tricky to parse as we don't know the
# exact repo base, so take it from the xml and pass it to nix
xml = "".join(info["text"].split())
new_data[scheme][rest][ext] = {
"groupId": xml.split("<groupId>")[1].split("</groupId>")[0],
}
if "<release>" in xml:
new_data[scheme][rest][ext]["release"] = xml.split("<release>")[1].split(
"</release>"
)[0]
if "<latest>" in xml:
latest = xml.split("<latest>")[1].split("</latest>")[0]
if latest != new_data[scheme][rest][ext].get("release"):
new_data[scheme][rest][ext]["latest"] = latest
if "<lastUpdated>" in xml:
new_data[scheme][rest][ext]["lastUpdated"] = xml.split("<lastUpdated>")[
1
].split("</lastUpdated>")[0]
else:
raise Exception("Unsupported key: " + repr(info))
# At this point, we have a map by part1 (initially the scheme), part2 (initially a
# slash-separated string without the scheme and with potential # substitution as
# seen above), extension.
# Now, push some segments from "part2" into "part1" like this:
# https # part1
# domain1/b # part2
# domain1/c
# domain2/a
# domain2/c
# ->
# https/domain1 # part1
# b # part2
# c
# https/domain2 # part1
# a # part2
# c
# This helps reduce the lockfile size because a Gradle project will usually use lots
# of files from a single Maven repo
data = new_data
changed = True
while changed:
changed = False
new_data = {}
for part1, info1 in data.items():
starts: Set[str] = set()
# by how many bytes the file size will be increased (roughly)
lose = 0
# by how many bytes the file size will be reduced (roughly)
win = 0
# how many different initial part2 segments there are
count = 0
for part2, info2 in info1.items():
if "/" not in part2:
# can't push a segment from part2 into part1
count = 0
break
st = part2.split("/", 1)[0]
if st not in starts:
lose += len(st) + 1
count += 1
starts.add(st)
win += len(st) + 1
if count == 0:
new_data[part1] = info1
continue
# only allow pushing part2 segments into path1 if *either*:
# - the domain isn't yet part of part1
# - the initial part2 segment is always the same
if count != 1 and "." in part1:
new_data[part1] = info1
continue
# some heuristics that may or may not work well (originally this was
# used when the above if wasn't here, but perhaps it's useless now)
lose += (count - 1) * max(0, len(part1) - 4)
if win > lose or ("." not in part1 and win >= lose):
changed = True
for part2, info2 in info1.items():
st, part3 = part2.split("/", 1)
new_part1 = part1 + "/" + st
if new_part1 not in new_data.keys():
new_data[new_part1] = {}
new_data[new_part1][part3] = info2
else:
new_data[part1] = info1
data = new_data
new_data["!comment"] = "This is a nixpkgs Gradle dependency lockfile. For more details, refer to the Gradle section in the nixpkgs manual." # type: ignore
new_data["!version"] = 1 # type: ignore
with open(sys.argv[2], "wt") as f:
json.dump(new_data, f, sort_keys=True, indent=1)
f.write("\n")

40
pkgs/development/tools/build-managers/gradle/default.nix
View File

@ -130,6 +130,7 @@ rec {
'';
};
};
passthru.jdk = defaultJava;
meta = with lib; {
inherit platforms;
@ -179,4 +180,43 @@ rec {
hash = "sha256-PiQCKFON6fGHcqV06ZoLqVnoPW7zUQFDgazZYxeBOJo=";
defaultJava = jdk11;
};
wrapGradle = {
lib, callPackage, mitm-cache, substituteAll, symlinkJoin, concatTextFile, makeSetupHook
}:
gradle-unwrapped:
lib.makeOverridable (args:
let
gradle = gradle-unwrapped.override args;
in symlinkJoin {
name = "gradle-${gradle.version}";
paths = [
(makeSetupHook { name = "gradle-setup-hook"; } (concatTextFile {
name = "setup-hook.sh";
files = [
(mitm-cache.setupHook)
(substituteAll {
src = ./setup-hook.sh;
# jdk used for keytool
inherit (gradle) jdk;
init_script = ./init-build.gradle;
})
];
}))
gradle
mitm-cache
];
passthru = {
fetchDeps = callPackage ./fetch-deps.nix { inherit mitm-cache; };
inherit (gradle) jdk;
};
meta = gradle.meta // {
# prefer normal gradle/mitm-cache over this wrapper, this wrapper only provides the setup hook
# and passthru
priority = (gradle.meta.priority or 0) + 1;
};
}) { };
}

222
pkgs/development/tools/build-managers/gradle/fetch-deps.nix Normal file
View File

@ -0,0 +1,222 @@
{ mitm-cache
, lib
, pkgs
, stdenv
, callPackage
}:
let
getPkg = attrPath:
lib.getAttrFromPath
(lib.splitString "." (toString attrPath))
pkgs;
in
# the derivation to fetch/update deps for
{ pkg ? getPkg attrPath
, pname ? null
, attrPath ? pname
# bwrap flags for the update script (this will be put in bash as-is)
# this is relevant for downstream users
, bwrapFlags ? "--ro-bind \"$PWD\" \"$PWD\""
# deps path (relative to the package directory, or absolute)
, data
# redirect stdout to stderr to allow the update script to be used with update script combinators
, silent ? true
, useBwrap ? stdenv.isLinux
} @ attrs:
let
data' = builtins.removeAttrs
(if builtins.isPath data then lib.importJSON data
else if builtins.isString data then lib.importJSON "${dirOf pkg.meta.position}/${data}"
else data)
[ "!comment" "!version" ];
parseArtifactUrl = url: let
extension = lib.last (lib.splitString "." url);
splitUrl = lib.splitString "/" url;
artifactId = builtins.elemAt splitUrl (builtins.length splitUrl - 3);
baseVer = builtins.elemAt splitUrl (builtins.length splitUrl - 2);
filename = builtins.elemAt splitUrl (builtins.length splitUrl - 1);
filenameNoExt = lib.removeSuffix ".${extension}" filename;
verCls = lib.removePrefix "${artifactId}-" filenameNoExt;
in rec {
inherit artifactId baseVer filename extension;
isSnapshot = lib.hasSuffix "-SNAPSHOT" baseVer;
version =
if isSnapshot && !lib.hasPrefix "SNAPSHOT" verCls
then builtins.concatStringsSep "-" (lib.take 3 (lib.splitString "-" verCls))
else baseVer;
classifier =
if verCls == version then null
else lib.removePrefix "${version}-" verCls;
# for snapshots
timestamp = builtins.elemAt (lib.splitString "-" version) 1;
buildNum = builtins.elemAt (lib.splitString "-" version) 2;
};
parseMetadataUrl = url: let
xmlBase = lib.removeSuffix "/maven-metadata.xml" url;
vMeta = lib.hasSuffix "-SNAPSHOT" xmlBase;
splitBase = lib.splitString "/" xmlBase;
in
if vMeta then {
vMeta = true;
baseVer = builtins.elemAt splitBase (builtins.length splitBase - 1);
artifactId = builtins.elemAt splitBase (builtins.length splitBase - 2);
} else {
vMeta = false;
baseVer = null;
artifactId = builtins.elemAt splitBase (builtins.length splitBase - 1);
};
extractHashArtifact = afterHash: let
nameVer = builtins.match "([^/]*)/([^/]*)(/SNAPSHOT)?(/.*)?" afterHash;
artifactId = builtins.elemAt nameVer 0;
version = builtins.elemAt nameVer 1;
isSnapshot = builtins.elemAt nameVer 2 != null;
cls = builtins.elemAt nameVer 3;
in rec {
inherit artifactId version isSnapshot;
baseVer =
if !isSnapshot then version
else builtins.head (builtins.match "(.*)-([^-]*)-([^-]*)" version) + "-SNAPSHOT";
classifier =
if cls == null then null
else lib.removePrefix "/" cls;
clsSuf =
if classifier == null then ""
else "-${classifier}";
};
# replace base#name/ver with base/name/ver/name-ver
decompressNameVer = prefix: let
splitHash = lib.splitString "#" (builtins.concatStringsSep "/" prefix);
inherit (extractHashArtifact (lib.last splitHash)) artifactId baseVer version clsSuf;
in
if builtins.length splitHash == 1 then builtins.head splitHash
else builtins.concatStringsSep "/${artifactId}/${baseVer}/" (lib.init splitHash ++ [ "${artifactId}-${version}${clsSuf}" ]);
# `visit` all elements in attrs and merge into a set
# attrs will be passed as parent1, parent1 will be passed as parent2
visitAttrs = parent1: prefix: attrs:
builtins.foldl'
(a: b: a // b)
{}
(lib.mapAttrsToList (visit parent1 attrs prefix) attrs);
# convert a compressed deps.json into an uncompressed json used for mitm-cache.fetch
visit = parent2: parent1: prefix: k: v:
# groupId being present means this is a metadata xml "leaf" and we shouldn't descend further
if builtins.isAttrs v && !v?groupId
then visitAttrs parent1 (prefix ++ [k]) v
else let
url = "${decompressNameVer prefix}.${k}";
in {
${url} =
if builtins.isString v then { hash = v; }
else {
text = let
xmlBase = lib.removeSuffix "/maven-metadata.xml" url;
meta = parseMetadataUrl url // v;
inherit (meta) groupId vMeta artifactId baseVer;
fileList = builtins.filter (x: lib.hasPrefix xmlBase x && x != url) (builtins.attrNames finalData);
jarPomList = map parseArtifactUrl fileList;
sortedJarPomList =
lib.sort
(a: b: lib.splitVersion a.version < lib.splitVersion b.version)
jarPomList;
uniqueVersionFiles =
builtins.map ({ i, x }: x)
(builtins.filter ({ i, x }: i == 0 || (builtins.elemAt sortedJarPomList (i - 1)).version != x.version)
(lib.imap0 (i: x: { inherit i x; }) sortedJarPomList));
uniqueVersions' = map (x: x.version) uniqueVersionFiles;
releaseVersions = map (x: x.version) (builtins.filter (x: !x.isSnapshot) uniqueVersionFiles);
latestVer = v.latest or v.release or (lib.last uniqueVersions');
releaseVer = v.release or (lib.last releaseVersions);
# The very latest version isn't necessarily used by Gradle, so it may not be present in the MITM data.
# In order to generate better metadata xml, if the latest version is known but wasn't fetched by Gradle,
# add it anyway.
uniqueVersions =
uniqueVersions'
++ lib.optional (!builtins.elem releaseVer uniqueVersions') releaseVer
++ lib.optional (!builtins.elem latestVer uniqueVersions' && releaseVer != latestVer) latestVer;
lastUpdated = v.lastUpdated or
(if vMeta then builtins.replaceStrings ["."] [""] snapshotTs
else "20240101123456");
# the following are only used for snapshots
snapshotTsAndNum = lib.splitString "-" latestVer;
snapshotTs = builtins.elemAt snapshotTsAndNum 1;
snapshotNum = lib.last snapshotTsAndNum;
indent = x: s: builtins.concatStringsSep "\n" (map (s: x + s) (lib.splitString "\n" s));
containsSpecialXmlChars = s: builtins.match ''.*[<>"'&].*'' s != null;
in
# make sure all user-provided data is safe
assert lib.hasInfix "${builtins.replaceStrings ["."] ["/"] groupId}/${artifactId}" url;
assert !containsSpecialXmlChars groupId;
assert !containsSpecialXmlChars lastUpdated;
if vMeta then ''
<?xml version="1.0" encoding="UTF-8"?>
<metadata modelVersion="1.1.0">
<groupId>${groupId}</groupId>
<artifactId>${artifactId}</artifactId>
<version>${baseVer}</version>
<versioning>
<snapshot>
<timestamp>${snapshotTs}</timestamp>
<buildNumber>${snapshotNum}</buildNumber>
</snapshot>
<lastUpdated>${lastUpdated}</lastUpdated>
<snapshotVersions>
${builtins.concatStringsSep "\n" (map (x: indent " " ''
<snapshotVersion>${
lib.optionalString
(x.classifier != null)
"\n <classifier>${x.classifier}</classifier>"
}
<extension>${x.extension}</extension>
<value>${x.version}</value>
<updated>${builtins.replaceStrings ["."] [""] x.timestamp}</updated>
</snapshotVersion>'') sortedJarPomList)}
</snapshotVersions>
</versioning>
</metadata>
''
else
assert !containsSpecialXmlChars latestVer;
assert !containsSpecialXmlChars releaseVer;
''
<?xml version="1.0" encoding="UTF-8"?>
<metadata modelVersion="1.1.0">
<groupId>${groupId}</groupId>
<artifactId>${artifactId}</artifactId>
<versioning>
<latest>${latestVer}</latest>
<release>${releaseVer}</release>
<versions>
${builtins.concatStringsSep "\n" (map (x: " <version>${x}</version>") uniqueVersions)}
</versions>
<lastUpdated>${lastUpdated}</lastUpdated>
</versioning>
</metadata>
'';
};
};
finalData = visitAttrs {} [] data';
in
mitm-cache.fetch {
name = "${pkg.pname or pkg.name}-deps";
data = finalData // { "!version" = 1; };
passthru = lib.optionalAttrs (!builtins.isAttrs data) {
updateScript = callPackage ./update-deps.nix { } {
inherit pkg pname attrPath bwrapFlags data silent useBwrap;
};
};
}

8
pkgs/development/tools/build-managers/gradle/init-build.gradle Normal file
View File

@ -0,0 +1,8 @@
gradle.projectsLoaded {
rootProject.allprojects {
tasks.withType(AbstractArchiveTask) {
preserveFileTimestamps = false
reproducibleFileOrder = true
}
}
}

10
pkgs/development/tools/build-managers/gradle/init-deps.gradle Normal file
View File

@ -0,0 +1,10 @@
gradle.projectsLoaded {
rootProject.allprojects {
task nixDownloadDeps {
doLast {
configurations.findAll{it.canBeResolved}.each{it.resolve()}
buildscript.configurations.findAll{it.canBeResolved}.each{it.resolve()}
}
}
}
}

70
pkgs/development/tools/build-managers/gradle/setup-hook.sh Normal file
View File

@ -0,0 +1,70 @@
gradleConfigureHook() {
if [ -z "${GRADLE_USER_HOME-}" ]; then
GRADLE_USER_HOME="$(mktemp -d)"
fi
export GRADLE_USER_HOME
export TERM=dumb
gradleFlagsArray+=(--no-daemon --console plain --init-script "${gradleInitScript:-@init_script@}")
if [ -n "${MITM_CACHE_CA-}" ]; then
if [ -z "${MITM_CACHE_KEYSTORE-}" ]; then
MITM_CACHE_KEYSTORE="$MITM_CACHE_CERT_DIR/keystore"
MITM_CACHE_KS_PWD="$(head -c10 /dev/random | base32)"
echo y | @jdk@/bin/keytool -importcert -file "$MITM_CACHE_CA" -alias alias -keystore "$MITM_CACHE_KEYSTORE" -storepass "$MITM_CACHE_KS_PWD"
fi
gradleFlagsArray+=(-Dhttp.proxyHost="$MITM_CACHE_HOST" -Dhttp.proxyPort="$MITM_CACHE_PORT")
gradleFlagsArray+=(-Dhttps.proxyHost="$MITM_CACHE_HOST" -Dhttps.proxyPort="$MITM_CACHE_PORT")
gradleFlagsArray+=(-Djavax.net.ssl.trustStore="$MITM_CACHE_KEYSTORE" -Djavax.net.ssl.trustStorePassword="$MITM_CACHE_KS_PWD")
else
gradleFlagsArray+=(--offline)
fi
if ! [[ -v enableParallelBuilding ]]; then
enableParallelBuilding=1
fi
if ! [[ -v enableParallelChecking ]]; then
enableParallelChecking=1
fi
if ! [[ -v enableParallelUpdating ]]; then
enableParallelUpdating=1
fi
}
gradle() {
command gradle $gradleFlags "${gradleFlagsArray[@]}" "$@"
}
gradleBuildPhase() {
runHook preBuild
gradle ${enableParallelBuilding:+--parallel} ${gradleBuildTask:-assemble}
runHook postBuild
}
gradleCheckPhase() {
runHook preCheck
gradle ${enableParallelChecking:+--parallel} ${gradleCheckTask:-test}
runHook postCheck
}
gradleUpdateScript() {
runHook preBuild
runHook preGradleUpdate
gradle ${enableParallelUpdating:+--parallel} ${gradleUpdateTask:-nixDownloadDeps}
runHook postGradleUpdate
}
if [ -z "${dontUseGradleConfigure-}" ]; then
preConfigureHooks+=(gradleConfigureHook)
fi
if [ -z "${dontUseGradleBuild-}" ] && [ -z "${buildPhase-}" ]; then
buildPhase=gradleBuildPhase
fi
if [ -z "${dontUseGradleCheck-}" ] && [ -z "${checkPhase-}" ]; then
checkPhase=gradleCheckPhase
fi

122
pkgs/development/tools/build-managers/gradle/update-deps.nix Normal file
View File

@ -0,0 +1,122 @@
{ lib
, runtimeShell
, srcOnly
, writeTextFile
, writeShellScript
, path
, bubblewrap
, coreutils
, curl
, jq
, mitm-cache
, nix
, openssl
, procps
, python3
}:
lib.makeOverridable
({ pkg, pname, attrPath, bwrapFlags, data, silent, useBwrap }:
let
keep = [ "MITM_CACHE_HOST" "MITM_CACHE_PORT" "MITM_CACHE_ADDRESS" "MITM_CACHE_CA" "MITM_CACHE_CERT_DIR" ];
gradleScript = writeShellScript "gradle-commands.sh" ''
set -eo pipefail
export http_proxy="$MITM_CACHE_ADDRESS"
export https_proxy="$MITM_CACHE_ADDRESS"
export SSL_CERT_FILE="$MITM_CACHE_CA"
export NIX_SSL_CERT_FILE="$MITM_CACHE_CA"
export GRADLE_USER_HOME="$(${coreutils}/bin/mktemp -d)"
export IN_GRADLE_UPDATE_DEPS=1
trap "${coreutils}/bin/rm -rf '$GRADLE_USER_HOME'" SIGINT SIGTERM ERR EXIT
cd "$(${coreutils}/bin/mktemp -d)"
${coreutils}/bin/mkdir out
export out="$PWD/out"
trap "${coreutils}/bin/rm -rf '$PWD'" SIGINT SIGTERM ERR EXIT
source "$stdenv/setup"
phases="''${prePhases[*]:-} unpackPhase patchPhase ''${preConfigurePhases[*]:-} configurePhase gradleUpdateScript" genericBuild
'';
source = srcOnly (pkg.overrideAttrs (old: {
mitmCache = "";
gradleInitScript = ./init-deps.gradle;
}));
sourceDrvPath = builtins.unsafeDiscardOutputDependency source.drvPath;
nixShellKeep = lib.concatMapStringsSep " " (x: "--keep ${x}") keep;
in
writeTextFile {
name = "fetch-deps.sh";
executable = true;
# see pkgs/common-updater/combinators.nix
derivationArgs.passthru =
{ supportedFeatures = lib.optional silent "silent"; }
// lib.optionalAttrs (attrPath != null) { inherit attrPath; };
text = ''
#!${runtimeShell}
set -eo pipefail
export PATH="${lib.makeBinPath [
bubblewrap coreutils curl jq mitm-cache openssl
procps python3.pkgs.ephemeral-port-reserve
]}:$PATH"
outPath="${
# if this is an absolute path in nix store, use path relative to the store path
if lib.hasPrefix "${builtins.storeDir}/" (toString data)
then builtins.concatStringsSep "/" (lib.drop 1 (lib.splitString "/" (lib.removePrefix "${builtins.storeDir}/" (toString data))))
# if this is an absolute path anywhere else, just use that path
else if lib.hasPrefix "/" (toString data)
then toString data
# otherwise, use a path relative to the package
else "${dirOf pkg.meta.position}/${data}"
}"
pushd "$(mktemp -d)" >/dev/null
MITM_CACHE_DIR="$PWD"
trap "rm -rf '$MITM_CACHE_DIR'" SIGINT SIGTERM ERR EXIT
openssl genrsa -out ca.key 2048
openssl req -x509 -new -nodes -key ca.key -sha256 -days 1 -out ca.cer -subj "/C=AL/ST=a/L=a/O=a/OU=a/CN=example.org"
export MITM_CACHE_HOST=127.0.0.1
export MITM_CACHE_PORT="''${mitmCachePort:-$(ephemeral-port-reserve "$MITM_CACHE_HOST")}"
export MITM_CACHE_ADDRESS="$MITM_CACHE_HOST:$MITM_CACHE_PORT"
# forget all redirects - this makes the lockfiles predictable
# not only does this strip CDN URLs, but it also improves security - since the redirects aren't
# stored in the lockfile, a malicious actor can't change the redirect URL stored in the lockfile
mitm-cache \
-l"$MITM_CACHE_ADDRESS" \
record \
--reject '\.(md5|sha(1|256|512:?):?)$' \
--forget-redirects-from '.*' \
--record-text '/maven-metadata\.xml$' >/dev/null 2>/dev/null &
MITM_CACHE_PID="$!"
# wait for mitm-cache to fully start
for i in {0..20}; do
ps -p "$MITM_CACHE_PID" >/dev/null || (echo "Failed to start mitm-cache" && exit 1)
curl -so/dev/null "$MITM_CACHE_ADDRESS" && break
[[ "$i" -eq 20 ]] && (echo "Failed to start mitm-cache" && exit 1)
sleep 0.5
done
trap "kill '$MITM_CACHE_PID'" SIGINT SIGTERM ERR EXIT
export MITM_CACHE_CERT_DIR="$PWD"
export MITM_CACHE_CA="$MITM_CACHE_CERT_DIR/ca.cer"
popd >/dev/null
useBwrap="''${USE_BWRAP:-${toString useBwrap}}"
if [ -n "$useBwrap" ]; then
# bwrap isn't necessary, it's only used to prevent messy build scripts from touching ~
bwrap \
--unshare-all --share-net --clearenv --chdir / --setenv HOME /homeless-shelter \
--tmpfs /home --bind /tmp /tmp --ro-bind /nix /nix --ro-bind /run /run --proc /proc --dev /dev \
--ro-bind ${toString path} ${toString path} --bind "$MITM_CACHE_CERT_DIR" "$MITM_CACHE_CERT_DIR" \
${builtins.concatStringsSep " " (map (x: "--setenv ${x} \"\$${x}\"") keep)} \
--setenv NIX_BUILD_SHELL bash ${bwrapFlags} ''${BWRAP_FLAGS:-} \
-- ${nix}/bin/nix-shell --pure --run ${gradleScript} ${nixShellKeep} ${sourceDrvPath}
else
NIX_BUILD_SHELL=bash nix-shell --pure --run ${gradleScript} ${nixShellKeep} ${sourceDrvPath}
fi${lib.optionalString silent " >&2"}
kill -s SIGINT "$MITM_CACHE_PID"
for i in {0..20}; do
# check for valid json
if jq -e 1 "$MITM_CACHE_DIR/out.json" >/dev/null 2>/dev/null; then
exec ${python3.interpreter} ${./compress-deps-json.py} "$MITM_CACHE_DIR/out.json" "$outPath"
fi
sleep 1
done
exit 1
'';
})

15
pkgs/top-level/all-packages.nix
View File

@ -18575,10 +18575,17 @@ with pkgs;
inherit jdk11 jdk17 jdk21;
};
gradleGen = gradle-packages.gen;
gradle_6 = callPackage gradle-packages.gradle_6 { };
gradle_7 = callPackage gradle-packages.gradle_7 { };
gradle_8 = callPackage gradle-packages.gradle_8 { };
gradle = gradle_8;
wrapGradle = callPackage gradle-packages.wrapGradle { };
gradle_6-unwrapped = callPackage gradle-packages.gradle_6 { };
gradle_7-unwrapped = callPackage gradle-packages.gradle_7 { };
gradle_8-unwrapped = callPackage gradle-packages.gradle_8 { };
gradle-unwrapped = gradle_8-unwrapped;
gradle_6 = wrapGradle gradle_6-unwrapped;
gradle_7 = wrapGradle gradle_7-unwrapped;
gradle_8 = wrapGradle gradle_8-unwrapped;
gradle = wrapGradle gradle-unwrapped;
grcov = callPackage ../development/tools/misc/grcov { };