By overriding each dependency on every level of the dependency tree we
are creating a lot of unnecessary instances of the same derivation
Looking at the output size of `nix-instantiate --trace-function-calls
-vvvv …` and the execution time I got about a 10x improvement after
applying this change.
It was probably good intentions that lead to these overrides but in
practice no tooling (that I know of) really needs this. `carnix` and
`crate2nix` are fine without those overrides. Furthermore I believe that
it is the job of the tooling around `buildRustCrate` to provide a
coherent set of overrides. By not enforcing all of the overrides, debug
flags, verbosity, … to be the same throughout the closure we also allow
consumers to override specific aspects of the crates. Some (older?)
crates might need different `crateOverrides` then newer crates with the
same name. Currently such situations can not (easily) be implemented
with the override in-place.
This has several advantages:
1. It takes up less space on disk in-between builds in the nix store.
2. It uses less space in the binary cache for vendor derivation packages.
3. It uses less network traffic downloading from the binary cache.
4. It plays nicely with hashed mirrors like tarballs.nixos.org, which only
substitute --flat hashes on single files (not recursive directory hashes).
5. It's consistent with how simple `fetchurl` src derivations work.
6. It provides a stronger abstraction between input src-package and output
package, e.g., it's harder to accidentally depend on the src derivation at
runtime by referencing something like `${src}/etc/index.html`. Likewise, in
the store it's harder to get confused with something that is just there as a
build-time dependency vs. a runtime dependency, since the build-time
src dependencies are tarred up.
Disadvantages are:
1. It takes slightly longer to untar at the start of a build.
As currently implemented, this attaches the compacted vendor.tar.gz feature as a
rider on `verifyCargoDeps`, since both of them are relatively newly implemented
behavior that change the `cargoSha256`.
If this PR is accepted, I will push forward the remaining rust packages with a
series of treewide PRs to update the `cargoSha256`s.
Previously I did use `runCommand` to do the same. Using
releaseTools.aggregate seems a lot saner and we might get nicer hydra
output of the tests that are failing.
It used to be the case (ref missing) that cargo did treat
`src/$libName.rs` as an alternative to `src/lib.rs` when the latter
wasn't present. Recently I failed to reproduce that with vanilla cargo
and it started to cause pain with some crates of the form:
some_crate/
`- src
`- main.rs
`- some_crate.rs
We would build `src/some_crate.rs` and thing it is a library while that
might not be the actual case. This crate is a valid `bin` crate not a
`lib` crate as far as I can tell from the samples I took.
I removed support for the previously required heuristic and commented
out the test cases in case we will need them again. We could crawl in
the Git history but chances are that the next person looking into this
doesn't know about the history.
When this fails, the user may want to copy-paste the path to the "bad"
Cargo.lock file to inspect. The trailing `.` on `$cargoDeps.` gets caught in
most terminal copy-pastes. Since half the lines already don't have it, this
removes it from all of them for consistent output.
This helps us instruct rustc to build tests instead of binaries. The
actual build will then ONLY produce test executables. This is a first
step towards having rust crate tests within nixpkgs.
We default back to only a single output in test cases since that is the
only reasonable thing to do here.
Producing libraries or binaries in addition to tests would theoretically
be feasible but usually generates different dependency trees. It is very
common to have some libraries in `[dev-depdendencies]` within Cargo.toml
just for your tests. To not start mixing things up going with a
dedicated derivation for the test build sounds like the best choice for
now.
To use this you must provide a proper test dependency chain to
`buildRustCrate` (as you would usually do with your non-test inputs).
And then set the `buildTests` attribute to `true`. The derivation will
then contain all tests that were built in `$out/tests`. All common test
patterns and directories should be supported and tested by this change.
Below is an example how you would run a single test from the derivation.
This commit contains some more examples in the `buildRustCrateTests`
attribute set that might be helpful.
```
let
drv = buildRustCrate {
…
buildTests true;
};
in runCommand "test-my-crate" {} ''
touch $out
exec ${drv}/tests/my-test
''
```
While unifying most of the lib function calls I accidentially changed
the filterSource functions as well. Since there were no tests I ended
up forgetting about this case (even thought I ran into it…).
Most stdenv wrappers already work like this -- it allows greater
customisation. We just have to be careful to remove arguments we're
using that shouldn't be passed to stdenv. I've been conservative
here, because fetchcargo checksums shouldn't change lightly.
The previous lines were only different in the kind of dependencies but
otherwise exactly the same. It makes the entire thing a bit more
readable by moving this into a function that takes care of this.
We can get rid of a bunch of workarounds that were in the build script
before by just passing on the `crateBin` attribute.
Before we converted the list of attributes to a string only to convert
it back in bash during the build phase. We can do the entire looping
through builds in Nix and thus need no conversion and parsing of
attributes over and over again.
The big part that still remains bash is the heuristic that cargo
introduced and that we can't do at eval time.
That code had been in the derivation for a while but no explanation was
given why that is needed. It might be helpful to our future selfs to
document why things are done the way they are.
The expression is already long and confusing enough without the color
stuff sprinkled in. Moving it to a dedicated file makes sense.
I switched a bit of the color support code to pure Nix since there
wasn't much point in doing that in bash while we can just do it in Nix.
We can just use `lib` instead of `builtins` in all cases but the
`hashString` case. Also changed a few lines to make use of some optional
helpers from lib.
This cuts down the dependency tree on some rust builds where a crate not
just exposes a binary but also a library. `$out/lib` contained a bunch
of extra support files that among other information carry linker flags
(including the full path to link-time dependencies). Worst case this led
to some binary outputs depending on the full build closure of rust
crates.
Moving all the `$out/lib` files to `$lib/lib` solves this nicely.
`lib` might be a bit weird here as they are most of the time just rlib
files (rust libraries). Those are essential only required during
compilation but they can also be shared objects (like with traditional
C-style packages). Which is why I went with `lib` for the new output.
One of the caveats we are running into here is that we do not (always)
know ahead of time of a crate produces just a library or just a binary.
Cargo allows for some ambiguity regarding whether or not a crate
provides one, two, … binaries and libraries as it's outputs. Ideally we
would be able to rely on the `crateType` entirely but so far that isn't
the case. More work on that area might show how difficult that actually
is.
One issue with cargoSha256 is that it's hard to detect when it needs to
be updated or not. It's possible to upgrade a package and forget to
update cargoSha256 and run with old versions of the program or
libraries.
This commit introduces `verifyCargoDeps` which, when enabled, will check
that the Cargo.lock is not out of date in the cargoDeps by comparing it
with the package source.
Quoting from the splitString docstring:
NOTE: this function is not performant and should never be used.
This replaces trivial uses of splitString for splitting version
strings with the (potentially builtin) splitVersion.
Before this change, buildRustCrate always called rustc with
--extern libName=[...]libName[...]
However, Cargo permits using a different name under which a dependency
is known to a crate. For example, rand 0.7.0 uses:
[dependencies]
getrandom_package = { version = "0.1.1", package = "getrandom", optional = true }
Which introduces the getrandom dependency such that it is known as
getrandom_package to the rand crate. In this case, the correct extern
flag is of the form
--extern getrandom_package=[...]getrandom[...]
which is currently not supported. In order to support such cases, this
change introduces a crateRenames argument to buildRustCrate. This
argument is an attribute set of dependencies that should be renamed. In
this case, crateRenames would be:
{
"getrandom" = "getrandom_package";
}
The extern options are then built such that if the libName occurs as
an attribute in this set, it value will be used as the local
name. Otherwise libName will be used as before.
buildRustCrate has a handy `include` helper, that only imports those whitelisted
files and folders to the store.
However, the function's matching logic is broken and includes all files,
regardless of whether or not they're whitelisted, as long as the whitelist
contains at least one name (regardless of whether that name exists). This is
because it doesn't take into account that
`lib.strings.removePrefix "foo" "bar" == "bar"` (that is, paths that don't match
the prefix are passed straight through).
While it is not obvious from the source, cargo sets CARGO_MANIFEST_DIR to an absolute directory. This let to a build problem with the popular "tera" crate using the "pest" crate.
## Cargo details
The variable is set here:
f7c91ba622/src/cargo/core/compiler/compilation.rs (L229)
and computed from the `manifest_path`:
f7c91ba622/src/cargo/core/package.rs (L163)
The manifest path is also exported via `cargo metadata` where you can see that it is absolute.
In combination with carnix we can now build crates that require a
specific edition of rust features. A few crates started requiring that
already and having this in nixpkgs is just logical.
This commit adds test based on real-world crates (brotli).
There were a few more edge cases that were missing beforehand. Also it
turned out that we can get rid of the `finalBins` list since that will
now be handled during runtime.
The build expression got quiet large over time and to make it a bit
easier to grasp the different scripts involved in the build are now
separated from the nix file.
Cargo has a few odd (old) ways of picking source files if the `bin.path`
attribute isn't given in the Cargo.toml. This commit adds support for
some of those. The previous behaviour always defaulted to `src/main.rs`
which was not always the right choice.
Since there is look-ahead into the unpacked sources before running the
actual builder the path selection logic has to be embedded within the
build script.
`buildRustCrate` currently supports two ways of running building
binaries when processing a crate:
- Explicit definition of all the binaries (& optionally the paths to
their respective `main.rs`) and,
- if not binary was explictly configured all files matching the patterns
`src/main.rs`, `src/bin/*.rs`.
When the explicit list is given without path information paths are now
being picked from a list of candidates. The first match wins. The order
is the same as within the cargo compatibility code.
If the crate does not provide any libraries the path `src/{bin_name}.rs`
is also considered.
All underscores within the binary names are translated into dashes (`-`)
before the lookups are made. This seems to be a common convention.
By setting useRealVendorConfig explicitly to true, the actual (slightly
modified) config generated by cargo-vendor is used.
This solves a problem, where the static vendor config in
pkgs/build-support/rust/default.nix would not sufficiently replace all
crates Cargo is looking for.
As useRealVendorConfig (and writeVendorConfig in fetchcargo) default to
false, there should be no breakage in existing cargoSha256 hashes.
Nethertheless, imho using this new feature should become standard. A
possible deprecation path could be:
- introduce this patch
- set useRealVendorConfig explicitly to false whereever cargoSha256 is
set but migration is not wanted yet.
- after some time, let writeVendorConfig default to true
- when useRealVendorConfig is true everywhere cargoSha256 is set and
enough time is passed, `assert cargoVendorDir == null ->
useRealVendorConfig;`, remove old behaviour
- after some time, remove all appearences of useRealVendorConfig and the
parameter itself
This makes the command ‘nix-env -qa -f. --arg config '{skipAliases =
true;}'’ work in Nixpkgs.
Misc...
- qtikz: use libsForQt5.callPackage
This ensures we get the right poppler.
- rewrites:
docbook5_xsl -> docbook_xsl_ns
docbook_xml_xslt -> docbook_xsl
diffpdf: fixup
Setting the hash to null is a convenient way to bypass the hash check
while developing. It looks like the ability to do this was inadvertently
removed while adding vendor directory support.
This still checks that the user is explicitly setting the value but
allows null as a valid option.
Previously, cargoUpdateHook was meaningful as it was used
in
[`cargo-fetch-deps`](19d3cf81d3/pkgs/build-support/rust/fetch-cargo-deps (L71)).
However, this entire file was removed in
5f8cf0048e. As far as I can
tell, nothing in the code is using it, but it is still
being passed around:
https://github.com/NixOS/nixpkgs/search?q=cargoUpdateHook&type=Code&utf8=%E2%9C%93
There are, however, legitimate use cases for it. For example,
in some software, some dependencies are not locked in Cargo.toml
and this causes Cargo to try fetching another version of them.
This doesn't work well with vendoring crates.
This hook allows to inject patching or whatever necessary workarounds
in the crate vendoring process. I suppose that's what it was for
in there in the first place.
This patch restores this hook and makes it usable again.
`find -executable` finds everything with the executable bit set,
including directories. Thats not harmful in this scenario as `cp` won't
copy those directories, but it does result in a few warning messages.
