Merge pull request #268408 from tweag/fileset.reference

This commit is contained in:
Valentin Gagarin 2023-11-20 09:05:43 +01:00 committed by GitHub
commit 4ce484e288
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23

View File

@ -3,10 +3,52 @@
[]{#sec-fileset} []{#sec-fileset}
The [`lib.fileset`](#sec-functions-library-fileset) library allows you to work with _file sets_. The [`lib.fileset`](#sec-functions-library-fileset) library allows you to work with _file sets_.
A file set is a mathematical set of local files that can be added to the Nix store for use in Nix derivations. A file set is a (mathematical) set of local files that can be added to the Nix store for use in Nix derivations.
File sets are easy and safe to use, providing obvious and composable semantics with good error messages to prevent mistakes. File sets are easy and safe to use, providing obvious and composable semantics with good error messages to prevent mistakes.
See the [function reference](#sec-functions-library-fileset) for function-specific documentation. ## Overview {#sec-fileset-overview}
Basics:
- [Implicit coercion from paths to file sets](#sec-fileset-path-coercion)
- [`lib.fileset.trace`](#function-library-lib.fileset.trace)/[`lib.fileset.traceVal`](#function-library-lib.fileset.trace):
Pretty-print file sets for debugging.
- [`lib.fileset.toSource`](#function-library-lib.fileset.toSource):
Add files in file sets to the store to use as derivation sources.
Combinators:
- [`lib.fileset.union`](#function-library-lib.fileset.union)/[`lib.fileset.unions`](#function-library-lib.fileset.unions):
Create a larger file set from all the files in multiple file sets.
- [`lib.fileset.intersection`](#function-library-lib.fileset.intersection):
Create a smaller file set from only the files in both file sets.
- [`lib.fileset.difference`](#function-library-lib.fileset.difference):
Create a smaller file set containing all files that are in one file set, but not another one.
Filtering:
- [`lib.fileset.fileFilter`](#function-library-lib.fileset.fileFilter):
Create a file set from all files that satisisfy a predicate in a directory.
Utilities:
- [`lib.fileset.fromSource`](#function-library-lib.fileset.fromSource):
Create a file set from a `lib.sources`-based value.
- [`lib.fileset.gitTracked`](#function-library-lib.fileset.gitTracked)/[`lib.fileset.gitTrackedWith`](#function-library-lib.fileset.gitTrackedWith):
Create a file set from all tracked files in a local Git repository.
If you need more file set functions,
see [this issue](https://github.com/NixOS/nixpkgs/issues/266356) to request it.
## Implicit coercion from paths to file sets {#sec-fileset-path-coercion} ## Implicit coercion from paths to file sets {#sec-fileset-path-coercion}
@ -106,6 +148,96 @@ let
in { in {
/*
Incrementally evaluate and trace a file set in a pretty way.
This function is only intended for debugging purposes.
The exact tracing format is unspecified and may change.
This function takes a final argument to return.
In comparison, [`traceVal`](#function-library-lib.fileset.traceVal) returns
the given file set argument.
This variant is useful for tracing file sets in the Nix repl.
Type:
trace :: FileSet -> Any -> Any
Example:
trace (unions [ ./Makefile ./src ./tests/run.sh ]) null
=>
trace: /home/user/src/myProject
trace: - Makefile (regular)
trace: - src (all files in directory)
trace: - tests
trace: - run.sh (regular)
null
*/
trace =
/*
The file set to trace.
This argument can also be a path,
which gets [implicitly coerced to a file set](#sec-fileset-path-coercion).
*/
fileset:
let
# "fileset" would be a better name, but that would clash with the argument name,
# and we cannot change that because of https://github.com/nix-community/nixdoc/issues/76
actualFileset = _coerce "lib.fileset.trace: Argument" fileset;
in
seq
(_printFileset actualFileset)
(x: x);
/*
Incrementally evaluate and trace a file set in a pretty way.
This function is only intended for debugging purposes.
The exact tracing format is unspecified and may change.
This function returns the given file set.
In comparison, [`trace`](#function-library-lib.fileset.trace) takes another argument to return.
This variant is useful for tracing file sets passed as arguments to other functions.
Type:
traceVal :: FileSet -> FileSet
Example:
toSource {
root = ./.;
fileset = traceVal (unions [
./Makefile
./src
./tests/run.sh
]);
}
=>
trace: /home/user/src/myProject
trace: - Makefile (regular)
trace: - src (all files in directory)
trace: - tests
trace: - run.sh (regular)
"/nix/store/...-source"
*/
traceVal =
/*
The file set to trace and return.
This argument can also be a path,
which gets [implicitly coerced to a file set](#sec-fileset-path-coercion).
*/
fileset:
let
# "fileset" would be a better name, but that would clash with the argument name,
# and we cannot change that because of https://github.com/nix-community/nixdoc/issues/76
actualFileset = _coerce "lib.fileset.traceVal: Argument" fileset;
in
seq
(_printFileset actualFileset)
# We could also return the original fileset argument here,
# but that would then duplicate work for consumers of the fileset, because then they have to coerce it again
actualFileset;
/* /*
Add the local files contained in `fileset` to the store as a single [store path](https://nixos.org/manual/nix/stable/glossary#gloss-store-path) rooted at `root`. Add the local files contained in `fileset` to the store as a single [store path](https://nixos.org/manual/nix/stable/glossary#gloss-store-path) rooted at `root`.
@ -250,75 +382,6 @@ in {
filter = sourceFilter; filter = sourceFilter;
}; };
/*
Create a file set with the same files as a `lib.sources`-based value.
This does not import any of the files into the store.
This can be used to gradually migrate from `lib.sources`-based filtering to `lib.fileset`.
A file set can be turned back into a source using [`toSource`](#function-library-lib.fileset.toSource).
:::{.note}
File sets cannot represent empty directories.
Turning the result of this function back into a source using `toSource` will therefore not preserve empty directories.
:::
Type:
fromSource :: SourceLike -> FileSet
Example:
# There's no cleanSource-like function for file sets yet,
# but we can just convert cleanSource to a file set and use it that way
toSource {
root = ./.;
fileset = fromSource (lib.sources.cleanSource ./.);
}
# Keeping a previous sourceByRegex (which could be migrated to `lib.fileset.unions`),
# but removing a subdirectory using file set functions
difference
(fromSource (lib.sources.sourceByRegex ./. [
"^README\.md$"
# This regex includes everything in ./doc
"^doc(/.*)?$"
])
./doc/generated
# Use cleanSource, but limit it to only include ./Makefile and files under ./src
intersection
(fromSource (lib.sources.cleanSource ./.))
(unions [
./Makefile
./src
]);
*/
fromSource = source:
let
# This function uses `._isLibCleanSourceWith`, `.origSrc` and `.filter`,
# which are technically internal to lib.sources,
# but we'll allow this since both libraries are in the same code base
# and this function is a bridge between them.
isFiltered = source ? _isLibCleanSourceWith;
path = if isFiltered then source.origSrc else source;
in
# We can only support sources created from paths
if ! isPath path then
if isStringLike path then
throw ''
lib.fileset.fromSource: The source origin of the argument is a string-like value ("${toString path}"), but it should be a path instead.
Sources created from paths in strings cannot be turned into file sets, use `lib.sources` or derivations instead.''
else
throw ''
lib.fileset.fromSource: The source origin of the argument is of type ${typeOf path}, but it should be a path instead.''
else if ! pathExists path then
throw ''
lib.fileset.fromSource: The source origin (${toString path}) of the argument is a path that does not exist.''
else if isFiltered then
_fromSourceFilter path source.filter
else
# If there's no filter, no need to run the expensive conversion, all subpaths will be included
_singleton path;
/* /*
The file set containing all files that are in either of two given file sets. The file set containing all files that are in either of two given file sets.
This is the same as [`unions`](#function-library-lib.fileset.unions), This is the same as [`unions`](#function-library-lib.fileset.unions),
@ -411,66 +474,6 @@ in {
_unionMany _unionMany
]; ];
/*
Filter a file set to only contain files matching some predicate.
Type:
fileFilter ::
({
name :: String,
type :: String,
...
} -> Bool)
-> Path
-> FileSet
Example:
# Include all regular `default.nix` files in the current directory
fileFilter (file: file.name == "default.nix") ./.
# Include all non-Nix files from the current directory
fileFilter (file: ! hasSuffix ".nix" file.name) ./.
# Include all files that start with a "." in the current directory
fileFilter (file: hasPrefix "." file.name) ./.
# Include all regular files (not symlinks or others) in the current directory
fileFilter (file: file.type == "regular") ./.
*/
fileFilter =
/*
The predicate function to call on all files contained in given file set.
A file is included in the resulting file set if this function returns true for it.
This function is called with an attribute set containing these attributes:
- `name` (String): The name of the file
- `type` (String, one of `"regular"`, `"symlink"` or `"unknown"`): The type of the file.
This matches result of calling [`builtins.readFileType`](https://nixos.org/manual/nix/stable/language/builtins.html#builtins-readFileType) on the file's path.
Other attributes may be added in the future.
*/
predicate:
# The path whose files to filter
path:
if ! isFunction predicate then
throw ''
lib.fileset.fileFilter: First argument is of type ${typeOf predicate}, but it should be a function instead.''
else if ! isPath path then
if path._type or "" == "fileset" then
throw ''
lib.fileset.fileFilter: Second argument is a file set, but it should be a path instead.
If you need to filter files in a file set, use `intersection fileset (fileFilter pred ./.)` instead.''
else
throw ''
lib.fileset.fileFilter: Second argument is of type ${typeOf path}, but it should be a path instead.''
else if ! pathExists path then
throw ''
lib.fileset.fileFilter: Second argument (${toString path}) is a path that does not exist.''
else
_fileFilter predicate path;
/* /*
The file set containing all files that are in both of two given file sets. The file set containing all files that are in both of two given file sets.
See also [Intersection (set theory)](https://en.wikipedia.org/wiki/Intersection_(set_theory)). See also [Intersection (set theory)](https://en.wikipedia.org/wiki/Intersection_(set_theory)).
@ -563,94 +566,133 @@ in {
(elemAt filesets 1); (elemAt filesets 1);
/* /*
Incrementally evaluate and trace a file set in a pretty way. Filter a file set to only contain files matching some predicate.
This function is only intended for debugging purposes.
The exact tracing format is unspecified and may change.
This function takes a final argument to return.
In comparison, [`traceVal`](#function-library-lib.fileset.traceVal) returns
the given file set argument.
This variant is useful for tracing file sets in the Nix repl.
Type: Type:
trace :: FileSet -> Any -> Any fileFilter ::
({
name :: String,
type :: String,
...
} -> Bool)
-> Path
-> FileSet
Example: Example:
trace (unions [ ./Makefile ./src ./tests/run.sh ]) null # Include all regular `default.nix` files in the current directory
=> fileFilter (file: file.name == "default.nix") ./.
trace: /home/user/src/myProject
trace: - Makefile (regular)
trace: - src (all files in directory)
trace: - tests
trace: - run.sh (regular)
null
*/
trace =
/*
The file set to trace.
This argument can also be a path, # Include all non-Nix files from the current directory
which gets [implicitly coerced to a file set](#sec-fileset-path-coercion). fileFilter (file: ! hasSuffix ".nix" file.name) ./.
# Include all files that start with a "." in the current directory
fileFilter (file: hasPrefix "." file.name) ./.
# Include all regular files (not symlinks or others) in the current directory
fileFilter (file: file.type == "regular") ./.
*/
fileFilter =
/*
The predicate function to call on all files contained in given file set.
A file is included in the resulting file set if this function returns true for it.
This function is called with an attribute set containing these attributes:
- `name` (String): The name of the file
- `type` (String, one of `"regular"`, `"symlink"` or `"unknown"`): The type of the file.
This matches result of calling [`builtins.readFileType`](https://nixos.org/manual/nix/stable/language/builtins.html#builtins-readFileType) on the file's path.
Other attributes may be added in the future.
*/ */
fileset: predicate:
let # The path whose files to filter
# "fileset" would be a better name, but that would clash with the argument name, path:
# and we cannot change that because of https://github.com/nix-community/nixdoc/issues/76 if ! isFunction predicate then
actualFileset = _coerce "lib.fileset.trace: Argument" fileset; throw ''
in lib.fileset.fileFilter: First argument is of type ${typeOf predicate}, but it should be a function instead.''
seq else if ! isPath path then
(_printFileset actualFileset) if path._type or "" == "fileset" then
(x: x); throw ''
lib.fileset.fileFilter: Second argument is a file set, but it should be a path instead.
If you need to filter files in a file set, use `intersection fileset (fileFilter pred ./.)` instead.''
else
throw ''
lib.fileset.fileFilter: Second argument is of type ${typeOf path}, but it should be a path instead.''
else if ! pathExists path then
throw ''
lib.fileset.fileFilter: Second argument (${toString path}) is a path that does not exist.''
else
_fileFilter predicate path;
/* /*
Incrementally evaluate and trace a file set in a pretty way. Create a file set with the same files as a `lib.sources`-based value.
This function is only intended for debugging purposes. This does not import any of the files into the store.
The exact tracing format is unspecified and may change.
This function returns the given file set. This can be used to gradually migrate from `lib.sources`-based filtering to `lib.fileset`.
In comparison, [`trace`](#function-library-lib.fileset.trace) takes another argument to return.
This variant is useful for tracing file sets passed as arguments to other functions. A file set can be turned back into a source using [`toSource`](#function-library-lib.fileset.toSource).
Type: :::{.note}
traceVal :: FileSet -> FileSet File sets cannot represent empty directories.
Turning the result of this function back into a source using `toSource` will therefore not preserve empty directories.
:::
Example: Type:
toSource { fromSource :: SourceLike -> FileSet
root = ./.;
fileset = traceVal (unions [ Example:
./Makefile # There's no cleanSource-like function for file sets yet,
./src # but we can just convert cleanSource to a file set and use it that way
./tests/run.sh toSource {
]); root = ./.;
} fileset = fromSource (lib.sources.cleanSource ./.);
=> }
trace: /home/user/src/myProject
trace: - Makefile (regular) # Keeping a previous sourceByRegex (which could be migrated to `lib.fileset.unions`),
trace: - src (all files in directory) # but removing a subdirectory using file set functions
trace: - tests difference
trace: - run.sh (regular) (fromSource (lib.sources.sourceByRegex ./. [
"/nix/store/...-source" "^README\.md$"
# This regex includes everything in ./doc
"^doc(/.*)?$"
])
./doc/generated
# Use cleanSource, but limit it to only include ./Makefile and files under ./src
intersection
(fromSource (lib.sources.cleanSource ./.))
(unions [
./Makefile
./src
]);
*/ */
traceVal = fromSource = source:
/*
The file set to trace and return.
This argument can also be a path,
which gets [implicitly coerced to a file set](#sec-fileset-path-coercion).
*/
fileset:
let let
# "fileset" would be a better name, but that would clash with the argument name, # This function uses `._isLibCleanSourceWith`, `.origSrc` and `.filter`,
# and we cannot change that because of https://github.com/nix-community/nixdoc/issues/76 # which are technically internal to lib.sources,
actualFileset = _coerce "lib.fileset.traceVal: Argument" fileset; # but we'll allow this since both libraries are in the same code base
# and this function is a bridge between them.
isFiltered = source ? _isLibCleanSourceWith;
path = if isFiltered then source.origSrc else source;
in in
seq # We can only support sources created from paths
(_printFileset actualFileset) if ! isPath path then
# We could also return the original fileset argument here, if isStringLike path then
# but that would then duplicate work for consumers of the fileset, because then they have to coerce it again throw ''
actualFileset; lib.fileset.fromSource: The source origin of the argument is a string-like value ("${toString path}"), but it should be a path instead.
Sources created from paths in strings cannot be turned into file sets, use `lib.sources` or derivations instead.''
else
throw ''
lib.fileset.fromSource: The source origin of the argument is of type ${typeOf path}, but it should be a path instead.''
else if ! pathExists path then
throw ''
lib.fileset.fromSource: The source origin (${toString path}) of the argument is a path that does not exist.''
else if isFiltered then
_fromSourceFilter path source.filter
else
# If there's no filter, no need to run the expensive conversion, all subpaths will be included
_singleton path;
/* /*
Create a file set containing all [Git-tracked files](https://git-scm.com/book/en/v2/Git-Basics-Recording-Changes-to-the-Repository) in a repository. Create a file set containing all [Git-tracked files](https://git-scm.com/book/en/v2/Git-Basics-Recording-Changes-to-the-Repository) in a repository.