The GeoIP databases from MaxMind have no stable URLs and change every
month (or so). Our current method of packaging these database in Nix and
playing catch-up with ever-changing file hashes is a bad idea. For
instance, it makes it impossible to realize old NixOS configurations.
This patch adds a NixOS service that periodically updates the GeoIP
databases in /var/lib/geoip-databases. Moving NixOS modules over can be
done in later patches.
I tried adding MD5 check, but not all databases have them, so i skipped
it. We are downloading over HTTPS though, it should be good. I also
tried adding zip support, but the first zip file I extracted had a
different filename inside than the archive name, which breaks an
assumption in this service, so I skipped that too.
Changes v9 -> v10:
- Pass "--max-time" to curl to set upper bound on downloads (ensures
no indefinite hanging if there's problem with networking).
Timeout for network connectivity check: 60s.
Timeout for geoip database (each): 15m.
Changes v8 -> v9:
- Mention the random timer delay in the documentation for the
'interval' option.
Changes v7 -> v8:
- Add "RemainAfterExit=true" for the setup service, so it won't be
restarted needlessly. (Thanks @danbst!)
Changes v6 -> v7:
- Add --skip-existing flag to geoip-updater, which skips updating
existing database files. Pass that flag when we run the service on
boot (and on any NixOS configuration change).
(IMHO, this is somewhat a workaround for systemd persistent timers
not being triggered immediately when a timer has never expired
before. But it does have the nice side effect of ensuring that the
installed databases always correspond to the configured ones, since
the service is now always run after configuration changes.)
Changes v5 -> v6:
- Update database files atomically (per DB)
- If a database is removed from the configuration, it'll be removed
from /var/lib/geoip-databases too (on next run).
- Add NixOS module assertion so that if user inputs non- .gz or .xz
file there will be a build time error instead of runtime.
- Run updater as user "nobody" instead of "root".
- Rename NixOS service from "geoip-databases" to "geoip-updater".
- Drop RemainAfterExit, or else the timer won't trigger the unit.
- Bring back "curl --fail", or else we won't catch and log curl
failures.
Changes v4 -> v5:
- Add "GeoLite2-City.mmdb.gz" to default database list.
Changes v3 -> v4:
- Remove unneeded geoip-updater-setup.service after adding
'wantedBy = [ "multi-user.target" ]' directly to
geoip-updater.service
- Drop unneeded "Service" name from service descriptions.
Changes v2 -> v3:
- Network may be down when starting from a cold boot, so try a few
times. Possibly, if using systemd-networkd, it'll pass on the first
try. But with default DHCP on NixOS, the service is started before
hostnames can be resolved and thus we need a few extra seconds.
- Add error handling and mark service as failed if fatal error.
- Add proper syslog log levels.
- Add RandomizedDelaySec=3600 to the timer to not put high load on the
MaxMind servers. Suggested by @Mic92.
- Set RemainAfterExit on geoip-updater.service instead of
geoip-updater-setup.service. (The latter is only a proxy that pulls
in the former service).
Changes v1 -> v2:
From Данило Глинський (Danylo Hlynskyi) <abcz2.uprola@gmail.com>:
nixos/geoip-databases: add `databases` option and fix initial setup
There were two great issues when using this service:
- When you just enable service, databases aren't downloaded, they are
downloaded when timer triggers. Fixed this with automatic download on
first system activation.
- When there is no internet, updater outputs nothing to logs, which is
IMO misbehavior. Fixed this with removing `--fail` option, better be
explicit here.
Pushing to gogs only works if the `gogs` user's shell is `bash`. For error and
solution, refer to [this SO thread](http://stackoverflow.com/a/22315659)
A default of 0 means that if you deploy two NixOS boxes with the default
configuration, the second will fail because the brokerId was already in
use. Using -1 instead tells it to pick one automatically at first start.
perlPackages.TextWrapI18N: init at 0.06
perlPackages.Po4a: init at 0.47
jade: init at 1.2.1
ding-libs: init at 0.6.0
Switch nscd to no-caching mode if SSSD is enabled.
abbradar: disable jade parallel building.
Closes#21150
Using the example before this commit resulted in the following error:
```
error: value is a string while a list was expected, at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/nixos/modules/services/misc/nix-daemon.nix:349:37
```
The matrix-synapse user has `createHome = true;` which runs before the
`preStart` script, so the home directory will always exist and the block
will never execute.
Also don't include default path to keys in the configuration file,
because synapse will choke if it tries to open them before they
exist (even with `--generate-keys`).
`startAt = ""` as in `startAt = optionalString false ...` results
in an invalid timer unit (due to "" being promoted to a singleton
list and not filtered out).
Ref: c9941c4b5e
This was one of the ways to build packages, we are trying
hard to minimize different ways so it's easier for newcomers
to learn only one way.
This also:
- removes texLive (old), fixes#14807
- removed upstream-updater, if that code is still used it should be in
separate repo
- changes a few packages like gitit/mit-scheme to use new texlive
The name gitlab-runner clashes with a component of Gitlab CI with the
same name and only confuses people. It's now called gitlab-bundle and
a convenience-script gitlab-rake for easier invocation of rake tasks
was added. This was the primary use case of gitlab-runner.
- currently pulled in from Git until the next release of PackageKit
has Nix support
- also: add in a service module to start packagekit properly
- nixos service can be enabled via services.packagekit.enable
- packagekit requires nixunstable to build properly
Move Subsonic state directory from `/var/subsonic` to
`/var/lib/subsonic`, since the general convention is for each
application to put its state directory there.
Also, automatically set the home directory of the `subsonic` user to the
value of `config.services.subsonic.home`, rather than setting it to a
value hardcoded in the module. This keeps the home directory of the
`subsonic` user and the state directory for the Subsonic application in
sync.
With the merge of the closure-size branch, most packages now have
multiple outputs. One of these packages is gnutls, so previously
everything that we needed was to reference "${gnutls}/bin/..." and now
we need to use "${gnutls.bin}/bin/...".
So it's not a very big issue to fix.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
This adds a Taskserver module along with documentation and a small
helper tool which eases managing a custom CA along with Taskserver
organisations, users and groups.
Taskserver is the server component of Taskwarrior, a TODO list
application for the command line.
The work has been started by @matthiasbeyer back in mid 2015 and I have
continued to work on it recently, so this merge contains commits from
both of us.
Thanks particularly to @nbp and @matthiasbeyer for reviewing and
suggesting improvements.
I've tested this with the new test (nixos/tests/taskserver.nix) this
branch adds and it fails because of the changes introduced by the
closure-size branch, so we need to do additional work on base of this.
Suggested by @nbp:
"Choose a better organization name in this example, such that it is less
confusing. Maybe something like my-company"
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
We have already revamped the CLI subcommands in commit
e2383b84f8.
This was just an artifact that was left because of this.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
The options client.allow and client.deny are gone since the commit
8b793d1916, so let's fix that.
No feature changes, only fixes the descriptions of allowedClientIDs and
disallowedClientIDs.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
This is the recommended way for long-running services and ensures that
Taskserver will keep running until it has been stopped manually.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Using requiredBy is a bad idea for the initialisation units, because
whenever the Taskserver service is restarted the initialisation units
get restarted as well.
Also, make sure taskserver-init.service will be ordered *before*
taskserver.service.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
The Taskserver doesn't need access to the full /dev nor does it need a
shared /tmp. In addition, the initialisation services don't need network
access, so let's constrain them to the loopback device.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Apart from the options manual, this should cover the basics for setting
up a Taskserver. I am not a native speaker so this can and (probably)
should be improved, especially the wording/grammar.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Try to match the subcommands to act more like the subcommands from the
taskd binary and also add a subcommand to list groups.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
As suggested by @matthiasbeyer:
"We might add a short note that this port has to be opened in the
firewall, or is this done by the service automatically?"
This commit now adds the listenPort to
networking.firewall.allowedTCPPorts as soon as the listenHost is not
"localhost".
In addition to that, this is now also documented in the listenHost
option declaration and I have removed disabling of the firewall from the
VM test.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
No changes in functionality but rather just restructuring the module
definitions to be one mkMerge, which now uses mkIf from the top-level
scope of the CA initialization service so we can better abstract
additional options we might need there.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
We want to make sure that the helper tool won't work if the automatic CA
wasn't properly set up. This not only avoids race conditions if the tool
is started before the actual service is running but it also fails if
something during CA setup has failed so the user can investigate what
went wrong.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
We need to explicitly make sure the CA is created before we actually
launch the main Taskserver service in order to avoid race conditions
where the preStart phase of the main service could possibly corrupt
certificates if it would be started in parallel.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
This is simply to add configuration lines to the generated configuration
file. The reason why I didn't went for an attribute set is that the
taskdrc file format doesn't map very well on Nix attributes, for example
the following can be set in taskdrc:
server = somestring
server.key = anotherstring
In order to use a Nix attribute set for that, it would be way too
complicated, for example if we want to represent the mentioned example
we'd have to do something like this:
{ server._top = somestring;
server.key = anotherstring;
}
Of course, this would work as well but nothing is more simple than just
appending raw strings.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
At least this should allow for some customisation of how the
certificates and keys are created. We now have two sub-namespaces within
PKI so it should be more clear which options you have to set if you want
to either manage your own CA or let the module create it automatically.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Whenever the nixos-taskserver tool was invoked manually for creating an
organisation/group/user we now add an empty file called .imperative to
the data directory.
During the preStart of the Taskserver service, we use process-json which
in turn now checks whether those .imperative files exist and if so, it
doesn't do anything with it.
This should now ensure that whenever there is a manually created user,
it doesn't get killed off by the declarative configuration in case it
shouldn't exist within that configuration.
In addition, we also add a small subtest to check whether this is
happening or not and fail if the imperatively created user got deleted
by process-json.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
We only print the output whenever there is an error, otherwise let's
shut it up because it only shows information the user can gather through
other means. For example by invoking certtool manually, or by just
looking at private key files (the whole blurb it's outputting is in
there as well).
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Now we finally can delete organisations, groups and users along with
certificate revocation. The new subtests now make sure that the client
certificate is also revoked (both when removing the whole organisation
and just a single user).
If we use the imperative way to add and delete users, we have to restart
the Taskserver in order for the CRL to be effective.
However, by using the declarative configuration we now get this for
free, because removing a user will also restart the service and thus its
client certificate will end up in the CRL.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Unfortunately we don't have a better way to check whether the reload has
been done successfully, but at least we now *can* reload it without
figuring out the exact signal to send to the process.
Note that on reload, Taskserver will not reload the CRL file. For that
to work, a full restart needs to be done.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
If we want to revoke client certificates and want the server to actually
notice the revocation, we need to have a valid certificate revocation
list.
Right now the expiration_days is set to 10 years, but that's merely to
actually get certtool to actually generate the CRL without trying to
prompt for user input.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
It doesn't do much harm to make the server certificate world readable,
because even though it's not accessible anymore via the file system,
someone can still get it by simply doing a TLS handshake with the
server.
So this is solely for consistency.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
We now no longer have the stupid --service-helper option, which silences
messages about already existing organisations, users or groups.
Instead of that option, we now have a new subcommand called
"process-json", which accepts a JSON file directly from the specified
NixOS module options and creates/deletes the users accordingly.
Note that this still has a two issues left to solve in this area:
* Deletion is not supported yet.
* If a user is created imperatively, the next run of process-json will
delete it once deletion is supported.
So we need to implement deletion and a way to mark organisations, users
and groups as "imperatively managed".
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
The Click functions really are for the command line and should be solely
used for that.
What I have in mind is that instead of that crappy --service-helper
argument, we should really have a new subcommand that is expecting JSON
which is directly coming from the services.taskserver.organisations
module option.
That way we can decrease even more boilerplate and we can also ensure
that organisations, users and groups get properly deleted if they're
removed from the NixOS configuration.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
With <olink/> support in place, we can now reference the Taskserver
section within the NixOS manual, so that users reading the manpage of
configuration.nix(5) won't miss this information.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
We're now using .pki.server.* and .pki.ca.* so that it's entirely clear
what these keys/certificates are for. For example we had just .pki.key
before, which doesn't really tell very much about what it's for except
if you look at the option description.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
The improvement here is just that we're adding a big <note/> here so
that users of these options are aware that whenever they're setting one
of these the certificates and keys are _not_ created automatically.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
This is clearly a server configuration option and has nothing to do with
certificate creation and signing, so let's move it away from the .pki
namespace.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
It's not necessarily related to the PKI options, because this is also
used for setting the server address on the Taskwarrior client.
So if someone doesn't have his/her own certificates from another CA, all
options that need to be adjusted are in .pki. And if someone doesn't
want to bother with getting certificates from another CA, (s)he just
doesn't set anything in .pki.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
After moving out the PKI-unrelated options, let's name this a bit more
appropriate, so we can finally get rid of the taskserver.server thing.
This also moves taskserver.caCert to taskserver.pki.caCert, because that
clearly belongs to the PKI options.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Having an option called services.taskserver.server.host is quite
confusing because we already have "server" in the service name, so let's
first get rid of the listening options before we rename the rest of the
options in that .server attribute.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
In the comments of the pull request @nbp wrote:
"Why is it implemented in 3 different languages: Nix, Bash and C?"
And he's right, it doesn't make sense, because we were using C as a
runuser replacement and used Nix to generate the shellscript
boilerplates.
Writing this in Python gets rid of all of this and we also don't need
the boilerplate as well, because we're using Click to handle all the
command line stuff.
Note that this currently is a 1:1 implementation of what we had before.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
The option is solely for debugging purposes (particularly the unit tests
of the project itself) and doesn't make sense to include it in the NixOS
module options.
If people want to use this, we might want to introduce another option so
that we can insert arbitrary configuration lines.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
It's not by any means exhaustive, but we're still going to change the
implementation, so let's just use this as a starting point.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
These values match against the client IDs only, so let's rename it to
something that actually reflects that. Having client.cert in the same
namespace also could lead to confusion, because the client.cert setting
is for the *debugging* client only.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Referring to the GnuTLS documentation isn't very nice if the user has to
use a search engine to find that documentation. So let's directly link
to it.
The type was "str" before, but it's actually a colon-separated string,
so if we set options in multiple modules, the result is one concatenated
string.
I know there is types.envVar, which does the same as separatedString ":"
but I found that it could confuse the reader of the Taskserver module.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
We already document that we allow special values such as "all" and
"none", but the type doesn't represent that. So let's use an enum in
conjuction with a loeOf type so that this becomes clear.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Using nixos-taskserver is more verbose but less cryptic and I think it
fits the purpose better because it can't be confused to be a wrapper
around the taskdctl command from the upstream project as
nixos-taskserver shares no commonalities with it.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
With a cluttered up module source it's really a pain to navigate through
it, so it's a good idea to put it into another file.
No changes in functionality here, just splitting up the files and fixing
references.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Finally, this is where we declaratively set up our organisations and
users/groups, which looks like this in the system configuration:
services.taskserver.organisations.NixOS.users = [ "alice" "bob" ];
This automatically sets up "alice" and "bob" for the "NixOS"
organisation, generates the required client keys and signs it via the
CA.
However, we still need to use nixos-taskdctl export-user in order to
import these certificates on the client.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
It's a helper for NixOS systems to make it easier to handle CA
certificate signing, similar to what taskd provides but comes preseeded
with the values from the system configuration.
The tool is very limited at the moment and only allows to *add*
organisations, users and groups. Deletion and suspension however is much
simpler to implement, because we don't need to handle certificate
signing.
Another limitation is that we don't take into account whether
certificates and keys are already set in the system configuration and if
they're set it will fail spectacularly.
For passing the commands to the taskd command, we're using a small C
program which does setuid() and setgid() to the Taskserver user and
group, because runuser(1) needs PAM (quite pointless if you're already
root) and su(1) doesn't allow for setting the group and setgid()s to the
default group of the user, so it even doesn't work in conjunction with
sg(1).
In summary, we now have a shiny nixos-taskdctl command, which lets us do
things like:
nixos-taskdctl add-org NixOS
nixos-taskdctl add-user NixOS alice
nixos-taskdctl export-user NixOS alice
The last command writes a series of shell commands to stdout, which then
can be imported on the client by piping it into a shell as well as doing
it for example via SSH:
ssh root@server nixos-taskdctl export-user NixOS alice | sh
Of course, in terms of security we need to improve this even further so
that we generate the private key on the client and just send a CSR to
the server so that we don't need to push any secrets over the wire.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
We want to declaratively specify users and organisations, so let's add
another module option "organisations", which allows us to specify users,
groups and of course organisations.
The implementation of this is not yet done and this is just to feed the
boilerplate.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Using just the host for the common name *and* for listening on the port
is quite a bad idea if you want to listen on something like :: or an
internal IP address which is proxied/tunneled to the outside.
Hence this separates host and fqdn.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
The server starts up without that option anyway, but it complains about
its value not being set. As we probably want to have access to that
configuration value anyway, let's expose this via the NixOS module as
well.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Now the service starts up if only the services.taskserver.enable option
is set to true.
We now also have three systemd services (started in this order):
* taskserver-init: For creating the necessary data directory and also
includes a refecence to the configuration file in
the Nix store.
* taskserver-ca: Only enabled if none of the server.key, server.cert,
server.crl and caCert options are set, so we can
allow for certificates that are issued by another
CA.
This service creates a new CA key+certificate and a
server key+certificate and signs the latter using
the CA key.
The permissions of these keys/certs are set quite
strictly to allow only the root user to sign
certificates.
* taskserver: The main Taskserver service which just starts taskd.
We now also log to stdout and thus to the journal.
Of course, there are still a few problems left to solve, for instance:
* The CA currently only signs the server certificates, so it's
only usable for clients if the server doesn't validate client certs
(which is kinda pointless).
* Using "taskd <command>" is currently still a bit awkward to use, so
we need to properly wrap it in environment.systemPackages to set the
dataDir by default.
* There are still a few configuration options left to include, for
example the "trust" option.
* We might want to introduce an extraConfig option.
* It might be useful to allow for declarative configuration of
organisations and users, especially when it comes to creating client
certificates.
* The right signal has to be sent for the taskserver service to reload
properly.
* Currently the CA and server certificates are created using
server.host as the common name and doesn't set additional certificate
information. This could be improved by adding options that explicitly
set that information.
As for the config file, we might need to patch taskd to allow for
setting not only --data but also a --cfgfile, which then omits the
${dataDir}/config file. We can still use the "include" directive from
the file specified using --cfgfile in order to chainload
${dataDir}/config.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
The descriptions for the options previously seem to be from the
taskdrc(5) manual page. So in cases where they didn't make sense for us
I changed the wording a bit (for example for client.deny we don't have a
"comma-separated list".
Also, I've reordered things a bit for consistency (type, default,
example and then description) and add missing types, examples and
docbook tags.
Options that are not used by default now have a null value, so that we
can generate a configuration file out of all the options defined for the
module.
The dataDir default value is now /var/lib/taskserver, because it doesn't
make sense to put just yet another empty subdirectory in it and "data"
doesn't quite make sense anyway, because it also contains the
configuration file as well.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
We're aiming for a proper integration into systemd/journald, so we
really don't want zillions of separate log files flying around in our
system.
Same as with the pidFile. The latter is only needed for taskdctl, which
is a SysV-style initscript and all of its functionality plus a lot more
is handled by systemd already.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
The service doesn't start with the "taskd" user being present, so we
really should add it. And while at it, it really makes sense to add a
default group as well.
I'm using a check for the user/group name as well, to allow the
taskserver to be run as an existing user.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
- Enforce that an option declaration has a "defaultText" if and only if the
type of the option derives from "package", "packageSet" or "nixpkgsConfig"
and if a "default" attribute is defined.
- Enforce that the value of the "example" attribute is wrapped with "literalExample"
if the type of the option derives from "package", "packageSet" or "nixpkgsConfig".
- Warn if a "defaultText" is defined in an option declaration if the type of
the option does not derive from "package", "packageSet" or "nixpkgsConfig".
- Warn if no "type" is defined in an option declaration.
Updates gitlab to the current stable version and fixes a lot of features that
were broken, at least with the current version and our configuration.
Quite a lot of sweat and tears has gone into testing nearly all features and
reading/patching the Gitlab source as we're about to deploy gitlab for our
whole company.
Things to note:
* The gitlab config is now written as a nix attribute set and will be
converted to JSON. Gitlab uses YAML but JSON is a subset of YAML.
The `extraConfig` opition is also an attribute set that will be merged
with the default config. This way *all* Gitlab options are supported.
* Some paths like uploads and configs are hardcoded in rails (at least
after my study of the Gitlab source). This is why they are linked from
the Gitlab root to /run/gitlab and then linked to the configurable
`statePath`.
* Backup & restore should work out of the box from another Gitlab instance.
* gitlab-git-http-server has been replaced by gitlab-workhorse upstream.
Push & pull over HTTPS works perfectly. Communication to gitlab is done
over unix sockets. An HTTP server is required to proxy requests to
gitlab-workhorse over another unix socket at
`/run/gitlab/gitlab-workhorse.socket`.
* The user & group running gitlab are now configurable. These can even be
changed for live instances.
* The initial email address & password of the root user can be configured.
Fixes#8598.
