nixos/foundationdb: convert manual chapter to MD
This commit is contained in:
pennae
parent
1ce4fde27b
commit
5b012f2c55
309
nixos/modules/services/databases/foundationdb.md
Normal file
309
nixos/modules/services/databases/foundationdb.md
Normal file
@ -0,0 +1,309 @@
|
||||
# FoundationDB {#module-services-foundationdb}
|
||||
|
||||
*Source:* {file}`modules/services/databases/foundationdb.nix`
|
||||
|
||||
*Upstream documentation:* <https://apple.github.io/foundationdb/>
|
||||
|
||||
*Maintainer:* Austin Seipp
|
||||
|
||||
*Available version(s):* 5.1.x, 5.2.x, 6.0.x
|
||||
|
||||
FoundationDB (or "FDB") is an open source, distributed, transactional
|
||||
key-value store.
|
||||
|
||||
## Configuring and basic setup {#module-services-foundationdb-configuring}
|
||||
|
||||
To enable FoundationDB, add the following to your
|
||||
{file}`configuration.nix`:
|
||||
```
|
||||
services.foundationdb.enable = true;
|
||||
services.foundationdb.package = pkgs.foundationdb52; # FoundationDB 5.2.x
|
||||
```
|
||||
|
||||
The {option}`services.foundationdb.package` option is required, and
|
||||
must always be specified. Due to the fact FoundationDB network protocols and
|
||||
on-disk storage formats may change between (major) versions, and upgrades
|
||||
must be explicitly handled by the user, you must always manually specify
|
||||
this yourself so that the NixOS module will use the proper version. Note
|
||||
that minor, bugfix releases are always compatible.
|
||||
|
||||
After running {command}`nixos-rebuild`, you can verify whether
|
||||
FoundationDB is running by executing {command}`fdbcli` (which is
|
||||
added to {option}`environment.systemPackages`):
|
||||
```ShellSession
|
||||
$ sudo -u foundationdb fdbcli
|
||||
Using cluster file `/etc/foundationdb/fdb.cluster'.
|
||||
|
||||
The database is available.
|
||||
|
||||
Welcome to the fdbcli. For help, type `help'.
|
||||
fdb> status
|
||||
|
||||
Using cluster file `/etc/foundationdb/fdb.cluster'.
|
||||
|
||||
Configuration:
|
||||
Redundancy mode - single
|
||||
Storage engine - memory
|
||||
Coordinators - 1
|
||||
|
||||
Cluster:
|
||||
FoundationDB processes - 1
|
||||
Machines - 1
|
||||
Memory availability - 5.4 GB per process on machine with least available
|
||||
Fault Tolerance - 0 machines
|
||||
Server time - 04/20/18 15:21:14
|
||||
|
||||
...
|
||||
|
||||
fdb>
|
||||
```
|
||||
|
||||
You can also write programs using the available client libraries. For
|
||||
example, the following Python program can be run in order to grab the
|
||||
cluster status, as a quick example. (This example uses
|
||||
{command}`nix-shell` shebang support to automatically supply the
|
||||
necessary Python modules).
|
||||
```ShellSession
|
||||
a@link> cat fdb-status.py
|
||||
#! /usr/bin/env nix-shell
|
||||
#! nix-shell -i python -p python pythonPackages.foundationdb52
|
||||
|
||||
import fdb
|
||||
import json
|
||||
|
||||
def main():
|
||||
fdb.api_version(520)
|
||||
db = fdb.open()
|
||||
|
||||
@fdb.transactional
|
||||
def get_status(tr):
|
||||
return str(tr['\xff\xff/status/json'])
|
||||
|
||||
obj = json.loads(get_status(db))
|
||||
print('FoundationDB available: %s' % obj['client']['database_status']['available'])
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
a@link> chmod +x fdb-status.py
|
||||
a@link> ./fdb-status.py
|
||||
FoundationDB available: True
|
||||
a@link>
|
||||
```
|
||||
|
||||
FoundationDB is run under the {command}`foundationdb` user and group
|
||||
by default, but this may be changed in the NixOS configuration. The systemd
|
||||
unit {command}`foundationdb.service` controls the
|
||||
{command}`fdbmonitor` process.
|
||||
|
||||
By default, the NixOS module for FoundationDB creates a single SSD-storage
|
||||
based database for development and basic usage. This storage engine is
|
||||
designed for SSDs and will perform poorly on HDDs; however it can handle far
|
||||
more data than the alternative "memory" engine and is a better default
|
||||
choice for most deployments. (Note that you can change the storage backend
|
||||
on-the-fly for a given FoundationDB cluster using
|
||||
{command}`fdbcli`.)
|
||||
|
||||
Furthermore, only 1 server process and 1 backup agent are started in the
|
||||
default configuration. See below for more on scaling to increase this.
|
||||
|
||||
FoundationDB stores all data for all server processes under
|
||||
{file}`/var/lib/foundationdb`. You can override this using
|
||||
{option}`services.foundationdb.dataDir`, e.g.
|
||||
```
|
||||
services.foundationdb.dataDir = "/data/fdb";
|
||||
```
|
||||
|
||||
Similarly, logs are stored under {file}`/var/log/foundationdb`
|
||||
by default, and there is a corresponding
|
||||
{option}`services.foundationdb.logDir` as well.
|
||||
|
||||
## Scaling processes and backup agents {#module-services-foundationdb-scaling}
|
||||
|
||||
Scaling the number of server processes is quite easy; simply specify
|
||||
{option}`services.foundationdb.serverProcesses` to be the number of
|
||||
FoundationDB worker processes that should be started on the machine.
|
||||
|
||||
FoundationDB worker processes typically require 4GB of RAM per-process at
|
||||
minimum for good performance, so this option is set to 1 by default since
|
||||
the maximum amount of RAM is unknown. You're advised to abide by this
|
||||
restriction, so pick a number of processes so that each has 4GB or more.
|
||||
|
||||
A similar option exists in order to scale backup agent processes,
|
||||
{option}`services.foundationdb.backupProcesses`. Backup agents are
|
||||
not as performance/RAM sensitive, so feel free to experiment with the number
|
||||
of available backup processes.
|
||||
|
||||
## Clustering {#module-services-foundationdb-clustering}
|
||||
|
||||
FoundationDB on NixOS works similarly to other Linux systems, so this
|
||||
section will be brief. Please refer to the full FoundationDB documentation
|
||||
for more on clustering.
|
||||
|
||||
FoundationDB organizes clusters using a set of
|
||||
*coordinators*, which are just specially-designated
|
||||
worker processes. By default, every installation of FoundationDB on NixOS
|
||||
will start as its own individual cluster, with a single coordinator: the
|
||||
first worker process on {command}`localhost`.
|
||||
|
||||
Coordinators are specified globally using the
|
||||
{command}`/etc/foundationdb/fdb.cluster` file, which all servers and
|
||||
client applications will use to find and join coordinators. Note that this
|
||||
file *can not* be managed by NixOS so easily:
|
||||
FoundationDB is designed so that it will rewrite the file at runtime for all
|
||||
clients and nodes when cluster coordinators change, with clients
|
||||
transparently handling this without intervention. It is fundamentally a
|
||||
mutable file, and you should not try to manage it in any way in NixOS.
|
||||
|
||||
When dealing with a cluster, there are two main things you want to do:
|
||||
|
||||
- Add a node to the cluster for storage/compute.
|
||||
- Promote an ordinary worker to a coordinator.
|
||||
|
||||
A node must already be a member of the cluster in order to properly be
|
||||
promoted to a coordinator, so you must always add it first if you wish to
|
||||
promote it.
|
||||
|
||||
To add a machine to a FoundationDB cluster:
|
||||
|
||||
- Choose one of the servers to start as the initial coordinator.
|
||||
- Copy the {command}`/etc/foundationdb/fdb.cluster` file from this
|
||||
server to all the other servers. Restart FoundationDB on all of these
|
||||
other servers, so they join the cluster.
|
||||
- All of these servers are now connected and working together in the
|
||||
cluster, under the chosen coordinator.
|
||||
|
||||
At this point, you can add as many nodes as you want by just repeating the
|
||||
above steps. By default there will still be a single coordinator: you can
|
||||
use {command}`fdbcli` to change this and add new coordinators.
|
||||
|
||||
As a convenience, FoundationDB can automatically assign coordinators based
|
||||
on the redundancy mode you wish to achieve for the cluster. Once all the
|
||||
nodes have been joined, simply set the replication policy, and then issue
|
||||
the {command}`coordinators auto` command
|
||||
|
||||
For example, assuming we have 3 nodes available, we can enable double
|
||||
redundancy mode, then auto-select coordinators. For double redundancy, 3
|
||||
coordinators is ideal: therefore FoundationDB will make
|
||||
*every* node a coordinator automatically:
|
||||
|
||||
```ShellSession
|
||||
fdbcli> configure double ssd
|
||||
fdbcli> coordinators auto
|
||||
```
|
||||
|
||||
This will transparently update all the servers within seconds, and
|
||||
appropriately rewrite the {command}`fdb.cluster` file, as well as
|
||||
informing all client processes to do the same.
|
||||
|
||||
## Client connectivity {#module-services-foundationdb-connectivity}
|
||||
|
||||
By default, all clients must use the current {command}`fdb.cluster`
|
||||
file to access a given FoundationDB cluster. This file is located by default
|
||||
in {command}`/etc/foundationdb/fdb.cluster` on all machines with the
|
||||
FoundationDB service enabled, so you may copy the active one from your
|
||||
cluster to a new node in order to connect, if it is not part of the cluster.
|
||||
|
||||
## Client authorization and TLS {#module-services-foundationdb-authorization}
|
||||
|
||||
By default, any user who can connect to a FoundationDB process with the
|
||||
correct cluster configuration can access anything. FoundationDB uses a
|
||||
pluggable design to transport security, and out of the box it supports a
|
||||
LibreSSL-based plugin for TLS support. This plugin not only does in-flight
|
||||
encryption, but also performs client authorization based on the given
|
||||
endpoint's certificate chain. For example, a FoundationDB server may be
|
||||
configured to only accept client connections over TLS, where the client TLS
|
||||
certificate is from organization *Acme Co* in the
|
||||
*Research and Development* unit.
|
||||
|
||||
Configuring TLS with FoundationDB is done using the
|
||||
{option}`services.foundationdb.tls` options in order to control the
|
||||
peer verification string, as well as the certificate and its private key.
|
||||
|
||||
Note that the certificate and its private key must be accessible to the
|
||||
FoundationDB user account that the server runs under. These files are also
|
||||
NOT managed by NixOS, as putting them into the store may reveal private
|
||||
information.
|
||||
|
||||
After you have a key and certificate file in place, it is not enough to
|
||||
simply set the NixOS module options -- you must also configure the
|
||||
{command}`fdb.cluster` file to specify that a given set of
|
||||
coordinators use TLS. This is as simple as adding the suffix
|
||||
{command}`:tls` to your cluster coordinator configuration, after the
|
||||
port number. For example, assuming you have a coordinator on localhost with
|
||||
the default configuration, simply specifying:
|
||||
|
||||
```
|
||||
XXXXXX:XXXXXX@127.0.0.1:4500:tls
|
||||
```
|
||||
|
||||
will configure all clients and server processes to use TLS from now on.
|
||||
|
||||
## Backups and Disaster Recovery {#module-services-foundationdb-disaster-recovery}
|
||||
|
||||
The usual rules for doing FoundationDB backups apply on NixOS as written in
|
||||
the FoundationDB manual. However, one important difference is the security
|
||||
profile for NixOS: by default, the {command}`foundationdb` systemd
|
||||
unit uses *Linux namespaces* to restrict write access to
|
||||
the system, except for the log directory, data directory, and the
|
||||
{command}`/etc/foundationdb/` directory. This is enforced by default
|
||||
and cannot be disabled.
|
||||
|
||||
However, a side effect of this is that the {command}`fdbbackup`
|
||||
command doesn't work properly for local filesystem backups: FoundationDB
|
||||
uses a server process alongside the database processes to perform backups
|
||||
and copy the backups to the filesystem. As a result, this process is put
|
||||
under the restricted namespaces above: the backup process can only write to
|
||||
a limited number of paths.
|
||||
|
||||
In order to allow flexible backup locations on local disks, the FoundationDB
|
||||
NixOS module supports a
|
||||
{option}`services.foundationdb.extraReadWritePaths` option. This
|
||||
option takes a list of paths, and adds them to the systemd unit, allowing
|
||||
the processes inside the service to write (and read) the specified
|
||||
directories.
|
||||
|
||||
For example, to create backups in {command}`/opt/fdb-backups`, first
|
||||
set up the paths in the module options:
|
||||
|
||||
```
|
||||
services.foundationdb.extraReadWritePaths = [ "/opt/fdb-backups" ];
|
||||
```
|
||||
|
||||
Restart the FoundationDB service, and it will now be able to write to this
|
||||
directory (even if it does not yet exist.) Note: this path
|
||||
*must* exist before restarting the unit. Otherwise,
|
||||
systemd will not include it in the private FoundationDB namespace (and it
|
||||
will not add it dynamically at runtime).
|
||||
|
||||
You can now perform a backup:
|
||||
|
||||
```ShellSession
|
||||
$ sudo -u foundationdb fdbbackup start -t default -d file:///opt/fdb-backups
|
||||
$ sudo -u foundationdb fdbbackup status -t default
|
||||
```
|
||||
|
||||
## Known limitations {#module-services-foundationdb-limitations}
|
||||
|
||||
The FoundationDB setup for NixOS should currently be considered beta.
|
||||
FoundationDB is not new software, but the NixOS compilation and integration
|
||||
has only undergone fairly basic testing of all the available functionality.
|
||||
|
||||
- There is no way to specify individual parameters for individual
|
||||
{command}`fdbserver` processes. Currently, all server processes
|
||||
inherit all the global {command}`fdbmonitor` settings.
|
||||
- Ruby bindings are not currently installed.
|
||||
- Go bindings are not currently installed.
|
||||
|
||||
## Options {#module-services-foundationdb-options}
|
||||
|
||||
NixOS's FoundationDB module allows you to configure all of the most relevant
|
||||
configuration options for {command}`fdbmonitor`, matching it quite
|
||||
closely. A complete list of options for the FoundationDB module may be found
|
||||
[here](#opt-services.foundationdb.enable). You should
|
||||
also read the FoundationDB documentation as well.
|
||||
|
||||
## Full documentation {#module-services-foundationdb-full-docs}
|
||||
|
||||
FoundationDB is a complex piece of software, and requires careful
|
||||
administration to properly use. Full documentation for administration can be
|
||||
found here: <https://apple.github.io/foundationdb/>.
|
||||
@ -424,6 +424,8 @@ in
|
||||
};
|
||||
};
|
||||
|
||||
# Don't edit the docbook xml directly, edit the md and generate it:
|
||||
# `pandoc foundationdb.md -t docbook --top-level-division=chapter --extract-media=media -f markdown-smart --lua-filter ../../../../doc/build-aux/pandoc-filters/myst-reader/roles.lua --lua-filter ../../../../doc/build-aux/pandoc-filters/docbook-writer/rst-roles.lua > foundationdb.xml`
|
||||
meta.doc = ./foundationdb.xml;
|
||||
meta.maintainers = with lib.maintainers; [ thoughtpolice ];
|
||||
}
|
||||
|
||||
@ -1,60 +1,56 @@
|
||||
<chapter xmlns="http://docbook.org/ns/docbook"
|
||||
xmlns:xlink="http://www.w3.org/1999/xlink"
|
||||
xmlns:xi="http://www.w3.org/2001/XInclude"
|
||||
version="5.0"
|
||||
xml:id="module-services-foundationdb">
|
||||
<title>FoundationDB</title>
|
||||
<para>
|
||||
<emphasis>Source:</emphasis>
|
||||
<filename>modules/services/databases/foundationdb.nix</filename>
|
||||
</para>
|
||||
<para>
|
||||
<emphasis>Upstream documentation:</emphasis>
|
||||
<link xlink:href="https://apple.github.io/foundationdb/"/>
|
||||
</para>
|
||||
<para>
|
||||
<emphasis>Maintainer:</emphasis> Austin Seipp
|
||||
</para>
|
||||
<para>
|
||||
<emphasis>Available version(s):</emphasis> 5.1.x, 5.2.x, 6.0.x
|
||||
</para>
|
||||
<para>
|
||||
FoundationDB (or "FDB") is an open source, distributed, transactional
|
||||
key-value store.
|
||||
</para>
|
||||
<section xml:id="module-services-foundationdb-configuring">
|
||||
<title>Configuring and basic setup</title>
|
||||
|
||||
<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xlink="http://www.w3.org/1999/xlink" xml:id="module-services-foundationdb">
|
||||
<title>FoundationDB</title>
|
||||
<para>
|
||||
To enable FoundationDB, add the following to your
|
||||
<filename>configuration.nix</filename>:
|
||||
<programlisting>
|
||||
<emphasis>Source:</emphasis>
|
||||
<filename>modules/services/databases/foundationdb.nix</filename>
|
||||
</para>
|
||||
<para>
|
||||
<emphasis>Upstream documentation:</emphasis>
|
||||
<link xlink:href="https://apple.github.io/foundationdb/" role="uri">https://apple.github.io/foundationdb/</link>
|
||||
</para>
|
||||
<para>
|
||||
<emphasis>Maintainer:</emphasis> Austin Seipp
|
||||
</para>
|
||||
<para>
|
||||
<emphasis>Available version(s):</emphasis> 5.1.x, 5.2.x, 6.0.x
|
||||
</para>
|
||||
<para>
|
||||
FoundationDB (or "FDB") is an open source, distributed,
|
||||
transactional key-value store.
|
||||
</para>
|
||||
<section xml:id="module-services-foundationdb-configuring">
|
||||
<title>Configuring and basic setup</title>
|
||||
<para>
|
||||
To enable FoundationDB, add the following to your
|
||||
<filename>configuration.nix</filename>:
|
||||
</para>
|
||||
<programlisting>
|
||||
services.foundationdb.enable = true;
|
||||
services.foundationdb.package = pkgs.foundationdb52; # FoundationDB 5.2.x
|
||||
</programlisting>
|
||||
</para>
|
||||
|
||||
<para>
|
||||
The <option>services.foundationdb.package</option> option is required, and
|
||||
must always be specified. Due to the fact FoundationDB network protocols and
|
||||
on-disk storage formats may change between (major) versions, and upgrades
|
||||
must be explicitly handled by the user, you must always manually specify
|
||||
this yourself so that the NixOS module will use the proper version. Note
|
||||
that minor, bugfix releases are always compatible.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
After running <command>nixos-rebuild</command>, you can verify whether
|
||||
FoundationDB is running by executing <command>fdbcli</command> (which is
|
||||
added to <option>environment.systemPackages</option>):
|
||||
<screen>
|
||||
<prompt>$ </prompt>sudo -u foundationdb fdbcli
|
||||
<para>
|
||||
The <option>services.foundationdb.package</option> option is
|
||||
required, and must always be specified. Due to the fact
|
||||
FoundationDB network protocols and on-disk storage formats may
|
||||
change between (major) versions, and upgrades must be explicitly
|
||||
handled by the user, you must always manually specify this
|
||||
yourself so that the NixOS module will use the proper version.
|
||||
Note that minor, bugfix releases are always compatible.
|
||||
</para>
|
||||
<para>
|
||||
After running <command>nixos-rebuild</command>, you can verify
|
||||
whether FoundationDB is running by executing
|
||||
<command>fdbcli</command> (which is added to
|
||||
<option>environment.systemPackages</option>):
|
||||
</para>
|
||||
<programlisting>
|
||||
$ sudo -u foundationdb fdbcli
|
||||
Using cluster file `/etc/foundationdb/fdb.cluster'.
|
||||
|
||||
The database is available.
|
||||
|
||||
Welcome to the fdbcli. For help, type `help'.
|
||||
<prompt>fdb> </prompt>status
|
||||
fdb> status
|
||||
|
||||
Using cluster file `/etc/foundationdb/fdb.cluster'.
|
||||
|
||||
@ -72,18 +68,17 @@ Cluster:
|
||||
|
||||
...
|
||||
|
||||
<prompt>fdb></prompt>
|
||||
</screen>
|
||||
</para>
|
||||
|
||||
<para>
|
||||
You can also write programs using the available client libraries. For
|
||||
example, the following Python program can be run in order to grab the
|
||||
cluster status, as a quick example. (This example uses
|
||||
<command>nix-shell</command> shebang support to automatically supply the
|
||||
necessary Python modules).
|
||||
<screen>
|
||||
<prompt>a@link> </prompt>cat fdb-status.py
|
||||
fdb>
|
||||
</programlisting>
|
||||
<para>
|
||||
You can also write programs using the available client libraries.
|
||||
For example, the following Python program can be run in order to
|
||||
grab the cluster status, as a quick example. (This example uses
|
||||
<command>nix-shell</command> shebang support to automatically
|
||||
supply the necessary Python modules).
|
||||
</para>
|
||||
<programlisting>
|
||||
a@link> cat fdb-status.py
|
||||
#! /usr/bin/env nix-shell
|
||||
#! nix-shell -i python -p python pythonPackages.foundationdb52
|
||||
|
||||
@ -101,343 +96,328 @@ def main():
|
||||
obj = json.loads(get_status(db))
|
||||
print('FoundationDB available: %s' % obj['client']['database_status']['available'])
|
||||
|
||||
if __name__ == "__main__":
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
<prompt>a@link> </prompt>chmod +x fdb-status.py
|
||||
<prompt>a@link> </prompt>./fdb-status.py
|
||||
a@link> chmod +x fdb-status.py
|
||||
a@link> ./fdb-status.py
|
||||
FoundationDB available: True
|
||||
<prompt>a@link></prompt>
|
||||
</screen>
|
||||
</para>
|
||||
|
||||
<para>
|
||||
FoundationDB is run under the <command>foundationdb</command> user and group
|
||||
by default, but this may be changed in the NixOS configuration. The systemd
|
||||
unit <command>foundationdb.service</command> controls the
|
||||
<command>fdbmonitor</command> process.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
By default, the NixOS module for FoundationDB creates a single SSD-storage
|
||||
based database for development and basic usage. This storage engine is
|
||||
designed for SSDs and will perform poorly on HDDs; however it can handle far
|
||||
more data than the alternative "memory" engine and is a better default
|
||||
choice for most deployments. (Note that you can change the storage backend
|
||||
on-the-fly for a given FoundationDB cluster using
|
||||
<command>fdbcli</command>.)
|
||||
</para>
|
||||
|
||||
<para>
|
||||
Furthermore, only 1 server process and 1 backup agent are started in the
|
||||
default configuration. See below for more on scaling to increase this.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
FoundationDB stores all data for all server processes under
|
||||
<filename>/var/lib/foundationdb</filename>. You can override this using
|
||||
<option>services.foundationdb.dataDir</option>, e.g.
|
||||
<programlisting>
|
||||
services.foundationdb.dataDir = "/data/fdb";
|
||||
a@link>
|
||||
</programlisting>
|
||||
</para>
|
||||
|
||||
<para>
|
||||
Similarly, logs are stored under <filename>/var/log/foundationdb</filename>
|
||||
by default, and there is a corresponding
|
||||
<option>services.foundationdb.logDir</option> as well.
|
||||
</para>
|
||||
</section>
|
||||
<section xml:id="module-services-foundationdb-scaling">
|
||||
<title>Scaling processes and backup agents</title>
|
||||
|
||||
<para>
|
||||
Scaling the number of server processes is quite easy; simply specify
|
||||
<option>services.foundationdb.serverProcesses</option> to be the number of
|
||||
FoundationDB worker processes that should be started on the machine.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
FoundationDB worker processes typically require 4GB of RAM per-process at
|
||||
minimum for good performance, so this option is set to 1 by default since
|
||||
the maximum amount of RAM is unknown. You're advised to abide by this
|
||||
restriction, so pick a number of processes so that each has 4GB or more.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
A similar option exists in order to scale backup agent processes,
|
||||
<option>services.foundationdb.backupProcesses</option>. Backup agents are
|
||||
not as performance/RAM sensitive, so feel free to experiment with the number
|
||||
of available backup processes.
|
||||
</para>
|
||||
</section>
|
||||
<section xml:id="module-services-foundationdb-clustering">
|
||||
<title>Clustering</title>
|
||||
|
||||
<para>
|
||||
FoundationDB on NixOS works similarly to other Linux systems, so this
|
||||
section will be brief. Please refer to the full FoundationDB documentation
|
||||
for more on clustering.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
FoundationDB organizes clusters using a set of
|
||||
<emphasis>coordinators</emphasis>, which are just specially-designated
|
||||
worker processes. By default, every installation of FoundationDB on NixOS
|
||||
will start as its own individual cluster, with a single coordinator: the
|
||||
first worker process on <command>localhost</command>.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
Coordinators are specified globally using the
|
||||
<command>/etc/foundationdb/fdb.cluster</command> file, which all servers and
|
||||
client applications will use to find and join coordinators. Note that this
|
||||
file <emphasis>can not</emphasis> be managed by NixOS so easily:
|
||||
FoundationDB is designed so that it will rewrite the file at runtime for all
|
||||
clients and nodes when cluster coordinators change, with clients
|
||||
transparently handling this without intervention. It is fundamentally a
|
||||
mutable file, and you should not try to manage it in any way in NixOS.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
When dealing with a cluster, there are two main things you want to do:
|
||||
</para>
|
||||
|
||||
<itemizedlist>
|
||||
<listitem>
|
||||
<para>
|
||||
Add a node to the cluster for storage/compute.
|
||||
FoundationDB is run under the <command>foundationdb</command> user
|
||||
and group by default, but this may be changed in the NixOS
|
||||
configuration. The systemd unit
|
||||
<command>foundationdb.service</command> controls the
|
||||
<command>fdbmonitor</command> process.
|
||||
</para>
|
||||
</listitem>
|
||||
<listitem>
|
||||
<para>
|
||||
Promote an ordinary worker to a coordinator.
|
||||
By default, the NixOS module for FoundationDB creates a single
|
||||
SSD-storage based database for development and basic usage. This
|
||||
storage engine is designed for SSDs and will perform poorly on
|
||||
HDDs; however it can handle far more data than the alternative
|
||||
"memory" engine and is a better default choice for most
|
||||
deployments. (Note that you can change the storage backend
|
||||
on-the-fly for a given FoundationDB cluster using
|
||||
<command>fdbcli</command>.)
|
||||
</para>
|
||||
</listitem>
|
||||
</itemizedlist>
|
||||
|
||||
<para>
|
||||
A node must already be a member of the cluster in order to properly be
|
||||
promoted to a coordinator, so you must always add it first if you wish to
|
||||
promote it.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
To add a machine to a FoundationDB cluster:
|
||||
</para>
|
||||
|
||||
<itemizedlist>
|
||||
<listitem>
|
||||
<para>
|
||||
Choose one of the servers to start as the initial coordinator.
|
||||
Furthermore, only 1 server process and 1 backup agent are started
|
||||
in the default configuration. See below for more on scaling to
|
||||
increase this.
|
||||
</para>
|
||||
</listitem>
|
||||
<listitem>
|
||||
<para>
|
||||
Copy the <command>/etc/foundationdb/fdb.cluster</command> file from this
|
||||
server to all the other servers. Restart FoundationDB on all of these
|
||||
other servers, so they join the cluster.
|
||||
FoundationDB stores all data for all server processes under
|
||||
<filename>/var/lib/foundationdb</filename>. You can override this
|
||||
using <option>services.foundationdb.dataDir</option>, e.g.
|
||||
</para>
|
||||
</listitem>
|
||||
<listitem>
|
||||
<programlisting>
|
||||
services.foundationdb.dataDir = "/data/fdb";
|
||||
</programlisting>
|
||||
<para>
|
||||
All of these servers are now connected and working together in the
|
||||
cluster, under the chosen coordinator.
|
||||
Similarly, logs are stored under
|
||||
<filename>/var/log/foundationdb</filename> by default, and there
|
||||
is a corresponding <option>services.foundationdb.logDir</option>
|
||||
as well.
|
||||
</para>
|
||||
</listitem>
|
||||
</itemizedlist>
|
||||
|
||||
<para>
|
||||
At this point, you can add as many nodes as you want by just repeating the
|
||||
above steps. By default there will still be a single coordinator: you can
|
||||
use <command>fdbcli</command> to change this and add new coordinators.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
As a convenience, FoundationDB can automatically assign coordinators based
|
||||
on the redundancy mode you wish to achieve for the cluster. Once all the
|
||||
nodes have been joined, simply set the replication policy, and then issue
|
||||
the <command>coordinators auto</command> command
|
||||
</para>
|
||||
|
||||
<para>
|
||||
For example, assuming we have 3 nodes available, we can enable double
|
||||
redundancy mode, then auto-select coordinators. For double redundancy, 3
|
||||
coordinators is ideal: therefore FoundationDB will make
|
||||
<emphasis>every</emphasis> node a coordinator automatically:
|
||||
</para>
|
||||
|
||||
<screen>
|
||||
<prompt>fdbcli> </prompt>configure double ssd
|
||||
<prompt>fdbcli> </prompt>coordinators auto
|
||||
</screen>
|
||||
|
||||
<para>
|
||||
This will transparently update all the servers within seconds, and
|
||||
appropriately rewrite the <command>fdb.cluster</command> file, as well as
|
||||
informing all client processes to do the same.
|
||||
</para>
|
||||
</section>
|
||||
<section xml:id="module-services-foundationdb-connectivity">
|
||||
<title>Client connectivity</title>
|
||||
|
||||
<para>
|
||||
By default, all clients must use the current <command>fdb.cluster</command>
|
||||
file to access a given FoundationDB cluster. This file is located by default
|
||||
in <command>/etc/foundationdb/fdb.cluster</command> on all machines with the
|
||||
FoundationDB service enabled, so you may copy the active one from your
|
||||
cluster to a new node in order to connect, if it is not part of the cluster.
|
||||
</para>
|
||||
</section>
|
||||
<section xml:id="module-services-foundationdb-authorization">
|
||||
<title>Client authorization and TLS</title>
|
||||
|
||||
<para>
|
||||
By default, any user who can connect to a FoundationDB process with the
|
||||
correct cluster configuration can access anything. FoundationDB uses a
|
||||
pluggable design to transport security, and out of the box it supports a
|
||||
LibreSSL-based plugin for TLS support. This plugin not only does in-flight
|
||||
encryption, but also performs client authorization based on the given
|
||||
endpoint's certificate chain. For example, a FoundationDB server may be
|
||||
configured to only accept client connections over TLS, where the client TLS
|
||||
certificate is from organization <emphasis>Acme Co</emphasis> in the
|
||||
<emphasis>Research and Development</emphasis> unit.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
Configuring TLS with FoundationDB is done using the
|
||||
<option>services.foundationdb.tls</option> options in order to control the
|
||||
peer verification string, as well as the certificate and its private key.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
Note that the certificate and its private key must be accessible to the
|
||||
FoundationDB user account that the server runs under. These files are also
|
||||
NOT managed by NixOS, as putting them into the store may reveal private
|
||||
information.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
After you have a key and certificate file in place, it is not enough to
|
||||
simply set the NixOS module options -- you must also configure the
|
||||
<command>fdb.cluster</command> file to specify that a given set of
|
||||
coordinators use TLS. This is as simple as adding the suffix
|
||||
<command>:tls</command> to your cluster coordinator configuration, after the
|
||||
port number. For example, assuming you have a coordinator on localhost with
|
||||
the default configuration, simply specifying:
|
||||
</para>
|
||||
|
||||
<programlisting>
|
||||
</section>
|
||||
<section xml:id="module-services-foundationdb-scaling">
|
||||
<title>Scaling processes and backup agents</title>
|
||||
<para>
|
||||
Scaling the number of server processes is quite easy; simply
|
||||
specify <option>services.foundationdb.serverProcesses</option> to
|
||||
be the number of FoundationDB worker processes that should be
|
||||
started on the machine.
|
||||
</para>
|
||||
<para>
|
||||
FoundationDB worker processes typically require 4GB of RAM
|
||||
per-process at minimum for good performance, so this option is set
|
||||
to 1 by default since the maximum amount of RAM is unknown. You're
|
||||
advised to abide by this restriction, so pick a number of
|
||||
processes so that each has 4GB or more.
|
||||
</para>
|
||||
<para>
|
||||
A similar option exists in order to scale backup agent processes,
|
||||
<option>services.foundationdb.backupProcesses</option>. Backup
|
||||
agents are not as performance/RAM sensitive, so feel free to
|
||||
experiment with the number of available backup processes.
|
||||
</para>
|
||||
</section>
|
||||
<section xml:id="module-services-foundationdb-clustering">
|
||||
<title>Clustering</title>
|
||||
<para>
|
||||
FoundationDB on NixOS works similarly to other Linux systems, so
|
||||
this section will be brief. Please refer to the full FoundationDB
|
||||
documentation for more on clustering.
|
||||
</para>
|
||||
<para>
|
||||
FoundationDB organizes clusters using a set of
|
||||
<emphasis>coordinators</emphasis>, which are just
|
||||
specially-designated worker processes. By default, every
|
||||
installation of FoundationDB on NixOS will start as its own
|
||||
individual cluster, with a single coordinator: the first worker
|
||||
process on <command>localhost</command>.
|
||||
</para>
|
||||
<para>
|
||||
Coordinators are specified globally using the
|
||||
<command>/etc/foundationdb/fdb.cluster</command> file, which all
|
||||
servers and client applications will use to find and join
|
||||
coordinators. Note that this file <emphasis>can not</emphasis> be
|
||||
managed by NixOS so easily: FoundationDB is designed so that it
|
||||
will rewrite the file at runtime for all clients and nodes when
|
||||
cluster coordinators change, with clients transparently handling
|
||||
this without intervention. It is fundamentally a mutable file, and
|
||||
you should not try to manage it in any way in NixOS.
|
||||
</para>
|
||||
<para>
|
||||
When dealing with a cluster, there are two main things you want to
|
||||
do:
|
||||
</para>
|
||||
<itemizedlist spacing="compact">
|
||||
<listitem>
|
||||
<para>
|
||||
Add a node to the cluster for storage/compute.
|
||||
</para>
|
||||
</listitem>
|
||||
<listitem>
|
||||
<para>
|
||||
Promote an ordinary worker to a coordinator.
|
||||
</para>
|
||||
</listitem>
|
||||
</itemizedlist>
|
||||
<para>
|
||||
A node must already be a member of the cluster in order to
|
||||
properly be promoted to a coordinator, so you must always add it
|
||||
first if you wish to promote it.
|
||||
</para>
|
||||
<para>
|
||||
To add a machine to a FoundationDB cluster:
|
||||
</para>
|
||||
<itemizedlist spacing="compact">
|
||||
<listitem>
|
||||
<para>
|
||||
Choose one of the servers to start as the initial coordinator.
|
||||
</para>
|
||||
</listitem>
|
||||
<listitem>
|
||||
<para>
|
||||
Copy the <command>/etc/foundationdb/fdb.cluster</command> file
|
||||
from this server to all the other servers. Restart
|
||||
FoundationDB on all of these other servers, so they join the
|
||||
cluster.
|
||||
</para>
|
||||
</listitem>
|
||||
<listitem>
|
||||
<para>
|
||||
All of these servers are now connected and working together in
|
||||
the cluster, under the chosen coordinator.
|
||||
</para>
|
||||
</listitem>
|
||||
</itemizedlist>
|
||||
<para>
|
||||
At this point, you can add as many nodes as you want by just
|
||||
repeating the above steps. By default there will still be a single
|
||||
coordinator: you can use <command>fdbcli</command> to change this
|
||||
and add new coordinators.
|
||||
</para>
|
||||
<para>
|
||||
As a convenience, FoundationDB can automatically assign
|
||||
coordinators based on the redundancy mode you wish to achieve for
|
||||
the cluster. Once all the nodes have been joined, simply set the
|
||||
replication policy, and then issue the
|
||||
<command>coordinators auto</command> command
|
||||
</para>
|
||||
<para>
|
||||
For example, assuming we have 3 nodes available, we can enable
|
||||
double redundancy mode, then auto-select coordinators. For double
|
||||
redundancy, 3 coordinators is ideal: therefore FoundationDB will
|
||||
make <emphasis>every</emphasis> node a coordinator automatically:
|
||||
</para>
|
||||
<programlisting>
|
||||
fdbcli> configure double ssd
|
||||
fdbcli> coordinators auto
|
||||
</programlisting>
|
||||
<para>
|
||||
This will transparently update all the servers within seconds, and
|
||||
appropriately rewrite the <command>fdb.cluster</command> file, as
|
||||
well as informing all client processes to do the same.
|
||||
</para>
|
||||
</section>
|
||||
<section xml:id="module-services-foundationdb-connectivity">
|
||||
<title>Client connectivity</title>
|
||||
<para>
|
||||
By default, all clients must use the current
|
||||
<command>fdb.cluster</command> file to access a given FoundationDB
|
||||
cluster. This file is located by default in
|
||||
<command>/etc/foundationdb/fdb.cluster</command> on all machines
|
||||
with the FoundationDB service enabled, so you may copy the active
|
||||
one from your cluster to a new node in order to connect, if it is
|
||||
not part of the cluster.
|
||||
</para>
|
||||
</section>
|
||||
<section xml:id="module-services-foundationdb-authorization">
|
||||
<title>Client authorization and TLS</title>
|
||||
<para>
|
||||
By default, any user who can connect to a FoundationDB process
|
||||
with the correct cluster configuration can access anything.
|
||||
FoundationDB uses a pluggable design to transport security, and
|
||||
out of the box it supports a LibreSSL-based plugin for TLS
|
||||
support. This plugin not only does in-flight encryption, but also
|
||||
performs client authorization based on the given endpoint's
|
||||
certificate chain. For example, a FoundationDB server may be
|
||||
configured to only accept client connections over TLS, where the
|
||||
client TLS certificate is from organization <emphasis>Acme
|
||||
Co</emphasis> in the <emphasis>Research and Development</emphasis>
|
||||
unit.
|
||||
</para>
|
||||
<para>
|
||||
Configuring TLS with FoundationDB is done using the
|
||||
<option>services.foundationdb.tls</option> options in order to
|
||||
control the peer verification string, as well as the certificate
|
||||
and its private key.
|
||||
</para>
|
||||
<para>
|
||||
Note that the certificate and its private key must be accessible
|
||||
to the FoundationDB user account that the server runs under. These
|
||||
files are also NOT managed by NixOS, as putting them into the
|
||||
store may reveal private information.
|
||||
</para>
|
||||
<para>
|
||||
After you have a key and certificate file in place, it is not
|
||||
enough to simply set the NixOS module options -- you must also
|
||||
configure the <command>fdb.cluster</command> file to specify that
|
||||
a given set of coordinators use TLS. This is as simple as adding
|
||||
the suffix <command>:tls</command> to your cluster coordinator
|
||||
configuration, after the port number. For example, assuming you
|
||||
have a coordinator on localhost with the default configuration,
|
||||
simply specifying:
|
||||
</para>
|
||||
<programlisting>
|
||||
XXXXXX:XXXXXX@127.0.0.1:4500:tls
|
||||
</programlisting>
|
||||
|
||||
<para>
|
||||
will configure all clients and server processes to use TLS from now on.
|
||||
</para>
|
||||
</section>
|
||||
<section xml:id="module-services-foundationdb-disaster-recovery">
|
||||
<title>Backups and Disaster Recovery</title>
|
||||
|
||||
<para>
|
||||
The usual rules for doing FoundationDB backups apply on NixOS as written in
|
||||
the FoundationDB manual. However, one important difference is the security
|
||||
profile for NixOS: by default, the <command>foundationdb</command> systemd
|
||||
unit uses <emphasis>Linux namespaces</emphasis> to restrict write access to
|
||||
the system, except for the log directory, data directory, and the
|
||||
<command>/etc/foundationdb/</command> directory. This is enforced by default
|
||||
and cannot be disabled.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
However, a side effect of this is that the <command>fdbbackup</command>
|
||||
command doesn't work properly for local filesystem backups: FoundationDB
|
||||
uses a server process alongside the database processes to perform backups
|
||||
and copy the backups to the filesystem. As a result, this process is put
|
||||
under the restricted namespaces above: the backup process can only write to
|
||||
a limited number of paths.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
In order to allow flexible backup locations on local disks, the FoundationDB
|
||||
NixOS module supports a
|
||||
<option>services.foundationdb.extraReadWritePaths</option> option. This
|
||||
option takes a list of paths, and adds them to the systemd unit, allowing
|
||||
the processes inside the service to write (and read) the specified
|
||||
directories.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
For example, to create backups in <command>/opt/fdb-backups</command>, first
|
||||
set up the paths in the module options:
|
||||
</para>
|
||||
|
||||
<programlisting>
|
||||
services.foundationdb.extraReadWritePaths = [ "/opt/fdb-backups" ];
|
||||
<para>
|
||||
will configure all clients and server processes to use TLS from
|
||||
now on.
|
||||
</para>
|
||||
</section>
|
||||
<section xml:id="module-services-foundationdb-disaster-recovery">
|
||||
<title>Backups and Disaster Recovery</title>
|
||||
<para>
|
||||
The usual rules for doing FoundationDB backups apply on NixOS as
|
||||
written in the FoundationDB manual. However, one important
|
||||
difference is the security profile for NixOS: by default, the
|
||||
<command>foundationdb</command> systemd unit uses <emphasis>Linux
|
||||
namespaces</emphasis> to restrict write access to the system,
|
||||
except for the log directory, data directory, and the
|
||||
<command>/etc/foundationdb/</command> directory. This is enforced
|
||||
by default and cannot be disabled.
|
||||
</para>
|
||||
<para>
|
||||
However, a side effect of this is that the
|
||||
<command>fdbbackup</command> command doesn't work properly for
|
||||
local filesystem backups: FoundationDB uses a server process
|
||||
alongside the database processes to perform backups and copy the
|
||||
backups to the filesystem. As a result, this process is put under
|
||||
the restricted namespaces above: the backup process can only write
|
||||
to a limited number of paths.
|
||||
</para>
|
||||
<para>
|
||||
In order to allow flexible backup locations on local disks, the
|
||||
FoundationDB NixOS module supports a
|
||||
<option>services.foundationdb.extraReadWritePaths</option> option.
|
||||
This option takes a list of paths, and adds them to the systemd
|
||||
unit, allowing the processes inside the service to write (and
|
||||
read) the specified directories.
|
||||
</para>
|
||||
<para>
|
||||
For example, to create backups in
|
||||
<command>/opt/fdb-backups</command>, first set up the paths in the
|
||||
module options:
|
||||
</para>
|
||||
<programlisting>
|
||||
services.foundationdb.extraReadWritePaths = [ "/opt/fdb-backups" ];
|
||||
</programlisting>
|
||||
|
||||
<para>
|
||||
Restart the FoundationDB service, and it will now be able to write to this
|
||||
directory (even if it does not yet exist.) Note: this path
|
||||
<emphasis>must</emphasis> exist before restarting the unit. Otherwise,
|
||||
systemd will not include it in the private FoundationDB namespace (and it
|
||||
will not add it dynamically at runtime).
|
||||
</para>
|
||||
|
||||
<para>
|
||||
You can now perform a backup:
|
||||
</para>
|
||||
|
||||
<screen>
|
||||
<prompt>$ </prompt>sudo -u foundationdb fdbbackup start -t default -d file:///opt/fdb-backups
|
||||
<prompt>$ </prompt>sudo -u foundationdb fdbbackup status -t default
|
||||
</screen>
|
||||
</section>
|
||||
<section xml:id="module-services-foundationdb-limitations">
|
||||
<title>Known limitations</title>
|
||||
|
||||
<para>
|
||||
The FoundationDB setup for NixOS should currently be considered beta.
|
||||
FoundationDB is not new software, but the NixOS compilation and integration
|
||||
has only undergone fairly basic testing of all the available functionality.
|
||||
</para>
|
||||
|
||||
<itemizedlist>
|
||||
<listitem>
|
||||
<para>
|
||||
There is no way to specify individual parameters for individual
|
||||
<command>fdbserver</command> processes. Currently, all server processes
|
||||
inherit all the global <command>fdbmonitor</command> settings.
|
||||
Restart the FoundationDB service, and it will now be able to write
|
||||
to this directory (even if it does not yet exist.) Note: this path
|
||||
<emphasis>must</emphasis> exist before restarting the unit.
|
||||
Otherwise, systemd will not include it in the private FoundationDB
|
||||
namespace (and it will not add it dynamically at runtime).
|
||||
</para>
|
||||
</listitem>
|
||||
<listitem>
|
||||
<para>
|
||||
Ruby bindings are not currently installed.
|
||||
You can now perform a backup:
|
||||
</para>
|
||||
</listitem>
|
||||
<listitem>
|
||||
<programlisting>
|
||||
$ sudo -u foundationdb fdbbackup start -t default -d file:///opt/fdb-backups
|
||||
$ sudo -u foundationdb fdbbackup status -t default
|
||||
</programlisting>
|
||||
</section>
|
||||
<section xml:id="module-services-foundationdb-limitations">
|
||||
<title>Known limitations</title>
|
||||
<para>
|
||||
Go bindings are not currently installed.
|
||||
The FoundationDB setup for NixOS should currently be considered
|
||||
beta. FoundationDB is not new software, but the NixOS compilation
|
||||
and integration has only undergone fairly basic testing of all the
|
||||
available functionality.
|
||||
</para>
|
||||
</listitem>
|
||||
</itemizedlist>
|
||||
</section>
|
||||
<section xml:id="module-services-foundationdb-options">
|
||||
<title>Options</title>
|
||||
|
||||
<para>
|
||||
NixOS's FoundationDB module allows you to configure all of the most relevant
|
||||
configuration options for <command>fdbmonitor</command>, matching it quite
|
||||
closely. A complete list of options for the FoundationDB module may be found
|
||||
<link linkend="opt-services.foundationdb.enable">here</link>. You should
|
||||
also read the FoundationDB documentation as well.
|
||||
</para>
|
||||
</section>
|
||||
<section xml:id="module-services-foundationdb-full-docs">
|
||||
<title>Full documentation</title>
|
||||
|
||||
<para>
|
||||
FoundationDB is a complex piece of software, and requires careful
|
||||
administration to properly use. Full documentation for administration can be
|
||||
found here: <link xlink:href="https://apple.github.io/foundationdb/"/>.
|
||||
</para>
|
||||
</section>
|
||||
<itemizedlist spacing="compact">
|
||||
<listitem>
|
||||
<para>
|
||||
There is no way to specify individual parameters for
|
||||
individual <command>fdbserver</command> processes. Currently,
|
||||
all server processes inherit all the global
|
||||
<command>fdbmonitor</command> settings.
|
||||
</para>
|
||||
</listitem>
|
||||
<listitem>
|
||||
<para>
|
||||
Ruby bindings are not currently installed.
|
||||
</para>
|
||||
</listitem>
|
||||
<listitem>
|
||||
<para>
|
||||
Go bindings are not currently installed.
|
||||
</para>
|
||||
</listitem>
|
||||
</itemizedlist>
|
||||
</section>
|
||||
<section xml:id="module-services-foundationdb-options">
|
||||
<title>Options</title>
|
||||
<para>
|
||||
NixOS's FoundationDB module allows you to configure all of the
|
||||
most relevant configuration options for
|
||||
<command>fdbmonitor</command>, matching it quite closely. A
|
||||
complete list of options for the FoundationDB module may be found
|
||||
<link linkend="opt-services.foundationdb.enable">here</link>. You
|
||||
should also read the FoundationDB documentation as well.
|
||||
</para>
|
||||
</section>
|
||||
<section xml:id="module-services-foundationdb-full-docs">
|
||||
<title>Full documentation</title>
|
||||
<para>
|
||||
FoundationDB is a complex piece of software, and requires careful
|
||||
administration to properly use. Full documentation for
|
||||
administration can be found here:
|
||||
<link xlink:href="https://apple.github.io/foundationdb/" role="uri">https://apple.github.io/foundationdb/</link>.
|
||||
</para>
|
||||
</section>
|
||||
</chapter>
|
||||
|
||||
Loading…
Reference in New Issue
Block a user