This seems to be the root cause of the random page allocation failures
and @wizeman did a very good job on not only finding the root problem
but also giving a detailed explanation of it in #10828.
Here is an excerpt:
The problem here is that the kernel is trying to allocate a contiguous
section of 2^7=128 pages, which is 512 KB. This is way too much:
kernel pages tend to get fragmented over time and kernel developers
often go to great lengths to try allocating at most only 1 contiguous
page at a time whenever they can.
From the error message, it looks like the culprit is unionfs, but this
is misleading: unionfs is the name of the userspace process that was
running when the system ran out of memory, but it wasn't unionfs who
was allocating the memory: it was the kernel; specifically it was the
v9fs_dir_readdir_dotl() function, which is the code for handling the
readdir() function in the 9p filesystem (the filesystem that is used
to share a directory structure between a qemu host and its VM).
If you look at the code, here's what it's doing at the moment it tries
to allocate memory:
buflen = fid->clnt->msize - P9_IOHDRSZ;
rdir = v9fs_alloc_rdir_buf(file, buflen);
If you look into v9fs_alloc_rdir_buf(), you will see that it will try
to allocate a contiguous buffer of memory (using kzalloc(), which is a
wrapper around kmalloc()) of size buflen + 8 bytes or so.
So in reality, this code actually allocates a buffer of size
proportional to fid->clnt->msize. What is this msize? If you follow
the definition of the structures, you will see that it's the
negotiated buffer transfer size between 9p client and 9p server. On
the client side, it can be controlled with the msize mount option.
What this all means is that, the reason for running out of memory is
that the code (which we can't easily change) tries to allocate a
contiguous buffer of size more or less equal to "negotiated 9p
protocol buffer size", which seems to be way too big (in our NixOS
tests, at least).
After that initial finding, @lethalman tested the gnome3 gdm test
without setting the msize parameter at all and it seems to have resolved
the problem.
The reason why I'm committing this without testing against all of the
NixOS VM test is basically that I think we can only go better but not
worse than the current state.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
This ensures that the intermediate machine is shut down only after the
migration has finished writing the memory dump to disk, to ensure we
don't end up with empty state files depending on how fast the migration
finished before we actually shut down the VM.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
This ensures that the builder isn't waiting forever if the Windows VM
drops dead while we're waiting for the controller VM to signal that a
particular command has been executed on the Windows VM. It won't ever
happen in such cases so it doesn't make sense to wait for the timeout.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
So far, we determined this based on stdenv.is64bit, but there are cases
where you want to run/build a 32bit program on a 64 bit Windows.
This is now possible, by passing windowsImage.arch = "i686" | "x86_64"
to runInWindowsVM. Based an what was passed, the corresponding Cygwin
packages and setup.exe are bootstrapped.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Another very annoying part. Unfortunately, the only option we might have
here is to include it in nixpkgs or maybe make a fixed Hash on the
result of the closure fetcher.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
As the official Cygwin setup binary download doesn't come in snapshots
or even versioned, the fetchurl of setup.exe will frequently fail, which
in turn will annoy us as hell (or at least me).
One warning though: The fetchurl is currently broken and the cross-build
might not work yet for example on mingw32 (mingw-w64 branch on its way),
but the upstream URL has already changed and the new version contains a
bug (not yet tracked down) which breaks our Windows bootstrap process.
So to conclude: If it's already broken, make it at least "less broken".
"Not broken" is coming soon with the merge of the mingw-w64 branch.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Coincidentally, even with this typo, most tests work anyway, so I didn't
notice it in the first place.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Also update 64bit setup.ini and check whether we have a 64 bit stdenv in
order to choose the proper Cygwin version. Otherwise we now have the
setup.ini for 32bit available as well.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
So far, the VMs have always been using the native architecture, because
it was reimporting <nixpkgs> several times. Now, we propagate a list of
packages down to all sub-imports, which not only makes clearer which
dependencies a part actually has, but also will make it easier in case
we want to refactor those parts to use callPackage.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
This now isolates the vmTools integration from the bootstrap process and
thus removes our fixed Windows ISO and product key. The latter can now
be provided by an attribute "windowsImage" to runInWindowsVM.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
This is the last item that was missing to get a fully working
runInWindowsVM function. Apart from checking exit codes, we also now
have preVM/postVM hooks which we can use to write arbitrary constructs
around this architecture, without the need to worry about specific
details.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
This function is quite similar to runInLinuxVM, but also ensures that
the builder is run decoupled of the Nix store and using the userland
inside the VM.
We're now picking up the environment variables saved in the previous
commit.
The reason we suppress all errors from the source operation is that it
would emit a ton of errors because we're trying to set read-only
variables.
Also, detecting whether the origBuilder is using the default builder
from the stdenv is currently a bit of a workaround until we have a
specialized pseudo-cross-stdenv someday in the future[TM].
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Later, when we start the actual builder, we're going to restore those
environment variables. We're using "(set; declare -p)", here, because
the former is just printing _all_ environment variables, even those not
supported, and the latter only lists specifically declared variables,
which also encludes exports.
The "declare -p" command also emits those variables in a format similar
to the "export" command.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
This is mainly to make it easier to quickly change mappings, without
making room for errors such as typos.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Cygwin initializes mounts on _every_ login via SSH and doesn't keep them
consistently like on Unix systems, that's why we need to also add fstab
entries for the bind mounts to the store and xchg shares.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
We now map all guest accounts to the root user, because in the end the
permissions of the current user boil down to the build user of the Nix
builder of the host. That way it's not possible to gain more permissions
at all and just makes the VM communication a lot easier.
However, setting "writable" to yes instead of "read only" to no doesn't
change anything here, I just found it to be clearer.
Also, we now no longer need to have a "nobody" user.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
This is done by putting the non-initrd /nix/store into a subdirectory,
which we then chroot to and pass along the SSH command.
Also, we now collect the exit code after the chroot command and power
off the VM thereafter, because the store is no longer shadowed and we
have still access to the busybox inside the initrd.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
This should trim down possible dependencies on the base installation and
hereby reduce the need for reinstallation of the damn VM to only changes
that affect the Windows installation and the base Cygwin + OpenSSH
setup.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
This now finally introduces our xchg share and also uses it for
exchanging state while suspending a VM. However, accessing the _real_
Nix store still isn't possible because we're shadowing the directory in
the initrd.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Now we're doing this at the point where we're saving the VM state.
Unfortunately it's not quite right, because the controller VM is shut
down _before_ we're saving the state, so the share gets disconnected
despite autodisconnect being deactivated during setup.
We can get around this issue by finally introducing the xchg share,
which is the last item to be implemented before we can merge to master.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Security-wise it's not a big issue because we're still sandboxed, but I
really don't want to write something like \\\\\\\\192.168.0.2\\\\share
in order to set up network shares.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
We're going to do this during the suspendedVM phase, so we're able to
more easily change the shares without reinstalling the whole VM.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
This could possibly cause flapping whenever qemu is too fast in starting
up. As we are running with the shell's -e flag, the socat check also
ensures that the VDE switch is properly started and causes the whole
build to fail, should it not start up within 20 seconds.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
These stages are in particular:
* Install of the bare Windows VM with Cygwin and shut down.
* Boot up the same VM again without the installation media and dump the
VMs memory to state.gz.
* Resume from state.gz and build whatever we want to build.
Every single stage involves a new "controller", which is more like an
abstraction on the Nix side that constructs the madness described in
276b72fb93.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
This is kinda stupid to do every little time the file is automatically
regenerated upstream. But let's see how often that happens and whether
it will become a major annoyance or not, and if yes, we might be forced
to include it in our source tree.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
This SSH key is specifically only for accessing the installed Cygwin
within the Windows VM, so we only need to expose the private key. Yes,
you heard right, the private key. It's not security-relevant because the
machine is completely read-only, only exposed to the filesystem and
networking is not available.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
At least the largest portion of the installer, because in the end we
don't want the installer to *actually* save the state but only prepare
the base image.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
After quite a lot of fighting against Windows and its various
limitations, this new is the base architecture for installing and
accessing the Windows VM and thus the Cygwin environment inside it:
.------------.
.---> | vde_switch |
| `-[#]----[#]-'
| | |
,' .' `---.___
,' 192.168.0.1 `.
| | 192.168.0.2
,' _____[#]____ |
,' | | ______[#]______
| | Windows VM | | .--' |
| |____________| | | |
| | /|\ | .-| |
| .---------. | | | | | |
.-|-| manager |-' | | | | |
| | `---------' | | | | |
| | | | | | |
| | .-------------. | | Samba |
| | | BOOTSTRAP | | | | | |
| | |-------------| | | | | .------|
| `-| spawn VMs |-+--> | | `---| xchg | <-------.
| |-------------| | | .---^------| |
| | install |---. | `-| nixstore | <----. |
| |-------------| | | `----------| | |
|---| suspend VM | | | | | |
| `------.------' | | Controller VM | | |
| | | |_______________| | |
| .--' | /|\ VirtIO
| | __|__________:____________ | |
| \|/ | | `. | | |
| .------------. | | : | | |
| | REAL BUILD | | | .-------^--------. | | |
| |------------| | `-> | serial console | | | |
`-| revive VM | | `----------------' | | |
|------------| |------------. | | |
| build |-->| /nix/store >>>-----------|-' |
|------------| |------------| | |
| collect |<--| xchg >>>-----------|----'
`-----.------' |------------' |
| | |
\|/ | | | __ ___ | |
| |--| | | (__ -|- |
F I N I S H E D | | | |__| ___) | |
|__________________________|
This might look a bit overwhelming, but let me try to explain:
We're starting at the base derivation ("BOOTSTRAP" above), where we
actually install the Cygwin envirenment. Over there we basically fire up
a vde_switch process and two virtual machines: One is the Windows
machine, the other is a NixOS machine, which serves as some kind of
proxy between the host and the Windows machine.
The reason we're doing this, is because we don't have a lot of options
for sharing files between a stock Windows machine and the host. In
earlier experiments, I've tried to communicate with the Windows guest by
using pipes and OpenSSH, but obviously this wasn't a big speed rush (or
to say it bluntly: It was fucking slow).
Using TCP/IP directly for accessing the guest would have been another
option, but it could lead to possible errors when the port or a range of
ports are in use at the Host system. Also, we would need to punch a hole
into the sandbox of the Nix builder (as it doesn't allow networking),
which in turn will possibly undermine deterministic builds/runs (well,
at least as deterministic as it can be, we're running Windows,
remember?).
So, let's continue: The responsibility of the NixOS (controller) VM is
to just wait until an SSH port becomes available on the Windows VM,
whereas the Windows VM itself is installed using an unattended
installation file provided via a virtual floppy image.
With the installation of the basic Windows OS, we directly install
Cygwin and start up an OpenSSH service.
At this point the bootstrapping is almost finished and as soon as the
port is available, the controller VM sets up Samba shares and makes it
available as drive letters within Windows and as bind mounts (for
example /nix/store) within Cygwin.
Finally we're making a snapshot of the memory of the Windows VM in order
to revive it within a few seconds when we want to build something.
Now, the build process itself is fairly straightforward: Revive VM and
build based on existing store derivations and collect the result _and_
the exit code from the xchg share/directory.
Conclusion: This architecture may sound a bit complicated, but we're
trying to achieve deterministic and reproducable builds and/or test
runs.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
