JakeHillion/nixpkgs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
250,501 Commits 2 Branches 0 Tags 13 GiB
440a3a497c
Commit Graph

19016 Commits

Author SHA1 Message Date
Graham Christensen
c06b97175b
Merge pull request #102173 from grahamc/create-amis 
create-amis.sh: fixup shellcheck issues, improve error logging, and add configurable service names
 2020-10-30 16:13:18 -04:00
Graham Christensen
82578fc725
Merge pull request #102172 from grahamc/stage-1-datestamps 
stage-1: add datestamps to logs
 2020-10-30 16:13:02 -04:00
Graham Christensen
b34cf366aa
Merge pull request #102171 from grahamc/faster-ext-resize 
stage-1: modprobe ext{2,3,4} before resizing (so resizing takes less than 45 minutes)
 2020-10-30 16:12:50 -04:00
WORLDofPEACE
214af51225
Merge pull request #101067 from deviant/remove-caddy-agree 
nixos/caddy: remove services.caddy.agree
 2020-10-30 16:02:44 -04:00
Graham Christensen
d77ddf2a40
nixos.amazonAmi: use legacy+gpt disk images to support partitions >2T 2020-10-30 15:50:25 -04:00
Graham Christensen
d78aa080f5
make-disk-image: support legacy+gpt 2020-10-30 15:50:24 -04:00
Doron Behar
77e081bb2b nixos/sddm: Use libsForQt514.sddm if needed (for lxqt) 
Currently lxqt is a desktop environment that's compiled against qt514.
To avoid possible issues (#101369), we (hopefully) use the same qt
version as the desktop environment at hand. LXQT should move to qt515,
and for the long term the correct qt version should be inherited by the
sddm module.
 2020-10-30 20:37:59 +02:00
Doron Behar
e681f442c9 nixos/plasma: Fix attribute path to kinit 2020-10-30 20:37:58 +02:00
Graham Christensen
c851030763
amazon-image: random.trust_cpu=on to cut 10s from boot 
Ubuntu and other distros already have this set via kernel config.
 2020-10-30 13:45:19 -04:00
Graham Christensen
74a577b293
create-amis: improve wording around the service name's IAM role 
Co-authored-by: Cole Helbling <cole.e.helbling@outlook.com>
 2020-10-30 12:40:17 -04:00
Graham Christensen
ece5c0f304
stage-1: modprobe ext{2,3,4} before resizing 
I noticed booting a system with an ext4 root which expanded to 5T took
quite a long time (12 minutes in some cases, 43(!) in others.)

I changed stage-1 to run `resize2fs -d 62` for extra debug output and
timing information. It revealed the adjust_superblock step taking
almost all of the time:

    [Fri Oct 30 11:10:15 UTC 2020] zero_high_bits_in_metadata: Memory used: 132k/0k (63k/70k), time:  0.00/ 0.00/ 0.00
    [Fri Oct 30 11:21:09 UTC 2020] adjust_superblock: Memory used: 396k/4556k (295k/102k), time: 654.21/ 0.59/ 5.13

but when I ran resize2fs on a disk with the identical content growing
to the identical target size, it would only take about 30 seconds. I
looked at what happened between those two steps in the fast case with
strace and found:

```
   235	getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=1795}, ru_stime={tv_sec=0, tv_usec=3590}, ...}) = 0
   236	write(1, "zero_high_bits_in_metadata: Memo"..., 84zero_high_bits_in_metadata: Memory used: 132k/0k (72k/61k), time:  0.00/ 0.00/ 0.00
   237	) = 84
   238	gettimeofday({tv_sec=1604061278, tv_usec=480147}, NULL) = 0
   239	getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=1802}, ru_stime={tv_sec=0, tv_usec=3603}, ...}) = 0
   240	gettimeofday({tv_sec=1604061278, tv_usec=480192}, NULL) = 0
   241	mmap(NULL, 2564096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fa3c7355000
   242	access("/sys/fs/ext4/features/lazy_itable_init", F_OK) = 0
   243	brk(0xf85000)                           = 0xf85000
   244	brk(0xfa6000)                           = 0xfa6000
   245	gettimeofday({tv_sec=1604061278, tv_usec=538828}, NULL) = 0
   246	getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=58720}, ru_stime={tv_sec=0, tv_usec=3603}, ...}) = 0
   247	write(1, "adjust_superblock: Memory used: "..., 79adjust_superblock: Memory used: 396k/2504k (305k/92k), time:  0.06/ 0.06/ 0.00
   248	) = 79
   249	gettimeofday({tv_sec=1604061278, tv_usec=539119}, NULL) = 0
   250	getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=58812}, ru_stime={tv_sec=0, tv_usec=3603}, ...}) = 0
   251	gettimeofday({tv_sec=1604061279, tv_usec=939}, NULL) = 0
   252	getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=520411}, ru_stime={tv_sec=0, tv_usec=3603}, ...}) = 0
   253	write(1, "fix_uninit_block_bitmaps 2: Memo"..., 88fix_uninit_block_bitmaps 2: Memory used: 396k/2504k (305k/92k), time:  0.46/ 0.46/ 0.00
   254	) = 88
```

In particular the access to /sys/fs seemed interesting. Looking
at the source of resize2fs:

```
[root@ip-172-31-22-182:~/e2fsprogs-1.45.5]# rg -B2 -A1 /sys/fs/ext4/features/lazy_itable_init .
./resize/resize2fs.c
923-	if (getenv("RESIZE2FS_FORCE_LAZY_ITABLE_INIT") ||
924-	    (!getenv("RESIZE2FS_FORCE_ITABLE_INIT") &&
925:	     access("/sys/fs/ext4/features/lazy_itable_init", F_OK) == 0))
926-		lazy_itable_init = 1;
```

I confirmed /sys is mounted, and then found a bug suggesting the
ext4 module is maybe not loaded:
https://bugzilla.redhat.com/show_bug.cgi?id=1071909

My home server doesn't have ext4 loaded and had 3T to play with, so
I tried (and succeeded with) replicating the issue locally:

```
[root@kif:/scratch]# lsmod | grep -i ext

[root@kif:/scratch]# zfs create -V 3G rpool/scratch/ext4

[root@kif:/scratch]# time mkfs.ext4 /dev/zvol/rpool/scratch/ext4
mke2fs 1.45.5 (07-Jan-2020)
Discarding device blocks: done
Creating filesystem with 786432 4k blocks and 196608 inodes
Filesystem UUID: 560a4a8f-93dc-40cc-97a5-f10049bf801f
Superblock backups stored on blocks:
	32768, 98304, 163840, 229376, 294912

Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done

real	0m2.261s
user	0m0.000s
sys	0m0.025s

[root@kif:/scratch]# zfs set volsize=3T rpool/scratch/ext4

[root@kif:/scratch]# time resize2fs -d 62 /dev/zvol/rpool/scratch/ext4
resize2fs 1.45.5 (07-Jan-2020)
fs has 11 inodes, 1 groups required.
fs requires 16390 data blocks.
With 1 group(s), we have 22234 blocks available.
Last group's overhead is 10534
Need 16390 data blocks in last group
Final size of last group is 26924
Estimated blocks needed: 26924
Extents safety margin: 49
Resizing the filesystem on /dev/zvol/rpool/scratch/ext4 to 805306368 (4k) blocks.
read_bitmaps: Memory used: 132k/0k (63k/70k), time:  0.00/ 0.00/ 0.00
read_bitmaps: I/O read: 1MB, write: 0MB, rate: 3802.28MB/s
fix_uninit_block_bitmaps 1: Memory used: 132k/0k (63k/70k), time:  0.00/ 0.00/ 0.00
resize_group_descriptors: Memory used: 132k/0k (68k/65k), time:  0.00/ 0.00/ 0.00
move_bg_metadata: Memory used: 132k/0k (68k/65k), time:  0.00/ 0.00/ 0.00
zero_high_bits_in_metadata: Memory used: 132k/0k (68k/65k), time:  0.00/ 0.00/ 0.00
```

here it got stuck for quite some time ... straceing this 20 minutes in revealed this in a tight loop:

```
getuid()                                = 0
geteuid()                               = 0
getgid()                                = 0
getegid()                               = 0
prctl(PR_GET_DUMPABLE)                  = 1 (SUID_DUMP_USER)
fallocate(3, FALLOC_FL_ZERO_RANGE, 2222649901056, 2097152) = 0
fsync(3)                                = 0
```

it finally ended 43(!) minutes later:

```
adjust_superblock: Memory used: 264k/3592k (210k/55k), time: 2554.03/ 0.16/15.07
fix_uninit_block_bitmaps 2: Memory used: 264k/3592k (210k/55k), time:  0.16/ 0.16/ 0.00
blocks_to_move: Memory used: 264k/3592k (211k/54k), time:  0.00/ 0.00/ 0.00
Number of free blocks: 755396/780023556, Needed: 0
block_mover: Memory used: 264k/3592k (216k/49k), time:  0.05/ 0.01/ 0.00
block_mover: I/O read: 1MB, write: 0MB, rate: 18.68MB/s
inode_scan_and_fix: Memory used: 264k/3592k (216k/49k), time:  0.00/ 0.00/ 0.00
inode_ref_fix: Memory used: 264k/3592k (216k/49k), time:  0.00/ 0.00/ 0.00
move_itables: Memory used: 264k/3592k (216k/49k), time:  0.00/ 0.00/ 0.00
calculate_summary_stats: Memory used: 264k/3592k (216k/49k), time: 16.35/16.35/ 0.00
fix_resize_inode: Memory used: 264k/3592k (222k/43k), time:  0.04/ 0.00/ 0.00
fix_resize_inode: I/O read: 1MB, write: 0MB, rate: 22.80MB/s
fix_sb_journal_backup: Memory used: 264k/3592k (222k/43k), time:  0.00/ 0.00/ 0.00
overall resize2fs: Memory used: 264k/3592k (222k/43k), time: 2570.90/16.68/15.07
overall resize2fs: I/O read: 1MB, write: 1MB, rate: 0.00MB/s
The filesystem on /dev/zvol/rpool/scratch/ext4 is now 805306368 (4k) blocks long.

real	43m1.943s
user	0m16.761s
sys	0m15.069s
```

I then cleaned up and recreated the zvol, loaded the ext4 module, created the ext4 fs,
resized the volume, and resize2fs'd and it went quite quickly:

```
[root@kif:/scratch]# zfs destroy rpool/scratch/ext4

[root@kif:/scratch]# zfs create -V 3G rpool/scratch/ext4

[root@kif:/scratch]# modprobe ext4

[root@kif:/scratch]# time resize2fs -d 62 /dev/zvol/rpool/scratch/ext4

[root@kif:/scratch]# time mkfs.ext4 /dev/zvol/rpool/scratch/ext4
mke2fs 1.45.5 (07-Jan-2020)
Discarding device blocks: done
Creating filesystem with 786432 4k blocks and 196608 inodes
Filesystem UUID: 5b415f2f-a8c4-4ba0-ac1d-78860de77610
Superblock backups stored on blocks:
	32768, 98304, 163840, 229376, 294912

Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done

real	0m1.013s
user	0m0.001s
sys	0m0.023s

[root@kif:/scratch]# zfs set volsize=3T rpool/scratch/ext4

[root@kif:/scratch]# time resize2fs -d 62 /dev/zvol/rpool/scratch/ext4
resize2fs 1.45.5 (07-Jan-2020)
fs has 11 inodes, 1 groups required.
fs requires 16390 data blocks.
With 1 group(s), we have 22234 blocks available.
Last group's overhead is 10534
Need 16390 data blocks in last group
Final size of last group is 26924
Estimated blocks needed: 26924
Extents safety margin: 49
Resizing the filesystem on /dev/zvol/rpool/scratch/ext4 to 805306368 (4k) blocks.
read_bitmaps: Memory used: 132k/0k (63k/70k), time:  0.00/ 0.00/ 0.00
read_bitmaps: I/O read: 1MB, write: 0MB, rate: 3389.83MB/s
fix_uninit_block_bitmaps 1: Memory used: 132k/0k (63k/70k), time:  0.00/ 0.00/ 0.00
resize_group_descriptors: Memory used: 132k/0k (68k/65k), time:  0.00/ 0.00/ 0.00
move_bg_metadata: Memory used: 132k/0k (68k/65k), time:  0.00/ 0.00/ 0.00
zero_high_bits_in_metadata: Memory used: 132k/0k (68k/65k), time:  0.00/ 0.00/ 0.00
adjust_superblock: Memory used: 264k/1540k (210k/55k), time:  0.02/ 0.02/ 0.00
fix_uninit_block_bitmaps 2: Memory used: 264k/1540k (210k/55k), time:  0.15/ 0.15/ 0.00
blocks_to_move: Memory used: 264k/1540k (211k/54k), time:  0.00/ 0.00/ 0.00
Number of free blocks: 755396/780023556, Needed: 0
block_mover: Memory used: 264k/3592k (216k/49k), time:  0.01/ 0.01/ 0.00
block_mover: I/O read: 1MB, write: 0MB, rate: 157.11MB/s
inode_scan_and_fix: Memory used: 264k/3592k (216k/49k), time:  0.00/ 0.00/ 0.00
inode_ref_fix: Memory used: 264k/3592k (216k/49k), time:  0.00/ 0.00/ 0.00
move_itables: Memory used: 264k/3592k (216k/49k), time:  0.00/ 0.00/ 0.00

calculate_summary_stats: Memory used: 264k/3592k (216k/49k), time: 16.20/16.20/ 0.00
fix_resize_inode: Memory used: 264k/3592k (222k/43k), time:  0.00/ 0.00/ 0.00
fix_resize_inode: I/O read: 1MB, write: 0MB, rate: 5319.15MB/s
fix_sb_journal_backup: Memory used: 264k/3592k (222k/43k), time:  0.00/ 0.00/ 0.00
overall resize2fs: Memory used: 264k/3592k (222k/43k), time: 16.45/16.38/ 0.00
overall resize2fs: I/O read: 1MB, write: 1MB, rate: 0.06MB/s
The filesystem on /dev/zvol/rpool/scratch/ext4 is now 805306368 (4k) blocks long.

real	0m17.908s
user	0m16.386s
sys	0m0.079s
```

Success!
 2020-10-30 12:18:23 -04:00
Graham Christensen
a179781696
stage-1: add datestamps to logs 
When the stage-1 logs get imported in to the journal, they all get
loaded with the same timestamp. This makes it difficult to identify
what might be taking a long time in early boot.
 2020-10-30 12:16:35 -04:00
Graham Christensen
2bf1fc0345
create-amis: allow customizing the service role name 
The complete setup on the AWS end can be configured
with the following Terraform configuration. It generates
a ./credentials.sh which I just copy/pasted in to the
create-amis.sh script near the top. Note: the entire stack
of users and bucket can be destroyed at the end of the
import.

    variable "region" {
      type = string
    }
    variable "availability_zone" {
      type = string
    }

    provider "aws" {
      region = var.region
    }

    resource "aws_s3_bucket" "nixos-amis" {
      bucket_prefix = "nixos-amis-"
      lifecycle_rule {
        enabled = true
        abort_incomplete_multipart_upload_days = 1
        expiration {
          days = 7
        }
      }
    }

    resource "local_file" "credential-file" {
      file_permission = "0700"
      filename = "${path.module}/credentials.sh"
      sensitive_content = <<SCRIPT
    export service_role_name="${aws_iam_role.vmimport.name}"
    export bucket="${aws_s3_bucket.nixos-amis.bucket}"
    export AWS_ACCESS_KEY_ID="${aws_iam_access_key.uploader.id}"
    export AWS_SECRET_ACCESS_KEY="${aws_iam_access_key.uploader.secret}"
    SCRIPT
    }

    # The following resources are for the *uploader*
    resource "aws_iam_user" "uploader" {
      name = "nixos-amis-uploader"
    }

    resource "aws_iam_access_key" "uploader" {
      user = aws_iam_user.uploader.name
    }

    resource "aws_iam_user_policy" "upload-to-nixos-amis" {
      user = aws_iam_user.uploader.name

      policy = data.aws_iam_policy_document.upload-policy-document.json
    }

    data "aws_iam_policy_document" "upload-policy-document" {
      statement {
        effect = "Allow"

        actions = [
          "s3:ListBucket",
          "s3:GetBucketLocation",
        ]

        resources = [
          aws_s3_bucket.nixos-amis.arn
        ]
      }

      statement {
        effect = "Allow"

        actions = [
          "s3:PutObject",
          "s3:GetObject",
          "s3:DeleteObject",
        ]

        resources = [
          "${aws_s3_bucket.nixos-amis.arn}/*"
        ]
      }

      statement {
        effect = "Allow"
        actions = [
          "ec2:ImportSnapshot",
          "ec2:DescribeImportSnapshotTasks",
          "ec2:DescribeImportSnapshotTasks",
          "ec2:RegisterImage",
          "ec2:DescribeImages"
        ]
        resources = [
          "*"
        ]
      }
    }

    # The following resources are for the *vmimport service user*
    # See: https://docs.aws.amazon.com/vm-import/latest/userguide/vmie_prereqs.html#vmimport-role
    resource "aws_iam_role" "vmimport" {
      assume_role_policy = data.aws_iam_policy_document.vmimport-trust.json
    }

    resource "aws_iam_role_policy" "vmimport-access" {
      role = aws_iam_role.vmimport.id
      policy = data.aws_iam_policy_document.vmimport-access.json
    }

    data "aws_iam_policy_document" "vmimport-access" {
      statement {
        effect = "Allow"
        actions = [
          "s3:GetBucketLocation",
          "s3:GetObject",
          "s3:ListBucket",
        ]
        resources = [
          aws_s3_bucket.nixos-amis.arn,
          "${aws_s3_bucket.nixos-amis.arn}/*"
        ]
      }
      statement {
        effect = "Allow"
        actions = [
          "ec2:ModifySnapshotAttribute",
          "ec2:CopySnapshot",
          "ec2:RegisterImage",
          "ec2:Describe*"
        ]
        resources = [
          "*"
        ]
      }
    }

    data "aws_iam_policy_document" "vmimport-trust" {
      statement {
        effect = "Allow"
        principals {
          type = "Service"
          identifiers = [ "vmie.amazonaws.com" ]
        }

        actions = [
          "sts:AssumeRole"
        ]

        condition {
          test = "StringEquals"
          variable = "sts:ExternalId"
          values = [ "vmimport" ]
        }
      }
    }
 2020-10-30 12:12:08 -04:00
Graham Christensen
e253de8a77
create-amis.sh: log the full response if describing the import snapshot tasks fails 2020-10-30 12:08:01 -04:00
Graham Christensen
f92a883ddb
nixos ec2/create-amis.sh: shellcheck: $ is not needed in arithmetic 2020-10-30 12:08:01 -04:00
Graham Christensen
7dac8470cf
nixos ec2/create-amis.sh: shellcheck: explicitly make the additions to block_device_mappings single strings 2020-10-30 12:08:00 -04:00
Graham Christensen
a66a22ca54
nixos ec2/create-amis.sh: shellcheck: read without -r mangles backslashes 2020-10-30 12:08:00 -04:00
Graham Christensen
baf7ed3f24
nixos ec2/create-amis.sh: shellcheck: SC2155: Declare and assign separately to avoid masking return values. 2020-10-30 12:07:59 -04:00
Graham Christensen
f5994c208d
nixos ec2/create-amis.sh: shellcheck: quote state_dir reference 2020-10-30 12:07:59 -04:00
Graham Christensen
c76692192a
nixos ec2/create-amis.sh: shellcheck: quote region references 2020-10-30 12:07:49 -04:00
Timo Kaufmann
83f48e8348
Merge pull request #95011 from Atemu/undervolt-pl 
undervolt: expose power limits as Nixopts
 2020-10-30 09:32:50 +01:00
Michele Guerini Rocco
1102a46ffe
Merge pull request #101724 from pickfire/patch-3 
fontdir: add ttc to font regex
 2020-10-30 08:41:34 +01:00
Benjamin Hipple
e00752079e
Merge pull request #102018 from 1000101/blockbook-frontend 
blockbook-frontend: fix&update extraConfig example
 2020-10-29 22:30:07 -04:00
Florian Klink
b8d59e93c8 nixos/networkd: allow RouteMetric= in [DHCPv6] section 2020-10-29 19:47:42 +01:00
talyz
89e83833af
nixos/keycloak: Add support for MySQL and external DBs with SSL 
- Add support for using MySQL as an option to PostgreSQL.
- Enable connecting to external DBs with SSL
- Add a database port config option
 2020-10-29 12:47:10 +01:00
talyz
d1d3c86c70
rl-2103: Note the addition of the Keycloak service 2020-10-29 12:08:06 +01:00
talyz
c6e4388449
nixos/keycloak: Add documentation 2020-10-29 12:08:01 +01:00
talyz
fe5a16aee6
nixos/keycloak: Document internal functions 2020-10-29 12:07:55 +01:00
talyz
31fe90d6ef
nixos/keycloak: Add test 2020-10-29 12:07:49 +01:00
1000101
4b8611c959 blockbook-frontend: fix&update extraConfig example 2020-10-29 11:41:41 +01:00
Philipp
fc856b89e5
nixos/murmur: add murmur group, don't run as nogroup 
fixes #101980
 2020-10-29 10:32:04 +01:00
Martin Weinelt
55746e0a4b
Merge pull request #98187 from mweinelt/nixos/babeld 
nixos/babeld: lock down service
 2020-10-29 01:24:11 +01:00
Minijackson
3fce272478 nixos/shiori: harden service with systemd 2020-10-28 20:46:30 +01:00
Maximilian Bosch
ca45bb574d
nixos/rl-2009: minor typo fix 2020-10-28 19:38:28 +01:00
Thomas Depierre
63caecee7d riak-cs: delete 2020-10-28 19:31:33 +01:00
Linus Heckemann
2b06415ca1
Merge pull request #101370 from m1cr0man/ssl-test-certs 
nixos/acme: Permissions and tests fixes
 2020-10-28 17:21:57 +01:00
Andreas Rammhold
c127653b72
Merge pull request #101887 from jonringer/minor-release-notes-adjustment 
nixos/docs/rl-2009.xml: grafana: description, example agreement
 2020-10-28 02:38:55 +01:00
Jonathan Ringer
3963954fc8
nixos/docs/rl-2009.xml: grafana: description, example agreement 2020-10-27 17:50:39 -07:00
Markus S. Wamser
a0cc1243cc doc: 20.09 release notes: remove duplicate service list entry 
opt-services.foldingathome.enable was listed twice
 2020-10-27 13:43:44 -07:00
davidak
4166a767de doc: improve 20.09 release notes 2020-10-27 21:11:22 +01:00
talyz
513599a6d7
nixos/keycloak: Init 2020-10-27 19:01:26 +01:00
AmineChikhaoui
8cae6703ef
ec2-amis: add stable NixOS 20.09 AMIs 
Fixes #101694
 2020-10-27 08:52:15 -04:00
WORLDofPEACE
5a08ab936b rl-2009: release on a Tuesday 
Because hydra took it's good old time
 2020-10-27 03:03:43 -04:00
Ryan Mulligan
178d373a8a
Merge pull request #83687 from primeos/wshowkeys 
wshowkeys: init at 2020-03-29
 2020-10-26 18:55:16 -07:00
WORLDofPEACE
d1b239703c
Merge pull request #101811 from jonringer/rl-2009-contributions 
release-notes-2009: add contributions section
 2020-10-26 21:49:20 -04:00
Jonathan Ringer
51ca426eb5
release-notes-2009: add contributions section 2020-10-26 18:36:12 -07:00
Andreas Rammhold
1dc37370c4
Merge pull request #101805 from andir/unbreak-tarball-job 
nixos/tests: fix wrong inherit that passes on the nodes attrs
 2020-10-27 01:29:36 +01:00
Jonathan Ringer
366bebd53a README.md: update stable release links 2020-10-26 20:10:29 -04:00
Andreas Rammhold
eba8f5425f
nixos/tests: fix wrong inherit that passes on the nodes attrs 
The hydra tarball step would fail due to the nodes attribute not being
properly inherited. Since we can't execute all the tests and release
steps locally anymore (thanks to the JSONification and faster hydra
eval) these errors will probably keep in appearing.

This is hopefully the last of those introduced by me test runner
refactoring.

Error was seen on hydra (https://hydra.nixos.org/build/129282411):
> unpacking sources
> unpacking source archive /nix/store/bp95x52h6nv3j8apxrryyj2rviw682k1-source
> source root is source
> patching sources
> autoconfPhase
> No bootstrap, bootstrap.sh, configure.in or configure.ac. Assuming this is not an GNU Autotools package.
> configuring
> release name is nixpkgs-21.03pre249116.1088f059401
> git-revision is 1088f05940
> building
> no Makefile, doing nothing
> running tests
> warning: you did not specify '--add-root'; the result might be removed by the garbage collector
> warning: you did not specify '--add-root'; the result might be removed by the garbage collector
> checking Nixpkgs on i686-linux
> checking Nixpkgs on x86_64-linux
> checking Nixpkgs on x86_64-darwin
> checking eval-release.nix
> trace: `mkStrict' is obsolete; use `mkOverride 0' instead.
> trace: `lib.nixpkgsVersion` is deprecated, use `lib.version` instead!
> trace: warning: lib.readPathsFromFile is deprecated, use a list instead
> trace: Warning: `showVal` is deprecated and will be removed in the next release, please use `traceSeqN`
> trace: lib.zip is deprecated, use lib.zipAttrsWith instead
> checking find-tarballs.nix
> trace: `mkStrict' is obsolete; use `mkOverride 0' instead.
> trace: `lib.nixpkgsVersion` is deprecated, use `lib.version` instead!
> trace: warning: lib.readPathsFromFile is deprecated, use a list instead
> trace: Warning: `showVal` is deprecated and will be removed in the next release, please use `traceSeqN`
> trace: lib.zip is deprecated, use lib.zipAttrsWith instead
> error: while evaluating anonymous function at /build/source/maintainers/scripts/find-tarballs.nix:6:1, called from undefined position:
> while evaluating 'operator' at /build/source/maintainers/scripts/find-tarballs.nix:27:16, called from undefined position:
> while evaluating 'immediateDependenciesOf' at /build/source/maintainers/scripts/find-tarballs.nix:39:29, called from /build/source/maintainers/scripts/find-tarballs.nix:27:44:
> while evaluating anonymous function at /build/source/lib/attrsets.nix:234:10, called from undefined position:
> while evaluating anonymous function at /build/source/maintainers/scripts/find-tarballs.nix:40:37, called from /build/source/lib/attrsets.nix:234:16:
> while evaluating 'derivationsIn' at /build/source/maintainers/scripts/find-tarballs.nix:42:19, called from /build/source/maintainers/scripts/find-tarballs.nix:40:40:
> while evaluating 'canEval' at /build/source/maintainers/scripts/find-tarballs.nix:48:13, called from /build/source/maintainers/scripts/find-tarballs.nix:43:9:
> while evaluating the attribute 'nodes' at /build/source/nixos/lib/testing-python.nix:195:23:
> attribute 'nodes' missing, at /build/source/nixos/lib/testing-python.nix:193:16
> build time elapsed:  0m0.122s 0m0.043s 17m51.526s 0m56.668s
> builder for '/nix/store/96rk3c74vrk6m3snm7n6jhis3j640pn4-nixpkgs-tarball-21.03pre249116.1088f059401.drv' failed with exit code 1
 2020-10-27 00:10:31 +01:00
Tim Steinbach
c851af868f
docker-edge: Fix test 2020-10-26 16:25:37 -04:00
WORLDofPEACE
ace69f768b Revert "nixos/pantheon: install nixos wallpaper" 
This reverts commit 5100e4f250.

Fixes https://github.com/NixOS/nixpkgs/issues/100293
Though it's only a workaround for now.
See https://github.com/elementary/switchboard-plug-pantheon-shell/issues/246#issuecomment-716713218
We trigger the broken scenario where we have two subdirectories. Reverting
that commit undoes this.
 2020-10-26 13:45:19 -04:00
Nick Hu
921287e7f0
Merge pull request #97726 from NickHu/pam_gnupg 
pam: add support for pam_gnupg
 2020-10-26 15:27:13 +00:00
Andreas Rammhold
1088f05940
Merge pull request #101598 from andir/nixos-build-vms-qemu 
nixos/tests: follow-up to the closure reduction PR
 2020-10-26 14:19:45 +01:00
rnhmjoj
9e04bba0af nixos/dnscrypt-wrapper: fix key rotation script 
Fix an error in the validation code when the public key is in a
nonstandard location. The check command fails and the key is
incorrectly assumed to be expiring.
 2020-10-26 13:07:49 +01:00
Jörg Thalheim
dfaa313d43
Merge pull request #101737 from aneeshusa/nginx-allow-unsetting-ssl_ciphers 
nixos/nginx: Allow unsetting ssl_ciphers
 2020-10-26 06:41:19 +01:00
Aneesh Agrawal
924035bb97 nixos/nginx: Allow unsetting ssl_ciphers 
When using the Modern config from the Mozilla SSL config generator,
the `ssl_ciphers` parameter does not need to be set
as only TLSv1.3 is permitted and all of its ciphers are reasonable.
 2020-10-26 00:35:29 -04:00
Jörg Thalheim
b7a2a5f967
nixos/nextcloud: fix several php endpoints 2020-10-26 05:06:43 +01:00
Klemens Nanni
0b8a6e787c
nixos/avahi: Enable IPv6 by default 
Treat it the same as IPv4 (I'm tempted to disable IPv4 by default);
this is the only option I still need to set manually to enjoy IPv6-only
networks including printer discovery!
 2020-10-26 04:06:26 +01:00
Ivan Tham
f6136d06ff
fontdir: add ttc to font regex 
.ttc fonts are used by noto-fonts-cjk
 2020-10-26 10:45:22 +08:00
Klemens Nanni
3216b85713 nixos/system-path: Add mkpasswd(1) 
Generating password hashes, e.g. when adding new users to the system
configuration, should work out-of-the-box and offline.
 2020-10-26 03:40:11 +01:00
Jonathan Ringer
37236c2a23 nixos/doc/rl-20.09: normalize highlights, reorder entries 2020-10-25 17:40:47 -07:00
WORLDofPEACE
4d71306596
Merge pull request #101516 from worldofpeace/gnome-polishing 
GNOME polishing from Q.A findings
 2020-10-25 18:41:34 -04:00
Andreas Rammhold
d4fb7daafd
nixos-build-vms: use the driverInteractive attribute instead 
This reverts commit aab534b894 & uses the
driverInteractive attribute for the test driver instead.

This has the same effect but removes the extra module in the
nixos-build-vms code.
 2020-10-25 20:14:53 +01:00
Andreas Rammhold
73635b859d
nixos/tests: fix runInMachine 
In 5500dc8 we introduced the --keep-vm-state flag and defaulted to that
flag not being set. This lead to the `runInMachine` tests not longer
working and that going unnoticed for quite some time now.
 2020-10-25 20:09:33 +01:00
Andreas Rammhold
fa25d84d13
nixos/tests: fix testDriver reference in runInMachine function 
In a previous commit I broke this as there is no longer one testDriver
but only a function to generate one based on some QEMU inputs.
 2020-10-25 20:09:33 +01:00
Andreas Rammhold
61b09f552c
nixos/tests: format the testing-python.nix file more consistenly 2020-10-25 20:09:33 +01:00
Andreas Rammhold
04100cd281
nixos/tests: restructure test driver so that QEMU is actually overriden 
Previously you would be able to override only the QEMU package to be
used in the test runner. Frankly that doesn't help a lot if you are
trying to get a graphical session. The graphical session requires the
option in the NixOS module system to bet set to the correct QEMU
package.

In this commit I moved most of the test node configuration and
transformations into the `mkDriver` function (previously called
`driver`). The motivation was to be able to create a `driver` instance
with a given QEMU package that will be used consistently througout the
test expression.
 2020-10-25 20:09:33 +01:00
Maximilian Bosch
a3041ab124
Merge pull request #101645 from andir/qemu-tests-fixup 
nixos/tests: only apply qemu parameters if the options are defined
 2020-10-25 19:25:50 +01:00
Konrad Borowski
254d30d4c9 test-driver.py: remove bufsize=1 from Popen calls 
According to Python documentation [0], `bufsize=1` is only meaningful in
text mode. As we don't pass in an argument called `universal_newlines`,
`encoding`, `errors` or `text` the file objects aren't opened in text
mode, which means the argument is ignored with a warning in Python 3.8.

    line buffering (buffering=1) isn't supported in binary mode,
    the default buffer size will be used

This commit removes this warning that appared when using
interactive test driver built with `-A driver`. This is done by
removing `bufsize=1` from Popen calls.

The default parameter when unspecified for `bufsize` is `-1` which
according to the documentation will be interpreted as
`io.DEFAULT_BUFFER_SIZE`. As mentioned by a warning, Python already
uses default buffer size when providing `buffering=1` parameter for
file objects not opened in text mode.

[0]: https://docs.python.org/3/library/subprocess.html#subprocess.Popen
 2020-10-25 16:22:07 +01:00
Benjamin Hipple
f98312fcb5
Merge pull request #79759 from lopsided98/syncoid-no-root 
nixos/syncoid: automatically setup privilege delegation
 2020-10-25 10:40:33 -04:00
Andreas Rammhold
f4d7493162
nixos/tests: only apply qemu parameters if the options are defined 
This fixes an eval error that occurred on hydra with the small channel
and the `nixos.tests.boot.biosCdrom.x86_64-linux` attribute:

> $ nix-instantiate nixos/release-small.nix -A nixos.tests.boot.biosCdrom.x86_64-linux
> warning: unknown setting 'experimental-features'
> error: The option `virtualisation.qemu' does not exist. Definition values:
> - In `/home/andi/dev/nixos/nixpkgs/nixos/modules/testing/test-instrumentation.nix':
>     {
>       consoles = [ ];
>       package = {
> 	_type = "override";
> 	content = <derivation /nix/store/q72h2cdcb9zjgiay5gdgzwddjkbjr7xq-qemu-host-cpu-only-for-vm-tests-5.1.0.drv>;
>     ...
> (use '--show-trace' to show detailed location information)

In bc2188b we changed test test-instrumentation to also set the QEMU
package that is being used. That change unfortunately caused us to
always assing values to the virtualisation.qemu.package option even when
the option is not defined. The original code was explicitly testing for
the consoles case but the then newly extended version did not adjust the
check as the intention was probably not clear.

With this commit we are always ensuring the entire virtualisation.qemu
section exists and can thus drop the individual tests for each of the
sections since the QEMU module always defines both the package and the
consoles option when it's root is defined..
 2020-10-25 13:42:01 +01:00
Scott Worley
f99b6369b1 nixos/tests/hadoop: Use curl --fail 2020-10-25 11:01:31 +01:00
Scott Worley
c25ccf6b4b nixos/tests/docker-tools: Use curl --fail 2020-10-25 11:01:31 +01:00
Scott Worley
f6ecfdac39 nixos/tests/bitcoind: Use curl --fail 2020-10-25 11:01:31 +01:00
Scott Worley
0812bb843d nixos/tests/corerad: Use curl --fail 2020-10-25 11:01:31 +01:00
Scott Worley
ad96a2e52b nixos/tests/cfssl: Use curl --fail 2020-10-25 11:01:31 +01:00
Scott Worley
8adaa71b52 nixos/tests/caddy: Use curl --fail 2020-10-25 11:01:31 +01:00
Scott Worley
ecbd2a8bc1 nixos/tests/cadvisor: Use curl --fail 2020-10-25 11:01:31 +01:00
Scott Worley
eff7338d98 nixos/tests/convos: Use curl --fail 2020-10-25 11:01:31 +01:00
Scott Worley
1abfb504b1 nixos/tests/hitch: Use curl --fail 2020-10-25 11:01:31 +01:00
Scott Worley
19034ed7a4 nixos/tests/metabase: Use curl --fail 2020-10-25 11:01:31 +01:00
Scott Worley
d16e547f2e nixos/tests/wordpress: Use curl --fail 2020-10-25 11:01:31 +01:00
Scott Worley
6e46a88dfb nixos/tests/unit-php: Use curl --fail 2020-10-25 11:01:31 +01:00
Scott Worley
726950775d nixos/tests/sympa: Use curl --fail 2020-10-25 11:01:31 +01:00
Scott Worley
238dbb4517 nixos/tests/limesurvey: Use curl --fail 2020-10-25 11:01:30 +01:00
Scott Worley
1d5130a97f nixos/tests/leaps: Use curl --fail 2020-10-25 11:01:30 +01:00
Scott Worley
22f52cdb79 nixos/tests/osrm-backend: Use curl --fail 2020-10-25 11:01:30 +01:00
Scott Worley
1332215d59 nixos/tests/php: Use curl --fail 2020-10-25 11:01:30 +01:00
Scott Worley
8bf2635afc nixos/tests/oci-containers: Use curl --fail 2020-10-25 11:01:30 +01:00
Scott Worley
7b51945dcb nixos/tests/trickster: Use curl --fail 2020-10-25 11:01:30 +01:00
Scott Worley
ebeb8d7287 nixos/tests/sslh: Use curl --fail 2020-10-25 11:01:30 +01:00
Scott Worley
8a9554bf9b nixos/tests/service-runner: Use curl --fail 2020-10-25 11:01:30 +01:00
Scott Worley
a6a9e3188d nixos/tests/spacecookie: Use curl --fail 2020-10-25 11:01:30 +01:00
Scott Worley
1959ab707c nixos/tests/victoriametrics: Use curl --fail 2020-10-25 11:01:30 +01:00
Scott Worley
c69d4eda3d nixos/tests/uwsgi: Use curl --fail 2020-10-25 11:01:30 +01:00
Scott Worley
ba0eda6cc5 nixos/tests/trezord: Use curl --fail 2020-10-25 11:01:30 +01:00
Scott Worley
47fd1c5356 nixos/tests/morty: Use curl --fail 2020-10-25 11:01:30 +01:00
Scott Worley
4fc64f27c7 nixos/tests/paperless: Use curl --fail 2020-10-25 11:01:30 +01:00
Scott Worley
bbd1f02b16 nixos/tests/hound: Use curl --fail 2020-10-25 11:01:30 +01:00
Scott Worley
bc4f47c19a nixos/tests/go-neb: Use curl --fail 2020-10-25 11:01:30 +01:00