Commit Graph

109 Commits

Author SHA1 Message Date
Artur M. Wolff
cc9b845a83 ci: switch back to pulling gateway@main
Switch back to pulling gateway@main for continuous integration because
https://review.dev.storj.io/c/storj/gateway/+/5834 was merged. This
change is a partial revert of
https://review.dev.storj.io/c/storj/storj/+/5832.

Change-Id: I0a87f6f18a9a863fe92003b76c830335f9253ced
2021-09-28 15:01:04 +00:00
Egon Elbre
9c232cebe1 ci: make deletion a separate step
Deletion as a separate step so it's more understandable what's
actually happening.

Change-Id: Icb3cefe7e4093c56604af633ac0e02226bae42ac
2021-09-28 10:24:43 +00:00
Clement Sam
118e64fa15 jenkinsfile: bump build timeout
Jenkins build consistently times out due to the enormous tests we have now.
Bumping the timeout to 40 fixes it for now.

Change-Id: Ie96cc5ec3e452744d5a7873853d14befe367bd7e
2021-09-27 18:31:16 +00:00
Egon Elbre
8036644422 ci: try fix builds
This chmod is combined with "Clean before checkout" behaviour
that deletes everything, except .git. This should give us
the sufficient ordering to ensure that everything has the
correct permissions.

Story so far:

Checkout with "Wipe out repository & force clone" didn't work,
because the chmod would've run after trying to delete.

Deleting as a post action doesn't work, because during timeouts
the always steps don't get executed sometimes.

We cannot easily run docker with non root, due to how our ci
image works.

Change-Id: Id4c1605d39ae8d7722ba3dd1ab5df8618de3309b
2021-09-27 16:18:04 +00:00
Egon Elbre
7fd3466925 ci: remove deleting workspace steps
Moved the deleting workspace to pipeline configuration.

Change-Id: I7c94d1a9bac514f1824988d21dfb3a144b10ca7e
2021-09-27 15:49:34 +03:00
Egon Elbre
8ef03b0967 ci: cleanup ws before build, fix gateway install
Jenkins uses the same folder for different PR-s. Sometimes other runs do
not cleanup after themselves (e.g. timeout), hence add a cleanWs step to
ensure we delete files in the workspace.

gateway-st introduced a replace directive in go.mod, which does not work
with go install. Hardcode to the last version without the directive.
Using this fix to unblock ci builds.

Change-Id: I5e5d75bf47e30a5a8b6d835867c0c9176f25e08a
2021-09-23 15:57:47 +03:00
Egon Elbre
bd4017b2a6 ci: fix logical races between tools
Currently some building may generate some additional files,
similary, we want to run check-clean-directory after everything
has finished.

Change-Id: I4d700896094257e65cc5197c468f6d752024ddd9
2021-09-23 12:16:31 +03:00
Egon Elbre
6e660cecdd Jenkinsfile: test cross-compile and bump deps
Change-Id: I47091de2e80bd96fcada616f75e1db07a59fb0c1
2021-09-16 18:59:31 +03:00
Egon Elbre
25971c581f Jenkinsfile: clarify modfile linting
Change-Id: I0ac0bfa48386d3430eb8fb5a92b57daefc23998f
2021-08-31 15:13:45 +03:00
storj-antonio
3510bc036e testsuite: fix testsuite/go.mod and enable linters
This change implements verification of the testsuite
and resolves missing go.sum module errors.

Change-Id: I7b513e74532581b8878d50aa2c5ddca12bbef2d0
2021-08-31 12:10:18 +00:00
Sean Harvey
d22ecf9ec0 Jenkinsfile.public: pull latest ci image instead of building
Change-Id: Ia0e6863fdd00127336839d4d0b81306611aee177
2021-08-17 06:47:29 +12:00
Egon Elbre
f807518477 ci: try fix go build scanning node_modules directory
Add a go.mod to node_modules folder, that should prevent

  go build ./...

from scanning it, however, it's slightly hacky.

Change-Id: I073150704bd90afd8ccf38a969e0a1c725d25137
2021-08-12 18:11:11 +03:00
Egon Elbre
6b153192a3 web/satellite: fix lint issues
After migrating to eslint some errors were disabled to make it easier to
migrate.

This enables all the lint rules and treats all warnings as a build
failure. Similarly, CI won't automatically try to fix mistakes.

Change-Id: I80f808af026fc51bed90421b3b24737994a52094
2021-08-10 09:22:19 +00:00
Egon Elbre
19852a767b web/storagenode: fix lint issues
After migrating to eslint some errors were disabled to make it easier to
migrate.

This enables all the lint rules and treats all warnings as a build
failure. Similarly, CI won't automatically try to fix mistakes.

Change-Id: Iff9c4a59401900fc395cd566dd328f3a9c688a12
2021-08-05 15:45:55 +00:00
Egon Elbre
a3e78491b9 web/multinode: fix lint issues
After migrating to eslint some errors were disabled to make it easier to
migrate.

This enables all the lint rules and treats all warnings as a build
failure. Similarly, CI won't automatically try to fix mistakes.

Change-Id: I6d5ef39d476a38dd9a6ab98e3fb94da4c3ab2e57
2021-08-04 17:00:00 +03:00
Egon Elbre
1cd13adc1c ci: clearer initial build
We can run a lot of the code building and downloading steps in parallel.
We were using Makefile, however, that doesn't integrate with Jenkins,
which means, when there's a failure, you have no clue where.

This separates the concurrent build steps and makes it clear what we are
running concurrently.

Change-Id: I1c1dccd09dab67d2ef428a58d5eb9e004b11a74c
2021-06-09 12:30:31 +00:00
Yaroslav Vorobiov
f0d60a6ec2 Jenkins: add multinode UI stage
Change-Id: I77ed791ff9dbb5b046450ab625c1763b34691888
2021-05-21 15:49:31 +03:00
Egon Elbre
2ae80690cb satellite/metabase/metabasetest: remove STORJ_TEST_DATABASES
Initially metabase was developed separately and it was useful to have a
separate environment flag for tests, however, it's more convenient to
use the same as rest of the testsuite.

Change-Id: Ia4d79be27ce5911cbae68d57cdf0b30f63459444
2021-05-11 13:31:01 +00:00
Egon Elbre
7802ab714f pkg/,private/: merge with private package
Initially there were pkg and private packages, however for all practical
purposes there's no significant difference between them. It's clearer to
have a single private package - and when we do get a specific
abstraction that needs to be reused, we can move it to storj.io/common
or storj.io/private.

Change-Id: Ibc2036e67f312f5d63cb4a97f5a92e38ae413aa5
2021-04-23 16:37:28 +03:00
Michał Niewrzał
a5bc38b197 build: replace 'multipart-upload' branch occurences
We removed `multipart-upload` branches but we have few leftovers.

Change-Id: I74f637748a248cb286cd190a3a8f895ef85f3940
2021-03-31 11:06:47 +02:00
Egon Elbre
74b243d1ca certificate: move protobuf to storj/storj repository
Change-Id: I4689318304b54121a65e28860430d39d94ac5bd6
2021-03-29 12:59:49 +00:00
Michał Niewrzał
237782813b Merge remote-tracking branch 'origin/multipart-upload'
Change-Id: If6c5a450b238adab55d1e0dea67d01e5f5768a9f
2021-03-23 09:44:49 +01:00
Ivan Fraixedes
536990dcf6 Jenkins: Run integration test Redis unavailability
Make Jenkins to run the integration tests that verifies that the
satellite can operates when Redis is unavailable.

Change-Id: Idace3ffb84c788f2af2c6e24eec1d63fb40c263a
2021-03-22 09:56:30 +00:00
Michał Niewrzał
d995fb497f Merge remote-tracking branch 'origin/main' into multipart-upload
Change-Id: I367da03351ab80f7343332420490dde9282aa47a
2021-02-23 12:31:31 +01:00
Egon Elbre
a34da8531b ci: bump test timeout
Currently testing with coverage exceeds the timeout very easily.

Change-Id: I4e77aa98d28bebe373416ac05ee39b4c70c3fed2
2021-02-22 17:31:21 +02:00
Egon Elbre
21a6dccd36 ci: set specific line coverage targets
Currently the auto-updating keeps moving the target upwards without
allowing for variations in the pipeline.

Let's try these settings:

* >=70% lines covered, to pass
* <=60% lines covered, to mark as unhealthy
* <=50% lines covered, to mark as unstable

Change-Id: Idc75e682345f6d62af34379362c75d321e564d2c
2021-02-22 15:13:18 +02:00
Egon Elbre
bc68898846 ci: enable coverage for multipart-upload branch
Change-Id: I4797d16bbc90a3fc7fc9627386b63124755f4730
2021-02-22 13:43:32 +02:00
Michał Niewrzał
341a4b7f59 metainfo-migration: extend test case
Change-Id: I9d1ac41a18bb08200ef20f7ff2f8df5531140f99
2021-02-11 19:28:41 +01:00
Michał Niewrzał
5a23d9c167 metainfo-migrator: improve performance
Change-Id: I258605737c37dfdd8f0070dbadf0e3a9c6358cc3
2021-02-11 17:33:22 +01:00
Egon Elbre
2848bf488f satellite/metainfo/metabase: use alias pieces in segments table
This makes all tables automatically convert between aliases and piece
ID-s.

Change-Id: I27fa42c82bbb09e05e3327f85e13a000b48faffd
2021-02-11 16:40:37 +02:00
Kaloyan Raev
d0612199f0 Merge remote-tracking branch 'origin/main' into multipart-upload
Conflicts:
	go.mod
	go.sum
	satellite/metainfo/config.go
	satellite/metainfo/metainfo_test.go

Change-Id: I95cf3c1d020a7918795b5eec63f36112fdb86749
2021-02-01 14:32:12 +02:00
Egon Elbre
a700a1bdab satellite/metainfo/metabase: add benchmark
Change-Id: I4289aac252f9c951548444aa5a82e007e9f47806
2021-01-25 14:59:24 +00:00
Jessica Grebenschikov
b7d8dee5e9 satellite/console/wasm: add js tests
Change-Id: I8b1b0e81500836e0408e0517edb6c696698ab5f7
2021-01-21 20:18:03 +00:00
Jessica Grebenschikov
1f1d9fce58 satellite/console/wasm: add test to confirm wasm size isnt growing
Change-Id: I975a9f8ac3f6b98cc213140fdd7a99557efe14c8
2021-01-21 15:48:49 +00:00
Michał Niewrzał
ac058e5ecc metainfo-migration: basic pointerdb->metabase migrator
Change-Id: If183b9c99ce816521fcae6e7189a6f85ddd4eb48
2021-01-12 12:59:53 +00:00
Michał Niewrzał
ad3e3a38c5 Merge 'main' branch
Change-Id: Ia0db1b1f9ef3e0671d3f2208881b0abc3064e200
2021-01-04 12:13:45 +01:00
Stefan Benten
02d7638eda
Jenkinsfile.public: specify main branch for docker build
Docker build defaults to the master branch, when specifying a git repo.
With this change, it should be smart enough to use the proper main branch.
2020-12-29 16:56:36 +02:00
Stefan Benten
7f1871b8f1 all: switch from master to main 2020-12-28 22:59:06 +01:00
Kaloyan Raev
bafc6af992 ci: remove workaround for failing tests
Change-Id: I3eb673fae6c81bee17d7437cb870d5f5ba6978d5
2020-12-21 18:07:40 +02:00
Michal Niewrzal
bc3be9c416 Jenkinsfile: enable build failure on integration tests failure
Change-Id: Ifabcd9e72cd51de1300b5f8a4453c87ca292c60e
2020-12-07 15:31:46 +00:00
Kaloyan Raev
402a581b86 ci: fail build if metabase tests fail
Change-Id: Ic263ad2dcf777d1fb7c03bbddc9edd09a2165c83
2020-11-17 13:30:15 +00:00
Michal Niewrzal
7c384c8293 Merge 'master' branch
Change-Id: I1eefd5a56449e577820977d61fa4a22bdd4fc230
2020-11-16 10:02:54 +01:00
Egon Elbre
c0b5e7ce3e ci: ensure cockroach doesn't pollute repo
Cockroach 20.2 started automatically profiling, this is a workaround to
disable it and ensure it doesn't create any folders and files in the
repository.

Change-Id: Ib65de01ea1fc619160d710c01602ced3a3a3492e
2020-11-13 16:07:01 +00:00
Egon Elbre
2ff7925e65 ci: set GOTRACEBACK=all
Currently when there's a timeout or panic, the culprit goroutines might
not be printed. Set traceback to all, which prints all user created
goroutines.

Change-Id: I29f87812d2a60f671b3eb172499e24cf70d990b5
2020-11-11 13:57:45 +02:00
Michal Niewrzal
dbf9f121ed ci: mark failures as unstable
Change-Id: I3cfa07a35cb789f409d95d4778c07ca3355c0f3c
2020-11-09 16:48:38 +01:00
Egon Elbre
754838f5d0 ci: convert test failures to unstable
Change-Id: Ib0252c666c086379bd6e2a3a47011479fe3988ed
2020-11-09 11:18:12 +00:00
Egon Elbre
0a77deec81 satellite/metainfo/metabase: basic migrate
Change-Id: Ia34f38e763af9ba88f75038b8ecfc9eb6550eae0
2020-11-05 16:31:02 +02:00
Michal Niewrzal
809eb14ac2 satellite/metainfo/metabase: move metainfo PoC into storj repo
Change-Id: I39356d8bc7305b4a8ea0c1fb5603010ad72a68b9
2020-10-29 09:44:55 +01:00
Egon Elbre
d508c4c985 scripts: remove duplicate check config lock
We have configlock_test for checking changes so the separate script and
makefile target are not needed.

Change-Id: I2bbc1c21ad849c9b7ec8bba43c0e11e94e04f6a6
2020-09-29 10:03:34 +00:00
Egon Elbre
2d27bc8787 satellite/satellitedb: separate cockroach for migration tests
Currently Cockroach migration test is the most heavy with regards to
schema changes. This causes other tests to time out. This adds an
alternate cockroach instance that is used for migration tests.

Change-Id: I01fe9313527ff002f0bb0914dd52c3645b8eaf6d
2020-09-29 09:31:33 +00:00