Commit Graph

127 Commits

Author SHA1 Message Date
Michał Niewrzał
ba74cb17a9 satellite: adjust to latest CRDB version
We have an issue with latest CRDB. Single query cannot modify
the same table multiple times. Now build is blocked.
This change is unblocking build by:
* adjusting query for inserting into repair queue
* temporary removing code for deletion for server-side copy
* temporary disable backward compatibility tests for CRDB

Change-Id: Idd9744ebd228e5dc05bdaf65cfc8f779472a975d
2022-03-23 07:16:38 +00:00
Egon Elbre
49dcec02e5 Jenkinsfile: optimize build caching
Change-Id: If3fb973b2d75672e2bb2eb09c381d48f5b5cfd4a
2022-03-07 22:59:15 +02:00
Mya
56c668a2e3 private/testplanet: use info level logs in jenkins
Change-Id: Ie500a7bfcd23ca7470139341e961b2e9986d6a33
2022-02-10 11:47:06 -06:00
Jeff Wendling
9061dd309f cmd/uplinkng: become cmd/uplink
Change-Id: If426c32219d32044d715ab6dfa9718807f32cb9f
2022-02-09 17:02:21 +00:00
Egon Elbre
c58a81e7c2 testsuite/ui: disable flaky UI tests
Currently UI tests are always failing.

Disable them for now, since they do not add any useful signal to the result.

Change-Id: Ifa4a5c70cb6954e355fa9f26912fb0ce03a62ddc
2022-01-03 15:54:15 +02:00
Egon Elbre
5f39f20766 web/: add check for change to eslint import
Change-Id: Idf94c8e0c92d34e840db549e9e704eb3373762ac
2021-12-21 15:59:23 +00:00
Ivan Fraixedes
5573ece848 satellite/admin/ui: Migrate to SvelteKit
Migrate the satellite admin UI web app from the Svelte template used to
generate a Svelte App scaffolding to SvelteKit.

There aren't any functional changes in the  application, however, the
commit has a lot because:

1. SvelteKit uses a different directory layout and constraints to it, so
   the files have been moved.
2. The files have changed its formatting due to the new default linter
   configurations that SvelteKit uses.
3. The linter detected some issues with using `object` and `any` types
   in Typescript, so they have been replaced by better general types
   (e.g. Record).

The  migration allows to use the new tooling rather than Rollup
directly, besides that will empower the future of it when it needs more
features (e.g. different routes, etc.).

Change-Id: Ifa6736c13585708337f6c5a59388077b784eaddd
2021-12-17 10:27:13 +00:00
Egon Elbre
dfd2977a01 testsuite/ui: fixes to setup
* Remove "enable-logging", because it ends up spawning consoles on
  Windows.
* Remove "disable-gpu", if we have a GPU, then let's use it.
* Create custom client, so we can attach logging to CDP.
* Ignore potential context.Canceled.
* Fix onboarding wizard test for new objects browser.
* Return an error on a context cancellation.
* Wire all loggers to planet.

Change-Id: I67eb138ba31252f55ac5b383679d033bcf71f1b2
2021-11-15 15:56:35 +02:00
Ivan Fraixedes
ab425d4248 build: Consider the Admin UI assets
Add into the Makefile and Jenkins the operations to build the Admin UI
assets and lint them.

Change-Id: I687c1adec796fd591664e34eb73a465e53096b58
2021-11-12 08:48:28 +00:00
Gabriel De Almeida
5ed3846e16 testsuite/ui/satellite: add browser tests
Also, we have multiple tests that contain the same sign up
and login logic and refactoring them would make tests
cleaner and more readable.

Change-Id: I25bf95ca2fffb7f494a3c93a4a4afe429ece70d3
2021-11-11 15:03:07 -05:00
Egon Elbre
58916d8ba9 ci: don't trigger UI tests when go.mod changes
Currently testsuite/go.mod is being updated automatically and
we don't want to trigger UI tests, since they are a bit flaky.

Change-Id: Ie1c847e262ad723a006071f7a8f5cd7f26f4244e
2021-11-10 17:16:08 +00:00
Artur M. Wolff
785eb93cd7 ci: install gateway@latest instead of @main
Gateway-ST frequent release cycle has been resurrected, which means it's
safer to use the latest release tag in the storj repository's CI now.

Change-Id: I9df1c789a9b9418ba7cceaec9cfec3cc6c448284
2021-11-01 12:16:38 +00:00
Egon Elbre
431f55d53b ci: cross-compile storagenode-updater
Change-Id: I1db757b64e61f185ba710bf669f2e62441b9ab71
2021-11-01 11:21:06 +00:00
Egon Elbre
52c950e429 testsuite/ui/uitest: add Edge testing
Change-Id: I1e592ac755b582d4fa73e95fd999a35a27f8a116
2021-10-15 13:27:13 +00:00
Egon Elbre
8f4550d621 ci: fix using npm run build-wasm
Newer npm seems to invoke the command without the current user
privileges. However, Go already initializes its cache using root.

Change-Id: I23b32ca3bbd7db06515c2af86b1bb5adc8e38a8f
2021-10-14 15:10:40 +00:00
Egon Elbre
9fd091831d ci: optimize benchmarks
We are not using the benchmark results for anything, they are mostly
there to ensure that we don't break the benchmarks. So we can disable
CockroachDB for them.

Similarly add short versions of other tests.

Also try to precompile test/benchmark code.

Change-Id: I60b501789f70c289af68c37a052778dc75ae2b69
2021-10-08 19:42:40 +03:00
Egon Elbre
6ab2647adb ci: run UI tests
We have implemented few UI tests using go-rod, but haven't been running
them regularly.

Change-Id: I6c89365f719882431842a2026b426860f1293b8b
2021-10-06 15:54:06 +00:00
Egon Elbre
f52f5931da ci: don't fail when nothing to delete
Change-Id: Ided795c3b60f053dfaff89be6e944cf2cce5faba
2021-09-29 13:41:50 +03:00
Artur M. Wolff
cc9b845a83 ci: switch back to pulling gateway@main
Switch back to pulling gateway@main for continuous integration because
https://review.dev.storj.io/c/storj/gateway/+/5834 was merged. This
change is a partial revert of
https://review.dev.storj.io/c/storj/storj/+/5832.

Change-Id: I0a87f6f18a9a863fe92003b76c830335f9253ced
2021-09-28 15:01:04 +00:00
Egon Elbre
9c232cebe1 ci: make deletion a separate step
Deletion as a separate step so it's more understandable what's
actually happening.

Change-Id: Icb3cefe7e4093c56604af633ac0e02226bae42ac
2021-09-28 10:24:43 +00:00
Clement Sam
118e64fa15 jenkinsfile: bump build timeout
Jenkins build consistently times out due to the enormous tests we have now.
Bumping the timeout to 40 fixes it for now.

Change-Id: Ie96cc5ec3e452744d5a7873853d14befe367bd7e
2021-09-27 18:31:16 +00:00
Egon Elbre
8036644422 ci: try fix builds
This chmod is combined with "Clean before checkout" behaviour
that deletes everything, except .git. This should give us
the sufficient ordering to ensure that everything has the
correct permissions.

Story so far:

Checkout with "Wipe out repository & force clone" didn't work,
because the chmod would've run after trying to delete.

Deleting as a post action doesn't work, because during timeouts
the always steps don't get executed sometimes.

We cannot easily run docker with non root, due to how our ci
image works.

Change-Id: Id4c1605d39ae8d7722ba3dd1ab5df8618de3309b
2021-09-27 16:18:04 +00:00
Egon Elbre
7fd3466925 ci: remove deleting workspace steps
Moved the deleting workspace to pipeline configuration.

Change-Id: I7c94d1a9bac514f1824988d21dfb3a144b10ca7e
2021-09-27 15:49:34 +03:00
Egon Elbre
8ef03b0967 ci: cleanup ws before build, fix gateway install
Jenkins uses the same folder for different PR-s. Sometimes other runs do
not cleanup after themselves (e.g. timeout), hence add a cleanWs step to
ensure we delete files in the workspace.

gateway-st introduced a replace directive in go.mod, which does not work
with go install. Hardcode to the last version without the directive.
Using this fix to unblock ci builds.

Change-Id: I5e5d75bf47e30a5a8b6d835867c0c9176f25e08a
2021-09-23 15:57:47 +03:00
Egon Elbre
bd4017b2a6 ci: fix logical races between tools
Currently some building may generate some additional files,
similary, we want to run check-clean-directory after everything
has finished.

Change-Id: I4d700896094257e65cc5197c468f6d752024ddd9
2021-09-23 12:16:31 +03:00
Egon Elbre
6e660cecdd Jenkinsfile: test cross-compile and bump deps
Change-Id: I47091de2e80bd96fcada616f75e1db07a59fb0c1
2021-09-16 18:59:31 +03:00
Egon Elbre
25971c581f Jenkinsfile: clarify modfile linting
Change-Id: I0ac0bfa48386d3430eb8fb5a92b57daefc23998f
2021-08-31 15:13:45 +03:00
storj-antonio
3510bc036e testsuite: fix testsuite/go.mod and enable linters
This change implements verification of the testsuite
and resolves missing go.sum module errors.

Change-Id: I7b513e74532581b8878d50aa2c5ddca12bbef2d0
2021-08-31 12:10:18 +00:00
Sean Harvey
d22ecf9ec0 Jenkinsfile.public: pull latest ci image instead of building
Change-Id: Ia0e6863fdd00127336839d4d0b81306611aee177
2021-08-17 06:47:29 +12:00
Egon Elbre
f807518477 ci: try fix go build scanning node_modules directory
Add a go.mod to node_modules folder, that should prevent

  go build ./...

from scanning it, however, it's slightly hacky.

Change-Id: I073150704bd90afd8ccf38a969e0a1c725d25137
2021-08-12 18:11:11 +03:00
Egon Elbre
6b153192a3 web/satellite: fix lint issues
After migrating to eslint some errors were disabled to make it easier to
migrate.

This enables all the lint rules and treats all warnings as a build
failure. Similarly, CI won't automatically try to fix mistakes.

Change-Id: I80f808af026fc51bed90421b3b24737994a52094
2021-08-10 09:22:19 +00:00
Egon Elbre
19852a767b web/storagenode: fix lint issues
After migrating to eslint some errors were disabled to make it easier to
migrate.

This enables all the lint rules and treats all warnings as a build
failure. Similarly, CI won't automatically try to fix mistakes.

Change-Id: Iff9c4a59401900fc395cd566dd328f3a9c688a12
2021-08-05 15:45:55 +00:00
Egon Elbre
a3e78491b9 web/multinode: fix lint issues
After migrating to eslint some errors were disabled to make it easier to
migrate.

This enables all the lint rules and treats all warnings as a build
failure. Similarly, CI won't automatically try to fix mistakes.

Change-Id: I6d5ef39d476a38dd9a6ab98e3fb94da4c3ab2e57
2021-08-04 17:00:00 +03:00
Egon Elbre
1cd13adc1c ci: clearer initial build
We can run a lot of the code building and downloading steps in parallel.
We were using Makefile, however, that doesn't integrate with Jenkins,
which means, when there's a failure, you have no clue where.

This separates the concurrent build steps and makes it clear what we are
running concurrently.

Change-Id: I1c1dccd09dab67d2ef428a58d5eb9e004b11a74c
2021-06-09 12:30:31 +00:00
Yaroslav Vorobiov
f0d60a6ec2 Jenkins: add multinode UI stage
Change-Id: I77ed791ff9dbb5b046450ab625c1763b34691888
2021-05-21 15:49:31 +03:00
Egon Elbre
2ae80690cb satellite/metabase/metabasetest: remove STORJ_TEST_DATABASES
Initially metabase was developed separately and it was useful to have a
separate environment flag for tests, however, it's more convenient to
use the same as rest of the testsuite.

Change-Id: Ia4d79be27ce5911cbae68d57cdf0b30f63459444
2021-05-11 13:31:01 +00:00
Egon Elbre
7802ab714f pkg/,private/: merge with private package
Initially there were pkg and private packages, however for all practical
purposes there's no significant difference between them. It's clearer to
have a single private package - and when we do get a specific
abstraction that needs to be reused, we can move it to storj.io/common
or storj.io/private.

Change-Id: Ibc2036e67f312f5d63cb4a97f5a92e38ae413aa5
2021-04-23 16:37:28 +03:00
Michał Niewrzał
a5bc38b197 build: replace 'multipart-upload' branch occurences
We removed `multipart-upload` branches but we have few leftovers.

Change-Id: I74f637748a248cb286cd190a3a8f895ef85f3940
2021-03-31 11:06:47 +02:00
Egon Elbre
74b243d1ca certificate: move protobuf to storj/storj repository
Change-Id: I4689318304b54121a65e28860430d39d94ac5bd6
2021-03-29 12:59:49 +00:00
Michał Niewrzał
237782813b Merge remote-tracking branch 'origin/multipart-upload'
Change-Id: If6c5a450b238adab55d1e0dea67d01e5f5768a9f
2021-03-23 09:44:49 +01:00
Ivan Fraixedes
536990dcf6 Jenkins: Run integration test Redis unavailability
Make Jenkins to run the integration tests that verifies that the
satellite can operates when Redis is unavailable.

Change-Id: Idace3ffb84c788f2af2c6e24eec1d63fb40c263a
2021-03-22 09:56:30 +00:00
Michał Niewrzał
d995fb497f Merge remote-tracking branch 'origin/main' into multipart-upload
Change-Id: I367da03351ab80f7343332420490dde9282aa47a
2021-02-23 12:31:31 +01:00
Egon Elbre
a34da8531b ci: bump test timeout
Currently testing with coverage exceeds the timeout very easily.

Change-Id: I4e77aa98d28bebe373416ac05ee39b4c70c3fed2
2021-02-22 17:31:21 +02:00
Egon Elbre
21a6dccd36 ci: set specific line coverage targets
Currently the auto-updating keeps moving the target upwards without
allowing for variations in the pipeline.

Let's try these settings:

* >=70% lines covered, to pass
* <=60% lines covered, to mark as unhealthy
* <=50% lines covered, to mark as unstable

Change-Id: Idc75e682345f6d62af34379362c75d321e564d2c
2021-02-22 15:13:18 +02:00
Egon Elbre
bc68898846 ci: enable coverage for multipart-upload branch
Change-Id: I4797d16bbc90a3fc7fc9627386b63124755f4730
2021-02-22 13:43:32 +02:00
Michał Niewrzał
341a4b7f59 metainfo-migration: extend test case
Change-Id: I9d1ac41a18bb08200ef20f7ff2f8df5531140f99
2021-02-11 19:28:41 +01:00
Michał Niewrzał
5a23d9c167 metainfo-migrator: improve performance
Change-Id: I258605737c37dfdd8f0070dbadf0e3a9c6358cc3
2021-02-11 17:33:22 +01:00
Egon Elbre
2848bf488f satellite/metainfo/metabase: use alias pieces in segments table
This makes all tables automatically convert between aliases and piece
ID-s.

Change-Id: I27fa42c82bbb09e05e3327f85e13a000b48faffd
2021-02-11 16:40:37 +02:00
Kaloyan Raev
d0612199f0 Merge remote-tracking branch 'origin/main' into multipart-upload
Conflicts:
	go.mod
	go.sum
	satellite/metainfo/config.go
	satellite/metainfo/metainfo_test.go

Change-Id: I95cf3c1d020a7918795b5eec63f36112fdb86749
2021-02-01 14:32:12 +02:00
Egon Elbre
a700a1bdab satellite/metainfo/metabase: add benchmark
Change-Id: I4289aac252f9c951548444aa5a82e007e9f47806
2021-01-25 14:59:24 +00:00