Jenkins doesn't do a very good job with identifying what has been changed.
While it has a syntax to defined patterns, it compares the current build with the previous build (in case of git-verify it can be a totally different branch) instead of checking the HEAD commit.
This patch introduces shell scripts to do this better:
* It doesn't depend on Jenkins any more
* It can be executed locally
* It can detect web changes properly (see the relation change as an example).
Change-Id: I9d37775e3818c08c4aa96ffb78f84d57f28a2c95
As we started to run integration tests one by one (instead of parallel). We need to bump the timeout as well.
(Later we can decrease it to a better value after understanding what is reasonable)
Change-Id: I4a7e18973311fe98b0ffb3d9ed496584b1697980
Our tests depend on timing (timeouts / sleeps). Therefore we must not overcommit the CPU with highly parallel execution as builds become flaky.
go test can do parallel execution (that's fine as it doesn't overcommit CPU, if only one job is running), but we shouldn't execute different type of
integration tests at the same time.
This commit changes only public (PR + main build) and pre-merge (one build per patch), which are less build time sensitive.
Change-Id: Ib9aaabd301cfa9cd3515a8aa4dd8ae82f58eb811
The builds for npm still fail, however, `--timing` does not provide
sufficient data to debug the situation.
Change-Id: I7e618ba8cac775748ebea6145cd5c180d2dc7883
`npm ci` deletes the node_modules directory, which also removes go.mod
from that folder.
Add --loglevel timing, so we can debug install slowness, whenever it
happens.
Change-Id: Ide613c4124bfdca9ae978876b2deed8abf86f987
Add go.mod to node_modules folder, that way Go compiler doesn't
need to scan the node_module directories for any Go code.
Change-Id: I747909416490c847d6b4bfa3438fea66660fcd53
I would like to start moving some of the integration tests from scripts
folder into testsuite. This allows integration tests to easily have
other and fewer dependencies.
This will also be useful for https://github.com/storj/storj/issues/4823.
Change-Id: Idbdb5aefc0e42d091ad7f74dbdc096281c1dfcba
Márton found out that DROP DATABASE is rather slow on CRDB, and it makes
a significant impact when running the whole testsuite. In sum of test
times it's ~2.5h compared to ~2h. And the end-to-end ~20m to ~16m.
This adds a new flag STORJ_TEST_COCKROACH_NODROP for enabling this
behavior in the CI environment.
Fixes https://github.com/storj/dev-enablement/issues/6
Change-Id: I5a6616c32dc6596a96ba3d203f409368307d7438
We had an issue with CRDB where newer version forbid type
of SQL query we were using. We disabled CRDB bc tests until we
will have release without problematic query. Now we can enable
tests.
Change-Id: I275cbecebdcbfef587281f2daaf677d01860b23d
To save load on DNS servers, the repair code first tries to dial the
last known good ip and port for a node, and then falls back to a DNS
lookup only if we fail to connect to the last known good ip and port.
However, it looks like we are seeing errors during the client stream
Close() call (probably due to quic-go code), and those are classified
the same as errors encountered during Dial. The repairer code sees this
error, assumes that we failed to contact the node, and retries- but
since we did actually succeed in connecting the first time around, this
results in submitting the same order limit (with the same serial number)
to the storage node, which (rightfully) rejects it.
So together with change I055c186d5fd4e79560f67763175bc3130b9bc7d2 in
storj/uplink, this should avoid the double submission and avoid dinging
nodes' suspension scores unfairly.
See https://github.com/storj/storj/issues/4687.
Also, moving the testsuite directory check up above check-monkit in the
Jenkins Lint task, so that a non-tidy testsuite/go.mod can be recognized
and handled before everything breaks weirdly and seemingly randomly
later on.
Change-Id: Icb2b05aaff921d0af6aba10e450ac7e0a7bb2655
Embedded files significantly increase the binary size for linking.
Add a tag that allows disabling embedding the build npm code.
Change-Id: I9d1fd7376d1fa035965c33d259faaa6c4770dfe1
Rather than starting all servers on 127.0.0.1 start them
on a random local host to try avoid port exhaustion.
The port exhaustion is just a guess.
Change-Id: Ibf31d6a017852238d836291d703642b44ff66c0c
This also fixes the build order. Unfortunately we need
to ensure that the web frontends are built before installing
Go binaries.
Fixes https://github.com/storj/storj/issues/4654
Change-Id: I5d1c83125fd3d1a454d3400b2cbdd44bd3f2250c
We have an issue with latest CRDB. Single query cannot modify
the same table multiple times. Now build is blocked.
This change is unblocking build by:
* adjusting query for inserting into repair queue
* temporary removing code for deletion for server-side copy
* temporary disable backward compatibility tests for CRDB
Change-Id: Idd9744ebd228e5dc05bdaf65cfc8f779472a975d
Currently UI tests are always failing.
Disable them for now, since they do not add any useful signal to the result.
Change-Id: Ifa4a5c70cb6954e355fa9f26912fb0ce03a62ddc
Migrate the satellite admin UI web app from the Svelte template used to
generate a Svelte App scaffolding to SvelteKit.
There aren't any functional changes in the application, however, the
commit has a lot because:
1. SvelteKit uses a different directory layout and constraints to it, so
the files have been moved.
2. The files have changed its formatting due to the new default linter
configurations that SvelteKit uses.
3. The linter detected some issues with using `object` and `any` types
in Typescript, so they have been replaced by better general types
(e.g. Record).
The migration allows to use the new tooling rather than Rollup
directly, besides that will empower the future of it when it needs more
features (e.g. different routes, etc.).
Change-Id: Ifa6736c13585708337f6c5a59388077b784eaddd
* Remove "enable-logging", because it ends up spawning consoles on
Windows.
* Remove "disable-gpu", if we have a GPU, then let's use it.
* Create custom client, so we can attach logging to CDP.
* Ignore potential context.Canceled.
* Fix onboarding wizard test for new objects browser.
* Return an error on a context cancellation.
* Wire all loggers to planet.
Change-Id: I67eb138ba31252f55ac5b383679d033bcf71f1b2
Also, we have multiple tests that contain the same sign up
and login logic and refactoring them would make tests
cleaner and more readable.
Change-Id: I25bf95ca2fffb7f494a3c93a4a4afe429ece70d3
Currently testsuite/go.mod is being updated automatically and
we don't want to trigger UI tests, since they are a bit flaky.
Change-Id: Ie1c847e262ad723a006071f7a8f5cd7f26f4244e
Gateway-ST frequent release cycle has been resurrected, which means it's
safer to use the latest release tag in the storj repository's CI now.
Change-Id: I9df1c789a9b9418ba7cceaec9cfec3cc6c448284
Newer npm seems to invoke the command without the current user
privileges. However, Go already initializes its cache using root.
Change-Id: I23b32ca3bbd7db06515c2af86b1bb5adc8e38a8f
We are not using the benchmark results for anything, they are mostly
there to ensure that we don't break the benchmarks. So we can disable
CockroachDB for them.
Similarly add short versions of other tests.
Also try to precompile test/benchmark code.
Change-Id: I60b501789f70c289af68c37a052778dc75ae2b69
Jenkins build consistently times out due to the enormous tests we have now.
Bumping the timeout to 40 fixes it for now.
Change-Id: Ie96cc5ec3e452744d5a7873853d14befe367bd7e
This chmod is combined with "Clean before checkout" behaviour
that deletes everything, except .git. This should give us
the sufficient ordering to ensure that everything has the
correct permissions.
Story so far:
Checkout with "Wipe out repository & force clone" didn't work,
because the chmod would've run after trying to delete.
Deleting as a post action doesn't work, because during timeouts
the always steps don't get executed sometimes.
We cannot easily run docker with non root, due to how our ci
image works.
Change-Id: Id4c1605d39ae8d7722ba3dd1ab5df8618de3309b
Jenkins uses the same folder for different PR-s. Sometimes other runs do
not cleanup after themselves (e.g. timeout), hence add a cleanWs step to
ensure we delete files in the workspace.
gateway-st introduced a replace directive in go.mod, which does not work
with go install. Hardcode to the last version without the directive.
Using this fix to unblock ci builds.
Change-Id: I5e5d75bf47e30a5a8b6d835867c0c9176f25e08a
Currently some building may generate some additional files,
similary, we want to run check-clean-directory after everything
has finished.
Change-Id: I4d700896094257e65cc5197c468f6d752024ddd9
Add a go.mod to node_modules folder, that should prevent
go build ./...
from scanning it, however, it's slightly hacky.
Change-Id: I073150704bd90afd8ccf38a969e0a1c725d25137
After migrating to eslint some errors were disabled to make it easier to
migrate.
This enables all the lint rules and treats all warnings as a build
failure. Similarly, CI won't automatically try to fix mistakes.
Change-Id: I80f808af026fc51bed90421b3b24737994a52094
After migrating to eslint some errors were disabled to make it easier to
migrate.
This enables all the lint rules and treats all warnings as a build
failure. Similarly, CI won't automatically try to fix mistakes.
Change-Id: Iff9c4a59401900fc395cd566dd328f3a9c688a12
After migrating to eslint some errors were disabled to make it easier to
migrate.
This enables all the lint rules and treats all warnings as a build
failure. Similarly, CI won't automatically try to fix mistakes.
Change-Id: I6d5ef39d476a38dd9a6ab98e3fb94da4c3ab2e57
We can run a lot of the code building and downloading steps in parallel.
We were using Makefile, however, that doesn't integrate with Jenkins,
which means, when there's a failure, you have no clue where.
This separates the concurrent build steps and makes it clear what we are
running concurrently.
Change-Id: I1c1dccd09dab67d2ef428a58d5eb9e004b11a74c
