Commit Graph

260 Commits

Author SHA1 Message Date
Clement Sam
6277d93c82 cmd/storagenode: remove storagenode dockerfiles
Updates https://github.com/storj/storagenode-docker/issues/1

Change-Id: I6b2f42194cb0e32f1b3978072fd6e1cdec753704
2023-01-26 13:49:12 +00:00
Erik van Velzen
cb01aca13d cmd/uplink: add docker image
Add a docker image for uplink-cli and push it to docker hub.
We used to have this before the change to uplinkng. I'm not
sure if the pushing works, we'll see after merge.

To test, build an image with `make uplink-image`, read the tag from the
output and run normal uplink-cli commands using
`docker run -it storjlabs/uplink:df9bbceca-uplink-docker-go1.18.8-amd64 [command]`

Part of https://github.com/storj/uplink/issues/109

Change-Id: I8a10aab2b778951ff42a22ba2f252c581eb66b65
2023-01-24 09:48:02 +00:00
Egon Elbre
947cdf0fb4 ci: fixes for new Go version
We need to make exceptions for older uplink versions, because it does
not compile with newer Go versions due to quic dependency.

Change-Id: I3e073694f0942029c56740f0689088058ee068c3
2022-11-02 15:17:46 +02:00
Michal Niewrzal
a3d9630336 scripts: move tests to separate folders
Doing some cleanup in "scripts" folder. All integration like tests are
moved under "test" directory (integration, bc, redis) and bash scripts
are adjusted to reflect new location.

As an addition "scripts/install-awscli.sh" was deleted as it was not
used.

Change-Id: I152905c4258f471a71f2d0e8731d91bb075e99c1
2022-09-16 11:10:18 +00:00
Sean Harvey
582dce1b36 Makefile: bump to Go 1.17.12
Change-Id: Ifa163a113da8c21a72a6127e53b331b574eea8dc
2022-07-14 20:57:26 +00:00
Egon Elbre
94ae196189 testsuite/ui: move go.mod
I would like to start moving some of the integration tests from scripts
folder into testsuite. This allows integration tests to easily have
other and fewer dependencies.

This will also be useful for https://github.com/storj/storj/issues/4823.

Change-Id: Idbdb5aefc0e42d091ad7f74dbdc096281c1dfcba
2022-06-21 14:12:35 +03:00
Márton Elek
7608ea7af0 Makefile: make test should work with any directory name
docker-compose exec is more reliable as it doesn't depend on the name of the directory (the first part of the container name).

I tried to run the `make test` comman on Jenkins, but it failed as the checkout directory was storj_main (or storj_debug).

Change-Id: I85b060ebd0705a2a9fcaefa17033c61a0509c4a2
2022-06-09 13:37:56 +00:00
Márton Elek
0831c8f390 makefile: use the ci-slim image for linting
Change-Id: I748d9fb6874a4b399985776e50202399467cfc2e
2022-06-08 14:03:10 +00:00
Márton Elek
7c854fcde4 Makefile: make test run more safe
This patch suggests small improvements for the previously committed (very nice <3) unit test helper

 * Use non-default ports for postgres + cockroach. This would help to run same services for ad-hoc unit test (for IDE development for example)
 * Use the same flags for cockroach what we use for unit tests (mem store supposed to make tests faster)
 * Use `docker-compose down -v` for removing all networks and annynmous volumes
 * Remove connection limit (I have seen related problems in my jenkins runs)
 * Omit unused databases to avoid getting failures (eg. omit psql when crdb is used)
 * Do not drop cockroach database (full database will be dropped, anyway...): it also makes testing faster

Change-Id: Iadc04f8617a2825ea4f10072bd023a9c86883680
2022-05-31 10:58:26 +00:00
Mya
9153f37055 Makefile: run lint locally in docker
Our linting process depends heavily on custom tools and linting
configuration. To help improve this process for running on local
developer machines, we can run the various tasks in our existing CI
container.

Change-Id: I60407686ce9233cc4f16e3724c5e8d44367aa200
2022-05-25 12:30:15 -05:00
Mya
4b61cc9e9c Makefile: update test target to support a local execution workflow
This change attempts to run tests as close to CI as possible. It
introduces a docker-compose file that deploys some supporting
services. While this change makes it possible to run tests locally,
it does not address the core time it takes to run tests end to end.

make test/postgres  1417.87s user 263.38s system 291% cpu 9:37.73 total

Resolves https://github.com/storj/dev-enablement/issues/10

Change-Id: I4d9adc125992c1b1c133e8fe108384fa3c66ff7a
2022-05-25 16:47:06 +00:00
Clement Sam
e486585853 Dockerfiles: switch base image from alpine to Debian
We've had a lot of issues with alpine and currently there's a broken
network issue on alpine for users running on RPI arm32 architechture
which requires a workaround before docker is able to sync time between
the host and the container: https://wiki.alpinelinux.org/wiki/Release_Notes_for_Alpine_3.13.0\#time64_requirements.

Since we're switching the base image of the storagenode to debian,
it's best to switch the base image of all our docker images to
debian as well for consistency; less drift across them and keeps
the push target consistent.

Change-Id: If3adf7a57dc59f19ef2221b892f340d919798fc5
2022-04-05 11:40:44 +00:00
Clement Sam
ee7f0d3a2a Makefile: fix wrong indentation
Indentations are off in the push-storagenode-images target in the makefile which is possibly causing build to fail

Change-Id: Ia25b8f700f49c551e3f201c988747a83e04ad83c
2022-04-05 09:11:55 +00:00
Clement Sam
d611c16891 cmd/storagenode: update storganode-base image version in Dockerfile
Change-Id: I2e40f75d7cf1f673ea16a633a427225f97ea92ed
2022-03-29 13:26:33 +00:00
Egon Elbre
4a6c2e2a9c satellite/admin/ui: avoid needing an additional copy
This also fixes the build order. Unfortunately we need
to ensure that the web frontends are built before installing
Go binaries.

Fixes https://github.com/storj/storj/issues/4654

Change-Id: I5d1c83125fd3d1a454d3400b2cbdd44bd3f2250c
2022-03-25 11:42:22 +02:00
Clement Sam
c641f4c9ac cmd/storagenode: use Debian as base image instead of alpine
We are switching from alpine to debian due to a network issue
introduced in alpine 3.13+ which fails to verify certificates
due to not all armhf boards meet the time64 requirement:
https://wiki.alpinelinux.org/wiki/Release_Notes_for_Alpine_3.13.0\#time64_requirements

Also, Debian does not have official imagess for arm32v6 architecture
so we are building with arm32v5 arch in the Makefile.

Change-Id: I3660c3f64b7c2b342dd4ccb876af5f4e3036ea9d
2022-03-25 08:19:20 +00:00
Clement Sam
52a80b2c67 cmd/storagenode: use custom storagenode-base image
Updates https://github.com/storj/storj/issues/4489

Change-Id: I3585b50df598c69e58137f61b6a272dbd077c5be
2022-03-15 14:22:35 +00:00
Egon Elbre
dc0f7b5f77 storagenode,web/storagenode: use go:embed for assets
Go can now directly embed files without relying on external tools.
This makes code use go:embed and avoid the external tooling.

go:embed requires files to be present in the embedded directory,
hence we need to add .keep to "dist" folder. We also add one to
public/.keep, such that it won't be deleted when building storagenode.

Change-Id: I8bef81236be6829ed37ed4c16ef693677b93a631
2022-03-11 16:01:28 +02:00
Egon Elbre
28c9403702 multinode,web/multinode: use go:embed for assets
Go can now directly embed files without relying on external tools.
This makes code use go:embed and avoid the external tooling.

go:embed requires files to be present in the embedded directory,
hence we need to add .keep to "dist" folder. We also add one to
public/.keep, such that it won't be deleted when building multinode.

Change-Id: I53ac3d5ac76e44f740d95221acf0da99fc256d42
2022-03-11 11:10:37 +02:00
Clement Sam
26b722e414 Makefile: create manifest for storagenode-base image
Change-Id: I7c0c920f745717965800b6019fc9fba07b8479b6
2022-03-08 13:30:58 +00:00
Clement Sam
15a1428828 {cmd/storagenode,Makefile}: add storagenode base image Dockerfile
Having the storagenode and storagenode-updater processes in one container
requires a process manager to properly handle the individual processes.

Using a process manager like supervisord requires that you package
supervisord and it configuration in the image, along with the storagenode
and storagenode-updater binaries.

Installing supervisord requires that we run apk to install it and its
dependencies at build time which makes it difficult to build multi-platoform
images; executing apk forces a requirement of the build system to run
foreign architechtures.

This change adds a dockerfile which will be used to build the base image
for the storagenode and has supervisord packaged. The base image will be
built manually using docker buildx, with QEMU binfmt support.

Updates https://github.com/storj/storj/issues/4489

Change-Id: I33f8f01398a7207bca08d8a4a43f4ed56b6a2473
2022-03-04 14:34:58 +00:00
Sean Harvey
d4a6524673 Makefile: use arm64 executable for arm64v8 storagenode images
This change fixes regression (one line of 7e63afb) that overwrote the
previous fix (10d7a63).

Change-Id: I892b0c0445ac0fbe6eb30c131edc96fd1d33734d
2022-03-02 14:30:29 +01:00
Stefan Benten
f89e030c3f final touches
Change-Id: I8fcc986f71f592507dc3f35ce4352647da46d461
2022-02-21 13:08:48 +02:00
Stefan Benten
76be9e6efd cmd/multinode,Makefile: build docker image for multinode dashboard
Closes #4547

We do not build an docker image for the multinode dashboard,
which makes monitoring for docker-focused environments harder.
This adds the basic image and ties it into CI/CD.

Change-Id: I14c01a7f1f0019f6f5c1b8fd75dc424fc362b18d
2022-02-21 13:08:48 +02:00
Jeff Wendling
ce3a405644 makefile: remove uplink-image
Change-Id: I3552de12fa99892bcd7e83aa5b0fdddb268f7db5
2022-02-17 16:01:53 -05:00
Jeff Wendling
9061dd309f cmd/uplinkng: become cmd/uplink
Change-Id: If426c32219d32044d715ab6dfa9718807f32cb9f
2022-02-09 17:02:21 +00:00
Clement Sam
7e63afbef6 storagenode: docker image autoupdate binaries
Get storagenode and storagenode-updater binaries during
run of the container to not to release new docker image
on each new version of the storagenode binary.

Fixes https://github.com/storj/storj/issues/4176

Change-Id: I994c4942136a2cc7298eb0346238689eb406ae5b
2022-02-02 11:40:04 +00:00
Ivan Fraixedes
8e80a51b64 Makefile: Set NodeJS version to use in var
Add new variable in the Makefile to specify the used NodeJS version so
it can be referenced wherever else in the file rather than hard-coding
it.

Having in to a variable makes more visible and quicker to find out which
NodeJS version the project uses.

Change-Id: I87abc3b3e5489407638618f8199aec7c2f40ca9f
2022-01-13 08:52:26 +00:00
Stefan Benten
35113634f6 Makefile: do not remove files to keep state clean
Currently the admin UI Makefile target deletes the .gitignore file.
While it being unnecessary to my findings it also causes and dirty
git repository, since files have been modified.
Removing this command lets the build process continue to work as expected
and since we are not commiting any files or assets later on this should
be fine.

Change-Id: Ibc1b7a8e456394ca19ea39dd8389c2ec03f066fc
2022-01-07 20:36:50 +01:00
Stefan Benten
0334465306
Makefile: fix variable used to trigger release drafting (#4319)
Small fix to properly trigger the script.
2021-12-22 13:33:07 +01:00
Stefan Benten
e2481fb510 scripts,Jenkinsfile,Makefile: upload binaries to drafted github release
This change adds a script and the needed build logic to create a draft github release and then upload the created binaries to it in order to make the actual publishing a lot less error prone.
The logic is currently that it only does this for all github tags, but not for every main build/push. This is handled by the checks in the script itself.

Change-Id: Ie172a8e4a97200de901a26a055aa5a8a54b60a2a
2021-12-21 13:55:38 +00:00
Ivan Fraixedes
c10cc302b5
Makefile: Fix target for building Admin UI
Fix the Makefile target for building the Satellite Admin UI.

Change-Id: Id9087c313c451fa68796c9a495704ea3b3728a5c
2021-12-20 12:11:10 +01:00
Ivan Fraixedes
5573ece848 satellite/admin/ui: Migrate to SvelteKit
Migrate the satellite admin UI web app from the Svelte template used to
generate a Svelte App scaffolding to SvelteKit.

There aren't any functional changes in the  application, however, the
commit has a lot because:

1. SvelteKit uses a different directory layout and constraints to it, so
   the files have been moved.
2. The files have changed its formatting due to the new default linter
   configurations that SvelteKit uses.
3. The linter detected some issues with using `object` and `any` types
   in Typescript, so they have been replaced by better general types
   (e.g. Record).

The  migration allows to use the new tooling rather than Rollup
directly, besides that will empower the future of it when it needs more
features (e.g. different routes, etc.).

Change-Id: Ifa6736c13585708337f6c5a59388077b784eaddd
2021-12-17 10:27:13 +00:00
Stefan Benten
6799354748 ci: bump to go 1.17.5
Change-Id: I072bebaac8c12b31bc68ab4058b1991c4f6d5e12
2021-12-13 11:25:22 +01:00
Ivan Fraixedes
cb499cb7db
Makefile: Fix target builds the satellite admin UI
Fix the Makefile target which build the satellite admin UI adding a
missed npm command which must run before the npm run build command.

Change-Id: I96c9b0bd2ad2d909fc680db2b22a6cf13931111a
2021-11-12 13:29:55 +01:00
Ivan Fraixedes
ab425d4248 build: Consider the Admin UI assets
Add into the Makefile and Jenkins the operations to build the Admin UI
assets and lint them.

Change-Id: I687c1adec796fd591664e34eb73a465e53096b58
2021-11-12 08:48:28 +00:00
Artur M. Wolff
785eb93cd7 ci: install gateway@latest instead of @main
Gateway-ST frequent release cycle has been resurrected, which means it's
safer to use the latest release tag in the storj repository's CI now.

Change-Id: I9df1c789a9b9418ba7cceaec9cfec3cc6c448284
2021-11-01 12:16:38 +00:00
Michał Niewrzał
d3a0364f21 Makefile: add uplinkng to build binaries
We want uplinkng binaries easily accessible.

Change-Id: I83e1d90f9413ebd97e1a51d0435548bc0cac8ad8
2021-10-28 09:47:36 +02:00
Egon Elbre
c54bcd63a1 Makefile: bump node to v16
Change-Id: I0587168f373230710c7b50b9aefd34718f4a5b3e
2021-10-14 12:46:28 +00:00
Egon Elbre
db2cb33db9 ci: use correct Go version for the image
Change-Id: Ieed044780a38ebd209662ddacb24ba5e2998a319
2021-09-30 14:36:21 +03:00
Artur M. Wolff
cc9b845a83 ci: switch back to pulling gateway@main
Switch back to pulling gateway@main for continuous integration because
https://review.dev.storj.io/c/storj/gateway/+/5834 was merged. This
change is a partial revert of
https://review.dev.storj.io/c/storj/storj/+/5832.

Change-Id: I0a87f6f18a9a863fe92003b76c830335f9253ced
2021-09-28 15:01:04 +00:00
Egon Elbre
8ef03b0967 ci: cleanup ws before build, fix gateway install
Jenkins uses the same folder for different PR-s. Sometimes other runs do
not cleanup after themselves (e.g. timeout), hence add a cleanWs step to
ensure we delete files in the workspace.

gateway-st introduced a replace directive in go.mod, which does not work
with go install. Hardcode to the last version without the directive.
Using this fix to unblock ci builds.

Change-Id: I5e5d75bf47e30a5a8b6d835867c0c9176f25e08a
2021-09-23 15:57:47 +03:00
Egon Elbre
6e660cecdd Jenkinsfile: test cross-compile and bump deps
Change-Id: I47091de2e80bd96fcada616f75e1db07a59fb0c1
2021-09-16 18:59:31 +03:00
Egon Elbre
6b3718f33e Makefile: skip darwin build for the time being
Mac builds need to be reworked to work with latest Go version.

Change-Id: Ifbe37cf8fb41c5e0d806c518a3be32e093b448b0
2021-09-16 12:53:41 +03:00
Igor
cd973fcb78
Revert "Makefile: disable building storagenode for darwin" (#4186)
This reverts commit d867a7007f.
2021-09-08 19:56:29 +03:00
Egon Elbre
d867a7007f Makefile: disable building storagenode for darwin
Also a few cleanups to ensure it wasm is built when needed.

Change-Id: I7612d969dddbfc1d8589cf00666c1c60d6312140
2021-09-08 13:46:58 +00:00
paul cannon
fda316b461 Makefile: update channel name of #team-data
Avoids confusion when metrics need to be changed or updated.

Change-Id: I1d0bf939bd30cd66128eb04ccee5c4133447bfe9
2021-07-30 09:58:57 -05:00
Artur M. Wolff
10d7a63ab0 Makefile: include arm64 (instead of arm) executables in arm64v8 images
Change-Id: Ic9cd5cd7b052efcb75499c338dd60480f960c4fa
2021-07-30 14:01:48 +02:00
Yaroslav Vorobiov
6db6b76b27 multinode/console: embed web assets
Embed web static files in multinode binary to be able to
release multinode as single binary.
Add make commands to build multinode binary with embeded web
assets.

Change-Id: I348aff7a7d847fae5c021cbf59abc7f892c0df80
2021-07-12 18:51:24 +03:00
Yaroslav Vorobiov
f0d60a6ec2 Jenkins: add multinode UI stage
Change-Id: I77ed791ff9dbb5b046450ab625c1763b34691888
2021-05-21 15:49:31 +03:00