Commit Graph

196 Commits

Author SHA1 Message Date
Clement Sam
d611c16891 cmd/storagenode: update storganode-base image version in Dockerfile
Change-Id: I2e40f75d7cf1f673ea16a633a427225f97ea92ed
2022-03-29 13:26:33 +00:00
Clement Sam
c641f4c9ac cmd/storagenode: use Debian as base image instead of alpine
We are switching from alpine to debian due to a network issue
introduced in alpine 3.13+ which fails to verify certificates
due to not all armhf boards meet the time64 requirement:
https://wiki.alpinelinux.org/wiki/Release_Notes_for_Alpine_3.13.0\#time64_requirements

Also, Debian does not have official imagess for arm32v6 architecture
so we are building with arm32v5 arch in the Makefile.

Change-Id: I3660c3f64b7c2b342dd4ccb876af5f4e3036ea9d
2022-03-25 08:19:20 +00:00
Egon Elbre
0d2d59f884 all: fix linting issues
Change-Id: Idfc93948e59a181321d79b365e638d63e256a16f
2022-03-21 15:26:42 +00:00
Clement Sam
36770448b1 cmd/storagenode: update base image for storagenode
Updating the version of the base image for the storagenode docker image.

Also fixes the non-root permission issue to /app directory

Change-Id: I8b55a1e3062f55ce6fc52e126ec1a18bfa24e669
2022-03-18 13:37:12 +00:00
Clement Sam
51e5e949a4 cmd/storagenode: bug fixes to storagenode docker image
This change fixes the following issues:

wget: Alpine docker image by default uses the builtin BusyBox wget which is not capable of handling SSL traffic via proxy unlike the GNU wget. We have to replace BusyBox wget with GNU wget.

updater failing to restart the node: supervisorctl pointing to wrong config file. We remove the default configuration file and point supervisorctl to custom config in systemctl

updates https://github.com/storj/storj/issues/4489

Change-Id: I24a7f18377ba723bbc377bb5d25aaa14f37021b1
2022-03-18 09:03:24 +00:00
Clement Sam
52a80b2c67 cmd/storagenode: use custom storagenode-base image
Updates https://github.com/storj/storj/issues/4489

Change-Id: I3585b50df598c69e58137f61b6a272dbd077c5be
2022-03-15 14:22:35 +00:00
Clement Sam
155c070837 cmd/storagenode: fix permission to /etc/supervisor in base image
This change fixes the `sed: can't create temp file '/etc/supervisor/supervisord.confXXXXXX': Permission denied` issue when editing the supervisord.conf file during runtime as a non-root user.

While editing the config file, Sed creates a temporary file, saves the result and then finally mv the original file with the temporary one. So we need to set the permission for the /etc/supervisor where the temporary file is created.

Change-Id: Ic9c147a9cf0a6ef94adf702e33054edce1828806
2022-03-15 13:01:23 +00:00
Clement Sam
053a38a46d cmd/storagenode: move supervisord config file to storagenode-base image
The supervisord.conf file is edited to set the args for the storagenode and storagenode-updater binaries at runtime. This change moves the config file to the base image so we can set the permission to allow non-root users edit the config file.

Non-root user permission is also needed for the /app directory so we can install/update the binaries when run as a non-root user.

Updates https://github.com/storj/storj/issues/4489

Change-Id: If7a51a00ea171253e41923501174a43393f4638c
2022-03-08 13:47:01 +00:00
Clement Sam
15a1428828 {cmd/storagenode,Makefile}: add storagenode base image Dockerfile
Having the storagenode and storagenode-updater processes in one container
requires a process manager to properly handle the individual processes.

Using a process manager like supervisord requires that you package
supervisord and it configuration in the image, along with the storagenode
and storagenode-updater binaries.

Installing supervisord requires that we run apk to install it and its
dependencies at build time which makes it difficult to build multi-platoform
images; executing apk forces a requirement of the build system to run
foreign architechtures.

This change adds a dockerfile which will be used to build the base image
for the storagenode and has supervisord packaged. The base image will be
built manually using docker buildx, with QEMU binfmt support.

Updates https://github.com/storj/storj/issues/4489

Change-Id: I33f8f01398a7207bca08d8a4a43f4ed56b6a2473
2022-03-04 14:34:58 +00:00
Erik van Velzen
4a26f0c4f1 cmd/storagenode: restore passing arguments
Through `docker run storjlabs/storagenode:latest --help` we have always
made available around 100 command-line arguments.

However if you now pass such an argument it will be passed to
storagenode-update and it may no longer be recognized. This will cause
the storagenode not to start.

This was introduced in
https://review.dev.storj.io/c/storj/storj/+/5426

This change restores previous functionality.

Change-Id: I06823283ff82ffda12aee48c4d83717bddfbfdac
2022-02-04 02:48:39 +00:00
Ivan Fraixedes
7f1dc74e37
cmd/storagenode: Change order load id in setup
Change the order of when the storage node setup node loads the identity
for avoiding to write anything in the disk in the case that there is an
error loading the identity.

This bug was reported by @onionjake Github username's and the specific
changes to make.

Closes #4387 #4396

Change-Id: I360fff3c23b160c9e055203d3526d749edfd9129
2022-02-03 16:00:29 +01:00
Clement Sam
7e63afbef6 storagenode: docker image autoupdate binaries
Get storagenode and storagenode-updater binaries during
run of the container to not to release new docker image
on each new version of the storagenode binary.

Fixes https://github.com/storj/storj/issues/4176

Change-Id: I994c4942136a2cc7298eb0346238689eb406ae5b
2022-02-02 11:40:04 +00:00
Stefan Benten
44a73b01d9 cmd/storagenode: use contact address instead of server address
Currently the address being used is most of the time just :28967, which is not the correct address to reach the node from the public on.
This change uses the designated contact external address value that contains the set and preferred way to reach the node.

Change-Id: I99e979c2541043755b81e65c36c4289bfa3f60f3
2022-01-12 15:15:09 +00:00
Clement Sam
ed9bde8f69 cmd/storagenode: add info command
The info command prints the details of the storagenode
to stdout.

It returns the storagenode info in JSON format
if --json flag is specified which can be piped
to the multinode add command.

Change-Id: I0163db8e02c4ec7346bfa69274d1772669357c6c
2021-12-14 17:05:52 +00:00
Ethan Adams
27c6c6aeae
ci: Golangci lint v1.43.0 changes (#4307)
Co-authored-by: Stefan Benten <mail@stefan-benten.de>
2021-12-09 20:49:48 +01:00
Egon Elbre
4a530ccffd cmd/storagenode: simplify windows service loop
Remove some testing code that is not necessary for the service.

Change-Id: Idd09d62bf022d6e66943983f98642fc3c9aa72f7
2021-11-05 15:56:18 +00:00
Egon Elbre
e721081354 cmd/storagenode: fix service initialization code
func init() code isn't that well defined and reordering of them
could cause problems when starting the whole process from it.

Change-Id: I4088a0db156ece15354877011a481f6f91c9b332
2021-11-05 01:09:41 +02:00
Egon Elbre
1aec831d98 satellite/audit,storage: increase sleep delay in TestMaxVerifyCount
Currently TextMaxVerifyCount flakes in some tests, try increasing the
sleep time to ensure that things are slow enough to trigger the error
condition.

Also pass ctx to all the funcs so we can handle sleep better.

Change-Id: I605b6ea8b14a0a66d81a605ce3251f57a1669c00
2021-09-10 15:30:37 +00:00
igor gaidaienko
d32ae0459b Revert "storagenode: docker image autoupdate binaries"
This reverts commit 244c488b10.
2021-05-13 14:21:06 +03:00
Egon Elbre
7802ab714f pkg/,private/: merge with private package
Initially there were pkg and private packages, however for all practical
purposes there's no significant difference between them. It's clearer to
have a single private package - and when we do get a specific
abstraction that needs to be reused, we can move it to storj.io/common
or storj.io/private.

Change-Id: Ibc2036e67f312f5d63cb4a97f5a92e38ae413aa5
2021-04-23 16:37:28 +03:00
Yaroslav Vorobiov
244c488b10 storagenode: docker image autoupdate binaries
Get storagenode and storagenode-updater binaries during
run of the container to not to release new docker image
on each new version of the storagenode binary.

Change-Id: Ic0eb4a9c18a98598dfd9b96c1d352c7399496fd2
2021-04-21 17:56:20 +00:00
Egon Elbre
d11c2b709e go.mod: bump storj.io/common
* Add missing endpoints.
* Fix deprecated packages and funcs.

Change-Id: I756090c46a4d15eabf6d413a593cdc64c5809bc7
2021-01-13 14:51:08 +00:00
Yaroslav Vorobiov
fb00d099cf multinode/console: list node infos
Change-Id: I5cac49feff2bac6fbd7ac61dfccffd672da8e8c0
2021-01-05 14:49:11 +00:00
Stefan Benten
7eab859030
cmd: ensure proper arch is used for docker container 2020-12-20 09:26:23 +02:00
Stefan Benten
3c77825b19 cmd: move ca-certificates build step to be arch independent
It turns out, that running a docker image build for specific
arches is not possible from amd64 (eg. installing ca-certificates).

Change-Id: I8b8f002b7e532fb4a0c6542d5b573c294c501068
2020-12-08 21:49:41 +01:00
Stefan Benten
a71c908c1e
cmd: add ca-certificates to Docker images (#3986) 2020-12-08 01:38:33 +01:00
Qweder93
a17cd9aa3e storageode/apikey: added service, CLI issue api key
Change-Id: I840cd0fdbd8dca884eefbd111f21fd3990c11e68
2020-11-18 10:40:17 +00:00
Cameron Ayer
5a337c48ec {cmd,private,storagenode}: create storage dir verification during setup
Previously, we created a new file to use for directory verification
every time the storage node starts. This is not helpful if the storage node
points to the wrong directory when restarting. Now we will only create the file
on setup. Now the file should be created only once and will be verified at
runtime.

Change-Id: Id529f681469138d368e5ea3c63159befe62b1a5b
2020-11-11 11:01:36 -05:00
Cameron Ayer
07acf0e574 cmd/storagenode: add docker env variable to toggle running setup
Previously, we ran setup if no config file was found in the expected dir.
However, there may be situations where a previously set up node's files
may be unreachable. In this case, we would prefer to exit with an error
rather than assume this node needs to be initialized.

The solution here is to add a new env variable to call the setup command.
If SETUP == true, the node will setup, but not run. If SETUP != true,
the node will run and not setup.

If a previously set up node runs with SETUP, it will return an error.
If a node runs without an initial SETUP, it will return an error.

Change-Id: Id2c796ec3d43f2add5e5f34fb777a563eae59f2f
2020-11-11 13:11:19 +00:00
Egon Elbre
1903b15474 storagenode/internalpb: move gracefulexit.proto
Change-Id: Ia3614846ed49a39c8f39331516d16d45a695240b
2020-10-30 15:24:56 +02:00
Egon Elbre
cda67a659a storagenode/internalpb: move inspector.proto
Change-Id: I951379c3b2ff00d1bc09d6a49c026a7e723432d6
2020-10-30 14:51:26 +02:00
Egon Elbre
89ce1fe626 storagenode/storagenodedb: add ctx to OpenNew and OpenExisting
Database opening usually dial and hence we should pass ctx to them.

Change-Id: I9160ae95829f22f347bd525904898a47279a7427
2020-10-29 09:52:37 +02:00
Egon Elbre
d0beaa4a87 pkg/revocation: pass ctx into opening the database
Opening a databases requires ctx, this is first step to passing ctx
to the appropriate level.

Change-Id: I12700f39a320206d8a2a4e054452319f8585b44b
2020-10-29 07:15:36 +00:00
Egon Elbre
cf2dd76db7 cmd/satellite: proper log usage
log.Fatal immediately terminates the program without running any defers.
We should properly close all the services and databases.

Change-Id: I5e959cef3eafedeacb3a2062e3da47e8d04e8e75
2020-10-13 16:56:35 +03:00
Egon Elbre
2268cc1df3 all: fix linter complaints
Change-Id: Ia01404dbb6bdd19a146fa10ff7302e08f87a8c95
2020-10-13 15:59:01 +03:00
Yaroslav Vorobiov
a840cb71e7 storagenode: check db version before run
Change-Id: I912f63fd62f2bff10341346c28dfb92fcd683806
2020-09-30 10:58:09 +00:00
Cameron Ayer
ca0c1a5f0c storagenode/{monitor,pieces}, storage/filestore: add loop to check storage directory writability
periodically create and delete a temp file in the storage directory
to verify writability. If this check fails, shut the node down.

Change-Id: I433e3a8d1d775fc779ae78e7cf3144a05ffd0574
2020-08-31 21:20:49 +00:00
Qweder93
6e90ca8b0e cmd/storagenode: CLI dashboard last contact renamed to status
Change-Id: I29bcd62d0d2970d6294dc642b442b60a3331bbd6
2020-08-06 13:54:07 +00:00
JT Olio
c89746a383 storagenode: use hardcoded metric application name
Change-Id: If11f226d03bc4bf70e2465569f8d578750718ae7
2020-07-28 16:24:15 +00:00
Yaroslav Vorobiov
4d2a505788 storagenode/db: explicitly open and create dbs
To prevent storagenode from implicitly recreating missing dbs and storage,
as such behaviour leads to audit failures. Do not allow storagenode to
start if any of dbs or storage is missing, corrupted, or dedicated storage disk is
unmounted, to get downtime instead.

Change-Id: Ic64e1f0ff4d8ef5b2fddbe7a7e53df4f4bd8652e
2020-07-24 14:08:47 +03:00
Egon Elbre
b67d7ecbc5 cmd/storagenode,storage/cockroachkv: better error handling
Change-Id: I6646aa046dc365c0dee38f23041be4fc2defb759
2020-07-16 20:03:50 +03:00
Egon Elbre
d8dcae3075 all: fix error checking
Change-Id: Ia0da1bbd6ce695139922f94096c2419281905e32
2020-07-16 19:13:14 +03:00
Egon Elbre
080ba47a06 all: fix dots
Change-Id: I6a419c62700c568254ff67ae5b73efed2fc98aa2
2020-07-16 14:58:28 +00:00
Ivan Fraixedes
37cfc01ac6 cmd/storagenode: Add long description to ge command
Add a long description to the graceful exit command to clearly mention
that the command is interactive asking which satellite the SNO wants to
exit.

Change-Id: Icd4056a470e707322f600133e63d9dc56eb877b7
2020-06-24 08:47:52 +00:00
Qweder93
e52809d53e cmd/storagenode: add check if satellites available to gracefulexit
Change-Id: I8747507593d810bbdec0d140de0600ee147011c3
2020-06-10 13:38:36 +00:00
Egon Elbre
0de1b5fa9f ci: fix golangci-lint failures
Change-Id: Ia8c4dd03439ce4e3f85890d45255b8ea1c94b50a
2020-06-05 12:55:35 +00:00
Ethan
159df8b2e4 Add logging listener for retrieving and setting log levels
See https://storjlabs.atlassian.net/browse/SM-752

These changes allow us to change the log level at runtime through a handler off of the debug endpoint.

Examples of changing the log level on storj-sim

To get the current level for the satellite api process:
curl -XGET 'http://127.0.0.1:10009/logging' --header 'Content-Type: text/plain'

To change the log level:
curl -XPUT 'http://127.0.0.1:10009/logging' --header 'Content-Type: text/plain' --data-raw '{"level":"error"}'

Change-Id: I05d164b290929fa06b6d78c01075ee41f8238044
2020-05-12 16:38:06 -04:00
Egon Elbre
ec589a8289 all: fix comments about grpc
Change-Id: Id830fbe2d44f083c88765561b6c07c5689afe5bd
2020-05-11 13:05:34 +03:00
Egon Elbre
c630cf2490 storagenode/pieces: implement buffering for writing
Currently uploads can cause a lot of IOPS, reduce this by introducing a
in-memory buffer on-top of the file.

Change-Id: I5f4e3e01c0a36258271d180b922107de447bcb59
2020-05-04 06:01:32 +00:00
Egon Elbre
8928399d02 all: rename CreateTables to MigrateToLatest
CreateTables hasn't been quite true for a while now, rename to
MigrateToLatest to be clearer in it's behavior.

Change-Id: Ida48e95122a5d9b7a814e922d3698e00024a2ba7
2020-04-30 07:21:17 +00:00