Commit Graph

1153 Commits

Author SHA1 Message Date
Jeremy Wharton
dba28f1d61 cmd/satellite: add project ID, bucket name to partner attribution CSV
This change adds project ID and bucket name columns to the generated
partner attribution report. Attribution values are now summed based on
their project ID and bucket name in addition to their user agent.
Additionally, the command to generate the attribution report has been
modified to optionally include only certain user agents.

Change-Id: I61a1d854379134f26b31467d9e83a787beb451dd
2022-06-01 20:27:12 +00:00
Márton Elek
84d02f7fbf metabase: use predefined snapshot for unit tests avoid migration
Change-Id: I6225ae32c312030e3559d8ed0faa137ffc3cc5b7
2022-06-01 12:07:00 +00:00
Ivan Fraixedes
aa913e63db cmd/storj-sim: Fix path to admin UI assets folder
The admin UI assets aren't inside of the `web` directory they are
directly in the `satellite` one.

The invalid path provoked that storj-sim generated a satellite
configuration with an invalid path to the Admin UI static assets
provoking that it didn't load the UI by default.

Change-Id: I49fb289377f51634057173690fbd8cf863ca9a9d
2022-05-31 18:54:54 +00:00
Michał Niewrzał
ffbb43ddbc cmd/uplink: fix how we are collecting errors while copy in parallel
Main issue was that when one part copy failed while being inside
goroutine (limiter) and another part was still collecting src/dst parts
it was possible to drop errors from failed part copy. It was possible
bacause on fail context was canceled and if we were still getting
part src/dst then it was returning error immediately and error
group with errors from goroutine was ignored.

Change-Id: I75c6799eba358741629795f2971c7a964cb2c9ce
2022-05-31 10:18:51 +00:00
Michał Niewrzał
4f2fae4f28 cmd/uplink: better error handling for parallel transfer
Few improvements were made to how we are handling errors
while doing parallel upload/download for single object:
* unhide error under 'context canceled' which was shown in most of
cases
* add part number to error message
* don't try to commit if any error occurs while operation
* combine errors into more readable form, example:

---
failed to download part 3: uplink: eestream: failed to download stripe 0:
error retrieving piece 00: ecclient: piecestore: rpc: tcp connector failed: rpc: dial tcp 97.119.158.36:28967: i/o timeout
...
error retrieving piece 89: ecclient: piecestore: rpc: tcp connector failed: rpc: dial tcp 161.129.152.194:28967: i/o timeout
failed to download part 1: uplink: eestream: failed to download stripe 0:
error retrieving piece 01: io: read/write on closed pipe
...
error retrieving piece 97: io: read/write on closed pipe
failed to download part 2: uplink: eestream: failed to download stripe 0:
error retrieving piece 00: io: read/write on closed pipe
...
error retrieving piece 01: ecclient: piecestore: rpc: tcp connector failed: rpc: dial tcp 180.183.132.234:28967: operation was canceled
error retrieving piece 96: io: read/write on closed pipe
	main.(*cmdCp).parallelCopy:418
	main.(*cmdCp).copyFile:262
	main.(*cmdCp).Execute:156
	main.(*external).Wrap:123
	github.com/zeebo/clingy.(*Environment).dispatchDesc:126
	github.com/zeebo/clingy.(*Environment).dispatch:53
	github.com/zeebo/clingy.Environment.Run:34
	main.main:26
	runtime.main:250
---

Change-Id: I9bb70b3f754567761fa8d17bef8ef59b0709e33b
2022-05-27 14:00:35 +00:00
Paul Willoughby
1d97b2c855 cmd/uplink: add use - for stdout,stdin to cp help
Change-Id: Ife3a0972d1be119a73eaefc0e23407b74fe03f54
2022-05-26 10:27:20 -06:00
Paul Willoughby
036d5a8619 cmd/storj-sim: fix printing node url
Fix printing satellite url to help configure authservice.
  storj-sim network env SATELLITE_0_URL

Change-Id: I786ab6723f837bbc35f1228eb73a82427d2f9250
2022-05-25 00:09:17 -06:00
Michał Niewrzał
d90ce467fc cmd/uplink: bring back --metadata for cp command
At some point uplink cli lost ability to set metadata. This change
brings back this functionality for 'cp' operation.

https://github.com/storj/storj/issues/3848

Change-Id: Ia5f60eb577fcab8a38d94730d8cdc6e0338d3b46
2022-05-18 15:58:53 +00:00
Michał Niewrzał
f9a3f19443 scripts: add 'uplink cp' tests for stdin/stdout
Uplink can upload from stdin and download to stdout. We had
such tests for old binary but now we were missing it.

Change-Id: I5110a9f531f5cc21277fa53611995fb5b556ff16
2022-05-18 08:42:04 +00:00
Jeff Wendling
836fa188fa cmd/uplink: allow listing invalid access grants
if you somehow get an invalid access grant in your
config file, it'd be nice to be able to list it
and delete it and stuff.

Change-Id: I7e335bf32353f294d5abb6a7c5f8f3aa18f2f6a7
2022-05-17 20:00:36 +00:00
Clement Sam
87ea2a4794 cmd/storagenode: make supervisor HTTP Server configurable
The current supervisord condifguration sets up the HTTP server
to listen on a tcp socket which is private i.e. available only
on localhost. This poses a regression where multiple containers
cannot be run if the host network interface is used when docker
container is run with `--network host` option.

This change adds a new env variable `SUPERVISOR_SERVER`, with
potential values `unix | private_port | public_port`, where
`unix` is set as the default value.

By default, the HTTP server is now set to listen on a UNIX
domain socket.
The file path is set to `/etc/supervisor/supervisor.sock`
instead of the /tmp directory since some systems
periodically delete older files in /tmp. If the socket file is
deleted, supervisorctl will be unable to connect to supervisord.

When SUPERVISOR_SERVER is set to `public_port` or `private_port`,
the HTTP server is set to listen on a TCP socket.

Resolves https://github.com/storj/storj/issues/4661

Change-Id: I224836dcae0293bcfe49874f2748be7723944687
2022-05-16 20:06:24 +00:00
JT Olio
5fb9ee3cfa uplink, satellite: use bgp hostnames
Change-Id: I58f5011d3019f8267fa8cbd3096b2cfe42eb5f8b
2022-05-16 18:14:37 +00:00
Stefan Benten
7afdb15fc8
cmd/uplink: adding Length Method to MultiReadHandle
This changes allows fetching the file size more easily (for supported
files) in order to afterwards calculate the multipart part size
accordingly.

Change-Id: Idabba4c2ee794ee471973889f5843174a7acad35
2022-05-13 21:31:45 +02:00
Stefan Benten
5e4ec0b3be
cmd/uplink: adjust multipart part size based on file size
This change allows the uplink to bump the part size based on the
content length that is being copied. This ensures we are staying
below the 10k part limit currently enforced on the satellites.

If the user specifies the flag, it will error out if the value
chosen by the user is too low. Otherwise it will use it.

Change-Id: I00d30f603d941c2f7703ba19d5923e668629a7b9
2022-05-13 21:31:23 +02:00
Egon Elbre
4791ba5d50 satellite/{mailservice,oidc}: improvements to debugging
Things that make debugging easier.
* Added logging to automatic link clicking to make it obvious, when it
  fails.
* Added monitoring to oidc.
* Made dbx create calls noreturn for oauth_*

Change-Id: I37397b4e84ce5bfd82954aed9c38fdfd52595f24
2022-05-11 19:59:42 +00:00
Jeff Wendling
89ccfe2dd7 cmd/uplink: fix recursive copy and improve tests
recursive copy had a bug with relative local paths.

this fixes that bug and changes the test framework
to use more of the code that actually runs in uplink
and only mocks out the direct interaction with the
operating system.

Change-Id: I9da2a80bfda8f86a8d05879b87171f299f759c7e
2022-05-11 15:17:16 -04:00
Erik van Velzen
928375a67c satellite/repair/queue: buffer batch insert
Implement a buffer for inserting repair items into the queue in a batch.

Part of https://github.com/storj/storj/issues/4727

Change-Id: I718472b2f2b1f4993c3d6f15c44923776407155a
2022-05-11 09:02:20 +00:00
Clement Sam
b471a5d8e3 cmd/storagenode/Dockerfile: bump storagenode-base image version
The new storagenode base image version contains the fix for the
failing "processes" supervisord event listener.

Resolves https://github.com/storj/storj/issues/4772

Change-Id: I6d67aa6f85ee33cd9abe6a663e4f9a84ea57fdbf
2022-05-04 16:00:26 +00:00
Clement Sam
9751ac9de2 cmd/storagenode: fix failing supervisord "processes" eventlistener
/bin/stop-supervisor fails in posix shell since the standard read utility
takes at least one variable's name as argument.

Changing the header #!bin/sh to #!/bin/bash fixes this issue.
`read` with no variable's name works in bash.

Looks like the shell in alpine isn't POSIX-compliant so we didn't
encounter this issue on alpine.

Also, I changed the name from "processes" to "processes-exit-eventlistener"
to make it clearer in the logs since supervisord spawns event listeners as
separate processes.

Change-Id: Ife9378c2013e2eb54f2adcd52a163d64eaacbbab
2022-05-04 02:44:23 +00:00
Clement Sam
87cb2e92b9 cmd/storagenode: bump storagenode-base image version
Change-Id: I8df391ee12c70cc599b54115b54eb987a2114b74
2022-05-04 01:59:00 +00:00
Clement Sam
a9faf756b0 cmd/storagenode: get rid of CRITICAL log when running supervisord as root
Closes https://github.com/storj/storj/issues/4729

Change-Id: Iea9a60ff60d753af375977f9bd90557574f80de0
2022-05-03 21:18:11 +00:00
Clement Sam
4a46d41c1c cmd/storagenode: change supervisord pidfile location
When running the docker auto-updater image as non-root user,
supervisord logs a "CRIT could not write pidfile /run/supervisord.pid"
since the user does not have permission to the /run directory.

Changing the location to /etc/supervisor fixes it because permissions
are set for non-root access of the /etc/supervisor directory.

Closes https://github.com/storj/storj/issues/4730

Change-Id: Id463f3a08db44dd9283921ece4575abdad9bd7f2
2022-05-03 12:54:40 +00:00
Stefan Benten
345ab87b5c cmd/uplink: adding output flag for ls command
With this change users can use the uplink cli in
scripts (ie. bash) more easily, since the output
can be switched to an easier processable json format.
It keeps the default of tabbed output.


Change-Id: I37e2c55f75c2250c3119fd8df8b66a766ff9096b
2022-04-29 10:32:04 +00:00
Paul Willoughby
8efed4b270 cmd/uplink: raise fd limits
Change-Id: I507c92c38d45d3d2be3edee08b6b85552ae21c43
2022-04-27 18:11:44 +00:00
Jeff Wendling
fead0db981 cmd/uplink: return nicer error message during import
Fixes #4750

Change-Id: Icef3340dd04d1c502b1651dcc70362895f258563
2022-04-27 09:30:57 +00:00
Jeff Wendling
f25ead5f98 cmd/uplink: set default parallelism to 1
Change-Id: Ic4198131c9958cc864fd861f983e32776bf56595
2022-04-26 22:55:11 +00:00
Cameron
dd6ab25cbd {cmd/satellite/reports, satellite/attribution}: type and variable name adjustments
Change-Id: I553b7ed9cb702479911baaf54d77d8f2662faaa2
2022-04-26 20:12:38 +00:00
Egon Elbre
1ed36e9fea cmd/uplink: make clearer ctx cancellation path in copy
When ctx is cancelled limiter won't start a new goroutine.
The code didn't immediately return an error in that case.

The dst.Commit(ctx) would fail anyways due to a cancelled ctx.
However, we can make the behavior clearer by returning immediately.

Change-Id: I65df7ca85de55813f3200a50db2eaaa7a297ba2c
2022-04-25 18:16:46 +03:00
Egon Elbre
c2bdd4effa cmd/uplink/ulfs: disallow writes after first failure
It was possible for the a previous write / part to fail or be aborted
and the next part write still happened. This causes a data ordering
corruption.

The whole write to parallel stdout fails, so there shouldn't be
confusion with regards to the output acceptability. However, it would
be clearer, if we avoided writing out-of-order data... mainly to be
clear that we didn't corrupt the data, just that it's incomplete.

Change-Id: I97b0d14404f29e8615e7d29b10cbd61ccb861e40
2022-04-25 18:16:46 +03:00
Egon Elbre
847ddaaab0 cmd/uplink: cancel on failed copy
Also ensure that abort is given at least 5 seconds to clear up any
pending uploads on cancellation.

Change-Id: I814aa407ee5783f2609a76b54de2879dcd5f89bb
2022-04-22 14:57:24 +03:00
Kaloyan Raev
978e0f1a26 cmd/uplink: cp sets connection pool capacity based on parallelism
If the cp command is executed with higher level of parallelism, it would
open more connections to storage nodes at the same time. Therefore, the
connection pool capacity should be expanded accordingly.

The pool capacity is set to 100 * parallelism.

Change-Id: Ia8b3ab6a99340d8cbb87a7b80c3354b2b21c1958
2022-04-21 14:10:08 +00:00
Clement Sam
b24e5cbc44 cmd/storagenode: silent authentication checks CRITICAL logs by supervisord
set dummy user/password in the [unix_http_server] and [supervisorctl] sections of the config file to remove the CRIT log entries as suggested here: https://github.com/Supervisor/supervisor/issues/717

Closes https://github.com/storj/storj/issues/4731

Change-Id: Iec10be39416f3aaa2d7febb414e15a63a4b23a4c
2022-04-21 13:10:42 +00:00
paul cannon
1422a1ff19 cmd/uplink: use 64 MiB for parallel chunk size, not 64 MB
I don't think it should matter for correctness whether this matches the
segment size or not, so I think there is something else wrong. However,
making this change seems to eliminate the "corruption when ulimit -n is
too low" problem we're seeing right now.

Change-Id: I232fe0d0a371b86ddf902e8c2d4778e140b2f1fc
2022-04-19 12:08:08 -05:00
Cameron
48fb3e947c cmd/satellite/reports: sum attribution data by user agent
Attribution is attached to bucket usage, but that's more granular than
necessary for the attribution report. This change iterates over the
bucket attributions, parses the user agent, converts the first entry
to lower case, and uses that as the key to a map which holds the
attribution totals for each unique user agent.

Change-Id: Ib2962ba0f57daa8a7298f11fcb1ac44a8bb97875
2022-04-18 13:23:37 +00:00
Yaroslav Vorobiov
a401450e3d cmd/storagenode-updater: add should-update cmd
This change add Separate CMD to check if a binary needs to be updated

Change-Id: I87f4214f84105464d283f1622dcb4ca9e796e771
2022-04-12 17:55:54 +00:00
Clement Sam
e9611801ad cmd/{storagenode,storagenode-updater}: add Process name to logs
Now that we have both the storagenode and updater processes running
in a single docker container, we need a way to know which log entry
is logged by any of the processes.

This change includes a Process field in the log entries.

Resolves https://github.com/storj/storj/issues/4648

Change-Id: I167b9ab65728a41136d264b5fe2c41bb64ed1785
2022-04-12 16:49:56 +00:00
Cameron
d4ad3a3ca6 satellite/satellitedb/attribution: update value attribution query to return byte-hours
Before, the VA query was summing the total and dividing by the number of
rows. This gives the average bytes stored per hour, but we charge for
usage with byte-hours. Why not do value attribution the same way?
To do that, we don't divide by the number of rows. We also have object
and segment fees so return segment-hours and object-hours too.

Change-Id: I1f18b7e1b2bae1d3fae1ca3b93bfc24db5b9b0e6
2022-04-08 16:22:21 -04:00
Clement Sam
e486585853 Dockerfiles: switch base image from alpine to Debian
We've had a lot of issues with alpine and currently there's a broken
network issue on alpine for users running on RPI arm32 architechture
which requires a workaround before docker is able to sync time between
the host and the container: https://wiki.alpinelinux.org/wiki/Release_Notes_for_Alpine_3.13.0\#time64_requirements.

Since we're switching the base image of the storagenode to debian,
it's best to switch the base image of all our docker images to
debian as well for consistency; less drift across them and keeps
the push target consistent.

Change-Id: If3adf7a57dc59f19ef2221b892f340d919798fc5
2022-04-05 11:40:44 +00:00
Erik van Velzen
61a47f3e95 cmd/uplink: refactor date parsing
Change-Id: I6a5cbdf86eecdc5578f3dae7a8ab1b0d4485e1da
2022-04-05 01:03:20 +00:00
Cameron
2b39df460c cmd/nullify-bad-user-agents: set user_agent to NULL where user_agent = partner_id
In the migration to migrate the corresponding name of the partner id to
user agent, part of the requirement was to migrate the partner id
itself if there was no partner name associated. This turned out to not
be so good. When we parse the user_agent column later, it is returning an
error if the user agent is one of these UUIDs.

Change-Id: I776ea458b82e1f99345005e5ba73d92264297bec
2022-03-31 16:57:50 -04:00
Qweder93
2a7b20e8e4 cmd/uplink: integrate server-side copy with uplink cp command
Resolves https://github.com/storj/storj/issues/4486

Change-Id: I42ac2ad2e1a05df4a83606f1990b639f08791403
2022-03-31 09:25:29 +00:00
Egon Elbre
e5972d8920 all: fix linting errors
strings.Title is deprecated in Go 1.18, replace it with
golang.org/x/text/cases.

Change-Id: I6185b97d37309dbe4a6715f794383ab259a0658b
2022-03-30 18:45:23 +03:00
Clement Sam
d611c16891 cmd/storagenode: update storganode-base image version in Dockerfile
Change-Id: I2e40f75d7cf1f673ea16a633a427225f97ea92ed
2022-03-29 13:26:33 +00:00
Clement Sam
c641f4c9ac cmd/storagenode: use Debian as base image instead of alpine
We are switching from alpine to debian due to a network issue
introduced in alpine 3.13+ which fails to verify certificates
due to not all armhf boards meet the time64 requirement:
https://wiki.alpinelinux.org/wiki/Release_Notes_for_Alpine_3.13.0\#time64_requirements

Also, Debian does not have official imagess for arm32v6 architecture
so we are building with arm32v5 arch in the Makefile.

Change-Id: I3660c3f64b7c2b342dd4ccb876af5f4e3036ea9d
2022-03-25 08:19:20 +00:00
Egon Elbre
0d2d59f884 all: fix linting issues
Change-Id: Idfc93948e59a181321d79b365e638d63e256a16f
2022-03-21 15:26:42 +00:00
paul cannon
d253b4c033 cmd/satellite: fix fetch-pieces segfaulting
When there is an error fetching a piece, the reader might be present or
it might not, depending on how far the fetch operation got. The
fetch-pieces code did not handle the "reader-not-present" case. Now it
should.

Change-Id: I263657d544d0ab8ba5d307a34ffc76bbf56835d0
2022-03-19 04:34:15 +00:00
Clement Sam
36770448b1 cmd/storagenode: update base image for storagenode
Updating the version of the base image for the storagenode docker image.

Also fixes the non-root permission issue to /app directory

Change-Id: I8b55a1e3062f55ce6fc52e126ec1a18bfa24e669
2022-03-18 13:37:12 +00:00
Clement Sam
51e5e949a4 cmd/storagenode: bug fixes to storagenode docker image
This change fixes the following issues:

wget: Alpine docker image by default uses the builtin BusyBox wget which is not capable of handling SSL traffic via proxy unlike the GNU wget. We have to replace BusyBox wget with GNU wget.

updater failing to restart the node: supervisorctl pointing to wrong config file. We remove the default configuration file and point supervisorctl to custom config in systemctl

updates https://github.com/storj/storj/issues/4489

Change-Id: I24a7f18377ba723bbc377bb5d25aaa14f37021b1
2022-03-18 09:03:24 +00:00
Cameron
9ebe388621 cmd/partnerid-to-useragent-migration: add ability to limit updates in migration
Add ability to limit updates in migrations.
To make sure things are looking okay in the migration, we can run it
with a limit of something like 10 or 30. We can look at the output of
the migrated columns to see if they are correct. This should have no
effect on subsequently running the full migration.

Change-Id: I2c74879c8909c7938f994e1bd972d19325bc01f0
2022-03-16 20:48:39 +00:00
Cameron
41a095a539 cmd/satellite: fix args indexing in cmdValueAttribution
Change-Id: Ife7432132563a8b6858d437e16025ce201a488b0
2022-03-16 16:42:04 +00:00