Commit Graph

1310 Commits

Author SHA1 Message Date
Jeff Wendling
567986e388 cmd/uplink: minimal tracing support
this allows one to specify a trace id and cause any remote
spans to be sent up to wherever. it doesn't collect any
local traces.

Change-Id: Ia87e294bb276d966f9f3dbfbaf6e7916b1ec7af9
2022-07-26 11:39:30 -04:00
Jennifer Johnson
61649d2850 cmd/node-cleanup: command to delete uncontacted nodes
Some nodes were added to the nodes table due to a bug in quic
based storagenode contact code. This is a tool to clean up these nodes.

Delete with batch-size 1k seems to take ~400ms on local CockroachDB.

Change-Id: Ic0c1180528c27952e19c431fc9cc327292a10a5f
2022-07-12 12:15:49 +00:00
Ivan Fraixedes
66f12992b0
cmd/storagenode: Don't abbreviate multinode in help
Don't abbreviate multinode in the command help message because there
isn't a need for it and the abbreviation isn't clear at all.

Change-Id: I7a1f2be6ae1f7d4b287c18c48b22c630549b731f
2022-07-05 19:29:12 +02:00
Egon Elbre
05e165283f storagenode/console/consoleapi: use fixed time.Now()
It seems the tests relied on time.Now(), which might cause some
discrepancies in calculations. Use a fixed time.Now() rather than
recalculating.

As a sidefix, remove "Test" prefix from t.Run. These are unnecessary.

Change-Id: I1de903fcf0fcf46fc8e3acf2463e17239b8e3cc6
2022-07-01 12:36:01 +03:00
Jeff Wendling
e00423e2e9 cmd/uplink: disable deleting local files
Change-Id: Id3687f3346f8ec9b5da5459a57d91d1d30bf5566
2022-06-27 10:30:04 -04:00
Paul Willoughby
de125eeb51 cmd/uplink: Fix typo in flag description
Change-Id: I6db38ecec2549d288285a9865f97badfe808b892
2022-06-21 11:55:02 -06:00
Michał Niewrzał
7e387af010 cmd/uplink: add buffering while writing to stdout
Current pipelining to stdout is synchronous so we don't have any
advantage from using --parallelism flag. This change adds buffer
while writing to stdout. Each part is first read into the buffer
and flushed only when all data was read from this part.

https://github.com/storj/uplink/issues/105

Change-Id: I07bec0f4864dc4fccb42224e450d85d4d196f2ee
2022-06-09 15:10:04 +00:00
Jeremy Wharton
dba28f1d61 cmd/satellite: add project ID, bucket name to partner attribution CSV
This change adds project ID and bucket name columns to the generated
partner attribution report. Attribution values are now summed based on
their project ID and bucket name in addition to their user agent.
Additionally, the command to generate the attribution report has been
modified to optionally include only certain user agents.

Change-Id: I61a1d854379134f26b31467d9e83a787beb451dd
2022-06-01 20:27:12 +00:00
Márton Elek
84d02f7fbf metabase: use predefined snapshot for unit tests avoid migration
Change-Id: I6225ae32c312030e3559d8ed0faa137ffc3cc5b7
2022-06-01 12:07:00 +00:00
Ivan Fraixedes
aa913e63db cmd/storj-sim: Fix path to admin UI assets folder
The admin UI assets aren't inside of the `web` directory they are
directly in the `satellite` one.

The invalid path provoked that storj-sim generated a satellite
configuration with an invalid path to the Admin UI static assets
provoking that it didn't load the UI by default.

Change-Id: I49fb289377f51634057173690fbd8cf863ca9a9d
2022-05-31 18:54:54 +00:00
Michał Niewrzał
ffbb43ddbc cmd/uplink: fix how we are collecting errors while copy in parallel
Main issue was that when one part copy failed while being inside
goroutine (limiter) and another part was still collecting src/dst parts
it was possible to drop errors from failed part copy. It was possible
bacause on fail context was canceled and if we were still getting
part src/dst then it was returning error immediately and error
group with errors from goroutine was ignored.

Change-Id: I75c6799eba358741629795f2971c7a964cb2c9ce
2022-05-31 10:18:51 +00:00
Michał Niewrzał
4f2fae4f28 cmd/uplink: better error handling for parallel transfer
Few improvements were made to how we are handling errors
while doing parallel upload/download for single object:
* unhide error under 'context canceled' which was shown in most of
cases
* add part number to error message
* don't try to commit if any error occurs while operation
* combine errors into more readable form, example:

---
failed to download part 3: uplink: eestream: failed to download stripe 0:
error retrieving piece 00: ecclient: piecestore: rpc: tcp connector failed: rpc: dial tcp 97.119.158.36:28967: i/o timeout
...
error retrieving piece 89: ecclient: piecestore: rpc: tcp connector failed: rpc: dial tcp 161.129.152.194:28967: i/o timeout
failed to download part 1: uplink: eestream: failed to download stripe 0:
error retrieving piece 01: io: read/write on closed pipe
...
error retrieving piece 97: io: read/write on closed pipe
failed to download part 2: uplink: eestream: failed to download stripe 0:
error retrieving piece 00: io: read/write on closed pipe
...
error retrieving piece 01: ecclient: piecestore: rpc: tcp connector failed: rpc: dial tcp 180.183.132.234:28967: operation was canceled
error retrieving piece 96: io: read/write on closed pipe
	main.(*cmdCp).parallelCopy:418
	main.(*cmdCp).copyFile:262
	main.(*cmdCp).Execute:156
	main.(*external).Wrap:123
	github.com/zeebo/clingy.(*Environment).dispatchDesc:126
	github.com/zeebo/clingy.(*Environment).dispatch:53
	github.com/zeebo/clingy.Environment.Run:34
	main.main:26
	runtime.main:250
---

Change-Id: I9bb70b3f754567761fa8d17bef8ef59b0709e33b
2022-05-27 14:00:35 +00:00
Paul Willoughby
1d97b2c855 cmd/uplink: add use - for stdout,stdin to cp help
Change-Id: Ife3a0972d1be119a73eaefc0e23407b74fe03f54
2022-05-26 10:27:20 -06:00
Paul Willoughby
036d5a8619 cmd/storj-sim: fix printing node url
Fix printing satellite url to help configure authservice.
  storj-sim network env SATELLITE_0_URL

Change-Id: I786ab6723f837bbc35f1228eb73a82427d2f9250
2022-05-25 00:09:17 -06:00
Michał Niewrzał
d90ce467fc cmd/uplink: bring back --metadata for cp command
At some point uplink cli lost ability to set metadata. This change
brings back this functionality for 'cp' operation.

https://github.com/storj/storj/issues/3848

Change-Id: Ia5f60eb577fcab8a38d94730d8cdc6e0338d3b46
2022-05-18 15:58:53 +00:00
Michał Niewrzał
f9a3f19443 scripts: add 'uplink cp' tests for stdin/stdout
Uplink can upload from stdin and download to stdout. We had
such tests for old binary but now we were missing it.

Change-Id: I5110a9f531f5cc21277fa53611995fb5b556ff16
2022-05-18 08:42:04 +00:00
Jeff Wendling
836fa188fa cmd/uplink: allow listing invalid access grants
if you somehow get an invalid access grant in your
config file, it'd be nice to be able to list it
and delete it and stuff.

Change-Id: I7e335bf32353f294d5abb6a7c5f8f3aa18f2f6a7
2022-05-17 20:00:36 +00:00
Clement Sam
87ea2a4794 cmd/storagenode: make supervisor HTTP Server configurable
The current supervisord condifguration sets up the HTTP server
to listen on a tcp socket which is private i.e. available only
on localhost. This poses a regression where multiple containers
cannot be run if the host network interface is used when docker
container is run with `--network host` option.

This change adds a new env variable `SUPERVISOR_SERVER`, with
potential values `unix | private_port | public_port`, where
`unix` is set as the default value.

By default, the HTTP server is now set to listen on a UNIX
domain socket.
The file path is set to `/etc/supervisor/supervisor.sock`
instead of the /tmp directory since some systems
periodically delete older files in /tmp. If the socket file is
deleted, supervisorctl will be unable to connect to supervisord.

When SUPERVISOR_SERVER is set to `public_port` or `private_port`,
the HTTP server is set to listen on a TCP socket.

Resolves https://github.com/storj/storj/issues/4661

Change-Id: I224836dcae0293bcfe49874f2748be7723944687
2022-05-16 20:06:24 +00:00
JT Olio
5fb9ee3cfa uplink, satellite: use bgp hostnames
Change-Id: I58f5011d3019f8267fa8cbd3096b2cfe42eb5f8b
2022-05-16 18:14:37 +00:00
Stefan Benten
7afdb15fc8
cmd/uplink: adding Length Method to MultiReadHandle
This changes allows fetching the file size more easily (for supported
files) in order to afterwards calculate the multipart part size
accordingly.

Change-Id: Idabba4c2ee794ee471973889f5843174a7acad35
2022-05-13 21:31:45 +02:00
Stefan Benten
5e4ec0b3be
cmd/uplink: adjust multipart part size based on file size
This change allows the uplink to bump the part size based on the
content length that is being copied. This ensures we are staying
below the 10k part limit currently enforced on the satellites.

If the user specifies the flag, it will error out if the value
chosen by the user is too low. Otherwise it will use it.

Change-Id: I00d30f603d941c2f7703ba19d5923e668629a7b9
2022-05-13 21:31:23 +02:00
Egon Elbre
4791ba5d50 satellite/{mailservice,oidc}: improvements to debugging
Things that make debugging easier.
* Added logging to automatic link clicking to make it obvious, when it
  fails.
* Added monitoring to oidc.
* Made dbx create calls noreturn for oauth_*

Change-Id: I37397b4e84ce5bfd82954aed9c38fdfd52595f24
2022-05-11 19:59:42 +00:00
Jeff Wendling
89ccfe2dd7 cmd/uplink: fix recursive copy and improve tests
recursive copy had a bug with relative local paths.

this fixes that bug and changes the test framework
to use more of the code that actually runs in uplink
and only mocks out the direct interaction with the
operating system.

Change-Id: I9da2a80bfda8f86a8d05879b87171f299f759c7e
2022-05-11 15:17:16 -04:00
Erik van Velzen
928375a67c satellite/repair/queue: buffer batch insert
Implement a buffer for inserting repair items into the queue in a batch.

Part of https://github.com/storj/storj/issues/4727

Change-Id: I718472b2f2b1f4993c3d6f15c44923776407155a
2022-05-11 09:02:20 +00:00
Clement Sam
b471a5d8e3 cmd/storagenode/Dockerfile: bump storagenode-base image version
The new storagenode base image version contains the fix for the
failing "processes" supervisord event listener.

Resolves https://github.com/storj/storj/issues/4772

Change-Id: I6d67aa6f85ee33cd9abe6a663e4f9a84ea57fdbf
2022-05-04 16:00:26 +00:00
Clement Sam
9751ac9de2 cmd/storagenode: fix failing supervisord "processes" eventlistener
/bin/stop-supervisor fails in posix shell since the standard read utility
takes at least one variable's name as argument.

Changing the header #!bin/sh to #!/bin/bash fixes this issue.
`read` with no variable's name works in bash.

Looks like the shell in alpine isn't POSIX-compliant so we didn't
encounter this issue on alpine.

Also, I changed the name from "processes" to "processes-exit-eventlistener"
to make it clearer in the logs since supervisord spawns event listeners as
separate processes.

Change-Id: Ife9378c2013e2eb54f2adcd52a163d64eaacbbab
2022-05-04 02:44:23 +00:00
Clement Sam
87cb2e92b9 cmd/storagenode: bump storagenode-base image version
Change-Id: I8df391ee12c70cc599b54115b54eb987a2114b74
2022-05-04 01:59:00 +00:00
Clement Sam
a9faf756b0 cmd/storagenode: get rid of CRITICAL log when running supervisord as root
Closes https://github.com/storj/storj/issues/4729

Change-Id: Iea9a60ff60d753af375977f9bd90557574f80de0
2022-05-03 21:18:11 +00:00
Clement Sam
4a46d41c1c cmd/storagenode: change supervisord pidfile location
When running the docker auto-updater image as non-root user,
supervisord logs a "CRIT could not write pidfile /run/supervisord.pid"
since the user does not have permission to the /run directory.

Changing the location to /etc/supervisor fixes it because permissions
are set for non-root access of the /etc/supervisor directory.

Closes https://github.com/storj/storj/issues/4730

Change-Id: Id463f3a08db44dd9283921ece4575abdad9bd7f2
2022-05-03 12:54:40 +00:00
Stefan Benten
345ab87b5c cmd/uplink: adding output flag for ls command
With this change users can use the uplink cli in
scripts (ie. bash) more easily, since the output
can be switched to an easier processable json format.
It keeps the default of tabbed output.


Change-Id: I37e2c55f75c2250c3119fd8df8b66a766ff9096b
2022-04-29 10:32:04 +00:00
Paul Willoughby
8efed4b270 cmd/uplink: raise fd limits
Change-Id: I507c92c38d45d3d2be3edee08b6b85552ae21c43
2022-04-27 18:11:44 +00:00
Jeff Wendling
fead0db981 cmd/uplink: return nicer error message during import
Fixes #4750

Change-Id: Icef3340dd04d1c502b1651dcc70362895f258563
2022-04-27 09:30:57 +00:00
Jeff Wendling
f25ead5f98 cmd/uplink: set default parallelism to 1
Change-Id: Ic4198131c9958cc864fd861f983e32776bf56595
2022-04-26 22:55:11 +00:00
Cameron
dd6ab25cbd {cmd/satellite/reports, satellite/attribution}: type and variable name adjustments
Change-Id: I553b7ed9cb702479911baaf54d77d8f2662faaa2
2022-04-26 20:12:38 +00:00
Egon Elbre
1ed36e9fea cmd/uplink: make clearer ctx cancellation path in copy
When ctx is cancelled limiter won't start a new goroutine.
The code didn't immediately return an error in that case.

The dst.Commit(ctx) would fail anyways due to a cancelled ctx.
However, we can make the behavior clearer by returning immediately.

Change-Id: I65df7ca85de55813f3200a50db2eaaa7a297ba2c
2022-04-25 18:16:46 +03:00
Egon Elbre
c2bdd4effa cmd/uplink/ulfs: disallow writes after first failure
It was possible for the a previous write / part to fail or be aborted
and the next part write still happened. This causes a data ordering
corruption.

The whole write to parallel stdout fails, so there shouldn't be
confusion with regards to the output acceptability. However, it would
be clearer, if we avoided writing out-of-order data... mainly to be
clear that we didn't corrupt the data, just that it's incomplete.

Change-Id: I97b0d14404f29e8615e7d29b10cbd61ccb861e40
2022-04-25 18:16:46 +03:00
Egon Elbre
847ddaaab0 cmd/uplink: cancel on failed copy
Also ensure that abort is given at least 5 seconds to clear up any
pending uploads on cancellation.

Change-Id: I814aa407ee5783f2609a76b54de2879dcd5f89bb
2022-04-22 14:57:24 +03:00
Kaloyan Raev
978e0f1a26 cmd/uplink: cp sets connection pool capacity based on parallelism
If the cp command is executed with higher level of parallelism, it would
open more connections to storage nodes at the same time. Therefore, the
connection pool capacity should be expanded accordingly.

The pool capacity is set to 100 * parallelism.

Change-Id: Ia8b3ab6a99340d8cbb87a7b80c3354b2b21c1958
2022-04-21 14:10:08 +00:00
Clement Sam
b24e5cbc44 cmd/storagenode: silent authentication checks CRITICAL logs by supervisord
set dummy user/password in the [unix_http_server] and [supervisorctl] sections of the config file to remove the CRIT log entries as suggested here: https://github.com/Supervisor/supervisor/issues/717

Closes https://github.com/storj/storj/issues/4731

Change-Id: Iec10be39416f3aaa2d7febb414e15a63a4b23a4c
2022-04-21 13:10:42 +00:00
paul cannon
1422a1ff19 cmd/uplink: use 64 MiB for parallel chunk size, not 64 MB
I don't think it should matter for correctness whether this matches the
segment size or not, so I think there is something else wrong. However,
making this change seems to eliminate the "corruption when ulimit -n is
too low" problem we're seeing right now.

Change-Id: I232fe0d0a371b86ddf902e8c2d4778e140b2f1fc
2022-04-19 12:08:08 -05:00
Cameron
48fb3e947c cmd/satellite/reports: sum attribution data by user agent
Attribution is attached to bucket usage, but that's more granular than
necessary for the attribution report. This change iterates over the
bucket attributions, parses the user agent, converts the first entry
to lower case, and uses that as the key to a map which holds the
attribution totals for each unique user agent.

Change-Id: Ib2962ba0f57daa8a7298f11fcb1ac44a8bb97875
2022-04-18 13:23:37 +00:00
Yaroslav Vorobiov
a401450e3d cmd/storagenode-updater: add should-update cmd
This change add Separate CMD to check if a binary needs to be updated

Change-Id: I87f4214f84105464d283f1622dcb4ca9e796e771
2022-04-12 17:55:54 +00:00
Clement Sam
e9611801ad cmd/{storagenode,storagenode-updater}: add Process name to logs
Now that we have both the storagenode and updater processes running
in a single docker container, we need a way to know which log entry
is logged by any of the processes.

This change includes a Process field in the log entries.

Resolves https://github.com/storj/storj/issues/4648

Change-Id: I167b9ab65728a41136d264b5fe2c41bb64ed1785
2022-04-12 16:49:56 +00:00
Cameron
d4ad3a3ca6 satellite/satellitedb/attribution: update value attribution query to return byte-hours
Before, the VA query was summing the total and dividing by the number of
rows. This gives the average bytes stored per hour, but we charge for
usage with byte-hours. Why not do value attribution the same way?
To do that, we don't divide by the number of rows. We also have object
and segment fees so return segment-hours and object-hours too.

Change-Id: I1f18b7e1b2bae1d3fae1ca3b93bfc24db5b9b0e6
2022-04-08 16:22:21 -04:00
Clement Sam
e486585853 Dockerfiles: switch base image from alpine to Debian
We've had a lot of issues with alpine and currently there's a broken
network issue on alpine for users running on RPI arm32 architechture
which requires a workaround before docker is able to sync time between
the host and the container: https://wiki.alpinelinux.org/wiki/Release_Notes_for_Alpine_3.13.0\#time64_requirements.

Since we're switching the base image of the storagenode to debian,
it's best to switch the base image of all our docker images to
debian as well for consistency; less drift across them and keeps
the push target consistent.

Change-Id: If3adf7a57dc59f19ef2221b892f340d919798fc5
2022-04-05 11:40:44 +00:00
Erik van Velzen
61a47f3e95 cmd/uplink: refactor date parsing
Change-Id: I6a5cbdf86eecdc5578f3dae7a8ab1b0d4485e1da
2022-04-05 01:03:20 +00:00
Cameron
2b39df460c cmd/nullify-bad-user-agents: set user_agent to NULL where user_agent = partner_id
In the migration to migrate the corresponding name of the partner id to
user agent, part of the requirement was to migrate the partner id
itself if there was no partner name associated. This turned out to not
be so good. When we parse the user_agent column later, it is returning an
error if the user agent is one of these UUIDs.

Change-Id: I776ea458b82e1f99345005e5ba73d92264297bec
2022-03-31 16:57:50 -04:00
Qweder93
2a7b20e8e4 cmd/uplink: integrate server-side copy with uplink cp command
Resolves https://github.com/storj/storj/issues/4486

Change-Id: I42ac2ad2e1a05df4a83606f1990b639f08791403
2022-03-31 09:25:29 +00:00
Egon Elbre
e5972d8920 all: fix linting errors
strings.Title is deprecated in Go 1.18, replace it with
golang.org/x/text/cases.

Change-Id: I6185b97d37309dbe4a6715f794383ab259a0658b
2022-03-30 18:45:23 +03:00
Clement Sam
d611c16891 cmd/storagenode: update storganode-base image version in Dockerfile
Change-Id: I2e40f75d7cf1f673ea16a633a427225f97ea92ed
2022-03-29 13:26:33 +00:00
Clement Sam
c641f4c9ac cmd/storagenode: use Debian as base image instead of alpine
We are switching from alpine to debian due to a network issue
introduced in alpine 3.13+ which fails to verify certificates
due to not all armhf boards meet the time64 requirement:
https://wiki.alpinelinux.org/wiki/Release_Notes_for_Alpine_3.13.0\#time64_requirements

Also, Debian does not have official imagess for arm32v6 architecture
so we are building with arm32v5 arch in the Makefile.

Change-Id: I3660c3f64b7c2b342dd4ccb876af5f4e3036ea9d
2022-03-25 08:19:20 +00:00
Egon Elbre
0d2d59f884 all: fix linting issues
Change-Id: Idfc93948e59a181321d79b365e638d63e256a16f
2022-03-21 15:26:42 +00:00
paul cannon
d253b4c033 cmd/satellite: fix fetch-pieces segfaulting
When there is an error fetching a piece, the reader might be present or
it might not, depending on how far the fetch operation got. The
fetch-pieces code did not handle the "reader-not-present" case. Now it
should.

Change-Id: I263657d544d0ab8ba5d307a34ffc76bbf56835d0
2022-03-19 04:34:15 +00:00
Clement Sam
36770448b1 cmd/storagenode: update base image for storagenode
Updating the version of the base image for the storagenode docker image.

Also fixes the non-root permission issue to /app directory

Change-Id: I8b55a1e3062f55ce6fc52e126ec1a18bfa24e669
2022-03-18 13:37:12 +00:00
Clement Sam
51e5e949a4 cmd/storagenode: bug fixes to storagenode docker image
This change fixes the following issues:

wget: Alpine docker image by default uses the builtin BusyBox wget which is not capable of handling SSL traffic via proxy unlike the GNU wget. We have to replace BusyBox wget with GNU wget.

updater failing to restart the node: supervisorctl pointing to wrong config file. We remove the default configuration file and point supervisorctl to custom config in systemctl

updates https://github.com/storj/storj/issues/4489

Change-Id: I24a7f18377ba723bbc377bb5d25aaa14f37021b1
2022-03-18 09:03:24 +00:00
Cameron
9ebe388621 cmd/partnerid-to-useragent-migration: add ability to limit updates in migration
Add ability to limit updates in migrations.
To make sure things are looking okay in the migration, we can run it
with a limit of something like 10 or 30. We can look at the output of
the migrated columns to see if they are correct. This should have no
effect on subsequently running the full migration.

Change-Id: I2c74879c8909c7938f994e1bd972d19325bc01f0
2022-03-16 20:48:39 +00:00
Cameron
41a095a539 cmd/satellite: fix args indexing in cmdValueAttribution
Change-Id: Ife7432132563a8b6858d437e16025ce201a488b0
2022-03-16 16:42:04 +00:00
Clement Sam
52a80b2c67 cmd/storagenode: use custom storagenode-base image
Updates https://github.com/storj/storj/issues/4489

Change-Id: I3585b50df598c69e58137f61b6a272dbd077c5be
2022-03-15 14:22:35 +00:00
Clement Sam
155c070837 cmd/storagenode: fix permission to /etc/supervisor in base image
This change fixes the `sed: can't create temp file '/etc/supervisor/supervisord.confXXXXXX': Permission denied` issue when editing the supervisord.conf file during runtime as a non-root user.

While editing the config file, Sed creates a temporary file, saves the result and then finally mv the original file with the temporary one. So we need to set the permission for the /etc/supervisor where the temporary file is created.

Change-Id: Ic9c147a9cf0a6ef94adf702e33054edce1828806
2022-03-15 13:01:23 +00:00
Clement Sam
053a38a46d cmd/storagenode: move supervisord config file to storagenode-base image
The supervisord.conf file is edited to set the args for the storagenode and storagenode-updater binaries at runtime. This change moves the config file to the base image so we can set the permission to allow non-root users edit the config file.

Non-root user permission is also needed for the /app directory so we can install/update the binaries when run as a non-root user.

Updates https://github.com/storj/storj/issues/4489

Change-Id: If7a51a00ea171253e41923501174a43393f4638c
2022-03-08 13:47:01 +00:00
Erik van Velzen
85fa78eae7 cmd/uplink: supporty expires in copy
When copying an object from cli you can now set the expiry.
It uses the same datetime format as restricting access grants.

Closes https://github.com/storj/storj/issues/4595

Change-Id: Icab73a64a9589817d6bc6d702b765b166ca1350d
2022-03-07 02:43:51 +01:00
Clement Sam
15a1428828 {cmd/storagenode,Makefile}: add storagenode base image Dockerfile
Having the storagenode and storagenode-updater processes in one container
requires a process manager to properly handle the individual processes.

Using a process manager like supervisord requires that you package
supervisord and it configuration in the image, along with the storagenode
and storagenode-updater binaries.

Installing supervisord requires that we run apk to install it and its
dependencies at build time which makes it difficult to build multi-platoform
images; executing apk forces a requirement of the build system to run
foreign architechtures.

This change adds a dockerfile which will be used to build the base image
for the storagenode and has supervisord packaged. The base image will be
built manually using docker buildx, with QEMU binfmt support.

Updates https://github.com/storj/storj/issues/4489

Change-Id: I33f8f01398a7207bca08d8a4a43f4ed56b6a2473
2022-03-04 14:34:58 +00:00
Erik van Velzen
b4e42ceb23 cmd/uplinkng: fix linkshare bugs
Fix various bugs by remove superflous steps and letting libuplink
build the url.

$ uplinkng share --not-after +1h --url sj://mybucket/myprefix/

before: https://link.us1.storjshare.io/s/jxcvcme2xkb44xsec235xb2ccmbq//myprefix//
after: https://link.us1.storjshare.io/s/jxcvcme2xkb44xsec235xb2ccmbq/mybucket/myprefix/

$ uplinkng share --not-after +1h --url $(printf "sj://waterbear/aa\x03bb")

before: https://link.us1.storjshare.io/s/jwbcxjiv4ept7t2g3qmxfb73ljga//aabb
after: https://link.us1.storjshare.io/s/jvod7pixige62yrp5tubn4ct47yq/waterbear/aa%03bb

$ uplinkng share --not-after +1h --url

before: no url generated, no error
after: error "Need at least a bucket to create a working linkshare URL"
Change-Id: Ibca6847f8fc5d5fad9ee4642d11d3081d75d77ca
2022-03-03 16:00:10 +00:00
Erik van Velzen
a9bd983f04 sql: capitalize keywords
Capitalize some keywords which were overlooked

Change-Id: Ie2ad283669e2ca2650fcddfd8c7395a81bac09a8
2022-03-01 15:19:38 +00:00
Michał Niewrzał
fbe2680500 satellite/metainfo: add feature flag for server side copy code
We would like to disable in production those parts of code
which are now mixed with new server-side copy logic.

Change-Id: Iff50682bc9545207330f58dd19b5eee53d404d7f
2022-02-24 10:43:49 +00:00
Sembeth
d015805d15
cmd/multinode: Add further documentation (#4556)
The text has been expanded a bit to clarify that it is necessary to create identity files with an example before using the Docker image.
Changed the <identity-dir> placeholder to <multinode-identity-dir> so no one confuses them with the storagenode identity files.
Changed the <storage-dir> placeholder to <multinode-config-dir> so no one confuses them with the storagenode 'config' folder.

fixed #4547
2022-02-22 14:42:54 +01:00
Stefan Benten
afa09b3c2e adding a short readme
Change-Id: I6fda2dd358895ae256a13dba6d033aa054795443
2022-02-21 13:08:48 +02:00
Stefan Benten
f89e030c3f final touches
Change-Id: I8fcc986f71f592507dc3f35ce4352647da46d461
2022-02-21 13:08:48 +02:00
Stefan Benten
76be9e6efd cmd/multinode,Makefile: build docker image for multinode dashboard
Closes #4547

We do not build an docker image for the multinode dashboard,
which makes monitoring for docker-focused environments harder.
This adds the basic image and ties it into CI/CD.

Change-Id: I14c01a7f1f0019f6f5c1b8fd75dc424fc362b18d
2022-02-21 13:08:48 +02:00
Stefan Benten
321bf26c85
cmd,private,satellite: add application_name to metabase connections
Currently the metainfo/metabase DB connections are missing the proper
application_name in order to differentiate and filter queries on the DB
side for analytics.
Without it, it is very time-consuming to correlate processes and their load.

This change adds the "check" on DB connection init and passes the fallbacks
in all places to catch connection strings, that do not set it.

Change-Id: Iea5cea8658bc63778ff89038e5c1c352bf482cfd
2022-02-20 22:29:41 +01:00
Cameron Ayer
1fa79d64f5 satellite/attribution: update value attribution report query
cmd/satellite/reports: show userAgent field and all partners

Change-Id: I9a763f6daa97358e9c22e1f75889ea14e0a3b7c2
2022-02-18 14:25:19 +00:00
paul cannon
12b3fb5fb0 cmd/satellite: add fetch-pieces command
The "satellite fetch-pieces" command allows a satellite operator to
fetch as many pieces of a segment as possible, along with their
original order limits and hashes as provided by the storage nodes. The
fetched pieces and associated info will be stored on in a specified
folder as they are, rather than being RS-decoded or decrypted.

It is hoped that this will allow easier debugging of certain one-off
problems we've observed in the wild.

Change-Id: I42ae0e9ef0023538e42473a9be5a2460a3ac0f3a
2022-02-18 00:13:53 +00:00
Jeff Wendling
e2e5882c86 cmd/uplink: fix migration for some old configs
some old configs had a value like

	access: <data>

in the yaml. this would end up causing migration to
create a json file where it had no access values and
a default name of the data. that's not what the command
expects to operate on, so now we fix that during
migration and add a little mini migration for any
users that may have hit it.

Change-Id: I4c98ca5d09d043fe9338738ef6b4f930f933892c
2022-02-16 21:13:52 +00:00
Mya
05a17ef42d deps: upgrade storj.io/common
In addition to upgrading the storj.io/common library, this change
moves off the TCPConnector in favor of the HybridConnector per
the deprecation warning.

Change-Id: I7e7e1e7568e8b95e4a99ad9caa158a799e68e1e3
2022-02-16 18:59:19 +00:00
Erik van Velzen
0796653b07 cmd/uplinkng: registeraccess via libuplink
Change the implementation of register and share so that it uses the
uplink method to contact the Auth Service. The network protocol switches
from HTTP to DRPC.

Closes https://github.com/storj/storj/issues/4324

Change-Id: Ib8fdb1665c6385bb39a546ba46a8df43a136df9c
2022-02-11 11:36:45 +00:00
Jeff Wendling
9061dd309f cmd/uplinkng: become cmd/uplink
Change-Id: If426c32219d32044d715ab6dfa9718807f32cb9f
2022-02-09 17:02:21 +00:00
Jeff Wendling
4f4b67c6f6 cmd/uplinkng: add quic support
Change-Id: Ia7b4569ed39ef79ad085358e711f3f3fc0d8c398
2022-02-07 15:22:43 -05:00
Erik van Velzen
4a26f0c4f1 cmd/storagenode: restore passing arguments
Through `docker run storjlabs/storagenode:latest --help` we have always
made available around 100 command-line arguments.

However if you now pass such an argument it will be passed to
storagenode-update and it may no longer be recognized. This will cause
the storagenode not to start.

This was introduced in
https://review.dev.storj.io/c/storj/storj/+/5426

This change restores previous functionality.

Change-Id: I06823283ff82ffda12aee48c4d83717bddfbfdac
2022-02-04 02:48:39 +00:00
Ivan Fraixedes
7f1dc74e37
cmd/storagenode: Change order load id in setup
Change the order of when the storage node setup node loads the identity
for avoiding to write anything in the disk in the case that there is an
error loading the identity.

This bug was reported by @onionjake Github username's and the specific
changes to make.

Closes #4387 #4396

Change-Id: I360fff3c23b160c9e055203d3526d749edfd9129
2022-02-03 16:00:29 +01:00
Clement Sam
7e63afbef6 storagenode: docker image autoupdate binaries
Get storagenode and storagenode-updater binaries during
run of the container to not to release new docker image
on each new version of the storagenode binary.

Fixes https://github.com/storj/storj/issues/4176

Change-Id: I994c4942136a2cc7298eb0346238689eb406ae5b
2022-02-02 11:40:04 +00:00
Cameron
0d03473e00 cmd/partnerid-to-useragent-migration: LIMIT 1 in MigrateUsers id select query
The select statement is scanning the entire table.
Limit to one row.

Change-Id: Ie50d9777fb054b197484a83a468d67fa24096bb5
2022-01-31 23:57:30 +00:00
Cameron
1743a72f21 cmd/partnerid-to-useragent-migration: parallelize tests, use db test migration method
Use satellite.DB method TestingMigrateToLatest instead of
MigrateToLatest. TestingMigrateToLatest is much faster.

Also, run package tests in parallel.

Change-Id: I18bc0926dcfb80ace30d0b401e64ed919bfb966f
2022-01-31 18:37:13 +00:00
Jeff Wendling
b7ca2289e0 cmd/uplinkng: improvements to prepare for doc update
* improve setup wizard
* added access grant file support
* improved consistency across commands
* a couple bug fixes
* added access import and export

Change-Id: I30ad9d4771f15430904a503a4d465bc40be471b5
2022-01-31 17:37:54 +00:00
Cameron Ayer
9d3614fc1b cmd/partnerid-to-useragent-migration: add value_attributions migration and tests
Change-Id: I21b8a752720c1567840b58cc9380eff0b8abeece
2022-01-31 16:34:31 +00:00
Egon Elbre
64c8de6ea5 mod: use vendored base58
Change-Id: I5aa29515928848c862500330218cc094618638d7
2022-01-31 15:54:33 +02:00
Cameron Ayer
6b6e9901e2 cmd/partnerid-to-useragent-migration: add bucket_metainfos migration and tests
Change-Id: I8c207fb4f28f45a30b3ca6e02122dd701f044b03
2022-01-28 11:51:08 -05:00
Cameron Ayer
1f1a054de4 cmd/partnerid-to-useragent-migration: add api_keys migration and tests
Change-Id: Id8616fb30c09b8f502e506a65cae7367b6e0872e
2022-01-28 10:52:00 -05:00
Cameron Ayer
6d6d6776d8 cmd/partnerid-to-useragent-migration: add projects migration and tests
Change-Id: I23f931cb6ff8e047aa9b30a45b4e05c41d87c04e
2022-01-28 10:16:50 -05:00
Jeff Wendling
67c58fdd56 cmd/uplinkng: add setup command
it's an alias to `access create` for backwards
compatibility.

Change-Id: I17b64da0b8c0ff56cd82b76af94ee578167a2269
2022-01-25 16:16:55 -05:00
Cameron Ayer
e6961e1fd7 cmd: add command for partner_id to user_agent satelliteDB migration
Value attribution codes were converted into UUIDs and stored
in the users, projects, api_keys, bucket_metainfos, and
value_attributions tables in the partner_id column. This
migration will lookup the appropriate partner name associated
with each of these UUIDs, and store the partner name directly
in the user_agent column within each table. If no corresponding
partner name exists for a partner_id, the partner_id value will
be stored instead.

Add migration for users table with tests.

Change-Id: I61254d9b81c474e76bcfc1c8cd863697c6ef44b6
2022-01-24 15:30:14 +00:00
Malcolm Bouzi
c30c7def44 satellite/{console,payments},web/satellite: save signup promo code to newly registered users
Users signing up through a url containing a promo code will have that code applied to their stripe account instead of the free tier coupon.

Change-Id: I071041b0934648ef3f5bdb05b6ec97c400f89ae4
2022-01-21 15:02:22 +00:00
Michał Niewrzał
6bbe2fc692 cmd/uplinkng: fix flag/error messages
This change fixes some small issues with flag descriptions
and error messages.

Change-Id: I74a669d2566bf51ad871f16d64a92984ae75e712
2022-01-13 13:36:11 +00:00
Erik van Velzen
4f96a85642 cmd/uplink/share: register access via DRPC
Convert registering access with the edge services
from the HTTP protocol to DRPC protocol

Change-Id: Iba88dd0758c26f613cf501be9a20ead07d122d0b
2022-01-13 13:24:05 +00:00
Stefan Benten
44a73b01d9 cmd/storagenode: use contact address instead of server address
Currently the address being used is most of the time just :28967, which is not the correct address to reach the node from the public on.
This change uses the designated contact external address value that contains the set and preferred way to reach the node.

Change-Id: I99e979c2541043755b81e65c36c4289bfa3f60f3
2022-01-12 15:15:09 +00:00
Mya
bc3392cda4 cmd/uplinkng: fix clingy argument usage
Change-Id: I779a084edf14e0ee1c600f8e564dfc39aae073cb
2022-01-10 16:24:53 -06:00
paul cannon
171f908d8b cmd/satellite: rename monkit stat with '_', not '-'
Change-Id: Idb7d2971fbef1145f281b728a492e40e04add155
2022-01-06 02:30:30 +00:00
Jeff Wendling
4426470173 cmd/uplinkng: rename parallelism flags, small fixes
this makes the flags match rclone nomenclature

fixes test-uplinkng to use the temporary config dir
instead of the machine default, and clean up some.

bumps clingy so that the command errors when an unknown
command is specified.

also fixes some printfs in share to use clingy stdout.
it still does some external actions that should be
passed through a ulext.External for mocking, but
that's ok for now.

Change-Id: Icc231e7e26393541c312396fec907b640b97718e
2021-12-22 13:13:30 +00:00
Clement Sam
137641f090 multinode/nodes: pass Node entity to Add method instead of set of a parameters
The Add method on the multinode DB interface accepts
a set of parameters which are already fields in the nodes.Node struct
excluding the name field.

When adding a new node, you're forced to call UpdateName() method
after calling the Add() method in order to save a node and update
the name.

This change allows passing the nodes.Node entity which includes
the name field. With this, a new node can be added together with
the name without invoking the UpdateName() method.

Change-Id: I281ec628dffaade35d6db4479a84f39636200072
2021-12-15 12:39:24 +00:00
Clement Sam
ed9bde8f69 cmd/storagenode: add info command
The info command prints the details of the storagenode
to stdout.

It returns the storagenode info in JSON format
if --json flag is specified which can be piped
to the multinode add command.

Change-Id: I0163db8e02c4ec7346bfa69274d1772669357c6c
2021-12-14 17:05:52 +00:00
Clement Sam
60c8280565 cmd/multinode: add add command to multinode
This change adds an add command to the multinode CLI.

The add command takes a json <file> as argument.
If dash (-) is specified, it reads data from stdin.

The <file> specified can be json file containing array of
nodes data or a single node object.

Change-Id: I44d68486dc9aea0bd0311a40e84d3262a0303aef
2021-12-14 17:05:52 +00:00
Michał Niewrzał
7fba79be3f cmd/uplinkng: add register command
Change-Id: I9add258249c618e804e3b006dc1afc5325019009
2021-12-14 10:03:55 -06:00
Michał Niewrzał
75acd6109d cmd/{uplink,uplinkng}: set user agent for uplink binaries
We want to monitor traffic and tools that are used
to interact with our network so we need to append
its user agent.
The same user agent is appended for current uplink
and uplinkng as eventually we will remove first.

Change-Id: I116080d6c2c6c85d591771facf01356de02a9392
2021-12-14 13:00:01 +00:00
Jeff Wendling
9615bd191f cmd/uplinkng: update clingy for better boolean support
this allows commands like

    uplinkng cp -r sj://foo sj://bar

to work correctly, rather than complain that sj://foo is
not a boolean.

Change-Id: I003e47aabb85566bc2b454851cf55043b17ee7ea
2021-12-14 09:41:38 +00:00
Jeff Wendling
e6aa52cd6b cmd/uplinkng: reorganize iterator helpers
Change-Id: I8cf27c8fcbc35bf75000669cb5c02e30b5663a39
2021-12-14 09:41:18 +00:00
Jeff Wendling
c8c4e8835e cmd/uplinkng: file level parallel copy
Change-Id: I074abca67ea6f4a2fd0983fe452c74f76ee62c6a
2021-12-13 12:14:39 -05:00
Jeff Wendling
baaa96c208 cmd/uplinkng: introduce MultiWriteHandle
Change-Id: I6acf93141ddfa62728164818a322120ed6956b00
2021-12-10 10:23:04 -05:00
Jeff Wendling
34890c9195 cmd/uplinkng: introduce MultiReadHandle
Change-Id: I57b98b5e1406e7b38edf3bc65907d9796a1a663b
2021-12-10 09:30:25 +00:00
Ethan Adams
27c6c6aeae
ci: Golangci lint v1.43.0 changes (#4307)
Co-authored-by: Stefan Benten <mail@stefan-benten.de>
2021-12-09 20:49:48 +01:00
Qweder93
b5194762f2 cmd/uplinkng: share command added
In uplink we have command uplink share and we need
to port it to uplinkng to have command with same functionality.

Command could be executed with parameter or without.
without any flags we share as readonly.

Change-Id: I973b11d00da237358834acf5a863ebab37e684cc
2021-12-02 13:02:14 +00:00
crawter
eec5ad4f5d cmd/uplinkng: access inspect command added
In uplink we have command uplink access inspect and we need
a command with the same functionality for uplinkng.

Command could be executed with parameter or without.
without parameter - we should show default access.
If parameter exists - it could be access name or value.
If access name or value is wrong - we show error.
F.e.
uplinkng access inspect
uplinkng access inspect accessName
uplinkng access inspect accessValue
https://storjlabs.atlassian.net/browse/PG-318

Change-Id: I85fd961283850feb8684db2d126441f6b9bf0270
2021-11-30 12:45:04 +00:00
Michał Niewrzał
8f07abafe0 cmd/uplinkng/ultest: make possible to test access commands
Test external implementation doesn't support OpenAccess
method. This makes imposible to test output of commands like
inspect or share. This implements only basic functionality.

Change-Id: I127ef0bb45a01634bd5265ed80840f8095c72794
2021-11-22 10:43:02 +00:00
Michał Niewrzał
24cf7e8ea6 cmd/uplinkng: add mv command
Add ability to move files and objects.

Change-Id: I4929da730984c06aa578678b1d8c8e9b4aceade8
2021-11-22 09:07:24 +00:00
Mya
814e3126fa satellite/buckets: add new buckets service
The main motivation is to wrap the bucket DB and metainfo DB, so we
could check if a bucket is empty before applying geofencing config.

Change-Id: I8bac21555e01d51a663fb557bc1acfc8106bc2e1
2021-11-16 12:36:17 +02:00
Egon Elbre
4a530ccffd cmd/storagenode: simplify windows service loop
Remove some testing code that is not necessary for the service.

Change-Id: Idd09d62bf022d6e66943983f98642fc3c9aa72f7
2021-11-05 15:56:18 +00:00
Egon Elbre
e721081354 cmd/storagenode: fix service initialization code
func init() code isn't that well defined and reordering of them
could cause problems when starting the whole process from it.

Change-Id: I4088a0db156ece15354877011a481f6f91c9b332
2021-11-05 01:09:41 +02:00
Clement Sam
16a334020f cmd/uplinkng: add ranged download
This change adds the ability to download byte ranges
to uplinkng.

Extended the uplinkng Filesystem interface with Stat
method and an OpenOptions struct as parameter for the
Open method.

Also added a few tests for the ranged download

Change-Id: I89a7276a75c51a4b22d7a450f15b3eb18ba838d4
2021-11-03 19:51:25 +00:00
Egon Elbre
6a69c66477 cmd/storj-sim: support printing node url
Add support for node url-s with PEER_URL to help configuring
authservice and gateway.

   storj-sim network env SATELLITE_0_URL

Change-Id: I8b595b398007730662f100a4e6b3529cc0a7512a
2021-11-03 16:11:41 +02:00
Jeff Wendling
bd2448bc4d cmd/uplinkng: expand version information
This includes the information that used to exist in uplink
as well as all of the module versions.

Change-Id: Ie6d6fa12da9c04b50611e5885e05ab0c24a2ec0d
2021-11-03 09:26:56 +00:00
Egon Elbre
431f55d53b ci: cross-compile storagenode-updater
Change-Id: I1db757b64e61f185ba710bf669f2e62441b9ab71
2021-11-01 11:21:06 +00:00
Jeff Wendling
4d023f773f cmd/storagenode-updater: update to correct version
Currently if the nodes is below the minimum version it will immediately
update to the suggested version, regardless if its eligible from the seed
or not. This change corrects the behaviour to update to the minimum
version only and then properly respect/wait for the rollout to include it.

Updates based on logic here: https://review.dev.storj.io/c/storj/private/+/6187

Change-Id: Ic6c91c48ae9b8a116378b2573fbfca7e7bd5cc32
2021-10-29 20:38:30 +00:00
Jeff Wendling
f94d8a200c cmd/uplinkng: mkdir config file folder
If the directory doesn't exist, then the first run in a non-migration
setting will error because it cannot create the config file. This
change creates the directory.

Change-Id: I439159c00047e9ae20e139318dad5a047c853253
2021-10-29 11:07:55 +00:00
mya
9749f9756c
cmd/uplinkng: port expanded option to show additional metadata (#4229) 2021-10-28 11:51:33 -05:00
Yingrong Zhao
d441c8da15 satellite: use segment count for billing
Change-Id: Iafaf41cb6cde4cb2abdac3f75ce0fead838cb758
2021-10-27 19:21:50 +00:00
paul cannon
1951450c57 cmd/satellite: add register-lost-segments command
Usage: from a host for the affected satellite, issue this command
and supply the number of segments that have been permanently lost. For
example, to indicate 2 segments permanently lost:

    satellite register-lost-segments 2

If the unthinkable happens and this is necessary to use, it is presumed
that we will also remove the non-recoverable segments from the metainfo
db, so they will not continue to appear as 'temporarily unavailable' to
the repair checker.

The influxQL query for this will probably look something like:

    SELECT sum(total) AS "sum_total"
    FROM "v3_stats_new"."autogen"."lost-segments"
    WHERE time > :dashboardTime: AND "scope" = 'segment-durability'
    GROUP BY time(:interval:), "application", "instance"
    FILL(null)

Or use the Flux language in order to get the benefit of the
cumulativeSum function:

    from(bucket: "v3_stats_new/autogen")
      |> range(start: dashboardTime)
      |> filter(fn: (r) => r._measurement == "lost-segments"
                           and r.scope == "segment-durability"
                           and r._field == "total")
      |> group(columns: ["application", "instance"])
      |> cumulativeSum()

Change-Id: I73d364937705aa815af7520ab79c00bc2aea09f6
2021-10-26 15:03:52 +00:00
Clement Sam
85b49bb27c cmd/uplink: add ranged download to uplink cli
Change-Id: Ib274df024a8ffc5db2d5c99f8f363efa3b43723f
2021-10-25 13:13:33 +00:00
Qweder93
50baefa10e satellite/metabase: limit maximum number of parts and size
Multipart upload limits added. Last part has no size limit.
Max number of parts: 10000, min part size: 5 MiB

Change-Id: Ic2262ce25f989b34d92f662bde720d4c4d0dc93d
2021-10-22 10:16:37 +00:00
crawter
9ed826eabb cmd/uplink/mv: moving files between folders
https://storjlabs.atlassian.net/browse/PG-305

we should extend method move of cmd/uplink/cmd/mv.go
if both parameters end with slash -
i should list all files and call move method in loop

parameters:
uplink mv sj://bucket/a/prefix/ sj://new-bucket/a/new-prefix/

Change-Id: Ic24c2af83153ea60ec74393e65736af094877151
2021-10-20 18:16:08 +00:00
Jeff Wendling
987cb6ab11 cmd/uplinkng: allow removing pending objects
Change-Id: I12457e7d6fb28492ed4b6c5816f78aa7820fed6b
2021-10-20 12:25:25 -04:00
dlamarmorgan
4bbf667ad1 satellite/{satellitedb,attribution,console}: value attribution changes that add userAgent field to buckets table and all tables that have partner_id
Change-Id: I36a13bb651b86bfc14fe5a0a2258f719e6cd2b48
2021-10-18 13:56:19 +00:00
Egon Elbre
c54bcd63a1 Makefile: bump node to v16
Change-Id: I0587168f373230710c7b50b9aefd34718f4a5b3e
2021-10-14 12:46:28 +00:00
Jeremy Wharton
3b751a35c5 satellite/{payments,satellitedb}: Remove custom coupon implementation
Removes database tables and functionality related to our custom
coupon implementation because it has been superseded by the Stripe
coupon and promo code system. Requires implementations of the
payments Invoices interface to return coupon usages along with
invoices.

Change-Id: Iac52d2ff64afca8cc4dbb2d1f20e6ad4b39ddfde
2021-10-11 19:47:00 +00:00
Michał Niewrzał
0209bc6ef7 cmd/uplink: add mv command
New command for cli to move object to different
location.

uplink mv sj://bucket/your-object sj://bucket/moved-object

Change-Id: I85a4961aa59f250819954e78f20363ac3c570938
2021-09-29 19:10:58 +00:00
Michał Niewrzał
5b66136312 cmd/uplinkng: fix mb command
Make bucket command was using full location
specified in command line instead only bucket name.

As an addition change contains basic integration tests
with storj-sim.

Change-Id: Ie3b5283468b7fbde0b1333f01dc4fc2a2952e1a1
2021-09-29 15:22:03 +00:00
Yaroslav Vorobiov
469ae72c19 satellite/repair: update audit records during repair
Change-Id: I788b2096968f043601aba6502a2e4e784f1f02a0
2021-09-24 00:48:13 +00:00
Egon Elbre
b2d724962c cmd/storagenode-updater: avoid depending on the storagenode code
Change-Id: If81927aee43ccdd678a0e42f91632f85e9fdefab
2021-09-16 11:44:55 +00:00
Egon Elbre
1aec831d98 satellite/audit,storage: increase sleep delay in TestMaxVerifyCount
Currently TextMaxVerifyCount flakes in some tests, try increasing the
sleep time to ensure that things are slow enough to trigger the error
condition.

Also pass ctx to all the funcs so we can handle sleep better.

Change-Id: I605b6ea8b14a0a66d81a605ce3251f57a1669c00
2021-09-10 15:30:37 +00:00
Michał Niewrzał
03d638bbb3 cmd/uplink: add parallel upload
Use multipart upload to upload single object in parts in
parallel. Its using parallelism flag added earlier.

Change-Id: I45b531a5db43c86f0112a5e3bb4a83bc1d65650f
2021-09-07 15:12:09 +00:00
Jeremy Wharton
eb11899acb cmd/satellite: Remove billing 'check-paid-tier' command
Now that the command has been run on all production satellites
(US1, AP1, EU1), we should not need it again.

Change-Id: I25a4ffb03a7172445d90a04ec539be36c4eb2c8e
2021-08-26 22:02:16 +00:00
Jeff Wendling
58bd85cbf4 uplinkng: some windows test fixes
Change-Id: Idbe66028ac77d45eff9ea1926e6e8bd66491ea19
2021-08-26 17:52:13 +00:00
Michał Niewrzał
211a630982 cmd/uplink: add parallelism flag for single object download
Adds support for new uplink method DownloadObjectAt which
gives ability to download single object in parallel.

Change-Id: I8388653429992b0d24c383d17d7e90904203fe77
2021-08-26 13:13:38 +00:00
Jeff Wendling
2782e000ac cmd/uplinkng: parallel rm
Change-Id: I4f911ce1f384f13812986a34b96a94ded70cfb4e
2021-08-19 17:48:53 +00:00
Jeff Wendling
d57583dfd4 cmd/uplinkng: initial setup
Change-Id: If4df3ec8b3b554f5228d43e97503eb8a87525b23
2021-08-19 17:48:49 +00:00
Jeff Wendling
7eaa7490a1 cmd/uplinkng: secret prompting
Change-Id: I1b407b59559281c2242b55ebcf835d465db38e1d
2021-08-19 17:48:45 +00:00
Moby von Briesen
ef9a5210a4 satellite/payments: Add command to ensure free tier coupons
This command is intended to be run as part of invoice generation - it
iterates over Stripe customers, and applies the free tier coupon to any
customer who doesn't already have a coupon.

This way, we can ensure that all customers have at least the free tier
coupon before and after invoice generation, in case a different coupon
has expired.

Change-Id: I33a4aff9174049f9e051de53ef65298ca65ed688
2021-08-18 13:48:44 +00:00
Artur M. Wolff
2cd68bf4fb private/lrucache: import from common
Change-Id: Ia1f43d0440fef21122b071b05da59b4cf2689d6c
2021-08-16 10:04:32 +00:00
Jeff Wendling
59b8422318 cmd/uplinkng: implement revoke and normalize some language
this makes the distinction between an "access name" and
an "access value" and talks about which is expected for
commands. most are "access name or value".

Change-Id: I43c0043a17d37e89ab5f87388ae9e890a8b59958
2021-08-11 02:30:13 +00:00
Jeff Wendling
08d860570b cmd/uplinkng: parallelsm and a ton of fixes
this was just supposed to add parallel uploads/downloads
and it does do that, but i then found a bunch of bugs
with respect to path handling that i thought i had under
control. oops.

so this adds a ton of tests and tries to make the logic
in ulloc to be more consistent. almost all of the actual
file handling bits and knowledge happens in cmd_cp now
where it should belong.

additionally, the s3 command has the behavior that if your
bucket has the file s3://bucket/file, then executing
s3 ls s3://bucket/fi returns nothing. this change makes
uplinkng match that behavior even if i don't personally
like it.

a big portion of the weirdness is the concept introduced
that i've named "directoryish", which intends to capture
the behavior that if a user copies a file to that location
then the base name of the source should be appended on
rather than a direct copy. this concept is entirely a
based on the string value and not the actual filesystem
state. hence, the cp command is responsible for checking
if local paths are actually a directory, and adding a
trailing slash if necessary to make them "directoryish".
additionally, the empty key for a bucket and the empty
string for local paths are considered "directoryish".

Change-Id: I9120d18616fd813b29ff81beed4f5993caa99fb6
2021-08-11 02:30:06 +00:00
Jeff Wendling
e33f8d7170 cmd/uplinkng: access creation/restriction and review fixes
Change-Id: I649ae3615363685c28c39d1efb6a65fcad507f46
2021-08-11 02:29:49 +00:00
Yingrong Zhao
077ec96d94 private/server: use quic implementation from storj.io/common
Change-Id: I820cf6444a3ddccee0d7c647dc84c80b2752068c
2021-08-10 13:32:21 +00:00
Clement Sam
1f353f3231 segment/{metabase,repair}: change segment created_at column to not accept nulls
This change adds a NOT NULL constraint to the created_at column in the segment table.
All occurrences of CreatedAt as a pointer are changed to non pointer version (metabase, segment loop, etc)

Change-Id: I3efd476ebd1edd3327b69c9223d9edc800e1cc52
2021-08-06 08:16:28 +00:00
Michał Niewrzał
55d7bcc59b satellite/metabase/segmentloop: don't shutdown satellite on loop error
We made decision to avoid satellite shutdown when segment loop
will return error. Loop still can reeturn error but it will be logged
and we will make monitoring/alert around that error.

Change-Id: I6aa8e284406edf644a09d6b1fe00c3155c5430c9
2021-07-30 06:49:10 +00:00
Yingrong Zhao
f8914ccce0 satellite/{repair, overlay}: use reputation store in repair
Change-Id: I48db9e68f48239d48621ccc77d33618ecb83ce1a
2021-07-28 13:22:05 -04:00
Moby von Briesen
8f072bdeee cmd/satellite: Skip non-existing users in paid tier conversion
There are some users in our QA satellite which are no longer in Stripe,
and there are some users in Stripe which are not on our QA satellite.
This change allows us to test the paid tier conversion script in QA
despite these problems.

Change-Id: If94c9e882327841d1fd294d75fd302e6a7feee41
2021-07-27 12:53:58 -04:00
Michał Niewrzał
420d2f6275 metabase-orphaned-segments: avoid processing recently created elements
To be sure we are comparing the same set of objects
and segments lets ignore objects and segments created
after processing was started.
Segmets without objects cannot be created in normal
way so we will have them only if we broke something in
the past.

Change-Id: I96c07caf9e5091775d4dc8dfc0fef2b08b87957c
2021-07-26 15:50:20 +00:00
Michał Niewrzał
b12d29935a satellite/metabase: remove metaloop package
We moved everything to segment loop so we can now
remove metaloop from code.

Change-Id: I9bd8d2349e5638d7cdad50f2f313f9bd89a8165c
2021-07-22 13:00:45 +00:00
Michał Niewrzał
237360e765 metabase-orphaned-segments: fix args binding for delete command
As an additon I clarify log message a little bit.

Change-Id: Id2487cc218e3dd62ad5a05b56e8b1d5d070676a7
2021-07-22 11:00:37 +02:00
Michał Niewrzał
f73fcee892 cmd: add tool for orphaned segments
This tool has two commands to execute. One is to 'report'
orphaned segments. Second to 'delete' orphaned
segments.

To find such segments tool is first finding all unique
segment stream ids. As a next step its removing from this
list stream ids of existing objects. What is left is a list of
orphaned segments.

Change-Id: I4a0ae3ad0b10a8d16572bfd22ac92cfa15ca19b3
2021-07-21 10:10:58 +00:00
Michał Niewrzał
27a714e8b0 satellite/accounting/tally: use objects iterator instead metaloop
Bucket tally calculation will be removed from metaloop and will
use metabase objects iterator directly.

At the moment only bucket tally needs objects so it make no sense
to implement separate objects loop.

Change-Id: Iee60059fc8b9a1bf64d01cafe9659b69b0e27eb1
2021-07-20 15:52:18 +00:00
Bill Thorp
18c3252025 cmd/uplink: recommend notAfter expiration caveat be set
The auth service has no way to remove access grant registrations that
lack expiration dates.  We want to encourage people to set them, so as
to slow the rate at which the auth service DB fills up.

Change-Id: I1ccf629cd995dc184d2d90333166eab34d34ae07
2021-07-14 17:05:28 +00:00
Moby von Briesen
5870502589 cmd/satellite: Add billing command for converting customers to paid tier
We have implemented the paid tier, but it currently only handles new
users entering paid tier. It does not convert users who have already
added a credit card previously. We still want to convert these users'
project limits. This billing command can be run once to convert all old
customers with a credti card. Afterwards, we should be able to safely
remove it.

Change-Id: Ia496580b8e72ef436375b74f590fe57cca704fa8
2021-07-12 13:37:37 +00:00
Jeremy Wharton
a5f6bb9cc0 satellite/console: Add reCAPTCHA verification step to registration
The user must complete a reCAPTCHA in order to register.
ReCAPTCHA verification failure results in rejection of the
registration attempt.

Change-Id: I34ba7db414d756fd1aaebdc3d19cccbfc7fc1ea3
2021-07-07 21:34:07 +00:00
Jeff Wendling
ef7b89cc03 cmd/uplinkng: remove global flags
this changes globalFlags to be a ulext.External
interface value that is passed to each command.

rather than have the ulext.External have a Setup
call in the way that the projectProvider used to
we make all of the state arguments to the functions
and have the commands call setup themselves.

the reason it is in its own package is so that
cmd/uplinkng can import cmd/uplinkng/ultest
but cmd/uplinkng/ultest needs to refer to whatever
the interface type is to call the function that
creates the commands.

there's also quite a bit of shuffling around of
code and names. sorry if that makes it tricky
to review. there should be no logic changes, though.

a side benefit is there's no longer a need to do
a type assertion in ultest to make it set the
fake filesystem to use. that can be passed in
directly now. additionally, this makes the
access commands much easier to test.

Change-Id: I29cf6a2144248a58b7a605a7ae0a5ada5cfd57b6
2021-07-06 17:26:51 -04:00
Jeff Wendling
f474bb6179 cmd/uplinkng: update for breaking clingy changes
clingy changed some and this is just fixes for that

Change-Id: I729aed6329fe0988fcb9b4407f16966a753b3204
2021-07-06 17:26:51 -04:00
Michał Niewrzał
7cbff95090 cmd: delete metabase-expireat-migration
This migration was applied to all satellites and can be removed.

Change-Id: I6c63eeb60da17f6bb8b2d4dd3cd5e6e3d9612638
2021-07-01 18:11:48 +02:00
Michał Niewrzał
d53aacc058 satellite/repair: migrate to new repair_queue table
We want to use StreamID/Position to identify injured
segment. As it is hard to alter existing injuredsegments
table we are adding a new table that will replace existing
one. Old table will be dropped later.

Change-Id: I0d3b06522645013178b6678c19378ebafe485c49
2021-06-30 17:12:24 +02:00
Moby von Briesen
4e95d27033 web,satellite: Remove paywall-related functionality
Because of our free/paid tier plan, we do not need a paywall anymore. We
have not used it in a while, but still have leftover code laying around.

Change-Id: Iaea8c39faf042a2f7a6b837727bb135c8bdf2907
2021-06-29 02:47:48 +02:00
Michał Niewrzał
b582c974c3 satellite: remove irreparabledb leftovers from code
Change-Id: Iabceea2733d6e0d3ddb26c235ef26ae132a44fc2
2021-06-22 11:26:17 +00:00
Michał Niewrzał
7815e647de cmd: add metabase-expiresat-migration tool
Expires_at column was added to segments table and
we need to migrate this value for existing segments
from corresponding objects. This standalone tool
will read all objects and if expires_at is set then
will send update query for this object segments.

Updates will be send in batches and in parallel.

Change-Id: I1fddf0af8cde0f560582d25c6d0e07a00b58e534
2021-06-22 07:42:49 +00:00
Michał Niewrzał
a93e47514a satellite: remove irreparabledb
This is part of metaloop refactoring. We plan to remove
irreparable at some point but there was not time for it.
Now instead refatoring it for segmentloop its just easier
to drop it.

Later we still need to drop table with migration step.

Change-Id: I270e77f119273d39a1ecdcf5e1c37a5662a29ab4
2021-06-17 07:20:15 +00:00
Jeff Wendling
ce87652a8c cmd/uplinkng: rm
this implements the rm command which has to add
a Remove method to the fileystem interface and
implement it for local, remote and test filesystems.

Change-Id: Id41add28f01938893530aae0b4b73c8954e9b715
2021-06-14 15:23:41 -04:00
Jeff Wendling
173d1e638c cmd/uplinkng: meta get
Change-Id: I6cd7e3fe408b42d257c35a6c715016f13559643f
2021-06-14 15:23:41 -04:00
Jeff Wendling
1323e1a666 cmd/uplinkng: rb
just implements the remove bucket command

Change-Id: I096a3c0c4a5fd1c7b018784d361221d1a1f5461a
2021-06-14 15:23:41 -04:00
Jeff Wendling
7fae5654ff cmd/uplinkng: access save: prompt for access
this adds a helper method to prompt for a line of
input using the clingy context to the global flag
state that errors if interactive mode is disabled.

Change-Id: Ie113c8920dfa4719e85cc24f11401d91b32812f9
2021-06-14 15:23:41 -04:00
Jeff Wendling
d73287f043 cmd/uplinkng: tests for cp
this adds some stuff to ultest so that the set of
files created by a test can be inspected after so
that we can write some tests for the cp command
to observe that it does what it is supposed to do.

Change-Id: I98b8fb214058140dfbb117baa7acea6a2cc340e1
2021-06-14 15:23:41 -04:00
Jeff Wendling
98be54b9a3 cmd/uplinkng: refactor into some focused packages
the directory was starting to get pretty large and
it was making it hard to pick concise names for
types and variables. this moves the location
stuff into a cmd/uplinkng/ulloc package, the
filesystem stuff into a cmd/uplinkng/ulfs package,
and the testing stuff into a cmd/uplinkng/ultest
package.

this should make the remaining stuff in cmd/uplinkng
only the business logic of how to implement the
commands, rather than also including a bunch of
helper utilities and scaffolding.

Change-Id: Id0901625ebfff9b1cf2dae52366aceb3b6c8f5b6
2021-06-14 15:23:41 -04:00
Jeff Wendling
b24ea2ead5 cmd/uplinkng: test framework and ls tests
this adds a test framework with fake implementations of a
filesystem so that unit tests can be written asserting
the output of different command invocations as well as
the effects they have on a hypothetical filesystem and
storj network.

it also implements the mb command lol

Change-Id: I134c7ea6bf34f46192956c274a96cb5df7632ac0
2021-06-14 15:23:41 -04:00
Jeff Wendling
46a3242ed4 cmd/uplinkng: cp
Change-Id: I9c251028d9f72572f4d42815de31d44517cd5525
2021-06-02 14:35:27 +00:00
paul cannon
8675cd178c cmd/uplink: add flags to 'put' subcommand
This enables use of the --progress, --expires, and --metadata flags with
'uplink put'. These flags work similarly to their counterparts in the
'uplink cp' command.

Small difference from cp: "progress" is on by default in 'uplink cp',
but (for backwards compatibility) off by default in 'uplink put'.

Why: Requested by @Toyoo on the forum:
https://forum.storj.io/t/explicit-data-expiration-questions/13854/2

..and it's an easy addition to make.

Change-Id: Id20aedd3a280db43e4883338f92f6beec7a400de
2021-06-01 19:16:22 +00:00
paul cannon
fbd11ba4e9 cmd/uplink: add expanded format for 'uplink ls'
This expanded format shows expiration times for objects and how much
custom metadata each object has.

This commit also organizes output formatting code together in one
section for simpler adjustments in the future.

Change-Id: Ica041c8a1de6ee73c104a0554c5c259e447536c4
2021-06-01 18:50:27 +00:00
Jeff Wendling
e460dc51f7 cmd/uplinkng: implement object listing
Change-Id: Ib5f6964a0c42718913a680529bb66c6f475aeac9
2021-05-24 23:02:19 +00:00
Michał Niewrzał
c9d4674859 cmd: remove metabase-createdat-migration tool
We finished migration for all satellites and we can now remove this
code.

Change-Id: If92091ba94b51c319528dc95845042c9dfd6d074
2021-05-24 09:37:40 +02:00
Egon Elbre
2d3c40fd71 cmd/internal/asset: delete unused code
Change-Id: I44970a4f5ac42dc3a297263cd64150e94c40f62a
2021-05-20 18:30:19 +03:00
Jeff Wendling
a1bf9ab6de cmd/uplinkng: initial commit with skeleton
Change-Id: I764618cc60c46882955e9b08b72b3c162aa4929f
2021-05-18 10:04:44 -04:00
Egon Elbre
10372afbe4 ci: fix lint errors
Change-Id: Ib5893440807811f77175ccd347aa3f8ca9cccbdf
2021-05-17 13:37:31 +00:00
Yaroslav Vorobiov
c08ca361d8 cmd/storj-sim: add multinode process
Change-Id: I7e6c8db045daa87c5d89f49e58fc82453d4024c6
2021-05-14 01:40:13 +03:00
Yaroslav Vorobiov
ea7fbdf843 multinode/multinodedb: add db migration
Change-Id: Ied1a7f3b951a25ab7a8edc25902c0479d2d08e87
2021-05-13 15:44:08 +00:00
Egon Elbre
910eec8eee satellite/metainfo: remove MetabaseDB interface
Currently the interface is not useful. When we need to vary the
implementation for testing purposes we can introduce a local interface
for the service/chore that needs it, rather than using the large api.

Unfortunately, this requires adding a cleanup callback for tests, there
might be a better solution to this problem.

Change-Id: I079fe4dbe297b0ae08c10081a1cea4dfbc277682
2021-05-13 13:22:14 +00:00
igor gaidaienko
d32ae0459b Revert "storagenode: docker image autoupdate binaries"
This reverts commit 244c488b10.
2021-05-13 14:21:06 +03:00
Egon Elbre
2ae80690cb satellite/metabase/metabasetest: remove STORJ_TEST_DATABASES
Initially metabase was developed separately and it was useful to have a
separate environment flag for tests, however, it's more convenient to
use the same as rest of the testsuite.

Change-Id: Ia4d79be27ce5911cbae68d57cdf0b30f63459444
2021-05-11 13:31:01 +00:00
Ivan Fraixedes
7fb86617fc satellite/satellitedb: Use CRDB AS OF SYSTEM & batch for GE
Use the 'AS OF SYSTEM TIME' Cockroach DB clause for the Graceful Exit
(a.k.a GE) queries that count the delete the GE queue items of nodes
which have already exited the network.

Split the subquery used for deleting all the transfer queue items of
nodes which has exited when CRDB is used and batch the queries because
CRDB struggles when executing in a single query unlike Postgres.

The new test which has been added to this commit to verify the CRDB
batch logic for deleting all the transfer queue items of the exited
nodes has raised that the Enqueue method has to run in baches when CRDB
is used otherwise CRDB has return the error "driver: bad connection"
when a big a amount of items are passed to be enqueued. This error
didn't happen with the current test implementation it was with an
initial one that it was creating a big amount of exited nodes and
transfer queue items for those nodes.

Change-Id: I6a099cdbc515a240596bc93141fea3182c2e50a9
2021-05-07 13:09:19 -04:00
JT Olio
a89a2b4b43 satellite/billing: make stripe invoice generation work with multiregion satellites
multiregion satellites have complex database connection strings
largely due to using a different backend for the repair queue than
cockroach.

billing stuff didn't work right with this.

Change-Id: Ie8759a8c47e71347c3a190abfc9d53945d7b8855
2021-05-06 11:51:03 -06:00
Egon Elbre
961e841bd7 all: fix error naming
errs.Class should not contain "error" in the name, since that causes a
lot of stutter in the error logs. As an example a log line could end up
looking like:

    ERROR node stats service error: satellitedbs error: node stats database error: no rows

Whereas something like:

    ERROR nodestats service: satellitedbs: nodestatsdb: no rows

Would contain all the necessary information without the stutter.

Change-Id: I7b7cb7e592ebab4bcfadc1eef11122584d2b20e0
2021-04-29 15:38:21 +03:00
Qweder93
a5c1e4b4a5 crashcollect: process for receiving panics
Added crashcollect server for tracking panics
on parent process and saving them in file.

Change-Id: I7926f9a16594227a3262e05d216199b7c2857385
2021-04-27 18:57:47 +03:00
Michał Niewrzał
7944df20d6 storj: use multipart API
Change-Id: I10b401434e3e77468d12ecd225b41689568fd197
2021-04-26 13:15:09 +00:00
Yingrong Zhao
08e1ce8930 cmd/connecttest: display more helpful error message and fix tls config
Change-Id: Ibedd643458ee8c84fd4668a23449d99eec43ba73
2021-04-23 14:27:34 +00:00
Egon Elbre
7802ab714f pkg/,private/: merge with private package
Initially there were pkg and private packages, however for all practical
purposes there's no significant difference between them. It's clearer to
have a single private package - and when we do get a specific
abstraction that needs to be reused, we can move it to storj.io/common
or storj.io/private.

Change-Id: Ibc2036e67f312f5d63cb4a97f5a92e38ae413aa5
2021-04-23 16:37:28 +03:00
Egon Elbre
c641ddcb54 pkg/lrucache: rename package
cache is really common variable and type name and we have already used
the package name alias in multiple places.

Change-Id: I6435785b7549b541d533de59ec94557b9bd11e04
2021-04-23 16:37:28 +03:00
Egon Elbre
a2e20c93ae private/dbutil: use dbutil and tagsql from storj.io/private
Initially we duplicated the code to avoid large scale changes to
the packages. Now we are past metainfo refactor we can remove the
duplication.

Change-Id: I9d0b2756cc6e2a2f4d576afa408a15273a7e1cef
2021-04-23 14:36:52 +03:00
Egon Elbre
c1fbecb96b satellite/metabase/metaloop: add Monitor
We need some chores to join without triggering the loop.

For example it's fine to run metrics, only when something else is
running.

Change-Id: I9d8bd16f59c28c540c8d72971bc4e233a8660c02
2021-04-23 12:20:22 +03:00
Egon Elbre
fff21b330d cmd/metabase-verify: tool for verifying metabase state
Currently the tool verifies:
* validity of plain_offset
* whether plain_size is smaller than encrypted_size

Change-Id: I9ec4fb5ead3356a196392c26ca377fcdb367138e
2021-04-23 07:50:36 +00:00