Commit Graph

1177 Commits

Author SHA1 Message Date
Márton Elek
1646a0ac9e cmd/uplink: experimental environment variable to change piece hash algorithm
Change-Id: I7420919146c64d29ff1023f7498867d3d30892d0
2022-09-08 10:41:04 +00:00
Egon Elbre
731916ffa4 cmd/inspector: move command back from cmd/tools/
Some of the makefiles require it to be in cmd/inspector.
This was initially moved in 381c1e1257.

Change-Id: I850f3bfa1d66e66fbd1d9069e353984a27a7c58c
2022-09-07 09:35:12 +00:00
Egon Elbre
381c1e1257 cmd/tools: move tooling to a separate folder
This helps to cleanup the cmd folder a bit.

Change-Id: I24025c3dbfd35966325d7d5aaa95cd9a1176a8b7
2022-09-02 18:25:36 +03:00
Cameron
7933e0c4c7 cmd: add command to migrate project public_ids
This is to update all projects to have a public_id if they do not have
one.

github issue: https://github.com/storj/storj/issues/4861

Change-Id: Icfa42b62e15ca75d3c04a0aab48a3c3b0f3a9d6e
2022-08-31 13:14:17 +00:00
Michal Niewrzal
2ae1db53b6 cmd/satellite: add command for collecting GC bloom filters
We would like to have separate process/command to collect bloom
filters from source different than production DBs. Such process will
use segment loop to build bloom filters for all storage nodes and
will send it to Storj bucket. This change is extending satellite binary
with appropriete command.

New GC service for collecting bloom filter will be a subsequent
change.

Updates https://github.com/storj/team-metainfo/issues/120

Change-Id: Ibc03e119c340919cf468fc1f5a4f3d187bb3a5a1
2022-08-31 11:40:36 +00:00
Márton Elek
ea1408f7a8 go.mod: bump clingy dependency
As a reminder: latest clingy removed the requirement of having custom context (which made the usage of context.WithValue harder) and uses simple context instead.

Clingy saves the stdin/stdout/stderr to the context (earlier to separated context type) to make it available for unit testing.

Change-Id: I8896574f4670721de43a577cd4b35952e3b5d00e
2022-08-31 10:24:27 +00:00
JT Olio
e621190b99 satellitedb: randomize nodes for restore from trash
Change-Id: I030bd9093ec13a882edc7e0d325b9b19c201e135
2022-08-26 17:19:41 +00:00
JT Olio
12ef68bdf4 satellitedb: restore-trash shouldn't bother with nodes we've never talked to
Change-Id: I3c631e92a9a2670c52c9fa23b2b1baf67d3c9a9c
2022-08-25 16:14:29 +00:00
Clement Sam
ea0124a183 cmd/multinode: handle JSON input with BOM to add command
Decode the JSON input string to its corresponding unicode
decoding if the special byte order mark is present.

Fixes https://github.com/storj/storj/issues/4950

Change-Id: If91bac22590c89b35c58bf54f6d3bdb8a67d7a4f
2022-08-25 03:05:49 +00:00
Márton Elek
3d79c5f3a1 storagenode/docker: allow to configure updater with separated env variables
We couldn't use environment variables safely to configure storagenode, since we introduced the embedded updater.

For example STORJ_DEBUG_ADDR=localhost:11111 would try to set debug port 11111 for both the storagenode and storagenode-updated, causing port conflict.

This small change enables to configure storagenode-updater with STORJUPDATER_... environment variables.

Tested with creating custom image and installing to my own storage node.

Change-Id: I6b0a601a4dc63d2d1ff3c191ae89981434e55c30
2022-08-24 06:50:30 +00:00
Jeremy Wharton
3f26cc599f satellite/console,web/satellite: invalidate sessions after inactivity
Sessions now expire after a much shorter amount of time, requiring
clients to issue API requests for session extension. This is handled
behind the scenes as the user interacts with the page, but once session
expiration is imminent, a modal appears which informs the user of his
inactivity and presents him with the choice of loging out or preserving
his session.

Change-Id: I68008d45859c814a835d65d882ad5ad2199d618e
2022-08-23 15:51:05 +00:00
Clement Sam
7e5025cac0 {storagenode,multinode/nodes}: use multinodeauth.Secret instead of []byte for APISecret
When enconding structs into JSON, byte slices are marshalled as base64
encoded string using the base64.StdEncoding.Encode():
ea9c3fd42d/src/encoding/json/encode.go (L833-L861)

We, however, expect API Secrets to be encoded as base64URL, so when
an marshalled secret (with byte slice type) is added to the multinode
dashboard, it fails with `illegal base64 data at input byte XX`.

This change changes the type of APISecret field in the
multinode/nodes.Nodes struct to use multinodeauth.Secret type instead
of []byte.
multinodeauth.Secret is extended with custom MarshalJSON and
UnmarshalJSON methods which implement the json.Marshaler and
json.Unmarshaler interfaces, respectively.

Resolves https://github.com/storj/storj/issues/4949

Change-Id: Ib14b5f49ceaac109620c25d7ff83be865c698343
2022-08-23 11:04:04 +00:00
Qweder93
04c0e5d135 cmd/uplink: fix access setup flags setup
Added Setup of access maker call into cmd_access_setup to use flags during cmd call

Closes https://github.com/storj/storj/issues/4766

Change-Id: I0c75f224414099573b021b18b87d9e17192cecc5
2022-08-09 16:05:44 +00:00
dlamarmorgan
92be1d878f satellite/payments/stripecoinpayments: storjscan invoice generation
Add line item with unclaimed Storjscan wallet balance during invoice generation.

Change-Id: I018bfa01abfcf7bfdffba0c5a1350a69188f63d5
2022-08-03 13:24:26 -07:00
Egon Elbre
1cb3cbaecf cmd/storj-sim: ignore context canceled and flush output
It's possible that content was not being flushed from processes.

For now, ignore other process failures under storj-sim network test.
Once we get other processes stable, we can repropagate the error.

Change-Id: I01ed572d7c779ab6451124f1e24e3d1168b3ea79
2022-08-02 20:48:46 +03:00
Jeff Wendling
75f19cb100 cmd/uplink: run the collector goroutine
without this goroutine, spans will not be collected or sent
from the uplink.

Change-Id: Ibbf2ab79dee7447a1d8ab62005e102e670114bf2
2022-07-27 14:38:30 +00:00
JT Olio
452c243d57 cmd/uplink: support sending traces
Change-Id: I10ef49a24a5b6821b17eb5d206aacfe7e0f54e9e
2022-07-26 11:39:30 -04:00
Jeff Wendling
567986e388 cmd/uplink: minimal tracing support
this allows one to specify a trace id and cause any remote
spans to be sent up to wherever. it doesn't collect any
local traces.

Change-Id: Ia87e294bb276d966f9f3dbfbaf6e7916b1ec7af9
2022-07-26 11:39:30 -04:00
Jennifer Johnson
61649d2850 cmd/node-cleanup: command to delete uncontacted nodes
Some nodes were added to the nodes table due to a bug in quic
based storagenode contact code. This is a tool to clean up these nodes.

Delete with batch-size 1k seems to take ~400ms on local CockroachDB.

Change-Id: Ic0c1180528c27952e19c431fc9cc327292a10a5f
2022-07-12 12:15:49 +00:00
Ivan Fraixedes
66f12992b0
cmd/storagenode: Don't abbreviate multinode in help
Don't abbreviate multinode in the command help message because there
isn't a need for it and the abbreviation isn't clear at all.

Change-Id: I7a1f2be6ae1f7d4b287c18c48b22c630549b731f
2022-07-05 19:29:12 +02:00
Egon Elbre
05e165283f storagenode/console/consoleapi: use fixed time.Now()
It seems the tests relied on time.Now(), which might cause some
discrepancies in calculations. Use a fixed time.Now() rather than
recalculating.

As a sidefix, remove "Test" prefix from t.Run. These are unnecessary.

Change-Id: I1de903fcf0fcf46fc8e3acf2463e17239b8e3cc6
2022-07-01 12:36:01 +03:00
Jeff Wendling
e00423e2e9 cmd/uplink: disable deleting local files
Change-Id: Id3687f3346f8ec9b5da5459a57d91d1d30bf5566
2022-06-27 10:30:04 -04:00
Paul Willoughby
de125eeb51 cmd/uplink: Fix typo in flag description
Change-Id: I6db38ecec2549d288285a9865f97badfe808b892
2022-06-21 11:55:02 -06:00
Michał Niewrzał
7e387af010 cmd/uplink: add buffering while writing to stdout
Current pipelining to stdout is synchronous so we don't have any
advantage from using --parallelism flag. This change adds buffer
while writing to stdout. Each part is first read into the buffer
and flushed only when all data was read from this part.

https://github.com/storj/uplink/issues/105

Change-Id: I07bec0f4864dc4fccb42224e450d85d4d196f2ee
2022-06-09 15:10:04 +00:00
Jeremy Wharton
dba28f1d61 cmd/satellite: add project ID, bucket name to partner attribution CSV
This change adds project ID and bucket name columns to the generated
partner attribution report. Attribution values are now summed based on
their project ID and bucket name in addition to their user agent.
Additionally, the command to generate the attribution report has been
modified to optionally include only certain user agents.

Change-Id: I61a1d854379134f26b31467d9e83a787beb451dd
2022-06-01 20:27:12 +00:00
Márton Elek
84d02f7fbf metabase: use predefined snapshot for unit tests avoid migration
Change-Id: I6225ae32c312030e3559d8ed0faa137ffc3cc5b7
2022-06-01 12:07:00 +00:00
Ivan Fraixedes
aa913e63db cmd/storj-sim: Fix path to admin UI assets folder
The admin UI assets aren't inside of the `web` directory they are
directly in the `satellite` one.

The invalid path provoked that storj-sim generated a satellite
configuration with an invalid path to the Admin UI static assets
provoking that it didn't load the UI by default.

Change-Id: I49fb289377f51634057173690fbd8cf863ca9a9d
2022-05-31 18:54:54 +00:00
Michał Niewrzał
ffbb43ddbc cmd/uplink: fix how we are collecting errors while copy in parallel
Main issue was that when one part copy failed while being inside
goroutine (limiter) and another part was still collecting src/dst parts
it was possible to drop errors from failed part copy. It was possible
bacause on fail context was canceled and if we were still getting
part src/dst then it was returning error immediately and error
group with errors from goroutine was ignored.

Change-Id: I75c6799eba358741629795f2971c7a964cb2c9ce
2022-05-31 10:18:51 +00:00
Michał Niewrzał
4f2fae4f28 cmd/uplink: better error handling for parallel transfer
Few improvements were made to how we are handling errors
while doing parallel upload/download for single object:
* unhide error under 'context canceled' which was shown in most of
cases
* add part number to error message
* don't try to commit if any error occurs while operation
* combine errors into more readable form, example:

---
failed to download part 3: uplink: eestream: failed to download stripe 0:
error retrieving piece 00: ecclient: piecestore: rpc: tcp connector failed: rpc: dial tcp 97.119.158.36:28967: i/o timeout
...
error retrieving piece 89: ecclient: piecestore: rpc: tcp connector failed: rpc: dial tcp 161.129.152.194:28967: i/o timeout
failed to download part 1: uplink: eestream: failed to download stripe 0:
error retrieving piece 01: io: read/write on closed pipe
...
error retrieving piece 97: io: read/write on closed pipe
failed to download part 2: uplink: eestream: failed to download stripe 0:
error retrieving piece 00: io: read/write on closed pipe
...
error retrieving piece 01: ecclient: piecestore: rpc: tcp connector failed: rpc: dial tcp 180.183.132.234:28967: operation was canceled
error retrieving piece 96: io: read/write on closed pipe
	main.(*cmdCp).parallelCopy:418
	main.(*cmdCp).copyFile:262
	main.(*cmdCp).Execute:156
	main.(*external).Wrap:123
	github.com/zeebo/clingy.(*Environment).dispatchDesc:126
	github.com/zeebo/clingy.(*Environment).dispatch:53
	github.com/zeebo/clingy.Environment.Run:34
	main.main:26
	runtime.main:250
---

Change-Id: I9bb70b3f754567761fa8d17bef8ef59b0709e33b
2022-05-27 14:00:35 +00:00
Paul Willoughby
1d97b2c855 cmd/uplink: add use - for stdout,stdin to cp help
Change-Id: Ife3a0972d1be119a73eaefc0e23407b74fe03f54
2022-05-26 10:27:20 -06:00
Paul Willoughby
036d5a8619 cmd/storj-sim: fix printing node url
Fix printing satellite url to help configure authservice.
  storj-sim network env SATELLITE_0_URL

Change-Id: I786ab6723f837bbc35f1228eb73a82427d2f9250
2022-05-25 00:09:17 -06:00
Michał Niewrzał
d90ce467fc cmd/uplink: bring back --metadata for cp command
At some point uplink cli lost ability to set metadata. This change
brings back this functionality for 'cp' operation.

https://github.com/storj/storj/issues/3848

Change-Id: Ia5f60eb577fcab8a38d94730d8cdc6e0338d3b46
2022-05-18 15:58:53 +00:00
Michał Niewrzał
f9a3f19443 scripts: add 'uplink cp' tests for stdin/stdout
Uplink can upload from stdin and download to stdout. We had
such tests for old binary but now we were missing it.

Change-Id: I5110a9f531f5cc21277fa53611995fb5b556ff16
2022-05-18 08:42:04 +00:00
Jeff Wendling
836fa188fa cmd/uplink: allow listing invalid access grants
if you somehow get an invalid access grant in your
config file, it'd be nice to be able to list it
and delete it and stuff.

Change-Id: I7e335bf32353f294d5abb6a7c5f8f3aa18f2f6a7
2022-05-17 20:00:36 +00:00
Clement Sam
87ea2a4794 cmd/storagenode: make supervisor HTTP Server configurable
The current supervisord condifguration sets up the HTTP server
to listen on a tcp socket which is private i.e. available only
on localhost. This poses a regression where multiple containers
cannot be run if the host network interface is used when docker
container is run with `--network host` option.

This change adds a new env variable `SUPERVISOR_SERVER`, with
potential values `unix | private_port | public_port`, where
`unix` is set as the default value.

By default, the HTTP server is now set to listen on a UNIX
domain socket.
The file path is set to `/etc/supervisor/supervisor.sock`
instead of the /tmp directory since some systems
periodically delete older files in /tmp. If the socket file is
deleted, supervisorctl will be unable to connect to supervisord.

When SUPERVISOR_SERVER is set to `public_port` or `private_port`,
the HTTP server is set to listen on a TCP socket.

Resolves https://github.com/storj/storj/issues/4661

Change-Id: I224836dcae0293bcfe49874f2748be7723944687
2022-05-16 20:06:24 +00:00
JT Olio
5fb9ee3cfa uplink, satellite: use bgp hostnames
Change-Id: I58f5011d3019f8267fa8cbd3096b2cfe42eb5f8b
2022-05-16 18:14:37 +00:00
Stefan Benten
7afdb15fc8
cmd/uplink: adding Length Method to MultiReadHandle
This changes allows fetching the file size more easily (for supported
files) in order to afterwards calculate the multipart part size
accordingly.

Change-Id: Idabba4c2ee794ee471973889f5843174a7acad35
2022-05-13 21:31:45 +02:00
Stefan Benten
5e4ec0b3be
cmd/uplink: adjust multipart part size based on file size
This change allows the uplink to bump the part size based on the
content length that is being copied. This ensures we are staying
below the 10k part limit currently enforced on the satellites.

If the user specifies the flag, it will error out if the value
chosen by the user is too low. Otherwise it will use it.

Change-Id: I00d30f603d941c2f7703ba19d5923e668629a7b9
2022-05-13 21:31:23 +02:00
Egon Elbre
4791ba5d50 satellite/{mailservice,oidc}: improvements to debugging
Things that make debugging easier.
* Added logging to automatic link clicking to make it obvious, when it
  fails.
* Added monitoring to oidc.
* Made dbx create calls noreturn for oauth_*

Change-Id: I37397b4e84ce5bfd82954aed9c38fdfd52595f24
2022-05-11 19:59:42 +00:00
Jeff Wendling
89ccfe2dd7 cmd/uplink: fix recursive copy and improve tests
recursive copy had a bug with relative local paths.

this fixes that bug and changes the test framework
to use more of the code that actually runs in uplink
and only mocks out the direct interaction with the
operating system.

Change-Id: I9da2a80bfda8f86a8d05879b87171f299f759c7e
2022-05-11 15:17:16 -04:00
Erik van Velzen
928375a67c satellite/repair/queue: buffer batch insert
Implement a buffer for inserting repair items into the queue in a batch.

Part of https://github.com/storj/storj/issues/4727

Change-Id: I718472b2f2b1f4993c3d6f15c44923776407155a
2022-05-11 09:02:20 +00:00
Clement Sam
b471a5d8e3 cmd/storagenode/Dockerfile: bump storagenode-base image version
The new storagenode base image version contains the fix for the
failing "processes" supervisord event listener.

Resolves https://github.com/storj/storj/issues/4772

Change-Id: I6d67aa6f85ee33cd9abe6a663e4f9a84ea57fdbf
2022-05-04 16:00:26 +00:00
Clement Sam
9751ac9de2 cmd/storagenode: fix failing supervisord "processes" eventlistener
/bin/stop-supervisor fails in posix shell since the standard read utility
takes at least one variable's name as argument.

Changing the header #!bin/sh to #!/bin/bash fixes this issue.
`read` with no variable's name works in bash.

Looks like the shell in alpine isn't POSIX-compliant so we didn't
encounter this issue on alpine.

Also, I changed the name from "processes" to "processes-exit-eventlistener"
to make it clearer in the logs since supervisord spawns event listeners as
separate processes.

Change-Id: Ife9378c2013e2eb54f2adcd52a163d64eaacbbab
2022-05-04 02:44:23 +00:00
Clement Sam
87cb2e92b9 cmd/storagenode: bump storagenode-base image version
Change-Id: I8df391ee12c70cc599b54115b54eb987a2114b74
2022-05-04 01:59:00 +00:00
Clement Sam
a9faf756b0 cmd/storagenode: get rid of CRITICAL log when running supervisord as root
Closes https://github.com/storj/storj/issues/4729

Change-Id: Iea9a60ff60d753af375977f9bd90557574f80de0
2022-05-03 21:18:11 +00:00
Clement Sam
4a46d41c1c cmd/storagenode: change supervisord pidfile location
When running the docker auto-updater image as non-root user,
supervisord logs a "CRIT could not write pidfile /run/supervisord.pid"
since the user does not have permission to the /run directory.

Changing the location to /etc/supervisor fixes it because permissions
are set for non-root access of the /etc/supervisor directory.

Closes https://github.com/storj/storj/issues/4730

Change-Id: Id463f3a08db44dd9283921ece4575abdad9bd7f2
2022-05-03 12:54:40 +00:00
Stefan Benten
345ab87b5c cmd/uplink: adding output flag for ls command
With this change users can use the uplink cli in
scripts (ie. bash) more easily, since the output
can be switched to an easier processable json format.
It keeps the default of tabbed output.


Change-Id: I37e2c55f75c2250c3119fd8df8b66a766ff9096b
2022-04-29 10:32:04 +00:00
Paul Willoughby
8efed4b270 cmd/uplink: raise fd limits
Change-Id: I507c92c38d45d3d2be3edee08b6b85552ae21c43
2022-04-27 18:11:44 +00:00
Jeff Wendling
fead0db981 cmd/uplink: return nicer error message during import
Fixes #4750

Change-Id: Icef3340dd04d1c502b1651dcc70362895f258563
2022-04-27 09:30:57 +00:00
Jeff Wendling
f25ead5f98 cmd/uplink: set default parallelism to 1
Change-Id: Ic4198131c9958cc864fd861f983e32776bf56595
2022-04-26 22:55:11 +00:00