Commit Graph

1340 Commits

Author SHA1 Message Date
dlamarmorgan
a010459520 satellite/payments/stripe/{client,server}: add set invoice status cmd
Change-Id: I6d9b4fe2ed4b17d63d858f7ceefd7c14adb062bb
2023-06-28 21:04:34 +00:00
Michal Niewrzal
cb9a7bdc71 satellite/repair/repairer: make DialTimeout configurable
This change makes dial timeout configurable and change it also from
defatul 20s to 5s. Main motivation is that during repair we often loose
lots of time to dial which eventually will fail. New timeout should be
still enough to dial but we will move forward quicker to next node if
that one will fail.

Timeout is also applied directly as context timeout in case we will
use noise of tcp fast open one day.

Change-Id: I021bf459af49b11241e314fa1a7887c81d5214ea
2023-06-16 12:23:25 +00:00
Egon Elbre
edbea5efe1 go.mod: bump to pgx/v5
Change-Id: I31cf3bec1d7db94f0f612f6ed04b782f8b04d876
2023-06-14 18:32:54 +03:00
Jeff Wendling
f3c58174c4 cmd/uplink: only use new code path for uploads
downloads still need the old copy code because they aren't
parallel in the same way uploads are. revert all the code
that removed the parallel copy, only use the non-parallel
copy for uploads, and add back the parallelism and chunk
size flags and have them set the maximum concurrent pieces
flags to values based on each other when only one is set
for backwards compatibility.

mostly reverts 54ef1c8ca2

Change-Id: I8b5f62bf18a6548fa60865c6c61b5f34fbcec14c
2023-06-09 23:45:30 +00:00
Márton Elek
61ad5751f6
cmd/uplink: exclude unnecessary spans from distributed tracing
We have a special method to exclude methods (which are called to frequently) from distributed traces.

https://github.com/storj/common/blob/main/tracing/excluded.go

But this works only, if we define the exclusion function during the initialization.

Let's do it in this patch.

Change-Id: Icf12202bd7213b5c0009332ce2755b267f2bdbae
2023-06-05 12:33:19 +02:00
Clement Sam
b64179c82a {storagenode/pieces,cmd/storagenode}: refactor lazyfilewalker commands and tests
With this change we are directly testing how the command
is executed when the args are passed

Change-Id: Ibb33926014c9d71c928e0fd374bf4edc5a8a1232
2023-06-02 00:11:53 +00:00
Jeff Wendling
ccfe5cae49 cmd/uplink: pass in a maximum concurrent segments value
also change the config creation to be more robust to
changes that add defaults in the future by not fully
reconstructing the config value passed in to the
project.

Change-Id: I673e8b54ce0b951ae735bf4658525c477c26ac5a
2023-05-25 10:53:50 -04:00
Jeff Wendling
a0fbc87b31 cmd/uplink: add upload log file flag
Change-Id: I740eaf431cfe5bb3dcb746e4128dea5efc935257
2023-05-25 14:14:00 +00:00
Márton Elek
d6615cb698
makefile, docker: purge deleted (satellite) inspector tool from build files
the inspector tool of satellite is removed recently. We need to remove it from the build/docker files.

Change-Id: Icd75474ea64fe5f9dd4ed76d8597982108f93536
2023-05-25 11:04:32 +02:00
Clement Sam
bf8072d3f3 cmd/storagenode: fix identity flag for graceful exit commands
Resolves https://github.com/storj/customer-issues/issues/780
Resolves https://github.com/storj/storj/issues/5807

Change-Id: I0af2b14f5b12ddc1e7a09afb2cf5ccf025f53502
2023-05-24 07:58:34 +00:00
Michal Niewrzal
eabd9dd994 satellite/orders: remove unsed argument
Change-Id: I6c5221fc19f97ae6db5627d7239795ff663289e0
2023-05-22 14:35:08 +00:00
Cameron
09ec5f107d satellite/payments: Remove expired package credits
During billing, before invoice creation, check if users are part of a
package plan. If so, and if the package plan is expired, remove unused
credit from the user's balance. If the user has credit in addition to
the package credit, send an analytics event to notify someone to handle
the credit removal manually.

Change-Id: Iad71d791f67c9733f9d9e42f962c64b2780264cc
2023-05-22 11:58:37 +00:00
JT Olio
b502310fe5 cmd/inspector: remove
this is a very old tool built in the very early days
of v3, when we didn't know how the network would be
used. this tool anticipated being able to query remote
nodes for internal state. we don't do that. i don't
think anyone uses this.

Change-Id: Ie1ded3ecbedb09313f2d6fc721039e0f15e4ee85
2023-05-19 09:11:29 +00:00
Michal Niewrzal
c0e7f463fe satellite/metabase: remove segmentsloop package
Last change to remove segments loop from codebase.

https://github.com/storj/storj/issues/5237

Change-Id: I77b12911b6b4e390a7385e6e8057c7587e74b70a
2023-05-18 19:08:29 +00:00
Michal Niewrzal
cf5ff537e3 cmd/tools/metabase-verify: switch to ranged loop
We would like to remove segments loop so we need to refactor
our tools to use ranged loop.

To simplify change ranged loop is used with single range only.

https://github.com/storj/storj/issues/5237

Change-Id: I94d96d54f9d0e37b06def4f4fc16b71c5b79baba
2023-05-18 14:27:38 +00:00
Clement Sam
018b6eeeaf storagenode: add tests for lazyfilewalker
Updates https://github.com/storj/storj/issues/5349

Change-Id: I9544c14ba2acacd5b304f151ab29c70ff61adc5b
2023-05-08 21:50:40 +00:00
Clement Sam
cf7ce81d09 cmd/storagenode: refactor lazyfilewalker commands to satisfy the execwrapper.Command interface
Follow-up change for https://review.dev.storj.io/c/storj/storj/+/10335

Updates https://github.com/storj/storj/issues/5349

Change-Id: Iadf55bae84ebc0803a0766830e596c396dfb332b
2023-05-08 15:09:53 +00:00
Márton Elek
56dbe7738d cmd/storagenode: use the right subcommand for restarting windows services
Commit 3cf89633e9 is changed how the cobra subcommands are created for storagenode (with prefering local variables instead of package level variables).

However, there is a bug which makes it impossible to restart Storagenode services on Windows: the refactored code creates the rootCmd/runCmd twice: therefore the ctx of the running process is not exactly the same as the ctx which supposed to be stopped / cancelled.

This patch fixes this problem with re-using exising, initialized command instead of creating a new one for cancellation.

Fixes: https://github.com/storj/storj/issues/5845

Change-Id: Ib8a4b80d4574e448f65c8558e927c0908c9c5eed
2023-05-08 14:09:37 +00:00
Jeremy Wharton
f61230a670 satellite/console/dbcleanup: create console DB cleanup chore
A chore responsible for purging data from the console DB has been
implemented. Currently, it removes old records for unverified user
accounts. We plan to extend this functionality to include expired
project member invitations in the future.

Resolves #5790
References #5816

Change-Id: I1f3ef62fc96c10a42a383804b3b1d2846d7813f7
2023-05-05 19:11:53 +00:00
Clement Sam
e0542c2d24 storagenode: run garbage collection filewalker as a low I/O subprocess
Updates https://github.com/storj/storj/issues/5349

Change-Id: I7d810d737b17f0b74943765f7f7cc30b9fcf1425
2023-05-02 19:43:38 +00:00
dlamarmorgan
efcae857ba satellite/main,stripe/{client,service}: stripe balance invoice item cmd
Add a new billing command that will convert stripe customer balances into invoice items so that the charges can be processed normally by the invoicing workflow.

Change-Id: Iaa8350e7aca80a0f14e94eb8ef8b7d6ce0b5b3b8
2023-04-25 21:08:23 +00:00
Márton Elek
768e69b55a cmd/storagenode: flag to configure identity dir for dashboard config
Change-Id: Ifc0af33749b67aad201a8fc90116bc5ddbded007
2023-04-20 14:22:00 +00:00
Egon Elbre
d94207048a go.mod: bump vbauerster/mpb/v8 for fixes
Change-Id: I2d24eec4389ca8e5effdbfc25d7012ce083d46f5
2023-04-18 15:53:35 +03:00
dlamarmorgan
d53a56cef6 cmd/uplink/initial_setup.go: fix logic on analytics prompt
Correct the logic for the analytics prompt on a new setup of uplink.

Change-Id: I86755711dd2f3c8418609bd7072ae5d55927b841
2023-04-18 03:02:12 +00:00
Clement Sam
f076238748 storagenode: run used-space filewalker as a low IO subprocess
As part of fixing the IO priority of filewalker related
processes such as the garbage collection and used-space
calculation, this  change allows the initial used-space
calculation to run as a separate subprocess with lower
IO priority.

This can be enabled with the `--storage2.enable-lazy-filewalker`
config item. It falls back to the old behaviour when the
subprocess fails.

Updates https://github.com/storj/storj/issues/5349

Change-Id: Ia6ee98ce912de3e89fc5ca670cf4a30be73b36a6
2023-04-14 04:16:14 +00:00
Jeff Wendling
54ef1c8ca2 cmd/uplink: use new upload code path
the parallelism and parallelism-chunk-size flags
which used to control how many parts to split a
segment into and many to perform in parallel
are now deprecated and replaced by
maximum-concurrent-pieces and long-tail-margin.

now, for an individual transfer, the total number
of piece uploads that transfer will perform is
controlled by maximum-concurrent-pieces, and
segments within that transfer will automatically
be performed in parallel. so if you used to set
your parallelism to n, a good value for the pieces
might be something approximately like 130*n, and
the parallelism-chunk-size is unnecessary.

Change-Id: Ibe724ca70b07eba89dad551eb612a1db988b18b9
2023-04-13 16:52:38 -04:00
Andrew Harding
e676b5c893 cmd/uplink: progress bars for recursive copy
```
$ uplink cp -r -t=3 files/ sj://files
uploading 6 files...
files/bar/buz   (2 of 6) 134.22 MB / 134.22 MB [============================================] 100.00% 36.43 MiB/s
files/bar/baz   (1 of 6) 67.11 MB / 67.11 MB [==============================================] 100.00% 18.39 MiB/s
files/boo       (3 of 6) 67.11 MB / 67.11 MB [==============================================] 100.00% 20.42 MiB/s
files/foo       (4 of 6) 67.11 MB / 67.11 MB [==============================================] 100.00% 57.83 MiB/s
files/glue/flew (5 of 6) 67.11 MB / 67.11 MB [==============================================] 100.00% 55.01 MiB/s
files/stew      (6 of 6) 67.11 MB / 67.11 MB [==============================================] 100.00% 91.43 MiB/s
```

Change-Id: Ibd9d07a1291f7a599bd27fba93c1b2e0f17dc787
2023-04-10 15:13:22 +00:00
paul cannon
915f3952af satellite/repair: repair pieces on the same last_net
We avoid putting more than one piece of a segment on the same /24
network (or /64 for ipv6). However, it is possible for multiple pieces
of the same segment to move to the same network over time. Nodes can
change addresses, or segments could be uploaded with dev settings, etc.
We will call such pieces "clumped", as they are clumped into the same
net, and are much more likely to be lost or preserved together.

This change teaches the repair checker to recognize segments which have
clumped pieces, and put them in the repair queue. It also teaches the
repair worker to repair such segments (treating clumped pieces as
"retrievable but unhealthy"; i.e., they will be replaced on new nodes if
possible).

Refs: https://github.com/storj/storj/issues/5391
Change-Id: Iaa9e339fee8f80f4ad39895438e9f18606338908
2023-04-06 17:34:25 +00:00
Michal Niewrzal
8a50a3baa3 satellite/payments: rename 'stripecoinpayments' package to 'stripe'
Automatic rename. May require some more cleanups later.

Change-Id: I18220a4278056d25c41fb137832bb81f2b876ac1
2023-04-06 16:51:43 +00:00
Clement Sam
3cf89633e9 cmd/storagenode: refactor main.go
The cmd/storagenode/main.go is a big mess right now with so many
unneeded config structures initialized and shared by several
subcommands.

There are many instances where the config structure of one subcommand
is mistakenly used for another subcommand.

This changes is an attempt to clean up the main.go by moving the
subcommands to a separate `cmd_*.go` files with separate config structures
for each subcommand.

Resolves https://github.com/storj/storj/issues/5756

Change-Id: I85adf2439acba271c023c269739f7fa3c6d49f9d
2023-04-06 12:48:23 +00:00
Sean Harvey
60444d145c
cmd/uplink: minor text fixes for "uplink share"
updates flag descriptions with correct punctuation, and fix errors
to not be capitalized.

Updates #5623

Change-Id: I9c6ef6d9888b2fb90b17db8775cc6abe803e102f
2023-04-03 11:34:35 +12:00
Egon Elbre
a1e920bbdf cmd/tools/segment-verify: satisfy wastedassign linter
The assignemnt to `err = nil` is not used in the rest of the code,
however, this was a protective err = nil assignment.

Change-Id: Id70fb2a2e68b91e2481952d865334e603ca41188
2023-03-31 08:52:46 +00:00
Sean Harvey
33c3e3176f
cmd/uplink: add --tls flag to share command
adds an additional flag to return an additional TXT record that will
enable TLS on custom domains with Linksharing.

Closes #5623

Change-Id: I941616362d7dcd9aec20dfd10346e483021516a4
2023-03-31 20:30:22 +13:00
Kaloyan Raev
2c28a0e599 cmd/uplink: ability to opt-out from object key encryption
Option added to `uplink access setup` and `uplink access create`
commands to disable object key encryption.

Related to https://github.com/storj/storj/issues/5678

Change-Id: I4789a94143742ff4b232fd60decc029ad2883c2a
2023-03-20 13:56:15 +00:00
Michal Niewrzal
38c7671acb satellite: bump capacity for API keys cache
We have lots of direct DB requests to get API keys. It should be handled
by cache but default value is very low at the moment.

Fixes https://github.com/storj/storj/issues/5665

Change-Id: I214ebebd6e397cacff80b2f36dc4a2eea388f93d
2023-03-13 16:32:50 +00:00
Márton Elek
0b66d22be4 cmd/uplink: support --trace-tags for custom tags
We do regular testing with executing uplink. But sometimes the recorded execution time showed spikes.

Would be nice to know the reason of the spikes (just internet blip, or something what we should be worried about).

We can collect distributed traces, but it's not easy to find the right trace in Jaeger.

 * We can provide a random trace-id, but it should be persisted / processed
 * We can also save standard output and use `--trace-verbose` which prints out the used trace id, but it's also complicated to collect all of them in a DB

Would be nice to attach additional metadata to traces to make sure that we can filter all traces of one specific kind of test.

This patch provides this feature:

  * It always adds hostname to the trace (if you opt-in to distributed tracing, which is turned off by default)
  * Additional tags can be defined with CLI flag

Tags can be used to find the right trace in Jaeger (or in Elastic search backend of Jaeger).

Change-Id: I08f10023bbebd783f812cfca95ac6237360ac2b0
2023-03-13 12:03:39 +00:00
Lizzy Thomson
7e2d98988b cmd/tools: remove migration tool code for generating missing salt
Remove generate-missing-project-salt migration tool code and related
tests. This migration has already been run and this code is no longer
needed.

Issue https://github.com/storj/storj-private/issues/163

Change-Id: I4e36dcd95a07c5305c597113a7fd08148e100ccc
2023-03-10 16:38:54 +00:00
paul cannon
fd6ce6b9a5 scripts/tests: fix test-sim-rolling-upgrade.sh
This test involves a satellite with dev defaults (DistinctIP=no) being
upgraded past commit 2522ff09b6, which
means we need to run the dev-defaults-satellite-upgrade migration SQL
to avoid getting DistinctIP=yes behavior (which breaks the tests).

Change-Id: I29fb596d1ffa568dad635d98cfe9abacd3aaa48f
2023-03-09 23:35:36 +00:00
Michal Niewrzal
0c177ef91f satellite: cleanup orders dependencies
Only API peer needs access to order DB (and rollups cache) because it's
only place where we are creating orders for PUT and GET operations. For
other peers like auditor and repairer we can set noop implementation to
reduce number of dependencies needed for them.

Change-Id: Ic32d1879f0b97ffc4516f401898e31e95ae892e4
2023-03-09 13:34:21 +00:00
Márton Elek
ffaf15a3b0 satellite/overlay: remove unused mail service from overlay
It was surprising that `satellite auditor` complained about SMTP mail settings, even if it's not supposed to sending any mail.

Looks like we can remove the mail service dependency, as it's not a hard requirement for overlay.Service.

Change-Id: I29a52eeff3f967ddb2d74a09458dc0ee2f051bd7
2023-03-09 12:17:35 +00:00
Sembeth
0408997e6c
cmd/multinode: Removes dependency on deprecated identity-dir flag, code and documentation. (#5646)
* The Multinode Dashboard no longer requires the identity files.

* Removed deprecated code that used the identity-dir.
2023-03-08 13:56:15 +01:00
JT Olio
d0408ee89b cmd/uplink: stop using transport.SetConnector
quic is still configurable based on the quic rollout
environment variables in storj.io/common. this stops
using a method removed in:
https://review.dev.storj.io/c/storj/uplink/+/9815

Change-Id: Ibfe28cfb19e5672630970b9e2c8c6ac0c98d4822
2023-03-06 21:48:26 +00:00
Márton Elek
6737d427e4
cmd/uplink: better usability for date specification
I use `uplink share` command but I always fail to set the --not-before parameter.

 * Usually I try +2d when I see in the help that +2h is possible --> fail
 * When it fails, I try to set explicit date, like 2012-12-23 --> fail

This patch makes it possible to use:

 * day duration (like +3d)
 * shorter date definition (like `2023-12-12` or `2023-12-12T12:40`)

Change-Id: I2243b36f59c8929eb0473c4bb4fed19220890c71
2023-03-06 13:07:27 +01:00
Egon Elbre
63fa386b0a satellite/payments/stripecoinpayments: avoid mock cross-talk in tests
The tests were using global variables for keeping the mock state, which
was indexed by the satellite ID. However, the satellite ID-s are
deterministic and it's possible for two tests end up using the same
mocks.

Instead make the mock creation not depend on the satellite ID and
instead require it being configured via paymentsconfig.

This fixes TestAutoFreezeChore failure.

Change-Id: I531d3550a934fbb36cff2973be96fd43b7edc44a
2023-03-03 20:20:48 +02:00
JT Olio
f3c5cd9df2 cmd/satellite: limit time spent restoring trash from a node
okay to do now that
https://review.dev.storj.io/c/storj/storj/+/7882
is widely deployed

Change-Id: I26c9a982b3999186e7c34a30515d151bf18a8648
2023-02-24 20:56:09 +00:00
Michal Niewrzal
16b7901fde satellite/metabase: add piece size calculation to segment
This code is essentially replacement for eestream.CalcPieceSize. To call
eestream.CalcPieceSize we need eestream.RedundancyStrategy which is not
trivial to get as it requires infectious.FEC. For example infectious.FEC
creation is visible on GE loop observer CPU profile because we were
doing this for each segment in DB.

New method was added to storj.Redundancy and here we are just wiring it
with metabase Segment.

BenchmarkSegmentPieceSize
BenchmarkSegmentPieceSize/eestream.CalcPieceSize
BenchmarkSegmentPieceSize/eestream.CalcPieceSize-8         	    5822	    189189 ns/op	    9776 B/op	       8 allocs/op
BenchmarkSegmentPieceSize/segment.PieceSize
BenchmarkSegmentPieceSize/segment.PieceSize-8              	94721329	        11.49 ns/op	       0 B/op	       0 allocs/op

Change-Id: I5a8b4237aedd1424c54ed0af448061a236b00295
2023-02-22 11:04:02 +00:00
Jeremy Wharton
3fa31c2077 satellite/console/consoleweb: remove trailing slash from URLs
This change removes the trailing slash from the account activation and
password recovery URLs, making them consistent with the rest. The URLs'
previous forms are still supported, however, in order to not invalidate
emails containing them.

Resolves storj/customer-issues#491

Change-Id: Ie774a87698d8e9edd1836611968fc3911c6cc56f
2023-02-21 19:15:36 +00:00
Egon Elbre
0cdef95d55 all: fix math/rand deprecations
Change-Id: I4b966375697c0d409ce24cc7604f806973f8f22a
2023-02-17 15:05:54 +02:00
Michal Niewrzal
94d341bcf3 satellite: use ranged loop with GC-GF peer
Peer for generating bloom filters will be able to use ranged loop.

As an addition some cleanup were made:
* remove unused parts of GC BF peer (identity, version control)
* added missing Close method for ranged loop service
* some additional tests added

https://github.com/storj/storj/issues/5545

Change-Id: I9a3d85f5fffd2ebc7f2bf7ed024220117ab2be29
2023-02-13 18:32:21 +00:00
Egon Elbre
0e00c7b8da certificate/authorization,cmd/certificates: remove gob code
Now that we've migrated data we can remove gob handling.

Change-Id: I55f772b7a0dda71c51db0683bad3db66b89867ac
2023-02-08 15:05:40 +00:00