Commit Graph

493 Commits

Author SHA1 Message Date
Michal Niewrzal
36e046375c satellite/repair/checker: remove segments loop parts
We are switching completely to ranged loop.

https://github.com/storj/storj/issues/5368

Change-Id: I8583549973cd36aa0e0c482c20d7a75cb7568ab3
2023-05-08 12:19:13 +00:00
Jeff Wendling
6ac72e42cd all: bump storj.io/drpc
this includes a fix to cancellation when soft cancel
is enabled.

Change-Id: I1b9ab071f623153a2bd715888f97ca20277a76d7
2023-05-06 01:02:50 +00:00
Egon Elbre
8b82dba602 storagenode/blobstore/filestore: add tracking of blobs
We've had issues with forgetting to close readers and writers.
Add leak tracking to find those pesky issues.

Change-Id: If6b0ad6e9958318a7e0affee9c6d0a1ece412b6d
2023-05-05 15:40:15 +03:00
Jeff Wendling
80b3edf1d1 storagenode/piecestore: respect maximum chunk size requests
See https://review.dev.storj.io/c/storj/common/+/10297 for
more details.

Change-Id: Id5d19f029ae872780a554874592679191c1b5b2f
2023-04-28 16:31:41 -04:00
Márton Elek
1588bf9606 go.mod: bump monkit
Change-Id: Icc286378a8b01518510a4b4dd4d213d0c5d3b5b6
2023-04-25 14:26:11 +00:00
Egon Elbre
2405bc8f3b satellite/metabase: stop using the common error type
Updates https://github.com/storj/storj/issues/5291

Change-Id: I7b57a4b454d3619cb5d8ae4cd92f818ad2839c8b
2023-04-19 19:18:18 +03:00
Egon Elbre
d94207048a go.mod: bump vbauerster/mpb/v8 for fixes
Change-Id: I2d24eec4389ca8e5effdbfc25d7012ce083d46f5
2023-04-18 15:53:35 +03:00
Jeff Wendling
df9cc4d10f all: bump storj.io/{common,uplink,drpc}
Change-Id: Ie98a7eacc11a492a20c76636f8044c81b7ed580d
2023-04-17 19:59:30 +00:00
Jeff Wendling
54ef1c8ca2 cmd/uplink: use new upload code path
the parallelism and parallelism-chunk-size flags
which used to control how many parts to split a
segment into and many to perform in parallel
are now deprecated and replaced by
maximum-concurrent-pieces and long-tail-margin.

now, for an individual transfer, the total number
of piece uploads that transfer will perform is
controlled by maximum-concurrent-pieces, and
segments within that transfer will automatically
be performed in parallel. so if you used to set
your parallelism to n, a good value for the pieces
might be something approximately like 130*n, and
the parallelism-chunk-size is unnecessary.

Change-Id: Ibe724ca70b07eba89dad551eb612a1db988b18b9
2023-04-13 16:52:38 -04:00
Andrew Harding
e676b5c893 cmd/uplink: progress bars for recursive copy
```
$ uplink cp -r -t=3 files/ sj://files
uploading 6 files...
files/bar/buz   (2 of 6) 134.22 MB / 134.22 MB [============================================] 100.00% 36.43 MiB/s
files/bar/baz   (1 of 6) 67.11 MB / 67.11 MB [==============================================] 100.00% 18.39 MiB/s
files/boo       (3 of 6) 67.11 MB / 67.11 MB [==============================================] 100.00% 20.42 MiB/s
files/foo       (4 of 6) 67.11 MB / 67.11 MB [==============================================] 100.00% 57.83 MiB/s
files/glue/flew (5 of 6) 67.11 MB / 67.11 MB [==============================================] 100.00% 55.01 MiB/s
files/stew      (6 of 6) 67.11 MB / 67.11 MB [==============================================] 100.00% 91.43 MiB/s
```

Change-Id: Ibd9d07a1291f7a599bd27fba93c1b2e0f17dc787
2023-04-10 15:13:22 +00:00
Márton Elek
e5995947d0 go.mod: bump storj.io/private and storj.io/uplink
Change-Id: I01790c1af8d6302b3c9ffee0482c952bb9a5d8da
2023-04-06 13:44:34 +00:00
Michal Niewrzal
c4c391e154 satellite/accounting/live: replace address parsing with redis util
With this change we are replacing parsing code with existing go-redis
util.

We also switch redis client to version 9.

Change-Id: Ie4a651e3ae6960e68958c690873925d319b70e10
2023-04-05 13:20:11 +00:00
Clement Sam
5ab2e6ac19 go.mod: bump storj.io/private
Updates https://github.com/storj/storj/issues/5349

Change-Id: I5b91a8e9c28d39c6ec432992a352de08aff886d9
2023-04-05 14:46:59 +03:00
Michal Niewrzal
4c05293d8b satellite: set names for lrucache metrics
Fixes https://github.com/storj/storj/issues/5685

Change-Id: I9c19b7644b0cd3297448982b920c97abcdce9119
2023-04-04 11:55:20 +00:00
JT Olio
bd238819b6 go.mod: bump storj.io/drpc
this includes improved connection reset error handling

Change-Id: I211221493b5dd3f4b95d4db44bf022302352f050
2023-03-29 19:27:57 +00:00
Márton Elek
5e6d1cc00a
go.mod: bump storj.io/{common,private,uplink}
Change-Id: I7af875b6ac73af3a9d77928f69a05b80a5321696
2023-03-27 16:13:52 +02:00
JT Olio
0b4b04900a private/server: debounce noise and tls connections
to support TCP_FAST_OPEN, we're considering just using
two TCP connections in parallel per request, one with
and one without. this allows us to safely fire both
concurrently without stressing out the node too much.

see https://review.dev.storj.io/c/storj/storj/+/9933

Change-Id: I9aa8a0252350db5ace04ee125bfe469203e980ec
2023-03-21 16:51:31 +00:00
Kaloyan Raev
2c28a0e599 cmd/uplink: ability to opt-out from object key encryption
Option added to `uplink access setup` and `uplink access create`
commands to disable object key encryption.

Related to https://github.com/storj/storj/issues/5678

Change-Id: I4789a94143742ff4b232fd60decc029ad2883c2a
2023-03-20 13:56:15 +00:00
Egon Elbre
0d981a6485 ci: check licenses
Change-Id: Ibe67bdb865fdb7c1059fe4bfb349299df86e2b62
2023-03-10 13:38:40 +00:00
JT Olio
4362761fc7 satellite/audit: fix go1.19 dial timeouts and log more
Change-Id: Ide17c1b8e0ca8c86f305bea1b4ae553cc4cb60d0
2023-02-28 17:09:47 +00:00
JT Olio
8befcaab27 go.mod: bump storj.io/{common,drpc,uplink}
this set of changes re-enables noise with fixes

Change-Id: I122142d202dd76a83b678ac8d709407b0791ed0b
2023-02-24 23:17:16 +00:00
Michal Niewrzal
16b7901fde satellite/metabase: add piece size calculation to segment
This code is essentially replacement for eestream.CalcPieceSize. To call
eestream.CalcPieceSize we need eestream.RedundancyStrategy which is not
trivial to get as it requires infectious.FEC. For example infectious.FEC
creation is visible on GE loop observer CPU profile because we were
doing this for each segment in DB.

New method was added to storj.Redundancy and here we are just wiring it
with metabase Segment.

BenchmarkSegmentPieceSize
BenchmarkSegmentPieceSize/eestream.CalcPieceSize
BenchmarkSegmentPieceSize/eestream.CalcPieceSize-8         	    5822	    189189 ns/op	    9776 B/op	       8 allocs/op
BenchmarkSegmentPieceSize/segment.PieceSize
BenchmarkSegmentPieceSize/segment.PieceSize-8              	94721329	        11.49 ns/op	       0 B/op	       0 allocs/op

Change-Id: I5a8b4237aedd1424c54ed0af448061a236b00295
2023-02-22 11:04:02 +00:00
JT Olio
f905878ff9 go.mod: bump storj.io/uplink
this change brings in:

 * private/piecestore: disable noise
 * uplink: fix noise version report
 * private/piecestore: fix noise
 * private/metaclient: honor context cancellation on retries
 * private/metaclient: fix multi-passphrase ListObjects paging
 * private/metaclient: fix broken ListUploads method

it also adds:

 * private/storage/streams/batchaggregator: introduce Aggregator
 * private/storage/streams/streambatcher: introduce Batcher
 * private/storage/streams: pieceupload package
 * private/storage/streams: segmenttracker package

but these commits provably change no existing behavior, since they
only add unused, new packages and types.

Change-Id: Id9efc65117b7e1ddcb3ffc8cc15d3e665ed11d7d
2023-02-21 16:09:19 +00:00
Cameron
dd0013bdda bump stripe-go version to v72.90.0
v72.90.0 grants access to InvoiceParams.PendingInvoiceItemsBehavior
which allows the caller to specify whether the invoice should apply all
pending invoice items.

Change-Id: I7c8295d38791ed7daa3d5b1fce15438eb42a962f
2023-02-15 21:16:56 +00:00
Márton Elek
ca6e3a9e88 satellite/orders: create mock based unit test
Most of our (~integration) tests based on testplanet runner.

However running testplanet for each test make the testing process slow.
It seems to be better to use real unit tests (without db dependency) when it's possible.

This patch makes small modification to make it possible to test orders.Service with real unit test.

As the existing unit test of `service.go` is isolated with `_test` package name, it's moved to an `_integration_test.go` file to make place for the unit test.

Change-Id: Ia69f26a34e2c48d230d8d36c2040dd02a60455a6
2023-02-13 13:24:30 +00:00
JT Olio
ca13eca718 go.mod: bump libuplink to include noise
Change-Id: If5bceb139ce6fdf6c0792b4bb536bc61b54e32bb
2023-02-10 17:03:51 +00:00
JT Olio
522aed083d private/server,satellite/contact,misc: use new storj/common noise helpers
this change uses the new storj/common noise helpers, which:
 * add a security fix (require an expected node id for validating
   noise key attestations)
 * stops doing an unnecessary order signature validation (it's
   already been done inside of PutPiece)
 * removes some duplicate code

Change-Id: I5e67a08ff216cd9c5b0b82e40b4d9de664b6b0fc
2023-02-07 09:53:45 -05:00
Egon Elbre
873a202530 mod: bump storj.io/common
This bumps common, such that things build with Go 1.20.

Also, adds `go vet` checks for testsuite/storjscan and testsuite/ui.

The latest golang.org/x/bcrypt has a check that the new password is less
than 72 bytes, because bcrypt silently discarded them. This means our
own password validation has the same limitation. Old passwords should
still work fine.

Change-Id: Ibb8735b15eeb91460145906b81ae4e365e9ac418
2023-02-03 16:49:41 +02:00
Egon Elbre
3976a2fd1d go.mod: bump common to latest
Temporarily allow larger wasm size, while we migrate to picobuf.

Change-Id: I55916f47486465e4223cbd79f412ab2876fee042
2023-02-01 12:33:37 +02:00
JT Olio
3e01e9c07a storj/private/server: listen for noiseconn requests
Change-Id: Ia74dcc576fa0c97460207d93d129aa7d88cd2fba
2023-01-30 13:23:08 -05:00
Michal Niewrzal
bd8867cd09 satellite: adjust code to handle context cancelation for SQL queries
Our DB support in storj/private was updated to enable basic context
support for executing SQL queries. This change requires some small
adjustments as not all parts were working correctly.

storj/private commit with change:
4bc77107b7acfcc2f7ad65796d5dd3d7c64801e4

Change-Id: I64d7ed92788ea0920d12cecd1aa0e414720e9b9c
2023-01-27 10:07:43 +01:00
Kaloyan Raev
674068a5f2 mod: bump go-redis to v8.11.5
This removes the indirect dependency to go.opentelemetry.io/otel, which
causes a dependency conflict in storj.io/ipfs-go-ds-storj.

Change-Id: I28c1e3637bb23f7949988511051d39c08b3fc852
2023-01-26 12:29:03 +00:00
Kaloyan Raev
805fcdf341 Revert "mod: bump go-redis to v8.11.5"
This reverts commit b2b564cc13.

Change-Id: I79a6b92758d1095113c85d49d673b243c31e6c82
2023-01-26 13:08:18 +02:00
Kaloyan Raev
b2b564cc13 mod: bump go-redis to v8.11.5
This removes the indirect dependency to go.opentelemetry.io/otel, which
causes a dependency conflict in storj.io/ipfs-go-ds-storj.

Change-Id: I28c1e3637bb23f7949988511051d39c08b3fc852
2023-01-26 13:07:13 +02:00
Andrew Harding
e7b35381f2 mod: bump common and uplink deps
Change-Id: Ie2d87f517b57a8760f50ac5c9f1b8b0448f5b3b5
2023-01-25 13:34:47 -07:00
JT Olio
e40191afd6 storj: upgrade to use latest storj/common NodeAddress
Change-Id: I5987391bcfe5f6dfd7b525698c337a4cbda9b76e
2023-01-25 01:37:26 +00:00
Egon Elbre
9544a670d7 storagenode/pieces: fix concurrent empty and restore trash
This ensures that empty trash and restore trash cannot run at the same
time.

Fixes https://github.com/storj/storj/issues/5416

Change-Id: I9d2e3aa3d66e61e5c8a7427a95208bb96089792d
2023-01-03 15:01:54 +00:00
Michal Niewrzal
5110803102 storagenode/piecestore: add Exists endpoint
Adds new method Exists which can be used to verify which
requested piece ids exists on storage node. Will verify only pieces
which belongs to the satellite that used that endpoint.

Minum WASM size was increased a bit.

https://github.com/storj/storj/issues/5415

Change-Id: Ia5f9cadeb526541b2776a8973eb7d50133ad8636
2022-12-17 04:08:26 +00:00
Márton Elek
ef2c0f4225 mod: use v1.10.0 tagged commit from storj.io/uplink
Change-Id: I475f2cf6447453dd78eb3549fdaacd286e1da494
2022-12-13 12:04:29 +00:00
Jeff Wendling
f2fdd6ca33 cmd/uplink: fix some issues with share
Because --readonly is default true, passing something like
--disallow-deletes=false would not actually update that
value because the readonly flag would override. this makes it
so that the --disallow-* flags override the --readonly and
--writeonly flags.

Also fixes some minor formatting issues with share like an
extra space after the "Public Access:" entry.

Simplifies the handling of the explicit "none" by making the
flags for the dates optional and using nil to signify that
the value was left unset.

Bump the go.mod to go1.18 to enable the use of generics and
add a small generic function. This can easily be backed out
if it causes problems.

Change-Id: I1c5f1321ad17b8ace778ce55561cbbfc24321a68
2022-12-08 17:46:02 +00:00
Márton Elek
27191d5b80 mod: bump storj.io/uplink
Change-Id: I8548f6bfafb86646bbc71b040190fad28579d92e
2022-12-07 14:07:18 +00:00
Michal Niewrzal
47125300dc mod: bump dependencies
* storj/common
* storj/private

Latests common version requires small refactoring for names and types
used by metainfo code.

Change-Id: I224fe93b4751c996ba6e846be0e5677252cf830f
2022-11-29 13:47:34 +00:00
Egon Elbre
5ee27fc42a mod: bump storj.io/uplink
Change-Id: I9692c3c8dd266b29d23a782f6aff3594a6423de3
2022-11-03 14:06:17 +00:00
Ethan
403003c294 storj: bump storj common package version
upgrade common to address connection leak

Change-Id: I4b8332120d803cace2856ef0248d73fa99469424
2022-10-28 21:38:47 +00:00
JT Olio
ec9d63d0ca mod: bump dependencies
bump storj.io/uplink to include small telemetry changes

Change-Id: Ia7767244e4eb8d1a86656a1e718576aa4b46218b
2022-10-28 14:53:42 +00:00
JT Olio
58a9c55f36 mod: bump dependencies
- storj.io/common

Change-Id: Ib78154acc253a13683495abfdd96d702625fdce8
2022-10-19 17:01:53 +00:00
JT Olio
52ede93086 mod: bump dependencies
- storj.io/uplink

Change-Id: Ic8a180dbc2f08a9300f3d122a012d84e58fa74b0
2022-10-19 09:47:24 -04:00
Egon Elbre
4fd59fb3c9 go.mod: bump common
Change-Id: I1ea76bad3d1fbe0cb78b8ab23a1512c6007a166f
2022-10-18 13:21:49 -04:00
Márton Elek
ea106f8894 go.mod: bump storj/private
Change-Id: I087b39c267c6492c4a5c85f470631b2f206cdf39
2022-10-13 11:23:57 +00:00
JT Olio
d632f23950 cmd/uplink: add eventkit
Change-Id: If109c8f7de257b77794e45599487ad2c46f2c3ec
2022-10-07 12:08:08 -04:00