Commit Graph

184 Commits

Author SHA1 Message Date
Vitalii
1eee2fad69 satellite/{db, admin}: added endpoints to update user's and project's user_agent
Added backend (for now) implementation for updating user's and projects's user_agent using admin API.
Updating both user and project also updates bucket_metainfo and value_attribution tables.

Issue:
https://github.com/storj/storj-private/issues/297

Change-Id: I40244bbaa08b46834c1b1d0720e7d84d0c2a0330
2023-06-16 19:41:05 +00:00
Egon Elbre
edbea5efe1 go.mod: bump to pgx/v5
Change-Id: I31cf3bec1d7db94f0f612f6ed04b782f8b04d876
2023-06-14 18:32:54 +03:00
Márton Elek
981dd4cbdc go.mod: bump dependencies (storj.io/private, storj.io/uplink)
Change-Id: Ie5a6a14a39786dd745116c3b6e4fdbc8e890d2c3
2023-06-14 16:36:45 +03:00
Jeff Wendling
c559df6a1b all: bump storj.io/uplink
this fixes a memory leak in the upload codepath

Change-Id: I6bf6214d331cbe4e8e8912942035ef01a285e3b4
2023-06-08 19:52:24 +00:00
Márton Elek
974b4f938a go.mod: bump storj.io/private
Change-Id: I2d130cf2e794c8e5e0a978433cb3d504e6747af0
2023-06-08 12:01:55 +00:00
JT Olio
8c8340f1bc go.mod: bump eventkit
Change-Id: I2ab54bf640fc8a2aeafab77418c63bcd9cabd1aa
2023-06-07 15:35:24 +00:00
JT Olio
08a63fa5d1 go.mod: bump storj.io/common and storj.io/uplink
Change-Id: Ifc33429ab3e6f5b52b685c800b7c96dff6b904f2
2023-06-02 18:01:16 -04:00
JT Olio
36038af3d1 go.mod: bump storj.io/common
Change-Id: I18d4d491cdca99f6694a20f3ead5fba3f75bb658
2023-06-02 11:23:02 -04:00
Jeff Wendling
a0fbc87b31 cmd/uplink: add upload log file flag
Change-Id: I740eaf431cfe5bb3dcb746e4128dea5efc935257
2023-05-25 14:14:00 +00:00
Clement Sam
018b6eeeaf storagenode: add tests for lazyfilewalker
Updates https://github.com/storj/storj/issues/5349

Change-Id: I9544c14ba2acacd5b304f151ab29c70ff61adc5b
2023-05-08 21:50:40 +00:00
Jeff Wendling
6ac72e42cd all: bump storj.io/drpc
this includes a fix to cancellation when soft cancel
is enabled.

Change-Id: I1b9ab071f623153a2bd715888f97ca20277a76d7
2023-05-06 01:02:50 +00:00
Egon Elbre
8b82dba602 storagenode/blobstore/filestore: add tracking of blobs
We've had issues with forgetting to close readers and writers.
Add leak tracking to find those pesky issues.

Change-Id: If6b0ad6e9958318a7e0affee9c6d0a1ece412b6d
2023-05-05 15:40:15 +03:00
Jeff Wendling
80b3edf1d1 storagenode/piecestore: respect maximum chunk size requests
See https://review.dev.storj.io/c/storj/common/+/10297 for
more details.

Change-Id: Id5d19f029ae872780a554874592679191c1b5b2f
2023-04-28 16:31:41 -04:00
Márton Elek
1588bf9606 go.mod: bump monkit
Change-Id: Icc286378a8b01518510a4b4dd4d213d0c5d3b5b6
2023-04-25 14:26:11 +00:00
Egon Elbre
2405bc8f3b satellite/metabase: stop using the common error type
Updates https://github.com/storj/storj/issues/5291

Change-Id: I7b57a4b454d3619cb5d8ae4cd92f818ad2839c8b
2023-04-19 19:18:18 +03:00
Egon Elbre
d94207048a go.mod: bump vbauerster/mpb/v8 for fixes
Change-Id: I2d24eec4389ca8e5effdbfc25d7012ce083d46f5
2023-04-18 15:53:35 +03:00
Jeff Wendling
f52ea2744b testsuite/ui: bump gateway-mt and enable vet
go vet was disabled while gateway-mt was not yet
updated to build with some breaking changes in
common. now it's updated, so enable.

Change-Id: Idbb0ab0e56b5bd6545273dd3d41b47e7c79eb4b6
2023-04-17 22:07:57 +00:00
Jeff Wendling
df9cc4d10f all: bump storj.io/{common,uplink,drpc}
Change-Id: Ie98a7eacc11a492a20c76636f8044c81b7ed580d
2023-04-17 19:59:30 +00:00
Andrew Harding
e676b5c893 cmd/uplink: progress bars for recursive copy
```
$ uplink cp -r -t=3 files/ sj://files
uploading 6 files...
files/bar/buz   (2 of 6) 134.22 MB / 134.22 MB [============================================] 100.00% 36.43 MiB/s
files/bar/baz   (1 of 6) 67.11 MB / 67.11 MB [==============================================] 100.00% 18.39 MiB/s
files/boo       (3 of 6) 67.11 MB / 67.11 MB [==============================================] 100.00% 20.42 MiB/s
files/foo       (4 of 6) 67.11 MB / 67.11 MB [==============================================] 100.00% 57.83 MiB/s
files/glue/flew (5 of 6) 67.11 MB / 67.11 MB [==============================================] 100.00% 55.01 MiB/s
files/stew      (6 of 6) 67.11 MB / 67.11 MB [==============================================] 100.00% 91.43 MiB/s
```

Change-Id: Ibd9d07a1291f7a599bd27fba93c1b2e0f17dc787
2023-04-10 15:13:22 +00:00
Jeremy Wharton
b2a4a3543f testsuite/playwright-ui: add .gitignore
This change adds a .gitignore file to the testsuite/playwright-ui
directory. It is configured to ignore Node dependencies and test
results.

Change-Id: Ib200128ebe30064b1827fd71a5a7df163dcb4f69
2023-04-07 17:44:13 +00:00
Márton Elek
e5995947d0 go.mod: bump storj.io/private and storj.io/uplink
Change-Id: I01790c1af8d6302b3c9ffee0482c952bb9a5d8da
2023-04-06 13:44:34 +00:00
Michal Niewrzal
c4c391e154 satellite/accounting/live: replace address parsing with redis util
With this change we are replacing parsing code with existing go-redis
util.

We also switch redis client to version 9.

Change-Id: Ie4a651e3ae6960e68958c690873925d319b70e10
2023-04-05 13:20:11 +00:00
Clement Sam
5ab2e6ac19 go.mod: bump storj.io/private
Updates https://github.com/storj/storj/issues/5349

Change-Id: I5b91a8e9c28d39c6ec432992a352de08aff886d9
2023-04-05 14:46:59 +03:00
Michal Niewrzal
4c05293d8b satellite: set names for lrucache metrics
Fixes https://github.com/storj/storj/issues/5685

Change-Id: I9c19b7644b0cd3297448982b920c97abcdce9119
2023-04-04 11:55:20 +00:00
JT Olio
bd238819b6 go.mod: bump storj.io/drpc
this includes improved connection reset error handling

Change-Id: I211221493b5dd3f4b95d4db44bf022302352f050
2023-03-29 19:27:57 +00:00
Márton Elek
5e6d1cc00a
go.mod: bump storj.io/{common,private,uplink}
Change-Id: I7af875b6ac73af3a9d77928f69a05b80a5321696
2023-03-27 16:13:52 +02:00
JT Olio
0b4b04900a private/server: debounce noise and tls connections
to support TCP_FAST_OPEN, we're considering just using
two TCP connections in parallel per request, one with
and one without. this allows us to safely fire both
concurrently without stressing out the node too much.

see https://review.dev.storj.io/c/storj/storj/+/9933

Change-Id: I9aa8a0252350db5ace04ee125bfe469203e980ec
2023-03-21 16:51:31 +00:00
Kaloyan Raev
2c28a0e599 cmd/uplink: ability to opt-out from object key encryption
Option added to `uplink access setup` and `uplink access create`
commands to disable object key encryption.

Related to https://github.com/storj/storj/issues/5678

Change-Id: I4789a94143742ff4b232fd60decc029ad2883c2a
2023-03-20 13:56:15 +00:00
Antonio Franco (He/Him)
3613bfe1af
testsuite/playwright: copys files to storj repo (#5648)
* testsuite/playwright: copys files to storj repo
This PR copies all the playwright ui tests to the storj repository.

* CI: Jenkinsfile.ui
Jenkinsfile.ui builds all necessary items for running storj binaries, installs all items for playwright ui tests and executes the test. Note that we aren't running regular unit tests and we are not linting the code as this is occuring in the public instance. Furthermore, we still need to implement the starting of storj services as well as typescript linters for the playwright tests.

* testsuite: playwright ui tests
This PR copies all playwright ui tests files from the qa repo.

* testsuite: playwright ui tests
This PR copies all playwright ui tests files from the qa repo.
2023-03-09 11:59:45 -05:00
Vitalii
a01d47b3ed satellite:{console, web}: remove old project dashboard
Removed old project dashboard along with a feature flag.

Change-Id: I9292b19caf101a84042d8c3006c27bc44db8d6b5
2023-03-07 13:34:59 +02:00
JT Olio
8befcaab27 go.mod: bump storj.io/{common,drpc,uplink}
this set of changes re-enables noise with fixes

Change-Id: I122142d202dd76a83b678ac8d709407b0791ed0b
2023-02-24 23:17:16 +00:00
Michal Niewrzal
16b7901fde satellite/metabase: add piece size calculation to segment
This code is essentially replacement for eestream.CalcPieceSize. To call
eestream.CalcPieceSize we need eestream.RedundancyStrategy which is not
trivial to get as it requires infectious.FEC. For example infectious.FEC
creation is visible on GE loop observer CPU profile because we were
doing this for each segment in DB.

New method was added to storj.Redundancy and here we are just wiring it
with metabase Segment.

BenchmarkSegmentPieceSize
BenchmarkSegmentPieceSize/eestream.CalcPieceSize
BenchmarkSegmentPieceSize/eestream.CalcPieceSize-8         	    5822	    189189 ns/op	    9776 B/op	       8 allocs/op
BenchmarkSegmentPieceSize/segment.PieceSize
BenchmarkSegmentPieceSize/segment.PieceSize-8              	94721329	        11.49 ns/op	       0 B/op	       0 allocs/op

Change-Id: I5a8b4237aedd1424c54ed0af448061a236b00295
2023-02-22 11:04:02 +00:00
JT Olio
f905878ff9 go.mod: bump storj.io/uplink
this change brings in:

 * private/piecestore: disable noise
 * uplink: fix noise version report
 * private/piecestore: fix noise
 * private/metaclient: honor context cancellation on retries
 * private/metaclient: fix multi-passphrase ListObjects paging
 * private/metaclient: fix broken ListUploads method

it also adds:

 * private/storage/streams/batchaggregator: introduce Aggregator
 * private/storage/streams/streambatcher: introduce Batcher
 * private/storage/streams: pieceupload package
 * private/storage/streams: segmenttracker package

but these commits provably change no existing behavior, since they
only add unused, new packages and types.

Change-Id: Id9efc65117b7e1ddcb3ffc8cc15d3e665ed11d7d
2023-02-21 16:09:19 +00:00
Egon Elbre
0cdef95d55 all: fix math/rand deprecations
Change-Id: I4b966375697c0d409ce24cc7604f806973f8f22a
2023-02-17 15:05:54 +02:00
Cameron
dd0013bdda bump stripe-go version to v72.90.0
v72.90.0 grants access to InvoiceParams.PendingInvoiceItemsBehavior
which allows the caller to specify whether the invoice should apply all
pending invoice items.

Change-Id: I7c8295d38791ed7daa3d5b1fce15438eb42a962f
2023-02-15 21:16:56 +00:00
JT Olio
ca13eca718 go.mod: bump libuplink to include noise
Change-Id: If5bceb139ce6fdf6c0792b4bb536bc61b54e32bb
2023-02-10 17:03:51 +00:00
JT Olio
522aed083d private/server,satellite/contact,misc: use new storj/common noise helpers
this change uses the new storj/common noise helpers, which:
 * add a security fix (require an expected node id for validating
   noise key attestations)
 * stops doing an unnecessary order signature validation (it's
   already been done inside of PutPiece)
 * removes some duplicate code

Change-Id: I5e67a08ff216cd9c5b0b82e40b4d9de664b6b0fc
2023-02-07 09:53:45 -05:00
Egon Elbre
873a202530 mod: bump storj.io/common
This bumps common, such that things build with Go 1.20.

Also, adds `go vet` checks for testsuite/storjscan and testsuite/ui.

The latest golang.org/x/bcrypt has a check that the new password is less
than 72 bytes, because bcrypt silently discarded them. This means our
own password validation has the same limitation. Old passwords should
still work fine.

Change-Id: Ibb8735b15eeb91460145906b81ae4e365e9ac418
2023-02-03 16:49:41 +02:00
Egon Elbre
3976a2fd1d go.mod: bump common to latest
Temporarily allow larger wasm size, while we migrate to picobuf.

Change-Id: I55916f47486465e4223cbd79f412ab2876fee042
2023-02-01 12:33:37 +02:00
JT Olio
3e01e9c07a storj/private/server: listen for noiseconn requests
Change-Id: Ia74dcc576fa0c97460207d93d129aa7d88cd2fba
2023-01-30 13:23:08 -05:00
Michal Niewrzal
bd8867cd09 satellite: adjust code to handle context cancelation for SQL queries
Our DB support in storj/private was updated to enable basic context
support for executing SQL queries. This change requires some small
adjustments as not all parts were working correctly.

storj/private commit with change:
4bc77107b7acfcc2f7ad65796d5dd3d7c64801e4

Change-Id: I64d7ed92788ea0920d12cecd1aa0e414720e9b9c
2023-01-27 10:07:43 +01:00
Kaloyan Raev
674068a5f2 mod: bump go-redis to v8.11.5
This removes the indirect dependency to go.opentelemetry.io/otel, which
causes a dependency conflict in storj.io/ipfs-go-ds-storj.

Change-Id: I28c1e3637bb23f7949988511051d39c08b3fc852
2023-01-26 12:29:03 +00:00
Kaloyan Raev
805fcdf341 Revert "mod: bump go-redis to v8.11.5"
This reverts commit b2b564cc13.

Change-Id: I79a6b92758d1095113c85d49d673b243c31e6c82
2023-01-26 13:08:18 +02:00
Kaloyan Raev
b2b564cc13 mod: bump go-redis to v8.11.5
This removes the indirect dependency to go.opentelemetry.io/otel, which
causes a dependency conflict in storj.io/ipfs-go-ds-storj.

Change-Id: I28c1e3637bb23f7949988511051d39c08b3fc852
2023-01-26 13:07:13 +02:00
Andrew Harding
e7b35381f2 mod: bump common and uplink deps
Change-Id: Ie2d87f517b57a8760f50ac5c9f1b8b0448f5b3b5
2023-01-25 13:34:47 -07:00
JT Olio
e40191afd6 storj: upgrade to use latest storj/common NodeAddress
Change-Id: I5987391bcfe5f6dfd7b525698c337a4cbda9b76e
2023-01-25 01:37:26 +00:00
Egon Elbre
9544a670d7 storagenode/pieces: fix concurrent empty and restore trash
This ensures that empty trash and restore trash cannot run at the same
time.

Fixes https://github.com/storj/storj/issues/5416

Change-Id: I9d2e3aa3d66e61e5c8a7427a95208bb96089792d
2023-01-03 15:01:54 +00:00
Michal Niewrzal
5110803102 storagenode/piecestore: add Exists endpoint
Adds new method Exists which can be used to verify which
requested piece ids exists on storage node. Will verify only pieces
which belongs to the satellite that used that endpoint.

Minum WASM size was increased a bit.

https://github.com/storj/storj/issues/5415

Change-Id: Ia5f9cadeb526541b2776a8973eb7d50133ad8636
2022-12-17 04:08:26 +00:00
Márton Elek
ef2c0f4225 mod: use v1.10.0 tagged commit from storj.io/uplink
Change-Id: I475f2cf6447453dd78eb3549fdaacd286e1da494
2022-12-13 12:04:29 +00:00
Márton Elek
27191d5b80 mod: bump storj.io/uplink
Change-Id: I8548f6bfafb86646bbc71b040190fad28579d92e
2022-12-07 14:07:18 +00:00