Commit Graph

7588 Commits

Author SHA1 Message Date
Wilfred Asomani
e751ee3c1b web/satellite: dismiss access grant modal tooltip
This change fixes an issue where the tooltip on the create access grant
modal will not dismiss even after hovering/clicking outside of it.

Issue: https://github.com/storj/storj/issues/5502

Change-Id: Ib6c042f3b18566bfeb4b05cb50acfa5e5ca1fbb9
2023-02-01 18:36:55 +00:00
Moby von Briesen
b4dbfa0cf8 web/satellite: Fix permission setting for access grants modal
Every case for the check/uncheck logic was not covered to actually
update the permissions list that gets sent to the worker to create the
access. This commit fixes these cases.

Solves https://github.com/storj/storj/issues/5512

Change-Id: I15f30f9f989039df110930b5a6e8914f5bf9d91e
2023-02-01 17:52:20 +00:00
Egon Elbre
6e298cca90 cmd/tools/node-cleanup: include tz information in test
Change-Id: Ic31b585b7df3fcb65fa37055ee62888fa43fa418
2023-02-01 17:21:38 +00:00
Jeremy Wharton
897de167a6 satellite/console,web/satellite: remove user input from welcome and invite emails
Emails should not contain user input that could be used by malicious
agents to deliver a message. Usernames have been removed from
account activation emails, and project names have been removed from
project invitation emails.

References storj-private#133

Change-Id: Ic05921149b409145df109c0966ea5dfd86d86eb1
2023-02-01 16:29:57 +00:00
Michal Niewrzal
15508d270c satellite/orders: don't store non user bandwidth actions for bucket
For bucket_bandwidth_rollups we are trying to insert lots of entries
with empty bucket name and project id. Those are inserts from orders
created by repair, audit and GE. High load on the same primary key
(the same range) is causing many retries and that's affect all inserts
as we are putting 1000 entries into DB a once.

This change solves this problem by not storing into
bucket_bandwidth_rollups other actions then GET and PUT. Those actions
are only important from bucket bandwidth usage perspective because
those are actions performed by users. Other actions (repair, audit or
GE) are also stored in storagenode_bandwdith_rollups so we will still
have access to them e.g. for statistic purposes.

https://github.com/storj/storj/issues/5332

Change-Id: Ibb5bf0a4c869b0439dc65da1c9342a38ca2890ba
2023-02-01 15:38:48 +00:00
Vitalii
7dfa379d77 web/satellite: show prompt for project passphrase on project create/switch
Show modal to enter project passphrase when switching between projects or creating new one.
Also fixed a bug when create project passphrase modal is shown twice after onboarding 'continue in web' click.

Change-Id: Iab64cf9f457b9739314610612608a5d8070810e3
2023-02-01 16:41:46 +02:00
Michal Niewrzal
3b6e1123b8 satellite/orders: fix sorting rollups before inserting
Sorting by primary key before inserting data into DB is fixed.
Earlier we were sorting input slice of BucketBandwidthRollup but then
we were putting all entries into map to rollup input data. Iteration
over map with a range loop doesn't guarantee any specific order so we
were loosing sorted order when we were creating with this map slices to
use with DB insert.

New code is also using map but when map is full its sorting map keys
separately and iterates over them to get data from map.

https://github.com/storj/storj/issues/5332

Change-Id: I5bf09489b0eecb6858bf854ab387b660124bf53f
2023-02-01 12:17:25 +00:00
Egon Elbre
9a6be7f1e7 private/server: avoid relying on timing
The previous test relied on timing, but instead let's try dialing
the server and see whether we can do something with the connection.

We probably should test all the supported protocols instead of just tcp.

Change-Id: I9217494859faea0a7b93515aad706da4fdd8a140
2023-02-01 12:33:56 +02:00
Egon Elbre
3976a2fd1d go.mod: bump common to latest
Temporarily allow larger wasm size, while we migrate to picobuf.

Change-Id: I55916f47486465e4223cbd79f412ab2876fee042
2023-02-01 12:33:37 +02:00
Egon Elbre
b8c7dcbf7b certificate: improve gob migration
`storage.KeyValueStore` requires ordered iteration, which redis
doesn't support natively. This would require loading all the keys
into memory and then processing them, rather than iterating over them
one-by-one.

This adds a temporary `IterateUnordered` to handle the migrations
more gracefully.

Change-Id: I55b763500523077c7ab8fdfad175c32cc7788e47
2023-02-01 10:21:48 +00:00
Vitalii
fa26ae85e9 web/satellite: create bucket without a passphrase feature
Allow user to create bucket without a passphrase if project level passphrase was not set.

Issue:
https://github.com/storj/storj/issues/5521

Change-Id: Ifc4a6724229ce0708db720edb2f8953098e346ed
2023-02-01 08:53:39 +00:00
JT Olio
ae9ea22193 storagenode/piecestore: return node certificate chain at upload conclusion
uplinks currently get the node's certificate chain over TLS. once Noise
is in use, uplinks will no longer be able to do this. we should start
having the upload request return the certificate chain in the same
release that starts supporting noise.

Change-Id: I619b23cb8e25691bcc62d760f884403a4ccd64a0
2023-02-01 01:49:50 +00:00
Jeremy Wharton
33bd929308 web/satellite: encode file browser URL components
File paths may contain characters that have special meaning when placed
in URLs. For example, a folder name may contain a pound symbol (#)
which is the URL fragment delimiter. This causes the characters that
succeed this symbol to not be considered part of the primary resource
identifier as they should be, resulting in navigation errors.
This change resolves this issue.

Resolves #5522

Change-Id: I59972321795375ec5981c3e9c505e35f998022d6
2023-02-01 00:08:54 +00:00
Vitalii
5377b9c314 satellite/{web, console}: removed old project level passphrase flow
Removed old behavior and bucket creation flow.

Issue:
https://github.com/storj/storj/issues/5521

Change-Id: I1a443911215cd1f611069fd9e6202598066b80b2
2023-01-31 23:02:44 +00:00
JT Olio
382af95499 storagenode/contact: send noise key and settings as contact info
Change-Id: I1e7a83de36d5cf16eed8874091b15af1e0b73df7
2023-01-31 21:49:20 +00:00
paul cannon
b6bcb32ecf satellite/reputation: more accurate "reputation changes" list
`overlay.(*Service).UpdateReputation()` takes a "reputationChanges"
parameter, a slice of node events indicating whether we think the node's
disqualification or suspension status is changing. This is necessary so
that the overlay service can notify the nodeevents DB about these
changes.

In several cases, however, this list of events is not constructed
correctly, because of missing information about the previous state.
In most cases, this is because the node was offline, and the order limit
creation functions (which usually obtain and return the prior reputation
status) ignored that node.

This change makes it so that all callers to
`overlay.(*Service).UpdateReputation()` can be expected to provide a
correct list of change events (as correct as feasible, given that we
can't lock the node's information in the database during the entire
operation).

It ended up that there was only one caller we needed to worry about, and
that was reputation.(*Service).ApplyAudit(). So the bulk of this change
is teaching that function how to recognize when the prior reputation
status was not filled in, and fill it in.

Refs: https://github.com/storj/storj/issues/5464
Change-Id: I52ce385fc9c0ce3b283b998d517998e7f4ec8792
2023-01-31 18:39:40 +00:00
JT Olio
c17ceef093 private/server: tcp_fastopen failures should not kill the server
Change-Id: Iec42524f83619c22d8be26f373978e8abd468ee8
2023-01-31 14:48:14 +00:00
Egon Elbre
053beef8c4 private/server: fix fastopen socket option
Change-Id: I7acbe0562ebfe9d78dae5c57f6057b8364dfa0e7
2023-01-31 14:48:08 +00:00
Egon Elbre
9bb9275acf private/testplanet: add some more error wrapping
Change-Id: I4cbdaafc0f6827e569d9b70acac3ce448359f125
2023-01-31 15:44:59 +02:00
Wilfred Asomani
3a714cefcf satellite: remove rewards package
Affected packages admin,attribution,console,metainfo,satellitedb,web,payments
This change removes the satellite/rewards package and its related usages.
It removes references to APIKeyInfo/PartnerID, Project/PartnerID
 and User/PartnerID.

Issue: https://github.com/storj/storj/issues/5432

Change-Id: Ieaa352ee848db45e94f85556febdbcf1444d8c3e
2023-01-31 11:46:50 +00:00
JT Olio
572c5b305b satellite/context: fix pingme test
Change-Id: I30fcfe1dea43d65e457c76dd44f475b5c197cbee
2023-01-30 15:01:32 -05:00
JT Olio
2a641b60d3 private/server: support tcp fastopen
we are not enabling it on client-side code yet, but
it will be hard to test this in the wild without
server-side support.

this is phase 2 of tcp fast open support. see
https://forum.storj.io/t/two-new-blueprints-design-drafts-seeking-feedback-replacing-tls-with-noise-and-tcp-fastopen/21053/12
for more details

Change-Id: I20362be4c49abab90afcc9b6572ef9f79816345b
2023-01-30 13:30:43 -05:00
JT Olio
3e01e9c07a storj/private/server: listen for noiseconn requests
Change-Id: Ia74dcc576fa0c97460207d93d129aa7d88cd2fba
2023-01-30 13:23:08 -05:00
Wilfred Asomani
73ffa0827f cmd/tools: remove nullify-bad-user-agents tool
This tool is being removed because it has served its purpose and was blocking another removal from being verified.

Change-Id: Ie888aa7ae1b153a34210af3a5d5a3682b381ba82
2023-01-30 15:57:33 +00:00
paul cannon
2f04e20627 storage/filestore: better error message on data corruption
A user on the forum was seeing the error "bad message", which was not
very helpful. This case from the ext4 filesystem using the code EBADMSG
to indicate it detected an invalid CRC, suggesting disk corruption.

This change adds some explanatory information about probable disk
corruption to all errors coming from the (*blobInfo).Stat() call, which
is where storagenode fs corruption problems will usually manifest.

Refs: https://github.com/storj/storj/issues/5375
Change-Id: I87f4a800236050415c4191ef1a0fc952f9def315
2023-01-30 08:54:06 -06:00
paul cannon
ed7c82439d storage/filestore: avoid stat() during walkNamespaceInPath
Calling stat() (really, lstat()) on every file during a directory walk
is the step that takes up the most time. Furthermore, not all directory
walk uses _need_ to have a stat done on every file. Therefore, in this
commit we avoid doing the stat at the lowest level of
walkNamespaceInPath. The stat will still be done when it is requested,
with the Stat() method on the blobInfo object.

The major upside of this is that we can avoid the stat call on most
files during a Retain operation. This should speed up garbage collection
considerably.

The major downside is that walkNamespaceInPath will no longer
automatically skip over directories that are named like blob files, or
blob files which are deleted between readdir() and stat(). Callers to
walkNamespaceInPath and its variants (WalkNamespace,
WalkSatellitePieces, etc) are now expected to handle these cases
individually.

Thanks to forum member Toyoo for the insight that this would speed up
garbage collection.

Refs: https://github.com/storj/storj/issues/5454
Change-Id: I72930573d58928fa25057ed89cd4ec474b884199
2023-01-30 13:47:03 +00:00
NickolaiYurchenko
facbd65882 web/satellite: EstimatedCostsAndCredits components migrated to use composition api
Change-Id: I11ee04880a5dbdc5494f7a51c0b4077e88827a11
2023-01-30 09:22:25 +00:00
Lizzy Thomson
1bff41e6b3 cmd/tools: add migration tool to update salt column when null
Add migration tool (and test) to update salt column in projects table
with the SHA-256 hash of the project ID when null

Issue https://github.com/storj/storj-private/issues/66

Change-Id: Ib8d484ac8d6ee25859064d803e2ac8fb46b45921
2023-01-27 22:04:07 +00:00
Andrew Harding
abd0ad92dc satellite/metainfo: RetryBeginSegmentPieces RPC implementation
Part of:
https://github.com/storj/uplink/issues/120

Change-Id: I2a2873455f7498ffd31f50ade16c173fe1d18157
2023-01-27 15:04:59 +00:00
Michal Niewrzal
bd8867cd09 satellite: adjust code to handle context cancelation for SQL queries
Our DB support in storj/private was updated to enable basic context
support for executing SQL queries. This change requires some small
adjustments as not all parts were working correctly.

storj/private commit with change:
4bc77107b7acfcc2f7ad65796d5dd3d7c64801e4

Change-Id: I64d7ed92788ea0920d12cecd1aa0e414720e9b9c
2023-01-27 10:07:43 +01:00
Vitalii
6f11c8b32c web/satellite: show create/enter passphrase modal after login
Show create/enter passphrase modal after login for new project level passphrase flow.
Also fixed buckets view mounted hook to load create bucket modal instead of old flow.

Issue:
https://github.com/storj/storj/issues/5510

Change-Id: If9ea70faaa2987f336d72d55a6ed2bbd02ced592
2023-01-26 18:06:59 +02:00
Egon Elbre
e4b325537e satellite/satellitedb: missing rows.Close in error case
Change-Id: Ib1cae87171c302653771507cdc31d0c11a68f504
2023-01-26 15:21:48 +00:00
Clement Sam
6277d93c82 cmd/storagenode: remove storagenode dockerfiles
Updates https://github.com/storj/storagenode-docker/issues/1

Change-Id: I6b2f42194cb0e32f1b3978072fd6e1cdec753704
2023-01-26 13:49:12 +00:00
Kaloyan Raev
674068a5f2 mod: bump go-redis to v8.11.5
This removes the indirect dependency to go.opentelemetry.io/otel, which
causes a dependency conflict in storj.io/ipfs-go-ds-storj.

Change-Id: I28c1e3637bb23f7949988511051d39c08b3fc852
2023-01-26 12:29:03 +00:00
Michal Niewrzal
b5c5c62d7b satellite/metabase: add missing error check
Change-Id: I6891f0647cb8e4a8dd6534eaa3588bbe76e2721d
2023-01-26 11:18:14 +00:00
Kaloyan Raev
805fcdf341 Revert "mod: bump go-redis to v8.11.5"
This reverts commit b2b564cc13.

Change-Id: I79a6b92758d1095113c85d49d673b243c31e6c82
2023-01-26 13:08:18 +02:00
Kaloyan Raev
b2b564cc13 mod: bump go-redis to v8.11.5
This removes the indirect dependency to go.opentelemetry.io/otel, which
causes a dependency conflict in storj.io/ipfs-go-ds-storj.

Change-Id: I28c1e3637bb23f7949988511051d39c08b3fc852
2023-01-26 13:07:13 +02:00
Michal Niewrzal
bb2ac4279a satellite/metainfo: enable multiple versions fix by default
Change-Id: I6cc7ba928e59cc8b8fa50f2ab19ec5418dc76507
2023-01-26 09:35:20 +00:00
Moby von Briesen
7c152f7ea0 satellite/console: Filter new characters out of user input
HTML and JS escape user input for create and update user.

Change-Id: I91d972f454341a5a7f333d006a87c6f854595490
2023-01-25 22:52:37 +00:00
NickolaiYurchenko
a53849f874 web/satellite: PeriodSelection, SmallDepositHistory, SortingHeader migrated to use composition api
Change-Id: I029b8daa8268aae1fd9a12ab83a368f5e6862ba7
2023-01-25 22:17:52 +00:00
NickolaiYurchenko
15efa1e319 web/satellite: CouponArea, DetailedHistory, PaymentHistoryItemDate, PaymentsItem migrated to use composition api
Change-Id: I3963835053e69dd3b989a417b295abaa0b645284
2023-01-25 21:47:32 +00:00
Andrew Harding
e7b35381f2 mod: bump common and uplink deps
Change-Id: Ie2d87f517b57a8760f50ac5c9f1b8b0448f5b3b5
2023-01-25 13:34:47 -07:00
Egon Elbre
d0686648db certificate/authorization: implement gob to pb migration
Change-Id: I6f33f5802e3f0a3c8a5f0c3cff61ef836a645c41
2023-01-25 11:09:05 +02:00
Egon Elbre
10c552fec4 certificate/authorization: add encode as pb
gob package is not stable across Go version, let's switch to protobuf
for encoding these. We still need backwards compatibility for the
moment.

Change-Id: If1da50658ab39a75d1b2b1f988356b56347cac14
2023-01-25 10:28:36 +02:00
Egon Elbre
e9bc06608e certificate/certificatepb: add definitions for migration
Also fixing the timestamp in .proto which was actually int64.

Change-Id: I65e3172793a49bf8bfd946b199016204ed72f221
2023-01-25 10:28:36 +02:00
JT Olio
e40191afd6 storj: upgrade to use latest storj/common NodeAddress
Change-Id: I5987391bcfe5f6dfd7b525698c337a4cbda9b76e
2023-01-25 01:37:26 +00:00
Vitalii
c98ef89311 web/satellite: fixed multiple passphrases notification for object browser
Fixed notifications link to toggle manage passphrase modal instead of redirecting to buckets screen.

Issue:
https://github.com/storj/storj/issues/5452

Change-Id: I84ce67da7445c6316eb0658606c0129bb4af680e
2023-01-24 16:21:35 +00:00
Erik van Velzen
eab595397f cmd/uplink: add --analytics flag
Add a flag to enable/disable analytics so uplink can be run
non-interactively. Also when run non-interactively for the first time
it will not error any more but instead default to disable analytics.

Part of https://github.com/storj/storj/issues/5126

Change-Id: I07ac8a040664334efcb4e2536f26c330c1751a6f
2023-01-24 15:33:55 +00:00
Yaroslav Vorobiov
3639c5ee1e satellite/payments/stripecoinpayments: skip paying invoices with due date
When running pay invoices command skip invoices that has due date set, so
customers can still pay it themselves until due date.

https://github.com/storj/storj/issues/5453

Change-Id: I8e557062491ab0c8246b28bc5ca57e845eb32e29
2023-01-24 15:12:29 +01:00
Clement Sam
95960572b3 storagenode/piecestore: improve logs for incoming requests
- Adds "Remote Address" field to all INFO logs related to GET,
PUT, and DELETE requests
- Adds Offset and Size fields to all info logs related to GET
requests

Resolves https://github.com/storj/storj/issues/5404

Change-Id: I5dab1867619385362e5f1e0455dfab17d295a37a
2023-01-24 10:23:34 +00:00