Commit Graph

3218 Commits

Author SHA1 Message Date
Moby von Briesen
19cb08b025 web/satellite: add "public linksharing URL"
* Update defaults for gateway credentials URL and linksharing URL to use
  storjsatelliteshare.io instead of storjshare.io
* Add new config for "public linksharing URL" and set it to
  link.storjshare.io
* Use "private" linksharing URL for actions within the object browser
* Use "public" linksharing URL for sharing files externally

Resolves https://github.com/storj/storj/issues/5805

Change-Id: I2c8fbd04141755b4751dcf4d054253a7ff8d6cf3
2023-05-18 07:02:26 +00:00
Vitalii
ace0ef89a2 web/satellite: migrated Vue 2 to Vue 3
Finally migrated web/satellite UI to use Vue 3.

Issue:
https://github.com/storj/storj/issues/5862

Change-Id: I609e2332142b9092d6efeb2d4b931e34cce846cc
2023-05-17 17:31:03 +00:00
paul cannon
958d8676d0 satellite/overlay: remove unnecessary test helper
Change-Id: I8439eec4ed440f60353fc620ca906a917a03613c
2023-05-17 17:04:54 +00:00
paul cannon
1f4f79b6b3 satellite/repair: don't mark clumped segments as irreparable
Clumped segments (segments with multiple pieces on the same subnet) may
need repair, but the clumped pieces are considered retrievable and we
don't need to call such segments irreparable.

We do want to know where they're coming from, though, if we can, because
we are seeing more than expected.

Change-Id: I41863b243f4bb007ef8929191a3fde1562565ef9
2023-05-17 16:24:15 +00:00
Jeremy Wharton
607b120116 satellite/satellitedb: add inviter ID column to project invites table
The project member invitations table has been modified to contain a
column for the ID of the user who sent the invitation. This ID is
required for us to return information about the inviter to the
satellite frontend.

References #5855

Change-Id: I928d987a8db2340f731ca65ce30173d4f90a9837
2023-05-17 15:46:40 +00:00
Lizzy Thomson
f87772a0d3 satellite/console: bug fix for using a API key to retrieve a session
Change-Id: I0202f75fb78f42af9e7cb00bb3ac73839340c977
2023-05-16 14:06:32 -06:00
paul cannon
75d10fe4fa satellite/overlay: use UploadSelectionCache for GetNodesNetworkInOrder
The query for GetNodesNetworkInOrder is causing far too much load on the
database. Since it is not critical that the repair checker have
perfectly up-to-date node network information, we can use a cache
instead.

Change-Id: I07ad45bfdeb46529da093941a06c2da8a00ce878
2023-05-16 17:32:09 +00:00
Márton Elek
f1b9f6b672 satellite/satellitedb: paginating on ordered invoice
87d0789691 replaces offset usage with cursors.

But to continue the interation from a specific cursor, we need to iterate over ordered records.

(at least this is what I understood based on the failing tests)

87d0789691

Change-Id: Ic4da3a7c5f03386dd4c373c05102f05871900a3a
2023-05-16 14:05:34 +00:00
Michal Niewrzal
4bdbb25d83 satellite/metabase/rangedloop: move Segment definition
We will remove segments loop soon so we need first to move
Segment definition to rangedloop package.

https://github.com/storj/storj/issues/5237

Change-Id: Ibe6aad316ffb7073cc4de166f1f17b87aac07363
2023-05-16 12:37:17 +00:00
Lizzy Thomson
7e69b22dd4 satellite/console: retrieve a web UI session using a API key
retrieve a web UI session using a API key and add new test

Issue https://github.com/storj/storj/issues/5688

Change-Id: I0ad9c783f0573b87b212b8fb065cdbf7074b782c
2023-05-15 10:24:58 -06:00
Wilfred Asomani
81b2b067e6 satellite/satellitedb/dbx/gen: update to work with updated dbx
This change updates the replacer in satellite/satellitedb/dbx/gen/main.go
to work with an updated dbx.

Change-Id: I08e89d6d27e6f1d435416105fe5f622009add7ad
2023-05-15 14:53:06 +00:00
Michal Niewrzal
2592aaef9c satellite/gc/bloomfilter: remove segments loop parts
We are switching completely to ranged loop.

https://github.com/storj/storj/issues/5368

Change-Id: I1a22ac4b242998e287b2b7d8167b64e850b61a0f
2023-05-15 11:46:26 +00:00
paul cannon
c2710cc78d satellite/audit: improve error handling
* Don't use rpcstatus.Unknown as an indicator of dial failure; instead,
  GetShare now indicates with a per-share field where a failure happened
  (DialFailure, RequestFailure, NoFailure). Use that information in
  Verify() to determine how to treat the source node.

* Add a test that replaces a storage node with a black hole, so that
  connections there will always time out. Make sure we handle that case
  correctly.

Refs: https://github.com/storj/storj/issues/5632
Change-Id: I513a53520fb48b7187d4c4d7e14e01e2cfc0a721
2023-05-11 22:55:26 +00:00
igor gaidaienko
bc30deee11 satellite/repair: update repair test to use blake3 algo
Update repair test to also use blake3 hashing algorithm
https://github.com/storj/storj/issues/5649

Change-Id: Id8299576f8be4cfd84ddf9a6b852e653628ada72
2023-05-11 08:45:48 +00:00
Michal Niewrzal
87d0789691 satellite/payments/stripe: avoid full table scan while listing records
Stripe invoice project records while listing are causing full table scan
because of OFFSET caluse. This change is refactoring query to list using
cursor.

Change-Id: I6b73b9b2815173d7ef02cf615408778476eb3b7b
2023-05-09 14:16:55 +00:00
Michal Niewrzal
fa16d947c9 satellite/satellitedb: add index on owner_id column for projects table
We have method which is getting projects owned by specific user but it's
causing full table scan because we don't have index on owner_id column.

Change-Id: Icb71c9ac5b73104a52241ed8ba126c995c10811f
2023-05-09 08:57:32 +00:00
Wilfred Asomani
5145be1eeb satellite/console: proper constraint error check for duplicate project members
The string check previously used to check for constraint errors is now
replaced with dbx.IsConstraintError check.

Change-Id: I553ccd69e3c02b6b54441bd9f929b85a155eaf00
2023-05-08 22:23:00 +00:00
dlamarmorgan
cf5d2d792c satellite/payments/stripe/service.go: fix payment for multiple invoices
Fix an error that can occur when processing multiple invoices for the same user in a single invoice cycle when the user is paying with Storj tokens.

Change-Id: I54af8c7dde1965d994f687fdfc4e4b5ef4deeb2d
2023-05-08 21:17:14 +00:00
Egon Elbre
8beb78ec3f {satellite,multinode}/console: fix WriteHeader
w.Header().Set needs to be called before WriteHeader,
because WriteHeader sends all the headers and calls to
Set won't have any effect afterwards.

Change-Id: Ia6b1c5e2cd54201a6c3980d63de04a0095b2db9a
2023-05-08 18:18:30 +00:00
Jeremy Wharton
9c753163c2 satellite/console/dbcleanup: make chore clean up project invites
The console DB cleanup chore has been extended to remove old project
member invitation records.

Resolves #5816

Change-Id: Id0a748e40f5acf03b9b903265c653b072846ba19
2023-05-08 16:36:16 +00:00
Michal Niewrzal
36e046375c satellite/repair/checker: remove segments loop parts
We are switching completely to ranged loop.

https://github.com/storj/storj/issues/5368

Change-Id: I8583549973cd36aa0e0c482c20d7a75cb7568ab3
2023-05-08 12:19:13 +00:00
Moby von Briesen
432b12d1c4 satellite/satellitedb: make value_attributions.partner_id nullable
There is still a reference to partner_id in a query, which we cannot
move until dropping the "not null" constraint for it. This change adds
that migration.

Related to https://github.com/storj/storj/issues/5432

Change-Id: I98802a6e1bd59f3d9214de3db6688d9daf664a70
2023-05-05 23:42:50 +00:00
Jeremy Wharton
f61230a670 satellite/console/dbcleanup: create console DB cleanup chore
A chore responsible for purging data from the console DB has been
implemented. Currently, it removes old records for unverified user
accounts. We plan to extend this functionality to include expired
project member invitations in the future.

Resolves #5790
References #5816

Change-Id: I1f3ef62fc96c10a42a383804b3b1d2846d7813f7
2023-05-05 19:11:53 +00:00
Wilfred Asomani
4ab433ba89 satellite/{web,console}: make duplicate member error readable
This change makes the error thrown when adding an existing member to a
project readable.

Issue: https://github.com/storj/storj/issues/5840

Change-Id: I4269495f9b7b09c77fbf1af1fc605e5c95bd7cbf
2023-05-05 16:46:12 +00:00
Wilfred Asomani
bf05040dd6 satellite/{console,db}: allow passphrasepromt toggling
This change adds the user's passphrase prompt setting to the
/account/settings endpoints.

Issue: https://github.com/storj/storj/issues/5616

Change-Id: I48d470d49e82096fd090b74da323b279e342546e
2023-05-04 19:26:54 +00:00
Paul Willoughby
d8081169cd satellite/metainfo: increase default MaxEncryptedObjectKeyLength
Allow a longer encrypted key length to reduce 'key length is too big'
errors in gateway-mt.  Gateway is enforcing an unencrypted key length
of 1024 bytes but when encrypted some keys are exceeding the current
limit.

updates https://github.com/storj/gateway-mt/issues/335

Change-Id: I861a2313e558c9f7d39569d21c7a3d429c83575c
2023-05-04 12:37:13 -06:00
Egon Elbre
511d74a76d satellite/metainfo: hardcode the prefix
This avoids a potential bug with regards to not returning the prefix
that the uplink expects.

Change-Id: I64683db57c5b22989db91b4f8e08530bf3b64ff1
2023-05-04 17:06:51 +03:00
Moby von Briesen
a229feac1e satellite/console: Pass pricing packages enabled config to frontend
Ensure that the value of "pricing packages enabled" flag on frontend is
the same as what is configured on the backend.

Change-Id: Id78771800a4973ebd3ad4e22f1953f6f71c75dd4
2023-05-03 18:13:38 -04:00
Egon Elbre
ee28103b29 satellite/metabase: fix error message
The "object not found" included an additional prefix "metabase:", which
broke uplink error message detection.

Ideally, changing an internal error message shouldn't break the uplink.

Change-Id: I5ce7789cc11742d3435af1ec555bc96927f1bedc
2023-05-02 13:53:57 +00:00
Michal Niewrzal
98562d06c8 satellite/gc/bloomfilter: add sync observer
Current observer used with ranged loop is using massive amount of
memory because each range is generating separate set of bloom filters.
Each bloom filter can be up to 2MB of memory. That's a lot.

This change is initial change to reduce used memory by sharing bloom
filters between ranges and just synchronize access to them. This
implementation is rather simple and even naive but maybe it will be
enough without doing something more complex.

https://github.com/storj/storj/issues/5803

Change-Id: Ie62d19276aa9023076b1c97f712b788bce963cbe
2023-04-28 07:40:56 +00:00
dlamarmorgan
b89e0e0272 payments/stripe: Remove apply coupon from generate invoice command
Remove the command to apply free tier coupons from the generate invoices
command. Applying free tier coupons should instead be done outside the
normal time window for invoice generation to save time during the
invoicing cycle.

Change-Id: If8fecc558411d5a6fff9d5689143d72f3b709e55
2023-04-27 08:24:12 -07:00
Michal Niewrzal
3cd79d987d satellite/gc/bloomfilter: extract BF upload logic
This is refactor/cleanup change before I will start working on adding
separate GC observer with optimized memory consumption.

https://github.com/storj/storj/issues/5803

Change-Id: I854cb3797802a32942c25f2765dbb72be88bacbd
2023-04-27 11:15:27 +02:00
Jeremy Wharton
7a17dc5e07 satellite/{console,satellitedb}: add methods for project invite table
This change immplements methods for interacting with the project member
invitations table.

Resolves #5766

Change-Id: I0090c50f9fde5bcdae4ebdaa72cdcaa84d341b54
2023-04-26 11:26:12 -05:00
Jeremy Wharton
defb9eae82 satellite/satellitedb: add table for project invitations
A table for storing pending project member invitations has been added.
This table is required to satisfy our new project invitation UX revamp.

References #5766

Change-Id: I6f948de66ed5b4dc81532564958ff7f48533cad2
2023-04-26 11:26:12 -05:00
dlamarmorgan
efcae857ba satellite/main,stripe/{client,service}: stripe balance invoice item cmd
Add a new billing command that will convert stripe customer balances into invoice items so that the charges can be processed normally by the invoicing workflow.

Change-Id: Iaa8350e7aca80a0f14e94eb8ef8b7d6ce0b5b3b8
2023-04-25 21:08:23 +00:00
dlamarmorgan
5fc493b276 metainfo: per-project and user-agent eventkit-based usage tracking
Add eventkit events for each satellite RPC type, tagged with user agent and public project id.

Change-Id: I838401683298d4594495de18563c55031e82f881
2023-04-25 15:13:43 +00:00
Vitalii
4d998970d4 satellite/admin: rework update user limits functionality
Fixed nil pointer dereference panic.
Updated naming conventions so that PUT request and GET response bodies are the same (bandwidth, storage and segment).
Allowed usage of notations like 150GB, 2TB for storage and bandwidth limits.
Updated tests.

Issue:
https://github.com/storj/storj/issues/5674

Change-Id: I7ac27c00721a9b4bf507afa34cb05c4475a809ad
2023-04-25 13:38:59 +00:00
Wilfred Asomani
260b71e70c satellite/{console,accountfreeze}: test freeze effects
This change adds more tests to the autofreeze chore and the freeze
service according to the testplan linked in the issue below.

Issue: https://github.com/storj/storj/issues/5738

Change-Id: Ib2afaa283961b2e7ef6fb6e5613ee083ac7d79eb
2023-04-25 13:04:37 +00:00
Michal Niewrzal
1aa24b9f0d satellite/audit: remove segments loop parts
We are switching completely to ranged loop.

https://github.com/storj/storj/issues/5368

Change-Id: I9cec0ac454f40f19d52c078a8b1870c4d192bd7a
2023-04-24 15:52:11 +00:00
Michal Niewrzal
6ac5bf0d7c satellite/gracefulexit: remove segments loop parts
We are switching completely to ranged loop.

https://github.com/storj/storj/issues/5368

Change-Id: Ia3e2d7879d91f7f5ffa99b8e8f108380e3b39f31
2023-04-24 15:00:26 +00:00
Michal Niewrzal
6a55682bc6 satellite/accounting/nodetally: remove segments loop parts
We are switching completely to ranged loop.

https://github.com/storj/storj/issues/5368

Change-Id: I6176a129ba14cf83fb635048d09e6748276b52a1
2023-04-24 14:25:53 +00:00
Egon Elbre
f40a0cb7ba satellite/*: use typed lrucache and ReadCache
Change-Id: Ieee535dd8735a95dd196a77413e4a25a6a72342c
2023-04-21 10:49:08 +00:00
Michal Niewrzal
fbfe5aaad7 satellite/metrics: remove code related to segments loop
We are switching completely to ranged loop.

Change-Id: I32120ef496addebec2de088fd10d0c1d02313c68
2023-04-20 13:47:22 +00:00
Egon Elbre
2405bc8f3b satellite/metabase: stop using the common error type
Updates https://github.com/storj/storj/issues/5291

Change-Id: I7b57a4b454d3619cb5d8ae4cd92f818ad2839c8b
2023-04-19 19:18:18 +03:00
Moby von Briesen
3679e29a93 satellite: Remove remaining references to "partner ID"
This field is deprecated in favor of UserAgent; Removing these
references is the final step necessary before dropping the columns from
the database.

https: //github.com/storj/storj/issues/5432
Change-Id: I3a6619170dcf382f82dc8eddb73b6547eaf636f0
2023-04-19 15:37:10 +00:00
Egon Elbre
41931970fd satellite/*: changes to stop using storj.ListDirection
Currently this duplicates the constants, because we cannot yet bump
common nor uplink.

Change-Id: I81818df8a9092e0b358eb946c31f2a42ac9dbf0a
2023-04-19 15:30:42 +03:00
Egon Elbre
54beea82da satellite/metabase: define a local ErrObjectNotFound
Updates https://github.com/storj/storj/issues/5291

Change-Id: If847cd95a9b500fb18535d20b46b1b1714021584
2023-04-19 12:03:01 +00:00
Márton Elek
9fbad53bbe
satellite/gracefulexit: extend GE tests using multple hash algo
Change-Id: I0b7accd5c0ae9ebb7adc7e5b8545bab639d70d36
2023-04-19 11:05:23 +02:00
Wilfred Asomani
9ac5183c7e satellite/consoleweb: improve freeze-status endpoint
This change adds a freeze status endpoint that gets whether a user is
frozen or warned.

Change-Id: I0bc498a128af969177c2cfcfda9b9ee3d8968305
2023-04-18 19:57:05 +00:00
Egon Elbre
daf5264f48 satellite/console/consoleweb: create a fallback error.html
Currently when error.html is not available, we should still start the
server and fallback to some simpler implementation. This template does
not require any external assets.

Change-Id: I76b660db988987e1e9ebadd966f60e149f26ff24
2023-04-18 14:36:51 +00:00