Commit Graph

487 Commits

Author SHA1 Message Date
Moby von Briesen
3216674c19 satellite/payments/accountfreeze: Increase exclusion threshold
Account freeze excludes accounts who have invoices which are unpaid and
are above a configured value. This change updates that configured value
from $100 to $1000.

Change-Id: Iff381deeb73ef2d8fccaf7a1612a33e8aeb08698
2023-10-06 14:39:20 +00:00
Márton Elek
58b98bc335 satellite/repair: repair is configurable to work only on included/excluded placements
This patch finishes the placement aware repair.

We already introduced the parameters to select only the jobs for specific placements, the remaining part is just to configure the exclude/include rules. + a full e2e unit test.

Change-Id: I223ba84e8ab7481a53e5a444596c7a5ae51573c5
2023-09-27 14:54:06 +00:00
Michal Niewrzal
e1215d5da8 satellite/overlay: add AOST to GetParticipatingNodes method
This method is sometimes ends with transaction error. Most probably
because it's trying to do full table scan on nodes table which is
heavily used. Adding AOST should help with DB contention.

Change-Id: Ibd4358d28dc26922b60c6b30862f20e7c0662cd1
2023-09-27 12:00:10 +00:00
paul cannon
72189330fd satellite/gracefulexit: revamp graceful exit
Currently, graceful exit is a complicated subsystem that keeps a queue
of all pieces expected to be on a node, and asks the node to transfer
those pieces to other nodes one by one. The complexity of the system
has, unfortunately, led to numerous bugs and unexpected behaviors.

We have decided to remove this entire subsystem and restructure graceful
exit as follows:

* Nodes will signal their intent to exit gracefully
* The satellite will not send any new pieces to gracefully exiting nodes
* Pieces on gracefully exiting nodes will be considered by the repair
  subsystem as "retrievable but unhealthy". They will be repaired off of
  the exiting node as needed.
* After one month (with an appropriately high online score), the node
  will be considered exited, and held amounts for the node will be
  released. The repair worker will continue to fetch pieces from the
  node as long as the node stays online.
* If, at the end of the month, a node's online score is below a certain
  threshold, its graceful exit will fail.

Refs: https://github.com/storj/storj/issues/6042
Change-Id: I52d4e07a4198e9cb2adf5e6cee2cb64d6f9f426b
2023-09-27 08:40:01 +00:00
Ivan Fraixedes
6555a68fa9 satellite/admin: Serve back-office static UI
Serve the front-end sources of the new back-office through the current
satellite admin server under the path `/back-office`.

The front-end is served in the same way than the current one, which is
through an indicated directory path with a configuration parameter or
embed in the binary when that configuration parameter is empty.

The commit also slightly changes the test that checks serving these
static assets for not targeting the empty file in the build folder.

build folders must remain because of the embed directive.

Change-Id: I3c5af6b75ec944722dbdc4c560d0e7d907a205b8
2023-09-26 13:18:29 +00:00
Jeremy Wharton
c8f4f5210d satellite/console: return edge URL overrides in project info responses
API responses containing project information now contain the edge
service URL overrides configured for that project. The overrides are
based on the project's default placement.

References #6188

Change-Id: Ifc3dc74e75c0f5daf0419ac3be184415c65b202e
2023-09-12 12:10:18 -05:00
Vitalii
6e1fd12930 web/satellite: paginate ListObjects request to show more than 1000 objects
With this change, we are able to fetch all objects to show in the object browser.
AWS SDK V3 provides paginator functionality to automatically make additional requests for every MaxKeys value (we use 500 objects at a time).
By initial request we fetch first 500 objects and save continuation tokens for the rest of the object batches.
Also, we save currently active (fetched) object range.
If user tries to open a page with objects which are out of currently active range then we look for needed continuation token and fetch needed objects batch.
Added a feature flag for this funtionality.

Issue:
https://github.com/storj/storj/issues/5595

Change-Id: If63e3c2ddaac3ea9f2bc1dc63cb49007f897e3e2
2023-09-06 12:47:15 +00:00
Wilfred Asomani
dcc4bd0d10 satellite/{console,payments}: freeze/warn storjscan users
This change enables the freezing/warning of users who use storjscan.

Issue: https://github.com/storj/storj/issues/6164

Change-Id: I7b00ee09d6527b3818b72326e9065c82ef5a2ac8
2023-08-31 13:22:21 +00:00
Wilfred Asomani
516241e406 cmd,satellite: remove Graphql code and dependencies
This change removes unused GraphQL code. It also updates storj sim code
to use the GraphQL replacement HTTP endpoints and removes the GraphQL
dependency.

Issue: https://github.com/storj/storj/issues/6142

Change-Id: Ie502553706c4b1282cd883a9275ea7332b8fc92d
2023-08-22 12:23:14 +00:00
Paul Willoughby
fdd4be80bf satellite/metainfo: increase default MaxEncryptedObjectKeyLength
Allow a longer encrypted key length to reduce 'key length is too big'
errors in gateway-mt.  Gateway is enforcing an unencrypted key length
of 1024 bytes but when encrypted some keys are exceeding the current
limit.

Updates https://github.com/storj/gateway-mt/issues/335

Change-Id: I38a0fbb0843fd782aeadca85f9a202821421b5a2
2023-08-21 14:03:16 -06:00
littleskunk
792bb113bc
satellite/console: Enable gallery view and limits area by default (#6177) 2023-08-14 17:05:19 +02:00
Moby von Briesen
b9206b1844 satellite/console: support hosting Vuetify POC on subdomain
This change allows you to host the vuetify app on <x>.example.com where
the main app is hosted on example.com. A configuration is added to
specify an exact subdomain for cookies. For example, if my production
app is hosted on us1.storj.io and my vuetify app is hosted on
vuetify.us1.storj.io, the cookie domain should be set to ".us1.storj.io"
so that any authentication cookie is accessible to lower-level
subdomains.

Since the vuetify app does not currently support login/signup on its
own, it is still required to first login to the main satellite UI, then
navigate to the Vuetify app after the session cookie is set.

If the "vuetifypoc" prefix is not desirable when using subdomain hosting
for vuetify, the VITE_VUETIFY_PREFIX variable can be modified in
web/satellite/.env before running `npm run build-vuetify`. For now, we
should keep this prefix because it makes developing on the vuetify app
significantly easier if subdomains are not being used.

Issue: https://github.com/storj/storj/issues/6144

Change-Id: Iba1a5737892c8ee8f38148a17b94e3222f8798e6
2023-08-14 13:15:41 +00:00
Michal Niewrzal
7be844351d satellite/metainfo: remove ServerSideCopyDuplicateMetadata
https://github.com/storj/storj/issues/5891

Change-Id: Ib5440169107acca6e832c2280e1ad12dfd380f28
2023-08-08 12:15:10 +00:00
Márton Elek
2ed08922d9 satellite/api: configuration option to set additional node tag authorities
Change-Id: Iba387e37cb586ce1378668c99beb3b4db8a9064d
2023-08-03 08:34:02 +00:00
Michal Niewrzal
cebf255d64 satellite/metabase: adjust BeginObjectNextVersion to use pending_objects
Change is adjusting BeginObjectNextVersion to create pending object in
`pending_objects` or `objects` table depends on configuration. This is
first change to move pending objects from objects table.

General goal is to support both tables until `objects` table will be
free from pending objects. Whenever it will be needed code will be
supporting both tables at once.

To be able to decide if we need to use `pending_objects` table or
`objects` table we extend satellite stream id to keep that information
for later use.

BeginObjectExactVersion will be not adjusted because at the moment it's
used only in tests.

Part of https://github.com/storj/storj/issues/6046

Change-Id: Ibf21965f63cca5e1775469994a29f1fd1261af4e
2023-08-02 14:42:26 +00:00
Michal Niewrzal
65aa9c11bc satellite/gc/sender: adjust some defaults
Change-Id: I94f5a18bc2d4218e834259b2ba22f3578745c975
2023-08-01 14:27:44 +00:00
Michal Niewrzal
73a279235a satellite/accounting/live: get project totals in batches
After infrastructure changes redis instance is not neccessay close
to core instance (where tally is calculated) and round trips to get
data from redis can be very costly. From less than hour calculation can take few hours for larger satellite.

This change combines 'segment' and 'storage' usage requests into
batches to reduce latency impact on tally calculation.

https://github.com/storj/storj/issues/5800

Change-Id: I87e57ec09e88fd167060a4ed51dc8b0274a095c5
2023-07-24 14:13:04 +00:00
Vitalii
5317135416 satellite/payments: fix config value for auto upgrade user tier flow
Fixed config value which indicates how many base units of US micro dollars are needed to auto upgrade user to paid tier.

Change-Id: I22821ac22fc3eaeeea21c6dec4e6912025df63aa
2023-07-19 10:43:07 +00:00
Vitalii
2ee0195eba satellite/payments: extend billing chore functionality to upgrade user
Added new observer for billing chore to check user's balance and upgrade their account if balance is more than or equal to needed amount for upgrade.
Added new config value which stands for needed amount of base units of US micro dollars needed to upgrade user.

Issue:
https://github.com/storj/storj/issues/5978

Change-Id: Ic3992cd3114397bfdd9e231ca090ff21ca66648b
2023-07-18 13:15:02 +03:00
Michal Niewrzal
47a4d4986d satellite/repair: enable declumping by default
This feature flag was disabled by default to test it slowly. Its enabled
for some time on one production satellite and test satellites without
any issue. We can enable it by default in code.

Change-Id: If9c36895bbbea12bd4aefa30cb4df912e1729e4c
2023-07-17 15:02:35 +00:00
Cameron
7e03ccfa46 satellite/console: optional separate web app server
This change creates the ability to run a server separate from the
console web server to serve the front end app. You can run it with
`satellite run ui`. Since there are now potentially two servers instead
of one, the UI server has the option to act as a reverse proxy to the
api server for local development by setting `--console.address` to the
console backend address and `--console.backend-reverse-proxy` to the
console backend's http url. Also, a feature flag has been implemented
on the api server to retain the ability to serve the front end app. It
is toggled with `--console.frontend-enable`.

github issue: https://github.com/storj/storj/issues/5843

Change-Id: I0d30451a20636e3184110dbe28c8a2a8a9505804
2023-07-11 12:17:35 -04:00
Moby von Briesen
bd4d57c604 satellite/payments: Exclude users who pay via storjscan from autofreeze
Add a configuration (default true) to exclude users who have made
storjscan payments from being auto-warned/frozen for an unpaid invoice.
This will allow us to reach out to these users and handle warning/freezing
manually. Auto account freeze still handles CC-only users.

Fixes https://github.com/storj/storj/issues/6027

Change-Id: I0c862785dad1c8febfa11100c0d30e621ce3ae9b
2023-07-10 13:39:01 +00:00
Márton Elek
97a89c3476 satellite: switch to use nodefilters instead of old placement.AllowedCountry
placement.AllowedCountry is the old way to specify placement, with the new approach we can use a more generic (dynamic method), which can check full node information instead of just the country code.

The 90% of this patch is just search and replace:

 * we need to use NodeFilters instead of placement.AllowedCountry
 * which means, we need an initialized PlacementRules available everywhere
 * which means we need to configure the placement rules

The remaining 10% is the placement.go, where we introduced a new type of configuration (lightweight expression language) to define any kind of placement without code change.

Change-Id: Ie644b0b1840871b0e6bbcf80c6b50a947503d7df
2023-07-07 16:55:45 +00:00
paul cannon
a4d68b9b7e satellite/metabase: server-side copy copies metadata
..instead of using segment_copies and ancestor_stream_id, etc.

This bypasses reference counting entirely, depending on our mark+sweep+
bloomfilter garbage collection strategy to get rid of pieces once they
are no longer part of a segment.

This is only safe to do after we have stopped passing delete requests on
to storage nodes.

Refs: https://github.com/storj/storj/issues/5889
Change-Id: I37bdcffaa752f84fd85045235d6875b3526b5ecc
2023-07-06 14:40:59 +00:00
Jeremy Wharton
80c5a628cb satellite/console/dbcleanup: remove project invite cleanup
This reverts 9c75316 which allowed the satellite console DB cleanup
chore to delete expired project member invitations. We now want such
invitations to be accessible indefinitely.

References #5752

Change-Id: I489a7e19df825dd14376d3d260b70b3eef643e03
2023-06-23 21:15:36 +00:00
Paul Willoughby
3180e09750 satellite/metainfo: increase default MaxEncryptedObjectKeyLength
Allow a longer encrypted key length to reduce 'key length is too big'
errors in gateway-mt.  Gateway is enforcing an unencrypted key length
of 1024 bytes but when encrypted some keys are exceeding the current
limit.

Updates https://github.com/storj/gateway-mt/issues/335

Change-Id: Ib02e2064c42e96b9d59936905832d8dd6068d2c7
2023-06-22 22:59:52 +00:00
Cameron
37e7eeb0e9 satellite/payments/accountfreeze: set grace period default to 15 days
Change-Id: Ied8f3758b579b83ebf04cba0fde9715c689bac4f
2023-06-22 18:53:03 +00:00
Clement Sam
1166fdfbab satellite/gc: add piece tracker ranged loop observer
Resolves https://github.com/storj/storj/issues/5798

Change-Id: I6fe2c57b3a247b085026feb8bee60c2d002db71b
2023-06-22 18:17:39 +00:00
Michal Niewrzal
2b2bca8e81 satellite/accounting/tally: save tallies in a batches
Because we are saving all tallies as a single SQL statement we finally
reached maximum message size. With this change we will call SaveTallies multiple times in batches.

https://github.com/storj/storj/issues/5977

Change-Id: I0c7dd27779b1743ede66448fb891e65c361aa3b0
2023-06-22 17:02:26 +00:00
Michal Niewrzal
bbdeb1eeb8 satellite/gc/bloomfilter: enable sync observer by default
After deploying to most of production satellites we want to enable
it by default. No issues found after few weeks of running.

Change-Id: I4d83c65edaa95b0e50eeab6ac5e2c6cbb7206c1e
2023-06-20 10:42:05 +00:00
Michal Niewrzal
cb9a7bdc71 satellite/repair/repairer: make DialTimeout configurable
This change makes dial timeout configurable and change it also from
defatul 20s to 5s. Main motivation is that during repair we often loose
lots of time to dial which eventually will fail. New timeout should be
still enough to dial but we will move forward quicker to next node if
that one will fail.

Timeout is also applied directly as context timeout in case we will
use noise of tcp fast open one day.

Change-Id: I021bf459af49b11241e314fa1a7887c81d5214ea
2023-06-16 12:23:25 +00:00
Moby von Briesen
6268c75d3f satellite/console: Enable all projects dashboard by default
Now that the table view has been implemented, the all projects dashboard
is ready to be turned on everywhere.

https://github.com/storj/storj/issues/5872

Change-Id: Iead684bf7d326d36d4d323eb63a3ed520602b4dc
2023-06-16 00:33:21 +00:00
Michal Niewrzal
2e0b687581 satellite/metainfo: drop piecedeletion package
We are not using this package anymore.

Change-Id: If32315d43d73c8deb096e93cb43c03881bd9aad1
2023-06-14 11:19:21 +00:00
JT Olio
3fff61f04a payments: don't redefine compensation rates twice
Change-Id: Ic00abe3795a000d4f0284c99f270180123a2f663
2023-06-10 12:40:43 +00:00
Vitalii
e83e98e46a web/satellite: vuetify POC
Built side Vuetify subproject inside web/satellite with limited functinality.
For now it has navigation side bar, simple project dashboard and team page (where you can list/add team members).

Issue:
https://github.com/storj/storj/issues/5854

Change-Id: I9ff3e80b8ace1dc31de6a788174c5ffc19f050f8
2023-06-10 00:59:40 +00:00
Michal Niewrzal
61933bc6f0 satellite/{metainfo,metabase}: optimize expired/zombie objects deletion
* optimize SQL for zombie objects deletion query by reducing some direct
selects to segments table
* set AOST for expired/zombie object deletion (was 0 since now)

https://github.com/storj/storj/issues/5881

Change-Id: I50482151d056a86fe0e31678a463f413d410759d
2023-06-09 11:22:46 +00:00
Jeremy Wharton
6359c537c0 satellite/console: add API methods for responding to project invites
API endpoints and associated methods have been implemented to allow
users to accept or decline their pending project member invitations
through the satellite frontend.

References #5855

Change-Id: Ic23721c64a65e741dc1015838e617fd1af5c8ca4
2023-06-06 10:38:00 +00:00
Michal Niewrzal
337eb9be6a satellite/repair/checker: put into queue segment off placement
Checker when qualifying segment for repair is now looking at pieces
location and if they are outisde segment placement puts them into
repair queue.

Fixes https://github.com/storj/storj/issues/5895

Change-Id: If0d941b30ad94c5ef02fb1a03c7f3d04a2df25c7
2023-06-05 15:53:49 +00:00
Michal Niewrzal
128b0a86e3 satellite/repair/repairer: repair pieces out of placement
Segment repairer should take into account segment 'placement' field
and remove or repair pieces from nodes that are outside this placement.

In case when after considering pieces out of placement we are still above
repair threshold we are only updating segment pieces to remove
problematic pieces. Otherwise we are doing regular repair.

https://github.com/storj/storj/issues/5896

Change-Id: I72b652aff2e6b20be3ac6dbfb1d32c2840ce3d59
2023-06-05 14:48:36 +00:00
Wilfred Asomani
2e3ed973de satellite/{consoleapi,web}: limit user input
This change limits the length of user input fields like search, email,
username. It also limits the receivable size of request payloads.
This is to prevent potential DDoS attacks resulting from receiving
large payloads.
Improvements are also made to the accounts page and register success
pages to display long names/emails better.

Issue: https://github.com/storj/storj-private/issues/201

Change-Id: I5d36eb83609b3605335a8d150b275c89deaf3b43
2023-06-05 11:43:56 +00:00
Vitalii
89457b3472 web/satellite: initial implementation of gallery view
Added new gallery view for object browser.
It is behind new feature flag.

TODO: add options dropdown and modals

Issue:
https://github.com/storj/storj/issues/5824

Change-Id: I21829c599cd904b833eaf429690c66c3da306a0f
2023-06-02 14:03:49 +03:00
Moby von Briesen
058dbd4cb7 web/satellite: Disable limits area of dashboard
Add a config flag (default false) to hide the new limit cards (e.g.
segment, storage, bandwidth limits) from the UI. We need to investigate
some queries the egress card is using before enabling these everywhere.

Change-Id: I762e7d9e6a0a4315f1520e688b2bad32b100e5a0
2023-05-31 13:40:54 +03:00
Moby von Briesen
037b07712d satellite/console: Disable all projects dashboard by default
Waiting for basic table view support before enabling

https://github.com/storj/storj/issues/5927

Change-Id: I48388f733138b5aa784c7b2fd3c4d3c813b9e40f
2023-05-30 15:07:43 -04:00
Moby von Briesen
714c9f40c1 satellite/console: Turn all projects dashboard on by default
Related issue: https://github.com/storj/storj/issues/5872

Change-Id: Idd78f78a63534e15f99ebf3015431bbe52ba18ef
2023-05-24 13:45:22 +00:00
Vitalii
98591f34f9 satellite/{web, console}: implemented new objects uploading hover component
Added new feature flag.
Implemented new objects uploading hover component.

Issue:
https://github.com/storj/storj/issues/5825

Change-Id: If43ade5db33aecba6c089ce727503518c59f6ffc
2023-05-23 15:52:41 +00:00
Jeremy Wharton
d8f64326f5 satellite/accounting/tally: remove objects loop
This change removes the use of the objects loop for calculating bucket
tallies. It has been superseded by a custom query.

Change-Id: I9ea4633006c9af3ea14d7de40871639e7b687c22
2023-05-22 15:18:03 -05:00
Cameron
09ec5f107d satellite/payments: Remove expired package credits
During billing, before invoice creation, check if users are part of a
package plan. If so, and if the package plan is expired, remove unused
credit from the user's balance. If the user has credit in addition to
the package credit, send an analytics event to notify someone to handle
the credit removal manually.

Change-Id: Iad71d791f67c9733f9d9e42f962c64b2780264cc
2023-05-22 11:58:37 +00:00
paul cannon
de737bdee9 satellite/repair: add flag for de-clumping behavior
It seems that the "what pieces are clumped" code does not work right, so
this logic is causing repair overload or other repair failures.

Hide it behind a flag while we figure out what is going on, so that
repair can still work in the meantime.

Change-Id: If83ef7895cba870353a67ab13573193d92fff80b
2023-05-18 21:02:36 +00:00
Michal Niewrzal
c0e7f463fe satellite/metabase: remove segmentsloop package
Last change to remove segments loop from codebase.

https://github.com/storj/storj/issues/5237

Change-Id: I77b12911b6b4e390a7385e6e8057c7587e74b70a
2023-05-18 19:08:29 +00:00
Moby von Briesen
19cb08b025 web/satellite: add "public linksharing URL"
* Update defaults for gateway credentials URL and linksharing URL to use
  storjsatelliteshare.io instead of storjshare.io
* Add new config for "public linksharing URL" and set it to
  link.storjshare.io
* Use "private" linksharing URL for actions within the object browser
* Use "public" linksharing URL for sharing files externally

Resolves https://github.com/storj/storj/issues/5805

Change-Id: I2c8fbd04141755b4751dcf4d054253a7ff8d6cf3
2023-05-18 07:02:26 +00:00