Account freeze excludes accounts who have invoices which are unpaid and
are above a configured value. This change updates that configured value
from $100 to $1000.
Change-Id: Iff381deeb73ef2d8fccaf7a1612a33e8aeb08698
This patch finishes the placement aware repair.
We already introduced the parameters to select only the jobs for specific placements, the remaining part is just to configure the exclude/include rules. + a full e2e unit test.
Change-Id: I223ba84e8ab7481a53e5a444596c7a5ae51573c5
This method is sometimes ends with transaction error. Most probably
because it's trying to do full table scan on nodes table which is
heavily used. Adding AOST should help with DB contention.
Change-Id: Ibd4358d28dc26922b60c6b30862f20e7c0662cd1
Currently, graceful exit is a complicated subsystem that keeps a queue
of all pieces expected to be on a node, and asks the node to transfer
those pieces to other nodes one by one. The complexity of the system
has, unfortunately, led to numerous bugs and unexpected behaviors.
We have decided to remove this entire subsystem and restructure graceful
exit as follows:
* Nodes will signal their intent to exit gracefully
* The satellite will not send any new pieces to gracefully exiting nodes
* Pieces on gracefully exiting nodes will be considered by the repair
subsystem as "retrievable but unhealthy". They will be repaired off of
the exiting node as needed.
* After one month (with an appropriately high online score), the node
will be considered exited, and held amounts for the node will be
released. The repair worker will continue to fetch pieces from the
node as long as the node stays online.
* If, at the end of the month, a node's online score is below a certain
threshold, its graceful exit will fail.
Refs: https://github.com/storj/storj/issues/6042
Change-Id: I52d4e07a4198e9cb2adf5e6cee2cb64d6f9f426b
Serve the front-end sources of the new back-office through the current
satellite admin server under the path `/back-office`.
The front-end is served in the same way than the current one, which is
through an indicated directory path with a configuration parameter or
embed in the binary when that configuration parameter is empty.
The commit also slightly changes the test that checks serving these
static assets for not targeting the empty file in the build folder.
build folders must remain because of the embed directive.
Change-Id: I3c5af6b75ec944722dbdc4c560d0e7d907a205b8
API responses containing project information now contain the edge
service URL overrides configured for that project. The overrides are
based on the project's default placement.
References #6188
Change-Id: Ifc3dc74e75c0f5daf0419ac3be184415c65b202e
With this change, we are able to fetch all objects to show in the object browser.
AWS SDK V3 provides paginator functionality to automatically make additional requests for every MaxKeys value (we use 500 objects at a time).
By initial request we fetch first 500 objects and save continuation tokens for the rest of the object batches.
Also, we save currently active (fetched) object range.
If user tries to open a page with objects which are out of currently active range then we look for needed continuation token and fetch needed objects batch.
Added a feature flag for this funtionality.
Issue:
https://github.com/storj/storj/issues/5595
Change-Id: If63e3c2ddaac3ea9f2bc1dc63cb49007f897e3e2
This change enables the freezing/warning of users who use storjscan.
Issue: https://github.com/storj/storj/issues/6164
Change-Id: I7b00ee09d6527b3818b72326e9065c82ef5a2ac8
This change removes unused GraphQL code. It also updates storj sim code
to use the GraphQL replacement HTTP endpoints and removes the GraphQL
dependency.
Issue: https://github.com/storj/storj/issues/6142
Change-Id: Ie502553706c4b1282cd883a9275ea7332b8fc92d
Allow a longer encrypted key length to reduce 'key length is too big'
errors in gateway-mt. Gateway is enforcing an unencrypted key length
of 1024 bytes but when encrypted some keys are exceeding the current
limit.
Updates https://github.com/storj/gateway-mt/issues/335
Change-Id: I38a0fbb0843fd782aeadca85f9a202821421b5a2
This change allows you to host the vuetify app on <x>.example.com where
the main app is hosted on example.com. A configuration is added to
specify an exact subdomain for cookies. For example, if my production
app is hosted on us1.storj.io and my vuetify app is hosted on
vuetify.us1.storj.io, the cookie domain should be set to ".us1.storj.io"
so that any authentication cookie is accessible to lower-level
subdomains.
Since the vuetify app does not currently support login/signup on its
own, it is still required to first login to the main satellite UI, then
navigate to the Vuetify app after the session cookie is set.
If the "vuetifypoc" prefix is not desirable when using subdomain hosting
for vuetify, the VITE_VUETIFY_PREFIX variable can be modified in
web/satellite/.env before running `npm run build-vuetify`. For now, we
should keep this prefix because it makes developing on the vuetify app
significantly easier if subdomains are not being used.
Issue: https://github.com/storj/storj/issues/6144
Change-Id: Iba1a5737892c8ee8f38148a17b94e3222f8798e6
Change is adjusting BeginObjectNextVersion to create pending object in
`pending_objects` or `objects` table depends on configuration. This is
first change to move pending objects from objects table.
General goal is to support both tables until `objects` table will be
free from pending objects. Whenever it will be needed code will be
supporting both tables at once.
To be able to decide if we need to use `pending_objects` table or
`objects` table we extend satellite stream id to keep that information
for later use.
BeginObjectExactVersion will be not adjusted because at the moment it's
used only in tests.
Part of https://github.com/storj/storj/issues/6046
Change-Id: Ibf21965f63cca5e1775469994a29f1fd1261af4e
After infrastructure changes redis instance is not neccessay close
to core instance (where tally is calculated) and round trips to get
data from redis can be very costly. From less than hour calculation can take few hours for larger satellite.
This change combines 'segment' and 'storage' usage requests into
batches to reduce latency impact on tally calculation.
https://github.com/storj/storj/issues/5800
Change-Id: I87e57ec09e88fd167060a4ed51dc8b0274a095c5
Fixed config value which indicates how many base units of US micro dollars are needed to auto upgrade user to paid tier.
Change-Id: I22821ac22fc3eaeeea21c6dec4e6912025df63aa
Added new observer for billing chore to check user's balance and upgrade their account if balance is more than or equal to needed amount for upgrade.
Added new config value which stands for needed amount of base units of US micro dollars needed to upgrade user.
Issue:
https://github.com/storj/storj/issues/5978
Change-Id: Ic3992cd3114397bfdd9e231ca090ff21ca66648b
This feature flag was disabled by default to test it slowly. Its enabled
for some time on one production satellite and test satellites without
any issue. We can enable it by default in code.
Change-Id: If9c36895bbbea12bd4aefa30cb4df912e1729e4c
This change creates the ability to run a server separate from the
console web server to serve the front end app. You can run it with
`satellite run ui`. Since there are now potentially two servers instead
of one, the UI server has the option to act as a reverse proxy to the
api server for local development by setting `--console.address` to the
console backend address and `--console.backend-reverse-proxy` to the
console backend's http url. Also, a feature flag has been implemented
on the api server to retain the ability to serve the front end app. It
is toggled with `--console.frontend-enable`.
github issue: https://github.com/storj/storj/issues/5843
Change-Id: I0d30451a20636e3184110dbe28c8a2a8a9505804
Add a configuration (default true) to exclude users who have made
storjscan payments from being auto-warned/frozen for an unpaid invoice.
This will allow us to reach out to these users and handle warning/freezing
manually. Auto account freeze still handles CC-only users.
Fixes https://github.com/storj/storj/issues/6027
Change-Id: I0c862785dad1c8febfa11100c0d30e621ce3ae9b
placement.AllowedCountry is the old way to specify placement, with the new approach we can use a more generic (dynamic method), which can check full node information instead of just the country code.
The 90% of this patch is just search and replace:
* we need to use NodeFilters instead of placement.AllowedCountry
* which means, we need an initialized PlacementRules available everywhere
* which means we need to configure the placement rules
The remaining 10% is the placement.go, where we introduced a new type of configuration (lightweight expression language) to define any kind of placement without code change.
Change-Id: Ie644b0b1840871b0e6bbcf80c6b50a947503d7df
..instead of using segment_copies and ancestor_stream_id, etc.
This bypasses reference counting entirely, depending on our mark+sweep+
bloomfilter garbage collection strategy to get rid of pieces once they
are no longer part of a segment.
This is only safe to do after we have stopped passing delete requests on
to storage nodes.
Refs: https://github.com/storj/storj/issues/5889
Change-Id: I37bdcffaa752f84fd85045235d6875b3526b5ecc
This reverts 9c75316 which allowed the satellite console DB cleanup
chore to delete expired project member invitations. We now want such
invitations to be accessible indefinitely.
References #5752
Change-Id: I489a7e19df825dd14376d3d260b70b3eef643e03
Allow a longer encrypted key length to reduce 'key length is too big'
errors in gateway-mt. Gateway is enforcing an unencrypted key length
of 1024 bytes but when encrypted some keys are exceeding the current
limit.
Updates https://github.com/storj/gateway-mt/issues/335
Change-Id: Ib02e2064c42e96b9d59936905832d8dd6068d2c7
Because we are saving all tallies as a single SQL statement we finally
reached maximum message size. With this change we will call SaveTallies multiple times in batches.
https://github.com/storj/storj/issues/5977
Change-Id: I0c7dd27779b1743ede66448fb891e65c361aa3b0
After deploying to most of production satellites we want to enable
it by default. No issues found after few weeks of running.
Change-Id: I4d83c65edaa95b0e50eeab6ac5e2c6cbb7206c1e
This change makes dial timeout configurable and change it also from
defatul 20s to 5s. Main motivation is that during repair we often loose
lots of time to dial which eventually will fail. New timeout should be
still enough to dial but we will move forward quicker to next node if
that one will fail.
Timeout is also applied directly as context timeout in case we will
use noise of tcp fast open one day.
Change-Id: I021bf459af49b11241e314fa1a7887c81d5214ea
Now that the table view has been implemented, the all projects dashboard
is ready to be turned on everywhere.
https://github.com/storj/storj/issues/5872
Change-Id: Iead684bf7d326d36d4d323eb63a3ed520602b4dc
Built side Vuetify subproject inside web/satellite with limited functinality.
For now it has navigation side bar, simple project dashboard and team page (where you can list/add team members).
Issue:
https://github.com/storj/storj/issues/5854
Change-Id: I9ff3e80b8ace1dc31de6a788174c5ffc19f050f8
* optimize SQL for zombie objects deletion query by reducing some direct
selects to segments table
* set AOST for expired/zombie object deletion (was 0 since now)
https://github.com/storj/storj/issues/5881
Change-Id: I50482151d056a86fe0e31678a463f413d410759d
API endpoints and associated methods have been implemented to allow
users to accept or decline their pending project member invitations
through the satellite frontend.
References #5855
Change-Id: Ic23721c64a65e741dc1015838e617fd1af5c8ca4
Checker when qualifying segment for repair is now looking at pieces
location and if they are outisde segment placement puts them into
repair queue.
Fixes https://github.com/storj/storj/issues/5895
Change-Id: If0d941b30ad94c5ef02fb1a03c7f3d04a2df25c7
Segment repairer should take into account segment 'placement' field
and remove or repair pieces from nodes that are outside this placement.
In case when after considering pieces out of placement we are still above
repair threshold we are only updating segment pieces to remove
problematic pieces. Otherwise we are doing regular repair.
https://github.com/storj/storj/issues/5896
Change-Id: I72b652aff2e6b20be3ac6dbfb1d32c2840ce3d59
This change limits the length of user input fields like search, email,
username. It also limits the receivable size of request payloads.
This is to prevent potential DDoS attacks resulting from receiving
large payloads.
Improvements are also made to the accounts page and register success
pages to display long names/emails better.
Issue: https://github.com/storj/storj-private/issues/201
Change-Id: I5d36eb83609b3605335a8d150b275c89deaf3b43
Added new gallery view for object browser.
It is behind new feature flag.
TODO: add options dropdown and modals
Issue:
https://github.com/storj/storj/issues/5824
Change-Id: I21829c599cd904b833eaf429690c66c3da306a0f
Add a config flag (default false) to hide the new limit cards (e.g.
segment, storage, bandwidth limits) from the UI. We need to investigate
some queries the egress card is using before enabling these everywhere.
Change-Id: I762e7d9e6a0a4315f1520e688b2bad32b100e5a0
This change removes the use of the objects loop for calculating bucket
tallies. It has been superseded by a custom query.
Change-Id: I9ea4633006c9af3ea14d7de40871639e7b687c22
During billing, before invoice creation, check if users are part of a
package plan. If so, and if the package plan is expired, remove unused
credit from the user's balance. If the user has credit in addition to
the package credit, send an analytics event to notify someone to handle
the credit removal manually.
Change-Id: Iad71d791f67c9733f9d9e42f962c64b2780264cc
It seems that the "what pieces are clumped" code does not work right, so
this logic is causing repair overload or other repair failures.
Hide it behind a flag while we figure out what is going on, so that
repair can still work in the meantime.
Change-Id: If83ef7895cba870353a67ab13573193d92fff80b
* Update defaults for gateway credentials URL and linksharing URL to use
storjsatelliteshare.io instead of storjshare.io
* Add new config for "public linksharing URL" and set it to
link.storjshare.io
* Use "private" linksharing URL for actions within the object browser
* Use "public" linksharing URL for sharing files externally
Resolves https://github.com/storj/storj/issues/5805
Change-Id: I2c8fbd04141755b4751dcf4d054253a7ff8d6cf3
