Currently slower storagenodes can slow down deletion queue.
To make piece deletion faster reduce the maximum time spent in
either dialing or piece deletion requests.
With this change:
* dial timeout is 3s
* request timeout is 15s
* fail threshold is set to 10min
Similarly, we'll mark storage node as failed when the timeout occurs.
The timeout usually indicates that the storagenode is overwhelmed.
Garbage collection will ensure that the pieces get deleted eventually.
Change-Id: Iec5de699f5917905f5807140e2c3252088c6399b
To make our free tier limits more clear, we will reduce the number of projects allowed from 3 to 1, and increase the storage and bandwidth limit of the free tier from 50 Gb to 150 GB. The total allotments across all projects for a given user are unchanged, just reduced to a single project.
Change-Id: Ic8dddb135f2b83a3f36e2b9fdcb477e351ec137b
Multipart upload limits added. Last part has no size limit.
Max number of parts: 10000, min part size: 5 MiB
Change-Id: Ic2262ce25f989b34d92f662bde720d4c4d0dc93d
https://storjlabs.atlassian.net/browse/PG-305
we should extend method move of cmd/uplink/cmd/mv.go
if both parameters end with slash -
i should list all files and call move method in loop
parameters:
uplink mv sj://bucket/a/prefix/ sj://new-bucket/a/new-prefix/
Change-Id: Ic24c2af83153ea60ec74393e65736af094877151
Removes database tables and functionality related to our custom
coupon implementation because it has been superseded by the Stripe
coupon and promo code system. Requires implementations of the
payments Invoices interface to return coupon usages along with
invoices.
Change-Id: Iac52d2ff64afca8cc4dbb2d1f20e6ad4b39ddfde
New command for cli to move object to different
location.
uplink mv sj://bucket/your-object sj://bucket/moved-object
Change-Id: I85a4961aa59f250819954e78f20363ac3c570938
Make bucket command was using full location
specified in command line instead only bucket name.
As an addition change contains basic integration tests
with storj-sim.
Change-Id: Ie3b5283468b7fbde0b1333f01dc4fc2a2952e1a1
Jenkins uses the same folder for different PR-s. Sometimes other runs do
not cleanup after themselves (e.g. timeout), hence add a cleanWs step to
ensure we delete files in the workspace.
gateway-st introduced a replace directive in go.mod, which does not work
with go install. Hardcode to the last version without the directive.
Using this fix to unblock ci builds.
Change-Id: I5e5d75bf47e30a5a8b6d835867c0c9176f25e08a
Use multipart upload to upload single object in parts in
parallel. Its using parallelism flag added earlier.
Change-Id: I45b531a5db43c86f0112a5e3bb4a83bc1d65650f
Adds support for new uplink method DownloadObjectAt which
gives ability to download single object in parallel.
Change-Id: I8388653429992b0d24c383d17d7e90904203fe77
Rate limits application of coupon codes by user ID to prevent
brute forcing. Refactors the rate limiter to allow limiting based
on arbitrary criteria and not just by IP.
Change-Id: I99d6749bd5b5e47d7e1aeb0314e363a8e7259dba
MFA is complete and we are good to enable it in production. This change
removes the flag that disables MFA by default.
Change-Id: I2f985ae501171bdab505d664b43c8cfc248bad8d
package in audit
This PR implements reputation store and replace overlay in audit service
to use such store for storing node's audit stats.
In order to keep the changeset smaller, most of the changes in this PR is for copying audit logic in overlay to
reputation package. In a following PR, the duplicating code will be
removed from overlay.
Change-Id: I16c12494a0970f44c422b26cf603c1dc489e5bc1
Full path: satellite/{payments,console},web/satellite
* Adds the ability to apply coupon codes from the billing page in the
satellite UI.
* Flag for coupon code UI is split into two flags - one for the billing
page and one for the signup page. This commit implements the first, but
not the second.
* Update the Stripe dependency to v72, which is necessary to
use Stripe's promo code functionality.
Change-Id: I19d9815c48205932bef68d87d5cb0b000498fa70
Bucket tally calculation will be removed from metaloop and will
use metabase objects iterator directly.
At the moment only bucket tally needs objects so it make no sense
to implement separate objects loop.
Change-Id: Iee60059fc8b9a1bf64d01cafe9659b69b0e27eb1
Added feature flag for MFA
Added new client-side api call to enable MFA returning secret
Updated users Vuex module to include new API call
Change-Id: Ia9e10f68c4a7da39b4f7c1073e657c2de98fb0db
The user must complete a reCAPTCHA in order to register.
ReCAPTCHA verification failure results in rejection of the
registration attempt.
Change-Id: I34ba7db414d756fd1aaebdc3d19cccbfc7fc1ea3
When a user adds a credit card, switch them to the paid tier and update
their projects with new bandwidth/storage limits. New projects for the
paid tier user will also have the updated limits.
The new limits are:
* storage per project - 50 GB free/25 TB paid
* bandwidth per project - 50 GB free/100 TB paid
Change-Id: I7d6467d077e8bb2bbe4bcf88ab8d75490f83165e
Because of our free/paid tier plan, we do not need a paywall anymore. We
have not used it in a while, but still have leftover code laying around.
Change-Id: Iaea8c39faf042a2f7a6b837727bb135c8bdf2907
Adding AS OF SYSTEM TIME to query that is calculating project bandiwdth.
As an addition method for setting interval is added as test doesn't
work well with default interval.
Change-Id: Id1e15be4f6afff13b9dc2b7f595e2edb6de28db9
We used this to reduce initial load on the core to avoid OOM. However,
this is not a problem anymore with garbage collection running
separately.
Change-Id: Ifd62c822a74974bc21a5913199334469a4bc0130
This adds verification for the processed count and before and after
segment/objects table counts.
This adds new flag:
metainfo.segment-loop.suspicious-processed-ratio: 0.03
This defaults to 3%, which at 100M segments is 3M segments.
Change-Id: I5ee03e913ddc4e67e94010ced126a2a9ea51f41b
This adds verification for the processed count and before and after
segment/objects table counts.
This adds new flag:
metainfo.loop.suspicious-processed-ratio: 0.03
This defaults to 3%, which at 100M objects is 3M objects.
Change-Id: Ife5522ecc97bcc5a55667f36868a0f1fc8e4c561
This is part of metaloop refactoring. We plan to remove
irreparable at some point but there was not time for it.
Now instead refatoring it for segmentloop its just easier
to drop it.
Later we still need to drop table with migration step.
Change-Id: I270e77f119273d39a1ecdcf5e1c37a5662a29ab4
Currently we did not limit the "as of system time" for iterating over
objects table. Using just an interval would cause problems with the
tests. That could be overcome skipping that interval for tests
altogether, however, we should probably test those more to ensure that
GC stays working as intended.
This is a safer code, however, maybe not as straigthforward as it could
be.
Change-Id: I374f77783b2af42bb6da846735ceea20a7ce5e60
Satellites set their configuration values to default values using
cfgstruct, however, it turns out our tests don't test these values
at all! Instead, they have a completely separate definition system
that is easy to forget about.
As is to be expected, these values have drifted, and it appears
in a few cases test planet is testing unreasonable values that we
won't see in production, or perhaps worse, features enabled in
production were missed and weren't enabled in testplanet.
This change makes it so all values are configured the same,
systematic way, so it's easy to see when test values are different
than dev values or release values, and it's less hard to forget
to enable features in testplanet.
In terms of reviewing, this change should be actually fairly
easy to review, considering private/testplanet/satellite.go keeps
the current config system and the new one and confirms that they
result in identical configurations, so you can be certain that
nothing was missed and the config is all correct.
You can also check the config lock to see what actual config
values changed.
Change-Id: I6715d0794887f577e21742afcf56fd2b9d12170e
We want to move some of current metainfo loop observers to
segment loop. This change adds new service, similar to metainfo
loop but which is iterating only over segments.
Change-Id: I67f7f461781723a4476e2b83377f31736d7c4870
Rather than applying our internal satellite implementation of coupons
when new accounts are created, use a configured Stripe coupon instead.
If no configuration is set, no coupon will be applied.
This change also removes logic for adding coupons to customers who pay
with crypto - they will already have the free tier coupon applied
anyway.
We will be phasing out our internal coupon implementation.
Change-Id: Ieb87ddb3412acbc74986aa9d18a4cbd93c29861a
Use the 'AS OF SYSTEM TIME' Cockroach DB clause for the Graceful Exit
(a.k.a GE) queries that count the delete the GE queue items of nodes
which have already exited the network.
Split the subquery used for deleting all the transfer queue items of
nodes which has exited when CRDB is used and batch the queries because
CRDB struggles when executing in a single query unlike Postgres.
The new test which has been added to this commit to verify the CRDB
batch logic for deleting all the transfer queue items of the exited
nodes has raised that the Enqueue method has to run in baches when CRDB
is used otherwise CRDB has return the error "driver: bad connection"
when a big a amount of items are passed to be enqueued. This error
didn't happen with the current test implementation it was with an
initial one that it was creating a big amount of exited nodes and
transfer queue items for those nodes.
Change-Id: I6a099cdbc515a240596bc93141fea3182c2e50a9
