This change introduces a new freeze event, ViolationFreeze, for ToS
violations and also prepends other events with Billing to clarify what
they signify.
Issue: https://github.com/storj/storj-private/issues/386
Change-Id: Ieef2abbbb03d06377dc7a73ba5ef0742ada75e8e
The yarn.lock file came from the former repository where the back office
UI was developed.
It seems that our build process complains about some dirty state related
to this file. Because we don't use Yarn, we delete the file, hoping to
resolve the build issues.
Change-Id: I5febd8292657289d0fc67e08151c6c8b5ac8b5dc
This change introduces a new user status, PendingDeletion to be used to
mark users before they're actually deleted. It also skips users with
this status or Deleted status when generating invoices.
Issue: https://github.com/storj/storj/issues/6302
Change-Id: I6a80d0ed1fe4f223ae00e0961f18f2f62f9b5213
This change removes the project name and inviter name from the
registration page's invitation text in order to prevent phishing
attempts using these values from succeeding.
Resolves storj-private#431
Change-Id: I08636d712b6b273d484cf0594d395c9d7c02ebfa
This fixes an inconsistency with error returned on copy and move
endpoints to match other endpoints. validateAuth() is already
wrapping the RPC status around the error, so this shouldn't be
doing it again.
This also ensures that rate limit errors for FinishCopyObject and
FinishMoveObject are correctly returned as rpcstatus.ResourceExhausted
so uplink can correctly map these to uplink.ErrTooManyRequests.
Change-Id: I6bf6185b456d6774b99d56cf3d7d8f8aa2afa0e8
The satellite admin API endpoint responsible for returning project
limits now includes the burst limit in its responses.
Resolves#6276
Change-Id: Ibb3f1fdebf2f9ffd62de2d7e7a60d978c25bb22a
This patch finishes the placement aware repair.
We already introduced the parameters to select only the jobs for specific placements, the remaining part is just to configure the exclude/include rules. + a full e2e unit test.
Change-Id: I223ba84e8ab7481a53e5a444596c7a5ae51573c5
This method is sometimes ends with transaction error. Most probably
because it's trying to do full table scan on nodes table which is
heavily used. Adding AOST should help with DB contention.
Change-Id: Ibd4358d28dc26922b60c6b30862f20e7c0662cd1
When the new back office UI sources where copied from former repository
I didn't realize that the .gitignore had the package-lock.json file.
This commit remove the package-lock.json file, so it can be tracked, in
order to have reproducible builds.
The lack of the file caused the build to fail due to `npm ci` requires
it.
Change-Id: Ibe493d0cd5762afe5caabe9b77a333fd6daa5373
Currently, graceful exit is a complicated subsystem that keeps a queue
of all pieces expected to be on a node, and asks the node to transfer
those pieces to other nodes one by one. The complexity of the system
has, unfortunately, led to numerous bugs and unexpected behaviors.
We have decided to remove this entire subsystem and restructure graceful
exit as follows:
* Nodes will signal their intent to exit gracefully
* The satellite will not send any new pieces to gracefully exiting nodes
* Pieces on gracefully exiting nodes will be considered by the repair
subsystem as "retrievable but unhealthy". They will be repaired off of
the exiting node as needed.
* After one month (with an appropriately high online score), the node
will be considered exited, and held amounts for the node will be
released. The repair worker will continue to fetch pieces from the
node as long as the node stays online.
* If, at the end of the month, a node's online score is below a certain
threshold, its graceful exit will fail.
Refs: https://github.com/storj/storj/issues/6042
Change-Id: I52d4e07a4198e9cb2adf5e6cee2cb64d6f9f426b
We set lifecyclestage in Hubspot without Segment now. If Segment tries
to set lifecyclestage, it interferes with the desired behavior in
Hubspot.
Change-Id: I817c0324ecc69529d8ca7f617cb97d2f4e84aee8
Serve the front-end sources of the new back-office through the current
satellite admin server under the path `/back-office`.
The front-end is served in the same way than the current one, which is
through an indicated directory path with a configuration parameter or
embed in the binary when that configuration parameter is empty.
The commit also slightly changes the test that checks serving these
static assets for not targeting the empty file in the build folder.
build folders must remain because of the embed directive.
Change-Id: I3c5af6b75ec944722dbdc4c560d0e7d907a205b8
The repair checker and repair worker both need to determine which pieces
are healthy, which are retrievable, and which should be replaced, but
they have been doing it in different ways in different code, which has
been the cause of bugs. The same term could have very similar but subtly
different meanings between the two, causing much confusion.
With this change, the piece- and node-classification logic is
consolidated into one place within the satellite/repair package, so that
both subsystems can use it. This ought to make decision-making code more
concise and more readable.
The consolidated classification logic has been expanded to create more
sets, so that the decision-making code does not need to do as much
precalculation. It should now be clearer in comments and code that a
piece can belong to multiple sets arbitrarily (except where the
definition of the sets makes this logically impossible), and what the
precise meaning of each set is. These sets include Missing, Suspended,
Clumped, OutOfPlacement, InExcludedCountry, ForcingRepair,
UnhealthyRetrievable, Unhealthy, Retrievable, and Healthy.
Some other side effects of this change:
* CreatePutRepairOrderLimits no longer needs to special-case excluded
countries; it can just create as many order limits as requested (by
way of len(newNodes)).
* The repair checker will now queue a segment for repair when there are
any pieces out of placement. The code calls this "forcing a repair".
* The checker.ReliabilityCache is now accessed by way of a GetNodes()
function similar to the one on the overlay. The classification methods
like MissingPieces(), OutOfPlacementPieces(), and
PiecesNodesLastNetsInOrder() are removed in favor of the
classification logic in satellite/repair/classification.go. This
means the reliability cache no longer needs access to the placement
rules or excluded countries list.
Change-Id: I105109fb94ee126952f07d747c6e11131164fadb
Add the front-end sources of the new back-office.
The front-end doesn't have any business logic, it only has the pages and
the components, so it's purely UI.
The front-end was developed in a separate repository until was
completed.
Change-Id: I382e50789d6b929a67b8a0b887563ef48cb1473d
When we do `satellite run api --placement '...'`, the placement rules are not parsed well.
The problem is based on `viper.AllSettings()`, and the main logic is sg. like this (from a new unit test):
```
r := ConfigurablePlacementRule{}
err := r.Set(p)
require.NoError(t, err)
serialized := r.String()
r2 := ConfigurablePlacementRule{}
err = r2.Set(serialized)
require.NoError(t, err)
require.Equal(t, p, r2.String())
```
All settings evaluates the placement rules in `ConfigurablePlacementRules` and stores the string representation.
The problem is that we don't have proper `String()` implementation (it prints out the structs instead of the original definition.
There are two main solutions for this problem:
1. We can fix the `String()`. When we parse a placement rule, the `String()` method should print out the original definition
2. We can switch to use pure string as configuration parameter, and parse the rules only when required.
I feel that 1 is error prone, we can do it (and in this patch I added a lot of `String()` implementations, but it's hard to be sure that our `String()` logic is inline with the parsing logic.
Therefore I decided to make the configuration value of the placements a string (or a wrapper around string).
That's the main reason why this patch seems to be big, as I updated all the usages.
But the main part is in beginning of the `placement.go` (configuration parsing is not a pflag.Value implementation any more, but a separated step).
And `filter.go`, (a few more String implementation for filters.
https://github.com/storj/storj/issues/6248
Change-Id: I47c762d3514342b76a2e85683b1c891502a0756a
send analytics event if project invite link is clicked and if user
signs up.
github issue: https://github.com/storj/storj/issues/5190
Change-Id: I41eee5e679a84b9ec325815655684a98624d5656
Currently each testplanet test is running ranged loop no matter if
it's used or not. This is small change with some benefits like:
* saves some cpu cycles
* less log entries
* ranged loop won't interfere with other systems
Change have no big impact on tests execration but I believe it's nice to
have.
Change-Id: I731846bf625cac47ed4f3ca3bc1d1a4659bdcce8
There's only one value that BeginExactObject should return and there's
no point in restating that.
Also, use a clearer value for the random object version.
Change-Id: I06b26ad87d64e1b04b48458f624edd630f7f2f1d
This change adds an alternate MFA code recovery endpoint that requires
MFA code to generate codes.
Issue: https://github.com/storj/storj-private/issues/433
Change-Id: I10d922e9ad1ace4300d4bcfea7f48494227f1ff8
We stoped returning lots of errors as is to avoid leaking our internals
but some errors were meanigful for client. Example of such error is
"exceeded maximum number of parts". With this change we are wrapping
some important commit object errors with new ErrFailedPrecondition
error to be able to return it easily to uplink.
Change-Id: Id834b78362ed1920f0c3f6f1c7d9587bfd27e36a
With pending_objects table support enabled we missed passing correctly
expiration time from pending object to committed object. This change
updates commit query to take into account expiration time.
Change-Id: I67146d5b2f7f0bda02925d16275fbc59acb705bd
Merged bandwidth graph lines to show only allocated-dead for last 3 days and settled for other days.
Issue:
https://github.com/storj/storj/issues/6072
Change-Id: Ic7f03d22ccd82d27ae6e6a85e73e144c9852e33b
add descriptions for the endpoint that removes a user from the waning
state.
Issue: https://github.com/storj/storj/issues/6118
Change-Id: I211cd3c41c7fefa295d0db1b9f43f53e33b984e6
To avoid enabling feature for every project at once we would like to
do this partially and control percentage of projects that will have
feature enabled.
https://github.com/storj/storj/issues/6258
Change-Id: Iaac7c42d39da76ed2ecc439847c3b210462befa5
Currently it wasn't quite clear what was a stub version and an actual
version. Use a PendingVersion constant to make this distinction clear.
Also use PendingVersion = NextVersion = 0, that way it's clearer that
the version hasn't been yet determined. DefaultVersion = 1 might imply
that the object will get that version once commited, however that will
entirely depend on whether use-pending-objects is used or versioning is
enabled or not.
Change-Id: I21398141f97035c48c778f23b542266b834c44f1
API responses containing project information now contain the edge
service URL overrides configured for that project. The overrides are
based on the project's default placement.
References #6188
Change-Id: Ifc3dc74e75c0f5daf0419ac3be184415c65b202e
Protobuf definition is read to support getting specific version of
object so we just need to wire requested version into metainfo.GetObject
endpoint.
https://github.com/storj/storj/issues/6221
Change-Id: If4568b82119a6c893788a0a86e598b05ff5951cf
While adding support for pending_objects table one case was missed.
When we are uploading object to location where old objects exists
we are not removing old object segments at all. Old object is
overwritten with new object metadata but segments remains without
corresponding object. This fix removes all existing committed objects
(with it's segments) before committing new object.
https://github.com/storj/storj/issues/6255
Change-Id: Id657840edf763fd6aec8191788d819191b074fb7
By this change we don't allow users to add credit cards that are already bind to their account.
We still allow the same CC number but with a different expiration date.
Issue:
https://github.com/storj/storj/issues/5597
Change-Id: Ifeb0cc5ae0c2f0f7596af4dead70ae7d20d30613
This change adds a new endpoint that submits limit increase requests
to segment.
Issue: https://github.com/storj/storj/issues/6233
Change-Id: Ie4f70aef31079acbe2f24771b3ea359d5769eb95
If MaxObjectTTL is set in the API key, BeginObject will use it for the
object expiration time, unless an explicit ExpireAt is available in the
request.
Context: https://github.com/storj/storj/issues/6249
Change-Id: I2adf57d979a9c68eec3a787f3739d2f1dbec1f7e
This partially reverts commit 516241e406.
Endpoints are added to the backend, as there are some customers who may
use these endpoints, even though they are no longer necessary for the
satellite UI.
Change-Id: I52a99912d9eacf269fbb2ddca603e53c4af6d6bf
This reverts commit 31ec421299.
This change made the usages endpoint slower for accounts with large
number of projects.
Change-Id: I95870e95c2bf3bc3050087532fd0d20cbb50748b
With zombie deletion chore we are removing inactive pending objects from
objects table but new we need also to do this for pending_objects table.
https://github.com/storj/storj/issues/6050
Change-Id: Ia29116c103673a1d9e10c2f16654022572210a8a
The easiest way to get node information WITH node tags is executing two queries:
1. select all nodes
2. select all tags
And we can pair them with a loop, using the in-memory data structures.
But this approach does work only, if we select all nodes, which is true when we use cache (upload, download, repair checker).
But repair process selects only the required nodes, where this approach is suboptimal. (full table scan for all tags, even if we need only tags for a few dozens nodes).
Possible solutions:
1. We can introduce a cache for repair (similar to upload cache)
2. Or we can select both node and tag information with one query (join).
This patch implements the second approach.
Note: repair itself is quite slow (10-20 seconds per segements to repair). With 15 seconds execution time and 3 minutes cache staleness, we would use the cache only 12 times per worker. Probably we don't need cache for now.
https://github.com/storj/storj/issues/6198
Change-Id: I0364d94306e9815a1c280b71e843b8f504e3d870
A new field is introduced to grant.Permission in storj.io/common. Having
a direct cast here leads to compilation problems when bumping
storj.io/uplink to the latest storj.io/common. Avoiding the direct cast
resolves the issue.
Context: https://github.com/storj/storj/issues/6249
Change-Id: I3b9bc14ebcce8e192e218c621b996300753b8de4
This change does two things:
* allow using either public ID or private ID to do project-related
requests in admin UI
* allow passing a UUID string not containing dashes (i.e. a pure hex
string) in order to do project-related requests in admin UI
Change-Id: I4807a5d7252a48f4a09e3966c406645d55c856e2
With this change, we are able to fetch all objects to show in the object browser.
AWS SDK V3 provides paginator functionality to automatically make additional requests for every MaxKeys value (we use 500 objects at a time).
By initial request we fetch first 500 objects and save continuation tokens for the rest of the object batches.
Also, we save currently active (fetched) object range.
If user tries to open a page with objects which are out of currently active range then we look for needed continuation token and fetch needed objects batch.
Added a feature flag for this funtionality.
Issue:
https://github.com/storj/storj/issues/5595
Change-Id: If63e3c2ddaac3ea9f2bc1dc63cb49007f897e3e2
This change filters the list of invoices sent to the frontend to only
contain open or paid invoices.
Change-Id: I9ac2640a7587a76b0baf46d941f799e742aa2b3b
This change refactors the way requests are sent in console API tests,
placing identical logic in a dedicated function to reduce code
duplication.
Change-Id: I7a5ac42d8d68a3fd9a9f8b9d61775659234e883f
as GetParticipatingNodes and GetNodes, respectively.
We now want these functions to include offline and suspended nodes as
well, so that we can force immediate repair when pieces are out of
placement or in excluded countries. With that change, the old names no
longer made sense.
Change-Id: Icbcbad43dbde0ca8cbc80a4d17a896bb89b078b7
When checking if invited user is unverified, initialize oldest
with unverified row rather than empty User, because empty User
CreatedAt is zero, so no real user could be created earlier.
Change-Id: I74dd8f7fc82951cbb61071632a74b1a9443b41fe
Some errors were returned as metabase errors, not pure drpc
errors because of how rpcstatus.Code method is working. Status
code was returned for errors like metabase context canceled but
we would like to not leak our internals to the client.
Change-Id: I3f0194755f8d7359b1e3d342fa3be3d984019ecb
This change updates our content security policy to include the domain
storjapi.io and all of its subdomains.
References #6188
Change-Id: I6f3073bc32aa99626c54caf00bf07d2253ccbb8f
As I learned, the `Include` supposed to communicate that some internal change also "included" to the filters during the check -> filters might be stateful.
But it's not the case any more after 552242387, where we removed the only one stateful filter.
Change-Id: I7c36ddadb2defbfa3b6b67bcc115e4427ba9e083
This change enables the freezing/warning of users who use storjscan.
Issue: https://github.com/storj/storj/issues/6164
Change-Id: I7b00ee09d6527b3818b72326e9065c82ef5a2ac8
Once uppon a time, at the dawn of the implementation of Storj, when all the nodes are read from the database directly, every time.
After a while -- due to performance reasons -- it has been changed for upload and download: where all the nodes are read for a short period of time, and used from memory.
This is the version which was improved recently to support advanced node selections using placement.
But stil we have an old configuration value `service.config.NodeSelectionCache.Disabled`, and the db based implementation: `service.FindStorageNodesWithPreferences(ctx, req, &service.config.Node)`.
For safety, we need to remove this option, to make sure that we use the cache, which has the advanced features.
This patch was supposed to be a very small one (just removing a method and a config: https://review.dev.storj.io/c/storj/storj/+/11074/1/satellite/overlay/service.go), but it turned out that we need to update a lot of unit tests.
These unit tests used the old implementation (which is not used in production any more).
The tests which used both implementation are just updated to use only the new one
The tests which used only the old implementation are refactored (but keeping the test cases).
Using real unit tests (without DB, working on OSX, fast)
Closes https://github.com/storj/storj/issues/6217
Change-Id: I023f92c7e34235665cf8474513e67b2fcc4763eb
This change addresses an issue where the /charges endpoint will take a
while to respond due to a project having a large number of buckets.
The method and queries involved have been optimized and benchmarks show
a performance improvement.
test name old ms/op new ms/op
Postgres/sum_all_partner_usages 3.659 1.101
Postgres/individual_partner_usages 3.74 1.299
Cockroach/sum_all_partner_usages 7.201 2.872
Cockroach/individual_partner_usages 7.247 2.852
Issue: https://github.com/storj/storj-private/issues/277
Change-Id: Ia5082a2e1c3e91120a9db7b01c18847fe04574fe
This small feature will give us ability to test pending_objects table
without enabling it globally.
Change-Id: I802f45987ad329f94adfc0f02957c802b21d8251
table
New method IteratePendingObjectsByKeyNew is used to provide results for
metainfo.ListPendingObjectStreams. This endpoint is used to list
pending objects with the same object key. In this case to support
both tables (objects, pending_objects) we need to do one query per table
and merge results.
Because existing metainfo protobuf API is missing some fields to have
proper listing cursor we are not able to make ListPendingObjectStreams
correct for returning more than single page. We need to fix it
separately.
With this change also turns out that approach to merge results from
listing objects for ListObjects method was wrong and this change is also
fixing this problem.
Handling both tables will be removed at some point and only
pending_objects will be used to look for results.
Part of https://github.com/storj/storj/issues/6047
Change-Id: I8a88a6f885ad529704e6c032f1d97926123c2909
When an unverified user is sent a project invitation it contains a
registration link currently. Instead, send an activation link.
github issue: https://github.com/storj/storj/issues/6033
Change-Id: I54b88de8347a2532f7a85372c0c5e4df4bf4eb38
This change adds a new endpoint for listing invoices for billing history
This endpoint will replace the billing-history endpoint used on the
front end since were only interested in listing invoices.
Issue: https://github.com/storj/storj/issues/5479
Change-Id: I4730f5dc497245c6730e60b7f9986554479d1d3b
Adjust metainfo.ListObjects method to use IteratePendingObjects to
support new pending_objects table. New method will be used only when
we are listing pending objects.
Because until objects table will be free from pending objects we can
have results in both tables we are merging listing results. This also
means that in some (rare?) cases we may return more results than
specified listing limit. This situation is temporary.
Part of https://github.com/storj/storj/issues/6047
Change-Id: I06389145e5d916c532dfdbd3dcc9ef68ef70e515
Previously the base path for the API was hardcoded to `/api` and the
specified version.
This was not obvious that the generated code was setting that base path
and it was not flexible for serving the API under a different path than
`/api`.
We will likely need to set a different base path if we pretend to serve
the new back office API that we are going to implement alongside the
current admin API until the new back office is fully implemented and
verified that works properly.
This commit also fix add the base path of the endpoints to the
documentation because it was even more confusing for somebody that wants
to use the API having to find out them through looking to the generated
code.
Change-Id: I6efab6b6f3d295129d6f42f7fbba8c2dc19725f4
We would like to make it easier to accept multiple annotations.
Examples:
```
country("GB") && annotation(...)
annotated(annotated(X,...),...)
```
Change-Id: I92e622e8b985b314dadddf83b17976c245eb2069
This change fixes an issue where the project limit could be exceeded if
multiple project creation requests were sent sufficiently close to one
another. This could also be used to bypass project name duplication
checking.
Change-Id: I61cde7abaf25dedc5601c6870275de9938d7b949
Make the link more human-friendly - to contain the text of the group and
endpoint names.
Also link back to list of endpoints from each endpoint.
Change-Id: Ia3e2ebe20b58b5f60ecefe9d35fb8fd90dd4b4d7
This change fixes an issue where the console DB cleanup chore was never
able to run when using a Cockroach database implementation because of
an inappropriate AS OF SYSTEM TIME clause in the relevant methods.
Resolves#6197
Change-Id: I8456b6df2128678e0eebeb416eb1a955cc9bd706
This change adds tests to ensure critical endpoints are not able to be
called by users for other users. It asserts that if cases like that
do happen, a 401 response will be sent.
Issue: https://github.com/storj/storj-private/issues/407
Change-Id: I70097a80f691a7d0fcb0bc5dbce8291144177720
Personal users, like business users, should now be classified with
a lifecycle stage of PQL ("product qualified lead") instead of "other"
Change-Id: Iff5139043da1c8e75559302320ff9ca43ea956e5
This change makes it easier for someone reading the documentation to see
a full list of supported endpoints, and have direct links to the
details.
Change-Id: I46e2f809cfa2760845898eaa3d99db9066d435ef
Remove outdated information from the generated API readme, and add a
link to the generated documentation.
Change-Id: Icc098c81f235464344895d2195444044831aac63
In some rare cases when two entities are trying to create the same
bucket at the same time it's possible that we will return internal
error instead of `bucket already exists`. It's because we are not
handling correctly DB error about constraint error. This change checks
if while inserting bucket into DB we got constraint error and propagate
correct error to metainfo API.
Change-Id: Ie6fd2c943b864b4ea7d71e4a162e74dc3510e386
This patch is a oneliner: rangedloop checker should check the subnets only if it's not turned off with placement annotation.
(see in satellite/repair/checker/observer.go).
But I didn't find any unit test to cover that part, so I had to write one, and I prefered to write it as a unit test not an integration test, which requires a mock repair queue (observer_unit_test.go mock.go).
Because it's small change, I also included a small change: creating a elper method to check if AutoExcludeSubnet annotation is defined
Change-Id: I2666b937074ab57f603b356408ef108cd55bd6fd
10 --> node tag inclusion in raw format
11 --> same, but using same subnet is enabled
12 --> same as 11 but with US restrictions
Change-Id: I20792689e0caf5fe190f566a770d70c3b3824793
This change removes unused GraphQL code. It also updates storj sim code
to use the GraphQL replacement HTTP endpoints and removes the GraphQL
dependency.
Issue: https://github.com/storj/storj/issues/6142
Change-Id: Ie502553706c4b1282cd883a9275ea7332b8fc92d
Allow a longer encrypted key length to reduce 'key length is too big'
errors in gateway-mt. Gateway is enforcing an unencrypted key length
of 1024 bytes but when encrypted some keys are exceeding the current
limit.
Updates https://github.com/storj/gateway-mt/issues/335
Change-Id: I38a0fbb0843fd782aeadca85f9a202821421b5a2
This change fixes an issue where multiple unverified users with the
same email address could be created if registration requests were
sent sufficiently close to one another.
Resolves#6156
Change-Id: If8b1a145bcab842ace718119183de59947430463
It's quite straightforward change, and AFAIK graceful exit will be decommissioned very soon.
Therefore I didn't create big unit tests, yet. But I can be convinced to invest more time.
Change-Id: Ia588e516d7af5171fa47f9bab100edd3bf2b2cf9
Extends metabase.BucketEmpty logic to check also pending_objects
table for any entry.
https://github.com/storj/storj/issues/6057
Change-Id: Ia26c272de24a983b308a0b692e6bd5800487eb98
While deleting bucket we need also to delete pending objects from
pending_objects table.
Part of https://github.com/storj/storj/issues/6048
Change-Id: Icc83eaecf8388704e0b6329c397e8028debcf672
New metabase method IteratePendingObjectsByKeyNew to iterate
over entries in pending_objects table with the same object key.
Implementation and tests are mostly copy of code for
IteratePendingObjectsByKey. Main difference is that pending_objects
table have StreamID column part of primary key instead Version.
Method will be used to support new table in
metainfo.ListPendingObjectStreams request.
After full transition to pending_objects table we should remove 'New'
suffix from methods names.
Part of https://github.com/storj/storj/issues/6047
Change-Id: Ifc1ecbc534f8510fbd70c4ec676cf2bf8abb94cb
New metabase method IteratePendingObjects to iterate over entries in
pending_objects table. Implementation and tests are mostly copy of
code that we are using to iterate over objects table. Main difference
is that pending_objects table have StreamID column part of primary
key instead Version. Also structure of pending object is smaller
than the one from object table but it's a detail.
Method will be used to support new table in metainfo.ListObjects
request.
Next step will be to port rest of iterator implementation to support
pending_objects table in metainfo.ListPendingObjectStreams.
Part of https://github.com/storj/storj/issues/6047
Change-Id: Ia578182f88840539f3668d4a242953e061eace02
We are deleting pending objects while aborting multipart upload. We are
using metainfo BeginDeleteObject to do that. This change starts using
DeletePendingObjectNew to delete entry from pending_objects table when
request indicates that object is in this table.
Part of https://github.com/storj/storj/issues/6048
Change-Id: I4478a9c13c8e3db48dc5de3087ef03d1b4c47a5c
This change adds an HTTP endpoint for creating projects, to be used in
place of the GraphQL version.
Issue: https://github.com/storj/storj/issues/6195
Change-Id: I0377353418df7c152db6a935e99a3ea7ab4ce625
It's statefull, therefore it can hit naive users. (NodeFilters couldn't be reused for more than one iterations).
But looks like we don't need it, as `SelectBySubnet` doest the same job.
Change-Id: Ie85b7f9c2bd9a47293f4e3b359f8b619215c7649
This patch makes it easier to configure existing placement rules only with string.
1. placement(n) rule can be used to reuse earlier definitions
2 .&& can be used in addition to all(n1,n2)
3. country(c) accepts exclusions (like '!RU'), regions ('EU','EEA'), all and none
See the 'full example' unit test, which uses all of these, in a realistic example.
https://github.com/storj/storj/issues/6126
Change-Id: Ica76f016ebd002eb7ea8103d4258bacd6a6d77bf
When we check the availability of the pieces, we do:
```
result.NumUnhealthyRetrievable = len(result.ClumpedPiecesSet) + len(result.OutOfPlacementPiecesSet)
// + some magic if there are overlaps between them
numHealthy := len(pieces) - len(piecesCheck.MissingPiecesSet) - piecesCheck.NumUnhealthyRetrievable
```
This works only if OutOfPlacementPieceSet doesn't contain the offline nodes (which are already included in MissingPieceSet).
But `result.OutOfPlacementPieces.Set` should include all the nodes (even offline), as in case of lucky conditions, we are able to remove those pieces from DB.
The solution is to remove all offline nodes from `NumUnhealthyRetrievable`.
Change-Id: I90baa0396352dd040e1e1516314b3271f8712034
This patch fixes the node tag based placement of rangedloop/repairchecker + repair process.
The main change is just adding the node tags for Reliable and KnownReliabel database calls + adding new tests to prove, it works.
https://github.com/storj/storj/issues/6126
Change-Id: I245d654a18c1d61b2c72df49afa0718d0de76da1
There are cases when we would like to override the default placement=0 rule.
For example when we would like to exclude tagged nodes from the selection (by default).
Therefore we couldn't use a shortcut any more, we should always check the placement rules, even if we use placement=0.
TODO: we need to update common, and rename `EveryCountry` to `DefaultPlacement`, just to avoid confusion.
https://github.com/storj/storj/issues/6126
Change-Id: Iba6c655bd623e04351ea7ff91fd741785dc193e4
This change allows you to host the vuetify app on <x>.example.com where
the main app is hosted on example.com. A configuration is added to
specify an exact subdomain for cookies. For example, if my production
app is hosted on us1.storj.io and my vuetify app is hosted on
vuetify.us1.storj.io, the cookie domain should be set to ".us1.storj.io"
so that any authentication cookie is accessible to lower-level
subdomains.
Since the vuetify app does not currently support login/signup on its
own, it is still required to first login to the main satellite UI, then
navigate to the Vuetify app after the session cookie is set.
If the "vuetifypoc" prefix is not desirable when using subdomain hosting
for vuetify, the VITE_VUETIFY_PREFIX variable can be modified in
web/satellite/.env before running `npm run build-vuetify`. For now, we
should keep this prefix because it makes developing on the vuetify app
significantly easier if subdomains are not being used.
Issue: https://github.com/storj/storj/issues/6144
Change-Id: Iba1a5737892c8ee8f38148a17b94e3222f8798e6
This commit adds a new endpoint on the console api to delete project
members and invitations.
issue: https://github.com/storj/storj/issues/6136
Change-Id: I980bb97afd1ed2ed8f0f27cc2e8dc1d80d7eef05
This change adds an endpoint update projects, to be used in place of
the graphql alternative.
Issue: https://github.com/storj/storj/issues/6134
Change-Id: I26c04f4400f71721cbddb7f64405e6c9b78edb4d
This change introduces an HTTP endpoint for retrieving bucket usage
totals. In the future, this will replace its GraphQL counterpart.
References #6141
Change-Id: Ic6a0069a7e58b90dc2b6c55f164393f036c6acf4
This commit adds a new endpoint on the satellite console api to get
project members and invitations.
issue: https://github.com/storj/storj/issues/6137
Change-Id: I66cb064eeaffb1c34878462b3e6b3be8f3629f4e
This change adds an endpoint to get a user's projects, similar to
the OwnedProjects GraphQL query.
The console.ProjectInfo struct has been renamed to UpsertProjectInfo
to more accurately reflect its use.
Issue: https://github.com/storj/storj/issues/6135
Change-Id: I802fe4694a5cc75a9df2b565476f6e6f473431d4
With this change we are removing code responsible for deleting objects
and supporting server side copies created with references. In practice
we are restoring delete queries that we had before server side copy
implementation (with small exception, see bellow).
From deletion queries we are also removing parts with segment metadata
as result because we are not longer sending explicit delete requests to
storage nodes.
https://github.com/storj/storj/issues/5891
Change-Id: Iee4e7a9688cff27a60fb95d60dd233d996f41c85
We don't need to support segment copies with references anymore.
We migrated to copies where all metadata are copied from original
segment to copy.
https://github.com/storj/storj/issues/5891
Change-Id: Ic91dc21b0386ddf5c51aea45530024cd463e8ba9
This change adds an endpoint to delete API keys, similar to GraphQL mutation.
Issue:
https://github.com/storj/storj/issues/6140
Change-Id: Ia4a808222a057a199d803d8ea6b944c411a4dc8d
With this change we are removing code responsible to handle deleting
bucket and supporting server side copies created with references. In practice we are restoring delete queries that we had before server side copy implementation (with small exception, see bellow).
From deletion queries we are also removing parts with segment metadata
as result because we are not longer sending explicit delete requests to
storage nodes.
https://github.com/storj/storj/issues/5891
Change-Id: If866d9f3a1b01e9ebd9b49c4740a6425ba06dd43
This change adds an endpoint to create new API key, similar to GraphQL mutation.
Issue:
https://github.com/storj/storj/issues/6139
Change-Id: I2b35d680fa8e019666c811ad3bdf16201e3b8946
This change adds an endpoint to get paged project API keys, similar to GraphQL query.
Issue:
https://github.com/storj/storj/issues/6138
Change-Id: I5dea9e4ac61e798cc8a2e56a2755d722c1b66bfa
This change adds an endpoint to get a user's projects, similar to
the MyProjects GraphQL query.
Issue: https://github.com/storj/storj/issues/6132
Change-Id: I91feb5a1ee8c1231a8a5e6de9b8dc5b256f857c5
In the repair subsystem, it is necessary to acquire several extra
properties of nodes that are holding pieces of things or may be
selected to hold pieces. We need to know if a node is 'online' (the
definition of "online" may change somewhat depending on the situation),
if a node is in the process of graceful exit, and whether a node is
suspended. We can't just filter out nodes with all of these properties,
because sometimes we need to know properties about nodes even when the
nodes are suspended or gracefully exiting.
I thought the best way to do this was to add fields to SelectedNode,
and (to avoid any confusion) arrange for the added fields to be
populated wherever SelectedNode is returned, whether or not the new
fields are necessarily going to be used.
If people would rather I use a separate type from SelectedNode, I can do
that instead.
Change-Id: I7804a0e0a15cfe34c8ff47a227175ea5862a4ebc
Current node selection logic (in case of using SelectBySubnet):
1. selects one subnet randomly
2. selects one node randomly from the subnet
3. applies the placement NodeFilters to the node and ignore it, if doesn't match
This logic is wrong:
1. Imagine that we have a subnet with two DE and one GB nodes.
2. We would like to select DE nodes
2. In case of GB node is selected (randomly) in step2, step3 will ignore the subnet, even if there are good (DE) nodes in there.
Change-Id: I7673f52c89b46e0cc7b20a9b74137dc689d6c17e
Change is adjusting CommitObject to use `pending_objects` table to
commit object.
Satellite stream id is used to determine if we need to use
`pending_objects` or `objects` table during commit.
General goal is to support both tables until `objects` table will be
free from pending objects.
Part of https://github.com/storj/storj/issues/6046
Change-Id: I2ebe0cd6b446727c98c8e210d4d00504dd0dacb6
Change is adjusting CommitSegment to check pending object existence in
`pending_objects` or `objects` table.
Satellite stream id is used to determine if we need to use
`pending_objects` or `objects` table.
General goal is to support both tables until `objects` table will be
free from pending objects. Whenever it will be needed code will be
supporting both tables at once.
Part of https://github.com/storj/storj/issues/6046
Change-Id: I954444a53b4733ae6fc909420573242b02746787
Change is adjusting BeginSegment to check pending object existence in
`pending_objects` or `objects` table.
Satellite stream id is used to determine if we need to use
`pending_objects` or `objects` table.
General goal is to support both tables until `objects` table will be
free from pending objects. Whenever it will be needed code will be
supporting both tables at once.
Part of https://github.com/storj/storj/issues/6046
Change-Id: I08aaa605c23d82695fde352fdbd0a7fd11f46bb5
Change is adjusting BeginObjectNextVersion to create pending object in
`pending_objects` or `objects` table depends on configuration. This is
first change to move pending objects from objects table.
General goal is to support both tables until `objects` table will be
free from pending objects. Whenever it will be needed code will be
supporting both tables at once.
To be able to decide if we need to use `pending_objects` table or
`objects` table we extend satellite stream id to keep that information
for later use.
BeginObjectExactVersion will be not adjusted because at the moment it's
used only in tests.
Part of https://github.com/storj/storj/issues/6046
Change-Id: Ibf21965f63cca5e1775469994a29f1fd1261af4e
We don't need it any more, as CountryCode uses location.Set, which supports exclusion (`Without`).
Change-Id: Ie311ae19fefa0bc9a0161496af1233ef4a6607df
stats/size/count is not used by any production code, and it's not required, as we can assert the state with other checks.
real motivation: next commits will make the Selector of the State configurable, therefore we won't have one single Stat, it depends on the request parameters.
(we plan to support both network and id based randomization)
Change-Id: I631828fc0046d2fef5b7a674fc0268a0446e9655
This change extends the autofreeze chore to go through users who have
been warned/frozen to check if they have no failed invoices. If they do
not, this extension unwarns/unfreezes them.
Issue: https://github.com/storj/storj/issues/6077
Change-Id: I570b1d4b2e29574bd8b9ae37eb2d4fb41d178336
For some test queries we are using workaround to filter them out from
full table scan detection. To avoid confustion what is this all about
we are changing label to be more descriptive.
Change-Id: I41a744e8faf400e3e8de7e416d8f4242f9093fce
This change adds only schema definition of pending_objects table and
small amount of supporting code which will be useful for testing later.
With this table we would like to achieve two major things:
* simplify `objects` table, before we will start working on object
versioning
* gain performance by removing need to filter `objects` results with `status` column, which is not indexed and we would like to avoid that
https://github.com/storj/storj/issues/6045
Change-Id: I6097ce1c644a8a3dad13185915fe01989ad41d90
Another try to fix calculation of used bandwidth which is displayed on Project Dashboard.
This change sums up allocated-dead traffic for the last 3 days and settled traffic for the period which is earlier than 3 days ago.
Issue:
https://github.com/storj/storj-private/issues/293
Change-Id: I91e652eba69f81bd21e0d053ac170e2b926b3cb4
After infrastructure changes redis instance is not neccessay close
to core instance (where tally is calculated) and round trips to get
data from redis can be very costly. From less than hour calculation can take few hours for larger satellite.
This change combines 'segment' and 'storage' usage requests into
batches to reduce latency impact on tally calculation.
https://github.com/storj/storj/issues/5800
Change-Id: I87e57ec09e88fd167060a4ed51dc8b0274a095c5
This change enables the admin UI to remove the warning status of users.
resolves: storj-private/issues/342
Change-Id: Ib960ffb33fdabc045884ce7fa2c55c3553db0fb0
Fixed config value which indicates how many base units of US micro dollars are needed to auto upgrade user to paid tier.
Change-Id: I22821ac22fc3eaeeea21c6dec4e6912025df63aa
This change patches a loophole that allowed accounts to remove cards
that belong to other users.
Closes #storj/storj-private#326
Change-Id: I33e9efe5c9cdb03aa48ad4c6b1d3283c396a7890
Tally ensures that live accounting has the latest information,
however, when there are concurrent updates to live-accounting
it may by off by a few segments. Disable tally for those tests.
Change-Id: I6fa8a1794334bba093e18f29cb76e7b8d1244979
