Some errors were returned as metabase errors, not pure drpc
errors because of how rpcstatus.Code method is working. Status
code was returned for errors like metabase context canceled but
we would like to not leak our internals to the client.
Change-Id: I3f0194755f8d7359b1e3d342fa3be3d984019ecb
This change updates our content security policy to include the domain
storjapi.io and all of its subdomains.
References #6188
Change-Id: I6f3073bc32aa99626c54caf00bf07d2253ccbb8f
As I learned, the `Include` supposed to communicate that some internal change also "included" to the filters during the check -> filters might be stateful.
But it's not the case any more after 552242387, where we removed the only one stateful filter.
Change-Id: I7c36ddadb2defbfa3b6b67bcc115e4427ba9e083
This change enables the freezing/warning of users who use storjscan.
Issue: https://github.com/storj/storj/issues/6164
Change-Id: I7b00ee09d6527b3818b72326e9065c82ef5a2ac8
Once uppon a time, at the dawn of the implementation of Storj, when all the nodes are read from the database directly, every time.
After a while -- due to performance reasons -- it has been changed for upload and download: where all the nodes are read for a short period of time, and used from memory.
This is the version which was improved recently to support advanced node selections using placement.
But stil we have an old configuration value `service.config.NodeSelectionCache.Disabled`, and the db based implementation: `service.FindStorageNodesWithPreferences(ctx, req, &service.config.Node)`.
For safety, we need to remove this option, to make sure that we use the cache, which has the advanced features.
This patch was supposed to be a very small one (just removing a method and a config: https://review.dev.storj.io/c/storj/storj/+/11074/1/satellite/overlay/service.go), but it turned out that we need to update a lot of unit tests.
These unit tests used the old implementation (which is not used in production any more).
The tests which used both implementation are just updated to use only the new one
The tests which used only the old implementation are refactored (but keeping the test cases).
Using real unit tests (without DB, working on OSX, fast)
Closes https://github.com/storj/storj/issues/6217
Change-Id: I023f92c7e34235665cf8474513e67b2fcc4763eb
This change addresses an issue where the /charges endpoint will take a
while to respond due to a project having a large number of buckets.
The method and queries involved have been optimized and benchmarks show
a performance improvement.
test name old ms/op new ms/op
Postgres/sum_all_partner_usages 3.659 1.101
Postgres/individual_partner_usages 3.74 1.299
Cockroach/sum_all_partner_usages 7.201 2.872
Cockroach/individual_partner_usages 7.247 2.852
Issue: https://github.com/storj/storj-private/issues/277
Change-Id: Ia5082a2e1c3e91120a9db7b01c18847fe04574fe
This small feature will give us ability to test pending_objects table
without enabling it globally.
Change-Id: I802f45987ad329f94adfc0f02957c802b21d8251
table
New method IteratePendingObjectsByKeyNew is used to provide results for
metainfo.ListPendingObjectStreams. This endpoint is used to list
pending objects with the same object key. In this case to support
both tables (objects, pending_objects) we need to do one query per table
and merge results.
Because existing metainfo protobuf API is missing some fields to have
proper listing cursor we are not able to make ListPendingObjectStreams
correct for returning more than single page. We need to fix it
separately.
With this change also turns out that approach to merge results from
listing objects for ListObjects method was wrong and this change is also
fixing this problem.
Handling both tables will be removed at some point and only
pending_objects will be used to look for results.
Part of https://github.com/storj/storj/issues/6047
Change-Id: I8a88a6f885ad529704e6c032f1d97926123c2909
When an unverified user is sent a project invitation it contains a
registration link currently. Instead, send an activation link.
github issue: https://github.com/storj/storj/issues/6033
Change-Id: I54b88de8347a2532f7a85372c0c5e4df4bf4eb38
This change adds a new endpoint for listing invoices for billing history
This endpoint will replace the billing-history endpoint used on the
front end since were only interested in listing invoices.
Issue: https://github.com/storj/storj/issues/5479
Change-Id: I4730f5dc497245c6730e60b7f9986554479d1d3b
Adjust metainfo.ListObjects method to use IteratePendingObjects to
support new pending_objects table. New method will be used only when
we are listing pending objects.
Because until objects table will be free from pending objects we can
have results in both tables we are merging listing results. This also
means that in some (rare?) cases we may return more results than
specified listing limit. This situation is temporary.
Part of https://github.com/storj/storj/issues/6047
Change-Id: I06389145e5d916c532dfdbd3dcc9ef68ef70e515
Previously the base path for the API was hardcoded to `/api` and the
specified version.
This was not obvious that the generated code was setting that base path
and it was not flexible for serving the API under a different path than
`/api`.
We will likely need to set a different base path if we pretend to serve
the new back office API that we are going to implement alongside the
current admin API until the new back office is fully implemented and
verified that works properly.
This commit also fix add the base path of the endpoints to the
documentation because it was even more confusing for somebody that wants
to use the API having to find out them through looking to the generated
code.
Change-Id: I6efab6b6f3d295129d6f42f7fbba8c2dc19725f4
We would like to make it easier to accept multiple annotations.
Examples:
```
country("GB") && annotation(...)
annotated(annotated(X,...),...)
```
Change-Id: I92e622e8b985b314dadddf83b17976c245eb2069
This change fixes an issue where the project limit could be exceeded if
multiple project creation requests were sent sufficiently close to one
another. This could also be used to bypass project name duplication
checking.
Change-Id: I61cde7abaf25dedc5601c6870275de9938d7b949
Make the link more human-friendly - to contain the text of the group and
endpoint names.
Also link back to list of endpoints from each endpoint.
Change-Id: Ia3e2ebe20b58b5f60ecefe9d35fb8fd90dd4b4d7
This change fixes an issue where the console DB cleanup chore was never
able to run when using a Cockroach database implementation because of
an inappropriate AS OF SYSTEM TIME clause in the relevant methods.
Resolves#6197
Change-Id: I8456b6df2128678e0eebeb416eb1a955cc9bd706
This change adds tests to ensure critical endpoints are not able to be
called by users for other users. It asserts that if cases like that
do happen, a 401 response will be sent.
Issue: https://github.com/storj/storj-private/issues/407
Change-Id: I70097a80f691a7d0fcb0bc5dbce8291144177720
Personal users, like business users, should now be classified with
a lifecycle stage of PQL ("product qualified lead") instead of "other"
Change-Id: Iff5139043da1c8e75559302320ff9ca43ea956e5
This change makes it easier for someone reading the documentation to see
a full list of supported endpoints, and have direct links to the
details.
Change-Id: I46e2f809cfa2760845898eaa3d99db9066d435ef
Remove outdated information from the generated API readme, and add a
link to the generated documentation.
Change-Id: Icc098c81f235464344895d2195444044831aac63
In some rare cases when two entities are trying to create the same
bucket at the same time it's possible that we will return internal
error instead of `bucket already exists`. It's because we are not
handling correctly DB error about constraint error. This change checks
if while inserting bucket into DB we got constraint error and propagate
correct error to metainfo API.
Change-Id: Ie6fd2c943b864b4ea7d71e4a162e74dc3510e386
This patch is a oneliner: rangedloop checker should check the subnets only if it's not turned off with placement annotation.
(see in satellite/repair/checker/observer.go).
But I didn't find any unit test to cover that part, so I had to write one, and I prefered to write it as a unit test not an integration test, which requires a mock repair queue (observer_unit_test.go mock.go).
Because it's small change, I also included a small change: creating a elper method to check if AutoExcludeSubnet annotation is defined
Change-Id: I2666b937074ab57f603b356408ef108cd55bd6fd
10 --> node tag inclusion in raw format
11 --> same, but using same subnet is enabled
12 --> same as 11 but with US restrictions
Change-Id: I20792689e0caf5fe190f566a770d70c3b3824793
This change removes unused GraphQL code. It also updates storj sim code
to use the GraphQL replacement HTTP endpoints and removes the GraphQL
dependency.
Issue: https://github.com/storj/storj/issues/6142
Change-Id: Ie502553706c4b1282cd883a9275ea7332b8fc92d
Allow a longer encrypted key length to reduce 'key length is too big'
errors in gateway-mt. Gateway is enforcing an unencrypted key length
of 1024 bytes but when encrypted some keys are exceeding the current
limit.
Updates https://github.com/storj/gateway-mt/issues/335
Change-Id: I38a0fbb0843fd782aeadca85f9a202821421b5a2
This change fixes an issue where multiple unverified users with the
same email address could be created if registration requests were
sent sufficiently close to one another.
Resolves#6156
Change-Id: If8b1a145bcab842ace718119183de59947430463
It's quite straightforward change, and AFAIK graceful exit will be decommissioned very soon.
Therefore I didn't create big unit tests, yet. But I can be convinced to invest more time.
Change-Id: Ia588e516d7af5171fa47f9bab100edd3bf2b2cf9
Extends metabase.BucketEmpty logic to check also pending_objects
table for any entry.
https://github.com/storj/storj/issues/6057
Change-Id: Ia26c272de24a983b308a0b692e6bd5800487eb98
While deleting bucket we need also to delete pending objects from
pending_objects table.
Part of https://github.com/storj/storj/issues/6048
Change-Id: Icc83eaecf8388704e0b6329c397e8028debcf672
New metabase method IteratePendingObjectsByKeyNew to iterate
over entries in pending_objects table with the same object key.
Implementation and tests are mostly copy of code for
IteratePendingObjectsByKey. Main difference is that pending_objects
table have StreamID column part of primary key instead Version.
Method will be used to support new table in
metainfo.ListPendingObjectStreams request.
After full transition to pending_objects table we should remove 'New'
suffix from methods names.
Part of https://github.com/storj/storj/issues/6047
Change-Id: Ifc1ecbc534f8510fbd70c4ec676cf2bf8abb94cb
New metabase method IteratePendingObjects to iterate over entries in
pending_objects table. Implementation and tests are mostly copy of
code that we are using to iterate over objects table. Main difference
is that pending_objects table have StreamID column part of primary
key instead Version. Also structure of pending object is smaller
than the one from object table but it's a detail.
Method will be used to support new table in metainfo.ListObjects
request.
Next step will be to port rest of iterator implementation to support
pending_objects table in metainfo.ListPendingObjectStreams.
Part of https://github.com/storj/storj/issues/6047
Change-Id: Ia578182f88840539f3668d4a242953e061eace02
We are deleting pending objects while aborting multipart upload. We are
using metainfo BeginDeleteObject to do that. This change starts using
DeletePendingObjectNew to delete entry from pending_objects table when
request indicates that object is in this table.
Part of https://github.com/storj/storj/issues/6048
Change-Id: I4478a9c13c8e3db48dc5de3087ef03d1b4c47a5c
This change adds an HTTP endpoint for creating projects, to be used in
place of the GraphQL version.
Issue: https://github.com/storj/storj/issues/6195
Change-Id: I0377353418df7c152db6a935e99a3ea7ab4ce625
It's statefull, therefore it can hit naive users. (NodeFilters couldn't be reused for more than one iterations).
But looks like we don't need it, as `SelectBySubnet` doest the same job.
Change-Id: Ie85b7f9c2bd9a47293f4e3b359f8b619215c7649
This patch makes it easier to configure existing placement rules only with string.
1. placement(n) rule can be used to reuse earlier definitions
2 .&& can be used in addition to all(n1,n2)
3. country(c) accepts exclusions (like '!RU'), regions ('EU','EEA'), all and none
See the 'full example' unit test, which uses all of these, in a realistic example.
https://github.com/storj/storj/issues/6126
Change-Id: Ica76f016ebd002eb7ea8103d4258bacd6a6d77bf
When we check the availability of the pieces, we do:
```
result.NumUnhealthyRetrievable = len(result.ClumpedPiecesSet) + len(result.OutOfPlacementPiecesSet)
// + some magic if there are overlaps between them
numHealthy := len(pieces) - len(piecesCheck.MissingPiecesSet) - piecesCheck.NumUnhealthyRetrievable
```
This works only if OutOfPlacementPieceSet doesn't contain the offline nodes (which are already included in MissingPieceSet).
But `result.OutOfPlacementPieces.Set` should include all the nodes (even offline), as in case of lucky conditions, we are able to remove those pieces from DB.
The solution is to remove all offline nodes from `NumUnhealthyRetrievable`.
Change-Id: I90baa0396352dd040e1e1516314b3271f8712034
This patch fixes the node tag based placement of rangedloop/repairchecker + repair process.
The main change is just adding the node tags for Reliable and KnownReliabel database calls + adding new tests to prove, it works.
https://github.com/storj/storj/issues/6126
Change-Id: I245d654a18c1d61b2c72df49afa0718d0de76da1
There are cases when we would like to override the default placement=0 rule.
For example when we would like to exclude tagged nodes from the selection (by default).
Therefore we couldn't use a shortcut any more, we should always check the placement rules, even if we use placement=0.
TODO: we need to update common, and rename `EveryCountry` to `DefaultPlacement`, just to avoid confusion.
https://github.com/storj/storj/issues/6126
Change-Id: Iba6c655bd623e04351ea7ff91fd741785dc193e4
This change allows you to host the vuetify app on <x>.example.com where
the main app is hosted on example.com. A configuration is added to
specify an exact subdomain for cookies. For example, if my production
app is hosted on us1.storj.io and my vuetify app is hosted on
vuetify.us1.storj.io, the cookie domain should be set to ".us1.storj.io"
so that any authentication cookie is accessible to lower-level
subdomains.
Since the vuetify app does not currently support login/signup on its
own, it is still required to first login to the main satellite UI, then
navigate to the Vuetify app after the session cookie is set.
If the "vuetifypoc" prefix is not desirable when using subdomain hosting
for vuetify, the VITE_VUETIFY_PREFIX variable can be modified in
web/satellite/.env before running `npm run build-vuetify`. For now, we
should keep this prefix because it makes developing on the vuetify app
significantly easier if subdomains are not being used.
Issue: https://github.com/storj/storj/issues/6144
Change-Id: Iba1a5737892c8ee8f38148a17b94e3222f8798e6
This commit adds a new endpoint on the console api to delete project
members and invitations.
issue: https://github.com/storj/storj/issues/6136
Change-Id: I980bb97afd1ed2ed8f0f27cc2e8dc1d80d7eef05
