We would like to make it easier to accept multiple annotations.
Examples:
```
country("GB") && annotation(...)
annotated(annotated(X,...),...)
```
Change-Id: I92e622e8b985b314dadddf83b17976c245eb2069
This change fixes an issue where the project limit could be exceeded if
multiple project creation requests were sent sufficiently close to one
another. This could also be used to bypass project name duplication
checking.
Change-Id: I61cde7abaf25dedc5601c6870275de9938d7b949
Make the link more human-friendly - to contain the text of the group and
endpoint names.
Also link back to list of endpoints from each endpoint.
Change-Id: Ia3e2ebe20b58b5f60ecefe9d35fb8fd90dd4b4d7
This change fixes an issue where the console DB cleanup chore was never
able to run when using a Cockroach database implementation because of
an inappropriate AS OF SYSTEM TIME clause in the relevant methods.
Resolves#6197
Change-Id: I8456b6df2128678e0eebeb416eb1a955cc9bd706
This change adds tests to ensure critical endpoints are not able to be
called by users for other users. It asserts that if cases like that
do happen, a 401 response will be sent.
Issue: https://github.com/storj/storj-private/issues/407
Change-Id: I70097a80f691a7d0fcb0bc5dbce8291144177720
Personal users, like business users, should now be classified with
a lifecycle stage of PQL ("product qualified lead") instead of "other"
Change-Id: Iff5139043da1c8e75559302320ff9ca43ea956e5
This change makes it easier for someone reading the documentation to see
a full list of supported endpoints, and have direct links to the
details.
Change-Id: I46e2f809cfa2760845898eaa3d99db9066d435ef
Remove outdated information from the generated API readme, and add a
link to the generated documentation.
Change-Id: Icc098c81f235464344895d2195444044831aac63
In some rare cases when two entities are trying to create the same
bucket at the same time it's possible that we will return internal
error instead of `bucket already exists`. It's because we are not
handling correctly DB error about constraint error. This change checks
if while inserting bucket into DB we got constraint error and propagate
correct error to metainfo API.
Change-Id: Ie6fd2c943b864b4ea7d71e4a162e74dc3510e386
This patch is a oneliner: rangedloop checker should check the subnets only if it's not turned off with placement annotation.
(see in satellite/repair/checker/observer.go).
But I didn't find any unit test to cover that part, so I had to write one, and I prefered to write it as a unit test not an integration test, which requires a mock repair queue (observer_unit_test.go mock.go).
Because it's small change, I also included a small change: creating a elper method to check if AutoExcludeSubnet annotation is defined
Change-Id: I2666b937074ab57f603b356408ef108cd55bd6fd
10 --> node tag inclusion in raw format
11 --> same, but using same subnet is enabled
12 --> same as 11 but with US restrictions
Change-Id: I20792689e0caf5fe190f566a770d70c3b3824793
This change removes unused GraphQL code. It also updates storj sim code
to use the GraphQL replacement HTTP endpoints and removes the GraphQL
dependency.
Issue: https://github.com/storj/storj/issues/6142
Change-Id: Ie502553706c4b1282cd883a9275ea7332b8fc92d
Allow a longer encrypted key length to reduce 'key length is too big'
errors in gateway-mt. Gateway is enforcing an unencrypted key length
of 1024 bytes but when encrypted some keys are exceeding the current
limit.
Updates https://github.com/storj/gateway-mt/issues/335
Change-Id: I38a0fbb0843fd782aeadca85f9a202821421b5a2
This change fixes an issue where multiple unverified users with the
same email address could be created if registration requests were
sent sufficiently close to one another.
Resolves#6156
Change-Id: If8b1a145bcab842ace718119183de59947430463
It's quite straightforward change, and AFAIK graceful exit will be decommissioned very soon.
Therefore I didn't create big unit tests, yet. But I can be convinced to invest more time.
Change-Id: Ia588e516d7af5171fa47f9bab100edd3bf2b2cf9
Extends metabase.BucketEmpty logic to check also pending_objects
table for any entry.
https://github.com/storj/storj/issues/6057
Change-Id: Ia26c272de24a983b308a0b692e6bd5800487eb98
While deleting bucket we need also to delete pending objects from
pending_objects table.
Part of https://github.com/storj/storj/issues/6048
Change-Id: Icc83eaecf8388704e0b6329c397e8028debcf672
New metabase method IteratePendingObjectsByKeyNew to iterate
over entries in pending_objects table with the same object key.
Implementation and tests are mostly copy of code for
IteratePendingObjectsByKey. Main difference is that pending_objects
table have StreamID column part of primary key instead Version.
Method will be used to support new table in
metainfo.ListPendingObjectStreams request.
After full transition to pending_objects table we should remove 'New'
suffix from methods names.
Part of https://github.com/storj/storj/issues/6047
Change-Id: Ifc1ecbc534f8510fbd70c4ec676cf2bf8abb94cb
New metabase method IteratePendingObjects to iterate over entries in
pending_objects table. Implementation and tests are mostly copy of
code that we are using to iterate over objects table. Main difference
is that pending_objects table have StreamID column part of primary
key instead Version. Also structure of pending object is smaller
than the one from object table but it's a detail.
Method will be used to support new table in metainfo.ListObjects
request.
Next step will be to port rest of iterator implementation to support
pending_objects table in metainfo.ListPendingObjectStreams.
Part of https://github.com/storj/storj/issues/6047
Change-Id: Ia578182f88840539f3668d4a242953e061eace02
We are deleting pending objects while aborting multipart upload. We are
using metainfo BeginDeleteObject to do that. This change starts using
DeletePendingObjectNew to delete entry from pending_objects table when
request indicates that object is in this table.
Part of https://github.com/storj/storj/issues/6048
Change-Id: I4478a9c13c8e3db48dc5de3087ef03d1b4c47a5c
This change adds an HTTP endpoint for creating projects, to be used in
place of the GraphQL version.
Issue: https://github.com/storj/storj/issues/6195
Change-Id: I0377353418df7c152db6a935e99a3ea7ab4ce625
It's statefull, therefore it can hit naive users. (NodeFilters couldn't be reused for more than one iterations).
But looks like we don't need it, as `SelectBySubnet` doest the same job.
Change-Id: Ie85b7f9c2bd9a47293f4e3b359f8b619215c7649
This patch makes it easier to configure existing placement rules only with string.
1. placement(n) rule can be used to reuse earlier definitions
2 .&& can be used in addition to all(n1,n2)
3. country(c) accepts exclusions (like '!RU'), regions ('EU','EEA'), all and none
See the 'full example' unit test, which uses all of these, in a realistic example.
https://github.com/storj/storj/issues/6126
Change-Id: Ica76f016ebd002eb7ea8103d4258bacd6a6d77bf
When we check the availability of the pieces, we do:
```
result.NumUnhealthyRetrievable = len(result.ClumpedPiecesSet) + len(result.OutOfPlacementPiecesSet)
// + some magic if there are overlaps between them
numHealthy := len(pieces) - len(piecesCheck.MissingPiecesSet) - piecesCheck.NumUnhealthyRetrievable
```
This works only if OutOfPlacementPieceSet doesn't contain the offline nodes (which are already included in MissingPieceSet).
But `result.OutOfPlacementPieces.Set` should include all the nodes (even offline), as in case of lucky conditions, we are able to remove those pieces from DB.
The solution is to remove all offline nodes from `NumUnhealthyRetrievable`.
Change-Id: I90baa0396352dd040e1e1516314b3271f8712034
This patch fixes the node tag based placement of rangedloop/repairchecker + repair process.
The main change is just adding the node tags for Reliable and KnownReliabel database calls + adding new tests to prove, it works.
https://github.com/storj/storj/issues/6126
Change-Id: I245d654a18c1d61b2c72df49afa0718d0de76da1
There are cases when we would like to override the default placement=0 rule.
For example when we would like to exclude tagged nodes from the selection (by default).
Therefore we couldn't use a shortcut any more, we should always check the placement rules, even if we use placement=0.
TODO: we need to update common, and rename `EveryCountry` to `DefaultPlacement`, just to avoid confusion.
https://github.com/storj/storj/issues/6126
Change-Id: Iba6c655bd623e04351ea7ff91fd741785dc193e4
This change allows you to host the vuetify app on <x>.example.com where
the main app is hosted on example.com. A configuration is added to
specify an exact subdomain for cookies. For example, if my production
app is hosted on us1.storj.io and my vuetify app is hosted on
vuetify.us1.storj.io, the cookie domain should be set to ".us1.storj.io"
so that any authentication cookie is accessible to lower-level
subdomains.
Since the vuetify app does not currently support login/signup on its
own, it is still required to first login to the main satellite UI, then
navigate to the Vuetify app after the session cookie is set.
If the "vuetifypoc" prefix is not desirable when using subdomain hosting
for vuetify, the VITE_VUETIFY_PREFIX variable can be modified in
web/satellite/.env before running `npm run build-vuetify`. For now, we
should keep this prefix because it makes developing on the vuetify app
significantly easier if subdomains are not being used.
Issue: https://github.com/storj/storj/issues/6144
Change-Id: Iba1a5737892c8ee8f38148a17b94e3222f8798e6
This commit adds a new endpoint on the console api to delete project
members and invitations.
issue: https://github.com/storj/storj/issues/6136
Change-Id: I980bb97afd1ed2ed8f0f27cc2e8dc1d80d7eef05
This change adds an endpoint update projects, to be used in place of
the graphql alternative.
Issue: https://github.com/storj/storj/issues/6134
Change-Id: I26c04f4400f71721cbddb7f64405e6c9b78edb4d
This change introduces an HTTP endpoint for retrieving bucket usage
totals. In the future, this will replace its GraphQL counterpart.
References #6141
Change-Id: Ic6a0069a7e58b90dc2b6c55f164393f036c6acf4
This commit adds a new endpoint on the satellite console api to get
project members and invitations.
issue: https://github.com/storj/storj/issues/6137
Change-Id: I66cb064eeaffb1c34878462b3e6b3be8f3629f4e
This change adds an endpoint to get a user's projects, similar to
the OwnedProjects GraphQL query.
The console.ProjectInfo struct has been renamed to UpsertProjectInfo
to more accurately reflect its use.
Issue: https://github.com/storj/storj/issues/6135
Change-Id: I802fe4694a5cc75a9df2b565476f6e6f473431d4
With this change we are removing code responsible for deleting objects
and supporting server side copies created with references. In practice
we are restoring delete queries that we had before server side copy
implementation (with small exception, see bellow).
From deletion queries we are also removing parts with segment metadata
as result because we are not longer sending explicit delete requests to
storage nodes.
https://github.com/storj/storj/issues/5891
Change-Id: Iee4e7a9688cff27a60fb95d60dd233d996f41c85
We don't need to support segment copies with references anymore.
We migrated to copies where all metadata are copied from original
segment to copy.
https://github.com/storj/storj/issues/5891
Change-Id: Ic91dc21b0386ddf5c51aea45530024cd463e8ba9
This change adds an endpoint to delete API keys, similar to GraphQL mutation.
Issue:
https://github.com/storj/storj/issues/6140
Change-Id: Ia4a808222a057a199d803d8ea6b944c411a4dc8d
With this change we are removing code responsible to handle deleting
bucket and supporting server side copies created with references. In practice we are restoring delete queries that we had before server side copy implementation (with small exception, see bellow).
From deletion queries we are also removing parts with segment metadata
as result because we are not longer sending explicit delete requests to
storage nodes.
https://github.com/storj/storj/issues/5891
Change-Id: If866d9f3a1b01e9ebd9b49c4740a6425ba06dd43
This change adds an endpoint to create new API key, similar to GraphQL mutation.
Issue:
https://github.com/storj/storj/issues/6139
Change-Id: I2b35d680fa8e019666c811ad3bdf16201e3b8946
This change adds an endpoint to get paged project API keys, similar to GraphQL query.
Issue:
https://github.com/storj/storj/issues/6138
Change-Id: I5dea9e4ac61e798cc8a2e56a2755d722c1b66bfa
This change adds an endpoint to get a user's projects, similar to
the MyProjects GraphQL query.
Issue: https://github.com/storj/storj/issues/6132
Change-Id: I91feb5a1ee8c1231a8a5e6de9b8dc5b256f857c5
In the repair subsystem, it is necessary to acquire several extra
properties of nodes that are holding pieces of things or may be
selected to hold pieces. We need to know if a node is 'online' (the
definition of "online" may change somewhat depending on the situation),
if a node is in the process of graceful exit, and whether a node is
suspended. We can't just filter out nodes with all of these properties,
because sometimes we need to know properties about nodes even when the
nodes are suspended or gracefully exiting.
I thought the best way to do this was to add fields to SelectedNode,
and (to avoid any confusion) arrange for the added fields to be
populated wherever SelectedNode is returned, whether or not the new
fields are necessarily going to be used.
If people would rather I use a separate type from SelectedNode, I can do
that instead.
Change-Id: I7804a0e0a15cfe34c8ff47a227175ea5862a4ebc
Current node selection logic (in case of using SelectBySubnet):
1. selects one subnet randomly
2. selects one node randomly from the subnet
3. applies the placement NodeFilters to the node and ignore it, if doesn't match
This logic is wrong:
1. Imagine that we have a subnet with two DE and one GB nodes.
2. We would like to select DE nodes
2. In case of GB node is selected (randomly) in step2, step3 will ignore the subnet, even if there are good (DE) nodes in there.
Change-Id: I7673f52c89b46e0cc7b20a9b74137dc689d6c17e
Change is adjusting CommitObject to use `pending_objects` table to
commit object.
Satellite stream id is used to determine if we need to use
`pending_objects` or `objects` table during commit.
General goal is to support both tables until `objects` table will be
free from pending objects.
Part of https://github.com/storj/storj/issues/6046
Change-Id: I2ebe0cd6b446727c98c8e210d4d00504dd0dacb6
Change is adjusting CommitSegment to check pending object existence in
`pending_objects` or `objects` table.
Satellite stream id is used to determine if we need to use
`pending_objects` or `objects` table.
General goal is to support both tables until `objects` table will be
free from pending objects. Whenever it will be needed code will be
supporting both tables at once.
Part of https://github.com/storj/storj/issues/6046
Change-Id: I954444a53b4733ae6fc909420573242b02746787
Change is adjusting BeginSegment to check pending object existence in
`pending_objects` or `objects` table.
Satellite stream id is used to determine if we need to use
`pending_objects` or `objects` table.
General goal is to support both tables until `objects` table will be
free from pending objects. Whenever it will be needed code will be
supporting both tables at once.
Part of https://github.com/storj/storj/issues/6046
Change-Id: I08aaa605c23d82695fde352fdbd0a7fd11f46bb5
Change is adjusting BeginObjectNextVersion to create pending object in
`pending_objects` or `objects` table depends on configuration. This is
first change to move pending objects from objects table.
General goal is to support both tables until `objects` table will be
free from pending objects. Whenever it will be needed code will be
supporting both tables at once.
To be able to decide if we need to use `pending_objects` table or
`objects` table we extend satellite stream id to keep that information
for later use.
BeginObjectExactVersion will be not adjusted because at the moment it's
used only in tests.
Part of https://github.com/storj/storj/issues/6046
Change-Id: Ibf21965f63cca5e1775469994a29f1fd1261af4e
We don't need it any more, as CountryCode uses location.Set, which supports exclusion (`Without`).
Change-Id: Ie311ae19fefa0bc9a0161496af1233ef4a6607df
stats/size/count is not used by any production code, and it's not required, as we can assert the state with other checks.
real motivation: next commits will make the Selector of the State configurable, therefore we won't have one single Stat, it depends on the request parameters.
(we plan to support both network and id based randomization)
Change-Id: I631828fc0046d2fef5b7a674fc0268a0446e9655
This change extends the autofreeze chore to go through users who have
been warned/frozen to check if they have no failed invoices. If they do
not, this extension unwarns/unfreezes them.
Issue: https://github.com/storj/storj/issues/6077
Change-Id: I570b1d4b2e29574bd8b9ae37eb2d4fb41d178336
For some test queries we are using workaround to filter them out from
full table scan detection. To avoid confustion what is this all about
we are changing label to be more descriptive.
Change-Id: I41a744e8faf400e3e8de7e416d8f4242f9093fce
This change adds only schema definition of pending_objects table and
small amount of supporting code which will be useful for testing later.
With this table we would like to achieve two major things:
* simplify `objects` table, before we will start working on object
versioning
* gain performance by removing need to filter `objects` results with `status` column, which is not indexed and we would like to avoid that
https://github.com/storj/storj/issues/6045
Change-Id: I6097ce1c644a8a3dad13185915fe01989ad41d90
Another try to fix calculation of used bandwidth which is displayed on Project Dashboard.
This change sums up allocated-dead traffic for the last 3 days and settled traffic for the period which is earlier than 3 days ago.
Issue:
https://github.com/storj/storj-private/issues/293
Change-Id: I91e652eba69f81bd21e0d053ac170e2b926b3cb4
After infrastructure changes redis instance is not neccessay close
to core instance (where tally is calculated) and round trips to get
data from redis can be very costly. From less than hour calculation can take few hours for larger satellite.
This change combines 'segment' and 'storage' usage requests into
batches to reduce latency impact on tally calculation.
https://github.com/storj/storj/issues/5800
Change-Id: I87e57ec09e88fd167060a4ed51dc8b0274a095c5
This change enables the admin UI to remove the warning status of users.
resolves: storj-private/issues/342
Change-Id: Ib960ffb33fdabc045884ce7fa2c55c3553db0fb0
Fixed config value which indicates how many base units of US micro dollars are needed to auto upgrade user to paid tier.
Change-Id: I22821ac22fc3eaeeea21c6dec4e6912025df63aa
This change patches a loophole that allowed accounts to remove cards
that belong to other users.
Closes #storj/storj-private#326
Change-Id: I33e9efe5c9cdb03aa48ad4c6b1d3283c396a7890
Tally ensures that live accounting has the latest information,
however, when there are concurrent updates to live-accounting
it may by off by a few segments. Disable tally for those tests.
Change-Id: I6fa8a1794334bba093e18f29cb76e7b8d1244979
Added new functionality to query storjscan for all wallet transactions (including pending).
Added new endpoint to query all wallet transactions.
Issue:
https://github.com/storj/storj/issues/5978
Change-Id: Id15fddfc9c95efcaa32aa21403cb177f9297e1ab
Added new observer for billing chore to check user's balance and upgrade their account if balance is more than or equal to needed amount for upgrade.
Added new config value which stands for needed amount of base units of US micro dollars needed to upgrade user.
Issue:
https://github.com/storj/storj/issues/5978
Change-Id: Ic3992cd3114397bfdd9e231ca090ff21ca66648b
This change adds an extra step to the auto freeze chore to attempt
payment before freezing/warning a user.
It also attempts payment after modifying user's cards whether the user
is frozen/warned or not.
Issue: https://github.com/storj/storj-private/issues/341
Change-Id: Ia9c0c5a2d37837bca5153fe720fef61f1385cb15
This feature flag was disabled by default to test it slowly. Its enabled
for some time on one production satellite and test satellites without
any issue. We can enable it by default in code.
Change-Id: If9c36895bbbea12bd4aefa30cb4df912e1729e4c
We are doing full bucket name validation for many requests but
we should do this only while creating bucket. Other requests will be
covered only by basic name length validation. Less strict validation for
other requests will make bucket usable in case of invalid bucket names
in DB (we have such cases from the past).
https://github.com/storj/storj/issues/6044
Change-Id: I3a41050e3637787f788705ef15b5dc4df4d01fc6
Sometimes DownloadSelectionCache doesn't keep up with all node
placement changes we are doing during this test.
Change-Id: Idbda6511e3324b560cee3be85f980bf8d5b9b7ef
This change adds request IDs to requests, logs them as part of audit
logs and sends to the client on error. This is to improve debugging
of customer issues.
Issue: https://github.com/storj/storj/issues/5898
Change-Id: I801514b547d28d810552d91aa7c8502051e552bf
Before this change, if a user creates a bucket with a user_agent attributed then deletes and recreates it, the row in bucket_metainfos
will not have the user_agent. This is because we skip setting the field
in bucket_metainfos if the bucket already exists in value_attributions.
This can be problematic, as we return the bucket's user agent during the
ListBuckets operation, and the client may be expecting this value to be
populated.
This change ensures the bucket table user_agent is set when (re)creating a bucket. To avoid decreasing BeginObject performance, which also
updates attribution, a flag has been added to determine whether to
make sure the buckets table is updated: `forceBucketUpdate`.
Change-Id: Iada2f233b327b292ad9f98c73ea76a1b0113c926
Segments loop implementation is using lots of memory to convert
alias pieces to pieces for each segment while iteration. To improve
situation this change is reusing Pieces between batch pages. This
should signifcantly reduce memory usage for ranged loop executions.
Change-Id: I469188779908facb19ad85c6bb7bc3657111cc9a
This change creates the ability to run a server separate from the
console web server to serve the front end app. You can run it with
`satellite run ui`. Since there are now potentially two servers instead
of one, the UI server has the option to act as a reverse proxy to the
api server for local development by setting `--console.address` to the
console backend address and `--console.backend-reverse-proxy` to the
console backend's http url. Also, a feature flag has been implemented
on the api server to retain the ability to serve the front end app. It
is toggled with `--console.frontend-enable`.
github issue: https://github.com/storj/storj/issues/5843
Change-Id: I0d30451a20636e3184110dbe28c8a2a8a9505804
* Handle failed country code conversion.
* Avoid potential issues with a data-race due to shared slice.
Updates #6028
Change-Id: If7beef2619abd084e1f4109de2d323f834a6090a
Having separate process/pod only for sending bloom filters once a week
is a bit waste. After reconfiguring production settings to use sender in
core we can remove also GC sender peer code.
https://github.com/storj/storj/issues/5979
Change-Id: I6efe3ec073f96545e1f70ad13843f8ccdf923ee8
* Fixes backend to use only a user's owned projects to determine if the
user has hit the project limit
* Makes frontend logic consistent (and simpler) for checking whether to
send user to the "Create Project" modal or the "upgrade account or
request limit increase" modal
Before this change, projects that a user is a member of would be
included in determining whether the user could create a project. Also,
the "create project" button in the projects menu in the navbar of the UI
did not enable a free tier user to create a new project, even if they
had not hit their limits.
Change-Id: Ia776eb627ca37b83f5bc63bed83ee83c9f7cc789
Add a configuration (default true) to exclude users who have made
storjscan payments from being auto-warned/frozen for an unpaid invoice.
This will allow us to reach out to these users and handle warning/freezing
manually. Auto account freeze still handles CC-only users.
Fixes https://github.com/storj/storj/issues/6027
Change-Id: I0c862785dad1c8febfa11100c0d30e621ce3ae9b
This change fixes an issue where a new project member invitation would
silently replace an older one that has the same project ID and email if
the email did not belong to a registered user. Additionally, the
satellite frontend has been updated to display more descriptive error
messages for project member invitations.
Change-Id: I32b582c40c0028b8eedf2aed4b5bfb43501594b4
