When we switched to segments loop we stopped
adding bucket name and project ID to order limit
metadata. This is causing lots of logging that we don't need.
Change-Id: Id3f878a170b7a6b801e8a838ee69165715985d60
For backward compatibility we are overriding pb.StreamMeta
values returned as encrypted metadata. It turns out that we
should do it not when target values are missing but when
values to override exists.
This was causing problems after move operation. Details can be
found here https://github.com/storj/uplink/issues/70
Backward compatibility test will be added to storj/uplink testsuite.
Change-Id: I72e7a01226b1dd62902cb0d6ebb1ff91a4693005
We want to set maximum number of segments per project. This change will
add functionality to get number of segments currently used by project.
To avoid often DB calls segment cound will be cached and refreshed every
few minutes.
Change-Id: I2ecb6484f5afc3875c0e0dfaea360e8872f9d196
Exposes functionality to get and update project segment
limit. It will be used to limit number of segments per project
while uploading object.
Change-Id: I971d48eebb4e7db8b01535c3091829e73437f48d
Previously, only valid partner IDs could be used for bucket level value attribution. Now that any useragent byte slice can be used, we should allow for empty useragent strings to be stored rather than throwing an error or leaving the bucket with no attribution.
Change-Id: I7043f835588dab1c401a27e31afd74b6b5a3e44b
Currently detecting if error NotFound is for object
or bucket is very fragile on uplink side. We cannot
change this messages for now.
Change-Id: I6ec4d98116477812f031134e4f1c9e73bdce8b27
We want to set maximum number of segments per
project. This change adds only column to projects table.
Default value 1M is set to make later migration easier as
we need to set 1M for paid tier users and 140K for free
tier users.
Change-Id: I8e83712e08c5bd91dfa59f652d17e45c14240a36
Added new query to get project object and segment count.
Added appropriate object and segment count view for new project dashboard.
Change-Id: I69a2e55442f318c51dc365c0c578b964f2f06c7f
Move an endpoint that was classified to return the geofence
configuration of a bucket to return the bucket information, which
also include the geofence configuration, because it's what it was
returning.
Change-Id: I0e0a6aac330296383a50a92d2352df9088df77d5
The satellite admin server was augmented with 3 new endpoints to manage
buckets geofencing configurations, however, they weren't created in the
admin UI API class for making them available in the web interface.
This commit adds these new endpoints to the admin UI.
Change-Id: If060d1f10a3bc6c365e16a891673d4ffc89e4b41
This reverts commit 2c0a360a14.
Avoid big transactions. We'll do it outside of the migration pipeline.
Change-Id: Iade810d81bb2453c9e351149cb84662b207ee527
Value attribution codes were converted into UUIDs and stored in the users, projects, api_keys, bucket_metainfos, and value_attributions tables in the partner_id column. This migration will lookup the appropriate partner name associated with each of these UUIDs, and store the partner name directly in the user_agent column within each table. If an error occurs during the partner ID to partner name conversion, the partner ID value will be migrated to user_agent.
A note on the migration test data, postgres.v182.sql:
With one exception, all preexisting rows in the relevant tables had a NULL partner_ID. Therefore, we needed to insert new rows with partner_ID set under the OLD DATA section in order to test that the migration works. For each affected table, we insert one row with a valid partner ID which has a corresponding partner name, and one row with a partner ID which would return an error during the conversion to the partner name.
Change-Id: Iad977d72df0ce95a0c5ca80a065c4276ec1f2354
This new method will be used with mechanism to limit number
of segments per project, similar to limiting buckets or bandwidth.
This is only one of multiple changes we will do to implement this
limitation.
Change-Id: Ia516c2a006ad1d7b4431d780679be9d809848f4b
Check if the context is done at the beginning of the download object and
segment and return right away if it's the case.
Check if Redis returned an error due to context cancellation for not
logging the error.
Change some logging messages of Redis errors to reflect on them if the
error happens while downloading an object or a segment.
Change-Id: I8ed8ff9ff7bb170b560f41356ea06820ce6c4e12
If satellite can't find enough nodes to successfully download a segment,
it probably is not the fault of storage nodes.
Change-Id: I681f66056df0bb940da9edb3a7dbb3658c0a56cb
Tests for checking different scenarios for enabling and disabling
geofence config on empty and non-empty buckets.
Change-Id: I0fe9abb1008d2daee660f22ffe4defe6226b9aa7
The main motivation is to wrap the bucket DB and metainfo DB, so we
could check if a bucket is empty before applying geofencing config.
Change-Id: I8bac21555e01d51a663fb557bc1acfc8106bc2e1
Add the project ID to error logs messages about not being able to
retrieve and track storage or bandwidth usage.
Some error logs related to these errors already contained the project ID
but others didn't, having it in the ones that didn't make them more
consistent and will provide more info which may be helpful for
troubleshooting them.
Change-Id: Ia9fc707a7f3aff0867645bb941badc199c2bf832
We implemented filtering of system metadata to
reduce DB load in case user will need only list of
objects but we missed that encryption configuration
is needed to decrypt object key.
This change is always including encryption configuration into
list objects results.
Change-Id: Iaf7b3d8441480e3ad787f19a28f1b648584d5f7d
Don't log as an Error when users make request that cannot be fulfilled
because they exceed their storage or bandwidth limits.
Those are not system errors and the service is handling them correctly
and showing them in the logs an "ERROR" is misleading.
Change-Id: Iac642b7e8ba92840bb943192ad0694b5f4930258
To allow for changing limits for new users, while leaving existing users limits as they are, we must store the project limits for each user. We currently store the limit for the number of projects a user can create in the user DB table. This change would also store the project bandwidth and storage limits in the same table.
Change-Id: If8d79b39de020b969f3445ef2fcc370e51d706c6
Change the satellite Admin HTTP server for:
* Embedding the UI assets into the Go binary.
* Serve the UI assets from the embedded file system or from a specific
directory path through a configuration flag, without requiring
authentication but keeping the authentication verification for the API
endpoints.
* Add tests to verify that the UI assets are served without
authentication.
Change-Id: I9003ac96f1ec585a189b67fc1cb315905403d557
Alter satellites DB nodes table to add `disqualification_reason` int column
which contains disqualification type enum of why node has been disqualified.
Change-Id: Ia514557018ca27e1984216dc5004346d59869d16
To resolve problem with lack of ability to set metadata while MPU on
gateway we are adding setting metadata with BeginObject. This
change makes also metadata optional while CommitObject. We need
this functionality to not override metadata set with BeginObject in
case when metadata is not set with CommitObject.
Another reson is that we would like to not set metadata at all if user
didn't specify metadata. At the moment we always setting some bytes
for metadata fields e.g. empty EncryptedMetadata field can have key
and nonce set.
Change-Id: Ifee25b7718eb1f919119db9b698b29d8b5ebe2ec
rollup does not handle virtual drives correctly, use preserveSymlinks to
avoid compilation issues.
Change-Id: I7ae9509572ff4e2e74b059db4909f2e612a1331b
Currently the slow db was sleeping for 1s and the timeout for audit was
1s. There's a slight chance that the timeout won't trigger on such a
small difference.
Increase the slow node sleep to 10x of the timeout.
Hopefully fixes#4268
Change-Id: Ifdab45141b3fc7c62bde11813dbc534b3255fe59
Currently the test threshold for unneeded node checkins are 30s.
The disqualification threshold was at 30s, which means, it was possible
for all the nodes to get disqualified.
Hopefully fixes#4267
Change-Id: I6b0a10c09b7fd90a9729794885c9e7a593781bad
This commit crate the Satellite Admin Web UI v1
* Bootstrap the files installing the project template:
https://github.com/sveltejs/template
* Add the prettier prettier-plugin-svelte NPM packages as dev
dependencies.
* Add an empty prettierrc.toml.
* Transform the project template to use Typescript rather than
Javasript. See: https://github.com/sveltejs/template#using-typescript
* Replace the default favicon by the Storj logo.
* Create components in companion of some Typescript code which allows to
generate a simple HTML page based on an opinionated definition of API.
* Implement all the Admin API client calls in the format that the UI
generator components requires for rendering the UI from them.
Change-Id: I58fa586d68dc8998e5d89db169b8e90204f0a96d
To reduce database load, add option to omit as many object properties
as possible when listing objects.
Change-Id: I817633801b00629a4042d1d1bd2389ee581953de
Removed PathCipher and DefaultSegmentSize from CreateBucketParams
since they are unused anymore and breaks Integration on uplink side
Change-Id: I1393a7f1f436940731aa59edd693043336383290
The UserAgent should be stored as is, with the exception of removing the trailing version from any libuplink user agents
Change-Id: If17ef2fc4b59480a3477300f2585a07d64cc2bf4
