Added new endpoint to get all bucket rollups by bucket ID.
Example of response:
vitalii:~/Documents$ ./testapi.sh
HTTP/1.1 200 OK
Content-Type: application/json
Date: Mon, 07 Mar 2022 11:18:55 GMT
Content-Length: 671
[{"projectID":"a9b2b1b6-714a-4c49-99f1-6a53d0852525","bucketName":"demo-bucket","totalStoredData":0.0026272243089674662,"totalSegments":0.05000107166666666,"objectCount":0.03333373083333333,"metadataSize":1.6750359008333334e-9,"repairEgress":0,"getEgress":0,"auditEgress":0,"since":"2022-03-01T11:00:00Z","before":"2022-03-07T11:17:07Z"},{"projectID":"a9b2b1b6-714a-4c49-99f1-6a53d0852525","bucketName":"qwe","totalStoredData":0.000018436725422435552,"totalSegments":0.016667081388888887,"objectCount":0.016667081388888887,"metadataSize":1.933381441111111e-9,"repairEgress":0,"getEgress":0,"auditEgress":0,"since":"2022-03-01T11:00:00Z","before":"2022-03-07T11:17:07Z"}]
Change-Id: I8b04b24dbc67b78be5c309ce542bf03d6f67e65d
Add a missing instruction step for allowing Go to embed the files
generated by the UI build process into the satellite binary.
Change-Id: Ie9223b8bb5317e53e692e3aa1d1086977daa17c9
Properly encode email sent to the signup success URL so that emails with
symbols like `+` are processed and displayed properly.
Change-Id: I2d24ae08dc6a540e1e24d6c7baf62d83a856cd68
We have an issue with latest CRDB. Single query cannot modify
the same table multiple times. Now build is blocked.
This change is unblocking build by:
* adjusting query for inserting into repair queue
* temporary removing code for deletion for server-side copy
* temporary disable backward compatibility tests for CRDB
Change-Id: Idd9744ebd228e5dc05bdaf65cfc8f779472a975d
Chronograph statistics indicate that much of our Gateway-MT traffic may
originate from and also is metriced as rclone traffic. This makes it
difficult to understand what our users are doing. This solution makes
it clear what products are actually being used, likely without
increasing the cardinality of our metrics by more than one.
Change-Id: I5d5e2af3715fa0864f69f1145fd78caf7e4a4224
Remove redundant suspension timestamp column from nodes and reputation tables.
Suspended timestamp was moved to unknown_audit_suspended and suspended column is
no longer used so there is no point in keeping both.
Change-Id: Ieea3f12141b33ec9efe7594f4c9dbc7e10675b0e
It seems that the github API is a little slow/laggy with regards to propagation of whether a release tag has been made or not.
This sleep should fix it and avoid having to retrigger builds.
If B is a copy of A, and C is a copy of B, then in the segment_copies table, it should appear that C is a copy of A.
Fixes https://github.com/storj/storj/issues/4538
Change-Id: I7e6b03f7584597cf616cd1e0cd0156386771d207
When there is an error fetching a piece, the reader might be present or
it might not, depending on how far the fetch operation got. The
fetch-pieces code did not handle the "reader-not-present" case. Now it
should.
Change-Id: I263657d544d0ab8ba5d307a34ffc76bbf56835d0
This change adds an integration test that performs an OAuth
workflow and verifies the OIDC endpoints are functioning as
expected.
Change-Id: I18a8968b4f0385a1e4de6784dee68e1b51df86f7
Updating the version of the base image for the storagenode docker image.
Also fixes the non-root permission issue to /app directory
Change-Id: I8b55a1e3062f55ce6fc52e126ec1a18bfa24e669
In the server-side copy initial implementation, we are inserting segments one by one. This PR inserts them all at once.
Fixes https://github.com/storj/storj/issues/4476
Change-Id: I776dba99be38a0eef73366e8e9287cbb794003dc
This change fixes the following issues:
wget: Alpine docker image by default uses the builtin BusyBox wget which is not capable of handling SSL traffic via proxy unlike the GNU wget. We have to replace BusyBox wget with GNU wget.
updater failing to restart the node: supervisorctl pointing to wrong config file. We remove the default configuration file and point supervisorctl to custom config in systemctl
updates https://github.com/storj/storj/issues/4489
Change-Id: I24a7f18377ba723bbc377bb5d25aaa14f37021b1
Remote closing during upload or download is entirely expected and
it shouldn't lead to an error in the log.
Bump drpc to get the version that contains correct error code
for it. Also bump errs, which contains a fix for .Has.
Fixes https://github.com/storj/storj/issues/4609
Change-Id: I9297cabcfdc4b3a2c19d478dc729f779a2aef0c3
For server-side copy we adjusted one method DeleteObjectExactVersion.
Other deletion methods won't be used directly in code at the moment.
We will adjust other methods later or decide if we will need them at
all.
To handle deletion of objects with copies or just copies correctly we
need to use DeleteObjectExactVersion method in two places while:
* removing object before upload
* explicit object deletion
This change is also changing DeleteObjectExactVersion method to
delete pending objects because we need this functionality to
delete object before new upload.
https://github.com/storj/storj/issues/4481
Change-Id: Ieff5cc95732bb70ed8cc0ecdd62e03c929857c02
Add ability to limit updates in migrations.
To make sure things are looking okay in the migration, we can run it
with a limit of something like 10 or 30. We can look at the output of
the migrated columns to see if they are correct. This should have no
effect on subsequently running the full migration.
Change-Id: I2c74879c8909c7938f994e1bd972d19325bc01f0
We were not checking if we were provided an empty StreamID.
Furthermore, this changes returns the object copy with the correct createdAt field.
Change-Id: Iefc563c34ae9d8c1e233895155c1718bf905df91
This change adds endpoints for supporting OpenID Connect (OIDC) and
OAuth requests. This allows application developers to easily
develop apps with Storj using common mechanisms for authentication
and authorization.
Change-Id: I2a76d48bd1241367aa2d1e3309f6f65d6d6ea4dc
Reworked email validation for new users (for old users trying to login or reset password validation remains the same).
Regular expression was built according to RFC 5322 and then extended to include international characters.
Change-Id: Id0224fee21a1ec0f8a2dcca5b8431197dee6b9d3
This change fixes the `sed: can't create temp file '/etc/supervisor/supervisord.confXXXXXX': Permission denied` issue when editing the supervisord.conf file during runtime as a non-root user.
While editing the config file, Sed creates a temporary file, saves the result and then finally mv the original file with the temporary one. So we need to set the permission for the /etc/supervisor where the temporary file is created.
Change-Id: Ic9c147a9cf0a6ef94adf702e33054edce1828806
When performing re-authorizations for OAuth, we need to pull up an
APIKey using it's project id and name. This change also updates the
APIKeyInfo struct to return the head value associated with an API
key.
Change-Id: I4b40f7f13fb9b58a1927dd283b42a39015ea550e
Update the user to the default paid tier project limit, which is currently 3 projects, when the user upgrades to a paid account.
Change-Id: I95b19d62cebc7d878b716355f2ebcaf0b51ca3f7
For nodes in excluded areas, we don't necessarily want to remove them
from the pointer, but we do want to increase the number of pieces in the
segment in case those excluded area nodes go down. To do that, we
increase the number of pieces repaired by the number of pieces in
excluded areas.
Change-Id: I0424f1bcd7e93f33eb3eeeec79dbada3b3ea1f3a
Go can now directly embed files without relying on external tools.
This makes code use go:embed and avoid the external tooling.
go:embed requires files to be present in the embedded directory,
hence we need to add .keep to "dist" folder. We also add one to
public/.keep, such that it won't be deleted when building storagenode.
Change-Id: I8bef81236be6829ed37ed4c16ef693677b93a631
Move storagnode/console caching headers to private/web. Also,
start using them in multinode/console/server.
Change-Id: I1f0f3c9833a183476009737cece515ae7537fb83
Go can now directly embed files without relying on external tools.
This makes code use go:embed and avoid the external tooling.
go:embed requires files to be present in the embedded directory,
hence we need to add .keep to "dist" folder. We also add one to
public/.keep, such that it won't be deleted when building multinode.
Change-Id: I53ac3d5ac76e44f740d95221acf0da99fc256d42
Copy object functionality should support setting new metadata for
copy. This change is adjusting FinishCopyObject method to set new
metadata when OverrideMetadata field is set to true.
Fixes https://github.com/storj/storj/issues/4483
Change-Id: Ica37cb57e8edae301cdc483fbda4f3ddba5d2702
The supervisord.conf file is edited to set the args for the storagenode and storagenode-updater binaries at runtime. This change moves the config file to the base image so we can set the permission to allow non-root users edit the config file.
Non-root user permission is also needed for the /app directory so we can install/update the binaries when run as a non-root user.
Updates https://github.com/storj/storj/issues/4489
Change-Id: If7a51a00ea171253e41923501174a43393f4638c
When copying an object from cli you can now set the expiry.
It uses the same datetime format as restricting access grants.
Closes https://github.com/storj/storj/issues/4595
Change-Id: Icab73a64a9589817d6bc6d702b765b166ca1350d
Added new endpoint to get project's single bucket usage rollup.
Extended generation code to handle service method args.
Change-Id: Ief768632a801c047c66e0617056fbd7b30427b33
Getting a copied segment by GetLatestObjectLastSegment needs to retrieve inline_data or remote_alias_pieces and other information from the original segment.
Resolves https://github.com/storj/storj/issues/4478
Change-Id: I8c7822c343b1ec3e04683f31a20f71e3097b4b4a
Having the storagenode and storagenode-updater processes in one container
requires a process manager to properly handle the individual processes.
Using a process manager like supervisord requires that you package
supervisord and it configuration in the image, along with the storagenode
and storagenode-updater binaries.
Installing supervisord requires that we run apk to install it and its
dependencies at build time which makes it difficult to build multi-platoform
images; executing apk forces a requirement of the build system to run
foreign architechtures.
This change adds a dockerfile which will be used to build the base image
for the storagenode and has supervisord packaged. The base image will be
built manually using docker buildx, with QEMU binfmt support.
Updates https://github.com/storj/storj/issues/4489
Change-Id: I33f8f01398a7207bca08d8a4a43f4ed56b6a2473
