We don't need to have every single test for both, only one for
each should be sufficient. For all other tests it doesn't matter
which one we use.
Change-Id: I9962206a4ee025d367332c29ea3e6bc9f0f9a1de
Embedded files significantly increase the binary size for linking.
Add a tag that allows disabling embedding the build npm code.
Change-Id: I9d1fd7376d1fa035965c33d259faaa6c4770dfe1
So far we assumes that metadata key/nonce cannot be empty at all
but at some point we adjusted code to accept empty metadata/key/nonce
to save DB space.
This change is adjusting how we are processing nonce while
FinishMoveObject/FinishCopyObject. We can use storj.Nonce directly
which makes code cleaner. It's also fixing issue in FinishMoveObject
where we didn't convert nonce correctly to []byte.
Part of change is disabling validation for key and nonce until
uplink will be adjusted. We need change uplink to send always
both key and nonce or non of them. Validation will be restored
as soon as change for uplink will be merged.
https://github.com/storj/storj/issues/4646
Change-Id: Ia1772bc430ae591f54c6a9ae0308a4968aa30bed
This also fixes the build order. Unfortunately we need
to ensure that the web frontends are built before installing
Go binaries.
Fixes https://github.com/storj/storj/issues/4654
Change-Id: I5d1c83125fd3d1a454d3400b2cbdd44bd3f2250c
Add uplink-php and nextcloud as user agents. These sending of these
user agents was added to recent releases of these clients.
Change-Id: Ia2732ade1d9e5cf8d4e41fe246faec3feaa58c25
We are in the process of creating an api to allow users to manage their
accounts programmatically. We would like to use api keys for
authorization. We were originally going to create an entirely new table
for these api keys, but seeing as we already have 2 other tables for
keys/tokens, api_keys and oauth_tokens, we thought it might be better to
use one of these. We're using oauth_tokens.
We create a new oidc.OAuthTokenKind for account management api keys:
KindAccountManagementTokenV0. We made the key versioned because we
likely want to improve the implementation in the future, but we want to
get something functional out the door ASAP because the account management
api feature is highly desired.
Add a new method to oidc.OAuthTokens interface for revoking v0 account
management api keys, RevokeAccountManagementTokenV0. Add update method
to dbx implementation to allow updating the expiration. We will revoke
these keys by setting the expiration to 0 so they are expired.
Change-Id: Ideb8ae04b23aa55d5825b064b5e43e32eadc1fba
Uplink have some types aliased from storj/common repo. It's like
that for easier type replacement if we decide to use custom type
instead of aliasing. Because in storj/storj we are not using aliases
it's impossible to do refactoring on uplink side. This change is
cleaning up this situation.
Change-Id: I20c8e31b9a821983483af1c67b2e7bb91397fd9d
Added new endpoint to get all bucket rollups by bucket ID.
Example of response:
vitalii:~/Documents$ ./testapi.sh
HTTP/1.1 200 OK
Content-Type: application/json
Date: Mon, 07 Mar 2022 11:18:55 GMT
Content-Length: 671
[{"projectID":"a9b2b1b6-714a-4c49-99f1-6a53d0852525","bucketName":"demo-bucket","totalStoredData":0.0026272243089674662,"totalSegments":0.05000107166666666,"objectCount":0.03333373083333333,"metadataSize":1.6750359008333334e-9,"repairEgress":0,"getEgress":0,"auditEgress":0,"since":"2022-03-01T11:00:00Z","before":"2022-03-07T11:17:07Z"},{"projectID":"a9b2b1b6-714a-4c49-99f1-6a53d0852525","bucketName":"qwe","totalStoredData":0.000018436725422435552,"totalSegments":0.016667081388888887,"objectCount":0.016667081388888887,"metadataSize":1.933381441111111e-9,"repairEgress":0,"getEgress":0,"auditEgress":0,"since":"2022-03-01T11:00:00Z","before":"2022-03-07T11:17:07Z"}]
Change-Id: I8b04b24dbc67b78be5c309ce542bf03d6f67e65d
Add a missing instruction step for allowing Go to embed the files
generated by the UI build process into the satellite binary.
Change-Id: Ie9223b8bb5317e53e692e3aa1d1086977daa17c9
We have an issue with latest CRDB. Single query cannot modify
the same table multiple times. Now build is blocked.
This change is unblocking build by:
* adjusting query for inserting into repair queue
* temporary removing code for deletion for server-side copy
* temporary disable backward compatibility tests for CRDB
Change-Id: Idd9744ebd228e5dc05bdaf65cfc8f779472a975d
Chronograph statistics indicate that much of our Gateway-MT traffic may
originate from and also is metriced as rclone traffic. This makes it
difficult to understand what our users are doing. This solution makes
it clear what products are actually being used, likely without
increasing the cardinality of our metrics by more than one.
Change-Id: I5d5e2af3715fa0864f69f1145fd78caf7e4a4224
Remove redundant suspension timestamp column from nodes and reputation tables.
Suspended timestamp was moved to unknown_audit_suspended and suspended column is
no longer used so there is no point in keeping both.
Change-Id: Ieea3f12141b33ec9efe7594f4c9dbc7e10675b0e
If B is a copy of A, and C is a copy of B, then in the segment_copies table, it should appear that C is a copy of A.
Fixes https://github.com/storj/storj/issues/4538
Change-Id: I7e6b03f7584597cf616cd1e0cd0156386771d207
This change adds an integration test that performs an OAuth
workflow and verifies the OIDC endpoints are functioning as
expected.
Change-Id: I18a8968b4f0385a1e4de6784dee68e1b51df86f7
In the server-side copy initial implementation, we are inserting segments one by one. This PR inserts them all at once.
Fixes https://github.com/storj/storj/issues/4476
Change-Id: I776dba99be38a0eef73366e8e9287cbb794003dc
For server-side copy we adjusted one method DeleteObjectExactVersion.
Other deletion methods won't be used directly in code at the moment.
We will adjust other methods later or decide if we will need them at
all.
To handle deletion of objects with copies or just copies correctly we
need to use DeleteObjectExactVersion method in two places while:
* removing object before upload
* explicit object deletion
This change is also changing DeleteObjectExactVersion method to
delete pending objects because we need this functionality to
delete object before new upload.
https://github.com/storj/storj/issues/4481
Change-Id: Ieff5cc95732bb70ed8cc0ecdd62e03c929857c02
We were not checking if we were provided an empty StreamID.
Furthermore, this changes returns the object copy with the correct createdAt field.
Change-Id: Iefc563c34ae9d8c1e233895155c1718bf905df91
This change adds endpoints for supporting OpenID Connect (OIDC) and
OAuth requests. This allows application developers to easily
develop apps with Storj using common mechanisms for authentication
and authorization.
Change-Id: I2a76d48bd1241367aa2d1e3309f6f65d6d6ea4dc
Reworked email validation for new users (for old users trying to login or reset password validation remains the same).
Regular expression was built according to RFC 5322 and then extended to include international characters.
Change-Id: Id0224fee21a1ec0f8a2dcca5b8431197dee6b9d3
When performing re-authorizations for OAuth, we need to pull up an
APIKey using it's project id and name. This change also updates the
APIKeyInfo struct to return the head value associated with an API
key.
Change-Id: I4b40f7f13fb9b58a1927dd283b42a39015ea550e
Update the user to the default paid tier project limit, which is currently 3 projects, when the user upgrades to a paid account.
Change-Id: I95b19d62cebc7d878b716355f2ebcaf0b51ca3f7
For nodes in excluded areas, we don't necessarily want to remove them
from the pointer, but we do want to increase the number of pieces in the
segment in case those excluded area nodes go down. To do that, we
increase the number of pieces repaired by the number of pieces in
excluded areas.
Change-Id: I0424f1bcd7e93f33eb3eeeec79dbada3b3ea1f3a
Copy object functionality should support setting new metadata for
copy. This change is adjusting FinishCopyObject method to set new
metadata when OverrideMetadata field is set to true.
Fixes https://github.com/storj/storj/issues/4483
Change-Id: Ica37cb57e8edae301cdc483fbda4f3ddba5d2702
Added new endpoint to get project's single bucket usage rollup.
Extended generation code to handle service method args.
Change-Id: Ief768632a801c047c66e0617056fbd7b30427b33
Getting a copied segment by GetLatestObjectLastSegment needs to retrieve inline_data or remote_alias_pieces and other information from the original segment.
Resolves https://github.com/storj/storj/issues/4478
Change-Id: I8c7822c343b1ec3e04683f31a20f71e3097b4b4a
We decided that we want to have segment limit for paying users high
enough to not have to change it too often.
Fixes https://github.com/storj/storj/issues/4590
Change-Id: Ic1c38bf3e2fcc000548ff4c7e7004647b39fbecf
There are two events in
web/satellite/src/utils/constants/analyticsEventNames.ts which did not
have corresponding entries in the backend analytics service.
Change-Id: If0f67cef2ed312953e580d855d63366e7c12786a
Users will be required to enter a MFA passcode or recovery code
upon attempting a password reset for an account with MFA enabled.
Change-Id: I08d07597035d5a25849dbc70f7fd686753530610
Create global config to specify a list of country codes that should be
excluded from node selection during uploads.
This exclusion is not implemented when the upload selection cache is
disabled.
Change-Id: Ic41e8b4f18857a11045668eac23107da99668a72
This change allows us to send newly registered users to a configured URL
to help us track user conversions for marketing campaigns.
Brave conversions continue to be tracked using the /signup-success page
within the satellite app.
Change-Id: I9b451947ce0f39d3c99b233cb4b806d361151823
Added new projectaccounting query to get project's single bucket usage rollup.
Added new service method to call new query.
Added implementation for IsAuthenticated method which is used by new generated API.
Change-Id: I7cde5656d489953b6c7d109f236362eb465fa64a
Add a RepairExcludedCountryCodes config flag for overlay for providing a list of country codes to exclude nodes from target repair selection.
Mark segments with less than repairThreshold pieces in countries not in the RepairExcludedCountryCodes as not healthy.
With this change, the repair process is not affected. The segment will be removed from the repair queue by the repairer.
Another change will handle the logic at the repairer level.
Fixes https://github.com/storj/team-metainfo/issues/95
Change-Id: I9231b32de117a116488de055a3e94efcabb46e81
Added a feture flag which will be used to indicate if new generated console api is used.
Fixed some comments from previous PR.
Change-Id: Ice31c998b0b347028a491c971a648fd1269bfd49
Return segments when creating a test object so that it can be checked
that the correct segments are remaining after a delete action.
Change-Id: Ifc245948935ba278806e887672c03abc5f2c2654
There was a defined type (`validationErrors`) for gathering several
validation errors and classify them with the `ErrValdiation errs.Class`.
`errs.Combine` doesn't maintain the classes of the errors to combine,
for example
```
var myClass errs.Class = "My error class"
err1 := myClass.Wrap(erros.New("error 1"))
err2 := myClass.Wrap(erros.New("error 2"))
err3 := errors.New("error 3")
combinedErr := errs.Combine(err1, err2, err3)
myClass.Has(combinedErr) // It returns false
// Even only passing errors with a class and with the same one for all
// of them
combinedErr := errs.Combine(err1, err2)
myClass.Has(combinedErr) // It returns false
```
Hence `validationErrors` didn't return what we expected to return when
calling its `Combine` method.
This commit delete the type and it replaces by `errs.Group` when there
are more than one error, and wrapping the `errs.Group.Err` returned
error with `ErrValiation` error class.
The bug caused the HTTP API server to return a 500 status code as you
can seee in the following log message extracted from the satellite
production logs:
```
code: 500
error: "console service: validation: full name can not be empty; validation: Your password needs at least 6 characters long; validation: mail: no address"
errorVerbose: "console service: validation: full name can not be empty; validation: Your password needs at least 6 characters long; validation: mail: no address
storj.io/storj/satellite/console.(*Service).CreateUser:593
storj.io/storj/satellite/console/consoleweb/consoleapi.(*Auth).Register:250
net/http.HandlerFunc.ServeHTTP:2047
storj.io/storj/private/web.(*RateLimiter).Limit.func1:90
net/http.HandlerFunc.ServeHTTP:2047
github.com/gorilla/mux.(*Router).ServeHTTP:210
storj.io/storj/satellite/console/consoleweb.(*Server).withRequest.func1:464
net/http.HandlerFunc.ServeHTTP:2047
net/http.serverHandler.ServeHTTP:2879
net/http.(*conn).serve:1930"
message: "There was an error processing your request"
```
The issues was that not being classified with `ErrValidation` class it
was not picked by the correct switch branch of the
`consoleapi.Auth.getStatusCode` method which is in the call chain to
`consoleapi.Auth.Register` method when it calls
`console.Service.CreateUser` and returns an error.
These changes should return the appropriated HTTP status code (Bad
Request) when `console.Service.CreateUser` returns a validation error.
Returning the appropriated HTTP statsus code also makes not to show this
as an error in the server logs because the Bad Request sttatus code gets
logged with debug level.
Change-Id: I869ea85788992ae0865c373860fbf93a40d2d387
