Create an error class for the "pending object error" for distinguishing
it from other errors for allowing to return it as a "Not Found" DRPC
status code instead an "Internal" status code.
"Internal" errors are logged in the satellite error so this was
polluting the server logs aside of returning an inappropriate status
code.
Change-Id: I10a81adfc887c030c08a228158adc8815834b23c
Respond with the appropriate HTTP status code when a request to the
analytics trigger event handler receive an authorized request.
A part of fixing the response status code this will stop to log these
response with ERROR level in our satellite logs.
Example of error message found in our satellite logs:
{
"insertId": "0ljf1cfn4xroxfd6",
"jsonPayload": {
"N": "console:endpoint",
"T": "2022-05-06T13:31:35.415Z",
"errorVerbose": "unauthorized: http: named cookie not present\n\tstorj.io/storj/satellite/console.GetAuth:72\n\tstorj.io/storj/satellite/console/consoleweb/consoleapi.(*Analytics).EventTriggered:60\n\tnet/http.HandlerFunc.ServeHTTP:2047\n\tstorj.io/storj/satellite/console/consoleweb.(*Server).withAuth.func1:488\n\tnet/http.HandlerFunc.ServeHTTP:2047\n\tgithub.com/gorilla/mux.(*Router).ServeHTTP:210\n\tstorj.io/storj/satellite/console/consoleweb.(*Server).withRequest.func1:495\n\tnet/http.HandlerFunc.ServeHTTP:2047\n\tnet/http.serverHandler.ServeHTTP:2879\n\tnet/http.(*conn).serve:1930",
"L": "ERROR",
"error": "unauthorized: http: named cookie not present",
"message": "unauthorized: http: named cookie not present",
"code": 500,
"S": "storj.io/storj/satellite/console/consoleweb/consoleapi.serveCustomJSONError\n\t/go/src/storj.io/storj/satellite/console/consoleweb/consoleapi/common.go:37\nstorj.io/storj/satellite/console/consoleweb/consoleapi.serveJSONError\n\t/go/src/storj.io/storj/satellite/console/consoleweb/consoleapi/common.go:23\nstorj.io/storj/satellite/console/consoleweb/consoleapi.(*Analytics).serveJSONError\n\t/go/src/storj.io/storj/satellite/console/consoleweb/consoleapi/analytics.go:75\nstorj.io/storj/satellite/console/consoleweb/consoleapi.(*Analytics).EventTriggered\n\t/go/src/storj.io/storj/satellite/console/consoleweb/consoleapi/analytics.go:62\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2047\nstorj.io/storj/satellite/console/consoleweb.(*Server).withAuth.func1\n\t/go/src/storj.io/storj/satellite/console/consoleweb/server.go:488\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2047\ngithub.com/gorilla/mux.(*Router).ServeHTTP\n\t/go/pkg/mod/github.com/gorilla/mux@v1.8.0/mux.go:210\nstorj.io/storj/satellite/console/consoleweb.(*Server).withRequest.func1\n\t/go/src/storj.io/storj/satellite/console/consoleweb/server.go:495\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2047\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2879\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:1930",
"M": "returning error to client"
},
"resource": {
"type": "k8s_container",
"labels": {
"location": "us-central1",
"pod_name": "us-central1-satellite-api-77c47f5c5-dzrpj",
"project_id": "storj-prod",
"namespace_name": "satellite",
"container_name": "satellite",
"cluster_name": "us-central1-gke-manatee"
}
},
"timestamp": "2022-05-06T13:31:35.416050390Z",
"severity": "ERROR",
"labels": {
"k8s-pod/version": "v3",
"k8s-pod/app": "us-central1-satellite-api",
"compute.googleapis.com/resource_name": "gke-us-central1-gke--terraform-202110-97ff1891-t0fv",
"k8s-pod/service": "api",
"k8s-pod/pod-template-hash": "77c47f5c5"
},
"logName": "projects/storj-prod/logs/stderr",
"receiveTimestamp": "2022-05-06T13:31:37.419991630Z"
}
Change-Id: I7cfcfb500b7878c59b1d259683c92e8963e2dc3f
Co-authored-by: Stefan Benten <mail@stefan-benten.de>
return storage and segment totals as a single result, instead of returning only storage
and bandwidth and segment values are filtered out, https://github.com/storj/storj/issues/4744
Change-Id: I624d67ed5205ae21ecd5a2f39775f63ed042e629
TestSegmentInExcludedCountriesRepair and TestSegmentInExcludedCountriesRepairIrreparable are using 20 storage nodes.
This change make them use 7 by adjusting the test redundancy scheme.
Change-Id: I1a44aa8b997d6edcc9a3305fdd0dac57e4d525b5
* Added new feature Flag for new Access Grant Flow.
* Added 3 cards to access grant view for S3, CLI and Access grant to replace old header
* Added new formatting, text and Icon for Access Grant Delete Popup modal
Added documentation.
Replaced PUT request with POST request.
Added inline param support for PATCH request.
Replaced unix timestamps handling with RFC-3339 timestampts handling.
Added 'Bearer' method requirement for Authorization header.
Change-Id: I4faa3864051dd18826c2c583ada53666d4aaec44
When an application wants to interact with resources on behalf of
an end-user, it needs to be granted access. In OAuth, this is done
when a user submits the consent screen.
Change-Id: Id838772f76999f63f5c9dbdda0995697b41c123a
Version collector previously returned errors and logged them in the
calling code. It is cleaner to log inside version collector.
Change-Id: I52cb49a1ef53f3f1f51692ddb26ec095cfd0f100
We were already able to override (or not) metadata with this method
but to be explicit we are introducting new option to control storing
metadata with object. Separate option should be less error prone.
https://github.com/storj/team-metainfo/issues/105
Change-Id: I4c5bce953a633a0009b05c5ca84266ca6ceefc26
"REST API" is a more accurate descriptor of the generated API in the
console package than "account management API". The generated API is very
flexible and will allow us to implement many more endpoints outside the
scope of "account management", and "account management" is not very well
defined to begin with.
Change-Id: Ie87faeaa3c743ef4371eaf0edd2826303d592da7
Extended user update query so prod owner can change user's paid tier status, bandwidth, storage and segment limits.
Change-Id: I82768afd1e50f653a50f7020310ce1e91578d746
it's a bit weird that these code definitions are in storj/storj
instead of storj/crypthopper-go, but as it stands, we should make
sure this package knows about all the codes in use.
Change-Id: I8df4666a015098e2d2e536d2f6c8ca5317a4369c
We implemented server-side copy feature and we would like to
confirm that it is not affecting expired deletion service.
Resolves: https://github.com/storj/storj/issues/4698
Change-Id: Ia8ca27a7ab7764a48a0c85dc7be80a58bfc83729
Last part of backwards compatible db migration to remove "suspended" column.
Removes exeption which removes "suspended" column in tests from `migrate_test.go`.
Adds DB migration to remove "suspended" column from 'nodes' and 'reputations' tables.
Change-Id: I02051279f6f4181e966c567919af0e774583f165
Set disqualification reason when reputations stats are updated on DB.Update.
Added tests for DisqualifyNode and for disqualification cases which happens during Update.
Change-Id: I00130ab5d9722422805159ad2f183c205de60f7e
When an api server is processing a graceful exit (node is connected and
getting lists of pieces to transfer), and the api server is shut down,
it was incorrectly marking all pending graceful exits as complete. The
GE then either passed or failed depending on the ratio of successfully
transferred pieces to unsuccessful pieces. In at least one case, _no_
pieces were transferred at all before the GE was marked a success.
Change-Id: I62cfab54a2296572c2e654eb460b62f772b7a60b
Attribution is attached to bucket usage, but that's more granular than
necessary for the attribution report. This change iterates over the
bucket attributions, parses the user agent, converts the first entry
to lower case, and uses that as the key to a map which holds the
attribution totals for each unique user agent.
Change-Id: Ib2962ba0f57daa8a7298f11fcb1ac44a8bb97875
Implemented new endpoint for project update using apigen.
Implemented new service method compatible with new generated api.
Change-Id: Ic0a7e0bbf3ea942275bd927d6e30cfb7e721e9c1
Migrate free tier users to have default limits if their limits were set to 0.
They were affected by incorrect working of Update user query.
Change-Id: I4c49c8d99b12dba2b9b0ab61b2175085976dcc95
We implemented server-side copy feature and we would like to
confirm that it is not affecting accounting/tally service.
Resolves https://github.com/storj/storj/issues/4697
Change-Id: I3944ea52c0acc68107ec15c1911750dc7d947501
Test case to verify if server-side copy doesn't affect
GE in any negative way.
Fixes https://github.com/storj/storj/issues/4699
Change-Id: I8c385767cca61499d46d9cb8de7318c56e5d7397
Added failed_login_count and login_lockout_expiration columns to users table to control users failed login attempts.
We want to prevent brute forcing of user login so this is the first step.
Change-Id: I06b0b9f5415a1922e08cd9908893b2fd3c26bca0
Use the same query when deleting a single object or multiple.
I have chosen not to deduplicate the row "scan" logic because
it is less complicated code and this change would expand to other
parts of the codebase.
Part of https://github.com/storj/storj/issues/4700
Change-Id: I7a958c78c903b2bddd72ca217971f7e8e02a0d0c
Initial space used for pieces is calcualted, not retrieved
from storage nodes and at the end of test we are deleting
also copies that become ancestors to verify that all data
was removed from storage nodes.
Change-Id: I9804adb9fa488dc0094a67a6e258c144977e7f5d
Before, the VA query was summing the total and dividing by the number of
rows. This gives the average bytes stored per hour, but we charge for
usage with byte-hours. Why not do value attribution the same way?
To do that, we don't divide by the number of rows. We also have object
and segment fees so return segment-hours and object-hours too.
Change-Id: I1f18b7e1b2bae1d3fae1ca3b93bfc24db5b9b0e6
We implemented server-side copy feature and we would like to
confirm that it is not affecting GC.
Fixes https://github.com/storj/storj/issues/4696
Change-Id: Id391f0badf5fce51f9910f0df732d477b07fa7ac
s3 allows for overwriting an object when using server-side copy.
This change makes overwriting the destination part of the atomic server-side copy operation so that
if copy fails, the old object is still available.
All the segments of the existing destination are deleted. If this destination object is an ancestor of another object, a new ancestor is promoted.
Fixes https://github.com/storj/storj/issues/4607
Change-Id: I85d1250850bb71867586ac230c8275a0cc1b63c3
Implemented new endpoint for project creation using apigen.
Implemented new service method compatible with new generated api.
Change-Id: I2bae22c8b046f21ec5bb6522f09b9c4e74bdba0c
When deleting a bucket, make sure that object copies in other buckets are
promoted to new ancestor and left in a working state.
Closes https://github.com/storj/storj/issues/4591
Change-Id: I019d916cd6de5ed51dd0dd25f47c35d0ec666af6
To save load on DNS servers, the repair code first tries to dial the
last known good ip and port for a node, and then falls back to a DNS
lookup only if we fail to connect to the last known good ip and port.
However, it looks like we are seeing errors during the client stream
Close() call (probably due to quic-go code), and those are classified
the same as errors encountered during Dial. The repairer code sees this
error, assumes that we failed to contact the node, and retries- but
since we did actually succeed in connecting the first time around, this
results in submitting the same order limit (with the same serial number)
to the storage node, which (rightfully) rejects it.
So together with change I055c186d5fd4e79560f67763175bc3130b9bc7d2 in
storj/uplink, this should avoid the double submission and avoid dinging
nodes' suspension scores unfairly.
See https://github.com/storj/storj/issues/4687.
Also, moving the testsuite directory check up above check-monkit in the
Jenkins Lint task, so that a non-tidy testsuite/go.mod can be recognized
and handled before everything breaks weirdly and seemingly randomly
later on.
Change-Id: Icb2b05aaff921d0af6aba10e450ac7e0a7bb2655
Moved invalid email testing to separate test.
Made all the emails used to have .test domain.
Added links to regex resources.
Change-Id: I26920ba7360064528256a6aeaea947bbe56ef618
