For the storagenode usage graph currently,
1. the graph is still likely to contain spikes on the first day of
the month for a newly setup storagenode because there's no previous
interval_end_time to deduct from.
2. if a node goes offline for too long, say the last usage report
in the storageusage cache has an interval_end_time of
2020-07-30 00:00:00+00:00 and later, it comes back online a few
days later, it requests for the storage usage from the satellite
starting from the current month, say 2021-01-01 00:00:00+00:00,
the calculated hours for the first day would be 48 hours and it
could be wrong because the cache is missing one day usage report.
This PR addresses second issue on the storagenode side by requesting
storage usage data, instead of just a month boundary, request for an
interval starting from the last day of the previous month to the
current day of the current month.
The first one will be a tradeoff and wouldn't really matter since
it will just be an issue on the first day the storagenode joined
the satellite.
Updates https://github.com/storj/storj/issues/4178
Change-Id: I041c56c412030ce013dd77dce11b0b5d6550927b
The satellite now returns the last interval_end_time for each
daily storage usage.
We need to store the interval_end_time in the storage usage cache.
Also, renamed interval_start to timestamp to avoid ambiguity since
the interval_start only stores just the date/day returned by the
satellite.
Updates https://github.com/storj/storj/issues/4178
Change-Id: I94138ba8a506eeedd6703787ee03ab3e072efa32
Added Setup of access maker call into cmd_access_setup to use flags during cmd call
Closes https://github.com/storj/storj/issues/4766
Change-Id: I0c75f224414099573b021b18b87d9e17192cecc5
Change from DEBUG level to INFO level the logs that the trace request
middleware logs because it looks that we don't log in DEBUG level in
production Satellite API pods.
For making that assumption I searched in the last 7 days logs collected
by Google Logging service for all the Satellite API pods in US1 and it
didn't show any line.
Change-Id: I620009d70d59df46d524c8cee93851bd13eceeee
During an update to the billing DB, there is a special case failure that can occur if multiple updates to the table happen concurrently. In this case, the update would normally fail silently due to the balance constraint during update, and the subsequent insert for a new record fails because the user already exists in the table. The solution for this case, is to simply retry the insert with some limit to prevent infinite loops.
Change-Id: Ibe70fec2c386c25bd2484fe91f49a6a962357706
- Previously unused struct Endpoint.Request now defines the form
of the request body.
- Path parameters (e.g. "id" in "/delete/{id}") are defined in
the Endpoint.PathParams field.
- Endpoint.Params has been renamed to Endpoint.QueryParams to
eliminate confusion.
Change-Id: Ifef51ca2f362c33086f0e43e936d50b0fdd18aa1
Logs out all current user sessions when a password is changed through both the
forgot password and change password methods.
Change-Id: Iaf9b4969aa45441591524906af326b9dec17939f
Split out the function to delete a batch of objects from a bucket, so
that we get metrics which give a rough indication how long this operation
takes.
Part of https://github.com/storj/storj/issues/4957
Change-Id: I20a4ed5894217f4cd0b2f25aee297f0ecda57ab5
Don't terminate the expired objects loop or the zombie objects loop when
there is a DB error when selecting the objects for deleting them because
it isn't critical and the loops will pick them up again in the next
iteration.
The exception is if the DB rows scan method returns an error because
that's a symptom of the passed arguments to the method don't match with
the columns order, number, or type of the query, or there is invalid
data in the DB.
Don't also terminate these loops if the there is a DB error when
deleting the objects because the loops will pick them up in the next
iteration.
Because we don't return those errors now for not terminating the loop,
we have to log them.
Change-Id: I86bcf83d619345255840ae8f3db61620f044d2af
* Allow configure initial piece scan
* Update tests to include new flag
* Update default config specification
* Rename configuration flag
* Rename variable and fix formatting
* Fix format
* Fix typo
Co-authored-by: Stefan Benten <mail@stefan-benten.de>
Co-authored-by: Clement Sam <clementsam75@gmail.com>
Co-authored-by: littleskunk <jens.heimbuerge@googlemail.com>
We have enabled the new project dashboard in production. Change the
default to true so that we do not need an explicit configuration in
prod.
Change-Id: I0f93773965283e7b0682f6586685224281cbf78c
This change fixes an issue where the file selector does not open when the "drop files" box is clicked in the file browser.
see: https://github.com/storj/storj/issues/5064
Change-Id: Ib3020cd7c2cfeb7740a1f7b68fd0620cb26cab28
We log metainfo object operations and it looks that the log's message
convention is `Object {operation}`, however the `Object Download`
operation didn't match with the actual operation and the one that was
representing it had was `Download Object`.
This commit changes the log's message for the download object operation
according to the other object operations log messages format and fixes
the log message for the Get Object operation.
For finding this I executed the following command at the root of the
repository to obtain the list of lines where we log object operations.
$> ag 'log\.Info\(".*Object.*",' --no-color git:(main)
satellite/metainfo/endpoint_object.go
179: endpoint.log.Info("Object Upload", zap.Stringer("Project ID", keyInfo.ProjectID), zap.String("operation", "put"), zap.String("type", "object"))
336: endpoint.log.Info("Object Download", zap.Stringer("Project ID", keyInfo.ProjectID), zap.String("operation", "get"), zap.String("type", "object"))
557: endpoint.log.Info("Download Object", zap.Stringer("Project ID", keyInfo.ProjectID), zap.String("operation", "download"), zap.String("type", "object"))
791: endpoint.log.Info("Object List", zap.Stringer("Project ID", keyInfo.ProjectID), zap.String("operation", "list"), zap.String("type", "object"))
979: endpoint.log.Info("Object Delete", zap.Stringer("Project ID", keyInfo.ProjectID), zap.String("operation", "delete"), zap.String("type", "object"))
`ag` is a command-line tool similar to `grep`
Change-Id: I9072c5967eb42c397a2c64761d843675dd4991ec
Adds test for storjcan chore which creates one tx for claimed
wallet, triggers chore to query new payments from storjscan and
uses service to retrieve payment.
Change-Id: Ic774213420f1b01d8adfeeb5ca8054d9c918e7b1
Create Run function which spins up storjscan, eth test network and
testplanet configured with storjscan endpoint and credentials.
Change-Id: I5bafd07b6b2e4f39228faf76af7dd7135bed5f77
Added code for SortingHeader2.vue, VerticalArrows.vue, and TokenTransactionItem.vue. This code is for some of the stubbed components on PaymentMethodsTab.vue.
Added error notification when user tries to upload object after storage limit is exceeded.
Slightly changed error handling on failed preview/map request.
Issue: https://github.com/storj/storj/issues/5003
Change-Id: Iee91c1037b5c6de7b9718adcdfcea850fd3056ca
Refactored MFA, project members, profile and change password modals to use common VModal component.
Didn't touch access grant modals.
Change-Id: I9af5277496a99ec907fbee57a84127064ca9c02b
We had a lot of flaky test failures from TestAuth. The error message (WHICH IS NOT VISIBLE IN JEKNINS, only in tests.json):
```
FAIL: TestAuth_Register_NameSpecialChars/Postgres (1.04s)
panic: runtime error: index out of range [0] with length 0 [recovered]
panic: runtime error: index out of range [0] with length 0
goroutine 3473 [running]:
testing.tRunner.func1.2({0x235fe40, 0xc000fe6a08})
/usr/local/go/src/testing/testing.go:1209 +0x36c
testing.tRunner.func1()
/usr/local/go/src/testing/testing.go:1212 +0x3b6
panic({0x235fe40, 0xc000fe6a08})
/usr/local/go/src/runtime/panic.go:1047 +0x266
storj.io/storj/satellite/console/consoleweb/consoleapi_test.TestAuth_Register_NameSpecialChars.func1(0xc001a281a0, 0x289d650, 0xc001a30000)
/var/lib/jenkins/workspace/storj-gerrit-verify/satellite/console/consoleweb/consoleapi/auth_test.go:773 +0x785
storj.io/storj/private/testplanet.Run.func1.1({0x289c770, 0xc0001b8008})
/var/lib/jenkins/workspace/storj-gerrit-verify/private/testplanet/run.go:67 +0x732
storj.io/storj/private/testmonkit.RunWith({0x289c770, 0xc0001b8008}, {0x28d89b0, 0xc001a281a0}, {0x1, {0x0, 0x0}, {0x0, 0x0, 0x0}}, ...)
```
The root cause:
testplanet uses a simulated mail sender which clicks to all the registration links by default (async).
These tests creat links and check the unverified users, but without enough luck the mail sender may already clicks to the link which makes the user verified.
Change-Id: I17cd6bf4ae3e7adc223ec693976bb609370f0c44
Added label for new project dashboard which tells user that there might be a delay before actual data is shown.
Issue: https://github.com/storj/storj/issues/5002
Change-Id: Ife317f68c4bd3c5eadc72e35ed4e77c97b1c038e
This change replaces the support email with a link in the "deposit storj token" message.
see: https://github.com/storj/storj/issues/5007
Change-Id: Icff1171e563a6b9184eb0e2828cd61e3c95b45e4
As we started to run integration tests one by one (instead of parallel). We need to bump the timeout as well.
(Later we can decrease it to a better value after understanding what is reasonable)
Change-Id: I4a7e18973311fe98b0ffb3d9ed496584b1697980
Added string length limits for registration partner and promo params.
Limitation added both on client and server sides.
Issue: https://github.com/storj/storj-private/issues/44
Change-Id: Ifae04caad1775e0a8ca72ae7f9abcf0ea5fb564b
Implemented Recaptcha and Hcaptcha for login screen.
Slightly refactored registration page implementation.
Made 2 different login/registration captcha configs on server side to easily swap between captchas independently.
Issue: https://github.com/storj/storj/issues/4982
Change-Id: I362bd5db2d59010e90a22301893bc3e1d860293a
In an effort to distribute load on the reputation database, the
reputation write cache scheduled nodes to be written at a time offset by
the local nodeID. The idea was that no two repair workers would have the
same nodeID, so they would not tend to write to the same row at the same
time.
Instead, since all satellite processes share the same satellite ID
(duh), this caused _all_ workers to try and write to the same row at the
same time _always_. This was not ideal.
This change uses a random number instead of the satellite ID. The random
number is sourced from the number of nanoseconds since the Unix epoch.
As long as workers are not started at the exact same nanosecond, they
ought to get well-distributed offsets.
Change-Id: I149bdaa6ca1ee6043cfedcf1489dd9d3e3c7a163
This change makes the authentication middleware reject any requests
that are not properly authenticated to prevent them from being
passed into endpoint-specific handlers.
Change-Id: I1f6b74f68fc7354e47fb825a128bad968129f420
Update the billing table to use generated IDs, to include the source and status fields, and to add metadata jsonb field that can be used for fields specific to a particular type of billing transaction. Additionally a new table was added to keep track of user balances
Change-Id: Ieb3a63aafd8fe21fc3386bafd43d52081b7d2838
satellite/{payments/billing,satellitedb}: refactor billing DB
Update the billing table to use generated IDs, to include the source and status fields, and to add metadata jsonb field that can be used for fields specific to a particular type of billing transaction. Additionally a new table was added to keep track of user balances
Change-Id: Ieb3a63aafd8fe21fc3386bafd43d52081b7d2838
metabasetest package utils can be used by both tests and benchmarks
if we will use interface TestingT from require package. This change
adjusts metabasettest.CreateObject method
Change-Id: I3c138e2ef9873b804ab5b3402804efa409397a9f
eslint-plugin-storj is uploaded to the main npm registry.
While we have no clear proof that it causes the random freeze of npm install, it can be fater anyway as fixed version can be cached safely.
Change-Id: Ie74d93c7c4863e46121a9ab6c4e6b787fa3899b3
Our tests depend on timing (timeouts / sleeps). Therefore we must not overcommit the CPU with highly parallel execution as builds become flaky.
go test can do parallel execution (that's fine as it doesn't overcommit CPU, if only one job is running), but we shouldn't execute different type of
integration tests at the same time.
This commit changes only public (PR + main build) and pre-merge (one build per patch), which are less build time sensitive.
Change-Id: Ib9aaabd301cfa9cd3515a8aa4dd8ae82f58eb811
It's possible that content was not being flushed from processes.
For now, ignore other process failures under storj-sim network test.
Once we get other processes stable, we can repropagate the error.
Change-Id: I01ed572d7c779ab6451124f1e24e3d1168b3ea79
In rare cases time frame between creating time.Now() variable and calling
service method that receives it (after API method call) was big enough to distort
current month expectations and make test fail with last digit difference in 1
Change-Id: Ib811492d62f6598a5c40a09de6a87bffeaa0a78e
`len(m.attached)` was used without locking.
Also, use gofumpt to make whitespace usage more consistent.
Change-Id: Ifa9deedc8451f0c54e84d6ac3c2bdc1807688989
When a someone tries to create an account with an email that is already
associated with a verified account, send them an email with options to
sign in, create an account on another satellite, or reset password.
Change-Id: I844144d88b7356bd7064c4840c9441347a5368b0
