Commit Graph

2833 Commits

Author SHA1 Message Date
Jeremy Wharton
4ab8031171 cmd/satellite: combine draft invoice generation commands
This change introduces the generate-invoices satellite billing
command whose functionality is equivalent to running
apply-free-coupons, prepare-invoice-records,
create-project-invoice-items, and create-invoices in order.
Invoice finalization must still be performed separately.

Change-Id: Ia3d80b95eef1f2776c38bd730ed731e42ec4c35e
2022-10-03 21:19:44 -05:00
Michal Niewrzal
6fcc5c5cb8 satellite/metabase/segmentloop: remove unused monkit calls
Monkit calls for fast methods which are executed very frequently can
slowdown whole process. This change removes monkit calls which are not
used.

See https://review.dev.storj.io/c/storj/storj/+/8498 as an example of
speed improvement after removing monkit calls.

Change-Id: If6567d80e05b748e6393b58a5142e43013107c61
2022-10-03 14:07:18 +00:00
Michal Niewrzal
91c3a45874 satellite/accounting/nodetally: remove monkit call from observer
Benchmark against 'main':
name                                         old time/op    new time/op    delta
RemoteSegment/Cockroach/multiple_segments-8    5.56µs ± 5%    0.69µs ±12%   -87.57%  (p=0.008 n=5+5)

name                                         old alloc/op   new alloc/op   delta
RemoteSegment/Cockroach/multiple_segments-8    2.72kB ± 0%    0.00kB           ~     (p=0.079 n=4+5)

name                                         old allocs/op  new allocs/op  delta
RemoteSegment/Cockroach/multiple_segments-8      50.0 ± 0%       0.0       -100.00%  (p=0.008 n=5+5)

Change-Id: I20527fb576cd81db667a81929fa95b810ee11b14
2022-10-03 13:32:20 +00:00
Michal Niewrzal
5dc5f076c9 satellite/repair/checker: remove monitoring from fast methods
It looks that monikt monitoring can give high CPU overhead for
segments loop observer. With this code we are changing how monitoring
is initialized for observer methods. This optimization affects mainly
path where segment is healthy and doesn't require repair. Benchmark
is also added to show difference between old and new approach.

Benchmark against 'main':
name                                       old time/op    new time/op    delta
RemoteSegment/Cockroach/healthy_segment-8    8.55µs ± 4%    1.37µs ± 6%  -84.03%  (p=0.008 n=5+5)

name                                       old alloc/op   new alloc/op   delta
RemoteSegment/Cockroach/healthy_segment-8    2.63kB ± 0%    0.17kB ± 0%  -93.62%  (p=0.008 n=5+5)

name                                       old allocs/op  new allocs/op  delta
RemoteSegment/Cockroach/healthy_segment-8      54.0 ± 0%       8.0 ± 0%  -85.19%  (p=0.008 n=5+5)

Change-Id: Ie138eab0d59e436395b13f57bdfb11f9871d4c18
2022-10-03 12:15:03 +00:00
Michal Niewrzal
e37435602f satellite/audit: optimize loop observer
Two things were done to optimize audit observer:
* monik call was removed as we have different way to track it
* no new allocation for audit.Segment struct inside observer

Benchmark against 'main':
name                                         old time/op    new time/op    delta
RemoteSegment/Cockroach/multiple_segments-8    5.85µs ± 1%    0.74µs ± 4%   -87.28%  (p=0.008 n=5+5)

name                                         old alloc/op   new alloc/op   delta
RemoteSegment/Cockroach/multiple_segments-8    2.72kB ± 0%    0.00kB           ~     (p=0.079 n=4+5)

name                                         old allocs/op  new allocs/op  delta
RemoteSegment/Cockroach/multiple_segments-8      50.0 ± 0%       0.0       -100.00%  (p=0.008 n=5+5)

Change-Id: Ib973e48782bad4346eee1cd5aee77f0a50f69258
2022-10-02 22:24:37 +00:00
Jennifer Johnson
47f9915a27 satellite/payments/storjscan: add info level log statement when a user successfully claims a wallet address.
If we need to restore the satelliteDB from a backup, we must preserve the user - storj token wallet address association. This commit adds a log statement of this information after a user successfully claims a wallet. We can perform a SQL update to reassign the wallet address to the user if needed.

Change-Id: Ia5c25d7ac57e59b35865d74068196e42bc4ffe87
2022-09-29 18:20:06 +00:00
Michal Niewrzal
6bb3ea3b6e satellite/metabase: add tests for ListVerifySegments
Change-Id: Idc97111ab61ac3b23303371a30e3a66ea702f96e
2022-09-29 11:52:51 +00:00
prerna-parashar
94c5aaf70e
satellite/analytics: Added analytics for user behavior on Billing page (#5205)
* Added analytics for Billing page

* Fixed Jenkins errors

* Fixed lint errors

* Fixed Lint errors

* Fixed Lint errors

Co-authored-by: Maximillian von Briesen <mobyvb@gmail.com>
2022-09-29 09:20:24 +03:00
Fadila Khadar
02924d0ded satellite/metainfo: temporary feature flag for listing query testing
Fixes: https://github.com/storj/storj/issues/5144

Change-Id: I7650f4d5dd0378e2246339e79710a695996a845c
2022-09-28 17:46:52 +00:00
paul cannon
802ff18bd8 satellite/audit: better handling of piece fetch errors
We have an alert on `not_enough_shares_for_audit` which fires too
frequently. Every time so far, it has been because of a network blip of
some nature on the satellite side.

Satellite operators are expected to have other means in place for
alerting on network problems and fixing them, so it's not necessary for
the audit framework to act in that way.

Instead, in this change, we add three new metrics,
`audit_not_enough_nodes_online`, `audit_not_enough_shares_acquired`, and
`audit_suspected_network_problem`. When an audit fails, and emits
`not_enough_shares_for_audit`, we will now determine whether it looks
like we are having network problems (most errors are connection
failures, possibly also some successful connections which subsequently
time out) or whether something else has happened.

After this is deployed, we can remove the alert on
`not_enough_shares_for_audit` and add new alerts on
`audit_not_enough_nodes_online` and `audit_not_enough_shares_acquired`.
`audit_suspected_network_problem` does not need an alert.

Refs: https://github.com/storj/storj/issues/4669

Change-Id: Ibb256bc19d2578904f71f5229111ac98e5212fcb
2022-09-28 17:02:06 +00:00
Ethan
77357f9877 satellite/{accounting,redis}: Use ARGV in redis scripts
While investigating high memory consumption on a Redis instance, it was found that much of the memory consumed was from cached redis scripts.

Redis caches scripts based on the hash of the script itself.  This change uses ARGV to reduce the number of cached scripts.

Change-Id: Ia878fe81552a3067f09e60c44bb4ace25c6b5f9a
2022-09-28 14:43:00 +00:00
Egon Elbre
2b2b6d2288 satellite/admin/ui: use a valid go.mod stub
Some tools check all the go.mod files and fail when it's incorrect.

Change-Id: I1be5bd1b502e15c90b2f69caedbf5545d366a87d
2022-09-28 12:41:30 +00:00
Fadila Khadar
f58129786b satellite/metabase: copy handles version when overwriting target location
When doing server-side copy, deletes the committed version of the target location if it already exists. It does not touch pending versions. The version of the copy is set to the highest already existing version + 1.

Fixes: https://github.com/storj/storj/issues/5071

Change-Id: I1d91ac17054834b1f4f0970a9fa5d58198c58a37
2022-09-28 11:39:24 +00:00
Michal Niewrzal
a97cd97789 satellite/orders: remove unused service dependency
Orders service doesn't need buckets service anymore.

Change-Id: I27853cda87e82b528f53667e4b4866801f7bfb62
2022-09-28 08:56:36 +00:00
dlamarmorgan
88328ab240 satellite/payments/billing/billing.go: change default loop interval
Change the default loop interval for querying for new payments and adding them into the billing table from 1 minute to 15 seconds.

Change-Id: I26cf4a764cbe1de4c9b839ad60352374d8231522
2022-09-27 11:42:28 -07:00
Cameron
d8fb082f89 satellite/console: create project salt endpoint on satellite web server
Introduces a new endpoint on the satellite web server to get the
project's salt. The endpoint utilizes a new console service method
GetSalt which in turn calls the project DB GetSalt method if the
user is authorized. It returns the project salt bytes as a base64
encoded string in the response.

Change-Id: Ia13b5a4b8580e7bdad0dbb98014a276b1c74b46d
2022-09-27 12:19:28 +00:00
Erik van Velzen
d2a67fb8f7 satellite/gc/sender: concurrency
Restore functionality where retain filters can be sent out to multiple
storage nodes simultaneously.

Fixes https://github.com/storj/team-metainfo/issues/121

Change-Id: I2bf86a166b09c6a277c1cb455cdca0165ce6b8af
2022-09-27 08:10:10 +00:00
Cameron
41c2bca096 satellite/{console,metainfo,satellitedb}: add projectdb method GetSalt
Add new project db method, GetSalt, to get project salt. If salt
column is empty, return the sha-256 hash of the project ID. This
new method is used in metainfo endpoint ProjectInfo to return the
project salt to the client. This is backwards compatible because
the salt column is not populated yet. The updated endpoint will
do the same thing as the current endpoint.

Change-Id: I7eba376c865e10995a5a916302feca7cd7c7efa2
2022-09-26 13:12:21 +00:00
Michal Niewrzal
1aecca1e76 satellite/repair/checker: tiny cleanup
* unused slice removed
* variable moved closer to place of use

Change-Id: I86126b8337225d4b31cabf89bc9640add7409398
2022-09-26 11:20:10 +00:00
Erik van Velzen
7c809e2955 satellite/metabase: test list objects with version >1
Assert that listing works with our new object consistency approach.

Half of https://github.com/storj/storj/issues/4868

Change-Id: I5e92f86122b50103cec7bf6d3b2c8ed103caceec
2022-09-26 08:07:11 +00:00
Erik van Velzen
6ee3993f6c satellite/gc: e2e gc test
Restore previously existing end-to-end garbage collection test using
the new separate services for bloom filter generation and storage node
communication.

Original tests can be found under:
https://github.com/storj/storj/blob/v1.63.1/satellite/gc/gc_test.go

Change-Id: I42d1ab0f9981dfe183140da4d08087f4a6cd9296
2022-09-26 07:56:35 +00:00
dlamarmorgan
3c4b2cb09a satellite/console/service.go: return billing balance as USD value
Return the balance as currency object with a value and currency. The values are returned in USDollarsMicro (6 digits after the decimal).

Change-Id: I88c87faf3311b72dedd293d4e754c2fd5c03c128
2022-09-23 09:35:26 -07:00
dlamarmorgan
0b468aca56 satellite/payments/storjscan/storjscan.go: change default block confirmations
Change the default number of required block confirmations for a payment to be confirmed from 12 to 15.

Change-Id: I44c258134c293e7691623bc00c504130aa69a96a
2022-09-23 14:30:05 +00:00
paul cannon
7f1cad6faf satellite/repair: better handling of piece fetch errors
We have an alert on `repair_too_many_nodes_failed` which fires too
frequently. Every time so far, it has been because of a network blip of
some nature on the satellite side.

Satellite operators are expected to have other means in place for
alerting on network problems and fixing them, so it's not necessary for
the repair framework to act in that way.

Instead, in this change, we change the way that
`repair_too_many_nodes_failed` works. When a repair fails, we collect
piece fetch errors by type and determine from them whether it looks like
we are having network problems (most errors are connection failures,
possibly also some successful connections which subsequently time out)
or whether something else has happened.

We will now only emit `repair_too_many_nodes_failed` when the outcome
does not look like a network failure. In the network failure case, we
will instead emit `repair_suspected_network_problem`.

Refs: https://github.com/storj/storj/issues/4669

Change-Id: I49df98da5df9c606b95ad08a2bdfec8092fba926
2022-09-23 09:35:06 +00:00
Michal Niewrzal
289daf6787 satellite/metainfo: close project while testing
Project was not closed in test cases.

Change-Id: I700f671121433167fce7f350f84aa0559233c43c
2022-09-23 07:45:38 +00:00
paul cannon
7d0885bbaa satellite/repair: move over audit.Pieces
This structure is entirely unused within the audit module, and is only
used by repair code. Accordingly, this change moves the structure from
audit code to repair code.

Also, we take the opportunity here to rename the structure to something
less generic.

Refs: https://github.com/storj/storj/issues/4669

Change-Id: If85b37e08620cda1fde2afe98206293e02b5c36e
2022-09-22 16:43:03 +00:00
Michal Niewrzal
70eda67bc1 satellite/metabase: don't count objects for loop
Before and after segments loop we are collecting stats about number of entries in segments and objects table. We are using number of segments
to validate loop execution but currently we are not using number of
objects anywhere. This change drops SQL query to count objects count as
it's not use anywhere at the moment.

Change-Id: I25ce77758870beb0daa5c0e21084a4c633a26f15
2022-09-22 16:02:08 +00:00
Jeremy Wharton
b7ae52fb35 web/satellite: add captcha verification to forgot password form
This change adds the option of requiring users to pass a captcha
verification before password recovery emails can be sent to them.
The captcha for the Forgot Password form shares a configuration with
the one for the login form.

Change-Id: I27fdb4ed42cd9da1b4b8b1e80357892dc4091765
2022-09-22 15:20:04 +00:00
Egon Elbre
0bfaadcc6c cmd/tools/segment-verify: fixes and more tests
* Disallow too large listing limit, which would cause a lot of memory to
  be consumed.
* Fix throttling logic and add a test.
* Fix read error handling; depending on the concurrency it can return
  the NotFound status either in the Read or Close.

Change-Id: I778f04a5961988b2480df5c7faaa22393fc5d760
2022-09-22 10:32:30 +00:00
Michal Niewrzal
a22e6bdf67 satellite/gc/bloomfilter: use int64 to count pieces
Pieces count in DB are stored as int64 and we would like to align bloom
filter processing with this type.

Change-Id: Iaec767e609a40d802077ae057520541805a7c44f
2022-09-22 09:39:53 +00:00
Qweder93
c210776a36 satellite/{metabase, metainfo}: merge logic for BeginCopyObject and BeginMoveObject
Closes https://github.com/storj/storj/issues/4514

Change-Id: I4d59fe9e3d30f67ce642d6266b878956e38f5db1
2022-09-22 09:02:20 +00:00
Michal Niewrzal
4e71dd302a satellite/{metainfo,metabase}: prepare feature flag MultipleVersions
We will introduce new logic for creating new objects (BeginObject).
Instead of using single version internally (1) we will be selecting first
available version during object creation. Because we need to be sure
that everything is wired up correctly we need a feature flag to be
able to control if new feature is enabled.

Change-Id: If0f8496397130811f43bf9db9fdcc2b30cd2e4ca
2022-09-21 13:07:13 +00:00
Erik van Velzen
cc39727f69 satellite/metabase: test list pending versioned objects
Assert that listing pending objects works with our new object consistency
approach.

Half of https://github.com/storj/storj/issues/4868

Change-Id: Ic7bf3b20db57e64853d0464d7dc0da5441efd56f
2022-09-21 13:00:25 +02:00
Michal Niewrzal
c3ca98f552 satellite/gc/bloomfilter: minor cleanups
* service was running wrong RunOnce method
* after doing integration with GC sender we concluded that we don't
need special flag "gc-sender" to be uploaded as its safe to consume
partial results by GC senger. This part was removed.
* prefix format for moving data after error was unified with GC
sender

https: //github.com/storj/team-metainfo/issues/120
Change-Id: I204b696b9c2def4874ad1d17d0e84231cc98d583
2022-09-20 18:29:00 +02:00
Erik van Velzen
e6b5501f9b satellite/gc/sender: new service to send retain filters
Implement a new service to read retain filter from a bucket and
send them out to storagenodes.

This allows the retain filters to be generated by a separate command on
a backup of the database.

Paralellism (setting ConcurrentSends) and end-to-end garbage collection
tests will be restored in a subsequent commit.

Solves https://github.com/storj/team-metainfo/issues/121

Change-Id: Iaf8a33fbf6987676cc3cf74a18a8078916fe673d
2022-09-20 11:49:40 +00:00
Fadila Khadar
7788170234 satellite/metabase: ListObjects
Current metainfo.ListObjects implementation is using metabase iterator to list objects.
In the non-recursive case, it used to retrieve all the corresponding rows and then discarded the entries that did not fit the listing request.
This can lead in some edge cases (each prefix contains more than batchsize objects/sub-prefixes) to make unecessary calls to the db.

This change defines the metabase.ListObjects and aims at retrieving only prefixes (but not objects under it) and objects by modifying the SQL query.

In this version, it is not optimized on the database side. Cockroach will still have to go through all rows under a prefix, so there is still room for improvement.

metainfo.ListObjects is not currently using this method as we would like to assess its performance on the QA satellite first.

Fixes https://github.com/storj/storj/issues/5088

Change-Id: Ied3a9210871871d9d4a3096888d3e40c2dceed61
2022-09-20 11:15:44 +00:00
Egon Elbre
cf50696745 cmd/tools/segment-verify: wire up overlay logic
Change-Id: I0a4c737a8b0995a1c3e3adeac728fe833d0ce684
2022-09-19 11:32:18 +03:00
dlamarmorgan
a3a3ffd123 satellite/payments/stripecoinpayments: update invoicing to use credit note
Rather than using Invoice Items to account for storjscan token payments, credit notes will be used and applied to the users finalized invoice. This credit note will reduce the amount due of the users invoice based on the amount of storj token balance the user has on the satellite. Applying credit notes to a finalized invoice also requires that the invoice not be automatically paid when finalized. Therefore, a new command (pay-invoices) was added to initiate payment for users invoices.

Change-Id: Ie539375a10e842e3cb64bf0140834bbab0774f54
2022-09-16 13:40:38 +00:00
Egon Elbre
9b520b2114 satellite/metabase: expose ConvertNodesToAliases and ConvertAliasesToNodes
They are needed for segment-verify tool.

Also rename some of the conversion methods to make clear,
which of them have side-effects.

Change-Id: Ie9a0952548e9ed5068c7a30c2fd2134b07139bca
2022-09-15 13:56:10 +00:00
Michal Niewrzal
90eded4d99 satellite/gc/bloomfilter: take CreationDate from latest segment
Bloom filter CreationDate is used to avoid deleting pieces that
where not processed by GC. Every piece created after that timestamp
won't be deleted. Current GC process is taking CreationDate as a
beginning of bloom filter creation. This is appraoch allows to avoid
issues with inconsistent view on DB as currently we are using live DB
to create bloom filters.

With appraoch were we will be using DB snaphot with segment loop
we can get CreationDate from latest created segment in DB. Every piece
created after latest created segment won't be touched by GC on storage
node.

Updates https://github.com/storj/team-metainfo/issues/120

Change-Id: I6aaf64948ab7f60cfea62195689ad77c25ea772e
2022-09-15 11:59:53 +00:00
Yaroslav Vorobiov
a60c83c1de satellite/payments/storjscan: use currency.Ammount for token and usd values
Update storjscan API payment type to use currency.Amount for token and
usd values.
Update storjscan version for testsuite intgration tests.

Change-Id: Ie4ba7277b25d0d297a0130a8b885d6d38ff5eea4
2022-09-14 15:15:26 +02:00
Qweder93
04411e6871 satellite/accounting: use single call to get project limits
segment and usage limits now taken by single call

Closes https://github.com/storj/team-metainfo/issues/124

Change-Id: I55e05e199d2575cb8d985f4d75ba061002c97f0f
2022-09-13 21:50:52 +00:00
Fadila Khadar
601cadb92c satellite/metabase: copy when source and destination are the same
If source and destination are the same, do nothing and return the original object.
Later, we will have to handle the case when metadata is changed, but for now it's not possible through libuplink.

An issue has been created for this (https://github.com/storj/storj/issues/5168)

Change-Id: I91cf48afeec498d3b2c219fa0d3baf2163cff384
2022-09-13 19:21:13 +00:00
Mya
6d017a1a92 satellite/oidc: include nodeURL in oidc configuration
By including NodeURL in the OIDC well-known configuration, we're
able to discover the NodeURL for a given HTTP address without
any client side assumptions or needing to make a drpc to discover
this. Instead, it's included in a call that is already made by an
OIDC/OAuth enabled client.

Change-Id: If00f31665ca69b1f522e26fec825b29ad03fe7f9
2022-09-13 13:46:30 +00:00
Egon Elbre
546aa315fa satellite/metabase: add ListVerifySegments
We need a method for getting a list of segments from the metabase,
without converting the aliases and omitting all inline segments.

Change-Id: I26d919c675fc285ab03a35b327edd9b5c8bbe4b0
2022-09-12 17:32:29 +03:00
Michal Niewrzal
464ab43558 satellite/metainfo: fix deletion tests
Main issue with those tests was that for case where all objects were
uploaded at once (case "some nodes down" and "all nodes down").
Because all objects had the same name while upload each new object
was overwriting existing object. Because of that instead had several
objects to delete by test explicitly we had just 1. Test were not
failing because while overwriting existing object we were deleting it
but it was not what this test should do.

Change-Id: I602116f00be66589c7c0e68fe28c25e5c03e6b5d
2022-09-12 11:23:53 +00:00
Michal Niewrzal
158eb2381e satellite/gc/bloomfilter: uploading bloom filters
We would like to have separate process/command to collect bloom
filters from source different than production DBs. Such process will
use segment loop to build bloom filters for all storage nodes and
will send it to Storj bucket.

This change add main logic to new service. After collecting all bloom
filters with segment loop and piece tracker all filters are marshaled
and packed into zip files. Each zip contains up to "ZipBatchSize" bloom
filters and it's uploaded to specified in configuration bucket.

All uploaded objects have specified expiration time to not delete them
manually.

Updates https://github.com/storj/team-metainfo/issues/120

Change-Id: I2b6bc02a7dd7c3a639e75810fd013ae4afdc80a2
2022-09-12 08:33:53 +00:00
Michal Niewrzal
eea3fac0d3 satellite/metabase: BeginObjectNextVersion returns object
We plan to replace metabase.BeginObjectExactVersion usage in
metainfo.BeginObject with metabase.BeginObjectNextVersion. To make this
switch as simple a possible would be nice to have the same results for
both methods. This change is extending return value for
BeginObjectNextVersion to whole object struct. Tests were also adjusted
to be more like metabase.BeginObjectExactVersion tests.

Part of https://github.com/storj/storj/issues/4871

Change-Id: I4db99d74af07e5a73757b55233e0bbdc7b99d565
2022-09-12 07:54:14 +00:00
Jeremy Wharton
92d81db183 satellite/console,web/satellite: limit new password length to 128
This change implements a requirement that all new passwords must be
no longer than 128 characters.

Change-Id: I7a8f23c87190d465eed59b3e0627ccd32d465bb9
2022-09-08 13:29:47 +00:00
Qweder93
98fb4c4f32 satellite/metainfo: added test for object TTL validation while upload
Closes https://github.com/storj/team-metainfo/issues/113

Change-Id: I6ff2a5f2cdab6fd42d94fbfc335084b927d86bea
2022-09-08 11:39:41 +03:00
Qweder93
2a1bcfed81 {satellite/metabase, satellite/metainfo, satellite/inspector} : Use metabase.GetObjectLastCommitted instead metabase.GetObjectExactVersion
adjusted metainfo, metabase and inspector methods to use GetObjectLastCommitted,
added test cases to confirm that this method will work correctly with version higher then 1

Closes https://github.com/storj/storj/issues/4869
Closes https://github.com/storj/storj/issues/4873

Change-Id: I8c338dcd1db82c141383f41339a295d54e2fa039
2022-09-08 07:27:22 +00:00
Yaroslav Vorobiov
ad7c5b1483 satellite/payments/monetary: remove pkg and all its references
Remove pkg satellite/payments/monetary as it moved to storj.io/common.
Update all code pkg references from monetary to common/currency.

Change-Id: If2519f4c80cf315a9299e6521a6b9bbc6c399156
2022-09-07 12:58:00 +00:00
prerna-parashar
54eaf79b5f
satellite/analytics: Re-added user click events in Access Management flow (#5151)
* Re-added user click events in Access Management flow
2022-09-06 09:55:33 -07:00
Egon Elbre
794a4cc365 private/apigen: make code compile with 1.16
Our Test Versions still requires 1.16 to be compatible with our oldest
uplink versions. These changes make the code compile with 1.16.

Also, it makes go generate work in private/apigen/example.

Change-Id: Ib2f7493941a16f361328fe01d2be293f26123719
2022-09-02 16:34:46 +03:00
Egon Elbre
1343528a43 satellite/console/.../consoleapi/gen: ensure go generate works
Currently the paths were set relative to the root of the module,
however the code did not ensure that we are running relative to the
module directory.

Also, ensure typescript output corresponds to our styling.

Change-Id: I2b3cbd4ea8f2615e35c7b58c6fb8851669c47885
2022-09-02 16:34:46 +03:00
Michal Niewrzal
d905931ed9 private/testplanet: integrate GC bloom filter service
We would like to have separate process/command to collect bloom
filters from source different than production DBs. Such process will
use segment loop to build bloom filters for all storage nodes and
will send it to Storj bucket.
This change adds integration with testplanet which makes writing
unit tests possible.

Updates https://github.com/storj/team-metainfo/issues/120

Change-Id: I7b335c5dafa8cffe265c56b75d8c8f8567580893
2022-09-02 11:52:45 +00:00
Michal Niewrzal
68f6d93f29 satellite/gc/bloomfilter: add service to collect bloom filters
We would like to have separate process/command to collect bloom
filters from source different than production DBs. Such process will
use segment loop to build bloom filters for all storage nodes and
will send it to Storj bucket. This this initial change to add such
service. Added service is joining segment loop and collects all
bloom filters.

Sending bloom filters to the bucket will be added as a subsequent
change.

Updates https://github.com/storj/team-metainfo/issues/120

Change-Id: I2551723605afa41bec84826b0c647cd1f61f3b14
2022-09-02 08:10:46 +00:00
Jeremy Wharton
1613d37466 satellite/console: add endpoints for listing and revoking api keys
This change adds the following endpoints:
- projects/apikeys/{id}: returns a paged list of API keys for the
	project specified by the given ID
- apikeys/delete/{id}: deletes the API key specified by the given ID

Additionally, the API Go code generator has been given the ability to
process unsigned integer parameters.

Change-Id: I5ff24e012da24a3f06bea1ebb62bae6ff62f951a
2022-09-01 14:24:00 -05:00
dlamarmorgan
335e11dacd satellitedb: add token balance to API endpoint
Add the users current wallet balance to the endpoints for claiming and listing storjscan wallets. Also prevent a user with a claimed wallet address from claiming a new wallet.

Change-Id: I0dbf1303699f924d05c8c52359038dc5ef6c42a1
2022-08-31 22:55:16 +00:00
Qweder93
df5b0d0044 satellite/metabase: added method DeleteObjectLastCommitted
method similar to metabase.DeleteObjectExactVersion which will delete last committed object

Closes 	https://github.com/storj/storj/issues/4872

Change-Id: Ia9f8c227dc59575bf8ed297886b35536097028b4
2022-08-31 12:26:56 +00:00
dlamarmorgan
47301e5718 billing: add USDollarsMicro to billing DB
Adds USDollarsMicro currency to the billing DB which support fraction of a cent with decimal places for better billing amounts accuracy.

Change-Id: Id07dfae104d94e27c7b22ab8f5781010e16c4c8e
2022-08-30 14:48:51 -07:00
dlamarmorgan
afe58323f9 satellitedb: remove use of batch payment insert
Removed batch insert of payments since they do not guarantee order. Order of payments sent to the payments DB is important, because the billing chore will request new payments based on the last received payment. If the last payment inserted is not the last payment received, duplicate payments will be inserted into the billing table.

Change-Id: Ic3335c89fa8031f7bc16f417ca23ed83301ef8f6
2022-08-30 14:45:55 -07:00
Michal Niewrzal
00c594362c satellite: skeleton for new GC peer
We would like to have separate process/command to collect bloom
filters from source different than production DBs. Such process will
use segment loop to build bloom filters for all storage nodes and
will send it to Storj bucket. This change is just initial code with peer
which will be used to build this new process.

Updates https://github.com/storj/team-metainfo/issues/120

Change-Id: I10a52b74865ce8ec4c29b7c6a2836f9232620422
2022-08-30 15:41:49 +00:00
Erik van Velzen
4393b8a5ce satellite/internalpb: new protobuf for retain filters
Add new protobuff message to store retain filters before sending them
out to storagenodes.

This allows the retain filters to be generated by a separate command on
a backup of the database.

Change-Id: I8a2892d9d34bed1dbd6e5a7ded000dcdea7ec061
2022-08-30 12:37:40 +02:00
Michal Niewrzal
fa76dfe577 satellite/metabase: use GetObjectLastCommitted with move/copy
We are preparing to use object versions internally and to do
that we need to prepare different parts of the system to handle
object versions different than '1'. This change adjust code
responsible for server-side move and copy.

What was done:
* begin methods for move and copy are now using GetObjectLastCommitted
to find object
* results from begin move and copy operation contains now version to
be able to map object correctly with finish operation
* begin methods are putting version into satellite stream id and
finish methods are using this version as parameter instead hardcoded
value

Fixes https://github.com/storj/storj/issues/4867

Change-Id: I1380911279c21e10a3fff0342793efd2e73eafad
2022-08-30 06:36:02 +00:00
Vitalii
d585f2550d web/satellite: share bucket feature
Implemented share bucket feature.
Refactored share object modal a bit (has to be refactored entirely).

Issue:
https://github.com/storj/storj/issues/4945

Change-Id: Icefd4bfe3eef9173ae824eea44d30450acde8044
2022-08-29 18:41:06 +03:00
paul cannon
a57cbc4e0b satellite/satellitedb: drop _gob columns
This concludes the saga that began with commit c053bdbd, migrating away
from using gob-encoded big.Float values in the database to using
integers with an implied number of decimal places.

Nothing is using or accessing or expecting the _gob columns to exist
anymore. The transition code and the migration chore are gone. All that
remains is to drop the columns.

Change-Id: I9b15ee52f7781510a6dc91cf7c54f7f9022b1210
2022-08-26 19:45:22 +00:00
Michal Niewrzal
2163d60037 satellite/metabase/segmentloop: add missing monitoring
Currently we are tracking loop time with RunOnce method
metrics but it may give inaccurate values if loop if waiting
for observer for a ver long time. We had such case with GC
which is only loop observer in GC peer and its joning loop
every 5 days.

Change-Id: I08546c912e00c3641488de6a5d75948fe75c8e99
2022-08-26 14:01:22 +02:00
paul cannon
07bbe7d340 satellite/overlay: don't insert new nodes if contact check failed
Currently, the satellite tracks connectivity information about all nodes
that have contacted it, even if we have never successfully contacted
the node back.

This behavior was leveraged during a security audit to create hundreds
of thousands of "junk nodes" in the nodes table on one satellite, which
affected performance of queries such as node selection.

With this change, we should no longer track information about nodes that
have never been successfully contacted.

Note that it will still be possible to cause the creation of "junk
node" entries in the db; the attacker just has to set up individual
publicly-routable IP+port pairs for each node as it is created, so it
can respond to a PingBack.

Change-Id: Ibb6da6cc908fd4fc85aae1ba00313ba2738409ab
2022-08-26 09:05:04 +00:00
Sean Harvey
5a5f609022
satellite/metainfo: add copy and move log and metrics
Change-Id: I579c03c389dea7f8e7e92ade5bcac53740ce69c8
2022-08-26 18:55:29 +12:00
JT Olio
12ef68bdf4 satellitedb: restore-trash shouldn't bother with nodes we've never talked to
Change-Id: I3c631e92a9a2670c52c9fa23b2b1baf67d3c9a9c
2022-08-25 16:14:29 +00:00
Michal Niewrzal
d6054214c3 satellite/metainfo: move metadata validation to a proper place
Metadata validation for CommitObject request was placed in a wrong
place. There is a case (old uplink) where encrypted key is bundled
inside encrypted metadata bytes and we need to extract it before we
can validate it. This change moves metadata validation to a place where
we are sure we have encrypted metadata and encrypted metadata encrypted
key ready to be checked.

"Run Versions Test" is covering this case and it was failing without
this change.

Change-Id: Ib709ad901fbb3fa4865a393195b7b3f4c0d87e7a
2022-08-24 10:50:43 +00:00
Michał Niewrzał
ea38860aab satellite/metainfo: align satellite StreamID protobuf version with metabase
For object Version in different places we are using different types.
Satellite StreamID is using int32 but metabase accepts int64. Metabase
type is correct one and we should align other places with it.

As a small addition this change is also passing version correctly
between requests instead of using hardcoded value.

Change-Id: I63634d73c0a48c009e4db5f203ff18b7f3218b02
2022-08-24 10:17:34 +00:00
Qweder93
2139d3966b satellite/metabase: update UpdateObjectMetadata to work with latest commited object
Updated metabase.UpdateObjectMetadata method to update set metdata always for last committed object

Closes https://github.com/storj/storj/issues/4870

Change-Id: I060683e31efcaf3e2531fea143cf0567e5ff5f73
2022-08-24 10:28:57 +03:00
Márton Elek
4b1be6bf8e storagenode/satellite: support different piece hash algorithms
Change-Id: I3db321e79f12f3ebaa249e6c32fa37fd9615687e
2022-08-23 18:15:06 +00:00
Jeremy Wharton
3f26cc599f satellite/console,web/satellite: invalidate sessions after inactivity
Sessions now expire after a much shorter amount of time, requiring
clients to issue API requests for session extension. This is handled
behind the scenes as the user interacts with the page, but once session
expiration is imminent, a modal appears which informs the user of his
inactivity and presents him with the choice of loging out or preserving
his session.

Change-Id: I68008d45859c814a835d65d882ad5ad2199d618e
2022-08-23 15:51:05 +00:00
Michał Niewrzał
b722c29e77 satellite/metainfo: add missing metadata validation
We are missing metadata validation for UpdateObjectMetadata and
FinishCopyOject requests.

Change-Id: Idca6a4d1fe108e1593405fd3913442f5b69d09e7
2022-08-23 16:56:45 +02:00
Michal Niewrzal
d1d98af098 satellite/metabase: jump after prefix at the end of iteration page
For non-recursive listing if cursor to the next iteration page
is a prefix then jump after this prefix to avoid listing objects from
this prefix.

Benchmark against 'main' - 7e5025cac0
name                                                 old time/op    new time/op    delta
NonRecursiveListing/Cockroach/listing_no_prefix-8      11.7ms ±29%     1.7ms ± 7%  -85.52%  (p=0.008 n=5+5)
NonRecursiveListing/Cockroach/listing_with_prefix-8    4.60ms ± 6%    3.23ms ± 3%  -29.72%  (p=0.008 n=5+5)
NonRecursiveListing/Cockroach/listing_only_prefix-8    7.38ms ± 7%    8.47ms ±34%     ~     (p=0.421 n=5+5)

name                                                 old alloc/op   new alloc/op   delta
NonRecursiveListing/Cockroach/listing_no_prefix-8      92.3kB ± 0%    16.9kB ± 0%  -81.72%  (p=0.008 n=5+5)
NonRecursiveListing/Cockroach/listing_with_prefix-8    38.0kB ± 0%    27.4kB ± 0%  -28.02%  (p=0.008 n=5+5)
NonRecursiveListing/Cockroach/listing_only_prefix-8    59.8kB ± 0%    60.1kB ± 0%   +0.38%  (p=0.008 n=5+5)

name                                                 old allocs/op  new allocs/op  delta
NonRecursiveListing/Cockroach/listing_no_prefix-8       1.78k ± 0%     0.31k ± 0%  -82.48%  (p=0.008 n=5+5)
NonRecursiveListing/Cockroach/listing_with_prefix-8       734 ± 0%       526 ± 0%  -28.34%  (p=0.008 n=5+5)
NonRecursiveListing/Cockroach/listing_only_prefix-8     1.15k ± 0%     1.16k ± 0%   +0.87%  (p=0.008 n=5+5)

Discussed here https://github.com/storj/team-metainfo/issues/116

Change-Id: Iff671f062d9af83ec419334089163dd204d0cac0
2022-08-23 13:32:30 +00:00
Wilfred Asomani
a4192acabb console/satellite: track signup captcha scores
This change tracks signup captcha scores in the signup_captcha column in the users table.
It slightly modifies the captcha verify method to return both the score and success.

see: https://github.com/storj/storj/issues/5067

Change-Id: I7b3993e44958cfcf179806c7df19d6887fe3eda9
2022-08-23 10:13:49 +00:00
dlamarmorgan
cd89e1e557 satellite/satellitedb: remove direct import of pgxerrcode
Use the provided ConstraintViolation method of the pgutil package rather than importing jackc pgxerrcode directly.

Change-Id: I4e86713000b3f5f0aadd54beee8ee239f0c8df8e
2022-08-22 11:31:48 -07:00
Artur M. Wolff
36ae8d4bb9 satellite/metabase: revert non-recursive listing optimization
This change reverts satellite/metabase/iterator.go of 7390f389c to the
previous version (without optimization) but leaves added benchmarks in
place.

We noticed that this optimization doesn't work and actually elevates
listing times for most buckets, hence the revert until we come up with a
better idea.

Benchmarks:

name                                                old time/op new time/op delta
NonRecursiveListing/Postgres/listing_no_prefix-8    1.30ms ± 4% 4.52ms ± 4% +246.92% (p=0.008 n=5+5)
NonRecursiveListing/Postgres/listing_with_prefix-8  3.26ms ± 3% 4.44ms ± 2%  +36.19% (p=0.008 n=5+5)
NonRecursiveListing/Cockroach/listing_no_prefix-8    618µs ± 3% 2225µs ± 2% +259.94% (p=0.008 n=5+5)
NonRecursiveListing/Cockroach/listing_with_prefix-8 1.81ms ± 5% 2.60ms ± 5%  +43.96% (p=0.008 n=5+5)

Updates storj/team-metainfo#115

Change-Id: I96e4e7a563b188df478f8489027dc0042469b839
2022-08-22 14:14:10 +02:00
Clement Sam
31242f3820 satellite: use table aliases for query in QueryStorageNodeUsage endpoint
Following the changes made to fix the storage usage graph
on the storagenode dashboard, we added a new
interval_end_time column to the accounting_rollups table.

We noticed a two-day delay in the graph, turns out the sub-query
was wrong due to the conflicting interval_end_time column
in both tables so we have to explicitly state which table
column we are referring to.

Also, set the default for interval_end_time in the accounting
rollups to the start_time if the interval_end_time is null
which will be removed once we backfill the column and alter
it to be a non-nullable column.

Updates https://github.com/storj/storj/issues/4178

Change-Id: Iff32b261d07b6ee219d2b6b6542377f0c54633a1
2022-08-22 11:39:45 +00:00
Michał Niewrzał
71c1d0fff2 satellite/metabase: reduce queries for iterator boundaries tests
This is longest metabase at the moment and would nice to speed
up is a bit to improve overal tests execution time.

Change-Id: I86da8e0e593d20024b3ec778cbeab34a4613151f
2022-08-22 10:47:28 +00:00
Egon Elbre
8ddd5557f5 satellite/satellitedb: reduce migration tests
Database migration tests are rather slow, reduce them to
only the last 10 migrations, which should be sufficient.

Change-Id: Ib9d964fe6ec86ddeeef26c66b6ea9207b7868855
2022-08-22 09:04:49 +00:00
Ivan Fraixedes
aa1f50197a satellite/metainfo: Don't log error due to ctx cancellation
Context cancellation that aborts a non-essential Redis operation must
not be logged as an error because the operation is intentionally
canceled.

We are actually considering them not to be an error in following
operation because of the same reason and we return a RPC canceled status
code.

On the other hand it doesn't make sense to continue if the context is
canceled because although this is a non-essential operation if this one
is canceled due to the context the next one will be canceled for the
same reason, hence, we return earlier.

Change-Id: Ib3331975adeb06367d1ea0a578263ef50ae3f079
2022-08-19 18:41:54 +00:00
Yaroslav Vorobiov
9c77c762cb satellite/payments/storjscan: change USD currency to USDollarsMicro
Changes all usd amounts to use monetary.USDollarsMicro to have better prescision.

Change-Id: I391e2680f7fa11a0493e5af77488acff6dbf4999
2022-08-19 16:05:17 +00:00
Yaroslav Vorobiov
7a9b2a707b satellite/payments/monetary: add USDMicro and json marshaling
Adds USDMicro currency which support fraction of a cent with decimal places
for better billing amounts accuracy.
Adds JSON marshaling and unmarshaling for monetary.Amount, so that it
can be converted to/from JSON.

Change-Id: I034eba120ed23b6ba00b2d81a4f1b9db5f9a203f
2022-08-18 18:38:01 +00:00
JT Olio
e0d3e48b66 satellite/consoleweb: monitor visits without blowing up influx namespace
Change-Id: I43de5409e66b9ef34b6c35f211b9a252d3cc3caa
2022-08-18 17:48:28 +00:00
Cameron
fa23e55654 {private,satellite}: add Typescript generation to REST API generation
github issue: https://github.com/storj/storj/issues/4984

Change-Id: Id3473a6642f7a4af80edda25a6242559efaf69e9
2022-08-18 16:37:48 +00:00
Wilfred Asomani
c59d0d415c satellitedb: add signup_captcha column to users table
The signup_captcha column will hold the captcha scores of new users.

see: https://github.com/storj/storj/issues/5067

Change-Id: Ia322af29a3b5b019b417843272506a3dbd1397e4
2022-08-17 14:31:52 -05:00
paul cannon
0dcc0a9ee0 satellite/reputation: reconfigure lambda and alpha
This is in response to community feedback that our existing reputation
calculation is too likely to disqualify storage nodes unfairly with
extreme swings up and down.

For details and analysis, please see the data_loss_vs_dq_chance_sim.py
tool, the "tuning reputation further.ipynb" Jupyter notebook in the
storj/datascience repository, and the discussion at

    https://forum.storj.io/t/tuning-audit-scoring/14084

In brief: changing the lambda and initial-alpha parameters in this way
causes the swings in reputation to be smaller and less likely to put a
node past the disqualification threshold unfairly.

Note: this change will cause a one-time reset of all (non-disqualified)
node reputations, because the new initial alpha value of 1000 is
dramatically different, and the disqualification threshold is going to
be much higher.

Change-Id: Id6dc4ba8fde1be3db4255b72282207bab5491ca3
2022-08-17 18:52:53 +00:00
Cameron
c82424c21d satellite/admin: add Burst field to Update Project Limits params
Since 6 fields were sent, but only 5 fields were used as input (no burst),
the Segments UI input field was being set to the burst field in the
object, leaving segments empty.

Change-Id: I4e58ed8a36e2022d1d0844e9f9001562db3dcbb8
2022-08-17 10:40:28 -04:00
dlamarmorgan
8eec61d315 satellitedb: add timestamp index to billing table
Adding an index to the timestamp field of the billing transaction table to improve query performance. This should prevent having to do a full table scan when we query for the last billing transaction of a particular source and/or type.

Change-Id: I581f09494cc8662a12efba4302022a07121ba309
2022-08-16 22:50:02 +00:00
dlamarmorgan
f596f72f44 satellite/payments/storjscan: add billing update to chore
Update chore to add confirmed transactions to the billing table.

Change-Id: I3c9a98c2ddc96f8a8905250376a1f5490d810277
2022-08-16 11:18:15 -07:00
Ivan Fraixedes
7299e9f92e
satellite/console: Add missing monkit calls
Add monkit monitoring calls to a couple of the service's methods because
they were missed.

Change-Id: I327a79634b7cdcaeb0e2701a737bb714a6478c4a
2022-08-16 13:20:18 +02:00
dlamarmorgan
ac241501ca satellitedb: add wallet address index to storjscan_wallet table
Adding an index to the wallet address field of the storjscan wallet table to improve query performance. This should prevent having to do a full table scan when we query for one or more wallet addresses for a given user in our queries.

Change-Id: Ic1b5d06c2258489e5464d186fed5270172f8cba5
2022-08-15 18:05:06 +00:00
Jeremy Wharton
1f0638719e private/apigen,cmd/apigentest: add tests for generated API code
This change implements a unit test for ensuring proper
processing of requests and responses by generated API code.
Additionally, this change requires API handlers to explicitly receive
Monkit scopes rather than assuming that `mon` will always exist in the
generated API code's namespace.

Change-Id: Iea56f139f9dad0050b7d09ea765189280c3466f2
2022-08-15 16:48:41 +00:00
Vitalii
0ec28ca623 web/satellite: add token funds modal
Created new modal which shows user their native STORJ token wallet address.
There are QR and copy buttons.
It will be used only in new billing screen.

Change-Id: Icef3c8668c548b779c07fe2b85eb5761cd1221a3
2022-08-15 14:17:30 +03:00
Qweder93
2abe709b6e satellite/metabase: add metabase.GetObjectLatestVersion back
Restored GetObjectLatestVersion and renamed it to GetObjectLastCommitted
Add test cases to cover server-side copy

Closes https://github.com/storj/storj/issues/4866

Change-Id: I343b339a60152b8fb92fda97baf80bd8fe60d631
2022-08-11 22:14:38 +00:00
kimbotsao
c3e7b1409d satellitedb: add salt to projects table
Change-Id: I53691849efc169c495566648c12ae3d639614c3d
2022-08-11 12:16:53 -04:00