Commit Graph

2425 Commits

Author SHA1 Message Date
Egon Elbre
2848bf488f satellite/metainfo/metabase: use alias pieces in segments table
This makes all tables automatically convert between aliases and piece
ID-s.

Change-Id: I27fa42c82bbb09e05e3327f85e13a000b48faffd
2021-02-11 16:40:37 +02:00
Egon Elbre
25f81f353c satellite/metainfo/metabase: add AliasPiece
This adds AliasPieces run length encoding. On average it should
make our pieces encoding:

   repair=50,optimal=85,total=90   152.0 bytes
   repair=16,optimal=37,total=50    65.4 bytes

Change-Id: I391a9183164828f05383a3cde9ab0e4549c2d440
2021-02-11 14:23:51 +02:00
Egon Elbre
63c7f8b7fc satellitedb/satellitedbtest: creating a database shouldn't auto-migrate
Some tests need to control the migration progress manually.

Change-Id: I776c69b6d56dc35c7cb88688c4b827d6bba4b7ac
2021-02-11 14:21:49 +02:00
Michał Niewrzał
908a96ae30 Merge remote-tracking branch 'origin/main' into multipart-upload
Change-Id: I075aaff42ca3f5dc538356cedfccd5939c75e791
2021-02-11 11:48:23 +01:00
Michał Niewrzał
dc2bec9f89 satellite/metainfo: collect uplink versions with monkit
This change adds metrics for uplink version and method which was used by this version.

Change-Id: I05fc425a024805cfcd6d8add810a2a0749405f4f
2021-02-10 17:47:13 +01:00
Cameron Ayer
4a797baa73 satellite/repair/repairer: a new set of rs_scheme tagged metrics
Change-Id: Ibecd9265da881247eeb85ba185ee8877a7243777
2021-02-09 14:19:22 +00:00
Yaroslav Vorobiov
1e328f3c30 satellite/compensation: add wallet features to invoice csv
Change-Id: I90a72c0a4c8d07604096913b6680263b6defc0a2
2021-02-08 22:13:48 +02:00
Yaroslav Vorobiov
966535e9de {storagenode,satellite}/nodeoperator: add wallet features
Change-Id: Iac7eb40a52b8fddcc573aebaad2e3a30a10cded9
2021-02-08 22:09:45 +02:00
VitaliiShpital
fcd18ef48a web/satellite: rework signup verify redirect flow
WHAT:
people who sign up on US2 are not redirected to verifying page. From now on we have to set verify URL to make redirect happen

WHY:
user experience

Change-Id: I96c51a2c4f9cb6376cbfea639675b32918b58bee
2021-02-08 18:00:57 +00:00
Yingrong Zhao
89529237a2 web/satellite, web/marketing: remove referral program related UI code
Change-Id: Ia815a72c0d17760bb63d9a37bb18b88952209368
2021-02-08 11:07:35 -05:00
Yingrong Zhao
3b49d3cddf satellite: remove referral program related code
This PR removes all back-end related referral program code including the
marketing portal.

We will have a separate PR for front-end code and database migration to
drop `offers` and `usercredits` table

Change-Id: If59f952cddfe0558a7dc03a0eac7cc1081517f88
2021-02-08 13:52:50 +00:00
Egon Elbre
0c223f288a satellite/metainfo/metabase: don't use random node ID-s
We will add a cache to nodes, so using completely random nodes wouldn't
show the actual performance.

Change-Id: I94f18283712812f05f7795efd3c7cf57499fa52c
2021-02-08 11:36:05 +02:00
Egon Elbre
d0a7ab60ec satellite/metainfo/metabase: add NodeAliasCache
This implements a node alias cache that automatically refreshes
and adds new nodes when needed.

Change-Id: I4af121f7307b7b7ac4f87ad465f02d4eeb17d42f
2021-02-08 11:36:03 +02:00
Egon Elbre
0260966de4 satellite/metainfo/metabase: add NodeAliasMap
We need to keep an inmemory cache to avoid lookups into aliases table.
This adds the inmemory state of the cache.

Change-Id: Ief2b9bb19e10b46839b9208472dfc3035eb49af3
2021-02-08 10:28:22 +02:00
Egon Elbre
322c3a167b satellite/metainfo/metabase: add node alias table
This is first step in supporting node aliases. It adds a table
that automatically assigns aliases to nodes inserted into the table.

Change-Id: Ibdf40097c3c1e5b371500203f8db203505a48adc
2021-02-05 14:34:15 +02:00
Jeff Wendling
e114cfe86d satellite/satellitedb: fix broken query
Change-Id: I6d412a673d75264bf9751c6f15b1fb0ab94e1394
2021-02-04 16:00:51 -05:00
Caleb Case
f011a5e8cc
Use WithNonce when creating caveats
This ensures the caveats are unique even when they contain the same
permissions and will result in unique macaroons. This is important to
ensure revocation doesn't impact more macaroons than intended.

Change-Id: I6354edd0119f2d85eaf580f2d1926a3de9151b88
2021-02-04 13:46:32 -05:00
Malcolm Bouzi
db3a3088f9
satellite/satellitedb: add professional user fields to db interface (#4034) 2021-02-04 10:00:15 -05:00
Egon Elbre
615586a471 {satellite,storagnode}/internalpb: bump gogo to v1.3.2
Change-Id: I585a3197630b9b675a4b540cf25612e1ed39b599
2021-02-04 11:09:13 +00:00
Michał Niewrzał
9a60011774 Merge remote-tracking branch 'origin/main' into multipart-upload
Change-Id: Ia90f29be432e207c4125f7f955c912978eabe59a
2021-02-04 09:38:08 +01:00
Yingrong Zhao
21dfd99883 bump storj.io/private to latest
Change-Id: If4f08cd8cd8cd7cb09cd7d7051d966b08fe96c2e
2021-02-04 00:54:28 +00:00
Ivan Fraixedes
9c9f481469 satellite/orders: Remove deprecated endpoint
Remove the orders Settlement endpoint because it isn't used and it was
already always returning an error.

Change-Id: I81486fbe7044a1444182173bc0693698ee7cfe7e
2021-02-03 23:47:07 +00:00
JT Olio
c8e42df2f8 satellitedb: reorder migrations 140-144
These changes are independently tracked on
https://github.com/storj/storj/tree/jt/migration-reorder

The point of this is to make the distributed column
migration, needed for SNO invoice generation, the very
next one, so we can release it as a point release.

Change-Id: I26e1c03629c4f079b9ad12485e2b71a715d82b3b
2021-02-03 18:28:42 +00:00
Yingrong Zhao
7e80badaf9 pkg/server,pkg/quic: accept an existing conn to create quic listener and
allow disabling tcp/quic

In order to have more control of a server so that we can
simulate connection failures in `testplanet`, this PR changes
quic.Listener to accept an existing UDPConn instead of relying on the
quic-go library to create the UDPConn.
This PR also adds two flags on the `server.Config` struct to allow
enabling/disabling tcp/tls listener and quic listener. By default, they
are both set to true.
    - `DisableTCPTLS`: internal flag, disables tcp/tls listener.
    - `DisableQUIC`: hidden flag, disables quic listener
By making the `DisableQUIC` a hidden flag, it allows storagenode operators to
have the ability to disable quic traffic in case their set up can't work
with udp traffic.

Change-Id: I853b12435d988b9c41ad9b873fd57480d792e378
2021-02-03 12:04:29 -05:00
Jeff Wendling
a754c3984b satellite/console/consolewasm: assert less about error
this changes from a satellite error to a local encryption
error with the upcoming permissions changes where we only
include keys for the paths that are allowed.

Change-Id: I7aa37cfbaee31a1e54afe0423b283b9f41d9345f
2021-02-03 10:36:58 -05:00
Ethan
9506b67ca2 satellite/projectaccounting: Improve performance of ProjectAccounting.getBuckets
Limit bucket name lookup to date range of the calling methods since we only need distinct bucket names for that time period.
Adds new index and removes an index specific to project ID since it is no longer needed.

Change-Id: Ic07bbfb1c32280e0c0e39f8da020b284e1e5d974
2021-02-03 14:05:12 +00:00
Fadila Khadar
f34519c969 metainfo/metabase: iterate through pending objects by key
Change-Id: I451453d52278a2ff15ade1a6fccede136ab314be
2021-02-03 11:44:05 +01:00
Kaloyan Raev
8d25c47897 satellite/repair: fix comment in TestRepairExpiredSegment
Change-Id: Ib91e81f6ba0a7f65daed157b78f7a1a108984930
2021-02-03 10:09:49 +02:00
Cameron Ayer
a17934cb51 satellite/satellitedb: remove reference to uptime counts
Change-Id: I26ac540b720a8ba5d6ca44526900228352dcaf4e
2021-02-02 14:51:27 -05:00
Jeff Wendling
759bdd6794 satellite/compensation: add total-paid and total-distributed to invoices
Change-Id: Id4414867917cbf8aad77795f764d6381e88d9a34
2021-02-02 18:14:31 +00:00
Kaloyan Raev
038bd0a4da satellite/repair/repairer: fix repair for pending objects
https://storjlabs.atlassian.net/browse/PG-160

Change-Id: Ice7a0dcfc591bcde85a355cf95fff1eb3411f508
2021-02-02 19:50:10 +02:00
Kaloyan Raev
6f3d0c4ad5 Merge remote-tracking branch 'origin/main' into multipart-upload
Conflicts:
	go.mod
	go.sum
	satellite/repair/repair_test.go
	satellite/repair/repairer/segments.go

Change-Id: Ie51a56878bee84ad9f2d31135f984881a882e906
2021-02-02 19:19:04 +02:00
Kaloyan Raev
339d1212cd satellite/repair: don't remove expired segments from repair queue
It's impossible to time correctly this check. The segment may expire
just at the time we upload the repaired pieces to new storage nodes.
They will reject this as expired and the repair will fail.

Also, we penalize storage nodes with audit failure only if they fail
piece hash verification, i.e. return incorrect data, but only if they
have already deleted the piece.

So, it would be best if the repair service does not care about object
expiration at all. This is a responsibility of another service.

Removing this check will also simplify how we migrate this code
correctly to the metabase.

Change-Id: I09f7b372ae2602daee919a8a73cd0475fb263cd2
2021-02-02 16:13:01 +00:00
Egon Elbre
0bbe2f1d61 satellite/metainfo: add unimplemented ListPendingObjectStreams
Change-Id: I6c0fd240ce5be82c1f464470a6f147289b1cdf9d
2021-02-02 16:51:36 +02:00
Ivan Fraixedes
cc0d88f9c3
satellite/satellitedb: Fix GE flaky test
Fix an issue due to copy-paste problem that made that the Graceful Exit
test to be flaky.

The test uses a time created at the beginning of the test for avoiding
to get undeterministic time differences due to the fact of the response
time variation by the DB queries, however some part of the test were
using a current time rather than this base time, so they have been
addressed.

Change-Id: I4786f06209e041269875c07798a44c2850478438
2021-02-02 13:24:42 +01:00
Kaloyan Raev
b519bb377d satellite/metainfo/metabase: GetSegmentByLocation
We need this method to fix repairing pending objects. In another PR, it
will replace the GetObjectLatestVersion + GetSegmentByPosition calls
that are currently executed.

Change-Id: I4c5c2ab604edf898452b6fd21b86d4d3f970ce79
2021-02-01 21:52:12 +02:00
Ivan Fraixedes
d93944c57b satellite/orders: Delete unused methods & DB tables
Delete satellite order methods and DB tables which aren't used anymore
after we have done a refactoring on the orders to stuck bucket
information in the orders' encrypted metadata.

There are also configuration parameters and a satellite chore that
aren't needed anymore after the orders refactoring.

Change-Id: Ida3682b95921df70792284b42c96d2508bf8ca9c
2021-02-01 18:01:29 +00:00
Ivan Fraixedes
076804eac9 cmd/satellite: Add command for GE data cleanup
Add a command to the satellite for cleaning up the Graceful Exit (a.k.a
GE) transfer queue items of nodes that have exited.

The commit adds to the GE satellite DB a couple of new methods, and its
corresponding test, for performing the operations of the new command.

Change-Id: I29a572a59689d63b24990ac13c52e76d65aaa917
2021-02-01 17:30:58 +00:00
Jeff Wendling
1cf3d89a56 satellite/satellitedb: add distributed column and migration
using redash i manually checked that the only times the sum of
the payments does not match the paid column is for 2020-12 and
if it does not match then there are no payments.

Change-Id: I71ce0571de7e38e21548d7d6757b25abc3bfa781
2021-02-01 16:33:14 +00:00
Natalie Villasana
91bd4191dd satellite/accounting: add rollup archiver chore
The rollup archiver chore moves bucket bandwidth rollups and
storagenode rollups that are older than a given duration
to two new archive tables.

Change-Id: I1626a3742ad4271bc744fbcefa6355a29d49c6a5
2021-02-01 09:29:54 -05:00
Kaloyan Raev
6f441960ec satellite/metainfo/metabase: fix ParseSegmentKey to parse part numbers
ParseSegmentKey did not parse part numbers correctly from an encoded
SegmentKey.

Change-Id: I4ec7e15e0a094ed2a659c8c09d2f024eaceda678
2021-02-01 15:18:18 +02:00
Kaloyan Raev
d0612199f0 Merge remote-tracking branch 'origin/main' into multipart-upload
Conflicts:
	go.mod
	go.sum
	satellite/metainfo/config.go
	satellite/metainfo/metainfo_test.go

Change-Id: I95cf3c1d020a7918795b5eec63f36112fdb86749
2021-02-01 14:32:12 +02:00
Egon Elbre
b7a0739219 satellite/overlay: use DownloadSelectionCache for getting node IPs
Change-Id: Ib8f4eedb2bf465767050693a1e961b37a294ca06
2021-01-29 16:47:10 +02:00
Egon Elbre
54e01d37f9 satellite/overlay: add DownloadSelectionCache
Change-Id: Ic0779280172325f8d03f55a2e9673722f72bdd44
2021-01-29 16:47:06 +02:00
Cameron Ayer
89e682b4d7 satellite/repair/checker: add 29/80/130-52 to default repair overrides
Change-Id: I2e5a7538fdf33f3869fcb65fc88f7abb10faad79
2021-01-28 16:55:16 -05:00
Egon Elbre
19e3dc4ec0 satellite/overlay: rename NodeSelectionCache to UploadSelectionCache
It wasn't obvious that NodeSelectionCache was only for uploads.

Change-Id: Ifeeaa6fdb50a4b7916245b48d8634d70ac54459c
2021-01-28 14:56:53 +02:00
Kaloyan Raev
4d32bdaefb satellite/satellitedb: drop bucket_metainfos_name_project_id_key index
This index is obsolete and duplicates a similiar (project_id, name)
index on the same table.

Moreover, it might confuse CockroachDB which of the two index to use,
which may might affect DB performance.

Change-Id: If8d1df8347714942cea9dca82864ba5f4973bed3
2021-01-28 09:06:22 +02:00
Jeff Wendling
ca86820b8b satellite/snopayouts: use dbx + some refactorings
Change-Id: I8f3973d2377f071bcea2f61e0fc21d913ffa7ea8
2021-01-27 17:59:16 +00:00
Jeff Wendling
66e15fb7f1 satellite/compensation: remove ytd paid amounts
they aren't right and we aren't using them.

Change-Id: I5ca024e38d055696696886278863e941b5bc51bf
2021-01-27 17:31:01 +00:00
Kaloyan Raev
0f1961b7d1 satellite/metainfo/metabase: improve subquery comparision performance
Comparing the result from a subquery with the "IN" operator instead of
"=" makes a huge difference in the execution time of the SQL query on
CockroachDB.

Change-Id: I76e8f75a7bc95951667345d1ed9bd60f9aef3edb
2021-01-27 17:45:27 +02:00
Malcolm Bouzi
24d60384c5
satellite/satellitedb: add columns for professional users (#4028)
Co-authored-by: Egon Elbre <egonelbre@gmail.com>
2021-01-26 11:38:53 -05:00
Isaac Hess
c92bda7e75 tally monkit: change location to monitor piecesize
When we observed the value for total piecesizes stored in the network,
we were doing it after converting them to byte-hours, rather than using
the actual piece sizes. This fixes that issue.

Change-Id: I1564d21b519f70eb59f298d97dbd777baf127723
2021-01-26 15:37:02 +00:00
littleskunk
0b2568d712
satellite/overlay/straynodes: increase development duration without contact
Stopping storj-sim for over 5 minutes caused nodes to be disqualified. Set development max duration without contact to 300 days.
2021-01-26 12:24:39 +02:00
Michał Niewrzał
3fc0d2a83e satellite/metainfo: add testing method from multipart-upload branch
We wanto have single uplink branch for standard and multipart-upload satellite but some tests are using helper methods from multipart. This change adds methods used by uplink test.

Change-Id: I82352ed56674ff7e8743b58061ba594018e78e3b
2021-01-26 09:13:12 +00:00
Michał Niewrzał
50bea3ab1a satellite/metainfo: adjust tests to changes in uplink
One of uplink method changed its signature and we need to fix test on satellite side.

Change-Id: Ib89ea6aa25c57bac11bc3e0e9c2c89a4b9debd7c
2021-01-26 08:54:35 +00:00
Moby von Briesen
8263f18321 satellite/console: Add graphql query for owned projects
Change-Id: If47183d46cb7552ecdddbb3e536c36d958fad6d0
2021-01-25 17:43:04 +00:00
Egon Elbre
a700a1bdab satellite/metainfo/metabase: add benchmark
Change-Id: I4289aac252f9c951548444aa5a82e007e9f47806
2021-01-25 14:59:24 +00:00
Kaloyan Raev
ba2907be1b satellite/metainfo: do not expire satStreamID
We are checking if satStreamID is created in the last 48 hours. If it is
older we treat is as expired an fail to unmarshal it.

Since the satStreamID is also the Upload ID for multipart uploads, this
means that all calls fail for multipart uploads older than 48 hours.
Even aborting old multipart uploads is not possible.

To resolve this issue, we should stop checking satStreamID for
expiration.

Change-Id: Ieaf53ed3cd800cdd08843676c2d9490b007d962e
2021-01-25 13:35:20 +00:00
Egon Elbre
b8fd8c775a satellite/metainfo/metabase: fix FixedSegmentSize for gapped parts
Parts that have segment index gaps should be treated similarly how
multipart objects are, because direct calculation of the segment does
not work.

Change-Id: I2717eac36f085b5100f3d600fcf0ce056202a9eb
2021-01-25 13:30:56 +02:00
Egon Elbre
2ce829a8b1 satellite/metainfo/metabase: fix segment index in tests
Change-Id: Iaf9591229454d54d6e6927893738be678926b31c
2021-01-25 13:27:24 +02:00
Jessica Grebenschikov
b7d8dee5e9 satellite/console/wasm: add js tests
Change-Id: I8b1b0e81500836e0408e0517edb6c696698ab5f7
2021-01-21 20:18:03 +00:00
Michał Niewrzał
3c13aae61e satellite/metainfo: remove unused method
CreateGetOrderLimits is not used anymore because we have CreateGetOrderLimits2. We need to remove old method and fix name of second.

Change-Id: I59148b8d28fc9dbab7d452c884319125a02745d1
2021-01-21 17:00:13 +01:00
Michał Niewrzał
ce675b7707 satellite/metainfo/metabase: remove sql transaction while committing segments
Change-Id: Ia2cb1df63e1cbb8b9d36b0161a75b9c9cb3a56f9
2021-01-21 13:20:42 +00:00
Michał Niewrzał
292caa9043 satellite/metainfo/metabase: allow to set Encryption while committing object
In some cases we need to set encryption parameters later, with CommitObject method. This change makes Encryption optional with BeginObject* methods and mandatory with CommitObject if not set earlier.

Change-Id: I812c9b0e8fc213ca32d4758e0e68227e0e9bdd32
2021-01-20 18:28:25 +01:00
Michał Niewrzał
2d087c54b1 satellite/metainfo/metabase: use PlainSize for fixed segment calculation
In the past we were storing fixed segment size with StreamInfo, encrypted in metadata. The value was unencrypted size of segment, not encrypted one.

Change-Id: Id6b18440c674223eabbb152b1636c83e1ab6462c
2021-01-20 16:40:27 +00:00
Moby von Briesen
0a48071854 satellite/console: Add pagination fields for ListProjectsByOwnerID
Add ProjectsCursor type for pagination
Add PageCount, CurrentPage, and TotalCount ProjectsPage
This allows us to mimic the logic of GetBucketTotals and the
implementation of BucketUsages in graphql for the new ProjectsByOwnerID
functionality.

Change-Id: I4e1613859085db65971b44fcacd9813d9ddad8eb
2021-01-20 16:15:29 +00:00
Brandon Iglesias
e6c0383477
adding hypernet to the list (#4025) 2021-01-20 10:52:29 -05:00
Kaloyan Raev
c24ada7114 Merge remote-tracking branch 'origin/main' into multipart-upload
Conflicts:
	go.mod
	go.sum

Change-Id: Icf7c029e9d800e5f6a9fdd208c36f28e05468690
2021-01-20 17:35:57 +02:00
Cameron Ayer
d14607a5f7 satellite/{contact,nodestats,overlay,satellitedb}: remove references to total_uptime_count and uptime_success_count columns
Change-Id: I1f92022909bc564e9b1e31bf937fdfe7c16554de
2021-01-19 15:43:02 -05:00
Cameron Ayer
75d828200c private,satellite: add chore to dq stray nodes
Full scope:
private/testplanet,satellite/{overlay,satellitedb}

Description:
In most cases, downtime tracking with audits will eventually lead
to DQ for nodes who are unresponsive. However, if a stray node has no
pieces, it will not be audited and will thus never be disqualified.
This chore will check for nodes who have not successfully been contacted
in some set time and DQ them.

There are some new flags for toggling DQ of stray nodes and the timeframes
for running the chore and how long nodes can go without contact.

Change-Id: Ic9d41fdbf214736798925e728245180fb3c55615
2021-01-19 14:21:56 -05:00
Kaloyan Raev
ea48322dd3 satellite/metainfo: use deterministic signing for satStreamID
So we can have stable UploadID for multipart uploads.

Change-Id: Iac6780394c8cc0f96c0b9c4b850b92ed3627a9b0
2021-01-19 15:56:29 +00:00
Michał Niewrzał
d54ae9f10f satellite/metainfo: close project in TestObjectOverrideOnUpload
Change-Id: If19245474f54cea1657f704e155e09e1511eaeec
2021-01-19 16:33:30 +01:00
Ivan Fraixedes
0f70574e3c
satellite/accounting/tally: Don't abort tally when cache is down
Tally shouldn't abort its cycle if the accounting cache return an error
because it isn't an essential requirement to update it.

Change-Id: I78fd2bd9cf253ddedfb9ada80c0fa2ddf438f647
2021-01-19 15:13:01 +01:00
Egon Elbre
8e47adfad5 satellite/metainfo/metabase: optimize a query
Change-Id: I6395225f68cd8fbc8e3078c80ac6cb079c3c6a4f
2021-01-19 15:37:03 +02:00
Qweder93
6ba8f6c8a9 storanode, satellite: payout renamed to payouts, expected estimation payouts added, console api for audits reworked
Change-Id: I4aa5e99bffaa87d0a800a429a4c83aa498ad4b7b
2021-01-18 10:56:03 +00:00
Ivan Fraixedes
678b07b314
satellite: Fix typos & code formatting
Fix some typos in the doc comments and readdress some code formatting
applied automatically.

Change-Id: I605b4eff2e7c6c58227ecf16be4c1d26f5322eb6
2021-01-15 16:40:26 +01:00
Moby von Briesen
c24f84914c satellite/console: Add ability to list projects by owner ID
Listing projects by owner ID also includes the number of members in each
project.

Change-Id: I53a09674b60c199ef378943851bb0f164e92e4e2
2021-01-15 14:22:22 +00:00
Michał Niewrzał
38beecc7ad satellite/metainfo: Override pending object on upload
On upload we need to override pending and committed object. This change is adjusting DeleteObjectAllVersions to delete both.

Change-Id: Ib66c2af207c618119f7bf0de7fa9d3e5145d8641
2021-01-14 15:16:02 +00:00
Michał Niewrzał
95320912b4 satellite/metainfo: simplify metainfo delete test
Simplifies tests for pending and committed objects.

Change-Id: I0c683198f8fce62898142c6c343aca985d91a77f
2021-01-14 15:06:06 +00:00
Kaloyan Raev
6dff40f5c5 Merge remote-tracking branch 'origin/main' into multipart-upload
Conflicts:
	go.mod
	go.sum
	satellite/metainfo/metainfo.go

Change-Id: Ib5c49f3c911c58319855a171f9ce73657da976d9
2021-01-14 14:33:59 +02:00
Egon Elbre
85fb964afe satellite/{metainfo,overlay}: improvements to GetObjectIPs
* Deduplicate NodeID list prior to fetching IPs.
* Use NodeSelectionCache for fetching reliable IPs.
* Return number of segements, reliable pieces and all pieces.

Change-Id: I13e679caab275488b4037624b840a4068dad9589
2021-01-14 09:12:45 +00:00
Egon Elbre
d11c2b709e go.mod: bump storj.io/common
* Add missing endpoints.
* Fix deprecated packages and funcs.

Change-Id: I756090c46a4d15eabf6d413a593cdc64c5809bc7
2021-01-13 14:51:08 +00:00
Ivan Fraixedes
a4d06b9b1e satellite/metainfo: Don't response errors when Redis down
For being able to have resilient multi-region satellites we cannot stop
processing uploads/download client request when Redis isn't responding
properly.

These changes avoid to stop the processing of the client requests when
we cannot check if the client exceeds its storage or bandwidth limits
and we cannot update its used storage/bandwidth limits because Redis is
not responding successfully or the satellite database returns an error.

Change-Id: Ia7f12c07fc9ffdfad0e7ff052ff3fd81eca0f0e3
2021-01-13 14:30:44 +00:00
Ivan Fraixedes
a73c59bbdd
satellite/console/consoleweb: Change status codes usage limits
Respond to the HTTP clients which request the project usage limits with
different status codes depending of the error class returned by the
satellite/accounting Service.

Change-Id: I6f486ea55517f616c7cec81dbbe77e997484180f
2021-01-13 15:00:12 +01:00
Cameron Ayer
0184d33e96 satellite/satellitedb: set default 0 on uptime columns
This is the first step in the removal of uptime columns on the
nodes table. These columns are no longer used:

uptime_success_count
total_uptime_count
uptime_reputation_alpha
uptime_reputation_beta

In order to avoid breaking backwards compatibility, we need to
remove all references to these columns before removing the columns
themselves from the database. However, since uptime_success_count
and total_uptime_count are NOT NULLABLE, we can't remove them from
the insert statements in the overlay. So we can't remove the columns
because of the references, and we can't remove the references because
the columns can't be null. What a pickle. To remedy this, we will set a
default on the columns. Then we should be able to remove them from the
insert statements

Change-Id: I75f6c56fb7897835bbf29869f86f39de1d9dd345
2021-01-12 17:44:37 +00:00
Ivan Fraixedes
ce26616647
satellite/accounting/live: Use Redis client directly
We have to adapt the live accounting to allow the packages that use it
to differentiate about errors for being able to ignore them and make our
satellite resilient to Redis downtime.

For differentiating errors we should make changes in the live accounting
but also in the storage/redis.Client, however, we may need to do some
dirty workarounds or break other parts of the implementation that
depends on it.

On the other hand we want to get rid of the storage/redis.Client because
it has more functionality that the one that we are using and some
process has been started to remove it.

Hence, we have refactored the live accounting to directly use the Redis
client library for later on (in a future commit) adapt the satellite for
being resilient to Redis downtime.

Last but not least, a test for expired bandwidth keys have been added
and with it a bug was spotted and fix it.

Change-Id: Ibd191522cd20f6a9a15e5ccb7beb83a678e530ff
2021-01-12 15:33:29 +01:00
Michał Niewrzał
ac058e5ecc metainfo-migration: basic pointerdb->metabase migrator
Change-Id: If183b9c99ce816521fcae6e7189a6f85ddd4eb48
2021-01-12 12:59:53 +00:00
JT Olio
1ad69b9f96 satellite: make external address calc more robust
Change-Id: Icbdc6fb4e3fc85076dcb7bcc4b3ec36baad308d2
2021-01-11 16:47:25 +00:00
Cameron Ayer
0403e99a5b satellite/{overlay,satellitedb}: remove unused methods for old downtime tracking
GetSuccessfulNodeNotCheckedInSince and GetOfflineNodesLimited are overlay methods
which were only used by the previous downtime tracking system which has been removed.
These methods should also be removed.

Change-Id: Idb829d742e1f987e095604423fff656fe581183e
2021-01-11 15:21:28 +00:00
Egon Elbre
0df0e980a0 satellite/metainfo/metabase: fix FixedSegmentSize calculation
Commiting multipart object with same sized segments should always
end with FixedSegmentSize = -1.

Change-Id: Iedeafb763b62b2513de0a753dce20ddad5b32d90
2021-01-11 14:47:49 +02:00
Egon Elbre
24833465e6 satellite/metainfo/metabase: avoid magic constant
Change-Id: I4f01e38f67e18ae9cb9845a8e75a987acba66427
2021-01-11 10:22:21 +00:00
Michał Niewrzał
ec88d21a3c Merge 'main' branch.
Change-Id: I6e8162d1a6caf75e89c9f9c9f9522730aebf83ae
2021-01-11 10:26:58 +01:00
Michał Niewrzał
71ca18134b satellite/metainfo: reject downloading a segment from multipart object
Non-multipart uplink implementation is always trying to download object
by downloading last segment first (PartNumber=0, Index=-1) but this
approach won't work with multipart object. We need to reject such old
style request with reasonable message.

Change-Id: I9221e019933565a8d25136bdfef3e054320bac3d
2021-01-11 08:04:33 +00:00
Jessica Grebenschikov
1709117b0d satellite/console/wasm: add more unit tests
Change-Id: Ie134f8a08d690ce013039ed1a4e484f8b6a1a6d5
2021-01-08 18:50:29 +00:00
Jeff Wendling
2d2359667d satellite/orders: remove unused satelliteAddress field
Change-Id: I58091769472688433c48becc8dfc9029bddd87aa
2021-01-08 12:25:39 -05:00
Egon Elbre
ba5461562d satellite/orders: remove satellite address
SatelliteAddress in OrderLimit is not being used anymore and some
satellite addresses may consume too much bytes.

Change-Id: Ic7a0efe5b6211c2f3b91af67b293cde98b29d074
2021-01-08 16:57:36 +00:00
Egon Elbre
51731db121 satellite/orders: use smaller encrypted metadata
Avoid using project uuid string representation, because
it uses more bandwidth.

This reduces the encrypted metadata size from 118 -> 97 bytes.

Change-Id: Ic53a81b83acc065f24f28cd404f9c0b1fe592594
2021-01-08 16:40:31 +00:00
Michał Niewrzał
da316c27aa satellite/metainfo/metabase: disable PlainSize validation
We need to accept segments with PlainSize == 0 to be backward compatible
for old uplinks.

Change-Id: I926abe751e9e0ad7e06e42a5301ab78a3f23aafa
2021-01-07 14:14:20 +00:00
JT Olio
8907180e81 satellite: pass contact.external-address config to web ui
Change-Id: I54978aa34aa9eb98876fab6460a5737d718d6135
2021-01-06 10:11:20 -07:00
Kaloyan Raev
4fc61f7afa satellite/metainfo/metabase: allow commmitting objects > 2 GiB
The total_plain_size and total_encrypted_size columns in the objects
table were set as INT4, which limits the size of committed objects to
just 2 GiB.

This patch migrates the DB to change the type of these fields to INT8.

Change-Id: Iad7e7b44a652e6c5b8e17b80588637bb48390fe6
2021-01-06 18:23:03 +02:00
Egon Elbre
9cb4466eb0 cmd/storj-sim: use dev setup by default for consistency
Fixes bug when using release binaries together with storj-sim.

Change-Id: I077bedc1486ac85aa1f04fcc0ed4098cd313f2fc
2021-01-05 13:47:30 +02:00
Moby von Briesen
a90d6fcad8 satellite/repair/checker: Use segment health on checker insert
Do not insert the number of healthy pieces for segment health anymore.
Rather, insert the segment health calculated by our new priority
function.

Change-Id: Ieee7fb2deee89f4d79ae85bac7f577befa2a0c7f
2021-01-04 11:48:17 -05:00
Moby von Briesen
6e2ef3b9ee Revert "satellite/satellitedb: Do not consider nodes with offline_suspended as reputable."
This reverts commit e24262c2c9.

Change-Id: I287deb2e52d03bbd698ed055f0f216b0b5bf2798
2021-01-04 14:28:37 +00:00
Michał Niewrzał
ad3e3a38c5 Merge 'main' branch
Change-Id: Ia0db1b1f9ef3e0671d3f2208881b0abc3064e200
2021-01-04 12:13:45 +01:00
Michał Niewrzał
d4ebdba48c satellite/payments/stripecoinpayments: fix tests failing in 2021
We had some tests with hardcoded year 2020.

Change-Id: I0184c3ece819cb764eb305751a1d8d4056b6af17
2021-01-04 10:47:31 +01:00
Moby von Briesen
edbee53888 satellite,storagenode: Pass audit history over GetStats endpoint
Full prefix: satellite/{overlay,nodestats},storagenode/{reputation,nodestats}

Allow the storagenode to receive its audit history data from the
satellite via the satellite's GetStats endpoint.

The storagenode does not save this data for use in the API yet.

Change-Id: I9488f4d7a4ccb4ccf8336b8e4aeb3e5beee54979
2020-12-30 19:13:26 +00:00
Rafael Gomes
8b2e4bfa7e satellite/metainfo/piecedeletion Remove spaces from metrics.
Change-Id: Iaf1d8a96a43087f2fcc579347f581e8a78a0fb58
2020-12-30 14:27:39 -03:00
paul cannon
7246368ca1 satellite/repair: clamp totalNodes to 100 or higher
Change-Id: I239418ed3671b1cee30b0b1797dc434244e72448
2020-12-30 10:39:14 -06:00
Moby von Briesen
825dc71227 satellite/{overlay, satellitedb}: Refactor audit history
* Separate audit history interface into its own file in the overlay
package
* Add overlay.AuditHistory struct so that internalpb.AuditHistory is
only used from within the database layer
* Add overlay.GetAuditHistory function for features that will require
access to detailed audit history information
* Do not return full audit history from UpdateAuditHistory - callers to
that function only need to know the online score and whether a full
tracking period has been completed
* Move audit history tests out of satellite/satellitedb, since they are
independent of database implementation

Change-Id: I35b0c4ac23bbaabd80624f8a9631c3cb1a1f33bd
2020-12-29 18:50:22 +00:00
Moby von Briesen
85ae13f11d satellite/satellitedb: Drop nodes_offline_times table.
Now that the deprecated downtime tracking service is removed
(3fc76f4ffe), we can safely remove
the nodes_offline_times table.

Change-Id: Ia7c6efe32ba104dff5a830af5f2beee3337eefe5
2020-12-29 18:17:50 +00:00
Moby von Briesen
e24262c2c9 satellite/satellitedb: Do not consider nodes with offline_suspended as reputable.
Nodes which are offline_suspended will no longer be considered for new
uploads. The current threshold that enters a node into offline
suspension is 0.6. Disqualification for offline suspension is still
disabled.

Change-Id: I0da9abf47167dd5bf6bb21e0bc2186e003e38d1a
2020-12-29 17:59:09 +00:00
Stefan Benten
ad58459198 satellite/admin: allow more than just "paid" invoice status during user deletion
Currently we do not allow anything other than the "paid" status for invoices when
trying to delete a user. However there can be a couple of other states that are
still fine to accept during deletion of a user. This change reverses the order to
check for the status that we do not want to allow.

Change-Id: I78d85af6438015c55100fa201ccffc731c91de1c
2020-12-23 16:40:44 +01:00
JT Olio
7faaeed2bf satellite/access grant wizard: don't hardcode the satellites
Change-Id: Id9fbf68882cdb2fce846b7a2604cf965cc53ab1a
2020-12-22 21:24:45 -07:00
JT Olio
efde103dba accounting: rollup test is broken for the hour before midnight UTC
this change isn't the real fix. it's just ignoring the problem.

i don't know what the real fix is. is the problem with the test, or
is there actually a problem with the rollup code?

Change-Id: I552bdd947deadc212cc56efc5f818942b9827126
2020-12-22 14:14:52 -07:00
Ethan Adams
6070018021
satellite/overlay: use AS OF SYSTEM TIME with Cockroach
Query nodes table using AS OF SYSTEM TIME '-10s' (by default) when on CRDB to alleviate contention on the nodes table and minimize CRDB retries. Queries for standard uploads are already cached, and node lookups for graceful exit uploads has retry logic so it isn't necessary for the nodes returned to be current.
2020-12-22 21:07:07 +02:00
Michal Niewrzal
9a8959d429 Merge 'master' branch
Change-Id: Iba69ea73ca4d3f1cd4ae94243eaaae033c5324e8
2020-12-22 14:55:57 +01:00
Kaloyan Raev
5934969dd6 satellite/orders: remove obsolete CreateDeleteOrderLimits method
Not used anywhere.

Change-Id: I878635d2d533ad4b06ba0d07a94908105546cb82
2020-12-22 13:15:11 +02:00
Kaloyan Raev
777f6e583b satellite/metainfo: replace a call to PointerDB with call to Metabase
Change-Id: I10aa89dacf91cb6c1528698031e6b53c52915bd9
2020-12-22 12:44:28 +02:00
Egon Elbre
c4578eb3ec satellite/gc: add test for pending object
Change-Id: Ifb076ab38442f88f94a3e0c2ae1b19528a55f724
2020-12-22 09:42:32 +00:00
Michal Niewrzal
66d4d5eb48 satellite/metainfo/metabase: implement IterateObjectsAllVersions for
pending/committed objects

Change-Id: Ibf390821b6a23919078de4f18c2653e308320904
2020-12-22 10:27:09 +01:00
Ethan Adams
563197c628
satellite/overlay: Add index on nodes table (#4012)
satellite/accounting: Add index for project_id on bucket_storage_tallies
2020-12-21 12:48:48 -05:00
Michal Niewrzal
dad8360b39 satellite/metainfo/metabase: rename IterateObjectsAllVersions to
IterateObjectsAllVersionsWithStatus

We need different implementation for IterateObjectsAllVersions because
we want to iterate over all object without specifying object status.
Existing method will have new name but implementation details are not
changed.

Change-Id: I01b987996772fa7f8fd73da9910d52db2d1aa0d7
2020-12-21 16:47:32 +00:00
Ethan Adams
9b52283570
satellite/accounting: Add index for project_id on bucket_storage_tallies (#4010)
Change-Id: I47ab2d1e24f94307c3383c497cffe2a150fa8ab7
2020-12-21 11:42:00 -05:00
Kaloyan Raev
bafc6af992 ci: remove workaround for failing tests
Change-Id: I3eb673fae6c81bee17d7437cb870d5f5ba6978d5
2020-12-21 18:07:40 +02:00
Ethan Adams
6e501898c3
satellite/accounting: Performance improvements to getNodeIds used by GetBandwidthSince (#4009) 2020-12-21 16:37:01 +01:00
Egon Elbre
529bae3052 satellite/accounting: fix TestBilling_AuditRepairTraffic
Change-Id: I1743b2ab9e1afeb16ca31338a6b067da88e906ee
2020-12-21 13:18:35 +00:00
Michal Niewrzal
f3ef8088e7 satellite/metainfo/metabase: add Verify method for Pieces
This change adds Verify method for pieces to do some basic checks.

Change-Id: I0ff4313b594d2cb3aad7da545f940e10ee654b77
2020-12-21 12:49:03 +00:00
Michal Niewrzal
18825d1e0b satellite/{metainfo,gracefulexit}: fix failing tests
Change-Id: I3428ea601255c36a316732c9f75135d6e5fa4d79
2020-12-21 12:22:32 +00:00
Kaloyan Raev
4d37d14929 satellite/{metrics,repair}: adjust monitoring to new metainfo loop
Change-Id: I87a2145daa5ed49bb2c08d6967baa09c0b14b4c6
2020-12-21 09:05:17 +02:00
Jessica Grebenschikov
d961437889 satellite/orders: remove the config IncludeEncryptedMetadata
Since the Satellite now requires the order encryption functionality (since serial_number table is deprecated) to properly function, we can remove the config flag to turn on/off the feature.

Change-Id: Ie973f72a9a05a81cef9e53dc9c99d22c940c2488
2020-12-18 10:39:29 -08:00
Jessica Grebenschikov
da0327c9b7 satellite/dbcleanup: remove expired serial chore
Change-Id: Ib71d41eb6679d6435e5bc10b6244dac66380a74e
2020-12-18 09:36:28 -08:00
Jessica Grebenschikov
97a5e6c814 satellite/orders: stop inserting/reading from serial_numbers table
This PR contains the minimum changes needed to stop inserting into the serial_numbers table. This is the first step in completely deprecating that table.
The next step is to create another PR to remove the expiredSerial chore, fix more tests, and remove any other methods on the serial_number table.

Change-Id: I5f12a56ebf3fa4d1a1976141d2911f25a98d2cc3
2020-12-18 08:35:13 -08:00
Egon Elbre
4e19839669 satellite/accounting: fix TestProjectUsage_ tests
Change-Id: Id3972ca592c4459cc95668a53daca09f818e34b7
2020-12-18 16:41:02 +02:00
Egon Elbre
65b22be417 satellite/accounting/tally: use metabase
Change-Id: I6d49dc103a18e8a110bfa7775d53a65d208b6c2c
2020-12-18 16:18:03 +02:00
Michal Niewrzal
f7a31308db satellite/repair: enable TestRemoveExpiredSegmentFromQueue test
Change adds ability to set `now` time during test for repair.

Change-Id: Idb8826b7b58b8789b0abc65817b888ecdc752a3f
2020-12-18 10:58:05 +00:00
Michal Niewrzal
b3aa28cc02 satellite/gracefulexit: migrate to metabase
Change-Id: I8be9cc68894124427e4a30d7631126b3afb1f281
2020-12-18 10:57:39 +00:00
Michal Niewrzal
311b082838 satellite/metainfo: fix metainfo loop
This fix issues with passing observers between iteration methods.

It's not best implementation but I think we will need to optimize it
soon one way or another.

Change-Id: I574599bfd10822d84e2d2f1800bcd88e176a76ea
2020-12-18 10:38:12 +00:00
Michal Niewrzal
2111740236 Merge 'master' branch
Change-Id: Ib73af0ff3ce0e9a1547b0b9fc55bf88704f6f394
2020-12-18 09:13:24 +01:00
littleskunk
2437d5b171
satellite/access-grants: default auth service url (#4002)
* satellite/access-grants: default auth service url
2020-12-17 23:38:16 +01:00
paul cannon
d3604a5e90 satellite/repair: use survivability model for segment health
The chief segment health models we've come up with are the "immediate
danger" model and the "survivability" model. The former calculates the
chance of losing a segment becoming lost in the next time period (using
the CDF of the binomial distribution to estimate the chance of x nodes
failing in that period), while the latter estimates the number of
iterations for which a segment can be expected to survive (using the
mean of the negative binomial distribution). The immediate danger model
was a promising one for comparing segment health across segments with
different RS parameters, as it is more precisely what we want to
prevent, but it turns out that practically all segments in production
have infinite health, as the chance of losing segments with any
reasonable estimate of node failure rate is smaller than DBL_EPSILON,
the smallest possible difference from 1.0 representable in a float64
(about 1e-16).

Leaving aside the wisdom of worrying about the repair of segments that
have less than a 1e-16 chance of being lost, we want to be extremely
conservative and proactive in our repair efforts, and the health of the
segments we have been repairing thus far also evaluates to infinity
under the immediate danger model. Thus, we find ourselves reaching for
an alternative.

Dr. Ben saves the day: the survivability model is a reasonably close
approximation of the immediate danger model, and even better, it is
far simpler to calculate and yields manageable values for real-world
segments. The downside to it is that it requires as input an estimate
of the total number of active nodes.

This change replaces the segment health calculation to use the
survivability model, and reinstates the call to SegmentHealth() where it
was reverted. It gets estimates for the total number of active nodes by
leveraging the reliability cache.

Change-Id: Ia5d9b9031b9f6cf0fa7b9005a7011609415527dc
2020-12-17 21:30:17 +00:00
littleskunk
3feee9f4f8
satellite/accounting: default project limits (#4001) 2020-12-17 22:27:05 +01:00
Cameron Ayer
28eaae66af satellite/satellitedb: drop num_healthy_pieces column from injuredsegments
This column is no longer used as it has been replaced by the segment_health
column.

Change-Id: I6b4df89cd4f994d8418976f88e8c5f57615f8115
2020-12-17 20:17:08 +00:00
VitaliiShpital
f4bbd0f5df web/satellite: use brotli instead of gzip
WHAT:
we'll use brotli instead of gzip from now on

WHY:
better compression

Change-Id: Ibeadd6bfc783e9c15cf3f62f719af692071a7721
2020-12-17 19:23:44 +00:00
Michal Niewrzal
70ba4deea9 satellite/repair/checker: adjust irreparable part of repair checker
Change-Id: I0732104a97ba18a5359de3966cd692677a0ff790
2020-12-17 14:11:22 +00:00
Kaloyan Raev
9aa61245d0 satellite/audits: migrate to metabase
Change-Id: I480c941820c5b0bd3af0539d92b548189211acb2
2020-12-17 14:38:48 +02:00
Michal Niewrzal
2381ca2810 Merge 'master' branch
Change-Id: I4a3e45a2a2cdacfd87d16b148cfb4c6671c20b15
2020-12-17 13:17:17 +01:00
Fadila Khadar
c705237beb satellite/inspector: migrate to metabase
Change-Id: Ibfc12bf9bce0f9f065f4859a6818e5c18bbd526a
2020-12-17 11:56:08 +00:00
Michal Niewrzal
8d3ea9c251 satellite/repair/repairer: implement SegmentRepairer with metabase
Change-Id: I647c625e00a626c44e812602ad9bc3e85a7b602c
2020-12-17 10:47:21 +00:00
Kaloyan Raev
6206aa88e4 {satellite/metainfo,private/testplanet} use TestingAllSegments in tests
Change-Id: I8c641a24fabf3ea537312978a42501eab8d6a339
2020-12-17 09:58:31 +00:00
Egon Elbre
1728e45e55 satellite/metainfo/metabase: optimize DeleteBucket
Change-Id: If34cdeae0f688cb96717905fc8287c66ea3034be
2020-12-16 14:39:33 +00:00
Egon Elbre
12055e7864 all: minor cleanups
Change-Id: I4248dbe36a62a223b06135254b32851485a2eec1
2020-12-16 10:47:46 +00:00
Fadila Khadar
724b0f91eb satellite/gc: update tests to use metabase
Change-Id: I13c6c02a46254ea1d7176c0c6045fd24dd117a58
2020-12-16 10:38:24 +00:00
Egon Elbre
4706f01876 satellite/metainfo: add TestingAll{Segments,Objects}
Change-Id: Ia758c119d5ebd7bbb21216a463c99c2e9afcdeb0
2020-12-16 10:21:36 +00:00
Cameron Ayer
8c52bb3a18 satellite/checker: use numHealthy as segment health in repair queue
A few weeks ago it was discovered that the segment health function
was not working as expected with production values. As a bandaid,
we decided to insert the number of healthy pieces into the segment
health column. This should have effectively reverted our means of
prioritizing repair to the previous implementation.

However, it turns out that the bandaid was placed into the code which
removes items from the irreparable db and inserts them into the repair
queue.

This change: insert number of healthy pieces into the repair queue in the
method, RemoteSegment

Change-Id: Iabfc7984df0a928066b69e9aecb6f615253f1ad2
2020-12-15 17:16:59 -05:00
Jeff Wendling
9db28b92e7 satellite/console/wasm: fix issue from merge
there were two changes to this package: one that modified
some things and renamed access.go to main.go, and one that
reduced the binary sizes against main.go.

somehow the latter change was included on this branch but
the former was not, which made the folder contain both
main.go and access.go. building the package then fails
with duplicate definitions.

the easiest fix is to remove access.go which makes the folder
match the contents of the current master branch.

Change-Id: I7070a0d25a0399cef3166b8b2189427bc55587e0
2020-12-15 16:29:04 -05:00
Cameron Ayer
2ac72eaf16 satellite/repair/checker: add new monkit stats tagged with rs scheme
There is a new checker field called statsCollector. This contains
a map of stats pointers where the key is a stringified redundancy
scheme. stats contains all tagged monkit metrics. These metrics exist
under the key name, "tagged_repair_stats", which is tagged with the
name of each metric and a corresponding rs scheme.

As the metainfo observer works on a segment, it checks statsCollector
for a stats corresponding to the segment's redundancy scheme. If one
doesn't exist, it is created and chained to the monkit scope. Now we can call
Observe, Inc, etc on the fields just like before, and they have tags!

durabilityStats has also been renamed to aggregateStats.

At the end of the metainfo loop, we insert the aggregateStats totals into the
corresponding stats fields for metric reporting.

Change-Id: I8aa1918351d246a8ef818b9712ed4cb39d1ea9c6
2020-12-15 14:08:01 +00:00
Kaloyan Raev
4fba9921f6 satellite/metainfo/metabase: define ErrSegmentNotFound error class
This makes it easier to callers of GetSegmentByPosition to determine if
the segment is missing.

Change-Id: I2d8546dddf07dcf790d2f7c08d308ed589b34f2f
2020-12-15 15:48:41 +02:00
Michal Niewrzal
934ae32ca4 satellite/repair/checker: fix checker tests
Change-Id: I63d3368a07b800fdb10bb93b847eb32927b8c0dc
2020-12-15 10:47:42 +00:00
Michal Niewrzal
57f374af24 Merge 'master' branch
Change-Id: Idf6b10ea7ca94e4d232e6a3b6a38ef2e646ba197
2020-12-15 08:26:53 +01:00
Stefan Benten
9fe477899b satellite/satellitedb: add lint ignore rule to support staticcheck 2020.2
staticcheck 2020.2 is not liking our dbx files, so we need to ignore them.

Change-Id: I6becc3619bb088473f9776d0878ce240d4935936
2020-12-14 21:16:31 +00:00
Jessica Grebenschikov
3cc98de3ee satellite/console/wasm: reduce size to <9MB
Make changes so that we only import the necessary files from the console package so that the generated wasm code is as small as possible.

This change gets the compiled wasm code down to 8.6MB uncompressed and 2MB when compressed with `gzip --best`.

https://review.dev.storj.io/c/storj/storj/+/3396

Change-Id: Ifdd4be285810757b46bbbe43327c0d0139e5f8f7
2020-12-14 16:41:39 +00:00
Ivan Fraixedes
2dddcffe43 satellite/accounting/rollout: Remove unused variable
Remove a declared variable that's set by never read nor passed to any
function so it's unused code.

Change-Id: I8daf9d1f71d29ab39d7a80011d1b4813ada1c67d
2020-12-14 14:11:41 +00:00
Kaloyan Raev
fc85179a19 satellite/metainfo: refactor SegmentLocation.Index to SegmentPosition
Change-Id: Ic9403c8126712693326dd83d6ba4f3b84be3e0c7
2020-12-14 13:35:53 +02:00
Kaloyan Raev
7d8f19e94d satellite/metainfo: metainfo loop should yield StreamID for segments
Change-Id: If6c86add75ce79ffcfe95353225719c7d4b5a459
2020-12-14 13:04:38 +02:00
Michal Niewrzal
e7e6985ae9 satellite/metainfo/metabase: add UpdateSegmentPieces method
We need to be able to update just remote_pieces column in DB. This is
needed at least for repair process.

Change-Id: I20dcc9b06babfefbbf102f32b1d14946379f26c2
2020-12-14 10:25:09 +00:00
Kaloyan Raev
2bb010e7c5 cmd: remove segment reaper
It was designed to detect and remove zombie segments in the PointerDB.
This tool should be not relevant with the MetabaseDB anymore.

Change-Id: I112552203b1329a5a659f69a0043eb1f8dadb551
2020-12-14 09:36:37 +00:00
Michal Niewrzal
7e6e0d3e2e satellite/metainfo: metainfo loop implementation with metabase
Change-Id: Iadac469519de605a88e624df23265289771b2006
2020-12-11 16:15:57 +01:00
Brandon Iglesias
ca1e6b9756
Adding Fastly (#3994) 2020-12-11 15:53:05 +02:00
Stefan Benten
8fe829d5fd
build: add wasm bits to Dockerfile and bump to go v1.15.6 (#3992) 2020-12-11 02:23:39 +01:00
Jessica Grebenschikov
0649d2b930 satellite/repair: improve contention for injuredsegments table on CRDB
We migrated satelliteDB off of Postgres and over to CockroachDB (crdb), but there was way too high contention for the injuredsegments table so we had to rollback to Postgres for the repair queue. A couple things contributed to this problem:
1) crdb doesn't support `FOR UPDATE SKIP LOCKED`
2) the original crdb Select query was doing 2 full table scans and not using any indexes
3) the SLC Satellite (where we were doing the migration) was running 48 repair worker processes, each of which run up to 5 goroutines which all are trying to select out of the repair queue and this was causing a ton of contention.

The changes in this PR should help to reduce that contention and improve performance on CRDB.
The changes include:
1) Use an update/set query instead of select/update to capitalize on the new `UPDATE` implicit row locking ability in CRDB.
- Details: As of CRDB v20.2.2, there is implicit row locking with update/set queries (contention reduction and performance gains are described in this blog post: https://www.cockroachlabs.com/blog/when-and-why-to-use-select-for-update-in-cockroachdb/).

2) Remove the `ORDER BY` clause since this was causing a full table scan and also prevented the use of the row locking capability.
- While long term it is very important to `ORDER BY segment_health`, the change here is only suppose to be a temporary bandaid to get us migrated over to CRDB quickly. Since segment_health has been set to infinity for some time now (re: https://review.dev.storj.io/c/storj/storj/+/3224), it seems like it might be ok to continue not making use of this for the short term. However, long term this needs to be fixed with a redesign of the repair workers, possible in the trusted delegated repair design (https://review.dev.storj.io/c/storj/storj/+/2602) or something similar to what is recommended here on how to implement a queue on CRDB https://dev.to/ajwerner/quick-and-easy-exactly-once-distributed-work-queues-using-serializable-transactions-jdp, or migrate to rabbit MQ priority queue or something similar..

This PRs improved query uses the index to avoid full scans and also locks the row its going to update and CRDB retries for us if there are any lock errors.

Change-Id: Id29faad2186627872fbeb0f31536c4f55f860f23
2020-12-10 09:51:26 -08:00
Michal Niewrzal
b3acc1101a Merge 'master' branch
Change-Id: Iee99400c7095770e61cde94b3b2c8eb0ddec463d
2020-12-10 15:42:52 +01:00
Michal Niewrzal
c2a97aeb14 satellite/satellitedb: add ListAllBuckets method
We need to be able to list all buckets in DB without knowing project ID.
This method will be used to list buckets for metainfo loop
implementation based on metabase.

Change-Id: Iac75af0eee4f31e80a15577575a8249cbca787b2
2020-12-10 14:19:27 +00:00
Fadila Khadar
15add3c752 satellite/metainfo: enable deletion of a pending object.
Change-Id: I1ac24ad340d411c52b92ef18f51a684d9214ed47
2020-12-10 13:56:14 +01:00
Egon Elbre
365410d10b satellite/metainfo/metabase: add commit with specifying a list of segments
Change-Id: Ibb9999545691d150f36e20ce70ac00a3802ad7b2
2020-12-09 15:16:38 +02:00
Michal Niewrzal
f077564bb7 satellite/metainfo: setup access to metabase in metainfo loop
This is initial change to replace metainfo loop implementation with
metabase.

Change-Id: Ib23cb047bbbe6cff146305935a16ba833d40351a
2020-12-09 09:50:04 +01:00
Kaloyan Raev
3f5e07fb1f satellite/metainfo: more tests fixed
- TestBucketNameValidation
- TestBatch
- TestCommitObjectMetadataSize
- TestIDs

TestOverwriteZombieSegments is removed as not relevant to metabase.

Change-Id: I13cf5abe342089960628f185061303fd4f9d09a4
2020-12-08 14:00:24 +02:00
Kaloyan Raev
d0d0a192cd satellite/metainfo: fix TestEndpoint_DeleteObjectPieces tests
This also removes the
TestEndpoint_DeleteObjectPieces_ObjectWithoutLastSegment test case as it
does not seem relevant to metabase.

Change-Id: I06a0ecaa8232c10c15e433517a7ba056933bf858
2020-12-08 10:22:05 +00:00
Michal Niewrzal
fb5244e8f1 satellite/metainfo: migrate GetObjectIP endpoint method to metabase
Change-Id: I0da346fc2ce84787a09d2bb7c4ffc02a5d8a3b2f
2020-12-08 08:50:15 +00:00
Michal Niewrzal
218bbeaffa Merge 'master' branch
Change-Id: Ica5c25607a951076dd9f77e35e308062f71ce3f0
2020-12-07 15:05:52 +01:00
Stefan Benten
494bd5db81
all: golangci-lint v1.33.0 fixes (#3985) 2020-12-05 17:01:42 +01:00
Ethan Adams
f90ea10a4a
Allow for DB application names per process. (#3983) 2020-12-04 11:24:39 +01:00
Moby von Briesen
d75e4be11f satellite/{accounting, contact}: Remove periods and spaces from metrics.
Change-Id: I84179c2931293e3a1eb0ff8050416d25e481ce07
2020-12-03 15:33:01 +00:00
Kaloyan Raev
a7685f50c9 satellite/metainfo/metabase: set maxParts to MaxListLimit if greater
We should set the client requested maxParts to MaxListLimit if it is
greater than that value instead of returning an error.

MinIO default value for maxParts is 10,000 while the satellite's
MaxListLimit is 1,000. If we return an error, the ListParts with default
maxParts will throw an error.

Change-Id: I06739e1d8d8f96803eba491585395da0443aec04
2020-12-03 11:32:07 +00:00
Fadila Khadar
fd7cc20a7e satellite/metainfo/metabase: add DeletePendingObject
Change-Id: I7ba7d3d7f179712940e91b05501310279bbb97a4
2020-12-03 10:21:04 +00:00
Moby von Briesen
3fc76f4ffe satellite/downtime: Remove deprecated downtime tracking service.
We are no longer planning on implementing downtime penalization using
the method described in
docs/blueprints/archive/storage-node-downtime-tracking-deprecated.md.
Now, we are implementing the design described in
docs/blueprints/storage-node-downtime-tracking-with-audits.md.

This change removes the downtime estimation chores from the satellite
core as well as the package satellite/downtime. A future change will
remove the database table.

Change-Id: I1a1d3cf9dceeba36255d25243294865b89925518
2020-12-02 15:16:13 -05:00
Kaloyan Raev
bc25cc8ddd satellite/metainfo: add encryption info in metadata of list items
This is needed to resolve 'encryption type 0 is not supported' errors in
gateway.

Change-Id: I33c486968d5a1a1310b82e9778a8ce01429f7043
2020-12-02 13:34:41 +02:00
Michal Niewrzal
83e7cd2a40 satellite/metainfo/metabase: replace SQL substring with go code
We have some issues with SUBSTRING function on cockroachdb so for now we
are removing it from SQL query and replacing with go code.

Change-Id: I5be921211067d42e7d1a4997076bcfdbed9617a1
2020-12-02 10:47:18 +01:00
JT Olio
1728c3a992 satellite/dbx: standardize on assignment
Change-Id: I8f87bc8391e765e4480b0590d92d3601248e1f93
2020-12-01 16:10:18 +00:00
Jessica Grebenschikov
b261110352 satellite/orders: get bucketID from encrypted metadata in order instead of serial_numbers table
We want to stop using the serial_numbers table in satelliteDB. One of the last places using the serial_numbers table is when storagenodes settle orders, we look up the bucket name and project ID from the serial number from the serial_numbers table.

Now that we have support to add encrypted metadata into the OrderLimit, this PR makes use of that and now attempts to read the project ID and bucket name from the encrypted orderLimit metadata instead of from the serial_numbers table. For backwards compatibility and to ensure no errors, we will still fallback to the old way of getting that info from the serial_numbers table, but this will be removed in the next release as long as there are no errors.

All processes that create orderLimits must have an orders.encryption-keys set. The services that create orderLimits (and thus need to encrypt the order metadata) are the satellite apiProcess, the repair process, audit service (core process), and graceful exit (core process). Only the satellite api process decrypts the order metadata when storagenodes settle orders. This means that the same encryption key needs to be provided in the config for the satellite api process, repair process, and the core process like so:
orders.include-encrypted-metadata=true
orders.encryption-keys="<"encryptionKeyID>=<encryptionKey>"

Change-Id: Ie2c037971713d6fbf69d697bfad7f8b672eedd66
2020-12-01 15:29:32 +00:00
Kaloyan Raev
e8378a87bf satellite/metainfo: include redundancy in satStreamID of ListObjects
This fixes the issue with using UploadID from the ListMultipartUploads
response.

Change-Id: If077a5804d03cc139f75febf472d5fb73f0f7929
2020-12-01 15:11:34 +00:00
Michal Niewrzal
d5c0264163 satellite/metainfo: implement deleteBucketObjects with metabase objects
iterator

This method replaces `deleteByPrefix` as at the moment only function of
this method was to delete objects in a bucket.

Change-Id: I5266103672003fbd64f3847f53760b1ba0016fe2
2020-12-01 14:54:23 +00:00
Michal Niewrzal
f08e34f15e satellite/metainfo: fix listing objects when cursor is specified
We need to combine prefix + cursor to iterate objects correctly.

Change-Id: I2f4a29ba3ee65ce52eb1e8b88a3b09cf93e0b2bb
2020-12-01 15:01:44 +01:00
Michal Niewrzal
e4c4ab06ba satellite/metainfo: add testing methods for getting objects and segments
Change-Id: I4229d80090609df27d998a90bdb3137469dba042
2020-12-01 12:44:08 +00:00
Kaloyan Raev
46839b992a satellite/metainfo: expired deletion service to use Metabase
It also sets the Expires time correctly in the Metabase when uploading
objects.

Change-Id: Iec1b3ec8d11346a91dfc2ba23a1b08edec4a84d3
2020-12-01 09:12:14 +00:00
JT Olio
70b91aac54 satellitedb: remove cruft caused by https://review.dev.storj.io/c/storj/storj/+/3223
Change-Id: I198bb2f869cc7177b9ecafdd8932bbf2b58be5b8
2020-12-01 00:16:26 +00:00
Yingrong Zhao
d8ba7b3057 satellite/console: only allow project member to get all bucket names
Change-Id: I8ceb0b7eb19e221072b4ff3411a4ec1a7817d16f
2020-11-30 15:41:35 -05:00
Michal Niewrzal
5a7bc9657d Merge 'master' branch
Change-Id: If583132a821274dc4b78cf5f72b853ba8460c619
2020-11-30 12:57:22 +01:00
Egon Elbre
f456d7ce03 satellite: remove implementation detail from DB interface
Which database access and how it internally does migrations is an
implementation detail and does not belong in the requirements interface.

Change-Id: Ia4a6994f39470063a96a8e5f3a1bd27aa79fe5cd
2020-11-30 13:29:20 +02:00
Egon Elbre
28ea63be92 satellite/repair: avoid TestDBAccess
Change-Id: I34adb58cd67fba5917032f2f328d75b1c4afdbbf
2020-11-30 13:29:08 +02:00
VitaliiShpital
0771cdb0b1 web/satellite: create access grant: generate gateway credentials step
WHAT:
generate gateway credentials step for create access grant flow

WHY:
part of the flow

Change-Id: I6496712b43f78a818ba0582b586cfae3a44683e6
2020-11-30 10:36:29 +00:00
VitaliiShpital
bb7677a85f web/satellite: get gateway credentials request using url from config
WHAT:
POST request to get gateway credentials using access grant.
Put request url to config and use it for request.

WHY:
to show gateway credentials on UI

Change-Id: I15ef43ecdeed69b0961d5796aacb47f36d560b1b
2020-11-30 10:36:23 +00:00
Michal Niewrzal
21602e0494 satellite/metainfo: enable commented test
Test was commented to make uplink refactoring possible. Now we can bring
back this test.

Change-Id: I0511b76073efaafed8aac97f8e845dcec93dd059
2020-11-30 10:49:23 +01:00
JT Olio
71e11b27f3 satellite/dbx: only retry with cockroach
Change-Id: Id3630c26dbfda36dcbece2849e2353d5ab2882af
2020-11-29 18:10:07 -07:00
JT Olio
bd23d12bb9 satellite/dbx: add cockroach retries for other QueryContext operations
Change-Id: Ia30fbba55c926892702fa96fb9dd01b75347d351
2020-11-29 18:09:56 -07:00
JT Olio
ea2f39ca7f satellite/dbx: add retries for QueryRowContext-based operations
Change-Id: Ie2527b673dd4ce5250cf5c0cbf8f14921262f665
2020-11-29 18:09:46 -07:00
JT Olio
d3b0691bbd satellite/dbx: import dbx templates
these are unchanged from storj.io/dbx.

we're importing them because in a later commit we
will change them, and it'd be nice to see that
diff as a separate commit.

Change-Id: I8315130ed6bab397bd65b9a1a90c29d130b8c02d
2020-11-29 18:09:33 -07:00
JT Olio
5d8a67a4f7 satellitedb: retry GetBandwidthSince on cockroach
Change-Id: I2bf20f3a19e7f3af97630d8a679410feba70661e
2020-11-29 16:36:15 -07:00
Ethan
5dc013d3bd satellite/overlay: Add retry to all selects in overlaycache
Change-Id: I0356d71a35701f8e0ca04a34b2bb2aea666c1394
2020-11-29 16:46:57 -05:00
JT Olio
6bce907cb0 satellite: try to stream rollups to aggregation function to use less memory
this change tries really hard to never have all of the storage node
rollups in memory at the same time, up until the rollups are actually
getting summed together.

Change-Id: If67f49e7d71106798d996a6850b3e48671bd9e18
2020-11-29 10:26:32 -07:00
JT Olio
6aae21541f satellitedb: do saverollup in batches
Change-Id: I78278a192cba60541eee2986f54a88d5a479bd3e
2020-11-28 19:26:46 -07:00
JT Olio
0ba516d405 satellite: support pointing db components at different databases
the immediate need is to be able to move the repair queue back out
of cockroach if we can't save it.

Change-Id: If26001a4e6804f6bb8713b4aee7e4fd6254dc326
2020-11-28 18:39:16 +00:00
Moby von Briesen
75f0f713a3 satellite/repair/checker/checker.go: Use number of healthy pieces instead of SegmentHealth for injured segments queue.
We did not test the SegmentHealth function with actual production
values, and it turns out that values such as 52 healthy, 35 minimum
result in +Inf segment health - so pretty much all segments put into the
repair queue have the same health, which means we effectively aren't
sorting by health.

This change inserts numHealthy as segment health into the database so
the segments are ordered as they were before. We need to refine the
SegmentHealth function before we can support multi RS.

Change-Id: Ief19bbfee3594c5dfe94ca606bc930f05f85ff74
2020-11-28 12:16:32 -05:00
Ivan Fraixedes
7eb3b2d6d0
satellite/gc: Init map with an aprox size
Because the PieceTracker receives a piece count per nodes which is an
approximation of the number of nodes that they are going to be reported
by the metainfo loop so we can use as a good guess of the map's size and
initialized with it.

Change-Id: I644db40926c03e4c457457fb41d2ec1da059cea6
2020-11-27 10:44:19 +01:00
Ivan Fraixedes
319d2cad11
satellite: Fix typo in a comment
Change-Id: I151b824e868db1cc1e8b8e8af9f35b027db1e6ff
2020-11-26 15:44:49 +01:00
Michal Niewrzal
cfb45a785c satellite/metainfo/metabase: add TotalPlainSize to objects table
We need to know how large is object in general and we shouldn't query
segments for that.

Change-Id: Ie316e1ec38bfc39065e6046b79b55d58f3ce9d53
2020-11-25 14:35:28 +01:00
Michal Niewrzal
8ceef9f357 satellite/metainfo: temporary disable one assertion in test
This is need to merge https://review.dev.storj.io/c/storj/uplink/+/3208
, after that this code will be back.

Change-Id: If9f2f1db95c7a1bba64a41c45a39bd3096a519e7
2020-11-25 13:21:41 +00:00
Michal Niewrzal
983b1737c2 satellite/metainfo: implement IsBucketEmpty with metabase
Use objects iterator to check if bucket is empty.

Change-Id: I590bb7258a47dd816f079039a1151df809dd2637
2020-11-25 12:20:30 +00:00
Egon Elbre
3792e2921c satellite/accounting/tally: make test less fragile
MetadataSize can slightly vary and checking for exact value makes
difficult to change what's being encoded in metadata.

Change-Id: I5f1ade41bc26d115e6743367ee35cf1ba74795c9
2020-11-25 13:33:24 +02:00
Michal Niewrzal
46102c1942 satellite/metainfo: change ListSegments required permission to
Read/Download

We should treat listing segments as a part of download process, not
listing.

Change-Id: Iae72c97c77976c427f668a96486af613d58da3b9
2020-11-24 17:50:33 +01:00
Egon Elbre
dad36179c6 satellite/metainfo/metabase: fix segment listing with cursor.Part
Change-Id: I910888d435b11d2d4687ea0042998a1078cd6443
2020-11-24 15:31:57 +02:00
Kaloyan Raev
67c210a64c satellite/metainfo: set list cursor to version 1
Otherwise, if left to default version 0, the iterator will include the
cursor item in the result, which fails some tests.

Change-Id: I85103a36852477f371ec46c673a82c2e129978b7
2020-11-24 10:17:42 +00:00
Kaloyan Raev
53b7fd7b00 satellite/{audit,gracefulexit}: remove logic for PieceHashesVerified
We now have the piece hashes verified for all segments on all production
satellites. We can remove the code that handles the case where piece
hashes are not verified. This would make easier the migration of
services from PointerDB to the new metabase.

For consistency, PieceHashesVerified is still set to true in PointerDB
for new segments.

Change-Id: Idf0ccce4c8d01ae812f11e8384a7221d90d4c183
2020-11-24 11:09:48 +02:00
Michal Niewrzal
efaba85c73 Merge 'master' branch
Change-Id: I3520b3e327732929f5167b07a15ddb92d26cae1b
2020-11-24 10:03:20 +01:00
Egon Elbre
9de1617db0 satellite/orders: ensure encryption keys handles set twice
Currently flag parsing seems to call Set twice, which causes problems
with encryption keys. We can clear for every set for now.

Change-Id: Id5c695b4020194ac1c50a2da9c7d2a896cb9216f
2020-11-23 19:47:22 +00:00
Moby von Briesen
575f50df84 satellite/repair: Update repair override config to support multiple RS schemes.
Rather than having a single repair override value, we will now support
repair override values based on a particular segment's RS scheme.

The new format for RS override values is
"k/o/n-override,k/o/n-override..."

Change-Id: Ieb422638446ef3a9357d59b2d279ee941367604d
2020-11-23 18:01:15 +00:00
Egon Elbre
55d5e1fd7d satellite/orders: ensure that expired deletion doesn't stall
Add checks to ensure that when somebody uses empty options, the deletion
doesn't loop infinitely.

Change-Id: I1738fb1e7e1f8efbbb954c491cb6489f7bcdc2db
2020-11-23 14:52:40 +02:00
Jessica Grebenschikov
5beb2f5737 satellite/orders: add factory function to encryption key
Change-Id: I9a1020c63e4ebc6d73683cf1749366e9b9f20f07
2020-11-20 11:40:15 -08:00
Ethan
2b92bba563 satellite/satellitedb/orders: Handle serial_numbers deletes in smaller increments on CRDB
CRDB doesn't like large deletes. While testing in the POC environment we found that deletes on the serial_numbers table could take hours.  This change limits deletes to 1000 at a time (configurable) to avoid blocking other queries.

Change-Id: I08455e25db1574579dd4d7b7125a08e9c913dff1
2020-11-20 13:44:52 +00:00
Michal Niewrzal
5be4d413b0 satellite/metainfo: set PlainSize for inline segment
Plain size is required by metabase implementation.

Change-Id: I6739744bff4ce485e0066f2a49acdf38560d31fb
2020-11-20 13:54:14 +01:00
Kaloyan Raev
05fe497a6f satellite/metainfo: add stream id when listing pending objects
Change-Id: Idcdffd70b82c0fe4f79635fc193604b66101ef76
2020-11-19 14:22:09 +02:00
Moby von Briesen
a8b66dce17 satellite/accounting: account for old orders that can be submitted in satellite rollup
With the new phase 3 order submission, orders can be added to the
storage and bandwidth rollup tables at timestamps before the most recent
rollup was run. This change shifts the start time of each new rollup
window to account for any unexpired orders that might have been added
since the previous rollup.

A satellitedb migration is necessary to allow upserts in the
accounting_rollups table when entries with identical node_ids and
start_times are inserted.

Change-Id: Ib3022081f4d6be60cfec8430b45867ad3c01da63
2020-11-18 14:46:00 -05:00
Michal Niewrzal
1465539220 satellite/metainfo: ignore empty inline segments for backward
compatibility

Change-Id: I2567d77e172391945dee24cea8e110fa1963d663
2020-11-18 14:12:03 +00:00
Egon Elbre
aeb801604e {satellite,storagenode}/orders: fix flaky tests
Before manipulating order information on storagenodes we need to wait
for the orders to propagate to the database. Some of that happens
async with uplink.

Change-Id: Iaacfd7db0909ab5d2831d06388e5fb27b6d4778f
2020-11-18 13:44:02 +00:00
Michal Niewrzal
70b1c7aa17 satellite/metainfo: check if EncryptedMetadataNonce was set
It possible that old uplinks won't set it.

Change-Id: Ifd3c4d4a45c5636cfeb70b0a37bcf1305a097727
2020-11-18 14:04:31 +01:00
Kaloyan Raev
a30f5d7ec0 satellite/metainfo: use Object.Status from ObjectListRequest
Change-Id: Ie2e48d2eedf69bd0f04b7c9d6e7e217fec6b0267
2020-11-18 13:31:52 +02:00
paul cannon
2b59640f18 cmd/satellite: ignore Canceled in exit from repair worker
Firstly, this changes the repair functionality to return Canceled errors
when a repair is canceled during the Get phase. Previously, because we
do not track individual errors per piece, this would just show up as a
failure to download enough pieces to repair the segment, which would
cause the segment to be added to the IrreparableDB, which is entirely
unhelpful.

Then, ignore Canceled errors in the return value of the repair worker.
Apparently, when the worker returns an error, that makes Cobra exit the
program with a nonzero exit code, which causes some piece of our
deployment automation to freak out and page people. And when we ask the
repair worker to shut down, "canceled" errors are what we _expect_, not
an error case.

Change-Id: Ia3eb1c60a8d6ec5d09e7cef55dea523be28e8435
2020-11-17 21:37:59 +00:00
Michal Niewrzal
9bdc8ebf75 satellite/metainfo: adjust GetObject to handle
EncryptedMetadataEncryptedKey

Change-Id: If159fdcc9f37e9449a7d8459a277366d6d6b5802
2020-11-17 16:38:10 +00:00
Egon Elbre
72fed3b3fb satellite/metainfo/metabase: add slow non-recursive listing
Change-Id: Ide60640400caea6474465af5224010fde60826c7
2020-11-17 16:47:29 +02:00
Egon Elbre
5e7d47b9cd satellite/metainfo: code cleaning
Rename iterator tests to iterator_test.go.
Add Delimiter to metabase package.

Change-Id: Icea6542b54176d716daba4afad901713f1dcac90
2020-11-17 15:50:40 +02:00
Egon Elbre
2e426ef0a8 satellite/metainfo/metabase: iterator compatibility
Old iterator returns object keys without prefixes, this helps to reduce
the bandwidth from the database. The endpoint also doesn't send the
prefixes.

Change-Id: I77d85dae671ee3a16abe75db14e19674e80abaf4
2020-11-17 15:50:39 +02:00
Fadila Khadar
c9bbd83f96 satellite/metainfo/metabase: align object status to those already in use
Change-Id: I33b2ce7282838e1172b7cb92d4e76169c96fddcb
2020-11-17 14:07:42 +01:00
Michal Niewrzal
b7df41a0a1 satellite/metainfo/metabase: add EncryptedMetainfoEncryptedKey parameter
to metabase

* EncryptedMetainfoEncryptedKey added to CommitSegment and
UpdateMetadata request
* EncryptedMetainfoEncryptedKey returned with GetObject response and all
delete responses
* EncryptedMetainfoEncryptedKey returned with object iterator results

Change-Id: I917541ab5f3e1863bc8f238d17a15fbf72a23025
2020-11-17 10:19:12 +00:00
Moby von Briesen
0ec685b173 satellite/{satellitedb, repair/{queue, checker}}: Use new column "segmentHealth" instead of "numHealthy" in injured segments queue
We plan to add support for a new Reed-Solomon scheme soon, but our
repair queue orders segments by least number of healthy pieces first.
With a second RS scheme, fewer healthy pieces will not necessarily
correlate to lower health.

This change just adds the new column in a migration. A separate change
will add the new health function.

Right now, since we only support one RS scheme, behavior will not
change. Number of healthy pieces is being inserted as "segment health"
until the new health function is merged.

Segment health is calculated with a new priority function created in
commit 3e5640359. In order to use the function, a new config value is
added, called NodeFailureRate, representing the approximate probability
of any individual node going down in the duration of one checker run.

Change-Id: I51c4202203faf52528d923befbe886dbf86d02f2
2020-11-16 21:18:09 +00:00
VitaliiShpital
51a712f9e8 satellite/console: get all bucket names endpoint and service method
WHAT:
new endpoint for fetching all bucket names

WHY:
used by new access grant flow

Change-Id: I356a3381359665fd2726120139b34b1e611fe3c4
2020-11-16 17:51:40 +02:00
Egon Elbre
402cfcb7c6 satellite/metainfo/metabase: add prefix to iterator
Change-Id: I2dd7b7917aed9def0be3f131ca60ab12f2077d83
2020-11-16 17:38:59 +02:00
Egon Elbre
7aba265db0 satellite/metainfo/metabase: add status option to iteration
Change-Id: Ie9d1a3704d8d08e3421ba2244a1f3be57d15ed8f
2020-11-16 15:41:20 +02:00
Kaloyan Raev
a015f41927 satellite/metainfo: ListObjects to use Metabase API
Change-Id: If75d04b9a44f08515be717a85f397b66f8a5c56f
2020-11-16 13:27:17 +02:00
Michal Niewrzal
7c384c8293 Merge 'master' branch
Change-Id: I1eefd5a56449e577820977d61fa4a22bdd4fc230
2020-11-16 10:02:54 +01:00
Michal Niewrzal
cd7d911b26 satellite/metainfo: implement ListSegments with metabase
Change-Id: I78fb2400fed0bb01c5e05bd5bbcddef5e4a3bdf0
2020-11-16 08:33:35 +00:00
Jessica Grebenschikov
f558cc825e satellite/orders: add storagenode_bw_phase2 table and dont delete tallies for longer
It turns out we need to make 2 more changes in order for the new order submission phase 3 to get deployed.

This PR makes 2 changes:
1) when the rollup service deletes tallies, we now keep tallies around until orders expire (vs 1 day like before).
2) the reported rollup chore will now write the storagenode_bandwidth_rollups to a new table _phase2 as an intermediary step so it doesn't conflict with phase 3 order settlement.

These changes need to be deployed for 2 days before we can turn on phase 3 of the new orders settlement workflow.

Change-Id: Iafbff577ba7d55f8f17b7db857311b2ce799de60
2020-11-13 17:15:24 +00:00
Kaloyan Raev
ff32268310 satellite/metainfo/metabase: DeleteExpiredObjects
Change-Id: I16293ce1a33f714aa005022daf3d40670cc66b04
2020-11-13 15:31:31 +02:00
Fadila Khadar
a749ac9f47 satellite/metainfo/metabase: iterateObjectsAllVersions
Iterate through all objects in a bucket
recursive, no prefix support

Change-Id: Ieaad9fbd0b5c1593554db9d543f4ee04851bac28
2020-11-13 13:50:08 +01:00
Yaroslav Vorobiov
1b4bfbb9d2 multinode/console: nodes addition and removal
Change-Id: I60c685953a8d0e24f78b1414c34a28d4b87863b0
2020-11-12 20:26:08 +02:00
Jessica Grebenschikov
226e13e616 satellite/cosole: add tests for wasm access code
Change-Id: I78f71b2f0bef03b6e87cd7d79ccaef5f45393b55
2020-11-12 08:03:36 -08:00
Egon Elbre
8182d8a726 satellite/metainfo/metabase: use txutil.WithTx in delete
Change-Id: I75965c4dcf57479ace2367f5d3069d785628e86a
2020-11-12 14:38:36 +02:00
Egon Elbre
bc460cd62d satellite/metainfo/metabase: use txutil.WithTx in commit
Change-Id: I84347291179c7042e22bda703a1c9bd80a1513a0
2020-11-12 14:38:33 +02:00
Michal Niewrzal
9d25b3a7d3 satellite/metainfo/metabase: add ListSegments request
Change-Id: Ie777a0f791685beab3973e76b10a9ce76398e7d7
2020-11-12 11:46:17 +01:00
paul cannon
3e56403599 satellite/repair: add a repair health function
This will be used to rank segments in need of repair for attention by
the repair workers.

Change-Id: I5b70650cec933696b4c6d73bb7efb97e3efdf24a
2020-11-11 18:48:51 +00:00
Michal Niewrzal
fd707392e7 satellite/metainfo: set RedundancyScheme for begin segment
response

Multipart upload needs it for uploading parts.

Change-Id: I34886a9dabd17ecb7d55d4c6e19914a1c821f638
2020-11-11 17:25:52 +00:00
Jeff Wendling
31533ed1a1 satellite/console/wasm: remove storj.io/uplink deependency
Change-Id: Iee95389e4ba24618e31aff7be44d05377b2e2419
2020-11-11 16:51:14 +00:00
Kaloyan Raev
9dcd0936d7 satellite/metainfo: MakeInlineSegment to use metabase
Change-Id: Ib32c1d9e91e3f13b4e44abefb2b5d0352b64bd79
2020-11-11 15:24:21 +02:00
Cameron Ayer
da9f1f0611 satellite/repair: add monkit counter for segments below minimum required
The current monkit reporting for "remote_segments_lost" is not usable for
triggering alerts, as it has reported no data. To allow alerting, two new
metrics "checker_segments_below_min_req" and "repairer_segments_below_min_req"
will increment by zero on each segment unless it is below the minimum
required piece count. The two metrics report what is found by the checker
and the repairer respectively.

Change-Id: I98a68bb189eaf68a833d25cf5db9e68df535b9d7
2020-11-11 12:48:23 +00:00
Michal Niewrzal
3fe16f4003 satellite/metainfo: upload/download with metabase
This change is adjusting metainfo endpoint to use metabase for uploading
and downloading remote objects. Inline segments will be added later.

Change-Id: I109d45bf644cd48096c47361043ebd8dfeaea0f3
2020-11-11 12:13:52 +00:00
Yingrong Zhao
2ce3170bb4 satellite/console/wasm: expose method to add caveats in the browser
This PR does the following three things:
    1. Defines a high-level interface for this wasm package
        - All return value from this package will be wrapped with an
          result object that contains a value field and an error field
    2. Exposes two new functions to allow users to add permissions for a
       given API key
        - newPermission()
        - setAPIKeyPermission()
    3. Adds API documentation for the newly added API functions

Change-Id: Id995189702b369bba18fa344bef4ddfb0f3f1f44
2020-11-10 20:10:53 +00:00
Fadila Khadar
2bd239bb75 satellite/metainfo/metabase: tests commit inline segment
Change-Id: Ifad461041c7b9d8c78fc9f998a4e708853087b02
2020-11-10 15:00:35 +00:00
Michal Niewrzal
48f1f7132e satellite/metainfo: set RS defaults
While resolving conflicts with `master` I missed this change which is
needed e.g. to run storj-sim.

Change-Id: I56a548ed92b978510526c26c81af03051acfde2f
2020-11-10 13:57:27 +01:00
Brandon Iglesias
3ba52b25a9
satellite/rewards: update partners to include MAXN 2020-11-10 14:08:32 +02:00
Michal Niewrzal
7dde184cb5 Merge 'master' branch
Change-Id: I6070089128a150a4dd501bbc62a1f8b394aa643e
2020-11-10 11:58:59 +00:00
Michal Niewrzal
92f9251074 satellite/metainfo/metabase: add GetSegmentByOffset request
Change-Id: Iea2ce439ec1f7285e447d590c1297294204edc2e
2020-11-10 11:33:34 +00:00
Kaloyan Raev
3ed4183e52 satellite/metainfo: delete object to use metabase
Change-Id: I2ab63a719fdbc1f8a7fbb4ad73d51a2d2dcfadc6
2020-11-10 09:55:23 +00:00
Moby von Briesen
db6bc6503d satellite/metainfo: Update metainfo RS config to more easily support multiple RS schemes.
Make metainfo.RSConfig a valid pflag config value. This allows us to
configure the RSConfig as a string like k/m/o/n-shareSize, which makes
having multiple supported RS schemes easier in the future.

RS-related config values that are no longer needed have been removed
(MinTotalThreshold, MaxTotalThreshold, MaxBufferMem, Verify).

Change-Id: I0178ae467dcf4375c504e7202f31443d627c15e1
2020-11-09 22:16:13 +00:00
Egon Elbre
184e1ffa9d satellite/metainfo/metabase: commit object without proofs
Change-Id: I788111b9e5bdc961be9e731f8203e52ca9da9dca
2020-11-09 15:18:07 +00:00
Cameron Ayer
d63b7658e8 satellite/repair: fix lastSeenSegmentKey bug in IrreparableProcess
A change was made to use a metabase.SegmentKey (a byte slice alias)
as the last seen item to iterate through the irreparable DB in a
for loop. However, this SegmentKey was not initialized, thus it was
nil. This caused the DB query to return nothing, and healthy segments
could not be cleaned out of the irreparable DB.

Change-Id: Idb30d6fef6113a30a27158d548f62c7443e65a81
2020-11-09 14:48:15 +00:00
VitaliiShpital
f8c3848c78 satellite/console: change user's email endpoint/feature
WHAT:
change user's email endpoint and appropriate service method was implemented

WHY:
make it possible to change user's email for temporary filezilla account

Change-Id: Ieea41bf49819a42b5f433e8dfaeec24c6d5ddc9f
2020-11-06 11:54:07 +00:00
Egon Elbre
0a77deec81 satellite/metainfo/metabase: basic migrate
Change-Id: Ia34f38e763af9ba88f75038b8ecfc9eb6550eae0
2020-11-05 16:31:02 +02:00
Kaloyan Raev
56e6bc884c metabase: add DeleteObjectsAllVersions method
Change-Id: I3252dd0aac595b233f4861c3c1375622326d955d
2020-11-05 12:17:54 +02:00
Michal Niewrzal
2398afe985 satellite/metainfo/metabase: add GetLatestObjectLastSegment request
Change-Id: I50c452e379f78864b38921f31e53a0554642ab02
2020-11-05 08:20:34 +00:00
jessicagreben
c4c29e370a wasm: add webassembly code for creating access grant in console web UI
Change-Id: I3c6d9afc660f3d959d6138db84341e9460b877a1
2020-11-04 12:08:30 -08:00
Kaloyan Raev
f986fdfe72 metabase: optimize delete segments
Segments are deleted in a single DB query instead of a separate one for
each segment.

Change-Id: Icb9ae92b4b8405030c9d9ef57ba30bee6a35515c
2020-11-04 11:59:56 +02:00
Ivan Fraixedes
2dffaebc6f satellite/accounting: Fix and enhance code doc comments
Fix and enhance the source code documentation comments for the
satellite/accounting packaged.

Change-Id: I965742cf378e8b6b80d18bc84a4ff76e9af1e8b7
2020-11-04 09:50:48 +00:00
paul cannon
8616fc146d satellite/orders: send IPs for graceful exit
Storage nodes undergoing Graceful Exit have up to now been receiving
hostnames for all other storage nodes they need to contact when
transferring pieces. This adds up to a lot of DNS lookups, which
apparently overwhelm some home routers. There does not seem to be any
need for us to send hostnames for graceful exit as opposed to IP
addresses; we already use IP addresses (as given by the last_ip_port
column in the nodes table) for all the GET and PUT orders we send out.

This change causes IP addresses to be used instead.

I started trying to construct a test to ensure that the behavior
changed, but it was rabbit-holing, so I've begun to feel that maybe this
change doesn't require one; it is a very simple change, and very much of
the same nature as what we already do for IPs in CreateGetOrderLimits
and CreatePutOrderLimits (and others).

Change-Id: Ib2b5ffe7a9310e9cdbe7464450cc7c934fa229a1
2020-11-04 00:17:20 +00:00
Michal Niewrzal
4134100d80 satellite/metainfo/metabase: add UpdateObjectMetadata request
Change-Id: Iec73000e58f373294f37ef3e4470412d1c5f1340
2020-11-03 11:08:41 +00:00
Kaloyan Raev
89cefa9045 metabase: add Objects info to DeleteObjectResult
Change-Id: If7ed2ed0bbba08ad971c64852d51ff1a066c4f1e
2020-11-03 12:45:49 +02:00
Kaloyan Raev
f7aa5b4e39 metabase: CommitObject return metabase.Object
An immediate purpose is to be used in test cases for deleting objects

Change-Id: I481d3be5800e55da231ecb628ef7eb3ea011d196
2020-11-03 09:57:44 +00:00
Kaloyan Raev
7e9ac2ec0a metabase: use storj.ErrObjectNotFound instead of custom error
Change-Id: Ie90e438f7d7b58e6ea986359849dd8319433ae9f
2020-11-03 09:35:22 +00:00
Cameron Ayer
dc67ce74c9 satellite: remove IsUp field from overlay.UpdateRequest
With the new overlay.AuditOutcome type for offline audits, the
IsUp field is redundant. If AuditOutcome != AuditOffline, then
the node is online.

In addition to removing the field itself, other changes needed
to be made regarding the relationship between 'uptime' and 'audits'.
Previously, uptime and audit outcome were completely separated. For
example, it was possible to update a node's stats to give it a
successful/failed/unknown audit while simultaneously indicating that
the node was offline by setting IsUp to false. This is no longer possible
under this changeset. Some test which did this have been changed slightly
in order to pass.

Also add new benchmarks for UpdateStats and BatchUpdateStats with different
audit outcomes.

Change-Id: I998892d615850b1f138dc62f9b050f720ea0926b
2020-11-02 15:34:17 -05:00
Michal Niewrzal
ae17ae73d7 satellite/metainfo/metabase: add metadata and nonce to CommitObject
request

Change-Id: I584775800963fc2167c749f3da20475036fa5325
2020-11-02 16:49:18 +01:00
Egon Elbre
7183dca6cb all: fix defers in loop
defer should not be called in a loop.

Change-Id: Ifa5a25a56402814b974bcdfb0c2fce56df8e7e59
2020-11-02 15:06:38 +02:00
Egon Elbre
716068a1e0 Merge branch 'master'.
Change-Id: Ic14325edc291573582dce0cea3e04991a820b48b
2020-11-02 13:02:01 +02:00
Egon Elbre
fd8e697ab2 {satellite,storagenode}/internalpb: use specific package name
Ensure we don't register types with the same name into protobuf.

Change-Id: I53d025863fff8c91a067ca5819befa87eb5e35bb
2020-10-30 17:31:08 +02:00
Michal Niewrzal
0205f0d807 satellite/metainfo: fix usage of types from internalpb
After moving SatStreamID and SatSegmentID from common I missed changing
some methods in metainfo endpoint. This change is a fix for that.

Change-Id: I34e121fce47371ee4cfd92cce03809520b68859f
2020-10-30 16:03:45 +02:00
Egon Elbre
77c4f99fa0 satellite/internalpb: move delegated_repair.proto
Change-Id: If4f37c52b151e09cf35d2145b463ef1e9ab529ae
2020-10-30 15:31:32 +02:00
Egon Elbre
11338e9beb satellite/internalpb: move audithistory.pb
Change-Id: I8eee84d49ed90459168ddaf04ae57f790c2a22c4
2020-10-30 15:30:11 +02:00
Michal Niewrzal
517212bfa8 satellite/metainfo: fix usage of types from internalpb
After moving SatStreamID and SatSegmentID from common I missed changing
some methods in metainfo endpoint. This change is a fix for that.

Change-Id: I3344623dc7acfa73db6c20cd3212301e74335857
2020-10-30 13:20:58 +00:00
Egon Elbre
7ce372c686 satellite/internalpb: add inspectors
Change-Id: Ib688e43d05135c0c31ae95df533f1e4535ea396a
2020-10-30 13:28:17 +02:00
Egon Elbre
004e610d0f satellite/internalpb: move datarepair.pb to internal
Change-Id: If901d9ff4e5ee6715b963eeeb46513a602a44b3d
2020-10-30 13:28:14 +02:00
Kaloyan Raev
b8c6fb764c satellite/metainfo: add metabase to metainfo service
Change-Id: Ie3ff238b138d8a57d99e32b13f7a71aa624d53e3
2020-10-30 12:49:47 +02:00
Egon Elbre
995900e02f Merge branch 'master' into multipart-upload
Change-Id: I8717f9f7cc54f3934402a98f21129e16841b25bd
2020-10-30 11:48:05 +02:00
Michal Niewrzal
8f26f66da0 internalpb: move satellite specific protobuf types storj/storj
We have some types that are only valid for satellite usage. Such types
are SatStreamID and SatSegmentID. This change moves those types to
storj/storj and adds basic infrastructure for generating code.

Change-Id: I1e643844f947ce06b13e51ff16b7e671267cea64
2020-10-30 08:49:16 +00:00
Egon Elbre
e0dca4042d all: add pprof labels for debugger
By using pprof.Labels debugger is able to show service/peer names in
goroutine names.

Change-Id: I5f55253470f7cc7e556f8e8b87f746394e41675f
2020-10-29 15:10:07 +00:00
Egon Elbre
53b396b901 satellite/metainfo/metabase: move Encryption to Begin
Change-Id: Ic1fb935ce7668e901f6cbe14a67c071a1f16a9ad
2020-10-29 16:04:21 +02:00
Egon Elbre
caefde6b32 private/{dbutil,tagsql}: pass ctx to database opening
Database opening usually dial and hence we should pass ctx to them.

Change-Id: Iaa2875981570d83e65be3710f841cf30349f807b
2020-10-29 10:51:29 +00:00
Egon Elbre
e3985799a1 storage/{cockroachkv,postgreskv}: add ctx to opening
Database opening usually dial and hence we should pass ctx to them.

Change-Id: Iecf41241aaa94d54506cbc80b0e53449848d8819
2020-10-29 10:49:08 +00:00
Michal Niewrzal
809eb14ac2 satellite/metainfo/metabase: move metainfo PoC into storj repo
Change-Id: I39356d8bc7305b4a8ea0c1fb5603010ad72a68b9
2020-10-29 09:44:55 +01:00
Egon Elbre
9b2e00a38b satellite: pass ctx into satellitedb.Open
Opening a database requires ctx, this is first step to passing ctx
to the appropriate level.

Change-Id: Ic303e69f868ef3449ae36377937a29670cf635e2
2020-10-29 06:38:37 +00:00
littleskunk
ed1f6d7973
satellite/config: move repair override from config to default (#3958)
Co-authored-by: Igor <38665104+ihaid@users.noreply.github.com>
2020-10-28 17:24:39 +02:00
Michal Niewrzal
cb1fea87f8 satellite/metainfo: mark unused methods as 'not implemented'
Some of metainfo endpoint methods are not used but we still have
implementation there. This change removes unused code and returns
unimplemented error for those methods.

Change-Id: I74e75e0caff76a4f5d119ee989b687b4e9d6e6f9
2020-10-28 12:42:47 +00:00
Michal Niewrzal
1adb497a71 satellite/metainfo: remove unused code
This change removed unused 'createRequests' struct. As far I remember it
was used to help validating old metainfo beginObject/commitObject flow.

Change-Id: I0f139b9934196d73f26eafa347ba5605722f3a55
2020-10-28 12:40:14 +01:00
Egon Elbre
76f4619a9c {satellite,storagenode}/gracefulexit: ensure client is closed
Change-Id: I576a955a5578caf7fcbee832beca28cef2b0c83e
2020-10-27 23:27:07 +02:00
Kaloyan Raev
92a2be2abd satellite/metainfo: get away from using pb.Pointer in Metainfo Loop
As part of the Metainfo Refactoring, we need to make the Metainfo Loop
working with both the current PointerDB and the new Metabase. Thus, the
Metainfo Loop should pass to the Observer interface more specific Object
and Segment types instead of pb.Pointer.

After this change, there are still a couple of use cases that require
access to the pb.Pointer (hence we have it as a field in the
metainfo.Segment type):
1. Expired Deletion Service
2. Repair Service

It would require additional refactoring in these two services before we
are able to clean this.

Change-Id: Ib3eb6b7507ed89d5ba745ffbb6b37524ef10ed9f
2020-10-27 13:06:47 +00:00
Cameron Ayer
bb7be23115 satellite/{audit,overlay,satellitedb}: enable reporting offline audits
- Remove flag for switching off offline audit reporting.
- Change the overlay method used from UpdateUptime to BatchUpdateStats, as this
is where the new online scoring is done.
- Add a new overlay.AuditOutcome type: AuditOffline. Since we now use the same
method to record offline audits as success, failure, and unknown, we need to
distinguish offline audits from the rest.

Change-Id: Iadcfe10cf13466fa1a1c2dc542db8994a6423355
2020-10-27 10:44:46 +00:00
Egon Elbre
9adde49e1a satellite/gracefulexit: ensure test doesn't timeout on failure
Change-Id: Id004f8a075592ffc19b12a9d666058b60cb7724d
2020-10-26 21:16:48 +02:00
Jessica Grebenschikov
f5880f6833 satellite/orders: rollout phase3 of SettlementWithWindow endpoint
Change-Id: Id19fae4f444c83157ce58c933a18be1898430ad0
2020-10-26 14:56:28 +00:00
Ethan
9a29ec5b3e Add index to graceful_exit_transfer_queue table
This fixes a slow query that was taking up to 4 seconds in production

SELECT node_id, path, piece_num, root_piece_id, durability_ratio, queued_at, requested_at, last_failed_at, last_failed_code, failed_count, finished_at, order_limit_send_count
	FROM graceful_exit_transfer_queue
	WHERE node_id = '[redacted]'
	AND finished_at is NULL
	AND last_failed_at is NULL
	ORDER BY durability_ratio asc, queued_at asc LIMIT 300 OFFSET 0;

Change-Id: Ib89743ca35f1d8d0a1456b20fa08c683ebdc1549
2020-10-26 14:47:48 +00:00
Ivan Fraixedes
4b61ca638b
satellite/console/consoleweb/consoleapi: Fix & add test DeleteAccount
Fix the DeleteAccount handler to return 501 HTTP status code because
it's what corresponds for a "Not Implemented" status.

Add a black box test for the DeleteAccount to ensure that always return
an error response because, at this time, we don't allow to delete
accounts through the API.

This test was not added to the corresponding commit
https://review.dev.storj.io/c/storj/storj/+/2712 due to the rush to
fix it.

Change-Id: Ibcf09e2ec52f182a8a580d606c457328d94c8b60
2020-10-23 09:14:50 +02:00
paul cannon
76d4977b6a storagenode/gracefulexit: logic moved from worker to service
Change-Id: I8b12606a96b712050bf40d587664fb1b2c578fbc
2020-10-22 23:19:30 +00:00
Ivan Fraixedes
9abdcc05e5 satellite/console/consoleweb/consoleapi: report err to monkit
Report the "Not Implemented" error response returned by DeleteAccount
API handler to monkit.

Change-Id: I17e319639c458cbe803b65b5a34111b8f74daece
2020-10-22 17:07:13 +00:00
Ivan Fraixedes
46b12c96bd satellite/console/consoleweb/consoleql: Fix typo
Fix a typo in the GraphQL mutation testing function.

Change-Id: I1c474795bfbaa3151b04cb768dfc506e654557ab
2020-10-22 13:30:20 +00:00
Kaloyan Raev
1f386db566
cmd/satellite: remove metainfo commands (#3955) 2020-10-22 13:33:09 +03:00
Kaloyan Raev
1aeb14e65e satellite/audit: do not delete expired segments
A year ago we made the audit service deleting expired segments.
Meanwhile, we introduced an expired deletetion sub-service in the
metainfo service which sole purpose is deleting expired segments.

Therefore, now we are removing this responsibility from the audit
service. It will continue to avoid reporting failures on expired
segments, but it would not delete them anymore.

We do this to cleanup responsibilities in advance of the metainfo
refactoring.

Change-Id: Id7aab2126f9289dbb5b0bdf7331ba7a3328730e4
2020-10-22 08:24:16 +00:00
Jessica Grebenschikov
89bdb20a62 storagenodedb/orders: select unsent satellite with expiration
In production we are seeing ~115 storage nodes (out of ~6,500) are not using the new SettlementWithWindow endpoint (but they are upgraded to > v1.12).

We analyzed data being reported by monkit for the nodes who were above version 1.11 but were not successfully submitting orders to the new endpoint.
The nodes fell into a few categories:
1. Always fail to list orders from the db; never get to try sending orders from the filestore
2. Successfully list/send orders from the db; never get to calling satellite endpoint for submitting filestore orders
3. Successfully list/send orders from the db; successfully list filestore orders, but satellite endpoint fails (with "unauthenticated" drpc error)

The code change here add the following to address these issues:
- modify the query for ordersDB.listUnsentBySatellite so that we no longer select expired orders from the unsent_orders table
- always process any orders that are in the ordersDB and also any orders stored in the filestore
- add monkit monitoring to filestore.ListUnsentBySatellite so that we can see the failures/successes

Change-Id: I0b473e5d75252e7ab5fa6b5c204ed260ab5094ec
2020-10-21 15:02:23 +00:00
paul cannon
360ab17869 satellite/audit: use LastIPAndPort preferentially
This preserves the last_ip_and_port field from node lookups through
CreateAuditOrderLimits() and CreateAuditOrderLimit(), so that later
calls to (*Verifier).GetShare() can try to use that IP and port. If a
connection to the given IP and port cannot be made, or the connection
cannot be verified and secured with the target node identity, an
attempt is made to connect to the original node address instead.

A similar change is not necessary to the other Create*OrderLimits
functions, because they already replace node addresses with the cached
IP and port as appropriate. We might want to consider making a similar
change to CreateGetRepairOrderLimits(), though.

The audit situation is unique because the ramifications are especially
powerful when we get the address wrong. Failing a single audit can have
a heavy cost to a storage node. We need to make extra effort in order
to avoid imposing that cost unfairly.

Situation 1: If an audit fails because the repair worker failed to make
a DNS query (which might well be the fault on the satellite side), and
we have last_ip_and_port information available for the target node, it
would be unfair not to try connecting to that last_ip_and_port address.

Situation 2: If a node has changed addresses recently and the operator
correctly changed its DNS entry, but we don't bother querying DNS, it
would be unfair to penalize the node for our failure to connect to it.

So the audit worker must try both last_ip_and_port _and_ the node
address as supplied by the SNO.

We elect here to try last_ip_and_port first, on the grounds that (a) it
is expected to work in the large majority of cases, and (b) there
should not be any security concerns with connecting to an out-or-date
address, and (c) avoiding DNS queries on the satellite side helps
alleviate satellite operational load.

Change-Id: I9bf6c6c79866d879adecac6144a6c346f4f61200
2020-10-21 13:34:40 +00:00
Ivan Fraixedes
979ee762ba
satellite/console/consoleweb: Fix typo in method name
Fix a typo in the graphQL handler method name.

Change-Id: I038c7783073f7bed95353f56a8a24520c724a5b6
2020-10-21 11:58:37 +02:00
Stefan Benten
334ae5b164 satellite/admin: add apikey endpoints
This change allows the creation and deletion of api keys via the admin API.
It adds two methods for deletion, one via the name and projectID and the
second one via the serialized apikey directly.

Change-Id: Ida8aa729e716db58c671a901e5f7e39253e89a0d
2020-10-20 11:26:56 +00:00
Moby von Briesen
7c3afe164b satellite/overlay: uncomment dq for offline and disable with feature flag
Change-Id: Ib39e2be32e880b822a94eddfb81af99a38843a27
2020-10-16 12:55:16 +00:00
Jessica Grebenschikov
205c39d404 satellite/orders: upgrade to phase 2 rollout ordersWithWindow
We are moving an error into rejectErr since its preventing storage nodes from being able to settle other orders.

Change-Id: I3ac97c340e491b127f5e0024c5e8bd9f4df8d5c3
2020-10-15 21:20:19 +00:00
Yaroslav Vorobiov
139a7ee959 private/migrate: add ablity to create dbs during migration
Use tagsql.DB pointer as step database, to propagate changes
back and forth between actual database and migration.
Adds CreateDB operation to the migration step to be able to
create new dbs before executing migration action.
Adjusts storagenode database migration to use inner tagsql.DB
pointer of each database as step.DB.
Adjusts satellite dabase migration, adds proxy migrationDB field
to satellite db that wraps itself as tagsql.DB, pointer of which
is used as step.DB.

Change-Id: Ifed4de5b01a356cf7b37db64d2eaeb7b61982c5c
2020-10-15 15:28:04 +03:00
Stefan Benten
0b43b93259 satellite/satellitedb: make limits per default NULL
This change completes the column migration of
5f6fccc6e8 and
2f648fd981.
It resets every users project limits who are below or equal to our
current production defaults.

Change-Id: Ie041d08bb67b62844f6023190fc00bc2dad5b1cb
2020-10-14 20:28:16 +00:00
VitaliiShpital
59d85aab5b web/satellite: take project amount limit from db instead of config
WHAT:
Now project amount limit is taken from users db instead of config. But if db value is 0 then default config value will be used instead.

WHY:
this will allow us to change user's project limit by changing db value.

Change-Id: I9edcd0bf9eaae5fe40e90a44cac82d9ce8519274
2020-10-14 14:17:45 +00:00
Kaloyan Raev
830817ec0d cmd/storj-sim: run gateway without --access flag
This makes it possible to remove of this obsolete flag from the
multi-tenant gateway.

As a consequence, displaying the GATEWAY_0_ACCESS env var will always
require a running storj-sim. Until now, it was required only the first
time. Then the value was stored in the 'access' config. But this is now
not possible anymore.

The changes in StripeMock are required to fix failures in integration
tests. StripeMock is in-memory and its data does not survive restarts of
storj-sim. The second and following starts of storj-sim had invalid
state of StripeMock, which failed requests that were required to
populate the GATEWAY_0_ACCESS env var. The changes in StripeMock makes
it repopulate the Stripe customers from the database.

Change-Id: I981a208172b76577f12ecdaae485f5ae4ea269bc
2020-10-13 14:45:04 +00:00
Egon Elbre
2268cc1df3 all: fix linter complaints
Change-Id: Ia01404dbb6bdd19a146fa10ff7302e08f87a8c95
2020-10-13 15:59:01 +03:00
Egon Elbre
0bdb952269 all: use keyed special comment
Change-Id: I57f6af053382c638026b64c5ff77b169bd3c6c8b
2020-10-13 15:13:41 +03:00
Stefan Benten
1d3b728766 satellite/{console/payments/satellitedb}: add validation for deletion of account and project
The same was that our Admin API handles project and account deletions currently, we would like
to have the same checks on the user-facing API. This PR adds the same checks to the console service.
General more applicable checks have been moved directly into the payments service.

In addition it adds the BucketsDB to the console DB, to have easier access and avoiding import cycles with
the metainfo package.

A small cleanup around our unnecessary monkit imports made it in as well.

Change-Id: I8769b01c2271c1687fbd2269a738a41764216e51
2020-10-13 07:55:26 +00:00
Jeff Wendling
4cbd4d52a9 satellite/orders: only hold the orders semaphore during database calls
holding it during node i/o means slow nodes can hold up order
processing for everyone else. this dramatically increases
the amount of tiem spent handling orders.

Change-Id: Iec999b7ed0817c921a0fd039097a75bdd3c70ea2
2020-10-10 15:40:50 -04:00
Jeff Wendling
0f0faf0a9f satellite/orders: do a better job limiting concurrent requests
Doing it at the ProcessOrders level was insufficient: the endpoints
make multiple database calls. It was a misguided attempt to only
have one spot enter the semaphore. By putting it in the endpoint
we can not only be sure that the concurrency is correctly limited
but it can be configurable easily.

Change-Id: I937149dd077adf9eb87fce52a1a17dc0afe96f64
2020-10-09 16:27:15 -04:00
Jeff Wendling
7c303208ff satellite/satellitedb: emergency temporary order processing semaphore
we have thundering herds of order submissions that take all of the
database connections causing temporary periodic outages. limit
the amount of concurrent order processing to 2.

Change-Id: If3f86cdbd21085a4414c2ff17d9ef6d8839a6c2b
2020-10-08 19:16:47 +00:00
Stefan Benten
b3cf12f567 satellite/console: Add more validation for console requests
Adds membership checks for the following calls:
- GetProject

Add ownership checks for the following calls:
- DeleteProject

It also disables the API endpoint to delete a project.

Furthermore it adds tests for the console service.

Change-Id: I1ffc8dcb44746a74ad06a7dbd064a29c57c25272
2020-10-07 15:33:28 +00:00
Kaloyan Raev
e7f2ec7ddf satellite/audit: fix sanity check for verify-piece-hashes command
The VerifyPieceHashes method has a sanity check for the number pieces to
be removed from the pointer after the audit for verifying the piece
hashes.

This sanity check failed when we executed the command on the production
satellites because the Verify command removes Fails and PendingAudits
nodes from the audit report if piece_hashes_verified = false.

A new temporary UsedToVerifyPieceHashes flag is added to
audits.Verifier. It is set to true only by the verify-piece-hashes
command. If the flag is true then the Verify method will always include
Fails and PendingAudits nodes in the report.

Test case is added to cover this use case.

Change-Id: I2c7cb6b12029d52b2fc565365eee0826c3de6ee8
2020-10-07 17:17:48 +03:00
Kaloyan Raev
4280142b24 satellite/console: remove unnecessary Error.Wrap
Change-Id: If851ccce7932cbf72c2fff3b51f4f9f2ea07c124
2020-10-07 09:22:41 +00:00
Stefan Benten
14a2050b8d pkg/auth: move package to consoleauth
To avoid further name collisions, the very broad named package gets moved into
the consoleauth package where its also mainly being used.

Change-Id: Ie563c9700adbf0553baca2b7b8ba4a1d9c29d144
2020-10-06 14:15:07 +02:00
Stefan Benten
44bd65795b satellite/console: ensure only project members can remove other project members
Change-Id: I815eb85f37631aaa65b5dc4cafa6851f241ca0f0
2020-10-06 11:03:12 +00:00
Stefan Benten
9deea2ffe2 satellite/console: disable account deletion via API
Change-Id: Ia8e43284c90fb2b833eb601e2c8f701cb5a4d9c0
2020-10-06 13:01:46 +02:00
Stefan Benten
0aaad88a44 satellite/{admin, console}: add test for projectLimit increase and update README
This change adds the capabilities to adjust the users project limit via the Admin API.
Adds a test for the new added function of the API and updates the existing tests.
It renames the json field on the user struct to be more consistent.

Change-Id: I9018acd80dae0af68d1d50526f20987132c654f3
2020-10-05 11:54:37 +00:00
Cameron Ayer
b39a99bae6 satellite/{overlay,satellitedb}: always show node's real online score
Previously if a node did not have audit history data for each of the
windows over the tracking period, we would give them the benefit of
the doubt and set their score to 1. This was to prevent nodes from
being suspended right out the gate. We need a minimum amount of data
to evaluate them.

However, a node who is actually failing at being online will have no
idea until they have received enough audits and we suspend them.

Instead, we will always use their real score, but use a flag to determine
whether they are eligible for suspension/dq.

Change-Id: I382218f12e8770f95d4bcddcf101ef348940cadf
2020-10-02 12:28:11 -04:00
Yingrong Zhao
c085a17a52 bump common and uplink to latest
Change-Id: I717f0214dd9973acd51b7732c5d64587f610c805
2020-10-01 15:38:58 +00:00
Moby von Briesen
000b1e6011 satellite/admin/project_test.go: Update TestCheckUsageLastMonthUnappliedInvoice
Test is failing because the mock date used in it is September 2020.
Update to 2030.

Change-Id: I4733487d0589dbeb0d2f3dd2abd88f767d59d0ca
2020-09-30 00:10:02 -04:00
Cameron Ayer
c2525ba2b5 satellite/{repair,satellitedb}: clean up healthy segments from repair queue at end of checker iteration
Repair workers prioritize the most unhealthy segments. This has the consequence that when we
finally begin to reach the end of the queue, a good portion of the remaining segments are
healthy again as their nodes have come back online. This makes it appear that there are more
injured segments than there actually are.

solution:
Any time the checker observes an injured segment it inserts it into the repair queue or
updates it if it already exists. Therefore, we can determine which segments are no longer
injured if they were not inserted or updated by the last checker iteration. To do this we
add a new column to the injured segments table, updated_at, which is set to the current time
when a segment is inserted or updated. At the end of the checker iteration, we can delete any
items where updated_at < checker start.

Change-Id: I76a98487a4a845fab2fbc677638a732a95057a94
2020-09-29 20:38:22 +00:00
Egon Elbre
c23a8e3b81 go.mod: update pgx to v4.9.0
Fix query to use TextArray instead of VarcharArray.
Fix queries to use the correct type.

Change-Id: Ibb7e55adba277d05778118d81ca697470e72c374
2020-09-29 19:03:08 +00:00
Brandon Iglesias
8783c2024a
satellite/rewards: add partner videocoin (#3946) 2020-09-29 14:30:46 +02:00
Kaloyan Raev
b409b53f7f cmd/satellite: command for verifying piece hashes
Jira: https://storjlabs.atlassian.net/browse/PG-69

There are a number of segments with piece_hashes_verified = false in
their metadata) on US-Central-1, Europe-West-1, and Asia-East-1
satellites. Most probably, this happened due to a bug we had in the
past. We want to verify them before executing the main migration to
metabase. This would simplify the main migration to metabase with one
less issue to think about.

Change-Id: I8831af1a254c560d45bb87d7104e49abd8242236
2020-09-29 10:58:24 +00:00
Egon Elbre
d508c4c985 scripts: remove duplicate check config lock
We have configlock_test for checking changes so the separate script and
makefile target are not needed.

Change-Id: I2bbc1c21ad849c9b7ec8bba43c0e11e94e04f6a6
2020-09-29 10:03:34 +00:00
Egon Elbre
2d27bc8787 satellite/satellitedb: separate cockroach for migration tests
Currently Cockroach migration test is the most heavy with regards to
schema changes. This causes other tests to time out. This adds an
alternate cockroach instance that is used for migration tests.

Change-Id: I01fe9313527ff002f0bb0914dd52c3645b8eaf6d
2020-09-29 09:31:33 +00:00
Stefan Benten
79eb682f9c satellite/console: allow coupons to be a valid payment option
Currently a user is only able to create a project if either
a STORJ deposit or CC was added to his account. With this change, an existing
coupon is also valid to let the user proceed.

Change-Id: I7be8d2d9ec58a15c50755b3fe33af04d2fd64ea2
2020-09-28 21:24:04 +00:00
Jessica Grebenschikov
4a2c66fa06 satellite/accounting: add cache for getting project storage and bw limits
This PR adds the following items:
1) an in-memory read-only cache thats stores project limit info for projectIDs

This cache is stored in-memory since this is expected to be a small amount of data. In this implementation we are only storing in the cache projects that have been accessed. Currently for the largest Satellite (eu-west) there is about 4500 total projects. So storing the storage limit (int64) and the bandwidth limit (int64), this would end up being about 200kb (including the 32 byte project ID) if all 4500 projectIDs were in the cache. So this all fits in memory for the time being. At some point it may not as usage grows, but that seems years out.

The cache is a read only cache. When requests come in to upload/download a file, we will read from the cache what the current limits are for that project. If the cache does not contain the projectID, it will get the info from the database (satellitedb project table), then add it to the cache.

The only time the values in the cache are modified is when either a) the project ID is not in the cache, or b) the item in the cache has expired (default 10mins), then the data gets refreshed out of the database. This occurs by default every 10 mins. This means that if we update the usage limits in the database, that change might not show up in the cache for 10 mins which mean it will not be reflected to limit end users uploading/downloading files for that time period..

Change-Id: I3fd7056cf963676009834fcbcf9c4a0922ca4a8f
2020-09-25 16:28:49 +00:00
Stefan Benten
9d0d0ad728 satellite/console: enable multiple projects all users
Change-Id: I42cc9f48cac387e1a67d21c1dd394f28cc5ff399
2020-09-23 16:18:28 +00:00
Stefan Benten
38108828ac satellite/satellitedb: enable multiple projects existing users
Change-Id: I2ef77182d5464d72574698c8abfbbfdbda3f5a9e
2020-09-23 18:17:38 +02:00
Stefan Benten
5f6fccc6e8 satellite/satellitedb: makes limits nullable change backwards compatible
Our current endpoints bail on us, if the column data is null. Thus we need
to take the intermediate step and set the default to a fixed value and
reset those with the following release.

It sets the default column value to our current config values of 50GB
for storage and bandwidth and 100 buckets, while still enabling the field to be nullable.
All 0 values are migrated to be the default as well to ensure they can
keep using their projects, as with the original change, 0 actually means 0.

Change-Id: I797be80ce2d2105091599dc1b3fc76f74336b66b
2020-09-23 17:54:42 +02:00
Stefan Benten
2f648fd981 satellite: make limits be nullable
Currently we have no way to actually set one
of the following limits to 0 (meaning not usable):

- maxBuckets
- usageLimit
- bandwidthLimit

With having the field nullable,
NULL corresponds to the global default,
0 now actually 0 and
a set value determines a custom limit.

Change-Id: I92bb77529dcbd0881ae8368921be9d246eb0919e
2020-09-21 19:34:19 +00:00
Kaloyan Raev
34613e4588 cmd/satellite: command for fixing old-style objects
Jira: https://storjlabs.atlassian.net/browse/PG-67

There are a number of old-style objects (without the number of segments
in their metadata) on US-Central-1, Europe-West-1, and Asia-East-1
satellites. We want to migrate their metadata to contain the number of
segments before executing the main migration to metabase. This would
simplify the main migration to metabase with one less issue to think
about.

Change-Id: I42497ae0375b5eb972aab08c700048b9a93bb18f
2020-09-21 14:46:19 +00:00
JT Olio
f46161cf20 consoleweb: log index template failures
Change-Id: I286ded309fed6198f1c450a2a31df36b6a015551
2020-09-16 16:26:14 +00:00
Qweder93
8182fdad0b storagenode: heldamount renamed to payouts, renamed some methods and structs to more meaningful names. grouped estimated payout with pathouts
satellite: heldamount renamed to SNOpayouts.

Change-Id: I244b4d2454e0621f4b8e22d3c0d3e602c0bbcb02
2020-09-16 14:57:35 +00:00
Michal Niewrzal
d199daa907 satellite/metainfo: replace local constants with metabase values
Change-Id: Ie855c10dd259464658952186251b7d3210eb49ce
2020-09-16 13:25:14 +02:00
VitaliiShpital
7d5e0259f6 satellite/projects: initial update project name functionality implemented
WHAT:
added functionality for user to update project name. Logic only, without actual GUI updates.

WHY:
better user experience

Change-Id: I1e38e33ba827b0bdf2c89e29de24e4e87edb474a
2020-09-15 12:21:56 +03:00
Cameron Ayer
e7c34a053d satellite/satellitedb: add column and index "updated_at" to injuredsegments
Change-Id: I59e9bb2077885f09e17795375fe98ed31bd83d54
2020-09-14 12:53:04 -04:00
Jeff Wendling
1fecaed7df satellite/orders: don't version check old endpoint
nodes are submitting using both the legacy and windowed endpoints
and thus having their legacy submissions rejected.

it is legal to use both the legacy and windowed endpoints
in phase1 since they use the same backend. the legacy endpoint
is disabled in phase2 and phase3.

therefore, if we wait an order expiration period (2 days) after
we determine enough nodes have started using the windowed
endpoint, we can be sure that any orders they did have to
submit with the legacy endpoint will have expired.

Change-Id: I4418a881bf8bb9377efaef4c651e6103a5dc6ed0
2020-09-09 10:23:48 -04:00
stefanbenten
2cf12b9409 satellite/admin: use correct period end time to get results
Change-Id: Ib0e76f96e61b39c98f0f5f683b39d622d235c4d8
2020-09-09 09:01:42 +00:00
Michal Niewrzal
27a9d14e2a satellite/repair: use metabase.SegmentKey type in repair package
Another change which is a part of refactoring to replace path parameter
(string/[]byte) with key paramter (metabase.SegmentKey)

Change-Id: I617878442442e5d59bbe5c995f913c3c93c16928
2020-09-08 19:35:20 +00:00
Michal Niewrzal
c753d17e8f satellite/metainfo/objectdeletion: replace
objectdeletion.ObjectIdentifier with metabase.ObjectLocation

Another change to use metabase.ObjectLocation across satellite codebase
to avoid duplication and provide better type safety.

Change-Id: I82cb52b94a9107ed3144255a6ef4ad9f3fc1ca63
2020-09-08 21:09:34 +02:00
VitaliiShpital
c4d6f472fc web/satellite: notification bar for reaching projects count limit
WHAT:
notification bar added to project dashboard page. It is shown when projects count limit is reached.
Create project button is removed after creating last available project

WHY:
inform user that their projects count limit was reached

Change-Id: If0d67148003be40cc9eb4d8b25cc17f8204008d4
2020-09-08 15:48:27 +00:00
JT Olio
249f98f4c1 audit logging: log important user activity
Change-Id: Ic423dab4b6c513fa44a48cb6fe2903c70fe916fb
2020-09-07 15:26:34 +00:00
Stefan Benten
2111d90684 satellite/admin: add get handler for projects
Change-Id: Ibfb761b1c907b83451570c52eea03ecfa75290ca
2020-09-06 01:40:11 +02:00
Jennifer Johnson
4e2413a99d satellite/satellitedb: uses vetted_at field to select for reputable nodes
Additionally, this PR changes NewNodeFraction devDefault and testplanet config from 0.05 to 1.
This is because many tests relied on selecting nodes that were reputable based on audit and uptime
counts of 0, in effect, selecting new nodes as reputable ones.
However, since reputation is now indicated by a vetted_at db field that is explicitly set
rather than implied by audit and uptime counts, it would be more complicated to try to
update all of the nodes' reputations before selecting nodes for tests.
Now we just allow all test nodes to be new if needed.

Change-Id: Ib9531be77408662315b948fd029cee925ed2ca1d
2020-09-04 16:45:32 +00:00
Michal Niewrzal
8649a00557 satellite/gracefulexit: replace Path []byte to `Key
metabaseSegmentKey` TransferQueueItem

We are unifying which name (and type) we are using for value we are
using to point to segment. We want to use `key` instead of `path`.
Dedicated type `metabase.SegmentKey` was created for this purposes also.
This change is doing refactoring around gracefulexit.

Change-Id: I90d51ff087b206179e61d5f1bc95f4709d76f917
2020-09-04 11:09:48 +00:00
Michal Niewrzal
9202295348 satellite/metainfo: replace ScopedPath with metabase.SegmentLocation
Change-Id: I7e89c9e8eaeae58be828a32ad47ed3028501f4c7
2020-09-04 10:06:52 +00:00
Yingrong Zhao
2cc3c19d55 satellite/piecedeletion: add test for disproportionate requests/nodes
ratio

Change-Id: I7902e4ffeca39c5a6a5b9170ce6f010d21e7e19e
2020-09-03 21:00:53 +00:00
Michal Niewrzal
aa47e70f03 satellite/metainfo: use metabase.SegmentKey with metainfo.Service
Instead of using string or []byte we will be using dedicated type
SegmentKey.

Change-Id: I6ca8039f0741f6f9837c69a6d070228ed10f2220
2020-09-03 15:11:32 +00:00
Egon Elbre
dc48197bd8 satellite/orders: add bucket id to order limit
Change-Id: I9019ec77d692e62ac17b67a1da71dc3535cde50c
2020-09-03 10:50:11 +03:00
Yingrong Zhao
af773ec8a6 cmd/uplink: use DeleteBucketWithObjects for bucket force deletion
This PR updates `uplink rb --force` command to use the new libuplink API
`DeleteBucketWithObjects`.
It also updates `DeleteBucket` endpoint to return a specific error
message when a given bucket has concurrent writes while being deleted.

Change-Id: Ic9593d55b0c27b26cd8966dd1bc8cd1e02a6666e
2020-09-02 16:39:20 +00:00
Qweder93
36d752e92d storagenode/reputation: offline_under_review_at added
Change-Id: Ia7ec79b2d6f20fe29de0c36223f9485380d2845c
2020-09-02 18:48:28 +03:00
Qweder93
7d9897b7af storagenode/nodestats: online_score added
Change-Id: I84b50a6cace306e5f10d53a2073fe8810d4d2960
2020-09-02 17:45:01 +03:00
Yingrong Zhao
7c7334e9d5 satellite/metainfo/piecedeletion: fix deadlock when waiting for threshold to reach
This PR fixes a deadlock that can happen when the number of piece
deletion requests is different from the distinct node count from those
requests. The success threshold should be based on the number of nodes
instead of the amount of requests

Change-Id: I83073a22eb1e111be1e27641cebcefecdc16afcb
2020-09-02 07:07:49 +00:00
Egon Elbre
61b17f1214 satellite/orders: add encryption keys flag to Service
Change-Id: Ie96e75bc96241b799d04654ef5e05b82e6a899bb
2020-09-02 05:02:14 +00:00
Natalie Villasana
95ff29cce1 satellite/metainfo: reduce lookupLimit default to 2500
Change-Id: I6569c6d1f145b127a9e8e1a65e4344dd62c989bb
2020-09-01 12:04:48 -04:00
JT Olio
b872fe52a1 satellite/repair: switch to piecestore.UploadReader
Change-Id: Ia99ad2cf5422e6ba1d98b32946740f9cadba7b6d
2020-09-01 09:26:54 -06:00
Michal Niewrzal
084e280a6d satellite/admin: fix time issues in tests
With day -1 test was creating tallies for two different months.

Change-Id: I10201f4e787e73bae40bf7120856fb7cfdcd66f7
2020-09-01 16:03:52 +02:00
Michal Niewrzal
0604a672c1 satellite/metainfo: use metabase in loop
Change-Id: I1bb0c6fe0a762895fde950690b06f7dd9d77e178
2020-09-01 10:06:16 +00:00
Isaac Hess
2d54447088 satellite/metainfo: Fix GetObjectIPs loop and test
This change forces the test of GetObjectIPs to use multiple remote
segments (earlier versions of the test were accidentally using inline
segments). This change also revealed a small bug in the for loop code,
which is fixed.

Change-Id: Ic486b079d221952ba13553acf0ca41a8873f3f21
2020-08-31 15:56:12 -06:00
Moby von Briesen
5d21e85529 satellite/audit/queue: Separate audit queue into two separate structs.
* The audit worker wants to get items from the queue and process them.
* The audit chore wants to create new queues and swap them in when the
old queue has been processed.

This change adds a "Queues" struct which handles the concurrency
issues around the worker fetching a queue and the chore swapping a new
queue in. It simplifies the logic of the "Queue" struct to its bare
bones, so that it behaves like a normal queue with no need to understand
the details of swapping and worker/chore interactions.

Change-Id: Ic3689ede97a528e7590e98338cedddfa51794e1b
2020-08-31 20:51:25 +00:00
Isaac Hess
ba1a113e2e satellite/metainfo: Add test for GetObjectIPs
This PR adds the integration test for GetObjectIPs, testing all repos.

Change-Id: Id4d4f19c427a2e320b2a82efa68150fa7f5f86fb
2020-08-31 10:27:56 -06:00
Isaac Hess
351aa70eb7 satellite/metainfo: Implement GetObjectIPs
Change-Id: Ibabbe7c555b790498d28a6ac4c95fcf2f7376978
2020-08-31 10:27:56 -06:00
Moby von Briesen
2d01dd9732 satellite/satellitedb: Add online_score column to nodes table
Add online score used for the new audit history offline tracking system
to the nodes table. This allows us easy access to the node's online
score for the storagenode dashboard as well as for data analysis.

Change-Id: Ie99be1192e5236862a5b3dbed2e5ef03b9169410
2020-08-31 15:07:07 +00:00
Bill Thorp
328004c0ef satellite/accounting: fix build - time rounding
We were seeing error on the last day of the month with TestProjectAllocatedBandwidthRetainTwo.
This is due to AddDate normalizes its result in the same way that Date does, so, for example,
adding one month to October 31 yields December 1, the normalized form for November 31."

I also fixed a minor UTC issue with this test as well.

Change-Id: I0157873e7befa57810e5f264a922b188890fa46a
2020-08-31 09:37:13 -04:00
Egon Elbre
c86c732fc0 satellite: simplify tests
satellite.DB.Console().Projects().GetAll database query
can be replaced with planet.Uplinks[0].Projects[0].ID

Change-Id: I73b82b91afb2dde7b690917345b798f9d81f6831
2020-08-28 22:28:04 +00:00
Moby von Briesen
60a95d0dc9 satellite/{satellitedb,overlay}: Enable offline suspension and review period
When a node's audit history "online score" passes below a configured
threshold, the node goes into "offline suspension" mode and begins a
review period, where the operator is given an opportunity to bring their
node back online.
After the review period passes, offline suspension is turned off for the
node.

In the future, if a node still has a bad online score at the end of the
review period, it will be disqualified. This is disabled right now.
In the future, if a node is in offline suspension, it will be treated as
"unhealthy". Right now, there are no consequences for being in offline
suspension.

Minor changes:
* Moves AuditHistoryConfig out of UpdateStats/BatchUpdateStats args and
into UpdateRequest.
* Adds "now" argument to UpdateStats/BatchUpdateStats args for easy
testing.
* Changes formatting strings inside buildUpdateStatement to use specific
types.

Change-Id: I032b60298840fc16e6ef831da750f2d57619a397
2020-08-28 16:35:48 +00:00
Egon Elbre
9225fc5aef satellite/accounting: simplify ExceedsBandwidthUsage call
Change-Id: I5376da2329e44da8f060226d2a76432df0acdaa0
2020-08-28 18:10:02 +03:00
Egon Elbre
3ca405aa97 satellite/orders: use metabase types as arguments
Change-Id: I7ddaad207c20572a5ea762667531770a56fd54ef
2020-08-28 15:52:37 +03:00
Brandon Iglesias
3bfb0a5246
Adding Kesque, MSP360, Innovoedge, Taloflow, Restic partners (#3933)
* Adding Kesque, MSP360, Innovoedge, Taloflow, Restic partners

* ABC order
2020-08-27 14:29:54 -04:00
Egon Elbre
d7c6ca6013 satellite/metainfo/metabase: add package for metainfo database
Currently there is confusion between responsibilities of
metainfo.Endpoint, metainfo.Service, PointerDB.

By separating database "service" into a separate package and
its types allows to disentagle them.

This gives us responsibilities:

1. metainfo.Endpoint - translates requests and permissions
2. metainfo.Service - handles requests and coordinates with
   objectdeletion, piecedeletion, metabase
3. metabase.Service - communication with the database interface and invariants

Currently metabase will contain the types necessary to coordinate
information.

Change-Id: If8c992b4b9d9e70a56bbd8a378a5af6b1a2ec34e
2020-08-27 17:31:11 +00:00
stefanbenten
086a3d5348 satellite/{payments,admin}: add deletion of user creditcards on account deletion
Change-Id: I38bf7e3995846150268f7b88a70f75b0ac871b62
2020-08-27 10:18:19 +00:00
Bill Thorp
729079965f satellite/satellitedb : remove migation steps 69-102
Jenkins has been failing a lot lately due to test timeouts with CockroachDB.
TestMigrateCockroach previously took around 5 minutes, now it takes 2.

Why 103?  I couldn't get 100 to work due to an error w/ NOT NULL and PKs.

Change-Id: Iec95d4e25f9d6cd36920e7f43272c486a17fa879
2020-08-27 07:36:05 +00:00
Bill Thorp
dbb53151f0 private/testplanet: Decrease metainfo MaxBuckets test value to speed testing.
TestMaxOutBuckets is one of our slower tests (50-90s).
This change seems to make it 2-12s.

It reduces the number of buckets that need to be created.
It also removes unnecessary storage nodes.

Change-Id: I1012fc6e9258b2f7674b16da4e8b418741c93eea
2020-08-26 17:31:31 +00:00
Moby von Briesen
4f28bf0720 satellite/audit: Do not return errors from Verify or Reverify on segment modified, expired, or deleted
If a segment is deleted, is modified, or expires during an audit, this
is not problematic, so we should not return errors. Functionally,
nothing changes, but our metrics around audit success rate will be
improved after this change.

Change-Id: Ic11df056b2c73894b67a55894bd4d58c00470606
2020-08-26 13:24:00 +00:00
Qweder93
88ff8829a1 satellite/gracefulexit: RecvTimeout increased to 2h, so slow nodes stop receiving lot of fails and as a result DQ
Change-Id: Id4c8a394162ba368aeb573a927f825bf7250aa52
2020-08-24 18:59:24 +03:00
Yingrong Zhao
bd5213f68b satellite/metainfo: implement batch delete for DeleteBucket
This PR changes DeleteBucket to be able to delete all objects within a
bucket if `DeleteAll` is set in `BucketDeleteRequest`.
It also changes `DeleteBucket` API to treat `ErrBucketNotFound` as a
successful delete operation instead of returning an error back to the
client.

Change-Id: I3a22c16224c7894f2d0c2a40ba1ae8717fa1005f
2020-08-24 13:28:09 +00:00
Moby von Briesen
959cd5cd83 satellite/satellitedb: Update audit history from overlay.UpdateStats and overlay.BatchUpdateStats
Change-Id: Ib530b61895ca4a8b12ba022c408a416b237b56d7
2020-08-20 22:46:28 +00:00
Moby von Briesen
5f0477ebe9 satellite/{overlay,satellitedb}: Create database functionality for updating audit history
Add a function to the overlay cache called UpdateAuditHistory, which
allows us to add online or offline audits to a particular node's audit
history, and get that node's "online score" for the configured tracking
period.

The next step will be to use UpdateAuditHistory from inside
BatchUpdateStats/UpdateStats, so that audit history is actually updated
when nodes get audited, and we can suspend nodes based on their online
score.

Change-Id: I2289105e6961e68e829a987ff756b0e576fab120
2020-08-20 17:34:27 +00:00
Jeff Wendling
91698207cf storagenode: live tracking of order window usage
This change accomplishes multiple things:

1. Instead of having a max in flight time, which means
   we effectively have a minimum bandwidth for uploads
   and downloads, we keep track of what windows have
   active requests happening in them.

2. We don't double check when we save the order to see if it
   is too old: by then, it's too late. A malicious uplink
   could just submit orders outside of the grace window and
   receive all the data, but the node would just not commit
   it, so the uplink gets free traffic. Because the endpoints
   also check for the order being too old, this would be a
   very tight race that depends on knowledge of the node system
   clock, but best to not have the race exist. Instead, we piggy
   back off of the in flight tracking and do the check when
   we start to handle the order, and commit at the end.

3. Change the functions that send orders and list unsent
   orders to accept a time at which that operation is
   happening. This way, in tests, we can pretend we're
   listing or sending far into the future after the windows
   are available to send, rather than exposing test functions
   to modify internal state about the grace period to get
   the desired effect. This brings tests closer to actual
   usage in production.

4. Change the calculation for if an order is allowed to be
   enqueued due to the grace period to just look at the
   order creation time, rather than some computation involving
   the window it will be in. In this way, you can easily
   answer the question of "will this order be accepted?" by
   asking "is it older than X?" where X is the grace period.

5. Increases the frequency we check to send up orders to once
   every 5 minutes instead of once every hour because we already
   have hour-long buffering due to the windows. This decreases
   the maximum latency that an order will be reported back to
   the satellite by 55 minutes.

Change-Id: Ie08b90d139d45ee89b82347e191a2f8db1b88036
2020-08-19 19:42:33 +00:00
Yingrong Zhao
14ad7a4f1c satellite/metainfo: add limiter for objectdeletion and piecedeletion
services

This PR adds a limiter on the amount of concurrent objects deletion can be handled so
we don't run out of memory.

Change-Id: Id2ce368af6f86845fcdfd34cb2f5e460efe9b272
2020-08-19 16:08:29 +00:00
Moby von Briesen
708cb48aa6 storagenode/orders: implement orders filestore on storagenode
* Add all new orders to the orders filestore instead of the database.
* Submit orders from the filestore to the new satellite SettleWindow
endpoint.

The orders filestore will eventually replace the orders DB completely.
For now, we will still be checking the orders DB and submitting those
orders if they exist. In a later release, we will completely remove the
orders DB, but we need both the DB and filestore for the transitionary
period.

Change-Id: Iac8780fd5ab770296181bbd313e1d335f072d4dc
2020-08-19 15:00:35 +00:00
Egon Elbre
b4c8e219c7 satellite/orders: calculate order expiration inside signer
Change-Id: I07f79eeb1ab41b061a1f3146f684bd21291cffb0
2020-08-18 13:21:16 +03:00
Egon Elbre
189ab07846 satellite/orders: use Signer in CreateGetOrderLimits
Change-Id: Icb7ed4f1af1dabbbb68cb6f6e1f86d93a9b5faa3
2020-08-18 13:20:00 +03:00
Egon Elbre
cd5e99ea6b satellite/orders: Signer for simplifying signing logic
Create a separate struct for signing order limits.

Change-Id: I8f8f5245040efa8c03138512be9248d4834f3f36
2020-08-18 13:19:16 +03:00
Qweder93
01bb2bd17d satellite/audit: verifier checks if node made sucess GE before auditing
Change-Id: Ia6cde4e9fcf11020a5301d38065f7159f276eb80
2020-08-17 23:37:57 +03:00
Yingrong Zhao
0518b16370 satellite/piecedeletion: move node info retrieval into the service
This change will require less work for the user of peiecedeletion
service by moving overlay database call into the package.

Change-Id: I14a150ab71fe885780e7a7a74db006a779507ae5
2020-08-13 16:07:54 +00:00
stefanbenten
c7b86a3481 satellite/admin: add check project usage endpoint and fix some leftover http.Error handling
Change-Id: I1ae3e7cb723a553f9c5a3a752beab0a27b0293bc
2020-08-13 13:48:34 +00:00
Yingrong Zhao
b71da59f8a satellite/metainfo: fix client context cancelation
We should still ignore client cancelation.
Also remove logger dependency on `Report`.

Change-Id: I59987c6b1aaa93202b07e5ef6e207d5191542b52
2020-08-11 20:58:23 +00:00
Isaac Hess
a14887e20b satellite/metainfo: Add GetObjectIPs method
This adds the unimplemented GetObjectIPs method to metainfo endpoint so
we can import new common protobuf definitions.

Change-Id: I154f26baccb6bb3c66de3eb25611930545c9754b
2020-08-11 11:39:15 -06:00
Egon Elbre
94a09ce20b all: add missing dots
Change-Id: I93b86c9fb3398c5d3c9121b8859dad1c615fa23a
2020-08-11 17:50:01 +03:00
Ethan
ab1d0f097d satellite/storageusage: Group accounting rollups at_rest_total by day
When investigating a gap in storage usage data in the SN dashboard, I noticed that there were 2 entries in the accounting_rollups table on the date of the gap.
This change accounts for multiple entries in the accounting_rollups table for a given day.

Change-Id: Ibf2b5d0455117cb0417163e8fcfb7e509d594171
2020-08-10 15:03:15 +00:00
Kaloyan Raev
7552ff26ec satellite/db: drop project_invoice_stamps table
It's an obsolete table from earlier state of Stripe invoices
implementation. No code is currently using it. It is confirmed that this
table is currently empty across all satellites.

Change-Id: I12d2756578faf8418ea8f3b09088e885694b8925
2020-08-10 13:22:10 +00:00
Qweder93
4ee1b2d45a storagenode/console: added list of all audits per satellite to sno dashboard/satellites
Change-Id: I52e58748d6467f372d9a308347fc77e400d137e2
2020-08-10 12:55:07 +00:00
Michal Niewrzal
6e1ff78cdd satellite/metainfo: extend TestAttributionReport
Small extension to test case where another partner is upload/downloading
to/from the same bucket as partner which creates this bucket.

Change-Id: Ib674fe5f95f868b71341e30aba5e2440847738f4
2020-08-07 11:17:52 +00:00
Yingrong Zhao
2b2cb468bf satellite/metainfo: implement DeleteObjectPieces with objectdeletion package
Use new objectdeletion package for deleting pointers.

In the best case scenario, it will make on database call to fetch
information about the number of segments. And another request to delete
and fetch information about other segments.

This PR also changes our object deletion API to return no error when an
object is not found but instead consider such operation as success. This
behavior is asligned with S3 API and makes the code less complex.

Change-Id: I280c56e8b5d815a8c4dafe8227689467e899775a
2020-08-06 16:10:13 -04:00
Michal Niewrzal
ceb2eee494 satellite/metainfo: make overwrites impossible without delete permission
https://storjlabs.atlassian.net/browse/USR-1007

Change-Id: Ic7fa600a00a53b80a37acee4ac994be6ea4cb279
2020-08-06 18:01:44 +00:00
Michal Niewrzal
88dcc93f3c satellite/metainfo: use user PartnerID for bucket attribution
Change-Id: I20f1bd432333f9b37ca8fb457c349eff94ffb392
2020-08-06 13:14:07 +00:00
stefanbenten
d654ab5fa0 satellite/admin: change returned data to be json encoded
Change-Id: I239779fe54328fe854fcbdd71f7906875cd76635
2020-08-05 15:33:39 +02:00
Moby von Briesen
e02adfe5e9 satellite/overlay/config.go: Add AuditHistoryConfig to overlay
Adds AuditHistory{WindowSize, TrackingPeriod, GracePeriod,
OfflineThreshold}. These values will be used to track offline audits over
time, and to suspend/disqualify nodes for being offline for too long.

Change-Id: I05f7dbc3c034bdc53c4fbd7719c71a44f37ec6a5
2020-08-04 18:18:56 +00:00
Moby von Briesen
5dfe27f175 satellite/{repair,overlay}: Use overlay NodeSelectionCache for repair uploads
This change removes the overlay function FindStorageNodesForRepair,
which skips using the node selection cache and hits the database
directly. Otherwise, it is functionally identical to
FindStorageNodesForUpload, which checks the node selection cache first.

When selecting nodes for PUT_REPAIRs, we now call
FindStorageNodesForUpload instead of FindStorageNodesForRepair to reduce
database load.

Change-Id: If34e109695b2ed2b8fb6759115bf769a3459684e
2020-08-04 12:50:12 -04:00
Jeff Wendling
85a74b47e7 satellite/orders: 3-phase rollout
This adds a config flag orders.window-endpoint-rollout-phase
that can take on the values phase1, phase2 or phase3.

In phase1, the current orders endpoint continues to work as
usual, and the windowed orders endpoint uses the same backend
as the current one (but also does a bit extra).

In phase2, the current orders endpoint is disabled and the
windowed orders endpoint continues to use the same backend.

In phase3, the current orders endpoint is still disabled and
the windowed orders endpoint uses the new backend that requires
much less database traffic and state.

The intention is to deploy in phase1, roll out code to nodes
to have them use the windowed endpoint, switch to phase2, wait
a couple days for all existing orders to expire, then switch
to phase3.

Additionally, it fixes a bug where a node could submit a bunch
of orders and rack up charges for a bucket.

Change-Id: Ifdc10e09ae1645159cbec7ace687dcb2d594c76d
2020-08-03 17:01:42 +00:00
Rafael Gomes
935f44ddb7 satellite/metainfo: Add Delete Service config
Change-Id: I0a6e3ce1adfe1488eb23da9dda92877af1834599
2020-08-03 14:28:02 +00:00
Rafael Gomes
50ac53f764 satellite/metainfo/objectdeletion: Implement batching
Implement an upper bound for how many objects the service
will handle at a time.

Change-Id: Id90a5c6197ed510a569f4f15134d06e1a9b315a1
2020-08-03 14:09:56 +00:00
Michal Niewrzal
20184d3604 satellite/metainfo: move TestAttributionReport to attribution tests
Additionally test was simplified by adding ability to set user agent for
testplanet uplink.

Change-Id: I82942c2280562b5118a42aa8e1e0f53092f8dbe1
2020-07-30 19:18:15 +00:00
Michal Niewrzal
4561d9bdb0 satellite/console/consoleweb: add support for partner parameter for
signup page

With this change partner id can be associated with user during creating
account by adding `?partner=partner_name` parameter to signup page url
e.g. https://tardigrade.io/signup?partner=mongodb

https://storjlabs.atlassian.net/browse/USR-999

Change-Id: I12a5ebec92a6f5135909447172ef24da57fb1c68
2020-07-30 11:29:31 +00:00
Kaloyan Raev
edfd3d7661 satellite/payments: delete credits and credits_spendings db tables
Jira: https://storjlabs.atlassian.net/browse/USR-822

This the last step of dropping these 2 db tables. It also deletes all
code associate with them.

Change-Id: I8be840dc2a7be255cf6308c9434b729fe4d9391e
2020-07-30 12:19:57 +03:00
Michal Niewrzal
14f49558ac satellite/admin: fix test failing at the end of month
Change-Id: I6db6080614eaa0d775c730a7eec571b02da03e7f
2020-07-29 14:17:11 +02:00
Ethan
dc971751ad satellite/stripemock: Fix race condition on customers and transactions
Failed tests:
https://build.dev.storj.io/job/storj-gerrit/7110/
https://build.dev.storj.io/job/storj-gerrit/7109/

Change-Id: Iaf66fcd1e40607d43691053a75658e4871a0997e
2020-07-28 17:51:28 +00:00
Moby von Briesen
76030a8237 satellite/audit/{queue,chore}: Wait for audit queue to be finished before swapping
* Do not swap the active audit queue with the pending audit queue until
the active audit queue is empty.
* Do not begin creating a new pending audit queue until the existing
pending audit queue has been swapped to the active queue.

Change-Id: I81db5bfa01458edb8cdbe71f5baeebdcb1b94317
2020-07-28 16:56:26 +00:00
Bill Thorp
b265b7f555 satellite/console: make paywall optional
Add a config so that some percent of users require credit cards /
account balances
in order to create a project or have a promotional coupon applied

UI was updated to match needed paywall status

At this point we decided not to use a field to store if a user is in an
A/B
test, and instead just use math to see if they're in a test.  We decided
to use MD5 (because its in Postgres too) and User UUID for that math.

Change-Id: I0fcd80707dc29afc668632d078e1b5a7a24f3bb3
2020-07-28 10:57:49 +00:00
Kaloyan Raev
cb0caa2e25 satellites/payments: resolve data race in StripeMock
Change-Id: I0dbcf043e88edab12704742ad9653c62fd7b08f4
2020-07-28 09:53:50 +00:00
Egon Elbre
36ed939b89 satellite/orders: add buckets db to service
We need to add bucket UUID into the order limit,
hence we need access to the buckets table.

Change-Id: I348ce1f709c9fcdec5c4034acaab59805b33da9f
2020-07-24 17:36:49 +03:00
Egon Elbre
44f9193404 satellite/orders: make optimal threshold multiplier into an argument
It feels weird having a repairer configuration part of order services.
Let's have a single source of truth for it.

Change-Id: I24f7c897aec80f3293f8af24876cbb6733d85a0b
2020-07-24 16:35:59 +03:00
Egon Elbre
ba4c3d9986 satellite/orders: remove unused node status logging flag
Change-Id: I24da78a11cc5d3d88cdf6aca85c4238e4086e59c
2020-07-24 16:35:59 +03:00
Cameron Ayer
46bdb1dbfe Revert "satellite/metainfo: log if replacing pieces in pointer goes wrong"
This reverts commit 9edb363558.

Change-Id: Ia6219f4cca4e766c7919a6d830dfa6900fd4f6e0
2020-07-23 13:42:50 +00:00
Cameron Ayer
e14f7a3fb4 satellite/repair: update healthyPieces and unhealthyPieces after CreateGetRepairOrderLimits
Inside CreateGetRepairOrderLimits we pass in a list of healthy pieces,
but when we query node info from this list we apply the "reliable" filter
again. We sometimes end up with nodes which at first were healthy, but then
became unhealthy, and thus can be repaired, but we do not update the 'unhealthyPieces'
list with these nodes.

This causes an error, 'piece to add already exists', as we fail to remove these
pieces from the pointer before replacing them with repaired pieces.

Change-Id: I6e2445f342ac117ded30351fa7e5e523c9ec26bd
2020-07-23 13:24:46 +00:00
Kaloyan Raev
4bcf308a04 satellite/payments: fetch old deposit bonuses from Stripe metadata
Jira: https://storjlabs.atlassian.net/browse/USR-822

The balance history in Satellite GUI display the deposit bonuses as
separate rows. These bonuses used to be stored in the satellite DB. We
recently started depositing the bonus directly to the Stripe balance and
migrated old bonuses to Stripe metadata.

This change displays all billing history entirely from Stripe, so we can
remove the `credits` and `credits_spendings` DB tables in a next step.

Change-Id: I14c304c66ec47c6a51f5b8508f11470cf36c4e24
2020-07-23 12:11:17 +00:00
Egon Elbre
20437b43a1 satellite/payments/stripecoinpayments: prevent data race
There's still a possibility of tests clashing due to the shared mock,
however it's slightly better, because it avoids the race.

Change-Id: I80eedf1ca50b6114ebe69ea3c4d61176452f4df0
2020-07-23 13:41:48 +03:00
Ethan
cfca021839 satellite/accounting: Add chore to cleanup old project bandwidth rollups data
Removes old project_bandwidth_rollups records that are no longer used.

Uses a retain months configuration to determine how many months to save.  Current month cannot be removed.
Tests retainMonths=-1, 0, 2

Change-Id: Ia4be2546cdb28802427acf41ecd85ad66df3e62c
2020-07-22 18:56:49 +00:00
Michal Niewrzal
18b3495248 satellite/metainfo: restore tests disabled to remove StreamID from
BeginDeleteObject method on uplink side

Change-Id: Id10cdeb367759f9949dc557b291726fd331d38ba
2020-07-22 13:54:39 +00:00
Qweder93
aa6afc3879 error handling in heldamount cash and collector delete fixed
Change-Id: I8fe58c50f844a6b819eacc14a40bc5c67268ed5c
2020-07-22 12:26:13 +00:00
Kaloyan Raev
96c83eb0b8 satellite/payments: log returned error
This makes debugging of issues easier as the error in the web browser
does not show all the info.

Change-Id: Ia79902b7275103a78a21d43ba14243bc32bb8be5
2020-07-22 12:00:27 +00:00
Kaloyan Raev
a20e85824a satellite/payments: add STORJ amount and rate to Stripe TX metadata
Jira: https://storjlabs.atlassian.net/browse/USR-968

We want to keep track of the STORJ amount and exchange rate in the
metadata of Stripe Customer Balance Transaction to be able to generate
reports without the need of requesting CoinPayments for this info.

Change-Id: Ia93af95706cd2312cf688f044874495279fe8fa2
2020-07-22 11:57:21 +03:00
Egon Elbre
ce7adc8d85 satellite/console: remove need for nolint
Change-Id: I1b7ab00720f630c4ad3f984adc5698b41368613b
2020-07-21 09:13:57 +00:00
Bill Thorp
65408db6e0 satellite/satellitedb: Coinpayments repeat insert bug fix
I introduced a bug with https://review.dev.storj.io/c/storj/storj/+/2216
Because the log change allowed insert to be called multiple times.
This changes the insert logic to do nothing if the PK already exists.

Change-Id: I90d192a0f6619bfbb360ea104066f00a3348f6dd
2020-07-20 20:21:35 +00:00
Rafael Gomes
375d76638d satellite/metainfo/objectdeletion: Object deletion implementation
Improve our delete logic to require fewer database requests.
This PR creates a new objectdeletion package

Change-Id: I0500173bb9b8c771accb350f076329ede6dbb42c
2020-07-20 16:09:48 +00:00
Michal Niewrzal
e69cec0b75 satellite/metainfo: stop generating StreamID for BeginDeleteObject
request

We are no longer using `BeginDeleteSegment` or `ListSegments` so we can
avoid generating StreamID as a result of `BeginDeleteObject`.

StreamID from `BeginDeleteObject` is also not used on Uplink side.

Change-Id: I3b068deab17068459849b5cf05811cad4b8a9034
2020-07-17 17:51:06 +00:00
Isaac Hess
67a292d135 satellite/satellitedb: Monitor node tallies
We are adding a monkit evaluation for the total sum of data stored on
the nodes before it is inserted into the database. This will give us a
time-series history of total data stored so we can see it change over
time.

Change-Id: I41145a2d7a09c8e63b42ae578bd081035b60e529
2020-07-17 10:21:42 -06:00
Egon Elbre
b719eecb28 satellite/admin: close http response body
Change-Id: I702e3a7e53d537d8be30b1aaabad4c4e70efe706
2020-07-16 16:38:10 +00:00
Egon Elbre
d8dcae3075 all: fix error checking
Change-Id: Ia0da1bbd6ce695139922f94096c2419281905e32
2020-07-16 19:13:14 +03:00
Egon Elbre
b84923558b satellite: fix scoping, formatting
Change-Id: I21ef9edc2d449d75ad74891df7f966fb150d80fd
2020-07-16 19:13:14 +03:00
Egon Elbre
e70da5cd4e all: fix comments
Change-Id: I2d2307e3fab87de47a72b3595d051e2c95ff4f8a
2020-07-16 19:13:14 +03:00
stefanbenten
1a6e25579c satellite/admin: check for existing user with given email
To prevent creating multiple users with the same email via API, we should check for an existing user with given email.

Change-Id: Ie35b85c4f94a7ca72d42951dab8ff475d7f0dd7c
2020-07-16 15:17:00 +00:00
stefanbenten
0534b91085 satellite/admin: setup payment account for user
Currently a customer created via the IP does not get an payment account until he signs in.
That causes issues if the account should be deleted again.

Change-Id: I393c8f301e426301bb713c423d6ce011138d4ae4
2020-07-16 15:16:50 +00:00
Egon Elbre
080ba47a06 all: fix dots
Change-Id: I6a419c62700c568254ff67ae5b73efed2fc98aa2
2020-07-16 14:58:28 +00:00
stefanbenten
9ace375ee0 satellite/{console,satellitedb}: change project limiting based on new users field
This change switches the backend logic to use the new DB column on the users table to restrict project creation.
Furthermore it back fills the existing limits from registration tokens to the new column to ensure no users are reset to the new default.

UI is updated to reflect ability to create several projects

Change-Id: Ie29157430ae6b065411ca4c4557c9f1be69cdc4f
2020-07-16 10:57:47 +00:00
Jeff Wendling
1944d734ef satellite/orders: check and enforce node api version
Change-Id: Ibdeb1a85dfed8b534bfed32a7cdaae5c3dc8b420
2020-07-16 10:38:12 +00:00
stefanbenten
0209a2095f satellite/{console,satellitedb}: add project_limit column to users table
Change-Id: I603f085f17ca5b413dd1c6837c2081f9e7e791a1
2020-07-15 17:27:31 +00:00
stefanbenten
2c2d284f3d satellite/admin: add bucket limit handling endpoint
Change-Id: I4b199277cff30f11f4a9fff3b0ac4017b694f2e8
2020-07-15 17:27:23 +00:00
Jennifer Johnson
784a156eea satellite: prevents uplink from creating a bucket once it exceeds the max bucket allocation.
Change-Id: I4b3822ed723c03dbbc0df136b2201027e19ba0cd
2020-07-15 17:27:05 +00:00
Jeff Wendling
3a8766936b satellite/orders: remove race condition in new endpoint tests
the flush batch size was set to 1 which means that a flush was
async scheduled after the first write. the explicit trigger wait
was then always flushing nothing, and the test would only
pass if the async flush was scheduled before the read.

remove that async flush and pause the flush loop so that we are
in full control of when the flushes happen so there are no races.

the tests are still disabled but that's because the endpoint is
still disabled.

Change-Id: I2b7b07fd5525388c30be8efbf4af7105087228da
2020-07-14 15:12:33 -04:00
Caleb Case
581579e80a
satellite/rewards: Add Duplicati to partners database.
Change-Id: I1d702504adc21cc7a243fe67f1856bd13928c369
2020-07-14 12:49:48 -04:00
stefanbenten
257855b5de all: replace == comparison with errors.Is
Change-Id: I05d9a369c7c6f144b94a4c524e8aea18eb9cb714
2020-07-14 15:50:25 +00:00
Jessica Grebenschikov
1f1e3f0604 satellite/orders: disable settlementwithwindow, skip flaky tests
Change-Id: Ia60d7e0f2d383919650cdc736ba4569bb26ff2d7
2020-07-14 07:17:18 -07:00
Qweder93
7d6973b5a2 satellite: heldamount and nodestats not returning error node not found by rpc
Change-Id: Ifb00b16a4a04603251de60da6a6612fd5e98d597
2020-07-14 16:31:02 +03:00
stefanbenten
1149417615 satellite/admin: cleanup parameter handling
We passed in revocationDB and metainfoDB for no reason.
Lets remove it from the dependency list to further reduce the footprint.

Change-Id: Ic0317bb92670fbd305d4a8b0ed1cb82858e2f6d3
2020-07-14 13:53:09 +02:00
stefanbenten
133fda4bef satellite/admin: add user deletion
Change-Id: Ideea978698183e25f5e0d73128c4f93c2caa1577
2020-07-14 11:37:00 +00:00
stefanbenten
0a32ba0e6b satellite/admin: add project rename functionality
Change-Id: I4c0f42d4c2c26859279f247f94cef97a8ff630a9
2020-07-14 11:36:49 +00:00
stefanbenten
f768302c91 satellite/admin: harden project deletion requirements
Change-Id: Ia7ea469f87469b16e464dc22af24b98a6ef1873d
2020-07-14 11:36:29 +00:00
Jessica Grebenschikov
8abb907010 satellite/orders: add settle orders with window
Why: We need a way to cut down on database traffic due to bandwidth
measurement and tracking.

What: This changeset is the Satellite side of settling orders in 1 hr windows.
See design doc for more details: https://review.dev.storj.io/c/storj/storj/+/1732

Change-Id: I2e1c151e2e65516ebe1b7f47b7c5f83a3a220b31
2020-07-13 15:41:29 -07:00
Egon Elbre
06a3510ae4 satellite/orders: add EncryptionKey encryption
Add encryption using nacl/secretbox.

Change-Id: I0add31a9fd2359be1bf9d3e43b6c4b9ff3b6fb03
2020-07-13 16:10:19 +00:00
paul cannon
bbdb351e5e all: use jackc/pgx in place of lib/pq
What:

Use the github.com/jackc/pgx postgresql driver in place of
github.com/lib/pq.

Why:

github.com/lib/pq has some problems with error handling and context
cancellations (i.e. it might even issue queries or DML statements more
than once! see https://github.com/lib/pq/issues/939). The
github.com/jackx/pgx library appears not to have these problems, and
also appears to be better engineered and implemented (in particular, it
doesn't use "exceptions by panic"). It should also give us some
performance improvements in some cases, and even more so if we can use
it directly instead of going through the database/sql layer.

Change-Id: Ia696d220f340a097dee9550a312d37de14ed2044
2020-07-13 15:54:41 +00:00
Michal Niewrzal
f532d4b25c satellite/payments/stripecoinpayments: add test for invoice items
values

Change-Id: I924b02e70d7ed36b0a769f1072410dd811390abb
2020-07-13 08:28:46 +00:00
Egon Elbre
9dc9cd8a17 tests: allow STORJ_TEST_POSTGRES
STORJ_POSTGRES_TEST naming was not consistent with STORJ_SIM_POSTGRES.

This allows to use STORJ_TEST_POSTGRES for clarity, it still has a
fallback to STORJ_POSTGRES_TEST.

Change-Id: I6f294c66c80fcfd6750fea2a89795f3b7f5dd691
2020-07-10 16:43:49 +03:00
Egon Elbre
5bdcd86fa7 ci: test benchmarks
This runs each benchmark for one iteration to ensure that they are
valid. Unfortunately, it does not give any useful metrics as output.

Change-Id: I68940398c8dd849aed656bd12656f48d5df10128
2020-07-10 13:26:49 +00:00
Egon Elbre
4869cfc9a4 satellite/vouchers: remove deprecated endpoint
Change-Id: I0a754217d9424253e448126face6594bc143f412
2020-07-10 12:38:46 +00:00
JT Olio
cdb0200280 satellite: collect hardware stats on satellites
Change-Id: Icc4d9920bb3d23be4d51058237da1baa0e1f3291
2020-07-09 19:17:20 +00:00
Jeff Wendling
885ef70c58 satellite/nodeapiversion: new table for tracking node api usage
This system tracks an abstract "api version" from nodes based on
their usage, allowing us to have latching behavior where if a node
ever uses a new api, it can be blocked from using the old api.

This is better than using self-reported semver version information
because the node cannot lie, there's no confusion about what semver
version implies which features, no questions about dev and ci
environments, and no dependencies between reporting the version
and using the new api.

Change-Id: Ifeced5c9ae8e0a16102d79635e176a7d3bdd8ed4
2020-07-09 15:02:25 +00:00
Jessica Grebenschikov
41497569cd satellite/orders: add settled amount to rollups write cache
https://storjlabs.atlassian.net/browse/SM-1109

Change-Id: Ic5859b141c1384157b33df0d7fb6c8b43cc8a6b1
2020-07-07 14:46:06 +00:00
Kaloyan Raev
12a15e5a6a satellite/payments: remove migare-credits billing command
This was a one-time command and it has been already executed on all
production satellites.

Change-Id: Ic58a151c3e5da9c139df875f6517f37275a1ebf5
2020-07-07 11:47:53 +03:00
Isaac Hess
fd740295ec satellite/satellitedb: Add comment to revocation
Change-Id: I1b65b7e46439c4788835ea5bfd4df3d32a713b44
2020-07-06 21:51:35 +00:00
Caleb Case
db55386fbc satellite/rewards: Add Rclone to partners database.
Change-Id: Ieab098f7fb411230b72a88533ae632bac8967266
2020-07-06 16:13:15 +00:00
Bill Thorp
00ae5ebbab satellite/payemnts: Credit coin payments earlier
Apply the coin payments when CoinPayments.net recieves the funds
Instead of the when STORJ gets them from CoinPayments.net

Based on 7/1/20 User Growth standup guidance by JG
Relates to: https://storjlabs.atlassian.net/browse/USR-801

Change-Id: I174ca23a585010f39464c45525e1dfe0179b7c1a
2020-07-06 13:24:26 +00:00
littleskunk
e1a18ce974
satellite/accounting: bugfix infinit storage usage increase (#3916)
* satellite/accounting: bugfix infinit storage usage increase

* support int64 values

Co-authored-by: JT Olio <hello@jtolio.com>
2020-07-03 12:34:13 -06:00
Cameron Ayer
e3088d9ad5 satellite/satellitedb: add new DB table audit_histories
Change-Id: I5f854514994cab9a68cf978f2dabfb588df695f5
2020-07-01 21:14:35 +00:00
Qweder93
b639ec08d4 satellite/heldamount: payments added, endpoind for payments added
Change-Id: Ia2b9580bc353ef614680230c6f82c5bf6ded49c4
2020-07-01 18:15:01 +03:00
Bill Thorp
66661c7486 satellite/coinpayments: query status of >25 coinpayments
Allow more than 25 coin payment statuses to be queries from coinpayments.net
Add some slight logging, a coinpayments duration metric, and a disabled test

These changes are small changes in support of https://storjlabs.atlassian.net/browse/USR-801

Change-Id: I5b176cdd5e513d8bd88b92f9b22a8bd2456bbdd5
2020-07-01 09:04:16 +00:00
Cameron Ayer
cadb435d25 {satellite/audit, private/testplanet}: remove ErrAlreadyExists, run 2 audit workers in testplanet
Since we increased the number of concurrent audit workers to two, there are going
to be instances of a single node being audited simultaneously for different segments.
If the node times out for both, we will try to write them both to the pending audits
table, and the second will return an error since the path is not the same as what
already exists. Since with concurrent workers this is expected, we will log the
occurrence rather than return an error.

Since the release default audit concurrency is 2, update testplanet default to run with
concurrent workers as well.

Change-Id: I4e657693fa3e825713a219af3835ae287bb062cb
2020-06-30 18:00:07 +00:00
Egon Elbre
735dc6e163 satellite/orders: add encryption keys
This adds EncryptionKey definition that can be used as a flag.
These order.EncryptionKey-s will be used to encrypt data in
order limits.

This helps to avoid storing lots of transient data in the
main database.

This code doesn't yet contain encryption itself.

Change-Id: I2efae102a89b851d33342a0106f8d8b3f35119bb
2020-06-30 15:03:14 +00:00
Egon Elbre
3567b49ef4 satellite/metainfo/piecedeletion: fix int to string conversion
Change-Id: I8d0cdec7cee182ade479c0cfb1d38f7f97af2ab7
2020-06-29 17:22:52 +03:00
paul cannon
4997fd55d0 satellite/repair: remove healthy from irreparabledb
Change-Id: Ia9d300d0359883f03734d0bdf204d56d6642ce34
2020-06-26 21:26:00 +00:00
Rafael Gomes
bdaabd611d satellite/metainfo/piecedeletion: add metrics for deletion requests
Change-Id: I48ad96e78dab84b9238c63d62bda679fc65c2072
2020-06-26 11:12:33 -03:00
Egon Elbre
d91cf5f4de satellite/satellitedb: add missing SeparateTx
Change-Id: I3ba5a4e0632a1e0e5e77c30e515953eadf05bc45
2020-06-26 12:27:05 +03:00
Egon Elbre
13a5854535 satellite/satellitedb: clarify test migration merging
Use a field to distinguish migration steps that need to use a
different transaction from previous steps. This is clearer than
using a func.

Change-Id: I2147369d05413f3e8ddb50c71a46ab1ba3ab5114
2020-06-25 14:32:45 +00:00
Egon Elbre
df8cf8f58a satellite/orders: delete unused code
Change-Id: I431c8cc2f23e538c676d6f742fb1faef7cc1d73e
2020-06-25 16:48:26 +03:00
Kaloyan Raev
091b49b921 cmd/satellite: command to move unspent bonuses to Stripe balance
Jira: https://storjlabs.atlassian.net/browse/USR-821

The `migrate-credits` billing command checks the available credits
balance for all users and moves it to the Stripe balance by creating a
new credit balance transaction.

Change-Id: Iafc7b95a4edad47f7c145a22e210f8c821ac183d
2020-06-24 13:20:24 +00:00
Cameron Ayer
3b4b5f45c7 satellite: replace references to Suspended with UnknownAuditSuspended
Change-Id: I3d2d00c95954c0546ad077702617895f262926ef
2020-06-23 14:19:22 +00:00
VitaliiShpital
5b3c8b2f1a web/satellite: google tag manager for signup pages
WHAT:
GTM added for partnered satellites sign up pages
csp values were extended to make GTM work at all:
1. googletagmanager.com for GTM script
2. google-analytics.com for GA script
3. hash was added to avoid using 'unsafe-inline' value in 'script-src' directive

Also config flag for GTM id was added

WHY:
Marketing team needs GTM and GA for their campaigns

Change-Id: Ibb2ace737feb971dda6c191599d479fe4a7af332
2020-06-23 10:45:04 +00:00
Isaac Hess
4ce1c099b1 satellite/metainfo: Add RevokeAPIKey to batch
Change-Id: I53b233a5f9f1c8ce7d0bd30e88998f2cdee37762
2020-06-22 13:50:07 -06:00
Isaac Hess
34b019d16a satellite/metainfo: Handle revocation request
Logic to handle a request to revoke a macaroon.

Change-Id: I5d5c93fcc2e026b0aaf82bfdfacc822185b10f9f
2020-06-22 13:50:07 -06:00
Isaac Hess
2d727bb14e satellite: Check macaroon revocation
When a request comes in on the satellite api and we validate the
macaroon, we now also check if any of the macaroon's tails have been
revoked.

Change-Id: I80ce4312602baf431cfa1b1285f79bed88bb4497
2020-06-22 13:50:07 -06:00
Egon Elbre
6cc7fd5f31 satellite/metainfo: remove old endpoints
Change-Id: Ic85c27caeea08879714e8303b285b6136f9a7fd1
2020-06-16 16:12:14 +03:00
Isaac Hess
05a981bc13 satellite/metainfo: Add RevokeAPIKey stub endpoint
Change-Id: I07fb997dda48ea204aab11f656a1a37762684339
2020-06-16 07:03:02 -06:00
Egon Elbre
f68e7b3fde satellite/overlay: replace pb.InfoResponse
pb.InfoResponse wasn't used for protocol buffer communication, but
instead as a satellite type.

Change-Id: I755619f2deec5b76c4fe488591b7d8c1b9fcdafb
2020-06-16 15:16:55 +03:00
Egon Elbre
6673125c03 satellite/metainfo: remove code for handling partner uuid
Change-Id: I2e0fdc6a23288b3d8e55548f1d97598e48588e4c
2020-06-16 11:31:41 +00:00
Egon Elbre
19d431ff38 satellite/metainfo: disable old endpoints
First step in removing endpoints that were used by pre 1.0 clients.

Change-Id: Id861c73ffac97139aa9540f4973ea849e572f906
2020-06-16 11:22:44 +00:00
Ivan Fraixedes
b72bf4a6d8 satellite/admin: Make minor improvements in README
Make some minimal improvements in the README about the API documentation
regarding the required requests and successful response bodies.

Change-Id: If7832f3c40166a55d9baefdb2395211ff9e8dc04
2020-06-15 10:23:20 +00:00
Cameron Ayer
0885ba5646 satellite/satellitedb: add new columns for offline suspension
add new columns `offline_suspended` and `under_review` to nodes table.
`unknown_audit_suspended` is a new column which will replace `suspended`

Change-Id: I22ddeb338ea0ff63f14332a7ebd0f3e9e4c06cdc
2020-06-15 04:00:20 +00:00
paul cannon
7b8e91ff28 satellite/satellitedb: no orders for exited nodes
We should not be sending any type of orders to nodes that have completed
graceful exit with the current satellite. In particular, we should not
be trying to audit them, because that would be silly.

Change-Id: Ie2153e5739914ab696feefcdef28545ed70f84e4
2020-06-13 13:49:33 +00:00
Rafael Gomes
958ea1b9df satellite/accounting: add download limit cache
Change-Id: I722930cab8bd5d240f4878dc6997e9bc7637311f
2020-06-12 16:33:46 -03:00
VitaliiShpital
8ecf01ece8 web/satellite: credit history page implemented
WHAT:
credit history page implemented.
can be visited by clicking specific button in a free credits dropdown.

WHY:
UI didn't display remaining coupon value.
coupons and referral items (in future) are displayed in the same place.

Change-Id: I495fd7a99f2ea5117152aaf8f495bd5322f02588
2020-06-12 14:31:02 +00:00
Michal Niewrzal
e48177ce81 satellite/payments/stripecoinpayments: add basic summary for invoice
generation commands

https://storjlabs.atlassian.net/browse/USR-823

Change-Id: I24a7dec56973abcbbb66e70842e3aa2b7c01a4b3
2020-06-12 09:52:41 +00:00
Egon Elbre
410d897840 satellite: fix string(int) conversions
Change-Id: I54c6ca8c2dad3c321175f72271b7536cc2a4df09
2020-06-12 06:41:34 +00:00
Egon Elbre
d88394c3d2 satellite/metainfo: verify pointer in service
Change-Id: Ib489a177c93739f48d3a9669a56d9892640c74ef
2020-06-11 13:45:58 +00:00
Egon Elbre
1ed5a1bac5 satellite/satellitedb/satellitedbtest: skip omitted database
The first implementation missed some changes.

Change-Id: I7ae696175e0a9ea46954970ba8547638a05ed5a9
2020-06-11 13:28:16 +00:00
Qweder93
0826c8d87f satellite/heldamount: fix dimension of usage_at_rest
Change-Id: If1518ad41736912d15fb2c882c9e236c16f85a51
2020-06-11 13:07:51 +00:00
stefanbenten
e8ef9b76bb satellite/rewards: add CloudBloq
Change-Id: Ic0b7554c68d9055acbde2745a445bfa883c237af
2020-06-10 21:14:45 +02:00
stefanbenten
c6c8b923af satellite/dbcleanup: run cleanup more frequently
As the tables that get cleaned up by this job get a lot of inserts and deletes over the course of a day, the autovacuum process on PostgreSQL struggles fairly easily/quickly.
Due to its limitation, it can only delete 180,000,000 tuples in one go, before it has to rescan the entire table/index.

With the current load, the most busy satellites accumulate about 1,000,000,000 tuples per day (consumed_serials). With our current 24h interval that results in ~6-7 scans, slowing the entire database down for a quite long time.
This PR reduces the interval to 4 hours, which under a constant load, results in less than 180,000,000 entries per run.
That way, we do not scan twice for only a small gain over said amount. Reducing the interval further would also increase the DB load unnecessary, as each run scans the entire tables at least once.

For future reference, we might need to adjust the interval, if the load is significantly changing.

Change-Id: I18fdd45d93d468cff126e719c8380c29a49f43dd
2020-06-10 18:32:15 +00:00
Cameron Ayer
26fce54b11 satellite/audit: increase MinDownloadTimeout in TestReverifySlowDownload
This test failed due to a timeout on a download which is supposed to
succeed. The testplanet default for the value is 5 seconds, but here
it is 500 milliseconds.

It looks like this is due to the fact that later in the test we need to
wait for a slow node to timeout, so we cut the timeout shorter to reduce
test time.

This PR increases the timeout to 1 second. Still not too long to wait, but
gives us twice as much time to download, decreasing the likelihood that we
see the timeout error.

Change-Id: I504db39ab5dc4d3c505520337b258265d6da7020
2020-06-10 17:43:50 +00:00
Cameron Ayer
bad299b541 satellite/satellitedb: serialize UpdateStats and BatchUpdateStats transactions
Since we increased the number of audit workers from 1 to 2, we need to make sure
concurrent updates do not trample each other. We can do this by serializing the
transactions.

Change-Id: If1b2f71cabe3c779c12ffa33c0c3271778ac3ae0
2020-06-10 17:11:28 +00:00
stefanbenten
dfeb4dc9c2 satellite/payments: add finalize invoice command
Change-Id: Ib74dc8fd352e9576a302f5923439f5e50f69c9f0
2020-06-10 14:48:05 +00:00
Qweder93
e52809d53e cmd/storagenode: add check if satellites available to gracefulexit
Change-Id: I8747507593d810bbdec0d140de0600ee147011c3
2020-06-10 13:38:36 +00:00
VitaliiShpital
47a766328f web/satellite: deposit and billing history splitted to be shown separately
WHAT:
1. Deposit & Billing history view was divided to be shown separately as Deposit History and Billing History
2. Datepicker was removed from billing page

WHY:
billing UX enhancements

Change-Id: Ie183849ef0965169997674ce37b71db38a562fc2
2020-06-09 12:32:02 +00:00
Michal Niewrzal
90e2e3c8c4 satellite/payments/stripecoinpayments: display more details about
invoice item generation

Part of https://storjlabs.atlassian.net/browse/USR-823

Change-Id: If0ffa56b68bb0a6b9b4e145fa21468006e7507f6
2020-06-08 23:33:38 +00:00
Egon Elbre
36c461bd59 private/tagsql: track proper closing of rows and statements
This ensures that rows are closed to avoid leaks.
Also verifies that Err() is called, to ensure that no
error is left behind.

Change-Id: Idd1bec9bf479f40021da67b2c80ce83033149469
2020-06-05 18:25:43 +00:00
Jeff Wendling
943eb872d3 satellite/audit: depend less on details of some error message
the error message changed when we removed spacemonkeygo/errors

Change-Id: I4904a072cfd84e4c39c881b58669325bcf51df46
2020-06-05 10:39:05 -06:00
Egon Elbre
34db4a80fd ci: fix staticcheck failures
Change-Id: I176fb24214755a1940a0a1a4e9cc8e39f184870b
2020-06-05 13:15:34 +00:00
Michal Niewrzal
2b2efcc662 satellite/payments/stripecoinpayments: move Coupons expiration date
sorting directly to listing method

Change-Id: I58d8a6ea1feba9ff2d19f21a1dbc87bfb8b49801
2020-06-04 09:47:42 +00:00
Michal Niewrzal
96286fde47 satellite/payments/stripecoinpayments: list projects by owner, not
by project members

This is a fix for listing the same project twice because project has
more than one member.

Change-Id: I3f6fe3456a6753d6d091a64436c22027dcbe2520
2020-06-04 08:42:01 +00:00
Jeff Wendling
254b42ff65 satellite/satellitedb: fix leaked rows from repairQueue.Insert
Change-Id: If5e62c49770f591ebe3f4d2dd4dd2658c229a022
2020-06-03 14:31:21 -06:00
Michal Niewrzal
b20ced9519 satellite/satellitedb: drop project_id column from coupons table
This is last part of https://storjlabs.atlassian.net/browse/USR-818

Change-Id: I053d11b37df962c12e46645bae2fc2dad49c9755
2020-06-03 14:56:41 +00:00
Kaloyan Raev
fbfc3e5d18 satellite/payments: adjust label for bonus
This change adjusts the label for STORJ deposit bonuses in billing
history to be more consistent with other labels.

Change-Id: I5e7179ae3ac52dafb0dcef084e9a7c4742491f9e
2020-06-03 13:15:14 +00:00
Cameron Ayer
6a60e1e96b satellite/satellitedb: inclusive interval_start in GetAllocatedBandwidthTotal
The DB query in GetAllocatedBandwidthTotal uses an exclusive range:

'WHERE interval_start > ?'

The value that is used for this condition is the first day of current the month,
00:00:00 UTC.

By using the exclusive '>', we exclude the entire first hour of the month from the
result set.

Change-Id: I3ed300f5230c7514dc9495a85e8166213cd0842e
2020-06-02 13:06:45 -04:00
VitaliiShpital
c272872d5d satellite/payments: available coupon value feature
Change-Id: Ieae9385fbd917230298afff91a6be2838ad9b313
2020-06-02 10:00:45 +00:00
Michal Niewrzal
e8c4010311 satellite/metainfo: add missing error check
Change-Id: I4043633505a3eaee8304f71f5ddea17de2b3d10f
2020-06-02 09:38:55 +00:00
Jeff Wendling
2b3545c0c3 satellite/satellitedb: use delete returning to query pending_serial_queue
this way we don't have to do 2 steps, and by using the ctid, postgres
is going to do two very efficient prefix scans.

Change-Id: Ia9d0546cdf0a1af67ceec9cd508d336a5fdcbdb9
2020-06-01 15:43:33 -06:00
Moby von Briesen
b82d04e618 satellite/metainfo: limit size of uplink-provided metadata to 2KiB
Change-Id: Id44a46046ddb4a12102525531f4502fcff2b6252
2020-06-01 16:51:29 -04:00
Jeff Wendling
44433f38be satellite/satellitedb: remove ORDER BY when reading from queue
also remove the continuation support from the queue, otherwise
we may end up sequential scanning the entire table to get
a few rows at the end.

then, in the core, instead of looping both to get a big enough
batch inside of the queue, as well as outside of it to ensure
we consume the whole queue, just get a single batch at a time.

also, make the queue size configurable because we'll need to
do some tuning in production.

Change-Id: If1a997c6012898056ace89366a847c4cb141a025
2020-06-01 18:31:14 +00:00
Yingrong Zhao
163c027a6d satellite/satellitedb: remove monkit trace from convertDBNode
In jaeger, it shows that this function gets called repetitively in
a single request. Most of the time, it's less than 1ms. Therefore, it
doesn't add much value in our trace but create noises.

Change-Id: I20234f36bbcf0fc22f91e5e1a5634c0cad577ed0
2020-06-01 17:58:43 +00:00
Kaloyan Raev
d66e646b57 satellite/payments: add deposit bonus to stripe balance
Jira issue: https://storjlabs.atlassian.net/browse/USR-820

The bonus for depositing STORJ tokens is now added as to the Stripe
balance instead of the to `credits` DB table on the satellite.

Existing unspent bonuses in the `credits` DB table are still processed
as usual when generating invoices. They will be migrated to the Stripe
balance with a separate change.

The bonus is added to the Stripe balance with a separate Credit
transaction. The balance transactions for the deposit and the bonus can
be differentiate by their different description.

The billing history is modified to list the bonus from the Stripe
transactions list.

The workflow for depositing STORJ tokens to the Stripe balance is
improved to survive failures in the middle of the process.

Change-Id: I6a1017984eae34e97c580f9093f7e51ca417b962
2020-06-01 17:41:10 +00:00
Kaloyan Raev
948cc7333d satellite/payments: mark coupon as expired on its last month
Jira issue: https://storjlabs.atlassian.net/browse/USR-110

When generating invoices, if a coupon is not fully spent, but this is
its last valid month, it will be marked as expired in the database.

Change-Id: Ib1b2375f6703f9514d1ae089a387aa029b62d1fb
2020-06-01 17:10:00 +00:00
Michal Niewrzal
a9f6489663 satellite/payments/stripecoinpayments: remove ProjectID from Coupon
struct

This change is removing ProjectID from code. Next change will be about
dropping this colum from DB table.

Change-Id: Idb949e2829e2c304a2b6b011259c7cc7667082e1
2020-06-01 11:37:20 +00:00
Michal Niewrzal
1b69d20eb3 satellite/payments/stripecoinpayments: apply coupons in the order of
their expiration date

Change-Id: Ib013899d82715adf7b3002f2067d6481ceb59de2
2020-06-01 11:01:04 +00:00
VitaliiShpital
ee9bb0d689 satellite/payments: coupon expiration bug fixed
Change-Id: Icc89e9ee6a1dd91109d34909490ee446b716e2ed
2020-06-01 12:20:08 +03:00
Jeff Wendling
839a6f15d1 stripecoinpayments: use fixed date for test
the test as written fails if it's the last day of the month because
the two tallies that are added span two different periods.

Change-Id: I294397278508035ce640500fc3b1b0ffd82b9b71
2020-06-01 09:02:52 +02:00
Michal Niewrzal
f6452e9c09 satellite/payments/stripecoinpayments: process customers instead of
projects

Currently, to generate invoices we are listing all projects. This is
problematic because we decided that elements like e.g. coupons should be
calculated per user. We want to have
more user oriented processing. With this change to prepare invoice
elements we are:
* listing Stripe customers
* listing projects for each customer
* processing projects to create project records
* calculating coupons per customer
* calculating credit spending per customer

Change-Id: I48b0063e44e4d301f97c87ecffb342649359968a
2020-05-30 13:16:07 +00:00
littleskunk
801a3ab90d
satellite/coinpayments: Reduce update interval to 2 minutes (#3897)
* satellite/coinpayments: Reduce update interval to 2 minutes

* satellite/coinpayments: Reduce balance update

Co-authored-by: paul cannon <thepaul@users.noreply.github.com>
2020-05-29 22:21:27 +02:00
VitaliiShpital
c9b9c686fc web/satellite: logic for new signup/login flow
WHAT:
1. updated verification page URL in config
2. added list of partnered satellites to config
3. added logic for satellites dropdown on new signup/login pages

WHY:
1. signup/login flow was reworked in tardigrade.io repo (iframe removed, new pages etc.)
2. new config flag was added to check if satellite name matches at least one member of partnered satellites list to redirect user to verification page
3. new pages will have dropdown with partnered satellites list. Appropriate logic was added.

Change-Id: I33399ab66ca31f07b297a433f6b1f41da4cb6e66
2020-05-29 17:11:44 +00:00
Moby von Briesen
e7e69f383a satellite/repair/repairer/ec.go: Add monkit tracing for ec repairer
Adds monkit tracing for ecrepairer.downloadAndVerifyPiece and
ecrepairer.putPiece so we can get more accurate estimates of node
performance during repair.

Change-Id: Ic05025bf3c493bb3d6f5d325d090c5b7c9e5465d
2020-05-29 14:00:45 +00:00
Egon Elbre
d6c90b7ab5 pkg/macaroon: remove aliased package
Change-Id: I8b06e9a794d00752948f190623006e1afffdb031
2020-05-29 13:34:20 +00:00
Egon Elbre
07050eea26 all: use common/storj
Change-Id: Id1e36d52f9807b5ffbb72ce73f4b60cb21b68a78
2020-05-29 11:57:32 +03:00
Moby von Briesen
acf8b72cd0 satellite/repair/repairer: cut off long tail when minimum number of required uploads is met
This will speed up the Put step of repair by not waiting to time out for
a handful of slow nodes, at the expense of a slightly less durable
pointer. It will still repair to the optimal threshold, but not every
node that is selected will end up in the pointer.

Change-Id: I02a0658e3fe6fc0383f26af0f50a065b8b11a651
2020-05-28 16:25:28 -04:00
Jeff Wendling
1e065fb450 satellite: migration to fix bad imported payment history
the initial calculations for the historical values of comp_at_rest
were wrong. because our historical data only included total amounts
as well as compensation for bandwidth, the at rest value was
calculated as

    at_rest = total - bandwidth

unfortunately, that calculation did not take surge pricing into
account correctly. the at rest and bandwidth values do not
include surge pricing, but the total that was used did. so what
we actually calculated was

    no_surge_at_rest = surge_total - no_surge_bandwidth

which will create a value that is too large. this migration
fixes the calculation for imports that are old enough and
of a non-negligable difference.

Change-Id: I61eb0b670510f6d7fb8fc3de39ba79150fac10eb
2020-05-28 12:59:08 -06:00
paul cannon
7395dd1e6e storagenode/gracefulexit: revalidate existing pieces
..before they are transferred to another node and submitted to the
satellite as successful piece transfers, because if we submit an invalid
signature, the node will be marked as a cheater and disqualified
immediately.

These signatures should have been validated when the piece was
originally stored, but bitrot does happen and needn't be cause for an
immediate DQ.

Change-Id: I8b0ebd5812ea8a2e60766005b7251fbb74ef7857
2020-05-28 09:50:14 -05:00
paul cannon
0c9a4a5e56 satellite/gracefulexit: better validation error messages
Change-Id: I14168dbeaf17302e5e853854956f8fbcb82a0900
2020-05-28 09:41:15 -05:00
Michal Niewrzal
84892631c8 private/testplanet: remove old libuplink from testplanet
Change-Id: Ib1553f84d0b3ae12a5b00382f0f53357b6a273e2
2020-05-28 13:50:23 +00:00
Ethan
d5dfeca84d satellite/payments: Return empty iterator instead of nil from StripeMock List methods
Some satellites do not have payment configured (ex. Salt Lake, Europe North). In this case the StripeMock is used, which returns nil on invoice and charges List methods. This causes a panic.

https://storjlabs.atlassian.net/browse/SM-978

Change-Id: Iec1b0bfd9b383e6f793d03dd63a3dec60e0fd9f3
2020-05-27 16:20:18 +00:00
Kaloyan Raev
ee7de0424b satellite/payments: use quantities in invoices
Jira issue: https://storjlabs.atlassian.net/browse/USR-719

Invoices now show the amount of used resources and their unit price.

This change also makes proper rounding to the nearest cent in few places
to resolve the "off-by-one-cent" issue observed in few invoices.

Change-Id: I2d70d6916b5cf4a9a9138c99c422f5f4d2deb35b
2020-05-27 09:33:12 +00:00
Michal Niewrzal
75b3db5426 satellite/payments/stripecoinpayments: test invoice user with more than
1 project

https://storjlabs.atlassian.net/browse/USR-291

Change-Id: I98286e40254e8868de9eb675a9c9a8cd0bf5f3b1
2020-05-27 09:12:23 +00:00
Moby von Briesen
290c006a10 satellite/repair/{checker,queue}: add metric for new segments added to repair queue
* add monkit stat new_remote_segments_needing_repair, which reports the
number of new unhealthy segments in the repair queue since the previous
checker iteration

Change-Id: I2f10266006fdd6406ece50f4759b91382059dcc3
2020-05-27 06:23:47 +00:00
Stefan Benten
ed857f0fb4
satellite/admin: Add the proper documentation for bandwidth limits (#3900) 2020-05-27 09:22:45 +03:00
Egon Elbre
c5d4a13158 satellite/nodeselection: use NodeURL
Change-Id: I2ebd4dbf993ff5c7864f3a3a665b5c8fc48aa7d1
2020-05-27 05:46:11 +00:00
Cameron Ayer
9edb363558 satellite/metainfo: log if replacing pieces in pointer goes wrong
Every now and then we see the repair error, "piece to add already exists".
With these new logs we should be able to verify if it is due to a change in
the pointer. These logs are only temporary

Change-Id: I029390cc4816668707546df14ed2cfe7ca192b0b
2020-05-26 21:48:40 +00:00
Michal Niewrzal
88a2561317 satellite/payments/stripecoinpayments: don't request anything from
Coinpayments if credentials are missing

Workaround for hitting Coinpayment API while tests.

Change-Id: I27114c820d4d7e17d05b066f55a13a709dac1711
2020-05-26 19:38:12 +02:00
Ethan
7014cf2083 satellite/gracefulexit: Change handleFailed to return nil if we can't get the pending transfer
https://storjlabs.atlassian.net/browse/SM-975

Change-Id: Ia1746b5e274e5011b6cd9f5a52f9a5faf703be51
2020-05-26 13:07:38 +00:00
littleskunk
2fbb34c3ea
nodeselection: Increase minimum free space to 500MB (#3898) 2020-05-25 12:13:28 +02:00
Jeff Wendling
074649835b satellite/satellitedb: add some docs and improve some snapshots
This attempts to add a README.md to help create consistent migrations
that maximize our test coverage and do not include unnecessary
statements.

It also adds a feature to have an `-- OLD DATA --` section as well
as a `-- NEW DATA --` section so that we can fix mistakes made in
previous snapshots (like a row that was forgotten to be added when a
table was created) without editing them going forward.

Change-Id: I28a786f8ef163cae1de1bb08f61af1e1104b0a88
2020-05-22 21:27:36 +00:00
Michal Niewrzal
5c10964040 satellite/payments/stripecoinpayments: add test for listing issues while
invoice generation

https://review.dev.storj.io/c/storj/storj/+/1853
https://review.dev.storj.io/c/storj/storj/+/1882

Change-Id: Ie71363b819866dd60dbe7117b42cfa8348479310
2020-05-22 17:24:16 +00:00
Michal Niewrzal
83cc80ef06 satellite/payments/stripecoinpayments: fix listing issue for coupon
usages and credit spendings

Test will be added as a separate change.

Change-Id: I9341d2ecb736fdbd45beb96f778f04448df68bd7
2020-05-21 07:48:04 +00:00
Egon Elbre
bef84a5f9d storagenode: remove dependency to overlay.NodeDossier
This is the last dependency from storage node to satellite.

Change-Id: I12f7abb91e84f823ba5af126c6e2979519838612
2020-05-21 08:37:13 +03:00
Egon Elbre
b42778c42e private/testplanet: remove some additional Local-s
Change-Id: I49701c41efb92efca27cc18d0a3f6d6b44d3cf8b
2020-05-21 08:37:13 +03:00
Egon Elbre
7f323754a4 metainfo/piecedeletion: use NodeURL-s
Change-Id: I247dbfe03e7864e940e4cd1d0f343f38e84099e0
2020-05-21 08:37:13 +03:00
Jennifer Johnson
03e5f922c3 satellite/overlay: updates node with a vetted_at timestamp if they meet the vetting criteria
What: As soon as a node passes the vetting criteria (total_audit_count and total_uptime_count
are greater than the configured thresholds), we set vetted_at to the current timestamp.

Why: We may want to use this timestamp in future development to select new vs vetted nodes.
It also allows flexibility in node vetting experiments and allows for better metrics around
vetting times.

Please describe the tests: satellitedb_test: TestUpdateStats and TestBatchUpdateStats make sure vetted_at is set appropriately
Please describe the performance impact: This change does add extra logic to BatchUpdateStats and UpdateStats and
commits another variable to the db (vetted_at), but this should be negligible.

Change-Id: I3de804549b5f1bc359da4935bc859758ceac261d
2020-05-20 16:30:26 -04:00
Michal Niewrzal
fe6a6f063f private/testplanet: cleanup predefined data generation
Use Console service to create user and project instead direct DB
modifications.

Change-Id: Ib0074b38313b3dc43b7d8d63ab2775d29028fb7b
2020-05-20 12:38:43 +00:00
Qweder93
49ad90dcd8 storagenode/reputation: unknown_score (unknown_alpha / unknown_alpha+unknown_beta) added to reputation stats, https://storjlabs.atlassian.net/browse/SG-326
Change-Id: I0b29ad736f7a11c7e57a846b6891f4b40755aa48
2020-05-20 11:25:14 +00:00
Egon Elbre
5d016425f1 satellite/{contact,downtime,overlay}: use NodeURL
Change-Id: I555a479a89e0ddbf0499898bdbc8574282cd6846
2020-05-20 11:09:05 +00:00
Egon Elbre
941d10cbc3 private/testplanet: remove Peer.Local()
Currently storagenode depends on overlay.NodeDossier, this is the first
step in removing it.

Change-Id: I034a3f1601835f8349bd41752455022e19bcc707
2020-05-20 11:05:34 +00:00
Egon Elbre
94b2b315f7 storagenode/trust: refactor GetAddress to GetNodeURL
Most places now need the NodeURL rather than the ID and Address
separately. This simplifies code in multiple places.

Change-Id: I52621d8ca52296a8b5bf7afbc1001cf8bfb44239
2020-05-20 11:05:15 +00:00
Egon Elbre
ed627144ed all: use DialNodeURL throughout the codebase
Change-Id: Iaf9ae3aeef7305c937f2660c929744db2d88776c
2020-05-20 10:36:30 +00:00
Stefan Benten
1a4ab3df48
satellite/admin: add user update endpoint (#3896) 2020-05-20 12:20:04 +03:00
littleskunk
8ec64f3daf
satellite/overlay: enable node selection cache on all satellites (#3895) 2020-05-19 19:25:53 +02:00
Jess G
6fac1ed130
satellite/contact: rm unhelpful verbose logging (#3894)
* unhelpful rm verbose logging

Change-Id: I74dc29f18969250fa6838fed9bec8ba7ea160b77

* change to debug

Change-Id: I4ba880e68b40bc3e0f29d9b1aa9154e63124c904
2020-05-19 08:09:00 -07:00
Stefan Benten
0a26c4af9a
satellite/admin: add coupon deletion (#3893) 2020-05-19 15:49:44 +03:00
Stefan Benten
671aca56b0
satellite/admin: add coupon creation and listing (#3891) 2020-05-19 12:36:13 +02:00
Michal Niewrzal
ac375d37bc satellite/payments: remove mockpayments and add Stripe client mock
instead

Change-Id: If3496f6abc16da90d2b43fa0c5be356847a39507
2020-05-19 09:35:37 +02:00
Egon Elbre
16abf02b35 satellite/{nodeselection,overlay}: use the new package
Change-Id: I034fdbe578dec2e5c906aca82231cd3e56f26aeb
2020-05-18 21:38:43 +00:00
Stefan Benten
17d3cb1551
satellite/admin: add user creation endpoint (#3889) 2020-05-18 22:37:18 +02:00
Egon Elbre
08692aef90 satellite/nodeselection: node selection with proper bias
Currently node selection cache is biased towards the same subnet. This
implements static node selection for distinct such that it selects with
equal probability subnets rather than id-s.

This is mostly a copy paste + modifications from previous node selection
state.

Change-Id: Ia5c0aaf68e7feca78fbbd7352ad369fcb77c3a05
2020-05-18 18:09:15 +00:00
littleskunk
1b06e2a928
satellite/metainfo: remove deprecated error validation (#3876) 2020-05-18 19:58:42 +02:00
Stefan Benten
70502658c8
satellite/admin: add delete project endpoint (#3888) 2020-05-18 19:36:09 +02:00
Jessica Grebenschikov
1ecbfc453f satellite/overlay: add random test for node selection from cache
Change-Id: I57b57db919b5afb3fc81176e078ac1e3a873565a
2020-05-18 16:59:52 +00:00
Kaloyan Raev
aac1e3c45f satellite/payments: move inspector commands to satellite cli
This allows to seeing logs in the output of the invoice commands.

Existing ensure-stripe-customer commands is moved from the 'reports' to
the new 'billing' root command.

Change-Id: I752c7ab6ca59bfac8e0f174a45d2ab45fc18e467
2020-05-18 16:12:45 +00:00
VitaliiShpital
47def02094 satellite: account balance divided into Free Credits and Coins
Change-Id: I2ae29701bab9f0b48f236ff94b366eb534b81c6b
2020-05-18 14:25:26 +00:00
Michal Niewrzal
9fb423ac19 satellite/payments/stripecoinpayments: fix listing issue in
InvoiceApplyProjectRecords

ListUnapplied method in listing loop  was using next offset as starting
point but applyProjectRecords method was changing project record state
to applied and we were missing some records to apply.

Test will be added as a separate change.

Change-Id: Id1ca33eeb66ec7f6ff1f05b45615a8935185568e
2020-05-18 13:11:30 +00:00
Egon Elbre
f6301612ac satellite/orders: use serial SerialNumber
By ensuring that they have less randomness it means that they can be
compressed better. Using a timestamp should be a good improvement here.

Change-Id: Ic4dabb53335a744ff1c332dd279f37ae2cd79357
2020-05-15 11:33:53 +00:00
Michal Niewrzal
2eb2c25e51 satellite/payments/stripecoinpayments: add StripeClient interface to be
able to cover more testing scenarios

Currently, its hard to implement test suite for payments because
mockpayments is on to high level and we cannot emulate many things e.g.
adding credit card. This change is first to be able to add mock for
Stripe client and do more granular tests.

Change-Id: Ied85d4bd0642debdffe1161657c1e475202e9d23
2020-05-15 10:52:44 +02:00
littleskunk
28a886f399
satellite/admin: obvious project name in documentation 2020-05-14 15:50:58 +03:00
Kaloyan Raev
49571f1a23 satellite/payments: all invoice commands require period
To avoid including multiple months in a single invoice, we need all
inspector's invoice commands to run in for specific period.

See https://storjlabs.atlassian.net/browse/USR-725

Change-Id: I3637dc189234f02350daca8d897c21765762ea55
2020-05-14 11:50:19 +00:00
Moby von Briesen
46df8c1977 satellite/gracefulexit: add log message when node fails validation for piece transfer
Change-Id: Ic5a53404ceb35003793aebc63637e7f8a58ef259
2020-05-13 16:58:50 +00:00
Jeff Wendling
6352d46100 satellite/satellitedb: do better ::date conversions
There is a subtle problem when one does a cast with `::date`. Observe:

    teststorj=# set timezone = 'US/Eastern';
    SET
    teststorj=# select (timestamp with time zone '2020-02-01 00:00:00+00')::date;
        date
    ------------
     2020-01-31
    (1 row)

    teststorj=# set timezone = 'UTC';
    SET
    teststorj=# select (timestamp with time zone '2020-02-01 00:00:00+00')::date;
        date
    ------------
     2020-02-01
    (1 row)

In order to correctly determine the date a timestamp is in, one has to
explicitly pick the time zone that the date truncation should use
otherwise postgres will use whatever setting the client has. These
tests were failing for me locally, because I run my postgres in
the US/Eastern time zone to try to tickle these bugs out. So it
should be `(x at time zone 'UTC')::date` instead of just `x::date`.

Change-Id: I4e9e32d4b53abc6165a4d0474f4702f8b9f801c7
2020-05-13 15:58:07 +00:00
igor gaidaienko
75ddecdf54 satellite/metainfo: Add test to verify payment report for attributed bucket
For this automated test I want to make sure the data in payment report is correct

Change-Id: I48a3b0e1dc9b2324435b763c87c363f1e310ff4e
2020-05-13 17:22:04 +03:00
Bill Thorp
5a7a4d2e98 satellite: add Go test version of satellite-config-lock tests
The current satellite config lock code relies on bash scripts and
gnu diff, it must be run as root and hence it typically requires
docker.  The old version will be removed at a later date..

I tried for several hours to run directly against cmdSetup() in
cmd/satellite/main.go, to avoid the ctx.Compile() call. I had no
luck.

Change-Id: I0a4888421e743b436d32b6af69d04759d7816751
2020-05-13 08:14:24 +00:00
Ethan
159df8b2e4 Add logging listener for retrieving and setting log levels
See https://storjlabs.atlassian.net/browse/SM-752

These changes allow us to change the log level at runtime through a handler off of the debug endpoint.

Examples of changing the log level on storj-sim

To get the current level for the satellite api process:
curl -XGET 'http://127.0.0.1:10009/logging' --header 'Content-Type: text/plain'

To change the log level:
curl -XPUT 'http://127.0.0.1:10009/logging' --header 'Content-Type: text/plain' --data-raw '{"level":"error"}'

Change-Id: I05d164b290929fa06b6d78c01075ee41f8238044
2020-05-12 16:38:06 -04:00
Egon Elbre
0e3be60b79 satellite/satellitedb: simplify migrate step
Change-Id: Ie4574144fb6ddd057d5fca740702c59fbdb2c5e4
2020-05-12 18:27:07 +03:00
Michal Niewrzal
0e263f0099 satellite/admin: add API endpoint for adding projects
Change-Id: I7970aa8bec810ee36727446d632a950c836f0597
2020-05-12 14:55:41 +00:00
Stefan Benten
e23bd806b4
satellite/accounting: separate usage and bandwidth limit (#3878) 2020-05-12 15:01:15 +02:00
Michal Niewrzal
0619f97e20 satellite/console: remove unused arguments
Change-Id: I3ef2b244137391daafee209e20cec383c6772e2e
2020-05-12 11:20:13 +00:00
Stefan Benten
65f3e26f80
satellite: Change Default Project Limits and minimum STORJ Payment (#3877) 2020-05-12 14:18:58 +03:00
Kaloyan Raev
073f9ed499 satellite/payments: deduct stripe discount first
We have 3 types of discounts:
1) Promotional credits/coupons
2) Bonus from depositing STORJ tokens
3) Stripe discounts (e.g. 100% off for Storjlings, 30% off for Early
Adopters, etc.)

So far the discounts were applied in the above order. But because the
Stripe discount is applied on all of the project usage fees, this could
sometimes lead to negative total in the invoice. Especially, if the
Stripe discount is 100% or all of the project fees are covered by
coupons and bonus.

To resolve this issue, before applying promotional coupons and deposit
bonuses, the Stripe discount will be applied first to the project fees.

Change-Id: I5dcbec04ec3a04e7f76b11e0a228ccb3195f2db0
2020-05-12 09:26:27 +00:00
Kaloyan Raev
b397dfc82d satellite/payments: better labels for discounts in invoices
Before:
- Discount from coupon: Promotional credits (limited time -
2 billing periods)
- Discount from credits

After:
- Promotional credits (limited time - 2 billing periods)
- Credits from STORJ deposit bonus

This way we don't mix the terms coupon and credit. And it is clearer
when the credit comes from a deposit bonus.

Change-Id: I4bba76a5501147f9de399eac41c4f157d6bda032
2020-05-11 16:12:04 +00:00
Michal Niewrzal
22fbe804e3 satellite/accounting: test if project bandwidth limits reset with
billing cycle

https://storjlabs.atlassian.net/browse/USR-287

Change-Id: I4dc5f6342417b6af3384da32d3d2ed8592904406
2020-05-11 15:11:53 +00:00
Egon Elbre
ec589a8289 all: fix comments about grpc
Change-Id: Id830fbe2d44f083c88765561b6c07c5689afe5bd
2020-05-11 13:05:34 +03:00
Egon Elbre
7d29f2e0d3 all: remove drpc wrappers
Change-Id: I45016f7d2a771dc00776196c1f531f3343e93b40
2020-05-11 08:20:34 +03:00
Egon Elbre
e6d5ce6b77 all: remove grpc
It seems everyone has migrated to drpc.

Change-Id: Ica6b2d0bdef68c6603083f2963458843eca71e9e
2020-05-10 06:36:09 +00:00
Egon Elbre
bcd93ee375 private/testplanet: add StopNodeAndUpdate
This was commonly used and code with it can be simplified.

Change-Id: I2f2b91f7de54269aee6ef027f97f9e8a7d222e39
2020-05-08 13:02:19 +00:00
Kaloyan Raev
12fa505692 satellite/payments: fix invoice amount in billing history
The billing history currently shows the Total amount from the Stripe
invoice. In fact, this value is just the amount deducted from the Stripe
balance. It does not reflect any deduction from promotional coupon or
bonus credits.

This patch adds the deducted amount form the promotional coupons and
bonus credits to the displayed amount in billing history. This way
customers have better understading of the total amount deducted from
their account balance on the satellite.

Change-Id: Ibd7f611a6cea0143a3059f39dd1d9ef21c264d8c
2020-05-08 11:43:41 +00:00
Egon Elbre
678b859172 satellite/overlay: remove MinimumRequiredNodes
In non-test code we were only using RequestedCount, not need to have
MinimumRequiredNodes.

Change-Id: I40736f4b028b41e94abfdeb221bce5aa86a5cb82
2020-05-07 15:41:23 +00:00
Egon Elbre
ce6a500b0c satellite/overlay: support DistinctIP=false in selection cache
Most of our tests and storj-sim are using DistinctIP.

Also fix bug with newNodeCount calculation.

Change-Id: I1a6d0efe7074908896a3322d19f487b929f0f0fc
2020-05-07 11:04:32 +00:00
Egon Elbre
c5452a87ec private/testplanet: use drpc referral manager server
Change-Id: I9e9e9a724c78c98859dd3e29416d766d8ffdca63
2020-05-07 07:03:11 +00:00
Egon Elbre
2955c50bc1 satellite/overlay: fix data-race in node selection cache
GetNodes returned references to nodes in the immutable state, however
some parts of code expect them to be modified.

Change-Id: I5be1866f95e0dbe062a6b6be60e29f2365c35faa
2020-05-06 20:03:06 +03:00
Egon Elbre
4e94da3fda satellite/overlay: add feature flag for node selection cache
Also distinguish the purpose for selecting nodes to avoid potential
confusion, what should allow caching and what shouldn't.

Change-Id: Iee2451c1f10d0f1c81feb1641507400d89918d61
2020-05-06 16:13:47 +03:00
Qweder93
b754806b23 storagenode/reputation: unknown_audit_reputation_alpha and beta added to db, and reputation endpoint
Change-Id: I5d92268851bb9a202cc5d6b4d403467e6f692726
2020-05-05 15:48:04 +00:00
Jennifer Johnson
18078bf7ee satellite/audit: increases audit worker concurrency to 2
Change-Id: Ibe3e3801b79accffbcfe9e2e02c96fc963894a7f
2020-05-05 11:31:55 +00:00
Egon Elbre
d98b8f6e23 satellite/metainfo,storage: use different limit for metainfo loop
Change-Id: I5ef7233930679b977b33f7b3e1dda45c907dcfad
2020-05-05 10:37:20 +00:00
Moby von Briesen
8f60cfc4fb satellite/overlay: Add flag for enabling/disabling disqualification from suspension mode
Add a flag that allows us to easily switch disqualification from
suspension mode on or off. A node will only be disqualified from
suspension mode if it has been suspended for longer than the grace
period AND the SuspensionDQEnabled flag is true.

Change-Id: I9e67caa727183cd52ab2042b0a370a1bcaebe792
2020-05-04 17:25:09 +00:00
Ethan
acf53bea4d satellite/orders;accounting: Add monthly project download bandwidth rollup
See https://storjlabs.atlassian.net/browse/SM-776

Change-Id: Ifd5cccea43c556fd59822d17344f399cfe9a7164
2020-05-04 15:49:57 +00:00
Michal Niewrzal
22691bd47f satellite/accounting: test project limits reset after file delete
https://storjlabs.atlassian.net/browse/USR-288

Change-Id: Ia63e58ef3b0f79b5edfb9a63f8edcdc17486b650
2020-05-04 15:19:19 +00:00
Egon Elbre
c630cf2490 storagenode/pieces: implement buffering for writing
Currently uploads can cause a lot of IOPS, reduce this by introducing a
in-memory buffer on-top of the file.

Change-Id: I5f4e3e01c0a36258271d180b922107de447bcb59
2020-05-04 06:01:32 +00:00
Stefan Benten
01e0ba2e0d
satellite/metainfo: move logging to debug for piece deletion (#3873) 2020-05-01 23:29:28 +02:00
Isaac Hess
5a85e8d749 satellite/audit: Fix flaky TestVerifierSlowDownload
TestVerifierSlowDownload would sometimes not have enough nodes finish in
the allotted deadline period. This increases the deadline and also does
not assert that exactly 3 have finished. Instead, in keeping with the
purpose of the test, it asserts that the slow download is never counted
as a success and is always counted as a pending audit in the final
report.

Change-Id: I180734fcc4a499420c75164bad6253ed155d87de
2020-04-30 15:58:47 -06:00
Egon Elbre
418a4cc926 satellite/console: fix flaky TestProjectsList
time.Now can return the exact time, ensure that it uses a later
timestamp in the listing.

Change-Id: I9bcbdf3d222804d949d3899cf3d1575f4de93f06
2020-04-30 13:48:29 +03:00
Egon Elbre
8928399d02 all: rename CreateTables to MigrateToLatest
CreateTables hasn't been quite true for a while now, rename to
MigrateToLatest to be clearer in it's behavior.

Change-Id: Ida48e95122a5d9b7a814e922d3698e00024a2ba7
2020-04-30 07:21:17 +00:00
Jessica Grebenschikov
6a6427526b satellite/overlay: remove old updateaddress method
The UpdateAddress method use to be used when storage node's checked in with the Satellite, but once the contact service was created this method was no longer used. This PR finally removes it.

Change-Id: Ib3f83c8003269671d97d54f21ee69665fa663f24
2020-04-30 06:41:48 +00:00
Egon Elbre
e26a5598ec satellite/overlay: fix UpdateCheckIn test
time.Now() in a short amount of time can return the exact same value.
Ensure that the test uses times that are distinct.

Change-Id: Ia653ce0af4bfcf7b5da133a9cf98b823033d9592
2020-04-29 17:59:00 +00:00
Isaac Hess
237d9da477 storagenode/pieces: Deleter can handle multiple tests
Before the deleter would close its done channel once, so if additional
tests shared a storagenode, even if not in parallel, the later waits
would not work properly. This fixes that problem.

Change-Id: I7dcacf6699cef7c2c2948ba0f4369ef520601bf5
2020-04-29 11:26:56 -06:00
Yingrong Zhao
518946fab9 satellite/metainfo/piecedeletion: add metrics for unhandled pieces
Change-Id: I0cd66e09a8de7c7c0a708b2a9fe44ed1739770b0
2020-04-29 13:30:51 +00:00
Egon Elbre
70fe974a95 satellite/payments/stripecoinpayments: fix test flakyness
CustomerRepositoryList relied on created at time for the output sorting,
however the granularity of the timer may cause multiple customers
having the same "created at" time.

Add a sleep so that every customer does get a unique time.

Change-Id: If2923174f304fa5f41260d500f6139e3fa7c3ba5
2020-04-29 12:06:35 +00:00
Egon Elbre
588826157b satellite/metainfo: reduce RateLimit test flakyness
Delay of 100ms could happen due to other things happening on the test
server. Increase the time to 1s.

Change-Id: I2c7c21f966101771633d73e84cf9850d28089e71
2020-04-29 14:20:02 +03:00
Egon Elbre
621b4e7f92 satellite: remove non-important debug lines
Change-Id: I9a0cbfbcfa6c9f8521c1367ea95a4d696d0b237b
2020-04-29 10:22:46 +00:00
Yingrong Zhao
004d82987c satellite/metainfo: fix TestEndpoint_DeleteObjectPieces_ObjectWithoutLastSegment test
Change-Id: I0dc20f74bc214aab8bf8f96443381de5f7a435d9
2020-04-29 09:18:44 +00:00
Moby von Briesen
de366537a8 satellite/satellitedb/overlaycache: fix behavior around gracefully exited nodes
Sometimes nodes who have gracefully exited will still be holding pieces
according to the satellite. This has some unintended side effects
currently, such as nodes getting disqualified after having successfully
exited.
* When the audit reporter attempts to update node stats, do not update
stats (alpha, beta, suspension, disqualification) if the node has
finished graceful exit (audit/reporter_test.go TestGracefullyExitedNotUpdated)
* Treat gracefully exited nodes as "not reputable" so that the repairer
and checker do not count them as healthy (overlay/statdb_test.go
TestKnownUnreliableOrOffline, repair/repair_test.go
TestRepairGracefullyExited)

Change-Id: I1920d60dd35de5b2385a9b06989397628a2f1272
2020-04-28 23:58:43 +00:00
Isaac Hess
baccfd36b1 private/testplanet: Mark sn peer deleter test mode
When running testplanet tests, mark storagenode peer PieceDeleter as in
testing mode so that you don't have to do it on each test.

Change-Id: I2592e02c63f8bcc9152ecf436bac4e798b08bccf
2020-04-28 15:57:29 -06:00
Jeff Wendling
42f63c6538 satellite/compensation: add offline status tracking
Change-Id: I52e615d3db186416ee95029dc72df626f0e69ad7
2020-04-28 19:35:59 +00:00
Egon Elbre
cafa7a5f0b satellite/admin: add a readme about endpoints
Change-Id: I720fcdf6af3860ad4c8566de85c39e9f06f7cb01
2020-04-28 19:13:45 +00:00
Egon Elbre
85c45cd56f private/dbutil/pgtest: support multiple databases for testing
Currently Cockroach isn't performant for concurrent database setup and
tear-down. Instead of a single instance allow setting multiple potential
connection strings and let the tests pick one connection string
randomly.

This improves test duration by ~10 minutes.

While we are at significantly changing how pgtest works, introduce
helper PickPostgres and PickCockroach for selecting the database to
reduce code duplications in multiple places.

Change-Id: I8ad171d5c4c8a4fc081ec2ae9bdd0cc948a80619
2020-04-28 21:55:49 +03:00
Natalie Villasana
6f84be133a satellite/metainfo: add MigrateToLatest to PointerDB
In cases like the segment reaper script connecting to the metainfodb,
we don't want a db migration to happen automatically when we call
metainfo.NewStore. This adds MigrateToLatest method for postgreskv
and cockroackv, and calls MigrateToLatest in places where NewStore used
to create tables.

Change-Id: I682d0f26d609af0601dfdb32a24866cdf5d32a7e
2020-04-28 17:26:35 +00:00
Isaac Hess
13bf0c62ab satellite/pieces: Fix race in piece deleter
There was a race in the test code for piece deleter, which made it
possible to broadcast on the condition variable before anyone was
waiting. This change fixes that and has Wait take a context so it times
out with the context.

Change-Id: Ia4f77a7b7d2287d5ab1d7ba541caeb1ba036dba3
2020-04-28 10:50:20 -06:00
Egon Elbre
ef913be234 satellite/satellitedb/satellitedbtest: don't use subtest naming
A/B indicates that B is a subtest of A, however in this case they
represent a configuration of the test, not a subtest.

Change-Id: I64eed5d5bcb12759e54fe4b5373f8e88488e50f7
2020-04-27 19:32:09 +03:00
Ivan Fraixedes
03871d17c3 satellite/satellitedb: Update ticket ref
Update a reference to a ticket in a code comment.

Change-Id: Ib82220e94527482c5ca1a58d8614b919d1113ab5
2020-04-27 08:50:41 +00:00
Stefan Benten
d73630fd4a
satellite/satellitedb: Ensure we just return bucket usage for buckets that exist (#3863) 2020-04-24 22:25:16 +02:00
Bill Thorp
341aecfe0f satellite/console: add rate limiter to login, register, password recovery
Added a per IP rate limiter to the console web.
Cleaned up password check to leak less bcyrpt info.

Change-Id: I3c882978bd8de3ee9428cb6434a41ab2fc405fb2
2020-04-24 17:15:49 +00:00
Jess G
825226c98e
satellite/overlay: use node selection cache for uploads (#3859)
* satellite/overlay: use node selection cache for uploads

Change-Id: Ibd16cccee979d0544f2f4a01749af9f36f02a6ad

* fix config lock

Change-Id: Idd307e4dee8ab92749f1ec3f996419ea0af829fd

* start fixing tests

Change-Id: I207d373a3b2a2d9312c9e72fe9bd0b01e06ad6cf

* fix test, add some more

Change-Id: I82b99c2004fca2510965f9b389f87dd4474bc722

* change config name

Change-Id: I0c0f7fc726b2565dc3828cb723f5459a940f2a0b

* add benchmarks

Change-Id: I05fa25bff8d5b65f94d918556855b95163d002e9

* revert bench to put in different PR

Change-Id: I0f6942296895594768f19614bd7b2e3b9b106ade

* add staleness to benchmark

Change-Id: Ia80a310623d5a342afa6d835402170b531b0f870

* add cache config to testplanet

Change-Id: I39abdab8cc442694da543115a9e470b2a8a25dff

* have repair select old way

Change-Id: I25a938457d7d1bcf89fd15130cb6b0ac19585252

* lower testplante config time

Change-Id: Ib56a2ed086c06bc6061388d15a10a2526a663af7

* fix test

Change-Id: I3868e9cacde2dfbf9c407afab04dc5fc2f286f69
2020-04-24 09:11:04 -07:00
Jess G
7a4dcd61f7
satellite/overlay: add changes to selected node benchmarks (#3862)
* add changes to selected node benchmarks

Change-Id: I0259af155f9151cc2c7830d10f8907634c5e494f

* fix lint

Change-Id: I6c7b82bbfa579b468712f90fc03b12a931874a54

* restart jenkins

Change-Id: I1d7300343e94e695cd1c93a3b59895f52bbcb11e
2020-04-23 15:30:50 -07:00
Isaac Hess
a785d37157 storagenode/pieces: Process deletes asynchronously
To improve delete performance, we want to process deletes asynchronously
once the message has been received from the satellite. This change makes
it so that storagenodes will send the delete request to a piece Deleter,
which will process a "best-effort" delete asynchronously and return a
success message to the satellite.

There is a configurable number of max delete workers and a max delete
queue size.

Change-Id: I016b68031f9065a9b09224f161b6783e18cf21e5
2020-04-23 11:51:19 -06:00
Moby von Briesen
720e26d235 satellite/satellitedb/overlaycache: update unknown alpha/beta values properly
Update unknown_audit_reputation_alpha and unknown_audit_reputation_beta.
Add test to verify that BatchUpdateStats properly modifies unknown audit
alpha/beta

Change-Id: I0d5f9cac96a99f64905cf575b772402db0756a9d
2020-04-23 10:40:53 -04:00
Moby von Briesen
72b93f3120 satellite/satellitedb: disqualify suspended nodes when the grace period passes
If a node is suspended and receives an unknown or failing audit,
disqualify them if the grace period (default 1w in production) has
passed.

Migrate the nodes table so any node that is currently suspended gets
unsuspended when the satellite starts up.

Change-Id: I7b81c68026f823417faa0bf5e5cb5e67c7156b82
2020-04-22 15:45:00 -04:00
Egon Elbre
676f3e8516 satellite/metainfo/piecedeletion: try to make batches larger
Currently it was possible that PopAll returns 1010 items, then
makes one RPC call with 1000 items, then RPC call 10 items. Meanwhile,
there have been added 500 new items added to the queue.

This change ensures that we pull items from the queue early and
try to make rpc batches as large as possible.

Change-Id: I1a30dde9164c2ff7b90c906a9544593c4f1cf0e9
2020-04-22 18:43:29 +00:00
Ethan Adams
60e07f0a8b Revert "satellite/accounting: Remove unnecessary index bucket_bandwidth_rollups_project_id_action_interval_index"
This reverts commit 105dc7acc6.

Reason for revert: Recent changes to the Postgres query plan seems to want to use this index now.  Reverting until we have time to analyze what's happening.

Change-Id: I74b4b5a8f15c3850d8a958a29f51dbc80e7c282c
2020-04-22 14:49:04 +00:00
Moby von Briesen
178aa8b5e0 satellite/{metainfo,repair}: Delete expired segments from metainfo
* Delete expired segments in expired segments service using metainfo
loop
* Add test to verify expired segments service deletes expired segments
* Ignore expired segments in checker observer
* Modify checker tests to verify that expired segments are ignored
* Ignore expired segments in segment repairer and drop from repair queue
* Add repair test to verify that a segment that expires after being
added to the repair queue is ignored and dropped from the repair queue

Change-Id: Ib2b0934db525fef58325583d2a7ca859b88ea60d
2020-04-22 13:02:31 +00:00
Qweder93
805e328c47 storagenode/heldamount payments removed
Change-Id: I87cc04f43d182a4190a571ef417be85d02db9d34
2020-04-21 17:15:31 +00:00
Ethan
105dc7acc6 satellite/accounting: Remove unnecessary index bucket_bandwidth_rollups_project_id_action_interval_index
See https://storjlabs.atlassian.net/browse/SM-738

Change-Id: I9ba3cc3fbff9f13fc0b95d25feee5a19e5a5c486
2020-04-21 16:43:09 +00:00
Qweder93
e999f24e54 storagenode/nodestats/cache: storagenodeDB/heldamount sync with satelliteDB/storagenode_paystub
Change-Id: If894166809bee8a5e036e618005d8141c2a0c594
2020-04-20 19:12:17 +00:00
Michal Niewrzal
c021b35879 private/testplanet: migrate testplanet to new libuplink
Replace most of old libuplink usages in testplanet. 100% migration will
be possible when we will be able to implement UploadWithClientConfig
with new libuplink.

Change-Id: I432d7d4917c7b67d46a058abd0a2a6a13f565ac4
2020-04-20 12:43:34 +00:00
Qweder93
6e3585e394 satellite/heldamount/endpoint : GetAllPaystubs added
Change-Id: Ic8cdd9db8b2a68796f9579c7fed2d49d9054bd64
2020-04-19 19:21:54 +03:00
Ethan
4cd86ff780 satellite/accounting: Add index on bucket_bandwidth_rollups for action, interval_start, and project_id
See https://storjlabs.atlassian.net/browse/SM-551 for details

Change-Id: I104c4e87d5aef500cc4a3893817763808f76c484
2020-04-17 19:14:45 +00:00
Egon Elbre
7e0e74c65c satellite/metainfo: attribution based on useragent
Automatically attach attribution information to bucket during
BeginObject or CreateBucket when the UserAgent is set.

Change-Id: I405cb26c5a2f7394b30e3f2cf5d2214c8781eb8b
2020-04-17 10:47:14 +03:00
Qweder93
c498e52cf2 satellite/nodestats/endoint: reputation stats updated
Change-Id: I03a44c4c3465d4f8f23fcbbc742148f5e49544e2
2020-04-16 22:28:58 +03:00
Egon Elbre
9052085f70 private/testplanet: simplify uplink usage
Change-Id: I3e488dc296f1094ce95e6d6597ca6d3f8da90a76
2020-04-16 16:45:55 +00:00
Egon Elbre
a129a8bd35 all: separate err check for http
We want to avoid net/http dependency in errs2 package, hence we removed
http.ErrServerClosed from IgnoreCanceled and IsCanceled check. Now we
need to add that check explicitly to every http endpoint.

Change-Id: I62b1cc0a0a2d3b43301d713a7951e5022145f88f
2020-04-16 18:50:24 +03:00
VitaliiShpital
2dce4c232c web/satellite: redirect to verification page on sign up if inside iframe
Change-Id: I606b63fd27bef46597697b491970523e8a3a0cae
2020-04-16 13:35:49 +00:00
Egon Elbre
93e39a6b33 satellite/metainfo: support co-joining the loop
This adds support for observers to join the loop together. This allows
to ensure that when multiple observers join, they will be part of the
same loop iteration.

Change-Id: Ie887d4cedfb074b65c782690a2c09c1704f56dfe
2020-04-16 13:31:17 +03:00
Egon Elbre
45d1ca87f5 satellite/metainfo/piecedeletion: handle zero case
During testing it's possible to get into a scenario where all nodes are
offline and list of requests is empty.

Change-Id: I271c0ca2c72009244df13e8bc1441fcd5f3da9e0
2020-04-16 13:00:32 +03:00
Jess G
75b9a5971e
satellite: update log levels (#3851)
* satellite: update log levels

Change-Id: I86bc32e042d742af6dbc469a294291a2e667e81f

* log version on start up for every service

Change-Id: Ic128bb9c5ac52d4dc6d6c4cb3059fbad73f5d3de

* Use monkit for tracking failed ip resolutions

Change-Id: Ia5aa71d315515e0c5f62c98d9d115ef984cd50c2

* fix compile errors

Change-Id: Ia33c8b6e34e780bd1115120dc347a439d99e83bf

* add request limit value to storage node rpc err

Change-Id: I1ad6706a60237928e29da300d96a1bafa94156e5

* we cant track storage node ids in monkit metrics so lets use logging to track that for expired orders

Change-Id: I1cc1d240b29019ae2f8c774792765df3cbeac887

* fix build errs

Change-Id: I6d0ffe058e9a38b7ed031c85a29440f3d68e8d47
2020-04-15 12:32:22 -07:00
Egon Elbre
d3ce845f82 satellite: revert log lines used to figure out node id
Currently storj-sim relies on the log lines to be exactly the same,
when they change it cannot find the necessary information from log.

Change-Id: Ia039915ef3375a7cf60f107b2c05c958de15b6d5
2020-04-15 17:07:56 +03:00
Kaloyan Raev
a2ce836761 remove sugar logging
Change-Id: I6b6ca9704837cb3f5f5449ba7f55661487814d9f
2020-04-15 12:37:47 +00:00
VitaliiShpital
158013a866 satellite/console: redirect on account activation
Change-Id: I2506ce0fd3832bf46fbcdcc5a42bb83dc926e99a
2020-04-15 11:49:50 +00:00
Egon Elbre
2c0d61b18e satellite/metainfo: avoid temporary list
Currently ListV2 loaded the whole data into memory, even when all the
data wasn't being used, using up more memory than needed.

Change-Id: I5846d979344729b447c108a6cc9f4227229ec981
2020-04-15 08:01:42 +00:00
Jess G
5ea1602ca5
satellite/overlay: add selected node cache (#3846)
* init implementation cache

Change-Id: Ia54a1943e0707a77189bc5f4a9aaa8339c98d99a

* one query to init cache

Change-Id: I7c04b3ae104b553ae23fca372351a4328f632c66

* add monit tracking of cache

Change-Id: I7d209e12c8f32d43708b23bf2126c5d5098e0a07

* add first test

Change-Id: I0646a9349d457a9eb3920f7cd2d62fb72ffc3ab5

* add staleness to cache

Change-Id: If002329bfdd53a4b200ad14dbd2ffc8b280aedb8

* add init test

Change-Id: I3a3d0aa74cfac1d125fa93cb749316ed2a74d5b1

* fix comment

Change-Id: I73353d00ccf0952b38c0f8ef7d1755c15cbfe9d9

* mv to nodeselection pkg

Change-Id: I62487f768296c7a7b597fa398a4c42daf6e9c5b7

* add state to cache

Change-Id: I081e77ec0e16706faee1a267de9a7fa643d6ac11

* add refresh concurrent test

Change-Id: Idcba72508291099f280edc65355273c0acc3d3ce

* add a few more tests

Change-Id: I9422e9eaa22bf01c11f14bdb892ebcf7b3e5e5fb

* fix tests, add min version to select allnodes

Change-Id: I926f41d568951ad4ff70c6d4ceb87abb1e3e5009

* update comments

Change-Id: I6ffe33e245ca65fb523c880cd72e63ce35776eb9

* fixes and rm Init

Change-Id: Ifbe09b668978b5d9af09ca38cb080d02a2154cf4

* fix format

Change-Id: I03cc217e28dc1839190c5c6dbdbb602c132a5a38
2020-04-14 13:50:02 -07:00
Moby von Briesen
d7794a4851 satellite/overlay: hardcode default values for audit alpha/beta
Alpha=1 and beta=0 are the expected first values for any alpha/beta
reputation system we are using in the codebase. So we are removing the
configurability of these values.

Change-Id: Ic61861b8ea5047fa1438ea6609b1d0048bf0abc3
2020-04-14 19:12:40 +00:00
Cameron Ayer
3ee6c14f54 satellite/downtime: add concurrency to downtime estimation
We want to increase our throughput for downtime estimation. This commit
adds the ability to reach out to multiple nodes concurrently for downtime
estimation. The number of concurrent routines is determined by a new config
flag, EstimationConcurrencyLimit. It also increases the default
EstimationBatchSize to 1000.

Change-Id: I800ce7ec1035885afa194c3c3f64eedd4f6f61eb
2020-04-14 14:39:13 +00:00
Egon Elbre
c97131ae78 satellite/metainfo: organize attribution methods
Change-Id: I4f35599c3f923861b5b05b128bf904480679f5f9
2020-04-14 11:27:43 +03:00
Jeff Wendling
83bbc7a37e satellite/accounting/tally: ignore expired pointers
until we do a good job of cleaning them up, we should at least
not charge or pay people for them. nodes already locally delete
expired segments.

subsumes the tests in 1112.

Change-Id: I5961185764e02f6136b3231b44ecc75a9a8832c9
2020-04-10 11:36:00 -06:00
Cameron Ayer
02613407ae satellite/satellitedb: only suspend node if not already suspended
Whenever the node's reputation is updated, if its unknown audit
reputation is below the suspension threshold, its suspension field
is set to the current time. This could overwrite the previous
"suspendedAt" value resulting a node that never reaches the end of
its suspension.

Also log whenever a node is disqualified or its suspension status
changes

Change-Id: I5e8c8f1c46f66d79cb279b5b16a84fe03f533deb
2020-04-10 09:37:37 +00:00
Egon Elbre
d86cce202c satellite/satellitedb: use arrays for arguments in node selection
This simplifies the code and makes queries faster:

name                               old time/op  new time/op  delta
SelectStorageNodes-32              7.72ms ± 6%  7.22ms ± 3%  -6.44%  (p=0.016 n=5+5)
SelectNewStorageNodes-32           7.75ms ± 2%  7.37ms ± 1%  -4.89%  (p=0.008 n=5+5)
SelectStorageNodesExclusion-32     16.9ms ± 0%  16.6ms ± 0%  -2.15%  (p=0.008 n=5+5)
SelectNewStorageNodesExclusion-32  17.2ms ± 0%  16.6ms ± 2%  -3.69%  (p=0.008 n=5+5)
FindStorageNodes-32                45.5ms ± 0%  45.1ms ± 1%    ~     (p=0.056 n=5+5)
FindStorageNodesExclusion-32       77.4ms ± 0%  75.9ms ± 0%  -1.91%  (p=0.008 n=5+5)

Change-Id: I38f77f6282b9738e8416113d42c6acb46c03da7b
2020-04-09 21:16:10 +03:00
Egon Elbre
ccf4f9ed2d satellite/satellitedb: node selection code cleanup
Reduce the number of non-methods to reduce funcs in the namespace also
combine a func to slightly condense the code more.

Change-Id: Ifbe728eb8c8ca4c981df648decd259c2097b6b40
2020-04-09 20:41:29 +03:00
Natalie Villasana
cf80b3caf3
satellite/overlay: combine SelectStorageNodes and SelectNewStorageNodes (#3831) 2020-04-09 11:19:44 -04:00
Moby von Briesen
14b3704f56 storagenode: add suspended status to storagenode dashboard/api
* Add migration to storagenode reputation table to add suspended
timestamp
* Send suspended info to storagenode from satellite nodestats endpoint
* Add suspended status to storagenode api
* Add an indicator on the storagenode dashboard informing operator of
the satellites the node is suspended on

Change-Id: Ie3669f6069cc0258ba76ec99d17006e1b5fd9c8a
2020-04-09 13:36:23 +00:00
Michal Niewrzal
f36e8548f1 satellite/metainfo: adjust max inline segment size validation to
potential encryption overhead.

This is the same approach we have for validating remote segment size.

https://storjlabs.atlassian.net/browse/USR-619

Change-Id: I2597ee734313a3068fd986001680bbedbf1bed2a
2020-04-09 12:34:10 +00:00
Egon Elbre
11a44cdd88 all: don't depend on gogo/proto directly
Change-Id: I8822dea0d1b7b99e0b828e0373a0308a42dde2be
2020-04-08 17:32:15 +00:00
Natalie Villasana
0a1bbc9824 satellite/overlay: add TestEnsureMinimumRequested
Adds a test to make sure that the correct amount of total nodes,
reputable nodes, and new nodes are returned by SelectStorageNodes
in different cases.

Change-Id: I0939159600afde8a46c35735f1edf0576fcdb4cd
2020-04-08 16:06:25 +00:00
Qweder93
3a9422cc9a satellite/nodestats: add pricing model to endpoint
Change-Id: Iddace8e437216a343458f440b543cee61164f233
2020-04-08 14:29:51 +00:00
Egon Elbre
cf26951a5b satellite/satellitedb/pbold: remove dead code
Change-Id: I7464773c20b8f99a601ca9cc4bee804f1ac14cf9
2020-04-08 15:22:31 +03:00