Commit Graph

848 Commits

Author SHA1 Message Date
Egon Elbre
cf7b22c466 satellite/satellitedb: add missing err check
Change-Id: I502838f78f1871315597b488602c0f1112612981
2020-01-19 19:24:12 +00:00
Yaroslav
48303bc0a9 satellite/console/limits: add content-type to response, fix error
Change-Id: I63dffb5372456569b100f97b04eacc1fc19bddf5
2020-01-19 15:42:55 +00:00
Egon Elbre
c207cd08fc satellite/satellitedb: gracefulexit, add missing Errs check
Change-Id: Iba4ba84fd57b3a0a0d15f13006566076045d6c11
2020-01-19 15:24:12 +00:00
Egon Elbre
1abfe42142 satellite: use tagsql
Change-Id: I2170dee409fb0c2fe85913ddd36e7811a3b853ed
2020-01-19 14:39:16 +02:00
Egon Elbre
59d06644b9 private/migrate: switch to tagsql
Also added temporary types withRebind and withTagTx,
which will be later removed. Currently they help to avoid
changing the whole codebase at the same time.

Change-Id: I7f07ba8f4709a23a463bfa67464628665a05808f
2020-01-19 14:39:16 +02:00
stefanbenten
f4097d518c satellite: reduce logging of node status
Change-Id: I6618cf4bf31b856acd7a28b54011a943c03ab22a
2020-01-18 17:47:59 +00:00
Yaroslav
d8368d0b30 satellite/payments: coinpayments add completed status, treat received status as pending, add balance for completed transactions only
Change-Id: I20494bdddfda6d4f37ba2c5b6f7955cd29a6d798
2020-01-17 17:26:34 +00:00
Bill Thorp
0c660f5490 satellite/payments: fixed test so that it passes on non-UTC systems, simplified date comparison
Change-Id: I5e1b31efc148701b9d4918bf9870fb72fbca75ac
2020-01-17 14:14:13 +00:00
Jessica Grebenschikov
955abd9293 satellite/satellitedb/orders: add multi row upserts to process orders
Change-Id: I00d8b55ee74b443fb328bd3a4378308cefa368e4
2020-01-16 23:51:46 +00:00
Stefan Benten
409d4123bb
Add proper Pathdata Index (#3750) 2020-01-17 00:48:59 +01:00
Isaac Hess
cd48dc369a satellite/satellitedb: Remove unused indexes
Change-Id: I875b94574eacf9d2df537bcf1f42f30e0bf60ab9
2020-01-16 16:06:21 -07:00
Jeff Wendling
47bb7a7a86 satellite/satellitedb/dbx: regenerate with default support
Change-Id: I0dab34f27af913795ef95ef92173844c2f53b29b
2020-01-16 22:13:57 +00:00
Jeff Wendling
696d98a232 satellite/satellitedb: fix nitpicks and timestamp issue found in review
warning: databases migrated to version 77 before this commit
is merged must be manually re-migrated. this should not be a
problem for anything but staging databases.

Change-Id: Ie1631c48379472352014183ee43f1465e22200f7
2020-01-16 21:22:38 +00:00
Cameron Ayer
4424697d7f satellite/accounting: refactor live accounting to hold current estimated totals
live accounting used to be a cache to store writes before they are picked up during
the tally iteration, after which the cache is cleared. This created a window in which
users could potentially exceed the storage limit. This PR refactors live accounting to
hold current estimations of space used per project. This should also reduce DB load
since we no longer need to query the satellite DB when checking space used for limiting.

The mechanism by which the new live accounting system works is as follows:

During the upload of any segment, the size of that segment is added to its respective
project total in live accounting. At the beginning of the tally iteration we record
the current values in live accounting as `initialLiveTotals`. At the end of the tally
iteration we again record the current totals in live accounting as `latestLiveTotals`.
The metainfo loop observer in tally allows us to get the project totals from what it
observed in metainfo DB which are stored in `tallyProjectTotals`. However, for any
particular segment uploaded during the metainfo loop, the observer may or may not
have seen it. Thus, we take half of the difference between `latestLiveTotals` and
`initialLiveTotals`, and add that to the total that was found during tally and set that
as the new live accounting total.

Initially, live accounting was storing the total stored amount across all nodes rather than
the segment size, which is inconsistent with how we record amounts stored in the project
accounting DB, so we have refactored live accounting to record segment size

Change-Id: Ie48bfdef453428fcdc180b2d781a69d58fd927fb
2020-01-16 10:26:49 -05:00
Egon Elbre
7d79aab14e satellite/satellitedb: fixes to row handling
Change-Id: I48fae692bcca152143a12f333296c42471538850
2020-01-16 17:07:26 +02:00
Jeff Wendling
f42851b1ab satellite/satellitedb: remove the big honkin mutex
no longer necessary/desired with reported_serials.

Change-Id: I69b5c535488eb5f98b250d73a7c8e6deaed0254e
2020-01-15 19:24:35 -07:00
Jeff Wendling
78c6d5bb32 satellite/satellitedb: reported_serials table for processing orders
this commit introduces the reported_serials table. its purpose is
to allow for blind writes into it as nodes report in so that we have
minimal contention. in order to continue to accurately account for
used bandwidth, though, we cannot immediately add the settled amount.
if we did, we would have to give up on blind writes.

the table's primary key is structured precisely so that we can quickly
find expired orders and so that we maximally benefit from rocksdb
path prefix compression. we do this by rounding the expires at time
forward to the next day, effectively giving us storagenode petnames
for free. and since there's no secondary index or foreign key
constraints, this design should use significantly less space than
the current used_serials table while also reducing contention.

after inserting the orders into the table, we have a chore that
periodically consumes all of the expired orders in it and inserts
them into the existing rollups tables. this is as if we changed
the nodes to report as the order expired rather than as soon as
possible, so the belief in correctness of the refactor is higher.

since we are able to process large batches of orders (typically
a day's worth), we can use the code to maximally batch inserts into
the rollup tables to make inserts as friendly as possible to
cockroach.

Change-Id: I25d609ca2679b8331979184f16c6d46d4f74c1a6
2020-01-15 19:21:21 -07:00
Jeff Wendling
9da16b1d9e satellite/satellitedb/dbx: name the package dbx
everyone was importing it as dbx anyway. why should it be
named satellitedb? so yeah just pass the "-p dbx" flag.

Change-Id: I5efa669f4f00f196b38a9acd0d402009475a936f
2020-01-15 15:16:39 -07:00
Yingrong Zhao
db8aee0806 satellite/contact; storagenode/preflight: add clock check on startup for storagenode
add config preflight.enabled-local-time

Change-Id: I7b942c9bee063aae409ee6721ae9d079dff0144f
2020-01-15 15:35:26 +00:00
Ivan Fraixedes
6dc948da46
satellite/metainfo: Create service for deleting pieces
Create a service for deleting pieces of storage nodes.

Currently the DeletePieces method returns after a success threshold,
completion or a timeout.

The end goal is to return when reaching the success threshold and
leaving the remaining goroutines running after DeletePieces method
returns and add a life cycle to the service that it waits for them when
it closes.

This is the first commit for ticket:
https://storjlabs.atlassian.net/browse/V3-3476

Change-Id: If740bbf57c741f880449980b8176b036dd956c7b
2020-01-15 15:24:20 +01:00
Egon Elbre
02ed2b5a19 satellite/metainfo: disallow cancelling deletion request
Change-Id: I05194d493632c9f454671ddf6b67b4901f2b7884
2020-01-15 14:23:46 +02:00
Egon Elbre
64fb2d3d2f Revert "dbutil: statically require all databases accesses to use contexts"
This reverts commit 8e242cd012.

Revert because lib/pq has known issues with context cancellation.
These issues need to be resolved before these changes can be merged.

Change-Id: I160af51dbc2d67c5449aafa406a403e5367bb555
2020-01-15 07:28:00 +00:00
JT Olio
8e242cd012 dbutil: statically require all databases accesses to use contexts
this will allow for some nice runtime analysis down the road.
also, this allows for wrapping database handles in a way that
can interact with these contexts

requires https://review.dev.storj.io/c/storj/dbx/+/514

Change-Id: Ib087b7cd73296dd2c1e0331314da34d861f61d2b
2020-01-14 18:20:47 -05:00
crawter
41d5e86306 satellite/payments: coupon addition removed
Change-Id: I92781d9133603fdefd58b19a6f0ac6b1c6df3ac6
2020-01-14 16:24:48 +00:00
NikolaiYurchenko
224025d714 web/satellite: password recovery page title fixed
Change-Id: I4e070315ec2e5f0b7ff0140f93dd6aacf1e68112
2020-01-14 13:34:54 +00:00
crawter
a57ce18f58 satellite/payments: coupons, coupons usage, invoice generation with pricing model applied
Change-Id: Ic5d5a2fc116388647efe46896cfccc2038c77537
2020-01-14 12:45:00 +00:00
Yingrong Zhao
ee87846f0b satellite/contact: add placeholder for GetTime endpoint
Change-Id: I42f8479708f0558350c2280a398d84d145e8118f
2020-01-14 06:38:47 +00:00
Jeff Wendling
3b99f03780 satellite/orders: add monitoring to bucket bandwidth cache operations
Change-Id: Ib14303fc9f97a133410e2d6e2cf532e468b3dcee
2020-01-13 17:36:40 -07:00
Isaac Hess
4950d7106a satellite/orders: Add write cache for bw rollups
Change-Id: I8ba454cb2ab4742cafd6ed09120e4240874831fc
2020-01-13 22:40:51 +00:00
JT Olio
e1ba3931ec postgres2: use cockroachkv impl against postgres
this allows for setting $STORJ_METAINFO_POSTGRESQL_USE_ALT=yes if you
want to use the cockroachkv implementation for metainfo against postgres

Change-Id: I0c9458c83fd67ee63ef4a78351e64a80a0647408
2020-01-13 14:51:56 -06:00
Jeff Wendling
71ec0ad374 satellite/satellitedb: add big honkin mutex to ProcessOrders
the hope is that it is mostly interfering with itself, so this
will make it not do that (well, N api servers, but hopefully
that's not enough to cause it to have issues).

Change-Id: Ifd0c9e6617457785ab25fe5b714d8556cdc8e2d3
2020-01-13 11:33:12 -07:00
Egon Elbre
ff267168c5 private/migrate: add ctx argument
Change-Id: I3d65912d89261386413c494c7ed1576fed4dcaf4
2020-01-13 15:52:26 +02:00
Egon Elbre
24958bd7d3 satellite: add ctx to DB.CreateTables
Change-Id: I9ecad624cf5a7fc9c86bb91c68f96a3a4efd2e92
2020-01-13 15:31:09 +02:00
Egon Elbre
0835b9024c private/dbutil/pgutil: add ctx argument
Change-Id: Icfd56ca8c1f831ad56c0195a0b883e8f0618daaf
2020-01-13 15:27:06 +02:00
littleskunk
bcc23f6869
Satellite/orders: remove allocated bandwith from storagenode_bandwidth_rollups
When an uplink requests an upload or download from the satellite we are trackig the
allocated bandwidth twice. The value in bucket_bandwidth_rollups is used
for project limits but the value in storagenode_bandwidth_rollups is not
used at all. We can increase the performance by removing it. Uplinks
will get a faster response from the satellite.

Change-Id: Icccd41f94107ef34668f30f99bf5f728c384b07e
2020-01-12 16:20:47 +01:00
Jeff Wendling
4aef0e3823 satellite/satellitedb: only reject orders if row not found
any database error doesn't mean the order wasn't found. for example
in cockroach it may say that the transaction is aborted. then what?
maybe we get big old row level deadlocks like we've observed? so
instead explicitly check for ErrNoRows to reject the order and bail
out otherwise. the surrounding logic will give it a retry.

Change-Id: I6e1f8f6e6a6def3e45b44f5088cbdc158e1098e4
2020-01-10 19:05:54 -07:00
Jeff Wendling
77fd41a02e satellite: add an expiring lru cache around api keys
Change-Id: I995429c66affd33da59b091f28f09ca122070b5e
2020-01-09 22:13:41 -07:00
Natalie Ventura Villasana
6b1829f3c3
satellite/downtime: new chore estimates downtime
Adds EstimationChore to the downtime package, which is an
independent chore that finds offline nodes given a configurable
limit, then uptime checks those nodes, and sets a last contact
success or failure given a response. For failed nodes, the chore
updates the amount of downtime the node has been offline in the
DowntimeTracking table.

Design doc section: https://github.com/storj/storj/blob/master/docs/blueprints/storage-node-downtime-tracking.md#estimating-offline-time
Jira: https://storjlabs.atlassian.net/browse/V3-2545

Change-Id: I60af95803930bf9b33232b248bb20cca6f0e0b5f
2020-01-09 15:05:13 -05:00
VitaliiShpital
a4e5c18877 satellite/payments: mock methods added to endpoint to match pb PaymentsServer
Change-Id: Ic8ff44cbe0b2368021a5d83cf86ce0dd2b670fd7
2020-01-09 14:30:21 +02:00
Egon Elbre
d3d75a597f satellite,storage: clean global ctx usage in tests
Change-Id: I89ea5c95fc6895518b464f8eb6a4c74c6ae37651
2020-01-09 10:37:21 +00:00
Ivan Fraixedes
5cd6058318 satellite/metainfo: Add back-pressure mechanism DeleteObjectPieces
Add a back-pressure mechanism to the satellite metainfo
DeleteObjectPieces method for returning once the 75% of successful
deleted pieces is reached.

Change-Id: Ia38df49fba5838f0605c40a77cfff8e3442cb5b0
2020-01-09 08:22:11 +00:00
Yingrong Zhao
76ee8a1b4c satellite: remove UptimeReputation configs from codebase
With the new storage node downtime tracking feature, we need remove current uptime reputation configs: UptimeReputationAlpha, UptimeReputationBeta, and
UptimeReputationDQ. This is the first step of removing the uptime
reputation columns from satellitedb

Change-Id: Ie8fab13295dbf545e33aeda0c4306cda4ba54e36
2020-01-08 18:54:15 +00:00
littleskunk
05e4a86654
satellite/metainfo: Close client DeleteObjectPieces
The DeleteObjectPieces should print out the warning on closing the
connections only if there was an error.

Change-Id: If3d7ab256d8508c08388c1f22c7dd1eb819d2509
2020-01-08 15:54:11 +01:00
Ivan Fraixedes
922c43f921
satellite/metainfo: Close client DeleteObjectPieces
The DeleteObjectPieces must close the storage node client once it has
finished deleting its pieces.

Change-Id: I08eb8af8e4215d77d59b52f5055211b918374ab4
2020-01-08 15:01:34 +01:00
Egon Elbre
082ec81714
uplink: move to storj.io/uplink (#3746) 2020-01-08 15:40:19 +02:00
Jeff Wendling
c740b82e66 satellitedb/dbx: remove sed usage for bash script
turns out portable sed is hard: it has to work with both
linux and bsd sed, etc. instead, use a really really basic
bash script and a temporary file. this should be much less
likely to cause issues on a wide range of machines.

Change-Id: Ia759789fb52aa1ee3361426bb6c02ed4eac3d23a
2020-01-08 01:40:24 +00:00
paul cannon
6b21334c47 satellite/satellitedb: use txutil.ExecuteInTx in dbx WithTx()
Change-Id: I42ec21fdf117c661b3e1687a04014650c3a6ab97
2020-01-07 17:00:08 -06:00
paul cannon
0c5e381434 satellite/console: use transaction helpers in consoledb
Transactions in our code that might need to work against CockroachDB
need to be retried in the event of a retryable error. The transaction
helper functions in dbutil do that automatically. I am changing this
code to use those helpers instead.

I also fleshed out consoledb_test.go to do actual inserts and gets to
make sure things were working correctly.

Change-Id: I089bf4c776d15dc8578080e26760bd6dff4beec9
2020-01-07 17:59:10 +00:00
paul cannon
22b6e9220a satellite/satellitedb: use transaction helpers in irreparabledb
Transactions in our code that might need to work against CockroachDB
need to be retried in the event of a retryable error. The transaction
helper functions in dbutil do that automatically. I am changing this
code to use those helpers instead.

Change-Id: I22b850ce5859fa07d13bf475be5140e6bde95b8a
2020-01-07 17:40:09 +00:00
paul cannon
723ed23298 satellite/satellitedb: use transaction helpers in orders
Transactions in our code that might need to work against CockroachDB
need to be retried in the event of a retryable error. The WithTx
helper functions in dbutil and dbx do that automatically. I am changing
this code to use those helpers instead.

Change-Id: Iaf492af35471931125f2b7365aa4338f44154881
2020-01-07 16:31:47 +00:00