Commit Graph

3463 Commits

Author SHA1 Message Date
stefanbenten
f4097d518c satellite: reduce logging of node status
Change-Id: I6618cf4bf31b856acd7a28b54011a943c03ab22a
2020-01-18 17:47:59 +00:00
Egon Elbre
76a6b28b33 ci: use tmpfs for running tests
Change-Id: I6278899bd158759c6fc0bceafd6a6c7244860f4a
2020-01-17 20:28:15 +00:00
Moby von Briesen
273eb66fae cmd/storagenode,storagenode/preflight: add config flag to disable
storagenode database preflight check.

Disable preflight database check by default, and have the option to
enable it. This will allow us to enable it once it is definitely
working.

Also change the name of the config flag for preflight  time sync.

Change-Id: Ie2e20f9e25dcb38794eafa7e1505e7c6ff287c99
2020-01-17 17:53:17 +00:00
Yaroslav
d8368d0b30 satellite/payments: coinpayments add completed status, treat received status as pending, add balance for completed transactions only
Change-Id: I20494bdddfda6d4f37ba2c5b6f7955cd29a6d798
2020-01-17 17:26:34 +00:00
Isaac Hess
614e04d055 storagenode/pieces: Cache inits trash info from db
On pieces usage cache init we now load the trash info from the db. Also
fixes a test that was masking the failure here.

Change-Id: I9ff7da5bc6c0f74cf0942e20931b40e0c88d70fa
2020-01-17 09:33:05 -07:00
Bill Thorp
6f2f97b313 storagenode\gracefulexit: broke worker deleteOnePieceOrAll into deleteOnePiece and deleteAllPieces and deletePiece
Change-Id: Ic3bd21e89fa71e962c2bb1c4943f4696bc4f83e5
2020-01-17 15:07:34 +00:00
Bill Thorp
0c660f5490 satellite/payments: fixed test so that it passes on non-UTC systems, simplified date comparison
Change-Id: I5e1b31efc148701b9d4918bf9870fb72fbca75ac
2020-01-17 14:14:13 +00:00
Egon Elbre
5d80e22af9 private/tagsql: implement wrapper for sql.DB
Wrapper adds tracing and fixes context usage issues.

Change-Id: Ie6f7650eac87e2a2b64b760198498ba5857ad535
2020-01-17 13:52:12 +00:00
VitaliiShpital
8bbb9083f0 web/satellite: error on logout from team page removed
Change-Id: I11ddf7a596250ba3cd8ea365ea41679ef9f2e470
2020-01-17 13:27:23 +00:00
Michal Niewrzal
22af78b62e cmd/uplink: fix 'must specify access' for old uplink configurations
Change-Id: I73b0483b1907a854048cdb1080a013844fb550f4
2020-01-17 12:02:31 +00:00
igaass
491cd8d8ab
scripts: automated test for testing uplink share command (#3736)
* scripts: automated test for testing uplink share command

* Replace "scope" to "access"

* Remove redundant access flag

* Rename variables
Remove retVal variable
2020-01-17 12:57:38 +02:00
Jessica Grebenschikov
955abd9293 satellite/satellitedb/orders: add multi row upserts to process orders
Change-Id: I00d8b55ee74b443fb328bd3a4378308cefa368e4
2020-01-16 23:51:46 +00:00
Stefan Benten
409d4123bb
Add proper Pathdata Index (#3750) 2020-01-17 00:48:59 +01:00
Isaac Hess
cd48dc369a satellite/satellitedb: Remove unused indexes
Change-Id: I875b94574eacf9d2df537bcf1f42f30e0bf60ab9
2020-01-16 16:06:21 -07:00
Jeff Wendling
47bb7a7a86 satellite/satellitedb/dbx: regenerate with default support
Change-Id: I0dab34f27af913795ef95ef92173844c2f53b29b
2020-01-16 22:13:57 +00:00
Jeff Wendling
696d98a232 satellite/satellitedb: fix nitpicks and timestamp issue found in review
warning: databases migrated to version 77 before this commit
is merged must be manually re-migrated. this should not be a
problem for anything but staging databases.

Change-Id: Ie1631c48379472352014183ee43f1465e22200f7
2020-01-16 21:22:38 +00:00
Moby von Briesen
e115bc1903 cmd/storagenode;storagenode/storagenodedb: add preflight database check
for storagenode

Ensure that database schema matches latest test migration schema before
allowing the node to start up.

Ensure minimal read/write functionality for each storagenode database
before allowing the node to start up.

This will eliminate many unhandled audit errors we are seeing.

Change-Id: Ic0e628b04a9c35b7a8243f6a81d4683918170ba9
2020-01-16 18:44:46 +00:00
littleskunk
b6f1a91c67 scripts/testversions,rollingupgrade: remove encrytion key
Change-Id: I6fd35fa4b29707f53e988bd00d6523b934767ecc
2020-01-16 16:01:34 +00:00
Cameron Ayer
4424697d7f satellite/accounting: refactor live accounting to hold current estimated totals
live accounting used to be a cache to store writes before they are picked up during
the tally iteration, after which the cache is cleared. This created a window in which
users could potentially exceed the storage limit. This PR refactors live accounting to
hold current estimations of space used per project. This should also reduce DB load
since we no longer need to query the satellite DB when checking space used for limiting.

The mechanism by which the new live accounting system works is as follows:

During the upload of any segment, the size of that segment is added to its respective
project total in live accounting. At the beginning of the tally iteration we record
the current values in live accounting as `initialLiveTotals`. At the end of the tally
iteration we again record the current totals in live accounting as `latestLiveTotals`.
The metainfo loop observer in tally allows us to get the project totals from what it
observed in metainfo DB which are stored in `tallyProjectTotals`. However, for any
particular segment uploaded during the metainfo loop, the observer may or may not
have seen it. Thus, we take half of the difference between `latestLiveTotals` and
`initialLiveTotals`, and add that to the total that was found during tally and set that
as the new live accounting total.

Initially, live accounting was storing the total stored amount across all nodes rather than
the segment size, which is inconsistent with how we record amounts stored in the project
accounting DB, so we have refactored live accounting to record segment size

Change-Id: Ie48bfdef453428fcdc180b2d781a69d58fd927fb
2020-01-16 10:26:49 -05:00
Egon Elbre
81d53b8097 storagenode/storagenodedb: fixes to row handling
Change-Id: I3813310b48337428f13678a9fcba5c8a0e0b2b2a
2020-01-16 15:08:37 +00:00
Egon Elbre
7d79aab14e satellite/satellitedb: fixes to row handling
Change-Id: I48fae692bcca152143a12f333296c42471538850
2020-01-16 17:07:26 +02:00
littleskunk
0c365d157f
scripts/testversions: replace apikey with access
Change-Id: I4f899cc49b63b2f04f31a6df478475d7bdbab30d
2020-01-16 14:37:31 +01:00
littleskunk
1e77cb88e7
scripts/rollingupgrade: replace apikey with access
Change-Id: I356587cdc417dabc6f15769592d70269d25051dc
2020-01-16 14:00:33 +01:00
Jeff Wendling
f42851b1ab satellite/satellitedb: remove the big honkin mutex
no longer necessary/desired with reported_serials.

Change-Id: I69b5c535488eb5f98b250d73a7c8e6deaed0254e
2020-01-15 19:24:35 -07:00
Jeff Wendling
78c6d5bb32 satellite/satellitedb: reported_serials table for processing orders
this commit introduces the reported_serials table. its purpose is
to allow for blind writes into it as nodes report in so that we have
minimal contention. in order to continue to accurately account for
used bandwidth, though, we cannot immediately add the settled amount.
if we did, we would have to give up on blind writes.

the table's primary key is structured precisely so that we can quickly
find expired orders and so that we maximally benefit from rocksdb
path prefix compression. we do this by rounding the expires at time
forward to the next day, effectively giving us storagenode petnames
for free. and since there's no secondary index or foreign key
constraints, this design should use significantly less space than
the current used_serials table while also reducing contention.

after inserting the orders into the table, we have a chore that
periodically consumes all of the expired orders in it and inserts
them into the existing rollups tables. this is as if we changed
the nodes to report as the order expired rather than as soon as
possible, so the belief in correctness of the refactor is higher.

since we are able to process large batches of orders (typically
a day's worth), we can use the code to maximally batch inserts into
the rollup tables to make inserts as friendly as possible to
cockroach.

Change-Id: I25d609ca2679b8331979184f16c6d46d4f74c1a6
2020-01-15 19:21:21 -07:00
Jeff Wendling
9da16b1d9e satellite/satellitedb/dbx: name the package dbx
everyone was importing it as dbx anyway. why should it be
named satellitedb? so yeah just pass the "-p dbx" flag.

Change-Id: I5efa669f4f00f196b38a9acd0d402009475a936f
2020-01-15 15:16:39 -07:00
NikolaiYurchenko
dc1db75ab6 web/satellite: forgot password error message removed
Change-Id: Id9b5a8975e4225f46ef8626d1eaf335fcfebc0bd
2020-01-15 16:24:38 +00:00
Yingrong Zhao
db8aee0806 satellite/contact; storagenode/preflight: add clock check on startup for storagenode
add config preflight.enabled-local-time

Change-Id: I7b942c9bee063aae409ee6721ae9d079dff0144f
2020-01-15 15:35:26 +00:00
Yingrong Zhao
07c2824d94 storagenode/gracefulexit: fix exit-status command output
When exit succeeded, cli should display `Y` in Successful column and `100%` in PercentComplete.

Change-Id: I6093eca207ecd618bb332af12e5e455bc8224dde
2020-01-15 14:58:15 +00:00
Ivan Fraixedes
6dc948da46
satellite/metainfo: Create service for deleting pieces
Create a service for deleting pieces of storage nodes.

Currently the DeletePieces method returns after a success threshold,
completion or a timeout.

The end goal is to return when reaching the success threshold and
leaving the remaining goroutines running after DeletePieces method
returns and add a life cycle to the service that it waits for them when
it closes.

This is the first commit for ticket:
https://storjlabs.atlassian.net/browse/V3-3476

Change-Id: If740bbf57c741f880449980b8176b036dd956c7b
2020-01-15 15:24:20 +01:00
Egon Elbre
02ed2b5a19 satellite/metainfo: disallow cancelling deletion request
Change-Id: I05194d493632c9f454671ddf6b67b4901f2b7884
2020-01-15 14:23:46 +02:00
Egon Elbre
08f63614be private/context2: add WithoutCancellation
Change-Id: I38557c16f41b8983886f256353cc6afb7634d9e6
2020-01-15 14:23:46 +02:00
VitaliiShpital
19d318ea9d web/satellite: password strength check added to change password popup
Change-Id: Ibea92b34ab52547caacdcff8a71e8b2042226134
2020-01-15 11:51:50 +00:00
Ivan Fraixedes
f9b2af934b cmd/gateway: Fix entrypoint setup command
Fix the setup command of the entrypoint script used by the gateway
docker container.

Issue: https://github.com/storj/storj/issues/3748#issuecomment-574257152
Change-Id: I94c4d7aa148761645dd065f86b7d8e2d72bb727c
2020-01-15 11:29:27 +00:00
Egon Elbre
64fb2d3d2f Revert "dbutil: statically require all databases accesses to use contexts"
This reverts commit 8e242cd012.

Revert because lib/pq has known issues with context cancellation.
These issues need to be resolved before these changes can be merged.

Change-Id: I160af51dbc2d67c5449aafa406a403e5367bb555
2020-01-15 07:28:00 +00:00
JT Olio
c01cbe0130 satellitedb: save out all db-touching traces
Change-Id: Ib1e192221f9da813fd9cbb55f620a047b82c9523
2020-01-14 18:47:45 -05:00
JT Olio
8e242cd012 dbutil: statically require all databases accesses to use contexts
this will allow for some nice runtime analysis down the road.
also, this allows for wrapping database handles in a way that
can interact with these contexts

requires https://review.dev.storj.io/c/storj/dbx/+/514

Change-Id: Ib087b7cd73296dd2c1e0331314da34d861f61d2b
2020-01-14 18:20:47 -05:00
Egon Elbre
5af1f9e6d1 storagenode/{piecestore,storagenodedb}: use context in queries
In endpoint.saveOrder, ensure we always try to save orders such
that they can be settled.

Change-Id: Ic9ac8f4bf684d8493282912ca97f386c1762e364
2020-01-14 20:27:26 +00:00
Egon Elbre
64f056bee4 private/dbutil/sqlutil: use context in queries
Change-Id: Icb92daa483d13e6d57013f3917571d476126bfd2
2020-01-14 20:27:09 +00:00
Egon Elbre
df9e53ea0b private: ensure we don't eat the underlying error
When error is formatted using %v it's not possible to check
whether the error was caused by a context cancellation.

Change-Id: I164d1c83cdf5e7e6eacf082145b5c6a47078d041
2020-01-14 20:26:51 +00:00
Egon Elbre
d80cfeb4ab storagenode: ensure we don't eat the underlying error
When error is formatted using %v it's not possible to check
whether the error was caused by a context cancellation.

Change-Id: Ia77dfb0817e49d9a7b168c12a6300d131007d0ee
2020-01-14 20:26:23 +00:00
Egon Elbre
23e2664327 storagenode/inspector: return rpcstatus
Change-Id: I7e13b6dc8c9c3f4550f77885b1ef99662f5a5727
2020-01-14 20:24:46 +00:00
JT Olio
86093d0940 postgreskv: drop not null on buckets
Change-Id: I2a2bd7709de211a9d1808248af573f1bb630cfd5
2020-01-14 12:07:53 -07:00
crawter
41d5e86306 satellite/payments: coupon addition removed
Change-Id: I92781d9133603fdefd58b19a6f0ac6b1c6df3ac6
2020-01-14 16:24:48 +00:00
Egon Elbre
cd4ff0722e private/testplanet: use defaultInterval
Change-Id: Ife2810be46faaaf8cd51b193a859a88fff894a0e
2020-01-14 16:07:36 +00:00
Michal Niewrzal
6c4e3b64df cmd/uplink: 'uplink access inspect' for displaying access fields
https://storjlabs.atlassian.net/browse/V3-3492

Change-Id: Id75aad869f8d06444f970f1da50e784179993894
2020-01-14 15:29:38 +00:00
NikolaiYurchenko
c51f9a22e2 web/satellite: recovery password fonts and styles fixed
Change-Id: I0673a2dbb1fdf002c8b9060c7b81bae203bf4f62
2020-01-14 16:35:22 +02:00
NikolaiYurchenko
224025d714 web/satellite: password recovery page title fixed
Change-Id: I4e070315ec2e5f0b7ff0140f93dd6aacf1e68112
2020-01-14 13:34:54 +00:00
Michal Niewrzal
c8ccd26e04 cmd/uplink: import imports 'access' into existing configuration
https://storjlabs.atlassian.net/browse/V3-3491

Change-Id: I9c5f649ded314bb3a2235588c746913a3ec2d203
2020-01-14 13:18:48 +00:00
crawter
a57ce18f58 satellite/payments: coupons, coupons usage, invoice generation with pricing model applied
Change-Id: Ic5d5a2fc116388647efe46896cfccc2038c77537
2020-01-14 12:45:00 +00:00