Commit Graph

3243 Commits

Author SHA1 Message Date
Egon Elbre
7d79aab14e satellite/satellitedb: fixes to row handling
Change-Id: I48fae692bcca152143a12f333296c42471538850
2020-01-16 17:07:26 +02:00
littleskunk
0c365d157f
scripts/testversions: replace apikey with access
Change-Id: I4f899cc49b63b2f04f31a6df478475d7bdbab30d
2020-01-16 14:37:31 +01:00
littleskunk
1e77cb88e7
scripts/rollingupgrade: replace apikey with access
Change-Id: I356587cdc417dabc6f15769592d70269d25051dc
2020-01-16 14:00:33 +01:00
Jeff Wendling
f42851b1ab satellite/satellitedb: remove the big honkin mutex
no longer necessary/desired with reported_serials.

Change-Id: I69b5c535488eb5f98b250d73a7c8e6deaed0254e
2020-01-15 19:24:35 -07:00
Jeff Wendling
78c6d5bb32 satellite/satellitedb: reported_serials table for processing orders
this commit introduces the reported_serials table. its purpose is
to allow for blind writes into it as nodes report in so that we have
minimal contention. in order to continue to accurately account for
used bandwidth, though, we cannot immediately add the settled amount.
if we did, we would have to give up on blind writes.

the table's primary key is structured precisely so that we can quickly
find expired orders and so that we maximally benefit from rocksdb
path prefix compression. we do this by rounding the expires at time
forward to the next day, effectively giving us storagenode petnames
for free. and since there's no secondary index or foreign key
constraints, this design should use significantly less space than
the current used_serials table while also reducing contention.

after inserting the orders into the table, we have a chore that
periodically consumes all of the expired orders in it and inserts
them into the existing rollups tables. this is as if we changed
the nodes to report as the order expired rather than as soon as
possible, so the belief in correctness of the refactor is higher.

since we are able to process large batches of orders (typically
a day's worth), we can use the code to maximally batch inserts into
the rollup tables to make inserts as friendly as possible to
cockroach.

Change-Id: I25d609ca2679b8331979184f16c6d46d4f74c1a6
2020-01-15 19:21:21 -07:00
Jeff Wendling
9da16b1d9e satellite/satellitedb/dbx: name the package dbx
everyone was importing it as dbx anyway. why should it be
named satellitedb? so yeah just pass the "-p dbx" flag.

Change-Id: I5efa669f4f00f196b38a9acd0d402009475a936f
2020-01-15 15:16:39 -07:00
NikolaiYurchenko
dc1db75ab6 web/satellite: forgot password error message removed
Change-Id: Id9b5a8975e4225f46ef8626d1eaf335fcfebc0bd
2020-01-15 16:24:38 +00:00
Yingrong Zhao
db8aee0806 satellite/contact; storagenode/preflight: add clock check on startup for storagenode
add config preflight.enabled-local-time

Change-Id: I7b942c9bee063aae409ee6721ae9d079dff0144f
2020-01-15 15:35:26 +00:00
Yingrong Zhao
07c2824d94 storagenode/gracefulexit: fix exit-status command output
When exit succeeded, cli should display `Y` in Successful column and `100%` in PercentComplete.

Change-Id: I6093eca207ecd618bb332af12e5e455bc8224dde
2020-01-15 14:58:15 +00:00
Ivan Fraixedes
6dc948da46
satellite/metainfo: Create service for deleting pieces
Create a service for deleting pieces of storage nodes.

Currently the DeletePieces method returns after a success threshold,
completion or a timeout.

The end goal is to return when reaching the success threshold and
leaving the remaining goroutines running after DeletePieces method
returns and add a life cycle to the service that it waits for them when
it closes.

This is the first commit for ticket:
https://storjlabs.atlassian.net/browse/V3-3476

Change-Id: If740bbf57c741f880449980b8176b036dd956c7b
2020-01-15 15:24:20 +01:00
Egon Elbre
02ed2b5a19 satellite/metainfo: disallow cancelling deletion request
Change-Id: I05194d493632c9f454671ddf6b67b4901f2b7884
2020-01-15 14:23:46 +02:00
Egon Elbre
08f63614be private/context2: add WithoutCancellation
Change-Id: I38557c16f41b8983886f256353cc6afb7634d9e6
2020-01-15 14:23:46 +02:00
VitaliiShpital
19d318ea9d web/satellite: password strength check added to change password popup
Change-Id: Ibea92b34ab52547caacdcff8a71e8b2042226134
2020-01-15 11:51:50 +00:00
Ivan Fraixedes
f9b2af934b cmd/gateway: Fix entrypoint setup command
Fix the setup command of the entrypoint script used by the gateway
docker container.

Issue: https://github.com/storj/storj/issues/3748#issuecomment-574257152
Change-Id: I94c4d7aa148761645dd065f86b7d8e2d72bb727c
2020-01-15 11:29:27 +00:00
Egon Elbre
64fb2d3d2f Revert "dbutil: statically require all databases accesses to use contexts"
This reverts commit 8e242cd012.

Revert because lib/pq has known issues with context cancellation.
These issues need to be resolved before these changes can be merged.

Change-Id: I160af51dbc2d67c5449aafa406a403e5367bb555
2020-01-15 07:28:00 +00:00
JT Olio
c01cbe0130 satellitedb: save out all db-touching traces
Change-Id: Ib1e192221f9da813fd9cbb55f620a047b82c9523
2020-01-14 18:47:45 -05:00
JT Olio
8e242cd012 dbutil: statically require all databases accesses to use contexts
this will allow for some nice runtime analysis down the road.
also, this allows for wrapping database handles in a way that
can interact with these contexts

requires https://review.dev.storj.io/c/storj/dbx/+/514

Change-Id: Ib087b7cd73296dd2c1e0331314da34d861f61d2b
2020-01-14 18:20:47 -05:00
Egon Elbre
5af1f9e6d1 storagenode/{piecestore,storagenodedb}: use context in queries
In endpoint.saveOrder, ensure we always try to save orders such
that they can be settled.

Change-Id: Ic9ac8f4bf684d8493282912ca97f386c1762e364
2020-01-14 20:27:26 +00:00
Egon Elbre
64f056bee4 private/dbutil/sqlutil: use context in queries
Change-Id: Icb92daa483d13e6d57013f3917571d476126bfd2
2020-01-14 20:27:09 +00:00
Egon Elbre
df9e53ea0b private: ensure we don't eat the underlying error
When error is formatted using %v it's not possible to check
whether the error was caused by a context cancellation.

Change-Id: I164d1c83cdf5e7e6eacf082145b5c6a47078d041
2020-01-14 20:26:51 +00:00
Egon Elbre
d80cfeb4ab storagenode: ensure we don't eat the underlying error
When error is formatted using %v it's not possible to check
whether the error was caused by a context cancellation.

Change-Id: Ia77dfb0817e49d9a7b168c12a6300d131007d0ee
2020-01-14 20:26:23 +00:00
Egon Elbre
23e2664327 storagenode/inspector: return rpcstatus
Change-Id: I7e13b6dc8c9c3f4550f77885b1ef99662f5a5727
2020-01-14 20:24:46 +00:00
JT Olio
86093d0940 postgreskv: drop not null on buckets
Change-Id: I2a2bd7709de211a9d1808248af573f1bb630cfd5
2020-01-14 12:07:53 -07:00
crawter
41d5e86306 satellite/payments: coupon addition removed
Change-Id: I92781d9133603fdefd58b19a6f0ac6b1c6df3ac6
2020-01-14 16:24:48 +00:00
Egon Elbre
cd4ff0722e private/testplanet: use defaultInterval
Change-Id: Ife2810be46faaaf8cd51b193a859a88fff894a0e
2020-01-14 16:07:36 +00:00
Michal Niewrzal
6c4e3b64df cmd/uplink: 'uplink access inspect' for displaying access fields
https://storjlabs.atlassian.net/browse/V3-3492

Change-Id: Id75aad869f8d06444f970f1da50e784179993894
2020-01-14 15:29:38 +00:00
NikolaiYurchenko
c51f9a22e2 web/satellite: recovery password fonts and styles fixed
Change-Id: I0673a2dbb1fdf002c8b9060c7b81bae203bf4f62
2020-01-14 16:35:22 +02:00
NikolaiYurchenko
224025d714 web/satellite: password recovery page title fixed
Change-Id: I4e070315ec2e5f0b7ff0140f93dd6aacf1e68112
2020-01-14 13:34:54 +00:00
Michal Niewrzal
c8ccd26e04 cmd/uplink: import imports 'access' into existing configuration
https://storjlabs.atlassian.net/browse/V3-3491

Change-Id: I9c5f649ded314bb3a2235588c746913a3ec2d203
2020-01-14 13:18:48 +00:00
crawter
a57ce18f58 satellite/payments: coupons, coupons usage, invoice generation with pricing model applied
Change-Id: Ic5d5a2fc116388647efe46896cfccc2038c77537
2020-01-14 12:45:00 +00:00
Yingrong Zhao
ee87846f0b satellite/contact: add placeholder for GetTime endpoint
Change-Id: I42f8479708f0558350c2280a398d84d145e8118f
2020-01-14 06:38:47 +00:00
Jeff Wendling
3b99f03780 satellite/orders: add monitoring to bucket bandwidth cache operations
Change-Id: Ib14303fc9f97a133410e2d6e2cf532e468b3dcee
2020-01-13 17:36:40 -07:00
Isaac Hess
4950d7106a satellite/orders: Add write cache for bw rollups
Change-Id: I8ba454cb2ab4742cafd6ed09120e4240874831fc
2020-01-13 22:40:51 +00:00
JT Olio
e1ba3931ec postgres2: use cockroachkv impl against postgres
this allows for setting $STORJ_METAINFO_POSTGRESQL_USE_ALT=yes if you
want to use the cockroachkv implementation for metainfo against postgres

Change-Id: I0c9458c83fd67ee63ef4a78351e64a80a0647408
2020-01-13 14:51:56 -06:00
Matt Robinson
dd234aad4c
jenkins: bump go to 1.13.6
Change-Id: I7218029a2264d1ef9d9d1f853c470e22137b5844
2020-01-13 13:57:56 -05:00
Jeff Wendling
71ec0ad374 satellite/satellitedb: add big honkin mutex to ProcessOrders
the hope is that it is mostly interfering with itself, so this
will make it not do that (well, N api servers, but hopefully
that's not enough to cause it to have issues).

Change-Id: Ifd0c9e6617457785ab25fe5b714d8556cdc8e2d3
2020-01-13 11:33:12 -07:00
VitaliiShpital
6bf40f3e59 web/satellite: npm packages updated
Change-Id: I87c1507423817283047d70d0d741307e03905556
2020-01-13 18:08:32 +00:00
Kaloyan Raev
172918596b cmd/uplink: output cleanup of uplink share command
JIRA: https://storjlabs.atlassian.net/browse/V3-3499

The `uplink share` command does not print the restricted API key and the
restricted encryption access anymore.

Change-Id: Ie4ebe0b27067ee00af97c775f4e06f558b894fe2
2020-01-13 16:53:36 +00:00
Egon Elbre
b9740f0c0a storage/cockroachkv: add ctx argument
Change-Id: Ib6c29f44722b0354afcd499a0e567f04aef7eb28
2020-01-13 15:57:47 +02:00
Egon Elbre
ff267168c5 private/migrate: add ctx argument
Change-Id: I3d65912d89261386413c494c7ed1576fed4dcaf4
2020-01-13 15:52:26 +02:00
Egon Elbre
24958bd7d3 satellite: add ctx to DB.CreateTables
Change-Id: I9ecad624cf5a7fc9c86bb91c68f96a3a4efd2e92
2020-01-13 15:31:09 +02:00
Egon Elbre
0835b9024c private/dbutil/pgutil: add ctx argument
Change-Id: Icfd56ca8c1f831ad56c0195a0b883e8f0618daaf
2020-01-13 15:27:06 +02:00
Egon Elbre
c7b846589e private/dbutil/sqliteutil: add ctx argument
Change-Id: If1caa9cde746817e62cae32a152eeec81959129c
2020-01-13 15:03:30 +02:00
littleskunk
bcc23f6869
Satellite/orders: remove allocated bandwith from storagenode_bandwidth_rollups
When an uplink requests an upload or download from the satellite we are trackig the
allocated bandwidth twice. The value in bucket_bandwidth_rollups is used
for project limits but the value in storagenode_bandwidth_rollups is not
used at all. We can increase the performance by removing it. Uplinks
will get a faster response from the satellite.

Change-Id: Icccd41f94107ef34668f30f99bf5f728c384b07e
2020-01-12 16:20:47 +01:00
Michal Niewrzal
36db00b2bf cmd/uplink: don't require setup or import if --access is set
We want to make using uplink as easy as possible. That's why we wan't to
avoid requiring setup or import command before normal usage if user
specified --access flag. If this flag is set then rest flags should be
set as defaults.

https://storjlabs.atlassian.net/browse/V3-3490

Change-Id: I95a7bd77a3f00b8d9981fee513e9e77aef298bca
2020-01-11 07:47:53 +00:00
Jeff Wendling
4aef0e3823 satellite/satellitedb: only reject orders if row not found
any database error doesn't mean the order wasn't found. for example
in cockroach it may say that the transaction is aborted. then what?
maybe we get big old row level deadlocks like we've observed? so
instead explicitly check for ErrNoRows to reject the order and bail
out otherwise. the surrounding logic will give it a retry.

Change-Id: I6e1f8f6e6a6def3e45b44f5088cbdc158e1098e4
2020-01-10 19:05:54 -07:00
Simon Guindon
5a1b2f49f4 storage/cockroachkv: add application name to the db connection string.
CockroachDB collects query metrics and separates them by application name and we were not setting the correct application name for the cockroachkv client. This PR calls our existing function that appends it to the connection string.
Change-Id: I4a97ed248c31f8b187c680d84b45472f0d50fd7e
2020-01-10 15:11:08 -05:00
NikolaiYurchenko
67b5528dc6 web/satellite: tab name on referral page fixed
Change-Id: I3a5e2fc9f79542984f3f7ca9eb9353bab9056239
2020-01-10 15:43:12 +00:00
Michal Niewrzal
b579c260ab cmd: rename "scope" flag to "access"
We decided that better name for "scope" will be "access". This change
refactors cmd part of code but don't touch libuplink. For backward
compatibility old configs with "scope" field will be loaded without any
issue. Old flag "scope" won't be supported directly from command line.

https://storjlabs.atlassian.net/browse/V3-3488

Change-Id: I349d6971c798380d147937c91e887edb5e9ae4aa
2020-01-10 15:27:53 +00:00
NikolaiYurchenko
57a31ac4e0 web/satellite: resend email timer fix
Change-Id: I44c00dfcfc8aed72f5542cd21ee8b4e363a5ff28
2020-01-10 14:59:59 +00:00