Commit Graph

3238 Commits

Author SHA1 Message Date
Jeff Wendling
9da16b1d9e satellite/satellitedb/dbx: name the package dbx
everyone was importing it as dbx anyway. why should it be
named satellitedb? so yeah just pass the "-p dbx" flag.

Change-Id: I5efa669f4f00f196b38a9acd0d402009475a936f
2020-01-15 15:16:39 -07:00
NikolaiYurchenko
dc1db75ab6 web/satellite: forgot password error message removed
Change-Id: Id9b5a8975e4225f46ef8626d1eaf335fcfebc0bd
2020-01-15 16:24:38 +00:00
Yingrong Zhao
db8aee0806 satellite/contact; storagenode/preflight: add clock check on startup for storagenode
add config preflight.enabled-local-time

Change-Id: I7b942c9bee063aae409ee6721ae9d079dff0144f
2020-01-15 15:35:26 +00:00
Yingrong Zhao
07c2824d94 storagenode/gracefulexit: fix exit-status command output
When exit succeeded, cli should display `Y` in Successful column and `100%` in PercentComplete.

Change-Id: I6093eca207ecd618bb332af12e5e455bc8224dde
2020-01-15 14:58:15 +00:00
Ivan Fraixedes
6dc948da46
satellite/metainfo: Create service for deleting pieces
Create a service for deleting pieces of storage nodes.

Currently the DeletePieces method returns after a success threshold,
completion or a timeout.

The end goal is to return when reaching the success threshold and
leaving the remaining goroutines running after DeletePieces method
returns and add a life cycle to the service that it waits for them when
it closes.

This is the first commit for ticket:
https://storjlabs.atlassian.net/browse/V3-3476

Change-Id: If740bbf57c741f880449980b8176b036dd956c7b
2020-01-15 15:24:20 +01:00
Egon Elbre
02ed2b5a19 satellite/metainfo: disallow cancelling deletion request
Change-Id: I05194d493632c9f454671ddf6b67b4901f2b7884
2020-01-15 14:23:46 +02:00
Egon Elbre
08f63614be private/context2: add WithoutCancellation
Change-Id: I38557c16f41b8983886f256353cc6afb7634d9e6
2020-01-15 14:23:46 +02:00
VitaliiShpital
19d318ea9d web/satellite: password strength check added to change password popup
Change-Id: Ibea92b34ab52547caacdcff8a71e8b2042226134
2020-01-15 11:51:50 +00:00
Ivan Fraixedes
f9b2af934b cmd/gateway: Fix entrypoint setup command
Fix the setup command of the entrypoint script used by the gateway
docker container.

Issue: https://github.com/storj/storj/issues/3748#issuecomment-574257152
Change-Id: I94c4d7aa148761645dd065f86b7d8e2d72bb727c
2020-01-15 11:29:27 +00:00
Egon Elbre
64fb2d3d2f Revert "dbutil: statically require all databases accesses to use contexts"
This reverts commit 8e242cd012.

Revert because lib/pq has known issues with context cancellation.
These issues need to be resolved before these changes can be merged.

Change-Id: I160af51dbc2d67c5449aafa406a403e5367bb555
2020-01-15 07:28:00 +00:00
JT Olio
c01cbe0130 satellitedb: save out all db-touching traces
Change-Id: Ib1e192221f9da813fd9cbb55f620a047b82c9523
2020-01-14 18:47:45 -05:00
JT Olio
8e242cd012 dbutil: statically require all databases accesses to use contexts
this will allow for some nice runtime analysis down the road.
also, this allows for wrapping database handles in a way that
can interact with these contexts

requires https://review.dev.storj.io/c/storj/dbx/+/514

Change-Id: Ib087b7cd73296dd2c1e0331314da34d861f61d2b
2020-01-14 18:20:47 -05:00
Egon Elbre
5af1f9e6d1 storagenode/{piecestore,storagenodedb}: use context in queries
In endpoint.saveOrder, ensure we always try to save orders such
that they can be settled.

Change-Id: Ic9ac8f4bf684d8493282912ca97f386c1762e364
2020-01-14 20:27:26 +00:00
Egon Elbre
64f056bee4 private/dbutil/sqlutil: use context in queries
Change-Id: Icb92daa483d13e6d57013f3917571d476126bfd2
2020-01-14 20:27:09 +00:00
Egon Elbre
df9e53ea0b private: ensure we don't eat the underlying error
When error is formatted using %v it's not possible to check
whether the error was caused by a context cancellation.

Change-Id: I164d1c83cdf5e7e6eacf082145b5c6a47078d041
2020-01-14 20:26:51 +00:00
Egon Elbre
d80cfeb4ab storagenode: ensure we don't eat the underlying error
When error is formatted using %v it's not possible to check
whether the error was caused by a context cancellation.

Change-Id: Ia77dfb0817e49d9a7b168c12a6300d131007d0ee
2020-01-14 20:26:23 +00:00
Egon Elbre
23e2664327 storagenode/inspector: return rpcstatus
Change-Id: I7e13b6dc8c9c3f4550f77885b1ef99662f5a5727
2020-01-14 20:24:46 +00:00
JT Olio
86093d0940 postgreskv: drop not null on buckets
Change-Id: I2a2bd7709de211a9d1808248af573f1bb630cfd5
2020-01-14 12:07:53 -07:00
crawter
41d5e86306 satellite/payments: coupon addition removed
Change-Id: I92781d9133603fdefd58b19a6f0ac6b1c6df3ac6
2020-01-14 16:24:48 +00:00
Egon Elbre
cd4ff0722e private/testplanet: use defaultInterval
Change-Id: Ife2810be46faaaf8cd51b193a859a88fff894a0e
2020-01-14 16:07:36 +00:00
Michal Niewrzal
6c4e3b64df cmd/uplink: 'uplink access inspect' for displaying access fields
https://storjlabs.atlassian.net/browse/V3-3492

Change-Id: Id75aad869f8d06444f970f1da50e784179993894
2020-01-14 15:29:38 +00:00
NikolaiYurchenko
c51f9a22e2 web/satellite: recovery password fonts and styles fixed
Change-Id: I0673a2dbb1fdf002c8b9060c7b81bae203bf4f62
2020-01-14 16:35:22 +02:00
NikolaiYurchenko
224025d714 web/satellite: password recovery page title fixed
Change-Id: I4e070315ec2e5f0b7ff0140f93dd6aacf1e68112
2020-01-14 13:34:54 +00:00
Michal Niewrzal
c8ccd26e04 cmd/uplink: import imports 'access' into existing configuration
https://storjlabs.atlassian.net/browse/V3-3491

Change-Id: I9c5f649ded314bb3a2235588c746913a3ec2d203
2020-01-14 13:18:48 +00:00
crawter
a57ce18f58 satellite/payments: coupons, coupons usage, invoice generation with pricing model applied
Change-Id: Ic5d5a2fc116388647efe46896cfccc2038c77537
2020-01-14 12:45:00 +00:00
Yingrong Zhao
ee87846f0b satellite/contact: add placeholder for GetTime endpoint
Change-Id: I42f8479708f0558350c2280a398d84d145e8118f
2020-01-14 06:38:47 +00:00
Jeff Wendling
3b99f03780 satellite/orders: add monitoring to bucket bandwidth cache operations
Change-Id: Ib14303fc9f97a133410e2d6e2cf532e468b3dcee
2020-01-13 17:36:40 -07:00
Isaac Hess
4950d7106a satellite/orders: Add write cache for bw rollups
Change-Id: I8ba454cb2ab4742cafd6ed09120e4240874831fc
2020-01-13 22:40:51 +00:00
JT Olio
e1ba3931ec postgres2: use cockroachkv impl against postgres
this allows for setting $STORJ_METAINFO_POSTGRESQL_USE_ALT=yes if you
want to use the cockroachkv implementation for metainfo against postgres

Change-Id: I0c9458c83fd67ee63ef4a78351e64a80a0647408
2020-01-13 14:51:56 -06:00
Matt Robinson
dd234aad4c
jenkins: bump go to 1.13.6
Change-Id: I7218029a2264d1ef9d9d1f853c470e22137b5844
2020-01-13 13:57:56 -05:00
Jeff Wendling
71ec0ad374 satellite/satellitedb: add big honkin mutex to ProcessOrders
the hope is that it is mostly interfering with itself, so this
will make it not do that (well, N api servers, but hopefully
that's not enough to cause it to have issues).

Change-Id: Ifd0c9e6617457785ab25fe5b714d8556cdc8e2d3
2020-01-13 11:33:12 -07:00
VitaliiShpital
6bf40f3e59 web/satellite: npm packages updated
Change-Id: I87c1507423817283047d70d0d741307e03905556
2020-01-13 18:08:32 +00:00
Kaloyan Raev
172918596b cmd/uplink: output cleanup of uplink share command
JIRA: https://storjlabs.atlassian.net/browse/V3-3499

The `uplink share` command does not print the restricted API key and the
restricted encryption access anymore.

Change-Id: Ie4ebe0b27067ee00af97c775f4e06f558b894fe2
2020-01-13 16:53:36 +00:00
Egon Elbre
b9740f0c0a storage/cockroachkv: add ctx argument
Change-Id: Ib6c29f44722b0354afcd499a0e567f04aef7eb28
2020-01-13 15:57:47 +02:00
Egon Elbre
ff267168c5 private/migrate: add ctx argument
Change-Id: I3d65912d89261386413c494c7ed1576fed4dcaf4
2020-01-13 15:52:26 +02:00
Egon Elbre
24958bd7d3 satellite: add ctx to DB.CreateTables
Change-Id: I9ecad624cf5a7fc9c86bb91c68f96a3a4efd2e92
2020-01-13 15:31:09 +02:00
Egon Elbre
0835b9024c private/dbutil/pgutil: add ctx argument
Change-Id: Icfd56ca8c1f831ad56c0195a0b883e8f0618daaf
2020-01-13 15:27:06 +02:00
Egon Elbre
c7b846589e private/dbutil/sqliteutil: add ctx argument
Change-Id: If1caa9cde746817e62cae32a152eeec81959129c
2020-01-13 15:03:30 +02:00
littleskunk
bcc23f6869
Satellite/orders: remove allocated bandwith from storagenode_bandwidth_rollups
When an uplink requests an upload or download from the satellite we are trackig the
allocated bandwidth twice. The value in bucket_bandwidth_rollups is used
for project limits but the value in storagenode_bandwidth_rollups is not
used at all. We can increase the performance by removing it. Uplinks
will get a faster response from the satellite.

Change-Id: Icccd41f94107ef34668f30f99bf5f728c384b07e
2020-01-12 16:20:47 +01:00
Michal Niewrzal
36db00b2bf cmd/uplink: don't require setup or import if --access is set
We want to make using uplink as easy as possible. That's why we wan't to
avoid requiring setup or import command before normal usage if user
specified --access flag. If this flag is set then rest flags should be
set as defaults.

https://storjlabs.atlassian.net/browse/V3-3490

Change-Id: I95a7bd77a3f00b8d9981fee513e9e77aef298bca
2020-01-11 07:47:53 +00:00
Jeff Wendling
4aef0e3823 satellite/satellitedb: only reject orders if row not found
any database error doesn't mean the order wasn't found. for example
in cockroach it may say that the transaction is aborted. then what?
maybe we get big old row level deadlocks like we've observed? so
instead explicitly check for ErrNoRows to reject the order and bail
out otherwise. the surrounding logic will give it a retry.

Change-Id: I6e1f8f6e6a6def3e45b44f5088cbdc158e1098e4
2020-01-10 19:05:54 -07:00
Simon Guindon
5a1b2f49f4 storage/cockroachkv: add application name to the db connection string.
CockroachDB collects query metrics and separates them by application name and we were not setting the correct application name for the cockroachkv client. This PR calls our existing function that appends it to the connection string.
Change-Id: I4a97ed248c31f8b187c680d84b45472f0d50fd7e
2020-01-10 15:11:08 -05:00
NikolaiYurchenko
67b5528dc6 web/satellite: tab name on referral page fixed
Change-Id: I3a5e2fc9f79542984f3f7ca9eb9353bab9056239
2020-01-10 15:43:12 +00:00
Michal Niewrzal
b579c260ab cmd: rename "scope" flag to "access"
We decided that better name for "scope" will be "access". This change
refactors cmd part of code but don't touch libuplink. For backward
compatibility old configs with "scope" field will be loaded without any
issue. Old flag "scope" won't be supported directly from command line.

https://storjlabs.atlassian.net/browse/V3-3488

Change-Id: I349d6971c798380d147937c91e887edb5e9ae4aa
2020-01-10 15:27:53 +00:00
NikolaiYurchenko
57a31ac4e0 web/satellite: resend email timer fix
Change-Id: I44c00dfcfc8aed72f5542cd21ee8b4e363a5ff28
2020-01-10 14:59:59 +00:00
Natalie Ventura Villasana
131c4d94f0 {cmd/uplink, lib/uplink}: change RS total to 110
Bump the uplink's RS Total value from 95 to 110.

Change-Id: I208beed17b5f53fe359847e0469a29fec80a4f7e
2020-01-10 14:27:32 +00:00
Yingrong Zhao
455e14adde Jenkins: only run versions test on master commit
Change-Id: I441774b6bedd5eb16c477d92c4c4468ec243b8bf
2020-01-10 14:13:40 +00:00
Jeff Wendling
77fd41a02e satellite: add an expiring lru cache around api keys
Change-Id: I995429c66affd33da59b091f28f09ca122070b5e
2020-01-09 22:13:41 -07:00
Natalie Ventura Villasana
6b1829f3c3
satellite/downtime: new chore estimates downtime
Adds EstimationChore to the downtime package, which is an
independent chore that finds offline nodes given a configurable
limit, then uptime checks those nodes, and sets a last contact
success or failure given a response. For failed nodes, the chore
updates the amount of downtime the node has been offline in the
DowntimeTracking table.

Design doc section: https://github.com/storj/storj/blob/master/docs/blueprints/storage-node-downtime-tracking.md#estimating-offline-time
Jira: https://storjlabs.atlassian.net/browse/V3-2545

Change-Id: I60af95803930bf9b33232b248bb20cca6f0e0b5f
2020-01-09 15:05:13 -05:00
Qweder93
cf19e141e0 storagenode/notifications: return unread count and fix json id, list-notifications method fix
Change-Id: Ic56beac1f388d91a29c9e8266161715d09364520
2020-01-09 17:56:00 +00:00