Commit Graph

4138 Commits

Author SHA1 Message Date
Bill Thorp
4b612a7906 cmd/segment-reaper: add support for >65 segments
Change the bitmask used by segment reaper to use []byte rather than uint64

This passes tests but I have literally no clue how to integration test this.

Change-Id: I393f4598b27cae6e427da2190dd3109bca721c34
2020-05-09 10:45:26 +00:00
Egon Elbre
bcd93ee375 private/testplanet: add StopNodeAndUpdate
This was commonly used and code with it can be simplified.

Change-Id: I2f2b91f7de54269aee6ef027f97f9e8a7d222e39
2020-05-08 13:02:19 +00:00
Kaloyan Raev
12fa505692 satellite/payments: fix invoice amount in billing history
The billing history currently shows the Total amount from the Stripe
invoice. In fact, this value is just the amount deducted from the Stripe
balance. It does not reflect any deduction from promotional coupon or
bonus credits.

This patch adds the deducted amount form the promotional coupons and
bonus credits to the displayed amount in billing history. This way
customers have better understading of the total amount deducted from
their account balance on the satellite.

Change-Id: Ibd7f611a6cea0143a3059f39dd1d9ef21c264d8c
2020-05-08 11:43:41 +00:00
Matt Robinson
94503db5a6
cmd/satellite: Enable admin service like other services, also sort environment variables (#3874)
Change-Id: I67592b7bf34a0e2ce9a445b58b7fa34396aa8c5d

Co-authored-by: Stefan Benten <mail@stefan-benten.de>
2020-05-08 07:38:21 -04:00
Egon Elbre
a549d523dc bump drpc and common
Change-Id: I231d72793cc07cefc59d013fce02c03d5638eabc
2020-05-08 10:50:57 +03:00
Egon Elbre
678b859172 satellite/overlay: remove MinimumRequiredNodes
In non-test code we were only using RequestedCount, not need to have
MinimumRequiredNodes.

Change-Id: I40736f4b028b41e94abfdeb221bce5aa86a5cb82
2020-05-07 15:41:23 +00:00
Kaloyan Raev
2af7138928 web/satellite: change invoice link title to 'Invoice PDF'
Change-Id: I820bb33f4ba88bb732d1c66ba3369ac4eac72f5c
2020-05-07 18:14:40 +03:00
VitaliiShpital
9837230f1e web/satellite: project dropdown reworked
Change-Id: I6267347693724ec34996af7625a917451aef631d
2020-05-07 12:51:34 +00:00
Egon Elbre
ce6a500b0c satellite/overlay: support DistinctIP=false in selection cache
Most of our tests and storj-sim are using DistinctIP.

Also fix bug with newNodeCount calculation.

Change-Id: I1a6d0efe7074908896a3322d19f487b929f0f0fc
2020-05-07 11:04:32 +00:00
Yaroslav Vorobiov
d8bdc60e19 storagenode/console: remove last ping id and address
Change-Id: I001e7d31d50aee7b9a07608158f8bd5c407c9c45
2020-05-07 13:34:29 +03:00
Egon Elbre
90d859fbb8 private/testplanet: use drpc piecestore mock for testing
Change-Id: Ia3f93f3c8b6584fb92f5d29025b7f0691120430e
2020-05-07 10:54:49 +03:00
Egon Elbre
c5452a87ec private/testplanet: use drpc referral manager server
Change-Id: I9e9e9a724c78c98859dd3e29416d766d8ffdca63
2020-05-07 07:03:11 +00:00
Egon Elbre
2955c50bc1 satellite/overlay: fix data-race in node selection cache
GetNodes returned references to nodes in the immutable state, however
some parts of code expect them to be modified.

Change-Id: I5be1866f95e0dbe062a6b6be60e29f2365c35faa
2020-05-06 20:03:06 +03:00
crawter
693f224ffd storagenode/web: Segment.io integration
Change-Id: I5b8054f39453bbfdebdbaaea1ffda4d5e706dab3
2020-05-06 15:02:53 +00:00
Jess G
73b2cd3fd5
docs/blueprints: blueprint nodes cache (#3838)
* init design for selecting nodes for uploads

Change-Id: I40920a88fc8c0c8591eaf3f4e711578ca02dad73

* spelling erro

Change-Id: I843945f356dbdfd5c8b7cefee91ab4172c41f3fb

* updates per comments

Change-Id: I2d6f0f9b4f22c4c7ae8ed80c4f533d2d9392badf

* add details

Change-Id: Id31ecfbe9f7ff068b127f04ba061a3fb9ad254cc

* mv redis to rationale section

Change-Id: I98d8565bb228555d92682d3b08c99c6056b0a84b

* add details about ReliabilityCache

Change-Id: I9f0ecf0a96a217c5e55c6a855b8e06e0a76ab574

* add option details

Change-Id: Icd229cbbdc263466d16856fdd82a021509a1c76a

* add opt for queue

Change-Id: I9d36e5e6f923440b6c22066a2aaf981aecc584d6

* final updates

Change-Id: Iee74985f3a45b3ef58ff23ef3ae26f609fb7f0d5
2020-05-06 07:44:21 -07:00
Egon Elbre
4e94da3fda satellite/overlay: add feature flag for node selection cache
Also distinguish the purpose for selecting nodes to avoid potential
confusion, what should allow caching and what shouldn't.

Change-Id: Iee2451c1f10d0f1c81feb1641507400d89918d61
2020-05-06 16:13:47 +03:00
Fadila Khadar
e4da7d65f0 pkg/macaroons: fix action types comments
Fix comments in ActionType definition

Change-Id: I29a300a0699697a10ebf5613ba36a2702aa67175
2020-05-06 11:58:56 +02:00
Jeff Wendling
efee3563a8 storagenode/storagenodedb: fix joined_at migration
Change-Id: I9091741bbec6e7a2fc6addc23249a40176ec040a
2020-05-05 15:01:07 -06:00
Qweder93
b754806b23 storagenode/reputation: unknown_audit_reputation_alpha and beta added to db, and reputation endpoint
Change-Id: I5d92268851bb9a202cc5d6b4d403467e6f692726
2020-05-05 15:48:04 +00:00
Qweder93
8a443bc9f5 storagenode/console: api notifications tests added
Change-Id: I472abc22746ea95841dcb0ae72dacee06fd98a2e
2020-05-05 17:53:19 +03:00
Kaloyan Raev
b5433322b4 cmd/storj-sim: enable open user registration
Change-Id: I695664279f42209245a82cc5bf103d037b844c32
2020-05-05 13:03:02 +00:00
Qweder93
16cd9b06ec storagenode/heldamount: added api for heldamount history separated by periods
Change-Id: I170010364269822848bc6cd051e0e0fb3df95d91
2020-05-05 12:29:44 +00:00
VitaliiShpital
0ae0cea93b web/satellite: billing history UX improved
added sorting header title
currency numbers are displayed with two digits now

Change-Id: I9531f112ff70ffb89b8a2dfe05ab02eb767dd107
2020-05-05 12:07:33 +00:00
Jennifer Johnson
18078bf7ee satellite/audit: increases audit worker concurrency to 2
Change-Id: Ibe3e3801b79accffbcfe9e2e02c96fc963894a7f
2020-05-05 11:31:55 +00:00
NickolaiYurchenko
c5871b83b0 web/storagenode: no data state added to payout table
Change-Id: I3f141debe2f7c04a20999c0d2e2a7cee1434f7d1
2020-05-05 11:10:10 +00:00
Egon Elbre
b7495d414d mod: update sqlite to v3.31.1
Change-Id: Ia2885f5385ffa227b8576c1d9e1c8bbba0c5d13d
2020-05-05 10:52:36 +00:00
Egon Elbre
d98b8f6e23 satellite/metainfo,storage: use different limit for metainfo loop
Change-Id: I5ef7233930679b977b33f7b3e1dda45c907dcfad
2020-05-05 10:37:20 +00:00
Jeff Wendling
13baacc479 lib/uplink: better diagnostics with invalid encryption access
Fixes #3841

Change-Id: I965b8fbb8fd221c740c2fc1421d66721c046f038
2020-05-05 09:45:57 +00:00
Brandon Iglesias
324fd410a0
updating Aha link on our readme 2020-05-04 14:21:58 -04:00
Moby von Briesen
8f60cfc4fb satellite/overlay: Add flag for enabling/disabling disqualification from suspension mode
Add a flag that allows us to easily switch disqualification from
suspension mode on or off. A node will only be disqualified from
suspension mode if it has been suspended for longer than the grace
period AND the SuspensionDQEnabled flag is true.

Change-Id: I9e67caa727183cd52ab2042b0a370a1bcaebe792
2020-05-04 17:25:09 +00:00
Jeff Wendling
f93d764998 cmd/uplink: properly path escape shares
Fixes #3829

Change-Id: I0f900a392239fcc064cea968fc7a5bc373ee26a1
2020-05-04 17:00:25 +00:00
Egon Elbre
31b322f07e pkg/server: log apikey head
Change-Id: Ib86987405ea73fe8a1b5cec93cf3b219e36074d1
2020-05-04 16:36:02 +00:00
Ethan
acf53bea4d satellite/orders;accounting: Add monthly project download bandwidth rollup
See https://storjlabs.atlassian.net/browse/SM-776

Change-Id: Ifd5cccea43c556fd59822d17344f399cfe9a7164
2020-05-04 15:49:57 +00:00
Michal Niewrzal
22691bd47f satellite/accounting: test project limits reset after file delete
https://storjlabs.atlassian.net/browse/USR-288

Change-Id: Ia63e58ef3b0f79b5edfb9a63f8edcdc17486b650
2020-05-04 15:19:19 +00:00
Yingrong Zhao
0b80d60491 cmd/uplink: disable tlemetry if user opt out
Change-Id: I793e6428d04e4f422d1f8292e99b233c70d9dd6e
2020-05-04 14:46:49 +00:00
VitaliiShpital
2284008b8c web/satellite: onboarding tour: api keys and upload data steps
Change-Id: I8ffa6d688a22c1568495a7e0e176096cadcd6eaa
2020-05-04 13:26:12 +00:00
Qweder93
7a83473f00 storagenode/console: API /dashboard disk-space data separated for UsedForPieces, Trash, Available
Change-Id: I7411da2c92c72a24af98e007efe476d8a023db82
2020-05-04 12:36:18 +00:00
Michal Niewrzal
16081f53d2 storj: update to uplink 1.0.5
Change-Id: Ie4f01356edc43c3ee2d124a69f4da1eb75a17618
2020-05-04 08:09:18 +00:00
Egon Elbre
2fad150952 storagenode/piecestore: remove piecestore.Client.Delete
This test was the last place using it. Replace it with a direct call so
we can remove the method from uplink piecestore.

Change-Id: I62e13028663a7e67aa2495f90ecc02d0d8657fbd
2020-05-04 10:14:55 +03:00
Jeff Wendling
57eb8a17e2 storagenode: allow configuring database path independently
Fixes #3852

Change-Id: I021c29c4dd7c393399f6abef41d8457514032833
2020-05-04 06:04:31 +00:00
Egon Elbre
c630cf2490 storagenode/pieces: implement buffering for writing
Currently uploads can cause a lot of IOPS, reduce this by introducing a
in-memory buffer on-top of the file.

Change-Id: I5f4e3e01c0a36258271d180b922107de447bcb59
2020-05-04 06:01:32 +00:00
Stefan Benten
01e0ba2e0d
satellite/metainfo: move logging to debug for piece deletion (#3873) 2020-05-01 23:29:28 +02:00
Jeff Wendling
4ad01e8170 storagenode/storagenodedb: backfill reputation.joined_at
it was being used in ways that implied it should be NOT NULL
even though it was possibly null. we used to get this data
from the satellite db's added_at column as seen in 30369b02,
so backfill it using that data where joined_at is NULL, and
then alter the table to constrain the column to be NOT NULL.

Fixes #3866.

Change-Id: If2d856189209740d985f71dada7b93525e625ef3
2020-05-01 17:59:53 +00:00
Qweder93
6c4d3f133f storagenode/dashboard: trash added to avaliable space calculations
Change-Id: Ia6f3af20dc98f569b86796ffa68428065d662c78
2020-05-01 15:26:02 +00:00
Jeff Wendling
202e65e408 storagenode/storagenodedb: do correct generalized alter table procedure
According to the docs at https://www.sqlite.org/lang_altertable.html
doing the steps

    1. Rename old table
    2. Create new table
    3. Copy data
    4. Drop old table

is incorrect and should be

    1. Create new table
    2. Copy data
    3. Drop old table
    4. Rename new into old

Additionally, each step was being run in a different transaction,
which could cause permanent failures if a problem happened during
the migration.

Avoid both of those problems by changing up some previous migrations
that ran in this way. Since they are semantically identical, it's
fine to change up these old migrations. It will help make newer
nodes coming up for the first time more robust.

Change-Id: I43fb004fa1b6cb2fe2554f9920925420da28fb4a
2020-05-01 13:53:26 +00:00
Yaroslav Vorobiov
516b8cf2be storagenode-updater: add recovery for windows service restart
Reimplement windows service restart part using svc, add recovery
for failed service startup. Added restart-service cmd, to execute
self restart in a separate process.

Addressed issues:
https://storjlabs.atlassian.net/browse/SG-49
https://storjlabs.atlassian.net/browse/SG-136
https://storjlabs.atlassian.net/browse/SG-137

Change-Id: Ic51d9a99e8c1c10800c6c60ff4e218321c674fea
2020-05-01 09:07:05 +00:00
Jeff Wendling
bb28851964 test-sim-backwards: refactor, reduce node count to 9 in mixed mode
This catches the bug that hit prod where we missed that a migration
was necessary for a column, but only on uploads, and only for nodes
that had not checked in yet.

So, this does more uploads, and stops a node from starting so that
we trigger that specific scenario. Hopefully it will catch other
similar ones in the future.

I confirmed that this locally caught the bug when the release under
test was v0.34.10 and HEAD did not include the fix for it.

Change-Id: If7d41e8241d6a042fa524b4aff956b0264ecb128
2020-05-01 04:56:06 +00:00
Isaac Hess
5a85e8d749 satellite/audit: Fix flaky TestVerifierSlowDownload
TestVerifierSlowDownload would sometimes not have enough nodes finish in
the allotted deadline period. This increases the deadline and also does
not assert that exactly 3 have finished. Instead, in keeping with the
purpose of the test, it asserts that the slow download is never counted
as a success and is always counted as a pending audit in the final
report.

Change-Id: I180734fcc4a499420c75164bad6253ed155d87de
2020-04-30 15:58:47 -06:00
Yaroslav Vorobiov
2b00d78add satellite/payments: add documentation
Change-Id: I75712ef282530b56ea7ca1f2bc31ee0af43dae33
2020-04-30 21:12:16 +00:00
Matt Robinson
4a95ad0b2f
storagenode/piecestore: remove error from info log for download canceled (#3869) 2020-04-30 18:42:16 +03:00