Commit Graph

4179 Commits

Author SHA1 Message Date
Ivan Fraixedes
d27cb60f81 cmd/segment-reaper: Remove unneeded test logic
After the segment-reaper was enhanced to support objects with any number
of segments, the test helper function to generate test data doesn't have
to support to generate objects with a number of segments up to or over
the previous limit.

The commit remove that unnecessary test logic.

Change-Id: I2897c5a96fb133f61de2cdb0ef7d13ee5e0151e8
2020-05-18 16:39:48 +00:00
Kaloyan Raev
aac1e3c45f satellite/payments: move inspector commands to satellite cli
This allows to seeing logs in the output of the invoice commands.

Existing ensure-stripe-customer commands is moved from the 'reports' to
the new 'billing' root command.

Change-Id: I752c7ab6ca59bfac8e0f174a45d2ab45fc18e467
2020-05-18 16:12:45 +00:00
VitaliiShpital
47def02094 satellite: account balance divided into Free Credits and Coins
Change-Id: I2ae29701bab9f0b48f236ff94b366eb534b81c6b
2020-05-18 14:25:26 +00:00
Michal Niewrzal
9fb423ac19 satellite/payments/stripecoinpayments: fix listing issue in
InvoiceApplyProjectRecords

ListUnapplied method in listing loop  was using next offset as starting
point but applyProjectRecords method was changing project record state
to applied and we were missing some records to apply.

Test will be added as a separate change.

Change-Id: Id1ca33eeb66ec7f6ff1f05b45615a8935185568e
2020-05-18 13:11:30 +00:00
VitaliiShpital
4544f2dffb web/satellite: navigation side bar reworked
Change-Id: I9be2d23a7a4af219d0b8888b7296fa7b0e57fc5a
2020-05-18 11:04:45 +00:00
Natalie Villasana
8d87a6efc9 cockroachutil/driver: handle retryable errors returned from Next
This will only work if retryable errors are returned on the first
call to Next. Otherwise if they're returned later, we will need
deeper changes at the application code level throughout the
codebase 😬👎

Change-Id: I46d795a13670f66b7f085605ba1b779f69c339c3
2020-05-15 14:49:43 -04:00
littleskunk
ef2671927d
storagenode/piecestore: move queue size defaults (#3881) 2020-05-15 19:10:26 +02:00
NickolaiYurchenko
ffbaaff179 web/storagenode: node module store tests added
Change-Id: I582fc1405263965e870a233def3f2d026490dd70
2020-05-15 14:41:48 +00:00
Stefan Benten
26f36d590e
cmd/satellite: ensure we only create stripe customers for activated users (#3884) 2020-05-15 16:00:46 +02:00
Egon Elbre
f6301612ac satellite/orders: use serial SerialNumber
By ensuring that they have less randomness it means that they can be
compressed better. Using a timestamp should be a good improvement here.

Change-Id: Ic4dabb53335a744ff1c332dd279f37ae2cd79357
2020-05-15 11:33:53 +00:00
Michal Niewrzal
2eb2c25e51 satellite/payments/stripecoinpayments: add StripeClient interface to be
able to cover more testing scenarios

Currently, its hard to implement test suite for payments because
mockpayments is on to high level and we cannot emulate many things e.g.
adding credit card. This change is first to be able to add mock for
Stripe client and do more granular tests.

Change-Id: Ied85d4bd0642debdffe1161657c1e475202e9d23
2020-05-15 10:52:44 +02:00
VitaliiShpital
1ec5eb06bd web/satellite: account dropdown reworked
Change-Id: I415f77c7c13546f467da62fc13feb2774dd54290
2020-05-14 17:08:02 +00:00
VitaliiShpital
a3459ec6b8 web/satellite: project dashboard page UI reworked
Change-Id: I134d70b3a351d2a104612727f7c24e21122a6730
2020-05-14 16:52:11 +00:00
Jeff Wendling
f03b23d2dc storage/postgreskv: monitor calls to sql.Next
Change-Id: I32c0c92b347a6d1cadfeb69117de58f4d9b41ad2
2020-05-14 16:35:25 +00:00
littleskunk
28a886f399
satellite/admin: obvious project name in documentation 2020-05-14 15:50:58 +03:00
Kaloyan Raev
49571f1a23 satellite/payments: all invoice commands require period
To avoid including multiple months in a single invoice, we need all
inspector's invoice commands to run in for specific period.

See https://storjlabs.atlassian.net/browse/USR-725

Change-Id: I3637dc189234f02350daca8d897c21765762ea55
2020-05-14 11:50:19 +00:00
Qweder93
ef87192120 storagenode/notifications: tests fixed: added time interval between inserts so created_at fields are different when running tests on Windows
Change-Id: I26ba059ab58d0216122ab2f49ae85f7ce7cfced4
2020-05-14 05:26:18 +00:00
Moby von Briesen
46df8c1977 satellite/gracefulexit: add log message when node fails validation for piece transfer
Change-Id: Ic5a53404ceb35003793aebc63637e7f8a58ef259
2020-05-13 16:58:50 +00:00
Jeff Wendling
6352d46100 satellite/satellitedb: do better ::date conversions
There is a subtle problem when one does a cast with `::date`. Observe:

    teststorj=# set timezone = 'US/Eastern';
    SET
    teststorj=# select (timestamp with time zone '2020-02-01 00:00:00+00')::date;
        date
    ------------
     2020-01-31
    (1 row)

    teststorj=# set timezone = 'UTC';
    SET
    teststorj=# select (timestamp with time zone '2020-02-01 00:00:00+00')::date;
        date
    ------------
     2020-02-01
    (1 row)

In order to correctly determine the date a timestamp is in, one has to
explicitly pick the time zone that the date truncation should use
otherwise postgres will use whatever setting the client has. These
tests were failing for me locally, because I run my postgres in
the US/Eastern time zone to try to tickle these bugs out. So it
should be `(x at time zone 'UTC')::date` instead of just `x::date`.

Change-Id: I4e9e32d4b53abc6165a4d0474f4702f8b9f801c7
2020-05-13 15:58:07 +00:00
igor gaidaienko
75ddecdf54 satellite/metainfo: Add test to verify payment report for attributed bucket
For this automated test I want to make sure the data in payment report is correct

Change-Id: I48a3b0e1dc9b2324435b763c87c363f1e310ff4e
2020-05-13 17:22:04 +03:00
Bill Thorp
5a7a4d2e98 satellite: add Go test version of satellite-config-lock tests
The current satellite config lock code relies on bash scripts and
gnu diff, it must be run as root and hence it typically requires
docker.  The old version will be removed at a later date..

I tried for several hours to run directly against cmdSetup() in
cmd/satellite/main.go, to avoid the ctx.Compile() call. I had no
luck.

Change-Id: I0a4888421e743b436d32b6af69d04759d7816751
2020-05-13 08:14:24 +00:00
NickolaiYurchenko
ff68a3c2bb web/storagenode: payout page scroll area extended
Change-Id: I937a5e7344a0c3507fe3aa2443458ee86c8f9af7
2020-05-12 21:27:14 +00:00
Ethan
159df8b2e4 Add logging listener for retrieving and setting log levels
See https://storjlabs.atlassian.net/browse/SM-752

These changes allow us to change the log level at runtime through a handler off of the debug endpoint.

Examples of changing the log level on storj-sim

To get the current level for the satellite api process:
curl -XGET 'http://127.0.0.1:10009/logging' --header 'Content-Type: text/plain'

To change the log level:
curl -XPUT 'http://127.0.0.1:10009/logging' --header 'Content-Type: text/plain' --data-raw '{"level":"error"}'

Change-Id: I05d164b290929fa06b6d78c01075ee41f8238044
2020-05-12 16:38:06 -04:00
NickolaiYurchenko
4b4222c2f9 web/storagenode: disposed held added to total held calculations
Change-Id: I593c9ce70c17b3b7a5ff0485a375fe2050ca8451
2020-05-12 17:51:17 +00:00
igor gaidaienko
1eab5e2980 satellite/console: Increase default webUI rate limit to 5
Previous limit is annoying for normal users

Change-Id: I7cb783e0b2515f415b2a055d5e811efab3810654
2020-05-12 16:12:17 +00:00
Egon Elbre
0e3be60b79 satellite/satellitedb: simplify migrate step
Change-Id: Ie4574144fb6ddd057d5fca740702c59fbdb2c5e4
2020-05-12 18:27:07 +03:00
Michal Niewrzal
0e263f0099 satellite/admin: add API endpoint for adding projects
Change-Id: I7970aa8bec810ee36727446d632a950c836f0597
2020-05-12 14:55:41 +00:00
Ivan Fraixedes
c58dccb7a8
cmd/segment-reaper: Remove segments of skipped objects
Reorganize some operations in the observer.processSegment method:

* to minimal reduce the memory usage removing the segments of the
  objects marked as skipped. Skipped objects aren't discarded of the
  analysis stage so the segments aren't needed.
* returns earlier when an object is skipped because isn't needed a
  further processing.

Change-Id: I210a26c394477ee411ff7f640507dcc07733a47f
2020-05-12 16:12:44 +02:00
Ivan Fraixedes
9314092841
cmd/segment-reaper: Rename Bitmask to BitArray
Rename a test name and an error class because the bitmask was renamed to
bitArray in a previous commit, however, they weren't updated.

Change-Id: I92afcf747dec9e0a15d4c59b896e07b942f3518b
2020-05-12 15:22:30 +02:00
Ivan Fraixedes
50e8314279 cmd/segment-reaper: Fix bitArray.IsSequence method
The bitArray.IsSequence method had a bug that caused that sometimes the
test "Bitmask/IsSequence/sequence starts at other index than 0" to fail
due to it uses random values.

The test mostly found a corner case, I've added new tests to catch the
bug and then apply the fix.

Change-Id: If01dc07006d261ba40bbd99d36ef776e09ed3efc
2020-05-12 13:02:52 +00:00
Stefan Benten
e23bd806b4
satellite/accounting: separate usage and bandwidth limit (#3878) 2020-05-12 15:01:15 +02:00
Michal Niewrzal
0619f97e20 satellite/console: remove unused arguments
Change-Id: I3ef2b244137391daafee209e20cec383c6772e2e
2020-05-12 11:20:13 +00:00
Stefan Benten
65f3e26f80
satellite: Change Default Project Limits and minimum STORJ Payment (#3877) 2020-05-12 14:18:58 +03:00
Kaloyan Raev
073f9ed499 satellite/payments: deduct stripe discount first
We have 3 types of discounts:
1) Promotional credits/coupons
2) Bonus from depositing STORJ tokens
3) Stripe discounts (e.g. 100% off for Storjlings, 30% off for Early
Adopters, etc.)

So far the discounts were applied in the above order. But because the
Stripe discount is applied on all of the project usage fees, this could
sometimes lead to negative total in the invoice. Especially, if the
Stripe discount is 100% or all of the project fees are covered by
coupons and bonus.

To resolve this issue, before applying promotional coupons and deposit
bonuses, the Stripe discount will be applied first to the project fees.

Change-Id: I5dcbec04ec3a04e7f76b11e0a228ccb3195f2db0
2020-05-12 09:26:27 +00:00
Egon Elbre
93f1fe49e3 mod: update to latest common
Change-Id: Ie938b40c82ce22d017a0566f03fd32dbef7d6829
2020-05-11 17:26:12 +00:00
Kaloyan Raev
b397dfc82d satellite/payments: better labels for discounts in invoices
Before:
- Discount from coupon: Promotional credits (limited time -
2 billing periods)
- Discount from credits

After:
- Promotional credits (limited time - 2 billing periods)
- Credits from STORJ deposit bonus

This way we don't mix the terms coupon and credit. And it is clearer
when the credit comes from a deposit bonus.

Change-Id: I4bba76a5501147f9de399eac41c4f157d6bda032
2020-05-11 16:12:04 +00:00
Kaloyan Raev
e0cf9ae888 storagenode/orders: set devDefault sender interval to 30s
Change-Id: Ic2877da12b5985b73d1565e8ccd65bc817a76448
2020-05-11 18:55:56 +03:00
Michal Niewrzal
22fbe804e3 satellite/accounting: test if project bandwidth limits reset with
billing cycle

https://storjlabs.atlassian.net/browse/USR-287

Change-Id: I4dc5f6342417b6af3384da32d3d2ed8592904406
2020-05-11 15:11:53 +00:00
Egon Elbre
ec589a8289 all: fix comments about grpc
Change-Id: Id830fbe2d44f083c88765561b6c07c5689afe5bd
2020-05-11 13:05:34 +03:00
Egon Elbre
7d29f2e0d3 all: remove drpc wrappers
Change-Id: I45016f7d2a771dc00776196c1f531f3343e93b40
2020-05-11 08:20:34 +03:00
Egon Elbre
e6d5ce6b77 all: remove grpc
It seems everyone has migrated to drpc.

Change-Id: Ica6b2d0bdef68c6603083f2963458843eca71e9e
2020-05-10 06:36:09 +00:00
Bill Thorp
4b612a7906 cmd/segment-reaper: add support for >65 segments
Change the bitmask used by segment reaper to use []byte rather than uint64

This passes tests but I have literally no clue how to integration test this.

Change-Id: I393f4598b27cae6e427da2190dd3109bca721c34
2020-05-09 10:45:26 +00:00
Egon Elbre
bcd93ee375 private/testplanet: add StopNodeAndUpdate
This was commonly used and code with it can be simplified.

Change-Id: I2f2b91f7de54269aee6ef027f97f9e8a7d222e39
2020-05-08 13:02:19 +00:00
Kaloyan Raev
12fa505692 satellite/payments: fix invoice amount in billing history
The billing history currently shows the Total amount from the Stripe
invoice. In fact, this value is just the amount deducted from the Stripe
balance. It does not reflect any deduction from promotional coupon or
bonus credits.

This patch adds the deducted amount form the promotional coupons and
bonus credits to the displayed amount in billing history. This way
customers have better understading of the total amount deducted from
their account balance on the satellite.

Change-Id: Ibd7f611a6cea0143a3059f39dd1d9ef21c264d8c
2020-05-08 11:43:41 +00:00
Matt Robinson
94503db5a6
cmd/satellite: Enable admin service like other services, also sort environment variables (#3874)
Change-Id: I67592b7bf34a0e2ce9a445b58b7fa34396aa8c5d

Co-authored-by: Stefan Benten <mail@stefan-benten.de>
2020-05-08 07:38:21 -04:00
Egon Elbre
a549d523dc bump drpc and common
Change-Id: I231d72793cc07cefc59d013fce02c03d5638eabc
2020-05-08 10:50:57 +03:00
Egon Elbre
678b859172 satellite/overlay: remove MinimumRequiredNodes
In non-test code we were only using RequestedCount, not need to have
MinimumRequiredNodes.

Change-Id: I40736f4b028b41e94abfdeb221bce5aa86a5cb82
2020-05-07 15:41:23 +00:00
Kaloyan Raev
2af7138928 web/satellite: change invoice link title to 'Invoice PDF'
Change-Id: I820bb33f4ba88bb732d1c66ba3369ac4eac72f5c
2020-05-07 18:14:40 +03:00
VitaliiShpital
9837230f1e web/satellite: project dropdown reworked
Change-Id: I6267347693724ec34996af7625a917451aef631d
2020-05-07 12:51:34 +00:00
Egon Elbre
ce6a500b0c satellite/overlay: support DistinctIP=false in selection cache
Most of our tests and storj-sim are using DistinctIP.

Also fix bug with newNodeCount calculation.

Change-Id: I1a6d0efe7074908896a3322d19f487b929f0f0fc
2020-05-07 11:04:32 +00:00