Commit Graph

1197 Commits

Author SHA1 Message Date
Egon Elbre
b42778c42e private/testplanet: remove some additional Local-s
Change-Id: I49701c41efb92efca27cc18d0a3f6d6b44d3cf8b
2020-05-21 08:37:13 +03:00
Egon Elbre
7f323754a4 metainfo/piecedeletion: use NodeURL-s
Change-Id: I247dbfe03e7864e940e4cd1d0f343f38e84099e0
2020-05-21 08:37:13 +03:00
Jennifer Johnson
03e5f922c3 satellite/overlay: updates node with a vetted_at timestamp if they meet the vetting criteria
What: As soon as a node passes the vetting criteria (total_audit_count and total_uptime_count
are greater than the configured thresholds), we set vetted_at to the current timestamp.

Why: We may want to use this timestamp in future development to select new vs vetted nodes.
It also allows flexibility in node vetting experiments and allows for better metrics around
vetting times.

Please describe the tests: satellitedb_test: TestUpdateStats and TestBatchUpdateStats make sure vetted_at is set appropriately
Please describe the performance impact: This change does add extra logic to BatchUpdateStats and UpdateStats and
commits another variable to the db (vetted_at), but this should be negligible.

Change-Id: I3de804549b5f1bc359da4935bc859758ceac261d
2020-05-20 16:30:26 -04:00
Michal Niewrzal
fe6a6f063f private/testplanet: cleanup predefined data generation
Use Console service to create user and project instead direct DB
modifications.

Change-Id: Ib0074b38313b3dc43b7d8d63ab2775d29028fb7b
2020-05-20 12:38:43 +00:00
Qweder93
49ad90dcd8 storagenode/reputation: unknown_score (unknown_alpha / unknown_alpha+unknown_beta) added to reputation stats, https://storjlabs.atlassian.net/browse/SG-326
Change-Id: I0b29ad736f7a11c7e57a846b6891f4b40755aa48
2020-05-20 11:25:14 +00:00
Egon Elbre
5d016425f1 satellite/{contact,downtime,overlay}: use NodeURL
Change-Id: I555a479a89e0ddbf0499898bdbc8574282cd6846
2020-05-20 11:09:05 +00:00
Egon Elbre
941d10cbc3 private/testplanet: remove Peer.Local()
Currently storagenode depends on overlay.NodeDossier, this is the first
step in removing it.

Change-Id: I034a3f1601835f8349bd41752455022e19bcc707
2020-05-20 11:05:34 +00:00
Egon Elbre
94b2b315f7 storagenode/trust: refactor GetAddress to GetNodeURL
Most places now need the NodeURL rather than the ID and Address
separately. This simplifies code in multiple places.

Change-Id: I52621d8ca52296a8b5bf7afbc1001cf8bfb44239
2020-05-20 11:05:15 +00:00
Egon Elbre
ed627144ed all: use DialNodeURL throughout the codebase
Change-Id: Iaf9ae3aeef7305c937f2660c929744db2d88776c
2020-05-20 10:36:30 +00:00
Stefan Benten
1a4ab3df48
satellite/admin: add user update endpoint (#3896) 2020-05-20 12:20:04 +03:00
littleskunk
8ec64f3daf
satellite/overlay: enable node selection cache on all satellites (#3895) 2020-05-19 19:25:53 +02:00
Jess G
6fac1ed130
satellite/contact: rm unhelpful verbose logging (#3894)
* unhelpful rm verbose logging

Change-Id: I74dc29f18969250fa6838fed9bec8ba7ea160b77

* change to debug

Change-Id: I4ba880e68b40bc3e0f29d9b1aa9154e63124c904
2020-05-19 08:09:00 -07:00
Stefan Benten
0a26c4af9a
satellite/admin: add coupon deletion (#3893) 2020-05-19 15:49:44 +03:00
Stefan Benten
671aca56b0
satellite/admin: add coupon creation and listing (#3891) 2020-05-19 12:36:13 +02:00
Michal Niewrzal
ac375d37bc satellite/payments: remove mockpayments and add Stripe client mock
instead

Change-Id: If3496f6abc16da90d2b43fa0c5be356847a39507
2020-05-19 09:35:37 +02:00
Egon Elbre
16abf02b35 satellite/{nodeselection,overlay}: use the new package
Change-Id: I034fdbe578dec2e5c906aca82231cd3e56f26aeb
2020-05-18 21:38:43 +00:00
Stefan Benten
17d3cb1551
satellite/admin: add user creation endpoint (#3889) 2020-05-18 22:37:18 +02:00
Egon Elbre
08692aef90 satellite/nodeselection: node selection with proper bias
Currently node selection cache is biased towards the same subnet. This
implements static node selection for distinct such that it selects with
equal probability subnets rather than id-s.

This is mostly a copy paste + modifications from previous node selection
state.

Change-Id: Ia5c0aaf68e7feca78fbbd7352ad369fcb77c3a05
2020-05-18 18:09:15 +00:00
littleskunk
1b06e2a928
satellite/metainfo: remove deprecated error validation (#3876) 2020-05-18 19:58:42 +02:00
Stefan Benten
70502658c8
satellite/admin: add delete project endpoint (#3888) 2020-05-18 19:36:09 +02:00
Jessica Grebenschikov
1ecbfc453f satellite/overlay: add random test for node selection from cache
Change-Id: I57b57db919b5afb3fc81176e078ac1e3a873565a
2020-05-18 16:59:52 +00:00
Kaloyan Raev
aac1e3c45f satellite/payments: move inspector commands to satellite cli
This allows to seeing logs in the output of the invoice commands.

Existing ensure-stripe-customer commands is moved from the 'reports' to
the new 'billing' root command.

Change-Id: I752c7ab6ca59bfac8e0f174a45d2ab45fc18e467
2020-05-18 16:12:45 +00:00
VitaliiShpital
47def02094 satellite: account balance divided into Free Credits and Coins
Change-Id: I2ae29701bab9f0b48f236ff94b366eb534b81c6b
2020-05-18 14:25:26 +00:00
Michal Niewrzal
9fb423ac19 satellite/payments/stripecoinpayments: fix listing issue in
InvoiceApplyProjectRecords

ListUnapplied method in listing loop  was using next offset as starting
point but applyProjectRecords method was changing project record state
to applied and we were missing some records to apply.

Test will be added as a separate change.

Change-Id: Id1ca33eeb66ec7f6ff1f05b45615a8935185568e
2020-05-18 13:11:30 +00:00
Egon Elbre
f6301612ac satellite/orders: use serial SerialNumber
By ensuring that they have less randomness it means that they can be
compressed better. Using a timestamp should be a good improvement here.

Change-Id: Ic4dabb53335a744ff1c332dd279f37ae2cd79357
2020-05-15 11:33:53 +00:00
Michal Niewrzal
2eb2c25e51 satellite/payments/stripecoinpayments: add StripeClient interface to be
able to cover more testing scenarios

Currently, its hard to implement test suite for payments because
mockpayments is on to high level and we cannot emulate many things e.g.
adding credit card. This change is first to be able to add mock for
Stripe client and do more granular tests.

Change-Id: Ied85d4bd0642debdffe1161657c1e475202e9d23
2020-05-15 10:52:44 +02:00
littleskunk
28a886f399
satellite/admin: obvious project name in documentation 2020-05-14 15:50:58 +03:00
Kaloyan Raev
49571f1a23 satellite/payments: all invoice commands require period
To avoid including multiple months in a single invoice, we need all
inspector's invoice commands to run in for specific period.

See https://storjlabs.atlassian.net/browse/USR-725

Change-Id: I3637dc189234f02350daca8d897c21765762ea55
2020-05-14 11:50:19 +00:00
Moby von Briesen
46df8c1977 satellite/gracefulexit: add log message when node fails validation for piece transfer
Change-Id: Ic5a53404ceb35003793aebc63637e7f8a58ef259
2020-05-13 16:58:50 +00:00
Jeff Wendling
6352d46100 satellite/satellitedb: do better ::date conversions
There is a subtle problem when one does a cast with `::date`. Observe:

    teststorj=# set timezone = 'US/Eastern';
    SET
    teststorj=# select (timestamp with time zone '2020-02-01 00:00:00+00')::date;
        date
    ------------
     2020-01-31
    (1 row)

    teststorj=# set timezone = 'UTC';
    SET
    teststorj=# select (timestamp with time zone '2020-02-01 00:00:00+00')::date;
        date
    ------------
     2020-02-01
    (1 row)

In order to correctly determine the date a timestamp is in, one has to
explicitly pick the time zone that the date truncation should use
otherwise postgres will use whatever setting the client has. These
tests were failing for me locally, because I run my postgres in
the US/Eastern time zone to try to tickle these bugs out. So it
should be `(x at time zone 'UTC')::date` instead of just `x::date`.

Change-Id: I4e9e32d4b53abc6165a4d0474f4702f8b9f801c7
2020-05-13 15:58:07 +00:00
igor gaidaienko
75ddecdf54 satellite/metainfo: Add test to verify payment report for attributed bucket
For this automated test I want to make sure the data in payment report is correct

Change-Id: I48a3b0e1dc9b2324435b763c87c363f1e310ff4e
2020-05-13 17:22:04 +03:00
Bill Thorp
5a7a4d2e98 satellite: add Go test version of satellite-config-lock tests
The current satellite config lock code relies on bash scripts and
gnu diff, it must be run as root and hence it typically requires
docker.  The old version will be removed at a later date..

I tried for several hours to run directly against cmdSetup() in
cmd/satellite/main.go, to avoid the ctx.Compile() call. I had no
luck.

Change-Id: I0a4888421e743b436d32b6af69d04759d7816751
2020-05-13 08:14:24 +00:00
Ethan
159df8b2e4 Add logging listener for retrieving and setting log levels
See https://storjlabs.atlassian.net/browse/SM-752

These changes allow us to change the log level at runtime through a handler off of the debug endpoint.

Examples of changing the log level on storj-sim

To get the current level for the satellite api process:
curl -XGET 'http://127.0.0.1:10009/logging' --header 'Content-Type: text/plain'

To change the log level:
curl -XPUT 'http://127.0.0.1:10009/logging' --header 'Content-Type: text/plain' --data-raw '{"level":"error"}'

Change-Id: I05d164b290929fa06b6d78c01075ee41f8238044
2020-05-12 16:38:06 -04:00
Egon Elbre
0e3be60b79 satellite/satellitedb: simplify migrate step
Change-Id: Ie4574144fb6ddd057d5fca740702c59fbdb2c5e4
2020-05-12 18:27:07 +03:00
Michal Niewrzal
0e263f0099 satellite/admin: add API endpoint for adding projects
Change-Id: I7970aa8bec810ee36727446d632a950c836f0597
2020-05-12 14:55:41 +00:00
Stefan Benten
e23bd806b4
satellite/accounting: separate usage and bandwidth limit (#3878) 2020-05-12 15:01:15 +02:00
Michal Niewrzal
0619f97e20 satellite/console: remove unused arguments
Change-Id: I3ef2b244137391daafee209e20cec383c6772e2e
2020-05-12 11:20:13 +00:00
Stefan Benten
65f3e26f80
satellite: Change Default Project Limits and minimum STORJ Payment (#3877) 2020-05-12 14:18:58 +03:00
Kaloyan Raev
073f9ed499 satellite/payments: deduct stripe discount first
We have 3 types of discounts:
1) Promotional credits/coupons
2) Bonus from depositing STORJ tokens
3) Stripe discounts (e.g. 100% off for Storjlings, 30% off for Early
Adopters, etc.)

So far the discounts were applied in the above order. But because the
Stripe discount is applied on all of the project usage fees, this could
sometimes lead to negative total in the invoice. Especially, if the
Stripe discount is 100% or all of the project fees are covered by
coupons and bonus.

To resolve this issue, before applying promotional coupons and deposit
bonuses, the Stripe discount will be applied first to the project fees.

Change-Id: I5dcbec04ec3a04e7f76b11e0a228ccb3195f2db0
2020-05-12 09:26:27 +00:00
Kaloyan Raev
b397dfc82d satellite/payments: better labels for discounts in invoices
Before:
- Discount from coupon: Promotional credits (limited time -
2 billing periods)
- Discount from credits

After:
- Promotional credits (limited time - 2 billing periods)
- Credits from STORJ deposit bonus

This way we don't mix the terms coupon and credit. And it is clearer
when the credit comes from a deposit bonus.

Change-Id: I4bba76a5501147f9de399eac41c4f157d6bda032
2020-05-11 16:12:04 +00:00
Michal Niewrzal
22fbe804e3 satellite/accounting: test if project bandwidth limits reset with
billing cycle

https://storjlabs.atlassian.net/browse/USR-287

Change-Id: I4dc5f6342417b6af3384da32d3d2ed8592904406
2020-05-11 15:11:53 +00:00
Egon Elbre
ec589a8289 all: fix comments about grpc
Change-Id: Id830fbe2d44f083c88765561b6c07c5689afe5bd
2020-05-11 13:05:34 +03:00
Egon Elbre
7d29f2e0d3 all: remove drpc wrappers
Change-Id: I45016f7d2a771dc00776196c1f531f3343e93b40
2020-05-11 08:20:34 +03:00
Egon Elbre
e6d5ce6b77 all: remove grpc
It seems everyone has migrated to drpc.

Change-Id: Ica6b2d0bdef68c6603083f2963458843eca71e9e
2020-05-10 06:36:09 +00:00
Egon Elbre
bcd93ee375 private/testplanet: add StopNodeAndUpdate
This was commonly used and code with it can be simplified.

Change-Id: I2f2b91f7de54269aee6ef027f97f9e8a7d222e39
2020-05-08 13:02:19 +00:00
Kaloyan Raev
12fa505692 satellite/payments: fix invoice amount in billing history
The billing history currently shows the Total amount from the Stripe
invoice. In fact, this value is just the amount deducted from the Stripe
balance. It does not reflect any deduction from promotional coupon or
bonus credits.

This patch adds the deducted amount form the promotional coupons and
bonus credits to the displayed amount in billing history. This way
customers have better understading of the total amount deducted from
their account balance on the satellite.

Change-Id: Ibd7f611a6cea0143a3059f39dd1d9ef21c264d8c
2020-05-08 11:43:41 +00:00
Egon Elbre
678b859172 satellite/overlay: remove MinimumRequiredNodes
In non-test code we were only using RequestedCount, not need to have
MinimumRequiredNodes.

Change-Id: I40736f4b028b41e94abfdeb221bce5aa86a5cb82
2020-05-07 15:41:23 +00:00
Egon Elbre
ce6a500b0c satellite/overlay: support DistinctIP=false in selection cache
Most of our tests and storj-sim are using DistinctIP.

Also fix bug with newNodeCount calculation.

Change-Id: I1a6d0efe7074908896a3322d19f487b929f0f0fc
2020-05-07 11:04:32 +00:00
Egon Elbre
c5452a87ec private/testplanet: use drpc referral manager server
Change-Id: I9e9e9a724c78c98859dd3e29416d766d8ffdca63
2020-05-07 07:03:11 +00:00
Egon Elbre
2955c50bc1 satellite/overlay: fix data-race in node selection cache
GetNodes returned references to nodes in the immutable state, however
some parts of code expect them to be modified.

Change-Id: I5be1866f95e0dbe062a6b6be60e29f2365c35faa
2020-05-06 20:03:06 +03:00