Commit Graph

249 Commits

Author SHA1 Message Date
Bill Thorp
b265b7f555 satellite/console: make paywall optional
Add a config so that some percent of users require credit cards /
account balances
in order to create a project or have a promotional coupon applied

UI was updated to match needed paywall status

At this point we decided not to use a field to store if a user is in an
A/B
test, and instead just use math to see if they're in a test.  We decided
to use MD5 (because its in Postgres too) and User UUID for that math.

Change-Id: I0fcd80707dc29afc668632d078e1b5a7a24f3bb3
2020-07-28 10:57:49 +00:00
Qweder93
92efffb48a storagenode/version: notification flow now based on cursor, chore_test added, versioncontrol added to reconfigure.
Change-Id: I70713def8d585228270ec5a8c586ecc5b4d510c4
2020-07-23 14:13:24 +00:00
Ethan
cfca021839 satellite/accounting: Add chore to cleanup old project bandwidth rollups data
Removes old project_bandwidth_rollups records that are no longer used.

Uses a retain months configuration to determine how many months to save.  Current month cannot be removed.
Tests retainMonths=-1, 0, 2

Change-Id: Ia4be2546cdb28802427acf41ecd85ad66df3e62c
2020-07-22 18:56:49 +00:00
paul cannon
fd7bfc94fe private/dbutil: don't sort column names in an index
The order in which column names appear in an index should be
deterministic (for both our sqlite and postgresql code). Also, the order
is very relevant as to whether a given schema is correct.

Change-Id: I227ea057fcd9c3e967dd241a7e1c787d1bc4baa1
2020-07-17 10:07:01 +00:00
Egon Elbre
b84923558b satellite: fix scoping, formatting
Change-Id: I21ef9edc2d449d75ad74891df7f966fb150d80fd
2020-07-16 19:13:14 +03:00
Egon Elbre
e70da5cd4e all: fix comments
Change-Id: I2d2307e3fab87de47a72b3595d051e2c95ff4f8a
2020-07-16 19:13:14 +03:00
Egon Elbre
080ba47a06 all: fix dots
Change-Id: I6a419c62700c568254ff67ae5b73efed2fc98aa2
2020-07-16 14:58:28 +00:00
stefanbenten
9ace375ee0 satellite/{console,satellitedb}: change project limiting based on new users field
This change switches the backend logic to use the new DB column on the users table to restrict project creation.
Furthermore it back fills the existing limits from registration tokens to the new column to ensure no users are reset to the new default.

UI is updated to reflect ability to create several projects

Change-Id: Ie29157430ae6b065411ca4c4557c9f1be69cdc4f
2020-07-16 10:57:47 +00:00
Jennifer Johnson
784a156eea satellite: prevents uplink from creating a bucket once it exceeds the max bucket allocation.
Change-Id: I4b3822ed723c03dbbc0df136b2201027e19ba0cd
2020-07-15 17:27:05 +00:00
stefanbenten
257855b5de all: replace == comparison with errors.Is
Change-Id: I05d9a369c7c6f144b94a4c524e8aea18eb9cb714
2020-07-14 15:50:25 +00:00
stefanbenten
1149417615 satellite/admin: cleanup parameter handling
We passed in revocationDB and metainfoDB for no reason.
Lets remove it from the dependency list to further reduce the footprint.

Change-Id: Ic0317bb92670fbd305d4a8b0ed1cb82858e2f6d3
2020-07-14 13:53:09 +02:00
Jessica Grebenschikov
8abb907010 satellite/orders: add settle orders with window
Why: We need a way to cut down on database traffic due to bandwidth
measurement and tracking.

What: This changeset is the Satellite side of settling orders in 1 hr windows.
See design doc for more details: https://review.dev.storj.io/c/storj/storj/+/1732

Change-Id: I2e1c151e2e65516ebe1b7f47b7c5f83a3a220b31
2020-07-13 15:41:29 -07:00
paul cannon
bbdb351e5e all: use jackc/pgx in place of lib/pq
What:

Use the github.com/jackc/pgx postgresql driver in place of
github.com/lib/pq.

Why:

github.com/lib/pq has some problems with error handling and context
cancellations (i.e. it might even issue queries or DML statements more
than once! see https://github.com/lib/pq/issues/939). The
github.com/jackx/pgx library appears not to have these problems, and
also appears to be better engineered and implemented (in particular, it
doesn't use "exceptions by panic"). It should also give us some
performance improvements in some cases, and even more so if we can use
it directly instead of going through the database/sql layer.

Change-Id: Ia696d220f340a097dee9550a312d37de14ed2044
2020-07-13 15:54:41 +00:00
Egon Elbre
9dc9cd8a17 tests: allow STORJ_TEST_POSTGRES
STORJ_POSTGRES_TEST naming was not consistent with STORJ_SIM_POSTGRES.

This allows to use STORJ_TEST_POSTGRES for clarity, it still has a
fallback to STORJ_POSTGRES_TEST.

Change-Id: I6f294c66c80fcfd6750fea2a89795f3b7f5dd691
2020-07-10 16:43:49 +03:00
Egon Elbre
4869cfc9a4 satellite/vouchers: remove deprecated endpoint
Change-Id: I0a754217d9424253e448126face6594bc143f412
2020-07-10 12:38:46 +00:00
Stefan Benten
9dbd511396
private/dbutil: reduce db connection defaults (#3920) 2020-07-08 19:59:42 +02:00
Qweder93
0521435e08 storagenode/gracefulexit: added deletion of all files left in storage/blobs/satellite after successful GE
https://storjlabs.atlassian.net/browse/SG-368

Change-Id: I29a978fe0d0153aedf2be91dc7f45b4ef386d447
2020-07-08 14:38:31 +03:00
Bill Thorp
4a98c9514c private/date: fix MonthsCountSince build issue
Change-Id: I58a70ea85f966dece4b3c75f54cfaa5238f9ecd9
2020-06-30 17:47:18 -04:00
Cameron Ayer
cadb435d25 {satellite/audit, private/testplanet}: remove ErrAlreadyExists, run 2 audit workers in testplanet
Since we increased the number of concurrent audit workers to two, there are going
to be instances of a single node being audited simultaneously for different segments.
If the node times out for both, we will try to write them both to the pending audits
table, and the second will return an error since the path is not the same as what
already exists. Since with concurrent workers this is expected, we will log the
occurrence rather than return an error.

Since the release default audit concurrency is 2, update testplanet default to run with
concurrent workers as well.

Change-Id: I4e657693fa3e825713a219af3835ae287bb062cb
2020-06-30 18:00:07 +00:00
Egon Elbre
13a5854535 satellite/satellitedb: clarify test migration merging
Use a field to distinguish migration steps that need to use a
different transaction from previous steps. This is clearer than
using a func.

Change-Id: I2147369d05413f3e8ddb50c71a46ab1ba3ab5114
2020-06-25 14:32:45 +00:00
Rafael Gomes
958ea1b9df satellite/accounting: add download limit cache
Change-Id: I722930cab8bd5d240f4878dc6997e9bc7637311f
2020-06-12 16:33:46 -03:00
Egon Elbre
1ed5a1bac5 satellite/satellitedb/satellitedbtest: skip omitted database
The first implementation missed some changes.

Change-Id: I7ae696175e0a9ea46954970ba8547638a05ed5a9
2020-06-11 13:28:16 +00:00
Ivan Fraixedes
dc5502cb81 private: Prepare pkg for enabling gosec
Prepare package for enabling gosec linter.

Change-Id: I0cce91d83969385f95e5bf82269d6c23629e04a0
2020-06-11 12:00:52 +00:00
Egon Elbre
1c30efd3a1 private/testplanet: allow setting "omit" as database to reduce output
Change-Id: I7af90fdefe2ff2df1340aa2b17f40806d889ca18
2020-06-09 12:41:58 +03:00
Egon Elbre
36c461bd59 private/tagsql: track proper closing of rows and statements
This ensures that rows are closed to avoid leaks.
Also verifies that Err() is called, to ensure that no
error is left behind.

Change-Id: Idd1bec9bf479f40021da67b2c80ce83033149469
2020-06-05 18:25:43 +00:00
Egon Elbre
10f8b5492c Revert "private/tagsql: add finalizer based leak checks during dev"
This reverts commit c6310b34d2.

The change was causing data-races that are hard to deal with.

Change-Id: I0d29d85af70dce7ee2e967b9d7854719b32cf216
2020-06-05 17:52:46 +03:00
Yaroslav Vorobiov
09ca382abf storagenode/db: preflight improve index discovery
Change-Id: I876b321f6cd4e91dfced87aa4d39f2cf9a8e63d0
2020-06-05 14:03:25 +03:00
Jeff Wendling
c6310b34d2 private/tagsql: add finalizer based leak checks during dev
what would win? thousands of man-hours spent trying to make
the best, most bug-free code possible, or one leaky boi?

this way we hopefully reduce the number of times we deadlock
everything by forgetting a single rows.Close.

Change-Id: I191727bbb75f74f5f4d0664e9e7b6ccf46c931f5
2020-06-03 15:06:58 -06:00
Moby von Briesen
b82d04e618 satellite/metainfo: limit size of uplink-provided metadata to 2KiB
Change-Id: Id44a46046ddb4a12102525531f4502fcff2b6252
2020-06-01 16:51:29 -04:00
Qweder93
89c9672ce0 storagenode/piecestore: available storage check added in Upload
Change-Id: I71e9e5f335d4320d5de8b374fe747fec43179f78
2020-06-01 16:55:22 +00:00
Michal Niewrzal
21518bcc92 private/testuplink: move tests to uplink
Tests will be deleted from storj repo and added to uplink.

Change-Id: I298d852325c8eb0df07df38fd7e1345623addd8d
2020-06-01 12:29:21 +02:00
Ethan
b1bb665c78 satellite/metainfo: Handle "server is not accepting clients" error during CRDB node rejoins
https: //storjlabs.atlassian.net/browse/SM-1035
Change-Id: I27243b0d8fc3250916c86ceb915f973cbf80f656
2020-05-29 16:21:56 +00:00
Moby von Briesen
dc57640d9c storagenode/piecestore: switch usedserials db for in-memory usedserials store
Part 2 of moving usedserials in memory
* Drop usedserials table in storagenodedb
* Use in-memory usedserials store in place of db for order limit
verification
* Update order limit grace period to be only one hour - this means
uplinks must send their order limits to storagenodes within an hour of
receiving them

Change-Id: I37a0e1d2ca6cb80854a3ef495af2d1d1f92e9f03
2020-05-28 12:52:52 -04:00
Michal Niewrzal
84892631c8 private/testplanet: remove old libuplink from testplanet
Change-Id: Ib1553f84d0b3ae12a5b00382f0f53357b6a273e2
2020-05-28 13:50:23 +00:00
Qweder93
8db848791f storagenode/console: added estimated payout for current month and estimated pay stub for previous month (until there's real data in satellite's table) + heldback percentage rate for previous month.
Change-Id: I9346f6d22ed6fbb7e5346b102fc898467678f384
2020-05-27 14:51:23 +03:00
Natalie Villasana
8bd4d7b43e storage/cockroachkv: add check if retry is needed during iteration
This changeset replaces https://review.dev.storj.io/c/storj/storj/+/1839
which did the same thing but Nat couldn't figure out how to fix conflicting
files the correct gerrity way.

Change-Id: If05a8902aca986ea9f6c9168a90b31beebab839a
2020-05-26 14:32:06 -04:00
Jeff Wendling
074649835b satellite/satellitedb: add some docs and improve some snapshots
This attempts to add a README.md to help create consistent migrations
that maximize our test coverage and do not include unnecessary
statements.

It also adds a feature to have an `-- OLD DATA --` section as well
as a `-- NEW DATA --` section so that we can fix mistakes made in
previous snapshots (like a row that was forgotten to be added when a
table was created) without editing them going forward.

Change-Id: I28a786f8ef163cae1de1bb08f61af1e1104b0a88
2020-05-22 21:27:36 +00:00
Michal Niewrzal
5c10964040 satellite/payments/stripecoinpayments: add test for listing issues while
invoice generation

https://review.dev.storj.io/c/storj/storj/+/1853
https://review.dev.storj.io/c/storj/storj/+/1882

Change-Id: Ie71363b819866dd60dbe7117b42cfa8348479310
2020-05-22 17:24:16 +00:00
Michal Niewrzal
3d332de228 private/testplanet: remove StorageNodeCount from testplanet uplink
definion

Small cleanup.

Change-Id: Icabdf1433c36cd0e9f8e10a67975e98391024e14
2020-05-21 14:51:58 +00:00
Egon Elbre
bef84a5f9d storagenode: remove dependency to overlay.NodeDossier
This is the last dependency from storage node to satellite.

Change-Id: I12f7abb91e84f823ba5af126c6e2979519838612
2020-05-21 08:37:13 +03:00
Egon Elbre
b42778c42e private/testplanet: remove some additional Local-s
Change-Id: I49701c41efb92efca27cc18d0a3f6d6b44d3cf8b
2020-05-21 08:37:13 +03:00
Natalie Villasana
2514d6328d dbutil/cockroachutil: add monkit to QueryContext
This will help us keep track of crdb errors in influx.

Change-Id: I997596aa4eb9a2b9b81305d123c3452ecdf5dde5
2020-05-20 14:56:25 -04:00
Bill Thorp
f43cb1688d private/tagsql: verify SQL connection with ping
Use ping to make sure the database connection is valid

Change-Id: I5217e28e186f487266c8f4a1d39cce0070dc1465
2020-05-20 13:12:16 +00:00
Michal Niewrzal
fe6a6f063f private/testplanet: cleanup predefined data generation
Use Console service to create user and project instead direct DB
modifications.

Change-Id: Ib0074b38313b3dc43b7d8d63ab2775d29028fb7b
2020-05-20 12:38:43 +00:00
Egon Elbre
941d10cbc3 private/testplanet: remove Peer.Local()
Currently storagenode depends on overlay.NodeDossier, this is the first
step in removing it.

Change-Id: I034a3f1601835f8349bd41752455022e19bcc707
2020-05-20 11:05:34 +00:00
Egon Elbre
ed627144ed all: use DialNodeURL throughout the codebase
Change-Id: Iaf9ae3aeef7305c937f2660c929744db2d88776c
2020-05-20 10:36:30 +00:00
Michal Niewrzal
705e82ea99 private/testplanet: add AddUser and AddProject to satellite
functionality

We want to start adding more complex test cases for billing/invoices and
we need more handy tooling to be able do this easily.

Change-Id: Ib22ac6b4ba9ee77cc91c88b0cfd2d2efc15657df
2020-05-19 13:02:04 +00:00
Michal Niewrzal
ac375d37bc satellite/payments: remove mockpayments and add Stripe client mock
instead

Change-Id: If3496f6abc16da90d2b43fa0c5be356847a39507
2020-05-19 09:35:37 +02:00
Natalie Villasana
8d87a6efc9 cockroachutil/driver: handle retryable errors returned from Next
This will only work if retryable errors are returned on the first
call to Next. Otherwise if they're returned later, we will need
deeper changes at the application code level throughout the
codebase 😬👎

Change-Id: I46d795a13670f66b7f085605ba1b779f69c339c3
2020-05-15 14:49:43 -04:00
littleskunk
ef2671927d
storagenode/piecestore: move queue size defaults (#3881) 2020-05-15 19:10:26 +02:00