Commit Graph

592 Commits

Author SHA1 Message Date
Egon Elbre
59d06644b9 private/migrate: switch to tagsql
Also added temporary types withRebind and withTagTx,
which will be later removed. Currently they help to avoid
changing the whole codebase at the same time.

Change-Id: I7f07ba8f4709a23a463bfa67464628665a05808f
2020-01-19 14:39:16 +02:00
Yaroslav
d8368d0b30 satellite/payments: coinpayments add completed status, treat received status as pending, add balance for completed transactions only
Change-Id: I20494bdddfda6d4f37ba2c5b6f7955cd29a6d798
2020-01-17 17:26:34 +00:00
Jessica Grebenschikov
955abd9293 satellite/satellitedb/orders: add multi row upserts to process orders
Change-Id: I00d8b55ee74b443fb328bd3a4378308cefa368e4
2020-01-16 23:51:46 +00:00
Isaac Hess
cd48dc369a satellite/satellitedb: Remove unused indexes
Change-Id: I875b94574eacf9d2df537bcf1f42f30e0bf60ab9
2020-01-16 16:06:21 -07:00
Jeff Wendling
47bb7a7a86 satellite/satellitedb/dbx: regenerate with default support
Change-Id: I0dab34f27af913795ef95ef92173844c2f53b29b
2020-01-16 22:13:57 +00:00
Jeff Wendling
696d98a232 satellite/satellitedb: fix nitpicks and timestamp issue found in review
warning: databases migrated to version 77 before this commit
is merged must be manually re-migrated. this should not be a
problem for anything but staging databases.

Change-Id: Ie1631c48379472352014183ee43f1465e22200f7
2020-01-16 21:22:38 +00:00
Egon Elbre
7d79aab14e satellite/satellitedb: fixes to row handling
Change-Id: I48fae692bcca152143a12f333296c42471538850
2020-01-16 17:07:26 +02:00
Jeff Wendling
78c6d5bb32 satellite/satellitedb: reported_serials table for processing orders
this commit introduces the reported_serials table. its purpose is
to allow for blind writes into it as nodes report in so that we have
minimal contention. in order to continue to accurately account for
used bandwidth, though, we cannot immediately add the settled amount.
if we did, we would have to give up on blind writes.

the table's primary key is structured precisely so that we can quickly
find expired orders and so that we maximally benefit from rocksdb
path prefix compression. we do this by rounding the expires at time
forward to the next day, effectively giving us storagenode petnames
for free. and since there's no secondary index or foreign key
constraints, this design should use significantly less space than
the current used_serials table while also reducing contention.

after inserting the orders into the table, we have a chore that
periodically consumes all of the expired orders in it and inserts
them into the existing rollups tables. this is as if we changed
the nodes to report as the order expired rather than as soon as
possible, so the belief in correctness of the refactor is higher.

since we are able to process large batches of orders (typically
a day's worth), we can use the code to maximally batch inserts into
the rollup tables to make inserts as friendly as possible to
cockroach.

Change-Id: I25d609ca2679b8331979184f16c6d46d4f74c1a6
2020-01-15 19:21:21 -07:00
Jeff Wendling
9da16b1d9e satellite/satellitedb/dbx: name the package dbx
everyone was importing it as dbx anyway. why should it be
named satellitedb? so yeah just pass the "-p dbx" flag.

Change-Id: I5efa669f4f00f196b38a9acd0d402009475a936f
2020-01-15 15:16:39 -07:00
Egon Elbre
64fb2d3d2f Revert "dbutil: statically require all databases accesses to use contexts"
This reverts commit 8e242cd012.

Revert because lib/pq has known issues with context cancellation.
These issues need to be resolved before these changes can be merged.

Change-Id: I160af51dbc2d67c5449aafa406a403e5367bb555
2020-01-15 07:28:00 +00:00
JT Olio
8e242cd012 dbutil: statically require all databases accesses to use contexts
this will allow for some nice runtime analysis down the road.
also, this allows for wrapping database handles in a way that
can interact with these contexts

requires https://review.dev.storj.io/c/storj/dbx/+/514

Change-Id: Ib087b7cd73296dd2c1e0331314da34d861f61d2b
2020-01-14 18:20:47 -05:00
crawter
a57ce18f58 satellite/payments: coupons, coupons usage, invoice generation with pricing model applied
Change-Id: Ic5d5a2fc116388647efe46896cfccc2038c77537
2020-01-14 12:45:00 +00:00
Jeff Wendling
3b99f03780 satellite/orders: add monitoring to bucket bandwidth cache operations
Change-Id: Ib14303fc9f97a133410e2d6e2cf532e468b3dcee
2020-01-13 17:36:40 -07:00
Isaac Hess
4950d7106a satellite/orders: Add write cache for bw rollups
Change-Id: I8ba454cb2ab4742cafd6ed09120e4240874831fc
2020-01-13 22:40:51 +00:00
Egon Elbre
ff267168c5 private/migrate: add ctx argument
Change-Id: I3d65912d89261386413c494c7ed1576fed4dcaf4
2020-01-13 15:52:26 +02:00
Egon Elbre
24958bd7d3 satellite: add ctx to DB.CreateTables
Change-Id: I9ecad624cf5a7fc9c86bb91c68f96a3a4efd2e92
2020-01-13 15:31:09 +02:00
Egon Elbre
0835b9024c private/dbutil/pgutil: add ctx argument
Change-Id: Icfd56ca8c1f831ad56c0195a0b883e8f0618daaf
2020-01-13 15:27:06 +02:00
littleskunk
bcc23f6869
Satellite/orders: remove allocated bandwith from storagenode_bandwidth_rollups
When an uplink requests an upload or download from the satellite we are trackig the
allocated bandwidth twice. The value in bucket_bandwidth_rollups is used
for project limits but the value in storagenode_bandwidth_rollups is not
used at all. We can increase the performance by removing it. Uplinks
will get a faster response from the satellite.

Change-Id: Icccd41f94107ef34668f30f99bf5f728c384b07e
2020-01-12 16:20:47 +01:00
Jeff Wendling
4aef0e3823 satellite/satellitedb: only reject orders if row not found
any database error doesn't mean the order wasn't found. for example
in cockroach it may say that the transaction is aborted. then what?
maybe we get big old row level deadlocks like we've observed? so
instead explicitly check for ErrNoRows to reject the order and bail
out otherwise. the surrounding logic will give it a retry.

Change-Id: I6e1f8f6e6a6def3e45b44f5088cbdc158e1098e4
2020-01-10 19:05:54 -07:00
Jeff Wendling
77fd41a02e satellite: add an expiring lru cache around api keys
Change-Id: I995429c66affd33da59b091f28f09ca122070b5e
2020-01-09 22:13:41 -07:00
Egon Elbre
d3d75a597f satellite,storage: clean global ctx usage in tests
Change-Id: I89ea5c95fc6895518b464f8eb6a4c74c6ae37651
2020-01-09 10:37:21 +00:00
Yingrong Zhao
76ee8a1b4c satellite: remove UptimeReputation configs from codebase
With the new storage node downtime tracking feature, we need remove current uptime reputation configs: UptimeReputationAlpha, UptimeReputationBeta, and
UptimeReputationDQ. This is the first step of removing the uptime
reputation columns from satellitedb

Change-Id: Ie8fab13295dbf545e33aeda0c4306cda4ba54e36
2020-01-08 18:54:15 +00:00
Egon Elbre
082ec81714
uplink: move to storj.io/uplink (#3746) 2020-01-08 15:40:19 +02:00
Jeff Wendling
c740b82e66 satellitedb/dbx: remove sed usage for bash script
turns out portable sed is hard: it has to work with both
linux and bsd sed, etc. instead, use a really really basic
bash script and a temporary file. this should be much less
likely to cause issues on a wide range of machines.

Change-Id: Ia759789fb52aa1ee3361426bb6c02ed4eac3d23a
2020-01-08 01:40:24 +00:00
paul cannon
6b21334c47 satellite/satellitedb: use txutil.ExecuteInTx in dbx WithTx()
Change-Id: I42ec21fdf117c661b3e1687a04014650c3a6ab97
2020-01-07 17:00:08 -06:00
paul cannon
0c5e381434 satellite/console: use transaction helpers in consoledb
Transactions in our code that might need to work against CockroachDB
need to be retried in the event of a retryable error. The transaction
helper functions in dbutil do that automatically. I am changing this
code to use those helpers instead.

I also fleshed out consoledb_test.go to do actual inserts and gets to
make sure things were working correctly.

Change-Id: I089bf4c776d15dc8578080e26760bd6dff4beec9
2020-01-07 17:59:10 +00:00
paul cannon
22b6e9220a satellite/satellitedb: use transaction helpers in irreparabledb
Transactions in our code that might need to work against CockroachDB
need to be retried in the event of a retryable error. The transaction
helper functions in dbutil do that automatically. I am changing this
code to use those helpers instead.

Change-Id: I22b850ce5859fa07d13bf475be5140e6bde95b8a
2020-01-07 17:40:09 +00:00
paul cannon
723ed23298 satellite/satellitedb: use transaction helpers in orders
Transactions in our code that might need to work against CockroachDB
need to be retried in the event of a retryable error. The WithTx
helper functions in dbutil and dbx do that automatically. I am changing
this code to use those helpers instead.

Change-Id: Iaf492af35471931125f2b7365aa4338f44154881
2020-01-07 16:31:47 +00:00
Natalie Ventura Villasana
1cb0f80a8d satellite/gracefulexit: dq node on exit fail
Disqualifies a node when the node fails to complete a graceful
exit.

Adds a new DisqualifyNode method to the overlay cache, since there
wasn't an existing method to disqualify a node but do nothing else
to its stats.

Adds checks to existing tests to make sure that a storage node that
fails a graceful exit is marked as disqualified in the overlay
cache.

https: //storjlabs.atlassian.net/browse/V3-3342
Change-Id: I4d554a519ab59db31ad3b8e28764c8683a6e3888
2020-01-06 19:16:26 -05:00
paul cannon
4a26fb5bd5 satellite/satellitedb: don't use crdb.ExecuteTx with postgres
crdb.ExecuteTx is great, but I don't think it will work right with
PostgreSQL. It works by way of cockroach savepoints, which allows
it to react to retryable errors, whereas tx.Commit() doesn't. But
I don't think PostgreSQL savepoints work exactly the same way. I'm not
100% sure, but it doesn't seem worth the risk.

So, I'm switching one case here to use the new dbutil.WithTx instead,
which will use crdb.ExecuteTx if appropriate. The other case doesn't
need a transaction at all.

Change-Id: I39283f3b5d8d47596db7aff5048bb74597e5918f
2020-01-06 23:51:35 +00:00
paul cannon
f3aee1b758 satellite/satellitedb: use transaction helpers in containment
Transactions in our code that might need to work against CockroachDB
need to be retried in the event of a retryable error. The transaction
helper functions in dbutil do that automatically. I am changing this
code to use those helpers instead.

Change-Id: I660540885a0784fae844cf99376d1537e208fa69
2020-01-06 23:07:38 +00:00
Moby von Briesen
6c2e4cc0cd satellite/overlay: Return NodeLastContact instead of a node dossier from
overlay.GetOfflineNodesLimited

We only care about node ID, address, and last contact success/failure
from the downtime service, so the overlay should only return these
values for the downtime-specific queries.

Change-Id: I08a6ecfdd2a12b82cae62e87d6adeab53975bfce
2020-01-06 17:12:30 -05:00
paul cannon
4203e25c54 satellite/satellitedb: use transaction helpers in overlaycache
Transactions in our code that might need to work against CockroachDB
need to be retried in the event of a retryable error. The transaction
helper functions in dbutil do that automatically. I am changing this
code to use those helpers instead.

Change-Id: Icd3da71448a84c582c6afdc6b52d1f345fe9469f
2020-01-06 21:42:57 +00:00
paul cannon
b072e16ff7 satellite/satellitedb: use transaction helpers in peeridentities
Transactions in our code that might need to work against CockroachDB
need to be retried in the event of a retryable error. The transaction
helper functions in dbutil do that automatically. I am changing this
code to use those helpers instead.

Change-Id: Ibaadd2c8540ba5c8cccd6ecbf529017ab98b78ca
2020-01-06 20:42:15 +00:00
paul cannon
eb81879d47 satellite/satellitedb: use transaction helpers in usercredits
Transactions in our code that might need to work against CockroachDB
need to be retried in the event of a retryable error. The transaction
helper functions in dbutil do that automatically. I am changing this
code to use those helpers instead.

Change-Id: Id24906f5f3ae83245dabb218e1f70e0bcb3b417a
2020-01-06 20:40:45 +00:00
paul cannon
a33734bee7 satellite/satellitedb/dbx: add cockroach driver type
Change-Id: I7a0da6e066c67a521fc1b23b085ab8554eee0d4c
2020-01-06 18:01:03 +00:00
Ethan
05b406e992 satellite:{downtime,overlay}: Implement offline node detection chore
https://storjlabs.atlassian.net/browse/V3-3398

Change-Id: I598c3bad819026377d1d113c099dc9bba8b02742
2020-01-03 17:10:03 +00:00
Ethan
8859c36234 satellite/{downtime,contact}: Add CheckNodeAvailability for use within the downtime tracking chores.
https://storjlabs.atlassian.net/browse/V3-2545

Change-Id: I1dd54a0c77cb4905bb1f350beeb82c6f7700ee70
2020-01-02 18:24:11 +00:00
Moby von Briesen
ff74b44c5f satellite/overlay: Add ability for overlay to get offline nodes ordered by last checked time
This is required for the downtime tracking service: https://storjlabs.atlassian.net/browse/V3-2545

Change-Id: I286cdc07d802393948eb10c25c45ba78cc3ceafc
2020-01-02 16:39:38 +00:00
Egon Elbre
3528c56c6f satellite/satellitedb/satellitedbtest: skip unconfigured db
Change-Id: Ib6ea58208ef19410146845e82eb08724888e85ae
2020-01-02 10:51:59 +00:00
Moby von Briesen
bb3baf5a4e satellite/satellitedb: Add nodes_offline_times table for downtime tracking
Change-Id: If6b80fe0a20d88cedacaf4b76b75aa21d0af2465
2019-12-30 15:45:02 -05:00
Egon Elbre
6615ecc9b6 common: separate repository
Change-Id: Ibb89c42060450e3839481a7e495bbe3ad940610a
2019-12-27 14:11:15 +02:00
Egon Elbre
acb4435a67 satellite/satellitedb: improve Cockroach migrate test
Load schemas in parallel instead of one-by-one.

Optimizes from 2m30s to 1m15s.

Change-Id: I0bf6381a0ae99b44271fe55d4ee658683064c097
2019-12-21 10:58:43 +00:00
Yingrong Zhao
6e71591b9b satellitedb;storagenodedb: remove unnecessary use of DB transactions in graceful exit
Change-Id: Ief0a28c6750c130896b48bfebfbea7fb3caa810f
2019-12-20 21:24:38 +00:00
Jeff Wendling
ffbc43d170 satellite/satellitedb/dbx: monitor the database calls
Change-Id: I9fbdc4a35fedfc7a1b4e1b630d2a3664b3218e67
2019-12-18 21:57:36 +00:00
Kaloyan Raev
5ee1a00857 satellite/overlay: filter reliable nodes from a list
Adds the KnownReliable method to Overlay Service that filters all nodes
from the given list to be only reliable nodes (online and qualified).
The method return []*pb.Node of reliable nodes. The pb.Node values are
ready for dialing.

The first use case is when deleting an object to efficiently dial all
reliable nodes holding a piece of that object and send them a delete
request.

Change-Id: I13e0a8666f3807c5c31ef1a1087476018a5d3acb
2019-12-17 21:20:08 +00:00
Cameron Ayer
a4f9865b47 satellite: adds and enables cockroachdb compatibility for tests
Change-Id: I85a3ad8c3b9d7e15ea8675b6c55af0002933db57
2019-12-16 22:29:25 +00:00
paul cannon
2f7465c294 private/dbutil: register "cockroach" as sql.DB driver
this will allow us to inspect the type of `db.Driver()` on *sql.DB
connections to correctly differentiate between pg and crdb conns.

as a bonus, this moves all concerns about when to replace "cockroach://"
with "postgres://" out of view, letting the thin shim "driver" take care
of that.

Change-Id: Ib24103ab7c508231e681f89a7321b623e4e125e9
2019-12-16 19:10:00 +00:00
paul cannon
b5ddfc6fa5 satellite/satellitedb: unexport satellitedb.DB
Backstory: I needed a better way to pass around information about the
underlying driver and implementation to all the various db-using things
in satellitedb (at least until some new "cockroach driver" support makes
it to DBX). After hitting a few dead ends, I decided I wanted to have a
type that could act like a *dbx.DB but which would also carry
information about the implementation, etc. Then I could pass around that
type to all the things in satellitedb that previously wanted *dbx.DB.

But then I realized that *satellitedb.DB was, essentially, exactly that
already.

One thing that might have kept *satellitedb.DB from being directly
usable was that embedding a *dbx.DB inside it would make a lot of dbx
methods publicly available on a *satellitedb.DB instance that previously
were nicely encapsulated and hidden. But after a quick look, I realized
that _nothing_ outside of satellite/satellitedb even needs to use
satellitedb.DB at all. It didn't even need to be exported, except for
some trivially-replaceable code in migrate_postgres_test.go. And once
I made it unexported, any concerns about exposing new methods on it were
entirely moot.

So I have here changed the exported *satellitedb.DB type into the
unexported *satellitedb.satelliteDB type, and I have changed all the
places here that wanted raw dbx.DB handles to use this new type instead.
Now they can just take a gander at the implementation member on it and
know all they need to know about the underlying database.

This will make it possible for some other pending code here to
differentiate between postgres and cockroach backends.

Change-Id: I27af99f8ae23b50782333da5277b553b34634edc
2019-12-16 19:09:30 +00:00
Jessica Grebenschikov
c5116cb2a0 satellitedb: fix migration cockroach test
Change-Id: Ie3b4a4b0795d156238d50a58078282cc0918a334
2019-12-16 18:02:31 +00:00
Vitalii Shpital
53d9bc4530
storagenode/notifications: db created (#3707) 2019-12-16 19:59:01 +02:00
Yaroslav Vorobiov
8cf1aa6e4f
satellite/accounting: fix project limits migration (#3717) 2019-12-10 18:12:49 +02:00
Jeff Wendling
48da8baab5 storj-sim: work with cockroach:// urls for satellite databases
for storj-sim to work, we need to avoid schemas in cockroach urls
so we have storj-sim create namespaced databases instead of schemas
and we have the migrate command create the database in the same way
that it would create a schema for postgres. then it works!

a follow up commit will move the creation of the database/schemas
into storj-sim's setup step so that we can avoid doing these icky
creations during normal migration calls. it will also make the
pointerdb have an explicit call to migrate instead of just doing
it every time it's opened.

Change-Id: If69ef5cb96b6866b0438c761bd445afb3597ae5f
2019-12-09 23:44:00 +00:00
Natalie Villasana
c3c02bec3c
satellite/satellitedb: reset storage node reputations to re-enable disqualification (#3693) 2019-12-09 12:04:00 -05:00
Egon Elbre
56a3b62bef satellite/satellitedb: ensure migration tests run (#3706)
satellitedb migration tests ran against multiple base versions, however after the merging all the steps the base versions didn't exists anymore - which meant none of the migration tests were actually running.
2019-12-09 09:26:58 -06:00
Vitalii Shpital
fa5288c254
satellitedb: bucket search fixed (#3594) 2019-12-09 14:46:30 +02:00
Jennifer Johnson
8e9532dbd9 satellitedb/repairqueue: use errs.New() instead of fmt.Errorf() to retain stack trace
Change-Id: I47f1985aaeace556e3e0f7a20d2718410936db17
2019-12-05 15:52:25 +00:00
paul cannon
378b863b2b private,satellite: unite all the "temp db schema" things
first, so that they all work the same way, because it's getting
complicated, and second, so that we can do the appropriate thing
instead of CREATE SCHEMA for cockroachdb.

Change-Id: I27fbaeeb6223a3e06d97bcf692a2d014b31465f7
2019-12-05 15:36:59 +00:00
Cameron
28a66747c9 satellite/satellitedb: cockroachDB compatible transaction in offersDB.Create (#3686)
* use crdb wrapper for create/update offer transaction, ensure times returned are UTC

* round to nearest microsecond to compare times
2019-12-04 21:54:43 -06:00
Jennifer Johnson
7c5f777a4f satellitedb/repairqueue: runs a different implementation of the query within Select() for postgres vs cockroach
Change-Id: Ie34dbdb9d870d7d9f8f269702b6b3bad0c55b98e
2019-12-04 21:32:02 +00:00
Jennifer Johnson
e9eff54473 satellitedb/attribution: updates valueAttrQuery to work for both postgres and cockroach
Change-Id: I55d6c21950a66bf782596e9152622efd700c324a
2019-12-04 18:42:02 +00:00
Jennifer Johnson
ecb960f506 private/dbutil: distinguishes between db drivers and implementations to allow for different implementations of SQL queries.
Change-Id: I2dc8d1d371139aa8bc805e92a2b80b71f580fd64
2019-12-04 18:31:26 +00:00
Cameron
c00f688930 format db name in query to match actual db name (#3688) 2019-12-04 08:55:10 -05:00
paul cannon
4d2881b711 satellite/satellitedb: return latest coupon time as UTC
it doesn't necessarily _have_ to be UTC; the time is correct as returned
either way, but this will make it a little less prone to variance.

also, there is a test that depends on the time being returned in UTC.

Change-Id: Ia71e24ecd9973ba70a1cfb5621a3030a5c82d004
2019-12-03 17:36:41 +00:00
Jess G
e0b9b5b317
satellitedb: fixes for cockroachdb compatibility (#3682)
* crdb compat fixes

* update sn accounting query
2019-12-03 08:24:46 -08:00
paul cannon
b1fa7cdfbb
satellite/satellitedb/satellitedbtest: flag to run cockroach tests (#3674)
This will make it so we don't need to comment out those lines every time
we want to enable the cockroachdb tests during development.

Once it's ready this flag can go away.
2019-12-02 14:06:21 -06:00
Yehor Butko
ae05fa0874
satellitedb/accounting: fixed query (#3672) 2019-11-29 17:53:57 +02:00
Yehor Butko
0f523e82a6
missing line returned (#3670) 2019-11-28 23:42:04 +02:00
Yehor Butko
b10eee6e7f
satellitedb/accounting: project usage optimization (#3668) 2019-11-28 20:45:31 +02:00
Yehor Butko
756b9b9e2b
satellite/payments: coupons and coupon usage (#3648) 2019-11-26 19:58:51 +02:00
Jess G
854e5507ab
crdb uses namespaced db for each test (#3646)
* crdb uses namespaced db for each test

* add test for me test

* fix lint and tests

* updates per cr comments

* rm all replaceall
2019-11-26 08:39:57 -08:00
Cameron
7abad3c6bb
refactor sql to be compatible with pq and cockroach (#3647) 2019-11-26 09:26:22 -05:00
Yaroslav Vorobiov
8a002e8c8e
satellite/accounting: separate project limit from project entity (#3632) 2019-11-25 16:18:04 +02:00
Jess G
388f33b84d
satellitedb: add support to testplanet for cockroachdb (#3634)
* update migration steps, add crdb support to testplanet

* add crdb support

* have jenkins run a bares bones crdb compat test

* skip crdb tests

* skip crdb tests

* fix root_piece_id column

* write crdb store to tmp dir

* escape
2019-11-22 11:59:46 -08:00
Matt Robinson
9af97d366a Make sed a little more cross platformable (#3629) 2019-11-22 11:17:02 -07:00
Nikolai Siedov
6a4389d3e1 satellite/console: apiKeys case-insensitive search added (#3621) 2019-11-21 00:56:55 +02:00
Yingrong Zhao
b995406ff9
satellite/satellitedb: separate uuid creation from db layer (#3600) 2019-11-20 14:16:27 -05:00
Jess G
e9c3194c82
satellitedb: merge migration into one step (#3551)
* merge migration

* rm migration versions

* rm unneeded migration test data

* create index w/postgres + crdb compatible syntax

* add default to offers.invitee_credit_duration_days

* changes so that schema matches from master to branch

* change to be crdb compatible

* add check to confirm db version

* mv version check to migration

* update tests

* add minversion to sadb migration, update tests

* confirm min version for all dbs in a migration

* add validate migration to sadb

* fix lint err

* rm min version check from migrate

* change sadb check

* hard code min db version

* fix comment
2019-11-19 12:52:57 -08:00
littleskunk
8b3444e088
satellite/nodeselection: don't select nodes that haven't checked in for a while (#3567)
* satellite/nodeselection: dont select nodes that havent checked in for a while

* change testplanet online window to one minute

* remove satellite reconfigure online window = 0 in repair tests

* pass timestamp into UpdateCheckIn

* change timestamp to timestamptz

* edit tests to set last_contact_success to 4 hours ago

* fix syntax error

* remove check for last_contact_success > last_contact_failure in IsOnline
2019-11-15 23:43:06 +01:00
Yaroslav Vorobiov
53c6741ba6
satellite/payments: add API for retrieving conversion ratio, convert tokens to USD before applying to balance (#3530) 2019-11-15 16:59:39 +02:00
Yehor Butko
a8e4e9cb03
satellite/payments: project usage charges (#3512) 2019-11-15 16:27:44 +02:00
Egon Elbre
ee6c1cac8a
private: rename internal to private (#3573) 2019-11-14 21:46:15 +02:00
Egon Elbre
1e64006e32 lint: add staticcheck as a separate step (#3569) 2019-11-14 10:31:30 +02:00
Natalie Villasana
1a9757a7f2 satellite/gracefulexit: add count for order limits sent from satellite to exiting node (#3544) 2019-11-13 09:54:50 -05:00
Yaroslav Vorobiov
36311a3a05
satellite/console: add token deposit API, populate billing history with transactions (#3500) 2019-11-12 13:14:34 +02:00
Egon Elbre
4b85d3d739
internal/testplanet: better error message when postgres is not defined (#3539) 2019-11-11 17:05:21 +02:00
Yingrong Zhao
6331f839ae
satellite/gracefulexit: not allow disqualified node to graceful exit (#3493) 2019-11-07 12:19:34 -05:00
Ethan Adams
f3dccb56b1
satellite/gracefulexit: Check if pointer has been overwritten or deleted before sending transfer message. (#3481) 2019-11-07 11:13:05 -05:00
Natalie Villasana
68a7790069 satellite/gracefulexit: select new node filtered by Distinct IP (#3435) 2019-11-06 16:38:51 -05:00
Egon Elbre
23c556ae15
satellite/rewards: fixes from review comments (#3495) 2019-11-06 20:37:53 +02:00
Yaroslav Vorobiov
35edc2bcc3 satellite/payments: invoice creation (#3468) 2019-11-05 15:16:02 +02:00
Yehor Butko
0c2e498f09
satellite/satellitedb: console tables archview comments updated (#3465) 2019-11-04 16:37:39 +02:00
Yehor Butko
761cec5ea3
satellite/payments: archview comments updated (#3464) 2019-11-04 14:30:07 +02:00
Jess G
8d92c288e2
satellitedb: separate migration into subcommand (#3436)
* separate sadb migration, add version check

* update checkversion to do same validation as migration

* changes per CR

* add sa migration to storj-sim

* add different debug port in storj-sim for migration

* add wait for exit for storj-sim migration

* update sa docker entrypoint to support migration

* storj-sim satellite parts all wait for migration

* upgrade golang-migrate/migrate to v4 because bug

* fix go mod tidy
2019-11-02 13:09:07 -07:00
Egon Elbre
aa761700af satellite/satellitedb: update nodes in sorted order (#3446) 2019-11-01 18:07:23 +01:00
Maximillian von Briesen
54594e79c3 satellite/gracefulexit: add metrics on satellite for graceful exit (#3355) 2019-10-29 16:22:20 -04:00
Yaroslav Vorobiov
30a3205745
satellite/payments: update account balance (#3379) 2019-10-29 18:04:34 +02:00
Egon Elbre
2e44a9fa6b
satellite/satellitedb/dbx: remove sqlite.sql file (#3409) 2019-10-29 12:35:27 +02:00
Ethan Adams
9905f2c61e add piece num to transfer queue PK (#3390) 2019-10-28 11:08:33 -04:00
Yingrong Zhao
fa1ac24e19
satellite/gracefulexit: add failure threshold check (#3329)
* add overall failure percentage check and inactive time frame check before sending a response to sno

* update comment

* delete node from transfer queue if it has been inactive for too long

* fix linting error

* add test config value

* fix nil pointer

* add config value into testplanet

* add unit test for overall failure threshold

* move timeframe threshold to chore

* update protolock

* add chore test

* add per peiece failure count logic

* change config name from EndpointMaxFailures to MaxFailuresPerPiece

* address comments

* fix linting error

* add error handling for no row returned from progress table

* fix test for graceful exit chore on storagenode

* fix typo InActive -> Inactive

* improve readability for failure threshold calculation

* update config lock

* change error handling for GetProgress in graceful exit endpoint on the satellite side

* return proper rpc error in endpoint

* add check in chore test for checking finish timestamp and queue
2019-10-24 12:24:42 -04:00
Yehor Butko
e82245e10e
satellite/payments: credit card selection (#3304) 2019-10-23 20:33:24 +03:00
Yaroslav Vorobiov
1a304f5ef9
satellite/payments: add payments loop, update pending transactions (#3318) 2019-10-23 15:04:54 +03:00
Maximillian von Briesen
abb567f6ae
cmd/satellite: add graceful exit reports command to satellite CLI (#3300)
* update lock file and add comment

* add created at and bytes transferred

* cleanup

* rename db func to GetGracefulExitNodesByTimeFrame

* fix flag

* split into two overlay functions

* := to =

* fix test

* add node not found error class

* fix overlay test

* suggested test changes

* review suggestions

* get exit status from overlay.Get()

* check rows.Err

* fix panic when ExitFinishedAt is nil

* fix comments in cmdGracefulExit
2019-10-22 21:06:01 -04:00
Bryan White
f468816f13
{internal/version,versioncontrol,cmd/storagenode-updater}: add rollout to storagenode updater (#3276) 2019-10-21 12:50:59 +02:00
Egon Elbre
3c438f31bd
satellite/satellitedb: remove sqlite support (#3296) 2019-10-19 00:27:57 +03:00
Egon Elbre
89ed997706
satellite/satellitedb: switch to postgres only (#3320) 2019-10-18 22:03:10 +03:00
Natalie Villasana
45c35d7c3f
satellite/satellitedb: add exit_status column to nodes table (#3301) 2019-10-17 11:01:39 -04:00
Yaroslav Vorobiov
24e72f35d3
satellite/payments: token deposit (#3283) 2019-10-17 17:04:50 +03:00
Yehor Butko
a5f4bbee22
satellite/payments: dbx scheme renamed, userID placed on Account level (#3281) 2019-10-15 21:05:45 +03:00
Natalie Villasana
cf430d2d73
scripts: add check-monitoring script to detect changes to monkit calls (#3114) 2019-10-15 13:00:14 -04:00
Vitalii Shpital
f1867a954b
web/satellite: project members sorting fixed (#3231) 2019-10-15 15:24:53 +03:00
Ethan Adams
a1275746b4
satellite/gracefulexit: Implement the 'process' endpoint on the satellite (#3223) 2019-10-11 17:18:05 -04:00
Yehor Butko
451909b3ec
satellite/payments: account balance (#3242) 2019-10-11 18:00:35 +03:00
Yehor Butko
0cc23add5b
satellite/payments - payment account setup (#3187) 2019-10-10 20:12:23 +03:00
Bogdan Artemenko
5f775b9e46
satellite/console: Added error for adding api key with existing name attempt. (#3185) 2019-10-10 16:28:35 +03:00
Maximillian von Briesen
f75893c1ba
satellite/overlay: do not include gracefully exiting nodes in node selection (#3211) 2019-10-08 15:03:38 -04:00
Maximillian von Briesen
0ea0d8c3da
satellite/overlay: remove overlay.IsVetted (#3203) 2019-10-08 09:25:41 -04:00
Michal Niewrzal
b25e0154c9
internal/testplanet: use postgres for pointerDB (#3139) 2019-10-04 07:12:21 -07:00
Natalie Villasana
4f2f8ae11b satellite/overlay: add UpdateExitStatus and GetExitingNodes for graceful exit (#3087) 2019-10-01 18:18:21 -04:00
Yaroslav Vorobiov
acbe449435 satellite/console: remove payments (#3074) 2019-09-27 12:46:37 +03:00
Cameron
fd72de211c satellite/satellitedb: update node version columns in UpdateCheckIn (#3129)
* update node version columns in UpdateCheckIn

* tests and fix sqlite implementation

* check timestamps

* edit timestamp check
2019-09-26 02:07:39 +02:00
Ethan Adams
9edfb6efe0
satellite/satellitedb: Initial GE Satellite DB Implementation (#3049)
Initial GE Satellite DB impl
Add basic CRUD operations for graceful_exit_progress and graceful_exit_transfer_queue tables.
2019-09-25 11:12:44 -06:00
Jess G
93788e5218
remove kademlia: create upsert query to update uptime (#2999)
* create upsert query for check-in method

* add tests

* fix lint err

* add benchmark test for db query

* fix lint and tests

* add a unit test, fix lint

* add address to tests

* replace print w/ b.Fatal

* refactor query per CR comments

* fix disqualified, only set if null

* fix query

* add version to updatecheckin query

* fix version

* fix tests

* change version for tests

* add version to tests

* add IP, add transport, mv unit test

* use node.address as arg

* add last ip

* fix lint
2019-09-19 11:37:31 -07:00
Ethan Adams
886041e0ba
satellite/satellitedb: add new graceful exit tables and add graceful exit fields to nodes (#3033)
DB schema changes for satellite to support Graceful Exit
2019-09-13 12:57:32 -04:00
Egon Elbre
7240e6cbb2
satellite: remove remote/inline file from BucketTally (#3041) 2019-09-13 16:51:41 +03:00
Egon Elbre
8ef57a2af3
satellite/satellitedb: use noreturn (#3022) 2019-09-12 20:31:50 +03:00
Bogdan Artemenko
bb6086aeab
web/satellite: API keys paged backend (#2839) 2019-09-12 17:19:30 +03:00
Egon Elbre
3d410add40
satellite/overlay: avoid large statement for piece counts (#3001) 2019-09-12 00:38:58 +03:00
Isaac Hess
0b32572ae6
migrate: Allow work on separate dbs (#2996) 2019-09-10 13:42:23 -06:00
Jess G
2fc4d61610
implement contact.checkin method (#2952)
* implement contact.checkin method

* add batching to update uptime checks

* rm batching

* rm other unneeded things

* fix lint

* fix unit test

* changes per CR comments

* couple more CR changes

* add identity check into grpcOpt

* fix lint

* why do you fix the test

* revert test change

* stop contact chore for repair test

* put node in cache

* comment out contact chore. See what happens

* Revert "comment out contact chore. See what happens"

This reverts commit 2e45008e36a50e0a842ae455ac83de77093d4daa.

* try stopping contact earlier

* stop contact chore in uplink_test

* replace self on chore with *RoutingTable for access to latest node info

* Revert "stop contact chore in uplink_test"

This reverts commit 302db70f4071112d1b9f7ee0279225ea12757723.

* Revert "try stopping contact earlier"

This reverts commit 806cc3b82f9d598899dafd83da9315a1cb0cb43c.

* Revert "stop contact chore for repair test"

This reverts commit dd34de1cfdfc09b972186c9ab9a4f1e822446b79.
2019-09-10 09:05:07 -07:00
Bogdan Artemenko
2c7813d40d
satellite/console: Added email normalization to users table (#2586) 2019-09-10 17:00:33 +03:00
Egon Elbre
a3e0955e16
satellite/satellitedb: ensure that we process orders in order (#2950)
When transactions are handled in different orders there is a potential for a deadlock.
2019-09-06 17:49:30 +03:00
Yaroslav Vorobiov
dadd7327df
satellite/nodestats: return storage usage in Byte*hours (#2858) 2019-09-04 19:05:34 +03:00
Ivan Fraixedes
83815ee7bf
satellitedb: always release savepoint processing orders (#2936)
* satellite/satellitedb: Always release savepoint
  Release the savepoint when processing orders in any case.
* satellite/satellitedb: Wrap errors exec savepoints
  Wrap the errors returned by the execution of savepoints operations when
  processing orders.
2019-09-04 15:02:20 +02:00
ethanadams
4ede12a2ab
satellite/orders: Fix for V3-2529: Release v0.19.0 storage nodes can't submit orders, duplicate key value violates unique constraint (#2900)
* V3-2529: Add DB savepoint to fix issue with postgres. Add test force a rejected order

Co-Authored-By: Ivan Fraixedes <ivan@fraixed.es>

* Update satellite/satellitedb/orders.go
2019-08-29 11:14:10 -04:00
Cameron
599324c364
satellite/dbcleanup: delete expired serials from satellite (#2867)
Creates a new chore, dbcleanup, which can be used for routine deletion of items from the satellite database and adds functionality for deletion of expired serial numbers
2019-08-27 13:12:38 -04:00
Bryan White
a33106df1c
satellite/satellitedb: persist piece counts to/from db (#2803) 2019-08-27 14:37:42 +02:00
aligeti
33aff71959 satellitedb/overlay: add database for storing peer identities (#2764) 2019-08-26 19:49:42 +03:00
Egon Elbre
6ff94caf22
satellite/satellitedb: move tests near the interface (#2863) 2019-08-26 13:19:02 +03:00
Egon Elbre
00b2e1a7d7 all: enable staticcheck (#2849)
* by having megacheck in disable it also disabled staticcheck

* fix closing body

* keep interfacer disabled

* hide bodies

* don't use deprecated func

* fix dead code

* fix potential overrun

* keep stylecheck disabled

* don't pass nil as context

* fix infinite recursion

* remove extraneous return

* fix data race

* use correct func

* ignore unused var

* remove unused consts
2019-08-22 13:40:15 +02:00
Egon Elbre
2d69d47655
all: fix Error.New formatting (#2840) 2019-08-21 19:30:29 +03:00
ethanadams
1a69ec8318
satellite/orders: document protocol and fix typos (#2813)
* Addressing comments from PR 2762
* Rebuild of orders.pb.go after comments added to proto file
* run update-satellite-config-lock for spelling fix.
2019-08-19 09:36:11 -04:00
Bryan White
6400d63a6c
satellite/satellitedb: Add piece count column to nodes table (#2795) 2019-08-19 12:58:13 +02:00
ethanadams
8df683a265
Update satellite settlement endpoint to batch order processing into transactions. (#2762)
Update satellite settlement endpoint to batch order processing into transactions
2019-08-15 15:05:43 -04:00
Yingrong Zhao
7db6851722
satellite/rewards: update current reward to be finished once redemption cap has reached (#2745)
* update offer once redemption cap has reached

* use transaction to get offer info before insert

* update offer status when redeemable capacity has reached

* fix format

* use pgutil to check constraint error

* change error message
2019-08-14 15:53:48 -04:00
Yaroslav Vorobiov
2c769fe9d9
satellite/overlaycache: add missing audit and uptime success count (#2788) 2019-08-14 20:53:39 +03:00
JT Olio
6ad7ca769d
satellite/satellitedb: save tallies in single transaction (#2758)
* tally: save tallies in single transaction

Change-Id: Ib4e18affc9d1d29cc7e50cb5a7dfae3761e02305

* fix compilation error
2019-08-13 16:13:56 -06:00
Yingrong Zhao
878a3c802b
satellite/console: store partner id on api key and project creation (#2743)
* init

* remove commented code

* add test

* remove unnecessary code

* add error log
2019-08-12 17:29:40 -04:00
aligeti
32f95a14fd
satellite/certdb: remove certdb that was used to store uplink certificates (#2760)
* satellitedb/certDB: refactors of the node certificate storage DB table

The existing implementation doesnt allow to store the complete certificate chain of uplinkIDs or storagenodeIDs, so the current table is dropped and new table will be added which addresses the storage and retrieval of certificates

pkg/identity: fixes spelling mistakes that I missed on PR#2754

Fixes V3-1992/V3-2388
2019-08-12 10:41:34 -04:00
Bogdan Artemenko
0decd1419b
satellite/console: project members refactoring (#2752) 2019-08-12 13:22:32 +03:00
Yaroslav Vorobiov
28a7778e9e
storagenode/nodestats: cache node stats (#2543) 2019-08-08 16:47:04 +03:00
Yehor Butko
b0e5964718
satellite/console: create owner_id column for project table (#2706)
* satellite/console: create owner_id column for project table
2019-08-07 15:28:13 +03:00
aligeti
65932ad692
Updating the certdb to support storage of multiple public keys for same node ID (#2692)
* updating the certdb to support storage of multiple public keys for same node ID
2019-08-02 13:47:35 -04:00
Yingrong Zhao
51833d0650
satellite/satellitedb: get active offer for partners (#2664)
* get partner offer

* fix lint

* fix deleting user account

* fix sqlite query

* add comments

* fix migration

* fix query_test

* add error logs

* add tests for user credits
2019-08-01 13:46:33 -04:00
ethanadams
c9b46f2fe2
V3-1987: Optimize audits stats persistence (#2632)
* Added batch update stats for recordAuditSuccessStatus
* Added batch update stats to recordAuditFailStatus
* added configurable batch size
* build individual update/delete statements so the statements can be batched into 1 call to the DB
* notified #config-changes channel and ran make update-satellite-config-lock
* updated tests to use batch update stats
2019-07-31 13:21:06 -04:00
Yingrong Zhao
4adafd056c
satellite/rewards: add generate referral link logic (#2655)
* add GeneratePartnerLink function

* remove unnecessary code

* add copyright

* fix typo and url path creation logic

* re-organize partners map

* fix creating offer for partners
2019-07-31 12:01:18 -04:00
Alexander Leitner
159ad439b1
Add count to repair queue (#2661)
* Add count to repair queue
2019-07-30 11:21:40 -04:00
Yingrong Zhao
efa2c776b7
satellite/satellitedb: add updateEarnedCredits method for user_credits table (#2609)
* parent 13dd501042
author Yingrong Zhao <yingrong.zhao@gmail.com> 1563560530 -0400
committer Yingrong Zhao <yingrong.zhao@gmail.com> 1563581673 -0400

parent 13dd501042
author Yingrong Zhao <yingrong.zhao@gmail.com> 1563560530 -0400
committer Yingrong Zhao <yingrong.zhao@gmail.com> 1563581428 -0400

satellite/console: add referral link logic (#2576)

* setup referral route

* referredBy

* add user id

* modify user query

* separate optional field from userInfo

* get current reward on init of satellite gui

* remove unsed code

* fix format

* only apply 0 credit on registration

* only pass required information for rewards

* fix time parsing

* fix test and linter

* rename method

* add todo

* remove user referral logic

* add null check and fix format

* get current offer

* remove partnerID on CreateUser struct

* fix storj-sim user creation

* only redeem credit when there's an offer

* fix default offer configuration

* fix migration

* Add helper function for get correct credit duration

* add comment

* only store userid into user_credit table

* add check for partner id to set correct offer type

* change free credit to use invitee credits

* remove unecessary code

* add credit update in activateAccount

* remove unused code

* fix format

* close reader and fix front-end build

* move create credit logic into CreateUser method

* when there's no offer set, user flow shouldn't be interrupted by referral program

* add appropriate error messages

* remove unused code

* add comment

* add error class for no current offer error

* add error class for credits update

* add comment for migration

* only log secret when it's in debug level

* fix typo

* add testdata
2019-07-30 09:21:00 -04:00
Marc Schubert
64199d37bd
Improve Node Selection Query (#2648)
* Update overlaycache.go

Removes one select statement and columns gets filtered in first query. 

Needs to be tested agains real database that this query is working and faster!

* Correct linting

reorder scans that this fit to new sql result order
2019-07-29 14:32:43 +02:00
Egon Elbre
5d0816430f
rename all the things (#2531)
* rename pkg/linksharing to linksharing
* rename pkg/httpserver to linksharing/httpserver
* rename pkg/eestream to uplink/eestream
* rename pkg/stream to uplink/stream
* rename pkg/metainfo/kvmetainfo to uplink/metainfo/kvmetainfo
* rename pkg/auth/signing to pkg/signing
* rename pkg/storage to uplink/storage
* rename pkg/accounting to satellite/accounting
* rename pkg/audit to satellite/audit
* rename pkg/certdb to satellite/certdb
* rename pkg/discovery to satellite/discovery
* rename pkg/overlay to satellite/overlay
* rename pkg/datarepair to satellite/repair
2019-07-28 08:55:36 +03:00
Faris Huskovic
2144181c99
satellite/marketingweb: Add Partner Offer Type (#2540) 2019-07-25 18:06:23 -04:00
Stefan Benten
e260f9f1cc
Remove unnecessary UNION and Selects (#2624) 2019-07-25 15:17:12 -04:00
Egon Elbre
0cdeae1922 add missing error handling (#2630) 2019-07-25 17:01:44 +02:00
Ivan Fraixedes
b24e60a33f
satellite/satellitedb: Use var block for single variable declar… (#2622)
* satellite/satellitedb: User var block for Error
  To follow with the code style of the majority of the sources of the
  current code base the Error variable should be in a block.
  Replacing a single var expression to a block one makes the godoc more
  consistent across the repository.

* satellite/satellitedb: Remove empty spaces end of line
2019-07-23 19:58:43 +02:00
Ivan Fraixedes
3c8f1370d2
[v3 2137] - Add more info to find out repair failures (#2623)
* pkg/datarepair/repairer: Track always time for repair
  Make a minor change in the worker function of the repairer, that when
  successful, always track the metric time for repair independently if the
  time since checker queue metric can be tracked.

* storage/postgreskv: Wrap error in Get func
  Wrap the returned error of the Get function as it is done when the
  query doesn't return any row.

* satellite/metainfo: Move debug msg to the right place
  NewStore function was writing a debug log message when the DB was
  connected, however it was always writing it out despite if an error
  happened when getting the connection.

* pkg/datarepair/repairer: Wrap error before logging it
  Wrap the error returned by process which is executed by the Run method
  of the repairer service to add context to the error log message.

* pkg/datarepair/repairer: Make errors more specific in worker
  Make the error messages of the "worker" method of the Service more
  specific and the logged message for such errors.

* pkg/storage/repair: Improve error reporting Repair
  In order of improving the error reporting by the
  pkg/storage/repair.Repair method, several errors of this method and
  functions/methods which this one relies one have been updated to be
  wrapper into their corresponding classes.

* pkg/storage/segments: Track path param of Repair method
  Track in monkit the path parameter passed to the Repair method.

* satellite/satellitedb: Wrap Error returned by Delete
  Wrap the error returned by repairQueue.Delete method to enhance the
  error with a class and stack and the
  pkg/storage/segments.Repairer.Repair method get a more contextualized
  error from it.
2019-07-23 16:28:06 +02:00
Jennifer Li Johnson
53d96be44a
Stylistic Go Cleanup (#2524) 2019-07-22 15:10:04 -04:00
Maximillian von Briesen
537d1cf09f
Add test to ensure correct order of repair queue selection (#2551) 2019-07-22 10:10:42 -04:00
Yingrong Zhao
6778caf846
satellite/console: add referral link logic (#2576)
* setup referral route

* referredBy

* add user id

* modify user query

* separate optional field from userInfo

* get current reward on init of satellite gui

* remove unsed code

* fix format

* only apply 0 credit on registration

* only pass required information for rewards

* fix time parsing

* fix test and linter

* rename method

* add todo

* remove user referral logic

* add null check and fix format

* get current offer

* remove partnerID on CreateUser struct

* fix storj-sim user creation

* only redeem credit when there's an offer

* fix default offer configuration

* fix migration

* Add helper function for get correct credit duration

* add comment

* only store userid into user_credit table

* add check for partner id to set correct offer type

* change free credit to use invitee credits

* remove unecessary code
2019-07-19 14:22:10 -04:00
aligeti
29b576961f
value attribution merge fix and more test cases (#2588)
* value attribution merge fix and more test cases
2019-07-19 11:17:34 -04:00
Bill Thorp
a7cc940776
Nodes should not be able to fail the same audit multiple times (#2404)
update pointer on audit failure
2019-07-18 14:08:15 -04:00
Yingrong Zhao
3e9f90b7ee
satellite/satellitedb: fix nil value from offers table (#2587)
* fix nil value

* Oops
2019-07-18 13:26:04 -04:00
Jess G
3af9250659
update irreparableDB.GetLimited query to use where instead of offset (#2585)
* update query to use where instead of offset, update tests

* update cmd/inspector irreparable

* add comment for offset
2019-07-18 09:21:21 -07:00
ethanadams
af7ffb8072 v3-2156: Add partner_id on user creation (#2571)
* Add partnerID on user creation

* added support for partner ID on create user in consoleql User

* add partner ID to api key if the user creating it has a partner ID associated with it

* updates for consoleal user and userinfo
2019-07-17 16:53:14 -04:00
Yingrong Zhao
c12a4aed3b satellite/console: add redeem reward method (#2484)
* add RedeemRewards method

* remove redeem from reward.db

* add redeemable cap check in redeem

* rename offerCap to redeemableCap

* remove redeem test

* update error message

* fix build

* Trigger Jenkins

* use correct credit setting for redeem

* fix comment

* change create qury to get redeemable_cap from offers table

* change referredBy to a pointer in user credit struct
2019-07-15 19:30:00 -04:00
aligeti
3a01106ab4
quick fix (#2548) 2019-07-12 14:12:02 -04:00
aligeti
daa3b32ee2
Add Attribution Columns to appropriate tables for OSPP referral link (#2516)
* adds "partner_id" column to user, project, api_key & bucket_metainfo tables
2019-07-12 13:59:19 -04:00
Maximillian von Briesen
b590e53d64
Order by attempted time in injured segments select (#2533) 2019-07-12 13:35:20 -04:00
Alexander Leitner
64b2769de3
discovery: parallelize refresh (#2535)
* parallelize discovery refresh

* add paginateQualifiedtest, address pr comments

* Remove duplicate uptime update

* Lower concurrency in Testplanet for discovery
2019-07-12 10:35:48 -04:00
Yingrong Zhao
d887ffec62
satellite/satellitedb: add default offer for offers table (#2522)
* add default offer for offers table

* fix migration test

* Trigger Jenkins

* set the default value to be correct type

* skip soon will deleted test

* fix test data

* add orderby for ListAll

* change durations, redeemable cap to be a nullable field

* remove unecessary code
2019-07-12 10:19:38 -04:00
Yingrong Zhao
6e57b102c5
satellite/satellitedb: remove num_redeemed in offers table (#2510)
* remove num_redeemed

* remove the field frol Offer struct

* remove the field in the UI templates

* fix merge conflict

* fix migration

* fix merge conflict

* fix typo
2019-07-12 09:19:18 -04:00
Jess G
f11bf46a11
Jg/1967 mv bucket metadata uplink (#2505)
* add bucketstore, add init uplink bucket

* update uplink to use bucket rpc

* fix tests

* wrap metainfo client errors

* add allowedBucket struct, fix tests

* update comment

* add paging

* updates per CR

* add test for pagination

* fix lint

* fix uplink test so its easier tyo understand

* fix gateway pagination bug

* changes per cr

* fix bug w/allowedBuckets, add test to catch
2019-07-12 08:57:02 -04:00
Michal Niewrzal
268c629ba8
Replace base64 encoding for path segments (#2345) 2019-07-11 13:26:07 -04:00
Bogdan Artemenko
32e0227c45
Project Payment methods (#2037) 2019-07-10 23:29:26 +03:00
Faris Huskovic
0d294103e9
satellite/rewards: nicer offers handling (#2390)
* organize offers

* revert changes to go.mod and go.sum

* change OfferStatus enums back to original

* revert modified auto-gen files

* don't render empty row if offers is empty

* change return val of ListAll to Offers

* fix build

* add method to check for empty offer when rendering template

* fix typo

* fix lint and typos

* lean out IsEmpty

* dont use named return vals

* better clarify offer statuses

* change back order of setting offer.Status

* lint

* satellite/marketingweb: allow disabling rewards (#2392)

* implement handler for stop offer endpoint

* use proper text and fix data-target for free-credit stop modal
2019-07-10 13:12:40 -04:00
JT Olio
a79c7d77f3 overlay cache: slight modification of node-is-online rules (#2490) 2019-07-09 22:36:09 -04:00
Alexander Leitner
1c5db71faf
Change protobuf expirations to use time.Time (#2509)
* Change protobuf expirations to use time.Time instead of timestamp.Timestamp
2019-07-09 17:54:00 -04:00
Egon Elbre
203f36a41f
rename order.go to order.pb.go (#2496) 2019-07-09 13:24:41 -04:00
Jess G
f9696d6c5e
satellite/metainfo: add buckets RPC and database (#2460)
* add db interface and methods, add sa metainfo endpoints and svc

* add bucket metainfo svc funcs

* add sadb bucekts

* bucket list gets all buckets

* filter buckets list on macaroon restrictions

* update pb cipher suite to be enum

* add conversion funcs

* updates per comments

* bucket settings should say default

* add direction to list buckets, add tests

* fix test bucket names

* lint err

* only support forward direction

* add comments

* minor refactoring

* make sure list up to limit

* update test

* update protolock file

* fix lint

* change per PR
2019-07-08 15:32:18 -07:00
Egon Elbre
674742d1a7
satellite/datarepair: use reliability cache (#1976) 2019-07-09 01:04:35 +03:00
Yingrong Zhao
e36f43c479
satellite/rewards: replace iota with hardcoded OfferType value (#2432)
* use 0 as the invalid offer type value

* add comment

* Update satellitedb.dbx.go

* Update satellitedb.dbx.go

* update comment
2019-07-08 14:39:56 -04:00
Alexander Leitner
19ab9852f2
Update node.proto to use time.Time instead of timestamp (#2482) 2019-07-08 14:24:42 -04:00
ethanadams
537c6021d5
fixes issue where both 36 byte and 16 uuids have duplicate time entries after deploying new version with 16 byte uuid (#2450) 2019-07-04 15:25:34 -04:00
Jess G
0587dd79d6 change bucket table to fix conflict with pointerDB (#2452)
* change bucket table to fix conflict with pointerDB

* update init query

* update migration for dbx psql

* fix migration testdata
2019-07-04 00:03:56 +02:00
Kaloyan Raev
d32c907440
overlay.UpdateStats removes node from containment mode (#2419) 2019-07-02 18:16:25 +03:00
Yaroslav Vorobiov
f4dfb8fb95
Add daily node space usage to satellite nodestats endpoint (#2361) 2019-07-02 13:42:09 +03:00
littleskunk
a2362f92dc
Rollback uptime disqualification (#2417) 2019-07-02 10:39:36 +02:00
Jess G
0158b1eb5a
add bucket metadata table in SA masterDB (#2369)
* add bucket metadata table in SA masterDB

* fix indentation

* update db model per CR comments

* update testdata

* add missing field on sql testdata

* fix args to testdata

* unique bucket name

* fix fkey constraint for test

* fix one too many commas

*  update timestamp type

* Trigger Jenkins

* Trigger Jenkins yet again
2019-07-01 13:45:21 -07:00
Egon Elbre
e8605d312e satellite/rewards: use USD type (#2384)
* fix String converison

* add method

* rename to USD

* fix types

* fix parsing of forms

* fix tests

* fix header

* use larger type

* use int64

* rename func

* move currency to separate package

* convert types, renames

* fix usercredits

* remove unnecessary conversion

* fix comment and named params
2019-07-01 15:16:49 -04:00
Egon Elbre
b6ad3e9c9f
internal/testrand: new package for random data (#2282) 2019-06-26 13:38:51 +03:00
ethanadams
0e528bc56e
Add attribution report to the satellite CLI (#2288)
* added satalite partner value attribution report. WIP

* WIP

* basic attribution report test completed. still a WIP

* cleanup

* fixed projectID conversion

* report display cleanup

* cleanup .added more test data

* added partnerID to query results

* fixed lint issues

* fix import order

* suggestions from PR review

* updated doc to reflect implementation

* clarification comments in the report SQL

* Changed based on PR suggestion

* More changes based on PR suggestions

* Changes based on PR suggestions

* reordered tests to make consistant with previous 2

* small comments cleanup

* More PR suggestions

* fixed lint issue and removed printf

* fixed var name

* Updates based on PR suggestions

* fixed message

* fixed test

* changes required after merge from master
2019-06-25 16:58:38 -04:00
ethanadams
605e3fb730
v3-2023: Add migration for project_id change in V3-2010 (#2332)
* v3-2023: add project_id migration for bucket_storage_tallies and bucket_bandwidth_rollups

* added test data for migration 37

* corrected data format

* test sql update

* migrate script updates

* adding previous data
2019-06-25 15:21:34 -04:00
Jess G
e5c48fab74
fix ordersDB methods to take correct args (#2314)
* fix orderdDB methods to take correct args

* update tally to save projectID in correct format

* update var names in splitBucket test

* changes per CR comments
2019-06-25 08:58:42 -07:00
Yingrong Zhao
bbedff12a6 satellite: rearrange marketing package (#2268)
* move offer out of marketing package and remove marketing package

* fix imports

* fix rename errors

* remove offer service

* change package name from offers to rewards

* fix linting

* remove unused code and use appropriate comment
2019-06-24 16:51:54 -04:00
Stefan Benten
01beaa289a
Mask IP Addresses to subnets (#2305) 2019-06-24 17:33:18 +02:00
paul cannon
8948459166
do ip filtering in a more correct way? (#2301)
This doesn't solve much of the performance difficulty but ought to be
lots more correct in terms of proper node selection semantics.
2019-06-23 16:16:45 -05:00
Bill Thorp
d1e9829e84 preserve reputation migration (#2295)
Adds a migration step to pull in old reputation success / total counts into modern alpha / beta scores

If audit success count is less than 50, audit alpha will be set to 50
If uptime success count is less than 100, uptime alpha will be set to 100
This helps us deal with cases where nodes have not been audited or checked for uptime yet, in which case alpha/beta values of 0/0 would cause a node to be considered disqualified.

A node with audit alpha/beta of 50/0 will be disqualified on the 19th check
A node with uptime alpha/beta of 100/0 will be disqualified on the 44th check

This does not affect brand new nodes (nodes that were not in the database before this change). The alpha/beta values for those nodes will be set to 1/0 as before
2019-06-22 21:18:54 -04:00
aligeti
d8e62bc06e
support value attribution endpoint (#2231)
Support Value attribution Endpoint
2019-06-21 15:14:34 -04:00
Bill Thorp
8f47fca5d3
Remove audit / uptime ratio fields (#2247)
* removed ratios
2019-06-21 13:14:53 -04:00
ethanadams
4f2e893e68
Fix the way project_id is stored in bucket_storage_tallies and bucket_bandwidth_rollups (#2283)
* fixing issues where projectID is stored as the byte representation of a UUID string, instead of bytes of the UUID

* added test for spitBucketID
2019-06-21 11:38:37 -04:00
Maximillian von Briesen
d103dd2c4e
Include information about disqualified nodes in the storage node payment report (#2239) 2019-06-21 10:21:15 -04:00
Natalie Villasana
9386187fe6
add disqualification and new reputation system into overlay cache (#2227) 2019-06-20 09:56:04 -04:00
Yingrong Zhao
d583ab707b add user credit usage method into console service (#2240) 2019-06-19 22:49:04 +02:00
aligeti
7b450927ff
code updates with comments (#2245) 2019-06-19 08:50:50 -04:00
Egon Elbre
8f4a6afc8a Fixes for value attribution (#2238)
* Move value attribution to satellite/

* remove unnecessary conversion
2019-06-19 08:02:37 -04:00
Maximillian von Briesen
ad8cad4909
Expand the inspector tool to provide node id's for each segment, rather than just numeric totals (#2205) 2019-06-18 18:22:14 -04:00
Bill Thorp
119a8fd3cc removed fields (#2234)
* removed fields

* sql tweaks
2019-06-18 13:40:28 -04:00
Yingrong Zhao
954ca3c6ee
add db implementation for user_credits table (#2169)
* add dbx queries

* add migration file

* start service

* Add TotalReferredCountByUserId and availableCreditsByUserID

* implement UserCredits interface and UserCredit struct type

* add UserCredits into consoledb

* add setupData helper function

* add test for update

* update lock file

* fix lint error

* add invalidUserCredits tests

* rename method

* adds comments

* add checks for erros in setupData

* change update method to only execute one query per request

* rename vairable

* should return a signal from Update method if the charge is not fully complete

* changes for readability

* prevent sql injection

* rename

* improve readability
2019-06-18 11:55:47 -04:00
Bill Thorp
81134e97cc
nodes db - adds alpha and beta reputation fields (#2215)
* node dbx and migrate adding alpha and beta reputation
2019-06-18 09:45:02 -04:00
aligeti
09940d4e0b
value attribution DB interface Insert & Get method support (#2200)
* value attribution DB interface methods support
2019-06-18 09:06:33 -04:00
Maximillian von Briesen
8398fae9b5
Add node churn and containment/reverify monkit stats (#2217)
* add counters for nodes that have/have not been seen in the past 24 hours/week

* add additional uptime counters

* add monkit stats for containment mode
2019-06-18 08:54:52 -04:00
Ivan Fraixedes
35c8648330
[v3-1914] Storage node disqualification: Change type from bool to timestamp (#2212)
* satellite/satellitedb: Alter nodes disqualification column
  Change the type of the 'disqualification' column of the nodes table from
  boolean to timestamp.
* overlay/cache: Change Disqualified field type
  Change the Disqualified field type the NodeDossier struct type from bool
  to time.Time to match with the disqualified type used by the DB layer.
* satellite/satellitedb: Update queries uses disqualified
  Update the queries which uses the disqualified column due to the column
  type has been changed from boolean to nullable timestamp.
* docs/design: Update disqualification due impl changes
  Update the disqualification design document to contain the architectural
  change required to be able to restore unfair disqualified nodes in case
  of an unexpected cause (bug, mistake, hard network disconnection, etc.).
2019-06-18 11:14:31 +02:00
Egon Elbre
f4f776d092 Use mail.test as domain in emails (#2224) 2019-06-18 02:28:40 +02:00
Faris Huskovic
c0fe061a67 add user credits table (#2085)
* add user credits table

* change primary key, change type for credit_type, and change relation kind of foreign keys from cascade to restrict

* modify table and query methods

* modify schema

* add dbx queries

* add migration file

* add orderby to read available credit entries
2019-06-13 09:52:33 -04:00
ethanadams
380684e766 Removing bucket_id and adding project_id and bucket_name as primary key (#2182)
* removing bucket_id and adding project_id and bucket_name as primary key

* cleanup
2019-06-13 08:28:55 -04:00
aligeti
8e8a5eb89c
initial value attribution check development (#2091)
* initial value attribution check development

* code reveiew updates
2019-06-12 21:35:37 -04:00
Yingrong Zhao
a4d6e8c8ca split update methods into two for offer data (#2133)
* change Offers interface to separate Update method into two.

* Implement Finish and Redeem method to avoid concurrent updates

* Implement FinishOffer and RedeemOffer service methods

* add tests

* fix linting issue

* add tests for checking Finish and Redeem's results to work as expected

* fix linting error
2019-06-12 11:53:19 -04:00
Egon Elbre
61c8c3fb49
satellite/orders: batch update storage node allocations (#2146) 2019-06-10 17:58:28 +03:00
Yaroslav Vorobiov
51db703b2b
Console add payments to service (#2100) 2019-06-06 19:07:14 +03:00
Egon Elbre
91a0ece959
Remove bwagreement leftovers (#2140) 2019-06-06 18:16:14 +03:00
Egon Elbre
28a1201590 pkg/bwagreements: remove service (#2138) 2019-06-06 15:57:58 +02:00
Natalie Villasana
25d7dda135 add disqualified check to node selection queries (#2102)
add disqualified check to queries, skip TestStatDB
2019-06-05 20:21:32 -04:00
Dennis Coyle
b9d586901e
Value Attribution DBX Model (#2116)
* adds model to satellite dbx

* cleans up model spacing

* generated golang from dbx

* added migration steps

* Added testdata

* changed node_id -> bucket_id

* adds -- NEW DATA -- to testdata

* more testdata changes

* adds -- NEW DATA -- line

* dbx makes the table plural

* missed a singular value_attribution

* restart jenkins

* Update satellitedb.dbx

* adjust to PR comments

* autogenerated dbx models

* restart jenkins
2019-06-05 12:06:14 -04:00
JT Olio
d02427e41a db: set max open conns, conn max lifetime, add db stat monitoring (#2117) 2019-06-04 23:30:21 +02:00
Yingrong Zhao
09b0c2a630
create db implementation for offer table (#2031)
* init marketing service

Fix linting error

Create offerdb implementation

Create offers service

Add update method

Create offer table and migration

Fix linting error

fix conflicts

Insert new data

Change duration to have clear indication to be based on days

add error wrapper

Change from using uuid to int for id field

* Create Marketing service

* make error virable name more readable

* add condition in update service method to check offer status

* generate lock file

Change get to listAllOffers

* Add method for getting current offer

wip

* add check for expires_at in update method

* Fix conflicts

* add copyright header

* Fix linting error

* only allow update to active offers

* add isDefault argument to GetCurrent

* Update lock file

* add migration file

* finish migrate for adding credit_in_cents for both award and invitee

* save 100 years as expiration date for default offers

* create crud test for offers

* add GetCurrent test

* modify doc

* Fix GetCurrent to work with default offer

* fix linting issue

* add more tests and address feedbacks

* fix migration file

* add type column back to match with mockup design

* add type column back to match with mockup design

* move doc changes to new pr

* add comments

* change GetCurrent to GetCurrentByType

* fix typo
2019-06-04 15:17:01 -04:00
JT Olio
29d16b4d68 satellite: add monkit task to missing places (#2108) 2019-06-04 13:55:37 +02:00
Cameron
e077b0d380
rename VetNode to IsVetted (#2097)
* rename VetNode to IsVetted
2019-06-03 10:53:30 -04:00
Egon Elbre
b8e0ac6377
satellitedb/overlaycache: avoid tx leak in case of error (#2095) 2019-06-03 17:37:43 +03:00
Yaroslav Vorobiov
79a0085103
Fix storage and object_count calculations on console (#2081) 2019-06-03 15:54:06 +03:00
Natalie Villasana
6db9388082 add disqualified column to nodes table (#2086)
* add disqualified column to nodes table, update migrate script and testdata

* fix crazy formatting of postgres.v25.sql
2019-05-30 17:38:23 -04:00
Cameron
590b1a5a1d
Satellite voucher service (#2043)
* set up voucher service skeleton, basic test

* add VetNode db method

* basic test for VetNode

* encode and sign voucher functions

* fill out and sign vouchers

* test pass/fail voucher request

* match EncodeVoucher to other Encode functions
2019-05-30 15:52:33 -04:00
ethanadams
16e3b77cf5
Enable Scopelint Linter (#2049)
* added scopelint and correcte issues found

* corrected scopelint issue

* made updates based on Ivan's suggestions

Most were around naming conventions
Some were false positives, but I kept them since the test.Run could eventually be changed to run in parallel, which could cause a bug
Others were false positives.  Added // nolint: scopelint
2019-05-29 09:30:16 -04:00
ethanadams
268dc6b7e4
Enable gocritic linter (#2051)
* first round cleanup based on go-critic

* more issues resolved for ifelsechain and unlambda checks

* updated from master and gocritic found a new ifElseChain issue

* disable appendAssign. i reports false positives

* re-enabled go-critic appendAssign and disabled lint check at code line level

* fixed go-critic lint error

* fixed // nolint add gocritic specifically
2019-05-29 09:14:25 -04:00
Michal Niewrzal
f731267e8c Per-project usage limiting (#2036)
What: Changes to support custom usage limit for the project. With this implementation by default project usage limit is taken from configuration flag. If project DB field usage_limit will be set to value larger than 0 it will become custom usage limit and we will be used to verify is limit was exceeded.

Whats changed:

usage_limit (bigint) field added to projects table (with migration)
things related to project usage moved from metainfo endpoint to project usage type
accounting.ProjectAccounting extended with GetProjectUsageLimits() method
Why: We need to have different usage limits per project. https://storjlabs.atlassian.net/browse/V3-1814
2019-05-28 09:36:52 -06:00
Jeff Wendling
1bd52b9f90 server side macaroons (#1945)
What: Adds macaroon support to the server side

Why: So that api keys are now macaroons
2019-05-24 10:51:27 -06:00
Kaloyan Raev
de8070730a Save hash of expected share from missing audited node (#2025) 2019-05-23 16:07:19 -04:00
Natalie Villasana
8b31c4b91f
remove unnecessary containmentDB wrapper (#2027) 2019-05-23 10:37:23 -04:00
Yingrong Zhao
f9045f8385 Create offer table in satellitedb for free credit program (#1975) 2019-05-22 23:41:55 +02:00
Cameron
4058c29ca4
filter duplicate node IPs (#1890)
* add last_ip field to dbx model node, generate dbx

* add last_ip to node proto, generate pb

* migrate

* resolve address in transport.DialNode, update lastIp in cache.UpdateAddress

* use net.SplitHostPort to isolate host address from port

* define DistinctIPs flag

* add test for GetIP

* select last_ip when querying for nodes

* if distinctIPs flag == true, query for nodes with distinct IPs

* some basic tests

* change last_ip to field 14 in proto

* remove comments

* check err

* change distinctIPs to distinctIP

* exclude IPs from newNodes in query for reputable nodes

* add index on last_ip

* only add to excludedIPs if flag is true

* test half new nodes returns distinct IPs

* fix alignment

* add test

* rework ip filter query, add retry logic, add switch for database driver

* add retry to SelectNewNodes

* change discovery intervals so IPs don't get overwritten

* remove TestGetIP

* edit updating node stats in test

* split exclude into nodeIDs and IPs

* separate non-distinct IP query into other function

* trigger checks

* remove else block
2019-05-22 16:06:27 -04:00
Natalie Villasana
acfb59ae48
adds db layer functions for containment (#1972) 2019-05-22 10:50:22 -04:00
Bill Thorp
a6c4019288
using DB time only (#2018)
* using DB time only, using UTC
2019-05-21 12:50:55 -04:00