JakeHillion/storj
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge 'master' branch

Browse Source 
Change-Id: I6070089128a150a4dd501bbc62a1f8b394aa643e
This commit is contained in:
Michal Niewrzal 2020-11-10 12:56:30 +01:00
parent 92f9251074
commit 7dde184cb5
101 changed files with 2948 additions and 1177 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
.clabot
View File

@ -1,4 +1,5 @@
{ {
"message": "Thank you for your pull request and welcome to our community. We require contributors to sign our [Contributor License Agreement](https://docs.google.com/forms/d/e/1FAIpQLSdVzD5W8rx-J_jLaPuG31nbOzS8yhNIIu4yHvzonji6NeZ4ig/viewform), and we don't seem to have the users {{usersWithoutCLA}} on file. Once you have signed the CLA, please let us know, so we can manually review and add you to the approved contributors list.",
"contributors": [ "contributors": [
"aleitner", "aleitner",
"aligeti", "aligeti",
@ -62,6 +63,7 @@
"montyanderson", "montyanderson",
"sixcorners", "sixcorners",
"alexottoboni", "alexottoboni",
"dominickmarino" "dominickmarino",
"hectorj2f"
] ]
} }

4
docs/blueprints/access-revocation.md
View File

@ -39,7 +39,7 @@ presents challenges when revoking a macaroon.
For example, if I hold the API key for Project A, I can create Macaroon A with a For example, if I hold the API key for Project A, I can create Macaroon A with a
caveat that it can only read and write files within Bucket A. I can caveat that it can only read and write files within Bucket A. I can
then share this macaroon with my own customer, Customer A. Customer A may then, then share this macaroon with my own customer, Customer A. Customer A may then,
if they wish, create Macaroon B which is futher caveated -- for example, if they wish, create Macaroon B which is further caveated -- for example,
restricted to read-only access in Bucket A -- and share Macaroon B with someone restricted to read-only access in Bucket A -- and share Macaroon B with someone
else. This can occur without my knowledge. else. This can occur without my knowledge.
@ -94,7 +94,7 @@ This approach was deemed best because it:
- Creates very little load on the database. - Creates very little load on the database.
- Is backwards compatible, and allows us to revoke existing macaroons. - Is backwards compatible, and allows us to revoke existing macaroons.
- Allows us to revoke an entire "macaroon tree" while maintaining the - Allows us to revoke an entire "macaroon tree" while maintaining the
distributive properies of macaroons. distributive properties of macaroons.
Disadvantages to this approach: Disadvantages to this approach:

2
docs/blueprints/cache-node-selection.md
View File

@ -30,7 +30,7 @@ When selecting nodes to store files to, the following criteria must be met:
- the node is not disqualified - the node is not disqualified
- the node is not suspended - the node is not suspended
- the node has not exited - the node has not exited
- the node has sufficent free disk space - the node has sufficient free disk space
- the node has been contacted recently - the node has been contacted recently
- the node has participated in a sufficient number of audit - the node has participated in a sufficient number of audit
- the nodes has sufficient uptime counts - the nodes has sufficient uptime counts

2
docs/blueprints/linux-setup.md
View File

@ -79,7 +79,7 @@ The process for building a package is as follows:
- make a source package - make a source package
- compile it to get binary packages. - compile it to get binary packages.
Only the binary package is used by the user for installation. It is not a recommended pratice to directly integrate binaries. Only the binary package is used by the user for installation. It is not a recommended practice to directly integrate binaries.
Building the source package is the most difficult part. But once it is done, we can use tools such as [fpm](https://github.com/jordansissel/fpm/wiki) to convert it to other package formats. Building the source package is the most difficult part. But once it is done, we can use tools such as [fpm](https://github.com/jordansissel/fpm/wiki) to convert it to other package formats.

2
docs/blueprints/password-key-derivation.md
View File

@ -15,7 +15,7 @@ That leaves open the question for how a root key is created. Some requirements o
These requirements allow users to be in full control of their encryption, and don't require users safely transporting high entropy (hard to remember) secrets to bootstrap new uplinks. These requirements allow users to be in full control of their encryption, and don't require users safely transporting high entropy (hard to remember) secrets to bootstrap new uplinks.
This design accomodates more requirements that allow for additional features: This design accommodates more requirements that allow for additional features:
3. A root key can be created for any encrypted path in a bucket, not just the bucket. 3. A root key can be created for any encrypted path in a bucket, not just the bucket.
4. A table of root keys for low entropy passwords should not be possible. In other words, an attacker with knowledge of the algorithm should not be able to use a dictionary of common passwords and pre-compute what keys to check in the event of a data breach. 4. A table of root keys for low entropy passwords should not be possible. In other words, an attacker with knowledge of the algorithm should not be able to use a dictionary of common passwords and pre-compute what keys to check in the event of a data breach.

2
docs/blueprints/repair-with-hashes.md
View File

@ -41,7 +41,7 @@ Downloading for repair is significantly different enough from streaming as to wa
Using only the minimum number of pieces means that Reed-Solomon does not act as a check during repair. Hence hashing is used instead. While an uplink could potentially send signed bogus data to a storage node, the storage node would not be penalized by these actions. This requires that Audit implements a similar piece hash check instead of relying solely on Reed-Solomon encoding. Using only the minimum number of pieces means that Reed-Solomon does not act as a check during repair. Hence hashing is used instead. While an uplink could potentially send signed bogus data to a storage node, the storage node would not be penalized by these actions. This requires that Audit implements a similar piece hash check instead of relying solely on Reed-Solomon encoding.
The size of all piece hashes downloaded should be roughly equal to a default maximum segment size : 64MiB. It seems preferable to keep this in memory over dealing with persistance to disk. The size of all piece hashes downloaded should be roughly equal to a default maximum segment size : 64MiB. It seems preferable to keep this in memory over dealing with persistence to disk.
## Implementation ## Implementation

6
docs/blueprints/satellite-billing-system.md
View File

@ -51,7 +51,7 @@ type Accounts interface {
``` ```
# Customer setup # Customer setup
Every satellite user has a corresponding customer entity on stripe which holds credit cards, balance which reflects the ammount of STORJ tokens, and is used for invoicing. Every time a user visits billing page on the satellite UI we try to create a customer for him if one doesn't exists. Every satellite user has a corresponding customer entity on stripe which holds credit cards, balance which reflects the amount of STORJ tokens, and is used for invoicing. Every time a user visits billing page on the satellite UI we try to create a customer for him if one doesn't exists.
```go ```go
// Setup creates a payment account for the user. // Setup creates a payment account for the user.
// If account is already set up it will return nil. // If account is already set up it will return nil.
@ -257,7 +257,7 @@ type StorjTokens interface {
``` ```
# Making a deposit # Making a deposit
STORJ cryptocurrency processing is done via coinpayments API. Every time a user wants to deposit some amount of STORJ token to his account balacne, new coinpayments transaction is created. Transaction amount is set in USD, and conversion rates is beeing locked(saved) after transaction is created and stored in the db. STORJ cryptocurrency processing is done via coinpayments API. Every time a user wants to deposit some amount of STORJ token to his account balacne, new coinpayments transaction is created. Transaction amount is set in USD, and conversion rates is being locked(saved) after transaction is created and stored in the db.
```go ```go
// Deposit creates new deposit transaction with the given amount returning // Deposit creates new deposit transaction with the given amount returning
// ETH wallet address where funds should be sent. There is one // ETH wallet address where funds should be sent. There is one
@ -403,7 +403,7 @@ func (tokens *storjTokens) ListTransactionInfos(ctx context.Context, userID uuid
``` ```
# Transaction update cycle # Transaction update cycle
There is a cycle that iterates over all `pending`(`pending` and `paid` statuses of coinpayments transaction respectively) transactions, list it's infos and updates tx status and received amount. If updated is status is set to `cancelled` or `completed`, that transactions won't take part in the next update cycle. When there is a status transation to `completed` along with the update `apply_balance_transaction_intent` is created. Transaction with status `completed` and present `apply_balance_transaction_intent` with state `unapplied` defines as `UnappliedTransaction` which is later processed in update balance cycle. If the received amount is greater that 50$ a promotional coupon for 55$ is created. There is a cycle that iterates over all `pending`(`pending` and `paid` statuses of coinpayments transaction respectively) transactions, list it's infos and updates tx status and received amount. If updated is status is set to `cancelled` or `completed`, that transactions won't take part in the next update cycle. When there is a status transaction to `completed` along with the update `apply_balance_transaction_intent` is created. Transaction with status `completed` and present `apply_balance_transaction_intent` with state `unapplied` defines as `UnappliedTransaction` which is later processed in update balance cycle. If the received amount is greater that 50$ a promotional coupon for 55$ is created.
```go ```go
// updateTransactions updates statuses and received amount for given transactions. // updateTransactions updates statuses and received amount for given transactions.
func (service *Service) updateTransactions(ctx context.Context, ids TransactionAndUserList) (err error) { func (service *Service) updateTransactions(ctx context.Context, ids TransactionAndUserList) (err error) {

4
docs/blueprints/storage-node-satellite-selection.md
View File

@ -274,7 +274,7 @@ and are left with the following:
The list of trusted Satellite URLs should be recalculated daily (with some jitter). The list of trusted Satellite URLs should be recalculated daily (with some jitter).
### Backwards Compatability ### Backwards Compatibility
The old piecestore configuration (i.e. `piecestore.OldConfig`) currently contains a The old piecestore configuration (i.e. `piecestore.OldConfig`) currently contains a
comma separated list of trusted Satellite URLs (`WhitelistedSatellites`). It comma separated list of trusted Satellite URLs (`WhitelistedSatellites`). It
@ -296,7 +296,7 @@ a fixed set of trusted Satellite URLs.
* Implement a `trust.ListConfig` configuration struct which: * Implement a `trust.ListConfig` configuration struct which:
* Contains the list of entries (with a release default of a single list containing `https://www.tardigrade.io/trusted-satellites`) * Contains the list of entries (with a release default of a single list containing `https://www.tardigrade.io/trusted-satellites`)
* Contains a refresh interval * Contains a refresh interval
* Maintains backwards compatability with `WhitelistedSatellites` in `piecestore.OldConfig` * Maintains backwards compatibility with `WhitelistedSatellites` in `piecestore.OldConfig`
* Implement `storj.io/storj/storagenode/trust.List` that: * Implement `storj.io/storj/storagenode/trust.List` that:
* Consumes `trust.ListConfig` for configuration * Consumes `trust.ListConfig` for configuration
* Performs the initial fetching and building of trusted Satellite URLs * Performs the initial fetching and building of trusted Satellite URLs

4
docs/blueprints/storagenode-graceful-exit/protocol.md
View File

@ -8,7 +8,7 @@ This document describes how storage node transfers its pieces during Graceful Ex
## Background ## Background
During Graceful Exit a storage node needs to transfer pieces to other nodes. During transfering the storage node or satellite may crash, hence it needs to be able to continue after a restart. During Graceful Exit a storage node needs to transfer pieces to other nodes. During transferring the storage node or satellite may crash, hence it needs to be able to continue after a restart.
Satellite gathers transferred pieces list asynchronously, which is described in [Gathering Pieces Document](pieces.md). This may consume a significant amount of time. Satellite gathers transferred pieces list asynchronously, which is described in [Gathering Pieces Document](pieces.md). This may consume a significant amount of time.
@ -28,7 +28,7 @@ The `worker` should continue to poll the satellite at a configurable interval un
The satellite should return pieces to transfer from the transfer queue if piece durability <= optimal. If durability > optimal, we remove the exiting node from the segment / pointer. The satellite should return pieces to transfer from the transfer queue if piece durability <= optimal. If durability > optimal, we remove the exiting node from the segment / pointer.
The storage node should concurrently transfer pieces returned by the satellite. The storage node should send a `TransferSucceeded` message as pieces are successfuly transfered. The Storage node should send a `TransferFailed`, with reason, on failure. The storage node should concurrently transfer pieces returned by the satellite. The storage node should send a `TransferSucceeded` message as pieces are successfully transferred. The Storage node should send a `TransferFailed`, with reason, on failure.
The satellites should set the `finished_at` on success, and respond with a `DeletePiece` message. Otherwise increment `failed_count` and set the `last_failed_at` and `last_failed_code` for reprocessing. The satellites should set the `finished_at` on success, and respond with a `DeletePiece` message. Otherwise increment `failed_count` and set the `last_failed_at` and `last_failed_code` for reprocessing.

7
go.mod
View File

@ -38,14 +38,13 @@ require (
go.etcd.io/bbolt v1.3.5 go.etcd.io/bbolt v1.3.5
go.uber.org/zap v1.16.0 go.uber.org/zap v1.16.0
golang.org/x/crypto v0.0.0-20200820211705-5c72a883971a golang.org/x/crypto v0.0.0-20200820211705-5c72a883971a
golang.org/x/net v0.0.0-20200226121028-0de0cce0169b // indirect
golang.org/x/sync v0.0.0-20200625203802-6e8e738ad208 golang.org/x/sync v0.0.0-20200625203802-6e8e738ad208
golang.org/x/sys v0.0.0-20200929083018-4d22bbb62b3c golang.org/x/sys v0.0.0-20200929083018-4d22bbb62b3c
golang.org/x/time v0.0.0-20200630173020-3af7569d3a1e golang.org/x/time v0.0.0-20200630173020-3af7569d3a1e
google.golang.org/api v0.20.0 // indirect google.golang.org/api v0.20.0 // indirect
storj.io/common v0.0.0-20201030120157-90ae6720d87e storj.io/common v0.0.0-20201106104920-372a344bdd45
storj.io/drpc v0.0.14 storj.io/drpc v0.0.16
storj.io/monkit-jaeger v0.0.0-20200518165323-80778fc3f91b storj.io/monkit-jaeger v0.0.0-20200518165323-80778fc3f91b
storj.io/private v0.0.0-20201026143115-bc926bfa3bca storj.io/private v0.0.0-20201026143115-bc926bfa3bca
storj.io/uplink v1.3.2-0.20201028181609-f6efc8fcf771 storj.io/uplink v1.3.2-0.20201106105834-ec8e5cc29f7e
) )

154
go.sum
View File

@ -1,5 +1,7 @@
cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw= cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
cloud.google.com/go v0.31.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
cloud.google.com/go v0.34.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw= cloud.google.com/go v0.34.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
cloud.google.com/go v0.37.0/go.mod h1:TS1dMSSfndXH133OKGwekG838Om/cQT0BUHV3HcBgoo=
cloud.google.com/go v0.37.4/go.mod h1:NHPJ89PdicEuT9hdPXMROBD91xc5uRDxsMtSB16k7hw= cloud.google.com/go v0.37.4/go.mod h1:NHPJ89PdicEuT9hdPXMROBD91xc5uRDxsMtSB16k7hw=
cloud.google.com/go v0.38.0/go.mod h1:990N+gfupTy94rShfmMCWGDn0LpTmnzTp2qbd1dvSRU= cloud.google.com/go v0.38.0/go.mod h1:990N+gfupTy94rShfmMCWGDn0LpTmnzTp2qbd1dvSRU=
cloud.google.com/go v0.44.1/go.mod h1:iSa0KzasP4Uvy3f1mN/7PiObzGgflwredwwASm/v6AU= cloud.google.com/go v0.44.1/go.mod h1:iSa0KzasP4Uvy3f1mN/7PiObzGgflwredwwASm/v6AU=
@ -17,7 +19,12 @@ cloud.google.com/go/pubsub v1.0.1 h1:W9tAK3E57P75u0XLLR82LZyw8VpAnhmyTOxW9qzmyj8
cloud.google.com/go/pubsub v1.0.1/go.mod h1:R0Gpsv3s54REJCy4fxDixWD93lHJMoZTyQ2kNxGRt3I= cloud.google.com/go/pubsub v1.0.1/go.mod h1:R0Gpsv3s54REJCy4fxDixWD93lHJMoZTyQ2kNxGRt3I=
cloud.google.com/go/storage v1.0.0 h1:VV2nUM3wwLLGh9lSABFgZMjInyUbJeaRSE64WuAIQ+4= cloud.google.com/go/storage v1.0.0 h1:VV2nUM3wwLLGh9lSABFgZMjInyUbJeaRSE64WuAIQ+4=
cloud.google.com/go/storage v1.0.0/go.mod h1:IhtSnM/ZTZV8YYJWCY8RULGVqBDmpoyjwiyrjsg+URw= cloud.google.com/go/storage v1.0.0/go.mod h1:IhtSnM/ZTZV8YYJWCY8RULGVqBDmpoyjwiyrjsg+URw=
dmitri.shuralyov.com/app/changes v0.0.0-20180602232624-0a106ad413e3/go.mod h1:Yl+fi1br7+Rr3LqpNJf1/uxUdtRUV+Tnj0o93V2B9MU=
dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU= dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU=
dmitri.shuralyov.com/html/belt v0.0.0-20180602232347-f7d459c86be0/go.mod h1:JLBrvjyP0v+ecvNYvCpyZgu5/xkfAUhi6wJj28eUfSU=
dmitri.shuralyov.com/service/change v0.0.0-20181023043359-a85b471d5412/go.mod h1:a1inKt/atXimZ4Mv927x+r7UpyzRUf4emIoiiSC2TN4=
dmitri.shuralyov.com/state v0.0.0-20180228185332-28bcc343414c/go.mod h1:0PRwlb0D6DFvNNtx+9ybjezNCa8XF0xaYcETyp6rHWU=
git.apache.org/thrift.git v0.0.0-20180902110319-2566ecd5d999/go.mod h1:fPE2ZNJGynbRyZ4dJvy6G277gSllfV2HJqblrnkyeyg=
github.com/Azure/go-ansiterm v0.0.0-20170929234023-d6e3b3328b78 h1:w+iIsaOQNcT7OZ575w+acHgRric5iCyQh+xv+KJ4HB8= github.com/Azure/go-ansiterm v0.0.0-20170929234023-d6e3b3328b78 h1:w+iIsaOQNcT7OZ575w+acHgRric5iCyQh+xv+KJ4HB8=
github.com/Azure/go-ansiterm v0.0.0-20170929234023-d6e3b3328b78/go.mod h1:LmzpDX56iTiv29bbRTIsUNlaFfuhWRQBWjQdVyAevI8= github.com/Azure/go-ansiterm v0.0.0-20170929234023-d6e3b3328b78/go.mod h1:LmzpDX56iTiv29bbRTIsUNlaFfuhWRQBWjQdVyAevI8=
github.com/BurntSushi/toml v0.3.1 h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ= github.com/BurntSushi/toml v0.3.1 h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ=
@ -40,6 +47,7 @@ github.com/alicebob/gopher-json v0.0.0-20200520072559-a9ecdc9d1d3a h1:HbKu58rmZp
github.com/alicebob/gopher-json v0.0.0-20200520072559-a9ecdc9d1d3a/go.mod h1:SGnFV6hVsYE877CKEZ6tDNTjaSXYUk6QqoIK6PrAtcc= github.com/alicebob/gopher-json v0.0.0-20200520072559-a9ecdc9d1d3a/go.mod h1:SGnFV6hVsYE877CKEZ6tDNTjaSXYUk6QqoIK6PrAtcc=
github.com/alicebob/miniredis/v2 v2.13.3 h1:kohgdtN58KW/r9ZDVmMJE3MrfbumwsDQStd0LPAGmmw= github.com/alicebob/miniredis/v2 v2.13.3 h1:kohgdtN58KW/r9ZDVmMJE3MrfbumwsDQStd0LPAGmmw=
github.com/alicebob/miniredis/v2 v2.13.3/go.mod h1:uS970Sw5Gs9/iK3yBg0l9Uj9s25wXxSpQUE9EaJ/Blg= github.com/alicebob/miniredis/v2 v2.13.3/go.mod h1:uS970Sw5Gs9/iK3yBg0l9Uj9s25wXxSpQUE9EaJ/Blg=
github.com/anmitsu/go-shlex v0.0.0-20161002113705-648efa622239/go.mod h1:2FmKhYUyUczH0OGQWaF5ceTx0UBShxjsH6f8oGKYe2c=
github.com/apache/thrift v0.12.0 h1:pODnxUFNcjP9UTLZGTdeh+j16A8lJbRvD3rOtrk/7bs= github.com/apache/thrift v0.12.0 h1:pODnxUFNcjP9UTLZGTdeh+j16A8lJbRvD3rOtrk/7bs=
github.com/apache/thrift v0.12.0/go.mod h1:cp2SuWMxlEZw2r+iP2GNCdIi4C1qmUzdZFSVb+bacwQ= github.com/apache/thrift v0.12.0/go.mod h1:cp2SuWMxlEZw2r+iP2GNCdIi4C1qmUzdZFSVb+bacwQ=
github.com/armon/circbuf v0.0.0-20150827004946-bbbad097214e/go.mod h1:3U/XgcO3hCbHZ8TKRvWD2dDTCfh9M9ya+I9JpbB7O8o= github.com/armon/circbuf v0.0.0-20150827004946-bbbad097214e/go.mod h1:3U/XgcO3hCbHZ8TKRvWD2dDTCfh9M9ya+I9JpbB7O8o=
@ -55,6 +63,7 @@ github.com/bketelsen/crypt v0.0.3-0.20200106085610-5cbc8cc4026c/go.mod h1:MKsuJm
github.com/blang/semver v3.5.1+incompatible h1:cQNTCjp13qL8KC3Nbxr/y2Bqb63oX6wdnnjpJbkM4JQ= github.com/blang/semver v3.5.1+incompatible h1:cQNTCjp13qL8KC3Nbxr/y2Bqb63oX6wdnnjpJbkM4JQ=
github.com/blang/semver v3.5.1+incompatible/go.mod h1:kRBLl5iJ+tD4TcOOxsy/0fnwebNt5EWlYSAyrTnjyyk= github.com/blang/semver v3.5.1+incompatible/go.mod h1:kRBLl5iJ+tD4TcOOxsy/0fnwebNt5EWlYSAyrTnjyyk=
github.com/bmizerany/assert v0.0.0-20160611221934-b7ed37b82869/go.mod h1:Ekp36dRnpXw/yCqJaO+ZrUyxD+3VXMFFr56k5XYrpB4= github.com/bmizerany/assert v0.0.0-20160611221934-b7ed37b82869/go.mod h1:Ekp36dRnpXw/yCqJaO+ZrUyxD+3VXMFFr56k5XYrpB4=
github.com/bradfitz/go-smtpd v0.0.0-20170404230938-deb6d6237625/go.mod h1:HYsPBTaaSFSlLx/70C2HPIMNZpVV8+vt/A+FMnYP11g=
github.com/btcsuite/btcd v0.20.1-beta/go.mod h1:wVuoA8VJLEcwgqHBwHmzLRazpKxTv13Px/pDuV7OomQ= github.com/btcsuite/btcd v0.20.1-beta/go.mod h1:wVuoA8VJLEcwgqHBwHmzLRazpKxTv13Px/pDuV7OomQ=
github.com/btcsuite/btclog v0.0.0-20170628155309-84c8d2346e9f/go.mod h1:TdznJufoqS23FtqVCzL0ZqgP5MqXbb4fg/WgDys70nA= github.com/btcsuite/btclog v0.0.0-20170628155309-84c8d2346e9f/go.mod h1:TdznJufoqS23FtqVCzL0ZqgP5MqXbb4fg/WgDys70nA=
github.com/btcsuite/btcutil v0.0.0-20190425235716-9e5f4b9a998d/go.mod h1:+5NJ2+qvTyV9exUAL/rxXi3DcLg2Ts+ymUAY5y4NvMg= github.com/btcsuite/btcutil v0.0.0-20190425235716-9e5f4b9a998d/go.mod h1:+5NJ2+qvTyV9exUAL/rxXi3DcLg2Ts+ymUAY5y4NvMg=
@ -66,12 +75,15 @@ github.com/btcsuite/goleveldb v0.0.0-20160330041536-7834afc9e8cd/go.mod h1:F+uVa
github.com/btcsuite/snappy-go v0.0.0-20151229074030-0bdef8d06723/go.mod h1:8woku9dyThutzjeg+3xrA5iCpBRH8XEEg3lh6TiUghc= github.com/btcsuite/snappy-go v0.0.0-20151229074030-0bdef8d06723/go.mod h1:8woku9dyThutzjeg+3xrA5iCpBRH8XEEg3lh6TiUghc=
github.com/btcsuite/websocket v0.0.0-20150119174127-31079b680792/go.mod h1:ghJtEyQwv5/p4Mg4C0fgbePVuGr935/5ddU9Z3TmDRY= github.com/btcsuite/websocket v0.0.0-20150119174127-31079b680792/go.mod h1:ghJtEyQwv5/p4Mg4C0fgbePVuGr935/5ddU9Z3TmDRY=
github.com/btcsuite/winsvc v1.0.0/go.mod h1:jsenWakMcC0zFBFurPLEAyrnc/teJEM1O46fmI40EZs= github.com/btcsuite/winsvc v1.0.0/go.mod h1:jsenWakMcC0zFBFurPLEAyrnc/teJEM1O46fmI40EZs=
github.com/buger/jsonparser v0.0.0-20181115193947-bf1c66bbce23/go.mod h1:bbYlZJ7hK1yFx9hf58LP0zeX7UjIGs20ufpu3evjr+s=
github.com/calebcase/tmpfile v1.0.1/go.mod h1:iErLeG/iqJr8LaQ/gYRv4GXdqssi3jg4iSzvrA06/lw= github.com/calebcase/tmpfile v1.0.1/go.mod h1:iErLeG/iqJr8LaQ/gYRv4GXdqssi3jg4iSzvrA06/lw=
github.com/calebcase/tmpfile v1.0.2-0.20200602150926-3af473ef8439/go.mod h1:iErLeG/iqJr8LaQ/gYRv4GXdqssi3jg4iSzvrA06/lw= github.com/calebcase/tmpfile v1.0.2-0.20200602150926-3af473ef8439/go.mod h1:iErLeG/iqJr8LaQ/gYRv4GXdqssi3jg4iSzvrA06/lw=
github.com/calebcase/tmpfile v1.0.2 h1:1AGuhKiUu4J6wxz6lxuF6ck3f8G2kaV6KSEny0RGCig= github.com/calebcase/tmpfile v1.0.2 h1:1AGuhKiUu4J6wxz6lxuF6ck3f8G2kaV6KSEny0RGCig=
github.com/calebcase/tmpfile v1.0.2/go.mod h1:iErLeG/iqJr8LaQ/gYRv4GXdqssi3jg4iSzvrA06/lw= github.com/calebcase/tmpfile v1.0.2/go.mod h1:iErLeG/iqJr8LaQ/gYRv4GXdqssi3jg4iSzvrA06/lw=
github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU= github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc= github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc=
github.com/cheekybits/genny v1.0.0 h1:uGGa4nei+j20rOSeDeP5Of12XVm7TGUd4dJA9RDitfE=
github.com/cheekybits/genny v1.0.0/go.mod h1:+tQajlRqAUrPI7DOSpB0XAqZYtQakVtB7wXkRAgjxjQ=
github.com/cheggaaa/pb/v3 v3.0.5 h1:lmZOti7CraK9RSjzExsY53+WWfub9Qv13B5m4ptEoPE= github.com/cheggaaa/pb/v3 v3.0.5 h1:lmZOti7CraK9RSjzExsY53+WWfub9Qv13B5m4ptEoPE=
github.com/cheggaaa/pb/v3 v3.0.5/go.mod h1:X1L61/+36nz9bjIsrDU52qHKOQukUQe2Ge+YvGuquCw= github.com/cheggaaa/pb/v3 v3.0.5/go.mod h1:X1L61/+36nz9bjIsrDU52qHKOQukUQe2Ge+YvGuquCw=
github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI= github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI=
@ -90,6 +102,7 @@ github.com/coreos/etcd v3.3.10+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc
github.com/coreos/etcd v3.3.13+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE= github.com/coreos/etcd v3.3.13+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE=
github.com/coreos/go-semver v0.2.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk= github.com/coreos/go-semver v0.2.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk=
github.com/coreos/go-semver v0.3.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk= github.com/coreos/go-semver v0.3.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk=
github.com/coreos/go-systemd v0.0.0-20181012123002-c6f51f82210d/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4=
github.com/coreos/go-systemd v0.0.0-20190321100706-95778dfbb74e/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4= github.com/coreos/go-systemd v0.0.0-20190321100706-95778dfbb74e/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4=
github.com/coreos/go-systemd v0.0.0-20190719114852-fd7a80b32e1f/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4= github.com/coreos/go-systemd v0.0.0-20190719114852-fd7a80b32e1f/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4=
github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f/go.mod h1:E3G3o1h8I7cfcXa63jLwjI0eiQQMgzzUDFVpN/nH/eA= github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f/go.mod h1:E3G3o1h8I7cfcXa63jLwjI0eiQQMgzzUDFVpN/nH/eA=
@ -123,6 +136,7 @@ github.com/docker/go-connections v0.4.0 h1:El9xVISelRB7BuFusrZozjnkIM5YnzCViNKoh
github.com/docker/go-connections v0.4.0/go.mod h1:Gbd7IOopHjR8Iph03tsViu4nIes5XhDvyHbTtUxmeec= github.com/docker/go-connections v0.4.0/go.mod h1:Gbd7IOopHjR8Iph03tsViu4nIes5XhDvyHbTtUxmeec=
github.com/docker/go-units v0.3.3 h1:Xk8S3Xj5sLGlG5g67hJmYMmUgXv5N4PhkjJHHqrwnTk= github.com/docker/go-units v0.3.3 h1:Xk8S3Xj5sLGlG5g67hJmYMmUgXv5N4PhkjJHHqrwnTk=
github.com/docker/go-units v0.3.3/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk= github.com/docker/go-units v0.3.3/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk=
github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk=
github.com/eapache/go-resiliency v1.1.0/go.mod h1:kFI+JgMyC7bLPUVY133qvEBtVayf5mFgVsvEsIPBvNs= github.com/eapache/go-resiliency v1.1.0/go.mod h1:kFI+JgMyC7bLPUVY133qvEBtVayf5mFgVsvEsIPBvNs=
github.com/eapache/go-xerial-snappy v0.0.0-20180814174437-776d5712da21/go.mod h1:+020luEh2TKB4/GOp8oxxtq0Daoen/Cii55CzbTV6DU= github.com/eapache/go-xerial-snappy v0.0.0-20180814174437-776d5712da21/go.mod h1:+020luEh2TKB4/GOp8oxxtq0Daoen/Cii55CzbTV6DU=
github.com/eapache/queue v1.1.0/go.mod h1:6eCeP0CKFpHLu8blIFXhExK/dRa7WDZfr6jVFPTqq+I= github.com/eapache/queue v1.1.0/go.mod h1:6eCeP0CKFpHLu8blIFXhExK/dRa7WDZfr6jVFPTqq+I=
@ -132,10 +146,16 @@ github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7
github.com/fatih/color v1.7.0/go.mod h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5KwzbycvMj4= github.com/fatih/color v1.7.0/go.mod h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5KwzbycvMj4=
github.com/fatih/color v1.9.0 h1:8xPHl4/q1VyqGIPif1F+1V3Y3lSmrq01EabUW3CoW5s= github.com/fatih/color v1.9.0 h1:8xPHl4/q1VyqGIPif1F+1V3Y3lSmrq01EabUW3CoW5s=
github.com/fatih/color v1.9.0/go.mod h1:eQcE1qtQxscV5RaZvpXrrb8Drkc3/DdQ+uUYCNjL+zU= github.com/fatih/color v1.9.0/go.mod h1:eQcE1qtQxscV5RaZvpXrrb8Drkc3/DdQ+uUYCNjL+zU=
github.com/flynn/go-shlex v0.0.0-20150515145356-3f9db97f8568/go.mod h1:xEzjJPgXI435gkrCt3MPfRiAkVrwSbHsst4LCFVfpJc=
github.com/francoispqt/gojay v1.2.13/go.mod h1:ehT5mTG4ua4581f1++1WLG0vPdaA9HaiDsoyrBGkyDY=
github.com/fsnotify/fsnotify v1.4.7 h1:IXs+QLmnXW2CcXuY+8Mzv/fWEsPGWxqefPtCP5CnV9I= github.com/fsnotify/fsnotify v1.4.7 h1:IXs+QLmnXW2CcXuY+8Mzv/fWEsPGWxqefPtCP5CnV9I=
github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo= github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
github.com/fsnotify/fsnotify v1.4.9 h1:hsms1Qyu0jgnwNXIxa+/V/PDsU6CfLf6CNO8H7IWoS4=
github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ=
github.com/fsouza/fake-gcs-server v1.7.0/go.mod h1:5XIRs4YvwNbNoz+1JF8j6KLAyDh7RHGAyAK3EP2EsNk= github.com/fsouza/fake-gcs-server v1.7.0/go.mod h1:5XIRs4YvwNbNoz+1JF8j6KLAyDh7RHGAyAK3EP2EsNk=
github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04= github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
github.com/gliderlabs/ssh v0.1.1/go.mod h1:U7qILu1NlMHj9FlMhZLlkCdDnU1DBEAqr0aevW3Awn0=
github.com/go-errors/errors v1.0.1/go.mod h1:f4zRHt4oKfwPJE5k8C9vpYG+aDHdBFUsgrm6/TyX73Q=
github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9AVAgeJqvqgH9Q5CA+iKCZ2gyEVpxRU= github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9AVAgeJqvqgH9Q5CA+iKCZ2gyEVpxRU=
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8= github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as= github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
@ -159,16 +179,28 @@ github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b h1:VKtxabqXZkF25pY9ekf
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q= github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
github.com/golang/groupcache v0.0.0-20190129154638-5b532d6fd5ef/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= github.com/golang/groupcache v0.0.0-20190129154638-5b532d6fd5ef/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/groupcache v0.0.0-20191027212112-611e8accdfc9/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7 h1:5ZkaAPbicIKTF2I64qf5Fh8Aa83Q/dnOafMYV0OMwjA= github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7 h1:5ZkaAPbicIKTF2I64qf5Fh8Aa83Q/dnOafMYV0OMwjA=
github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/lint v0.0.0-20180702182130-06c8688daad7/go.mod h1:tluoj9z5200jBnyusfRPU2LqT6J+DAorxEvtC7LHB+E=
github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A= github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
github.com/golang/mock v1.2.0/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A= github.com/golang/mock v1.2.0/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
github.com/golang/mock v1.3.1 h1:qGJ6qTW+x6xX/my+8YUVl4WNpX9B7+/l2tRsHGZ7f2s= github.com/golang/mock v1.3.1 h1:qGJ6qTW+x6xX/my+8YUVl4WNpX9B7+/l2tRsHGZ7f2s=
github.com/golang/mock v1.3.1/go.mod h1:sBzyDLLjw3U8JLTeZvSv8jJB+tU5PVekmnlKIyFUx0Y= github.com/golang/mock v1.3.1/go.mod h1:sBzyDLLjw3U8JLTeZvSv8jJB+tU5PVekmnlKIyFUx0Y=
github.com/golang/mock v1.4.0/go.mod h1:UOMv5ysSaYNkG+OFQykRIcU/QvvxJf3p21QfJ2Bt3cw=
github.com/golang/mock v1.4.4 h1:l75CXGRSwbaYNpl/Z2X1XIIAMSCquvXgpVZDhwEIJsc=
github.com/golang/mock v1.4.4/go.mod h1:l3mdAwkq5BuhzHwde/uurv3sEJeZMXNpwsxVWU71h+4=
github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.2 h1:6nsPYzhq5kReh6QImI3k5qWzO4PEbvbIW2cwSfR/6xs= github.com/golang/protobuf v1.3.2 h1:6nsPYzhq5kReh6QImI3k5qWzO4PEbvbIW2cwSfR/6xs=
github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.4.0-rc.1/go.mod h1:ceaxUfeHdC40wWswd/P6IGgMaK3YpKi5j83Wpe3EHw8=
github.com/golang/protobuf v1.4.0-rc.1.0.20200221234624-67d41d38c208/go.mod h1:xKAWHe0F5eneWXFV3EuXVDTCmh+JuBKY0li0aMyXATA=
github.com/golang/protobuf v1.4.0-rc.2/go.mod h1:LlEzMj4AhA7rCAGe4KMBDvJI+AwstrUpVNzEA03Pprs=
github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0/go.mod h1:WU3c8KckQ9AFe+yFwt9sWVRKCVIyN9cPHBJSNnbL67w=
github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvqG2KuDX0=
github.com/golang/protobuf v1.4.2 h1:+Z5KGCizgyZCbGh1KZqA0fcLLkwbsjIzS4aV2v7wJX0=
github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI=
github.com/golang/snappy v0.0.0-20170215233205-553a64147049/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q= github.com/golang/snappy v0.0.0-20170215233205-553a64147049/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q= github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
github.com/golang/snappy v0.0.1/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q= github.com/golang/snappy v0.0.1/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
@ -176,6 +208,7 @@ github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c/go.mod h1:lNA+9X1NB3Z
github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ= github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M= github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M=
github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.2 h1:X2ev0eStA3AbceY54o37/0PQ/UWqKEiiO2dKL5OPaFM= github.com/google/go-cmp v0.5.2 h1:X2ev0eStA3AbceY54o37/0PQ/UWqKEiiO2dKL5OPaFM=
github.com/google/go-cmp v0.5.2/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= github.com/google/go-cmp v0.5.2/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
@ -189,6 +222,9 @@ github.com/google/pprof v0.0.0-20191218002539-d4f498aebedc/go.mod h1:ZgVRPoUq/hf
github.com/google/pprof v0.0.0-20200229191704-1ebb73c60ed3 h1:SRgJV+IoxM5MKyFdlSUeNy6/ycRUF2yBAKdAQswoHUk= github.com/google/pprof v0.0.0-20200229191704-1ebb73c60ed3 h1:SRgJV+IoxM5MKyFdlSUeNy6/ycRUF2yBAKdAQswoHUk=
github.com/google/pprof v0.0.0-20200229191704-1ebb73c60ed3/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM= github.com/google/pprof v0.0.0-20200229191704-1ebb73c60ed3/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI= github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI=
github.com/googleapis/gax-go v2.0.0+incompatible h1:j0GKcs05QVmm7yesiZq2+9cxHkNK9YM6zKx4D2qucQU=
github.com/googleapis/gax-go v2.0.0+incompatible/go.mod h1:SFVmujtThgffbyetf+mdk2eWhX2bMyUtNHzFKcPA9HY=
github.com/googleapis/gax-go/v2 v2.0.3/go.mod h1:LLvjysVCY1JZeum8Z6l8qUty8fiNwE08qbEPm1M08qg=
github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg= github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg=
github.com/googleapis/gax-go/v2 v2.0.5 h1:sjZBwGj9Jlw33ImPtvFviGYvseOtDM7hkSKB7+Tv3SM= github.com/googleapis/gax-go/v2 v2.0.5 h1:sjZBwGj9Jlw33ImPtvFviGYvseOtDM7hkSKB7+Tv3SM=
github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk= github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk=
@ -205,8 +241,10 @@ github.com/gorilla/websocket v1.4.0/go.mod h1:E7qHFY5m1UJ88s3WnNqhKjPHQ0heANvMoA
github.com/gorilla/websocket v1.4.2/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE= github.com/gorilla/websocket v1.4.2/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
github.com/graphql-go/graphql v0.7.9 h1:5Va/Rt4l5g3YjwDnid3vFfn43faaQBq7rMcIZ0VnV34= github.com/graphql-go/graphql v0.7.9 h1:5Va/Rt4l5g3YjwDnid3vFfn43faaQBq7rMcIZ0VnV34=
github.com/graphql-go/graphql v0.7.9/go.mod h1:k6yrAYQaSP59DC5UVxbgxESlmVyojThKdORUqGDGmrI= github.com/graphql-go/graphql v0.7.9/go.mod h1:k6yrAYQaSP59DC5UVxbgxESlmVyojThKdORUqGDGmrI=
github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7/go.mod h1:FecbI9+v66THATjSRHfNgh1IVFe/9kFxbXtjV0ctIMA=
github.com/grpc-ecosystem/go-grpc-middleware v1.0.0/go.mod h1:FiyG127CGDf3tlThmgyCl78X/SZQqEOJBCDaAfeWzPs= github.com/grpc-ecosystem/go-grpc-middleware v1.0.0/go.mod h1:FiyG127CGDf3tlThmgyCl78X/SZQqEOJBCDaAfeWzPs=
github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0/go.mod h1:8NvIoxWQoOIhqOTXgfV/d3M/q6VIi02HzZEHgUlZvzk= github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0/go.mod h1:8NvIoxWQoOIhqOTXgfV/d3M/q6VIi02HzZEHgUlZvzk=
github.com/grpc-ecosystem/grpc-gateway v1.5.0/go.mod h1:RSKVYQBd5MCa4OVpNdGskqpgL2+G+NZTnrVHpWWfpdw=
github.com/grpc-ecosystem/grpc-gateway v1.9.0/go.mod h1:vNeuVxBJEsws4ogUvrchl83t/GYV9WGTSLVdBhOQFDY= github.com/grpc-ecosystem/grpc-gateway v1.9.0/go.mod h1:vNeuVxBJEsws4ogUvrchl83t/GYV9WGTSLVdBhOQFDY=
github.com/hailocab/go-hostpool v0.0.0-20160125115350-e80d13ce29ed/go.mod h1:tMWxXQ9wFIaZeTI9F+hmhFiGpFmhOHzyShyFUhRm0H4= github.com/hailocab/go-hostpool v0.0.0-20160125115350-e80d13ce29ed/go.mod h1:tMWxXQ9wFIaZeTI9F+hmhFiGpFmhOHzyShyFUhRm0H4=
github.com/hashicorp/consul/api v1.1.0/go.mod h1:VmuI/Lkw1nC05EYQWNKwWGbkg+FbDBtguAZLlVdkD9Q= github.com/hashicorp/consul/api v1.1.0/go.mod h1:VmuI/Lkw1nC05EYQWNKwWGbkg+FbDBtguAZLlVdkD9Q=
@ -294,6 +332,7 @@ github.com/jackc/puddle v0.0.0-20190608224051-11cab39313c9/go.mod h1:m4B5Dj62Y0f
github.com/jackc/puddle v1.1.0/go.mod h1:m4B5Dj62Y0fbyuIc15OsIqK0+JU8nkqQjsgx7dvjSWk= github.com/jackc/puddle v1.1.0/go.mod h1:m4B5Dj62Y0fbyuIc15OsIqK0+JU8nkqQjsgx7dvjSWk=
github.com/jackc/puddle v1.1.1/go.mod h1:m4B5Dj62Y0fbyuIc15OsIqK0+JU8nkqQjsgx7dvjSWk= github.com/jackc/puddle v1.1.1/go.mod h1:m4B5Dj62Y0fbyuIc15OsIqK0+JU8nkqQjsgx7dvjSWk=
github.com/jackc/puddle v1.1.2/go.mod h1:m4B5Dj62Y0fbyuIc15OsIqK0+JU8nkqQjsgx7dvjSWk= github.com/jackc/puddle v1.1.2/go.mod h1:m4B5Dj62Y0fbyuIc15OsIqK0+JU8nkqQjsgx7dvjSWk=
github.com/jellevandenhooff/dkim v0.0.0-20150330215556-f50fe3d243e1/go.mod h1:E0B/fFc00Y+Rasa88328GlI/XbtyysCtTHZS8h7IrBU=
github.com/jessevdk/go-flags v0.0.0-20141203071132-1679536dcc89/go.mod h1:4FA24M0QyGHXBuZZK/XkWh8h0e1EYbRYJSGM75WSRxI= github.com/jessevdk/go-flags v0.0.0-20141203071132-1679536dcc89/go.mod h1:4FA24M0QyGHXBuZZK/XkWh8h0e1EYbRYJSGM75WSRxI=
github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af/go.mod h1:Nht3zPeWKUH0NzdCt2Blrr5ys8VGpn0CEB0cQHVjt7k= github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af/go.mod h1:Nht3zPeWKUH0NzdCt2Blrr5ys8VGpn0CEB0cQHVjt7k=
github.com/jonboulle/clockwork v0.1.0/go.mod h1:Ii8DK3G1RaLaWxj9trq07+26W01tbo22gdxWY5EU2bo= github.com/jonboulle/clockwork v0.1.0/go.mod h1:Ii8DK3G1RaLaWxj9trq07+26W01tbo22gdxWY5EU2bo=
@ -321,6 +360,7 @@ github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515/go.mod h1:+0opPa2QZZtGFB
github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI= github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI=
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo= github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
github.com/kr/pty v1.1.3/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
github.com/kr/pty v1.1.8/go.mod h1:O1sed60cT9XZ5uDucP5qwvh+TE3NnUj51EiZO/lmSfw= github.com/kr/pty v1.1.8/go.mod h1:O1sed60cT9XZ5uDucP5qwvh+TE3NnUj51EiZO/lmSfw=
github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE= github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
@ -330,9 +370,18 @@ github.com/lib/pq v1.1.0/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo=
github.com/lib/pq v1.2.0/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo= github.com/lib/pq v1.2.0/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo=
github.com/lib/pq v1.3.0 h1:/qkRGz8zljWiDcFvgpwUpwIAPu3r07TDvs3Rws+o/pU= github.com/lib/pq v1.3.0 h1:/qkRGz8zljWiDcFvgpwUpwIAPu3r07TDvs3Rws+o/pU=
github.com/lib/pq v1.3.0/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo= github.com/lib/pq v1.3.0/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo=
github.com/lucas-clemente/quic-go v0.7.1-0.20201102053916-272229abf044 h1:M6zB4Rs4SJDk9IBIvC3ozl23+b0d1Q7NOlHnbxuc3AY=
github.com/lucas-clemente/quic-go v0.7.1-0.20201102053916-272229abf044/go.mod h1:ZUygOqIoai0ASXXLJ92LTnKdbqh9MHCLTX6Nr1jUrK0=
github.com/lunixbochs/vtclean v1.0.0/go.mod h1:pHhQNgMf3btfWnGBVipUOjRYhoOsdGqdm/+2c2E2WMI=
github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ= github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
github.com/magiconair/properties v1.8.1 h1:ZC2Vc7/ZFkGmsVC9KvOjumD+G5lXy2RtTKyzRKO2BQ4= github.com/magiconair/properties v1.8.1 h1:ZC2Vc7/ZFkGmsVC9KvOjumD+G5lXy2RtTKyzRKO2BQ4=
github.com/magiconair/properties v1.8.1/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ= github.com/magiconair/properties v1.8.1/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
github.com/mailru/easyjson v0.0.0-20190312143242-1de009706dbe/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc=
github.com/marten-seemann/qpack v0.2.1/go.mod h1:F7Gl5L1jIgN1D11ucXefiuJS9UMVP2opoCp2jDKb7wc=
github.com/marten-seemann/qtls v0.10.0 h1:ECsuYUKalRL240rRD4Ri33ISb7kAQ3qGDlrrl55b2pc=
github.com/marten-seemann/qtls v0.10.0/go.mod h1:UvMd1oaYDACI99/oZUYLzMCkBXQVT0aGm99sJhbT8hs=
github.com/marten-seemann/qtls-go1-15 v0.1.1 h1:LIH6K34bPVttyXnUWixk0bzH6/N07VxbSabxn5A5gZQ=
github.com/marten-seemann/qtls-go1-15 v0.1.1/go.mod h1:GyFwywLKkRt+6mfU99csTEY1joMZz5vmB1WNZH3P81I=
github.com/mattn/go-colorable v0.0.9/go.mod h1:9vuHe8Xs5qXnSaW/c/ABM9alt+Vo+STaOChaDxuIBZU= github.com/mattn/go-colorable v0.0.9/go.mod h1:9vuHe8Xs5qXnSaW/c/ABM9alt+Vo+STaOChaDxuIBZU=
github.com/mattn/go-colorable v0.1.1/go.mod h1:FuOcm+DKB9mbwrcAfNl7/TZVBZ6rcnceauSikq3lYCQ= github.com/mattn/go-colorable v0.1.1/go.mod h1:FuOcm+DKB9mbwrcAfNl7/TZVBZ6rcnceauSikq3lYCQ=
github.com/mattn/go-colorable v0.1.2/go.mod h1:U0ppj6V5qS13XJ6of8GYAs25YV2eR4EVcfRqFIhoBtE= github.com/mattn/go-colorable v0.1.2/go.mod h1:U0ppj6V5qS13XJ6of8GYAs25YV2eR4EVcfRqFIhoBtE=
@ -353,6 +402,7 @@ github.com/mattn/go-sqlite3 v1.10.0/go.mod h1:FPy6KqzDD04eiIsT53CuJW3U88zkxoIYsO
github.com/mattn/go-sqlite3 v2.0.3+incompatible h1:gXHsfypPkaMZrKbD5209QV9jbUTJKjyR5WD3HYQSd+U= github.com/mattn/go-sqlite3 v2.0.3+incompatible h1:gXHsfypPkaMZrKbD5209QV9jbUTJKjyR5WD3HYQSd+U=
github.com/mattn/go-sqlite3 v2.0.3+incompatible/go.mod h1:FPy6KqzDD04eiIsT53CuJW3U88zkxoIYsOqkbpncsNc= github.com/mattn/go-sqlite3 v2.0.3+incompatible/go.mod h1:FPy6KqzDD04eiIsT53CuJW3U88zkxoIYsOqkbpncsNc=
github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0= github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0=
github.com/microcosm-cc/bluemonday v1.0.1/go.mod h1:hsXNsILzKxV+sX77C5b8FSuKF00vh2OMYv+xgHpAMF4=
github.com/miekg/dns v1.0.14/go.mod h1:W1PPwlIAgtquWBMBEV9nkV9Cazfe8ScdGz/Lj7v3Nrg= github.com/miekg/dns v1.0.14/go.mod h1:W1PPwlIAgtquWBMBEV9nkV9Cazfe8ScdGz/Lj7v3Nrg=
github.com/minio/sha256-simd v0.0.0-20190328051042-05b4dd3047e5/go.mod h1:2FMWW+8GMoPweT6+pI63m9YE3Lmw4J71hV56Chs1E/U= github.com/minio/sha256-simd v0.0.0-20190328051042-05b4dd3047e5/go.mod h1:2FMWW+8GMoPweT6+pI63m9YE3Lmw4J71hV56Chs1E/U=
github.com/mitchellh/cli v1.0.0/go.mod h1:hNIlj7HEI86fIcpObd7a0FcrxTWetlwJDGcceTlRvqc= github.com/mitchellh/cli v1.0.0/go.mod h1:hNIlj7HEI86fIcpObd7a0FcrxTWetlwJDGcceTlRvqc=
@ -370,22 +420,32 @@ github.com/morikuni/aec v0.0.0-20170113033406-39771216ff4c h1:nXxl5PrvVm2L/wCy8d
github.com/morikuni/aec v0.0.0-20170113033406-39771216ff4c/go.mod h1:BbKIizmSmc5MMPqRYbxO4ZU0S0+P200+tUnFx7PXmsc= github.com/morikuni/aec v0.0.0-20170113033406-39771216ff4c/go.mod h1:BbKIizmSmc5MMPqRYbxO4ZU0S0+P200+tUnFx7PXmsc=
github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U= github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
github.com/nakagami/firebirdsql v0.0.0-20190310045651-3c02a58cfed8/go.mod h1:86wM1zFnC6/uDBfZGNwB65O+pR2OFi5q/YQaEUid1qA= github.com/nakagami/firebirdsql v0.0.0-20190310045651-3c02a58cfed8/go.mod h1:86wM1zFnC6/uDBfZGNwB65O+pR2OFi5q/YQaEUid1qA=
github.com/neelance/astrewrite v0.0.0-20160511093645-99348263ae86/go.mod h1:kHJEU3ofeGjhHklVoIGuVj85JJwZ6kWPaJwCIxgnFmo=
github.com/neelance/sourcemap v0.0.0-20151028013722-8c68805598ab/go.mod h1:Qr6/a/Q4r9LP1IltGz7tA7iOK1WonHEYhu1HRBA7ZiM=
github.com/nsf/jsondiff v0.0.0-20200515183724-f29ed568f4ce h1:RPclfga2SEJmgMmz2k+Mg7cowZ8yv4Trqw9UsJby758= github.com/nsf/jsondiff v0.0.0-20200515183724-f29ed568f4ce h1:RPclfga2SEJmgMmz2k+Mg7cowZ8yv4Trqw9UsJby758=
github.com/nsf/jsondiff v0.0.0-20200515183724-f29ed568f4ce/go.mod h1:uFMI8w+ref4v2r9jz+c9i1IfIttS/OkmLfrk1jne5hs= github.com/nsf/jsondiff v0.0.0-20200515183724-f29ed568f4ce/go.mod h1:uFMI8w+ref4v2r9jz+c9i1IfIttS/OkmLfrk1jne5hs=
github.com/nsf/termbox-go v0.0.0-20200418040025-38ba6e5628f1 h1:lh3PyZvY+B9nFliSGTn5uFuqQQJGuNrD0MLCokv09ag= github.com/nsf/termbox-go v0.0.0-20200418040025-38ba6e5628f1 h1:lh3PyZvY+B9nFliSGTn5uFuqQQJGuNrD0MLCokv09ag=
github.com/nsf/termbox-go v0.0.0-20200418040025-38ba6e5628f1/go.mod h1:IuKpRQcYE1Tfu+oAQqaLisqDeXgjyyltCfsaoYN18NQ= github.com/nsf/termbox-go v0.0.0-20200418040025-38ba6e5628f1/go.mod h1:IuKpRQcYE1Tfu+oAQqaLisqDeXgjyyltCfsaoYN18NQ=
github.com/nxadm/tail v1.4.4 h1:DQuhQpB1tVlglWS2hLQ5OV6B5r8aGxSrPc5Qo6uTN78=
github.com/nxadm/tail v1.4.4/go.mod h1:kenIhsEOeOJmVchQTgglprH7qJGnHDVpk1VPCcaMI8A=
github.com/oklog/ulid v1.3.1/go.mod h1:CirwcVhetQ6Lv90oh/F+FBtV6XMibvdAFo93nm5qn4U= github.com/oklog/ulid v1.3.1/go.mod h1:CirwcVhetQ6Lv90oh/F+FBtV6XMibvdAFo93nm5qn4U=
github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
github.com/onsi/ginkgo v1.7.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= github.com/onsi/ginkgo v1.7.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
github.com/onsi/ginkgo v1.10.3 h1:OoxbjfXVZyod1fmWYhI7SEyaD8B00ynP3T+D5GiyHOY= github.com/onsi/ginkgo v1.10.3 h1:OoxbjfXVZyod1fmWYhI7SEyaD8B00ynP3T+D5GiyHOY=
github.com/onsi/ginkgo v1.10.3/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= github.com/onsi/ginkgo v1.10.3/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
github.com/onsi/ginkgo v1.12.1/go.mod h1:zj2OWP4+oCPe1qIXoGWkgMRwljMUYCdkwsT2108oapk=
github.com/onsi/ginkgo v1.14.0 h1:2mOpI4JVVPBN+WQRa0WKH2eXR+Ey+uK4n7Zj0aYpIQA=
github.com/onsi/ginkgo v1.14.0/go.mod h1:iSB4RoI2tjJc9BBv4NKIKWKya62Rps+oPG/Lv9klQyY=
github.com/onsi/gomega v1.4.3/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY= github.com/onsi/gomega v1.4.3/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY=
github.com/onsi/gomega v1.7.1 h1:K0jcRCwNQM3vFGh1ppMtDh/+7ApJrjldlX8fA0jDTLQ= github.com/onsi/gomega v1.7.1 h1:K0jcRCwNQM3vFGh1ppMtDh/+7ApJrjldlX8fA0jDTLQ=
github.com/onsi/gomega v1.7.1/go.mod h1:XdKZgCCFLUoM/7CFJVPcG8C1xQ1AJ0vpAezJrB7JYyY= github.com/onsi/gomega v1.7.1/go.mod h1:XdKZgCCFLUoM/7CFJVPcG8C1xQ1AJ0vpAezJrB7JYyY=
github.com/onsi/gomega v1.10.1 h1:o0+MgICZLuZ7xjH7Vx6zS/zcu93/BEp1VwkIW1mEXCE=
github.com/onsi/gomega v1.10.1/go.mod h1:iN09h71vgCQne3DLsj+A5owkum+a2tYe+TOCB1ybHNo=
github.com/opencontainers/go-digest v1.0.0-rc1 h1:WzifXhOVOEOuFYOJAW6aQqW0TooG2iki3E3Ii+WN7gQ= github.com/opencontainers/go-digest v1.0.0-rc1 h1:WzifXhOVOEOuFYOJAW6aQqW0TooG2iki3E3Ii+WN7gQ=
github.com/opencontainers/go-digest v1.0.0-rc1/go.mod h1:cMLVZDEM3+U2I4VmLI6N8jQYUd2OVphdqWwCJHrFt2s= github.com/opencontainers/go-digest v1.0.0-rc1/go.mod h1:cMLVZDEM3+U2I4VmLI6N8jQYUd2OVphdqWwCJHrFt2s=
github.com/opencontainers/image-spec v1.0.1 h1:JMemWkRwHx4Zj+fVxWoMCFm/8sYGGrUVojFA6h/TRcI= github.com/opencontainers/image-spec v1.0.1 h1:JMemWkRwHx4Zj+fVxWoMCFm/8sYGGrUVojFA6h/TRcI=
github.com/opencontainers/image-spec v1.0.1/go.mod h1:BtxoFyWECRxE4U/7sNtV5W15zMzWCbyJoFRP3s7yZA0= github.com/opencontainers/image-spec v1.0.1/go.mod h1:BtxoFyWECRxE4U/7sNtV5W15zMzWCbyJoFRP3s7yZA0=
github.com/openzipkin/zipkin-go v0.1.1/go.mod h1:NtoC/o8u3JlF1lSlyPNswIbeQH9bJTmOf0Erfk+hxe8=
github.com/openzipkin/zipkin-go v0.1.6/go.mod h1:QgAqvLzwWbR/WpD4A3cGpPtJrZXNIiJc5AZX7/PBEpw= github.com/openzipkin/zipkin-go v0.1.6/go.mod h1:QgAqvLzwWbR/WpD4A3cGpPtJrZXNIiJc5AZX7/PBEpw=
github.com/pascaldekloe/goe v0.0.0-20180627143212-57f6aae5913c/go.mod h1:lzWF7FIEvWOWxwDKqyGYQf6ZUaNfKdP144TG7ZOy1lc= github.com/pascaldekloe/goe v0.0.0-20180627143212-57f6aae5913c/go.mod h1:lzWF7FIEvWOWxwDKqyGYQf6ZUaNfKdP144TG7ZOy1lc=
github.com/pelletier/go-toml v1.2.0 h1:T5zMGML61Wp+FlcbWjRDT7yAxhJNAiPPLOFECq181zc= github.com/pelletier/go-toml v1.2.0 h1:T5zMGML61Wp+FlcbWjRDT7yAxhJNAiPPLOFECq181zc=
@ -398,6 +458,7 @@ github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINE
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/posener/complete v1.1.1/go.mod h1:em0nMJCgc9GFtwrmVmEMR/ZL6WyhyjMBndrE9hABlRI= github.com/posener/complete v1.1.1/go.mod h1:em0nMJCgc9GFtwrmVmEMR/ZL6WyhyjMBndrE9hABlRI=
github.com/prometheus/client_golang v0.8.0/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXPKyh/dDVn+NZz0KFw=
github.com/prometheus/client_golang v0.9.1/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXPKyh/dDVn+NZz0KFw= github.com/prometheus/client_golang v0.9.1/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXPKyh/dDVn+NZz0KFw=
github.com/prometheus/client_golang v0.9.3-0.20190127221311-3c4408c8b829/go.mod h1:p2iRAGwDERtqlqzRXnrOVns+ignqQo//hLXqYxZYVNs= github.com/prometheus/client_golang v0.9.3-0.20190127221311-3c4408c8b829/go.mod h1:p2iRAGwDERtqlqzRXnrOVns+ignqQo//hLXqYxZYVNs=
github.com/prometheus/client_golang v0.9.3/go.mod h1:/TN21ttK/J9q6uSwhBd54HahCDft0ttaMvbicHlPoso= github.com/prometheus/client_golang v0.9.3/go.mod h1:/TN21ttK/J9q6uSwhBd54HahCDft0ttaMvbicHlPoso=
@ -405,9 +466,11 @@ github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910/go.mod h1:
github.com/prometheus/client_model v0.0.0-20190115171406-56726106282f/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo= github.com/prometheus/client_model v0.0.0-20190115171406-56726106282f/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo=
github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA= github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA= github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
github.com/prometheus/common v0.0.0-20180801064454-c7de2306084e/go.mod h1:daVV7qP5qjZbuso7PdcryaAu0sAZbrN9i7WWcTMWvro=
github.com/prometheus/common v0.0.0-20181113130724-41aa239b4cce/go.mod h1:daVV7qP5qjZbuso7PdcryaAu0sAZbrN9i7WWcTMWvro= github.com/prometheus/common v0.0.0-20181113130724-41aa239b4cce/go.mod h1:daVV7qP5qjZbuso7PdcryaAu0sAZbrN9i7WWcTMWvro=
github.com/prometheus/common v0.2.0/go.mod h1:TNfzLD0ON7rHzMJeJkieUDPYmFC7Snx/y86RQel1bk4= github.com/prometheus/common v0.2.0/go.mod h1:TNfzLD0ON7rHzMJeJkieUDPYmFC7Snx/y86RQel1bk4=
github.com/prometheus/common v0.4.0/go.mod h1:TNfzLD0ON7rHzMJeJkieUDPYmFC7Snx/y86RQel1bk4= github.com/prometheus/common v0.4.0/go.mod h1:TNfzLD0ON7rHzMJeJkieUDPYmFC7Snx/y86RQel1bk4=
github.com/prometheus/procfs v0.0.0-20180725123919-05ee40e3a273/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
github.com/prometheus/procfs v0.0.0-20181005140218-185b4288413d/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk= github.com/prometheus/procfs v0.0.0-20181005140218-185b4288413d/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
github.com/prometheus/procfs v0.0.0-20190117184657-bf6a532e95b1/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk= github.com/prometheus/procfs v0.0.0-20190117184657-bf6a532e95b1/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
github.com/prometheus/procfs v0.0.0-20190507164030-5867b95ac084/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsTZCD3I8kEA= github.com/prometheus/procfs v0.0.0-20190507164030-5867b95ac084/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsTZCD3I8kEA=
@ -418,15 +481,39 @@ github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFR
github.com/rs/xid v1.2.1/go.mod h1:+uKXf+4Djp6Md1KODXJxgGQPKngRmWyn10oCKFzNHOQ= github.com/rs/xid v1.2.1/go.mod h1:+uKXf+4Djp6Md1KODXJxgGQPKngRmWyn10oCKFzNHOQ=
github.com/rs/zerolog v1.13.0/go.mod h1:YbFCdg8HfsridGWAh22vktObvhZbQsZXe4/zB0OKkWU= github.com/rs/zerolog v1.13.0/go.mod h1:YbFCdg8HfsridGWAh22vktObvhZbQsZXe4/zB0OKkWU=
github.com/rs/zerolog v1.15.0/go.mod h1:xYTKnLHcpfU2225ny5qZjxnj9NvkumZYjJHlAThCjNc= github.com/rs/zerolog v1.15.0/go.mod h1:xYTKnLHcpfU2225ny5qZjxnj9NvkumZYjJHlAThCjNc=
github.com/russross/blackfriday v1.5.2/go.mod h1:JO/DiYxRf+HjHt06OyowR9PTA263kcR/rfWxYHBV53g=
github.com/russross/blackfriday/v2 v2.0.1/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM= github.com/russross/blackfriday/v2 v2.0.1/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
github.com/ryanuber/columnize v0.0.0-20160712163229-9b3edd62028f/go.mod h1:sm1tb6uqfes/u+d4ooFouqFdy9/2g9QGwK3SQygK0Ts= github.com/ryanuber/columnize v0.0.0-20160712163229-9b3edd62028f/go.mod h1:sm1tb6uqfes/u+d4ooFouqFdy9/2g9QGwK3SQygK0Ts=
github.com/satori/go.uuid v1.2.0/go.mod h1:dA0hQrYB0VpLJoorglMZABFdXlWrHn1NEOzdhQKdks0= github.com/satori/go.uuid v1.2.0/go.mod h1:dA0hQrYB0VpLJoorglMZABFdXlWrHn1NEOzdhQKdks0=
github.com/sean-/seed v0.0.0-20170313163322-e2103e2c3529/go.mod h1:DxrIzT+xaE7yg65j358z/aeFdxmN0P9QXhEzd20vsDc= github.com/sean-/seed v0.0.0-20170313163322-e2103e2c3529/go.mod h1:DxrIzT+xaE7yg65j358z/aeFdxmN0P9QXhEzd20vsDc=
github.com/sergi/go-diff v1.0.0/go.mod h1:0CfEIISq7TuYL3j771MWULgwwjU+GofnZX9QAmXWZgo=
github.com/shopspring/decimal v0.0.0-20180709203117-cd690d0c9e24/go.mod h1:M+9NzErvs504Cn4c5DxATwIqPbtswREoFCre64PpcG4= github.com/shopspring/decimal v0.0.0-20180709203117-cd690d0c9e24/go.mod h1:M+9NzErvs504Cn4c5DxATwIqPbtswREoFCre64PpcG4=
github.com/shopspring/decimal v0.0.0-20200227202807-02e2044944cc/go.mod h1:DKyhrW/HYNuLGql+MJL6WCR6knT2jwCFRcu2hWCYk4o= github.com/shopspring/decimal v0.0.0-20200227202807-02e2044944cc/go.mod h1:DKyhrW/HYNuLGql+MJL6WCR6knT2jwCFRcu2hWCYk4o=
github.com/shopspring/decimal v1.2.0 h1:abSATXmQEYyShuxI4/vyW3tV1MrKAJzCZ/0zLUXYbsQ= github.com/shopspring/decimal v1.2.0 h1:abSATXmQEYyShuxI4/vyW3tV1MrKAJzCZ/0zLUXYbsQ=
github.com/shopspring/decimal v1.2.0/go.mod h1:DKyhrW/HYNuLGql+MJL6WCR6knT2jwCFRcu2hWCYk4o= github.com/shopspring/decimal v1.2.0/go.mod h1:DKyhrW/HYNuLGql+MJL6WCR6knT2jwCFRcu2hWCYk4o=
github.com/shurcooL/component v0.0.0-20170202220835-f88ec8f54cc4/go.mod h1:XhFIlyj5a1fBNx5aJTbKoIq0mNaPvOagO+HjB3EtxrY=
github.com/shurcooL/events v0.0.0-20181021180414-410e4ca65f48/go.mod h1:5u70Mqkb5O5cxEA8nxTsgrgLehJeAw6Oc4Ab1c/P1HM=
github.com/shurcooL/github_flavored_markdown v0.0.0-20181002035957-2122de532470/go.mod h1:2dOwnU2uBioM+SGy2aZoq1f/Sd1l9OkAeAUvjSyvgU0=
github.com/shurcooL/go v0.0.0-20180423040247-9e1955d9fb6e/go.mod h1:TDJrrUr11Vxrven61rcy3hJMUqaf/CLWYhHNPmT14Lk=
github.com/shurcooL/go-goon v0.0.0-20170922171312-37c2f522c041/go.mod h1:N5mDOmsrJOB+vfqUK+7DmDyjhSLIIBnXo9lvZJj3MWQ=
github.com/shurcooL/gofontwoff v0.0.0-20180329035133-29b52fc0a18d/go.mod h1:05UtEgK5zq39gLST6uB0cf3NEHjETfB4Fgr3Gx5R9Vw=
github.com/shurcooL/gopherjslib v0.0.0-20160914041154-feb6d3990c2c/go.mod h1:8d3azKNyqcHP1GaQE/c6dDgjkgSx2BZ4IoEi4F1reUI=
github.com/shurcooL/highlight_diff v0.0.0-20170515013008-09bb4053de1b/go.mod h1:ZpfEhSmds4ytuByIcDnOLkTHGUI6KNqRNPDLHDk+mUU=
github.com/shurcooL/highlight_go v0.0.0-20181028180052-98c3abbbae20/go.mod h1:UDKB5a1T23gOMUJrI+uSuH0VRDStOiUVSjBTRDVBVag=
github.com/shurcooL/home v0.0.0-20181020052607-80b7ffcb30f9/go.mod h1:+rgNQw2P9ARFAs37qieuu7ohDNQ3gds9msbT2yn85sg=
github.com/shurcooL/htmlg v0.0.0-20170918183704-d01228ac9e50/go.mod h1:zPn1wHpTIePGnXSHpsVPWEktKXHr6+SS6x/IKRb7cpw=
github.com/shurcooL/httperror v0.0.0-20170206035902-86b7830d14cc/go.mod h1:aYMfkZ6DWSJPJ6c4Wwz3QtW22G7mf/PEgaB9k/ik5+Y=
github.com/shurcooL/httpfs v0.0.0-20171119174359-809beceb2371/go.mod h1:ZY1cvUeJuFPAdZ/B6v7RHavJWZn2YPVFQ1OSXhCGOkg=
github.com/shurcooL/httpgzip v0.0.0-20180522190206-b1c53ac65af9/go.mod h1:919LwcH0M7/W4fcZ0/jy0qGght1GIhqyS/EgWGH2j5Q=
github.com/shurcooL/issues v0.0.0-20181008053335-6292fdc1e191/go.mod h1:e2qWDig5bLteJ4fwvDAc2NHzqFEthkqn7aOZAOpj+PQ=
github.com/shurcooL/issuesapp v0.0.0-20180602232740-048589ce2241/go.mod h1:NPpHK2TI7iSaM0buivtFUc9offApnI0Alt/K8hcHy0I=
github.com/shurcooL/notifications v0.0.0-20181007000457-627ab5aea122/go.mod h1:b5uSkrEVM1jQUspwbixRBhaIjIzL2xazXp6kntxYle0=
github.com/shurcooL/octicon v0.0.0-20181028054416-fa4f57f9efb2/go.mod h1:eWdoE5JD4R5UVWDucdOPg1g2fqQRq78IQa9zlOV1vpQ=
github.com/shurcooL/reactions v0.0.0-20181006231557-f2e0b4ca5b82/go.mod h1:TCR1lToEk4d2s07G3XGfz2QrgHXg4RJBvjrOozvoWfk=
github.com/shurcooL/sanitized_anchor_name v0.0.0-20170918181015-86672fcb3f95/go.mod h1:1NzhyTcUVG4SuEtjjoZeVRXNmyL/1OwPU0+IJeTBvfc=
github.com/shurcooL/sanitized_anchor_name v1.0.0/go.mod h1:1NzhyTcUVG4SuEtjjoZeVRXNmyL/1OwPU0+IJeTBvfc= github.com/shurcooL/sanitized_anchor_name v1.0.0/go.mod h1:1NzhyTcUVG4SuEtjjoZeVRXNmyL/1OwPU0+IJeTBvfc=
github.com/shurcooL/users v0.0.0-20180125191416-49c67e49c537/go.mod h1:QJTqeLYEDaXHZDBsXlPCDqdhQuJkuw4NOtaxYe3xii4=
github.com/shurcooL/webdavfs v0.0.0-20170829043945-18c3829fa133/go.mod h1:hKmq5kWdCj2z2KEozexVbfEZIWiTjhE0+UjmZgPqehw=
github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo= github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
github.com/sirupsen/logrus v1.4.1/go.mod h1:ni0Sbl8bgC9z8RoU9G6nDWqqs/fq4eDPysMBDgk/93Q= github.com/sirupsen/logrus v1.4.1/go.mod h1:ni0Sbl8bgC9z8RoU9G6nDWqqs/fq4eDPysMBDgk/93Q=
github.com/sirupsen/logrus v1.4.2 h1:SPIRibHv4MatM3XXNO2BJeFLZwZ2LvZgfQ5+UNI2im4= github.com/sirupsen/logrus v1.4.2 h1:SPIRibHv4MatM3XXNO2BJeFLZwZ2LvZgfQ5+UNI2im4=
@ -436,6 +523,8 @@ github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d/go.mod h1
github.com/smartystreets/goconvey v1.6.4 h1:fv0U8FUIMPNf1L9lnHLvLhgicrIVChEkdzIKYqbNC9s= github.com/smartystreets/goconvey v1.6.4 h1:fv0U8FUIMPNf1L9lnHLvLhgicrIVChEkdzIKYqbNC9s=
github.com/smartystreets/goconvey v1.6.4/go.mod h1:syvi0/a8iFYH4r/RixwvyeAJjdLS9QV7WQ/tjFTllLA= github.com/smartystreets/goconvey v1.6.4/go.mod h1:syvi0/a8iFYH4r/RixwvyeAJjdLS9QV7WQ/tjFTllLA=
github.com/soheilhy/cmux v0.1.4/go.mod h1:IM3LyeVVIOuxMH7sFAkER9+bJ4dT7Ms6E4xg4kGIyLM= github.com/soheilhy/cmux v0.1.4/go.mod h1:IM3LyeVVIOuxMH7sFAkER9+bJ4dT7Ms6E4xg4kGIyLM=
github.com/sourcegraph/annotate v0.0.0-20160123013949-f4cad6c6324d/go.mod h1:UdhH50NIW0fCiwBSr0co2m7BnFLdv4fQTgdqdJTHFeE=
github.com/sourcegraph/syntaxhighlight v0.0.0-20170531221838-bd320f5d308e/go.mod h1:HuIsMU8RRBOtsCgI77wP899iHVBQpCmg4ErYMZB+2IA=
github.com/spacemonkeygo/monkit/v3 v3.0.0-20191108235033-eacca33b3037/go.mod h1:JcK1pCbReQsOsMKF/POFSZCq7drXFybgGmbc27tuwes= github.com/spacemonkeygo/monkit/v3 v3.0.0-20191108235033-eacca33b3037/go.mod h1:JcK1pCbReQsOsMKF/POFSZCq7drXFybgGmbc27tuwes=
github.com/spacemonkeygo/monkit/v3 v3.0.4/go.mod h1:JcK1pCbReQsOsMKF/POFSZCq7drXFybgGmbc27tuwes= github.com/spacemonkeygo/monkit/v3 v3.0.4/go.mod h1:JcK1pCbReQsOsMKF/POFSZCq7drXFybgGmbc27tuwes=
github.com/spacemonkeygo/monkit/v3 v3.0.5/go.mod h1:JcK1pCbReQsOsMKF/POFSZCq7drXFybgGmbc27tuwes= github.com/spacemonkeygo/monkit/v3 v3.0.5/go.mod h1:JcK1pCbReQsOsMKF/POFSZCq7drXFybgGmbc27tuwes=
@ -475,9 +564,12 @@ github.com/stripe/stripe-go v70.15.0+incompatible h1:hNML7M1zx8RgtepEMlxyu/FpVPr
github.com/stripe/stripe-go v70.15.0+incompatible/go.mod h1:A1dQZmO/QypXmsL0T8axYZkSN/uA/T/A64pfKdBAMiY= github.com/stripe/stripe-go v70.15.0+incompatible/go.mod h1:A1dQZmO/QypXmsL0T8axYZkSN/uA/T/A64pfKdBAMiY=
github.com/subosito/gotenv v1.2.0 h1:Slr1R9HxAlEKefgq5jn9U+DnETlIUa6HfgEzj0g5d7s= github.com/subosito/gotenv v1.2.0 h1:Slr1R9HxAlEKefgq5jn9U+DnETlIUa6HfgEzj0g5d7s=
github.com/subosito/gotenv v1.2.0/go.mod h1:N0PQaV/YGNqwC0u51sEeR/aUtSLEXKX9iv69rRypqCw= github.com/subosito/gotenv v1.2.0/go.mod h1:N0PQaV/YGNqwC0u51sEeR/aUtSLEXKX9iv69rRypqCw=
github.com/tarm/serial v0.0.0-20180830185346-98f6abe2eb07/go.mod h1:kDXzergiv9cbyO7IOYJZWg1U88JhDg3PB6klq9Hg2pA=
github.com/tidwall/pretty v0.0.0-20180105212114-65a9db5fad51/go.mod h1:XNkn88O1ChpSDQmQeStsy+sBenx6DDtFZJxhVysOjyk= github.com/tidwall/pretty v0.0.0-20180105212114-65a9db5fad51/go.mod h1:XNkn88O1ChpSDQmQeStsy+sBenx6DDtFZJxhVysOjyk=
github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U= github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U=
github.com/ugorji/go v1.1.4/go.mod h1:uQMGLiO92mf5W77hV/PUCpI3pbzQx3CRekS0kk+RGrc= github.com/ugorji/go v1.1.4/go.mod h1:uQMGLiO92mf5W77hV/PUCpI3pbzQx3CRekS0kk+RGrc=
github.com/viant/assertly v0.4.8/go.mod h1:aGifi++jvCrUaklKEKT0BU95igDNaqkvz+49uaYMPRU=
github.com/viant/toolbox v0.24.0/go.mod h1:OxMCG57V0PXuIP2HNQrtJf2CjqdmbrOx5EkMILuUhzM=
github.com/vivint/infectious v0.0.0-20200605153912-25a574ae18a3 h1:zMsHhfK9+Wdl1F7sIKLyx3wrOFofpb3rWFbA4HgcK5k= github.com/vivint/infectious v0.0.0-20200605153912-25a574ae18a3 h1:zMsHhfK9+Wdl1F7sIKLyx3wrOFofpb3rWFbA4HgcK5k=
github.com/vivint/infectious v0.0.0-20200605153912-25a574ae18a3/go.mod h1:R0Gbuw7ElaGSLOZUSwBm/GgVwMd30jWxBDdAyMOeTuc= github.com/vivint/infectious v0.0.0-20200605153912-25a574ae18a3/go.mod h1:R0Gbuw7ElaGSLOZUSwBm/GgVwMd30jWxBDdAyMOeTuc=
github.com/xanzy/go-gitlab v0.15.0/go.mod h1:8zdQa/ri1dfn8eS3Ir1SyfvOKlw7WBJ8DVThkpGiXrs= github.com/xanzy/go-gitlab v0.15.0/go.mod h1:8zdQa/ri1dfn8eS3Ir1SyfvOKlw7WBJ8DVThkpGiXrs=
@ -508,6 +600,7 @@ go.etcd.io/bbolt v1.3.2/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU=
go.etcd.io/bbolt v1.3.5 h1:XAzx9gjCb0Rxj7EoqcClPD1d5ZBxZJk0jbuoPHenBt0= go.etcd.io/bbolt v1.3.5 h1:XAzx9gjCb0Rxj7EoqcClPD1d5ZBxZJk0jbuoPHenBt0=
go.etcd.io/bbolt v1.3.5/go.mod h1:G5EMThwa9y8QZGBClrRx5EY+Yw9kAhnjy3bSjsnlVTQ= go.etcd.io/bbolt v1.3.5/go.mod h1:G5EMThwa9y8QZGBClrRx5EY+Yw9kAhnjy3bSjsnlVTQ=
go.mongodb.org/mongo-driver v1.1.0/go.mod h1:u7ryQJ+DOzQmeO7zB6MHyr8jkEQvC8vH7qLUO4lqsUM= go.mongodb.org/mongo-driver v1.1.0/go.mod h1:u7ryQJ+DOzQmeO7zB6MHyr8jkEQvC8vH7qLUO4lqsUM=
go.opencensus.io v0.18.0/go.mod h1:vKdFvxhtzZ9onBp9VKHK8z/sRpBMnKAsufL7wlDrCOA=
go.opencensus.io v0.20.1/go.mod h1:6WKK9ahsWS3RSO+PY9ZHZUfv2irvY6gN279GOPZjmmk= go.opencensus.io v0.20.1/go.mod h1:6WKK9ahsWS3RSO+PY9ZHZUfv2irvY6gN279GOPZjmmk=
go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU= go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU=
go.opencensus.io v0.22.0/go.mod h1:+kGneAE2xo2IficOXnaByMWTGM9T73dGwxeWcUqIpI8= go.opencensus.io v0.22.0/go.mod h1:+kGneAE2xo2IficOXnaByMWTGM9T73dGwxeWcUqIpI8=
@ -531,10 +624,14 @@ go.uber.org/zap v1.10.0/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q=
go.uber.org/zap v1.14.1/go.mod h1:Mb2vm2krFEG5DV0W9qcHBYFtp/Wku1cvYaqPsS/WYfc= go.uber.org/zap v1.14.1/go.mod h1:Mb2vm2krFEG5DV0W9qcHBYFtp/Wku1cvYaqPsS/WYfc=
go.uber.org/zap v1.16.0 h1:uFRZXykJGK9lLY4HtgSw44DnIcAM+kRBP7x5m+NpAOM= go.uber.org/zap v1.16.0 h1:uFRZXykJGK9lLY4HtgSw44DnIcAM+kRBP7x5m+NpAOM=
go.uber.org/zap v1.16.0/go.mod h1:MA8QOfq0BHJwdXa996Y4dYkAqRKB8/1K1QMMZVaNZjQ= go.uber.org/zap v1.16.0/go.mod h1:MA8QOfq0BHJwdXa996Y4dYkAqRKB8/1K1QMMZVaNZjQ=
go4.org v0.0.0-20180809161055-417644f6feb5/go.mod h1:MkTOUMDaeVYJUOUsaDXIhWPZYa1yOyC1qaOBpL57BhE=
golang.org/x/build v0.0.0-20190111050920-041ab4dc3f9d/go.mod h1:OWs+y06UdEOHN4y+MfF/py+xQ/tYqIWW03b70/CG9Rw=
golang.org/x/crypto v0.0.0-20170930174604-9419663f5a44/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= golang.org/x/crypto v0.0.0-20170930174604-9419663f5a44/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
golang.org/x/crypto v0.0.0-20181029021203-45a5f77698d3/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= golang.org/x/crypto v0.0.0-20181029021203-45a5f77698d3/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
golang.org/x/crypto v0.0.0-20181030102418-4d3f4d9ffa16/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20190313024323-a1f597ede03a/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20190325154230-a5d413f7728c/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= golang.org/x/crypto v0.0.0-20190325154230-a5d413f7728c/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20190411191339-88737f569e3a/go.mod h1:WFFai1msRO1wXaEeE5yQxYXgSfI8pQAWXbQop6sCtWE= golang.org/x/crypto v0.0.0-20190411191339-88737f569e3a/go.mod h1:WFFai1msRO1wXaEeE5yQxYXgSfI8pQAWXbQop6sCtWE=
golang.org/x/crypto v0.0.0-20190426145343-a29dc8fdc734/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= golang.org/x/crypto v0.0.0-20190426145343-a29dc8fdc734/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
@ -545,6 +642,7 @@ golang.org/x/crypto v0.0.0-20190911031432-227b76d455e7/go.mod h1:yigFU9vqHzYiE8U
golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20200115085410-6d4e4cb37c7d/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= golang.org/x/crypto v0.0.0-20200115085410-6d4e4cb37c7d/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20200220183623-bac4c82f6975/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= golang.org/x/crypto v0.0.0-20200220183623-bac4c82f6975/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20200221231518-2aa609cf4a9d/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20200323165209-0ec3e9974c59/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= golang.org/x/crypto v0.0.0-20200323165209-0ec3e9974c59/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20200604202706-70a84ac30bf9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= golang.org/x/crypto v0.0.0-20200604202706-70a84ac30bf9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
@ -559,6 +657,7 @@ golang.org/x/exp v0.0.0-20191227195350-da58074b4299 h1:zQpM52jfKHG6II1ISZY1Zcpyg
golang.org/x/exp v0.0.0-20191227195350-da58074b4299/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4= golang.org/x/exp v0.0.0-20191227195350-da58074b4299/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4=
golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js= golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js=
golang.org/x/image v0.0.0-20190802002840-cff245a6509b/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0= golang.org/x/image v0.0.0-20190802002840-cff245a6509b/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0=
golang.org/x/lint v0.0.0-20180702182130-06c8688daad7/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU= golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU=
golang.org/x/lint v0.0.0-20190301231843-5614ed5bae6f/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= golang.org/x/lint v0.0.0-20190301231843-5614ed5bae6f/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
@ -577,6 +676,8 @@ golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73r
golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20181023162649-9b4f9f5ad519/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20181023162649-9b4f9f5ad519/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20181029044818-c44066c5c816/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20181106065722-10aee1819953/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20181108082009-03003ca0c849/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20181108082009-03003ca0c849/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20181201002055-351d144fa1fc/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20181201002055-351d144fa1fc/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
@ -585,6 +686,7 @@ golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73r
golang.org/x/net v0.0.0-20190125091013-d26f9f9a57f3/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20190125091013-d26f9f9a57f3/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190313220215-9f648a60d977/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190424112056-4829fb13d2c6/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= golang.org/x/net v0.0.0-20190424112056-4829fb13d2c6/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190501004415-9ce7a6920f09/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= golang.org/x/net v0.0.0-20190501004415-9ce7a6920f09/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
@ -595,15 +697,19 @@ golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLL
golang.org/x/net v0.0.0-20190813141303-74dc4d7220e7/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20190813141303-74dc4d7220e7/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20190923162816-aa69164e4478/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20190923162816-aa69164e4478/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200114155413-6afb5195e5aa/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20200114155413-6afb5195e5aa/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200226121028-0de0cce0169b h1:0mm1VjtFUOIlE1SbDlwjYaDxZVDP2S5ou6y0gSgXHu8= golang.org/x/net v0.0.0-20200520004742-59133d7f0dd7/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20200707034311-ab3426394381 h1:VXak5I6aEWmAXeQjA+QSZzlgNrpq9mjcfDemuexIKsU=
golang.org/x/net v0.0.0-20200707034311-ab3426394381/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20181017192945-9dcd33a902f4/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20181106182150-f42d05182288/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= golang.org/x/oauth2 v0.0.0-20181106182150-f42d05182288/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20181203162652-d668ce993890/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20190402181905-9f3314589c9a/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= golang.org/x/oauth2 v0.0.0-20190402181905-9f3314589c9a/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d h1:TzXSXBo42m9gQenoE3b9BGiEpg5IG2JkU5FkPIawgtw= golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d h1:TzXSXBo42m9gQenoE3b9BGiEpg5IG2JkU5FkPIawgtw=
golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/perf v0.0.0-20180704124530-6e6d33e29852/go.mod h1:JLpeXjPJfIyPr5TlbXLkXWLhP8nz10XfvxElABhCtcw=
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
@ -618,6 +724,7 @@ golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5h
golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20181026203630-95b1ffbd15a5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20181026203630-95b1ffbd15a5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20181029174526-d69651ed3497/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20181107165924-66b7b1311ac8/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20181107165924-66b7b1311ac8/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20181122145206-62eef0e2fa9b/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20181122145206-62eef0e2fa9b/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
@ -626,6 +733,7 @@ golang.org/x/sys v0.0.0-20190204203706-41f3e6584952/go.mod h1:STP8DvDyc/dI5b8T5h
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190312061237-fead79001313/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190312061237-fead79001313/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190316082340-a2f829d7f35f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190403152447-81d4e9dc473e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190403152447-81d4e9dc473e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190422165155-953cdadca894/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190422165155-953cdadca894/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
@ -636,7 +744,10 @@ golang.org/x/sys v0.0.0-20190606165138-5da285871e9c/go.mod h1:h1NjWce9XRLGQEsW7w
golang.org/x/sys v0.0.0-20190624142023-c5567b49c5d0/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190624142023-c5567b49c5d0/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190813064441-fde4db37ae7a/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190813064441-fde4db37ae7a/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190826190057-c7b8b68b1456/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190826190057-c7b8b68b1456/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190904154756-749cb33beabd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191005200804-aed5e4c7ecf9/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191026070338-33540a1f6037/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20191026070338-33540a1f6037/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191120155948-bd437916bb0e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191204072324-ce4227a45e2e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20191204072324-ce4227a45e2e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191210023423-ac6580df4449/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20191210023423-ac6580df4449/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200107144601-ef85f5a75ddf/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200107144601-ef85f5a75ddf/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
@ -645,14 +756,17 @@ golang.org/x/sys v0.0.0-20200116001909-b77594299b42/go.mod h1:h1NjWce9XRLGQEsW7w
golang.org/x/sys v0.0.0-20200202164722-d101bd2416d5/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200202164722-d101bd2416d5/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200519105757-fe76b779f299/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200610111108-226ff32320da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200610111108-226ff32320da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200929083018-4d22bbb62b3c h1:/h0vtH0PyU0xAoZJVcRw1k0Ng+U0JAy3QDiFmppIlIE= golang.org/x/sys v0.0.0-20200929083018-4d22bbb62b3c h1:/h0vtH0PyU0xAoZJVcRw1k0Ng+U0JAy3QDiFmppIlIE=
golang.org/x/sys v0.0.0-20200929083018-4d22bbb62b3c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200929083018-4d22bbb62b3c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk= golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
golang.org/x/text v0.3.3 h1:cokOdA+Jmi5PJGXLlLllQSgYigAEfHXJAERHVMaCc2k= golang.org/x/text v0.3.3 h1:cokOdA+Jmi5PJGXLlLllQSgYigAEfHXJAERHVMaCc2k=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/time v0.0.0-20180412165947-fbb02b2291d2/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20200630173020-3af7569d3a1e h1:EHBhcS0mlXEAVwNyO2dLfjToGsyY4j24pTs2ScHnX7s= golang.org/x/time v0.0.0-20200630173020-3af7569d3a1e h1:EHBhcS0mlXEAVwNyO2dLfjToGsyY4j24pTs2ScHnX7s=
@ -660,6 +774,7 @@ golang.org/x/time v0.0.0-20200630173020-3af7569d3a1e/go.mod h1:tRJNPiyCQ0inRvYxb
golang.org/x/tools v0.0.0-20180221164845-07fd8470d635/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20180221164845-07fd8470d635/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20180828015842-6cd1fcedba52/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20180828015842-6cd1fcedba52/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20181030000716-a0a13e073c7b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20181030221726-6c7e314b6563/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20181030221726-6c7e314b6563/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY= golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY=
@ -691,6 +806,9 @@ golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8T
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543 h1:E7g+9GITq07hpfrRu66IVDexMakfv52eLZ2CXBWiKr4= golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543 h1:E7g+9GITq07hpfrRu66IVDexMakfv52eLZ2CXBWiKr4=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
google.golang.org/api v0.0.0-20180910000450-7ca32eb868bf/go.mod h1:4mhQ8q/RsB7i+udVvVy5NUi08OU8ZlA0gRVgrF7VFY0=
google.golang.org/api v0.0.0-20181030000543-1d582fd0359e/go.mod h1:4mhQ8q/RsB7i+udVvVy5NUi08OU8ZlA0gRVgrF7VFY0=
google.golang.org/api v0.1.0/go.mod h1:UGEZY7KEX120AnNLIHFMKIo4obdJhkp2tPbaPlQx13Y=
google.golang.org/api v0.3.1/go.mod h1:6wY9I6uQWHQ8EM57III9mq/AjF+i8G65rmVagqKMtkk= google.golang.org/api v0.3.1/go.mod h1:6wY9I6uQWHQ8EM57III9mq/AjF+i8G65rmVagqKMtkk=
google.golang.org/api v0.3.2/go.mod h1:6wY9I6uQWHQ8EM57III9mq/AjF+i8G65rmVagqKMtkk= google.golang.org/api v0.3.2/go.mod h1:6wY9I6uQWHQ8EM57III9mq/AjF+i8G65rmVagqKMtkk=
google.golang.org/api v0.4.0/go.mod h1:8k5glujaEP+g9n7WNsDg8QP6cUVNI86fCNMcbazEtwE= google.golang.org/api v0.4.0/go.mod h1:8k5glujaEP+g9n7WNsDg8QP6cUVNI86fCNMcbazEtwE=
@ -702,6 +820,7 @@ google.golang.org/api v0.15.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsb
google.golang.org/api v0.20.0 h1:jz2KixHX7EcCPiQrySzPdnYT7DbINAypCqKZ1Z7GM40= google.golang.org/api v0.20.0 h1:jz2KixHX7EcCPiQrySzPdnYT7DbINAypCqKZ1Z7GM40=
google.golang.org/api v0.20.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE= google.golang.org/api v0.20.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE=
google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM= google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
google.golang.org/appengine v1.2.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
google.golang.org/appengine v1.3.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= google.golang.org/appengine v1.3.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
google.golang.org/appengine v1.5.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= google.golang.org/appengine v1.5.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
@ -709,6 +828,10 @@ google.golang.org/appengine v1.6.1/go.mod h1:i06prIuMbXzDqacNJfV5OdTW448YApPu5ww
google.golang.org/appengine v1.6.5 h1:tycE03LOZYQNhDpS27tcQdAzLCVMaj7QT2SXxebnpCM= google.golang.org/appengine v1.6.5 h1:tycE03LOZYQNhDpS27tcQdAzLCVMaj7QT2SXxebnpCM=
google.golang.org/appengine v1.6.5/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc= google.golang.org/appengine v1.6.5/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc=
google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc= google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
google.golang.org/genproto v0.0.0-20180831171423-11092d34479b/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
google.golang.org/genproto v0.0.0-20181029155118-b69ba1387ce2/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
google.golang.org/genproto v0.0.0-20181202183823-bd91e49a0898/go.mod h1:7Ep/1NZk928CDR8SjdVbjWNpdIf6nzjE3BTgJDr2Atg=
google.golang.org/genproto v0.0.0-20190306203927-b5d61aea6440/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
google.golang.org/genproto v0.0.0-20190307195333-5fe7a883aa19/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE= google.golang.org/genproto v0.0.0-20190307195333-5fe7a883aa19/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
google.golang.org/genproto v0.0.0-20190404172233-64821d5d2107/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE= google.golang.org/genproto v0.0.0-20190404172233-64821d5d2107/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
google.golang.org/genproto v0.0.0-20190418145605-e7d98fc518a7/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE= google.golang.org/genproto v0.0.0-20190418145605-e7d98fc518a7/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
@ -720,6 +843,8 @@ google.golang.org/genproto v0.0.0-20190911173649-1774047e7e51/go.mod h1:IbNlFCBr
google.golang.org/genproto v0.0.0-20191108220845-16a3f7862a1a/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc= google.golang.org/genproto v0.0.0-20191108220845-16a3f7862a1a/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20200115191322-ca5a22157cba h1:pRj9OXZbwNtbtZtOB4dLwfK4u+EVRMvP+e9zKkg2grM= google.golang.org/genproto v0.0.0-20200115191322-ca5a22157cba h1:pRj9OXZbwNtbtZtOB4dLwfK4u+EVRMvP+e9zKkg2grM=
google.golang.org/genproto v0.0.0-20200115191322-ca5a22157cba/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc= google.golang.org/genproto v0.0.0-20200115191322-ca5a22157cba/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/grpc v1.14.0/go.mod h1:yo6s7OP7yaDglbqo1J04qKzAhqBH6lvTonzMVmEdcZw=
google.golang.org/grpc v1.16.0/go.mod h1:0JHn/cJsOMiMfNA9+DeHDlAU7KAAB5GDlYFpa9MZMio=
google.golang.org/grpc v1.17.0/go.mod h1:6QZJwpn2B+Zp71q/5VxRsJ6NXXVCE5NRUHRo+f3cWCs= google.golang.org/grpc v1.17.0/go.mod h1:6QZJwpn2B+Zp71q/5VxRsJ6NXXVCE5NRUHRo+f3cWCs=
google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c= google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
google.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiqXj38= google.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiqXj38=
@ -730,6 +855,13 @@ google.golang.org/grpc v1.26.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8
google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk= google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
google.golang.org/grpc v1.27.1 h1:zvIju4sqAGvwKspUQOhwnpcqSbzi7/H6QomNNjTL4sk= google.golang.org/grpc v1.27.1 h1:zvIju4sqAGvwKspUQOhwnpcqSbzi7/H6QomNNjTL4sk=
google.golang.org/grpc v1.27.1/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk= google.golang.org/grpc v1.27.1/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8=
google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0=
google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM=
google.golang.org/protobuf v1.20.1-0.20200309200217-e05f789c0967/go.mod h1:A+miEFZTKqfCUM6K7xSMQL9OKL/b6hQv+e19PK+JZNE=
google.golang.org/protobuf v1.21.0/go.mod h1:47Nbq4nVaFHyn7ilMalzfO3qCViNmqZ2kzikPIcrTAo=
google.golang.org/protobuf v1.23.0 h1:4MY060fB1DLGMB/7MBTLnwQUY6+F09GEiz6SsrNqyzM=
google.golang.org/protobuf v1.23.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw= gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 h1:qIbj1fsPNlZgppZ+VLlY7N33q108Sa+fhmuc+sWQYwY= gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 h1:qIbj1fsPNlZgppZ+VLlY7N33q108Sa+fhmuc+sWQYwY=
@ -749,10 +881,13 @@ gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.4 h1:/eiJrUcujPVeJ3xlSWaiNi3uSVmDGBK1pDHUHAnao1I= gopkg.in/yaml.v2 v2.2.4 h1:/eiJrUcujPVeJ3xlSWaiNi3uSVmDGBK1pDHUHAnao1I=
gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.3.0 h1:clyUAQHOM3G0M3f5vQj7LuJrETvjVot3Z5el9nffUtU=
gopkg.in/yaml.v2 v2.3.0/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c h1:dUUwHk2QECo/6vqA44rthZ8ie2QXMNeKRTHCNY2nXvo= gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c h1:dUUwHk2QECo/6vqA44rthZ8ie2QXMNeKRTHCNY2nXvo=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gotest.tools v2.2.0+incompatible h1:VsBPFP1AI068pPrMxtb/S8Zkgf9xEmTLJjfM+P5UIEo= gotest.tools v2.2.0+incompatible h1:VsBPFP1AI068pPrMxtb/S8Zkgf9xEmTLJjfM+P5UIEo=
gotest.tools v2.2.0+incompatible/go.mod h1:DsYFclhRJ6vuDpmuTbkuFWG+y2sxOXAzmJt81HFBacw= gotest.tools v2.2.0+incompatible/go.mod h1:DsYFclhRJ6vuDpmuTbkuFWG+y2sxOXAzmJt81HFBacw=
grpc.go4.org v0.0.0-20170609214715-11d0a25b4919/go.mod h1:77eQGdRu53HpSqPFJFmuJdjuHRquDANNeA4x7B8WQ9o=
honnef.co/go/tools v0.0.0-20180728063816-88497007e858/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= honnef.co/go/tools v0.0.0-20180728063816-88497007e858/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.0-20190106161140-3f1c8253044a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= honnef.co/go/tools v0.0.0-20190106161140-3f1c8253044a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
@ -761,18 +896,23 @@ honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWh
honnef.co/go/tools v0.0.1-2019.2.3 h1:3JgtbtFHMiCmsznwGVTUWbgGov+pVqnlf1dEJTNAXeM= honnef.co/go/tools v0.0.1-2019.2.3 h1:3JgtbtFHMiCmsznwGVTUWbgGov+pVqnlf1dEJTNAXeM=
honnef.co/go/tools v0.0.1-2019.2.3/go.mod h1:a3bituU0lyd329TUQxRnasdCoJDkEUEAqEt0JzvZhAg= honnef.co/go/tools v0.0.1-2019.2.3/go.mod h1:a3bituU0lyd329TUQxRnasdCoJDkEUEAqEt0JzvZhAg=
rsc.io/binaryregexp v0.2.0/go.mod h1:qTv7/COck+e2FymRvadv62gMdZztPaShugOCi3I+8D8= rsc.io/binaryregexp v0.2.0/go.mod h1:qTv7/COck+e2FymRvadv62gMdZztPaShugOCi3I+8D8=
rsc.io/quote/v3 v3.1.0/go.mod h1:yEA65RcK8LyAZtP9Kv3t0HmxON59tX3rD+tICJqUlj0=
rsc.io/sampler v1.3.0/go.mod h1:T1hPZKmBbMNahiBKFy5HrXp6adAjACjK9JXDnKaTXpA=
sourcegraph.com/sourcegraph/go-diff v0.5.0/go.mod h1:kuch7UrkMzY0X+p9CRK03kfuPQ2zzQcaEFbx8wA8rck=
sourcegraph.com/sqs/pbtypes v0.0.0-20180604144634-d3ebe8f20ae4/go.mod h1:ketZ/q3QxT9HOBeFhu6RdvsftgpsbFHBF5Cas6cDKZ0=
storj.io/common v0.0.0-20200424175742-65ac59022f4f/go.mod h1:pZyXiIE7bGETIRXtfs0nICqMwp7PM8HqnDuyUeldNA0= storj.io/common v0.0.0-20200424175742-65ac59022f4f/go.mod h1:pZyXiIE7bGETIRXtfs0nICqMwp7PM8HqnDuyUeldNA0=
storj.io/common v0.0.0-20201026135900-1aaeec90670b/go.mod h1:GqdmNf3fLm2UZX/7Zr0BLFCJ4gFjgm6eHrk/fnmr5jQ= storj.io/common v0.0.0-20201026135900-1aaeec90670b/go.mod h1:GqdmNf3fLm2UZX/7Zr0BLFCJ4gFjgm6eHrk/fnmr5jQ=
storj.io/common v0.0.0-20201027143432-3718579e12bf/go.mod h1:9iobNl9eI6C2M23FS/b37yFYOdHpoeJ8BFFcxsmv538= storj.io/common v0.0.0-20201106104920-372a344bdd45 h1:pv552R7MiRA8VLQC4qXczLjbl2Qb/MNyus2E9NBSXgI=
storj.io/common v0.0.0-20201030120157-90ae6720d87e h1:6baDicBbR0/2XgcQ068KN+B4dF6akkdh2vemmXka1ns= storj.io/common v0.0.0-20201106104920-372a344bdd45/go.mod h1:ZkQZup2jpFZvvTgz+yPc7K4Vr4bBHM8AA66P57MZkjk=
storj.io/common v0.0.0-20201030120157-90ae6720d87e/go.mod h1:9iobNl9eI6C2M23FS/b37yFYOdHpoeJ8BFFcxsmv538=
storj.io/drpc v0.0.11/go.mod h1:TiFc2obNjL9/3isMW1Rpxjy8V9uE0B2HMeMFGiiI7Iw= storj.io/drpc v0.0.11/go.mod h1:TiFc2obNjL9/3isMW1Rpxjy8V9uE0B2HMeMFGiiI7Iw=
storj.io/drpc v0.0.11/go.mod h1:TiFc2obNjL9/3isMW1Rpxjy8V9uE0B2HMeMFGiiI7Iw= storj.io/drpc v0.0.11/go.mod h1:TiFc2obNjL9/3isMW1Rpxjy8V9uE0B2HMeMFGiiI7Iw=
storj.io/drpc v0.0.14 h1:GCBdymTt1BRw4oHmmUZZlxYXLVRxxYj6x3Ivide2J+I= storj.io/drpc v0.0.14 h1:GCBdymTt1BRw4oHmmUZZlxYXLVRxxYj6x3Ivide2J+I=
storj.io/drpc v0.0.14/go.mod h1:82nfl+6YwRwF6UG31cEWWUqv/FaKvP5SGqUvoqTxCMA= storj.io/drpc v0.0.14/go.mod h1:82nfl+6YwRwF6UG31cEWWUqv/FaKvP5SGqUvoqTxCMA=
storj.io/drpc v0.0.16 h1:9sxypc5lKi/0D69cR21BR0S21+IvXfON8L5nXMVNTwQ=
storj.io/drpc v0.0.16/go.mod h1:zdmQ93nx4Z35u11pQ+GAnBy4DGOK3HJCSOfeh2RryTo=
storj.io/monkit-jaeger v0.0.0-20200518165323-80778fc3f91b h1:Bbg9JCtY6l3HrDxs3BXzT2UYnYCBLqNi6i84Y8QIPUs= storj.io/monkit-jaeger v0.0.0-20200518165323-80778fc3f91b h1:Bbg9JCtY6l3HrDxs3BXzT2UYnYCBLqNi6i84Y8QIPUs=
storj.io/monkit-jaeger v0.0.0-20200518165323-80778fc3f91b/go.mod h1:gj4vuCeyCRjRmH8LIrgoyU9Dc9uR6H+/GcDUXmTbf80= storj.io/monkit-jaeger v0.0.0-20200518165323-80778fc3f91b/go.mod h1:gj4vuCeyCRjRmH8LIrgoyU9Dc9uR6H+/GcDUXmTbf80=
storj.io/private v0.0.0-20201026143115-bc926bfa3bca h1:ekR7vtUYC5+cDyim0ZJaSZeXidyzQqDYsnFPYXgTozc= storj.io/private v0.0.0-20201026143115-bc926bfa3bca h1:ekR7vtUYC5+cDyim0ZJaSZeXidyzQqDYsnFPYXgTozc=
storj.io/private v0.0.0-20201026143115-bc926bfa3bca/go.mod h1:EaLnIyNyqWQUJB+7+KWVez0In9czl0nHHlm2WobebuA= storj.io/private v0.0.0-20201026143115-bc926bfa3bca/go.mod h1:EaLnIyNyqWQUJB+7+KWVez0In9czl0nHHlm2WobebuA=
storj.io/uplink v1.3.2-0.20201028181609-f6efc8fcf771 h1:jPbw74xt8bvv8nOfBaM4g9Ts4moX8mqfD4N/B8vEJrA= storj.io/uplink v1.3.2-0.20201106105834-ec8e5cc29f7e h1:a58hzcYciTvBtWST+Byoj76NWuYdnPcz2GK8ynyEyfA=
storj.io/uplink v1.3.2-0.20201028181609-f6efc8fcf771/go.mod h1:5do8jvbs4ao4tLdIZKzNFJPVKOH1oDfvVf8OIsR5Z9E= storj.io/uplink v1.3.2-0.20201106105834-ec8e5cc29f7e/go.mod h1:mrdt4I4EhPRC7cnvCD5490IBm423pgKrVoUiC9a5Srg=

40
multinode/console/members.go Normal file
View File

@ -0,0 +1,40 @@
// Copyright (C) 2020 Storj Labs, Inc.
// See LICENSE for copying information.
package console
import (
"context"
"github.com/zeebo/errs"
"storj.io/common/uuid"
)
// Members exposes needed by MND MembersDB functionality.
//
// architecture: Database
type Members interface {
// Invite will create empty row in membersDB.
Invite(ctx context.Context, member Member) error
// Update updates all updatable fields of member.
Update(ctx context.Context, member Member) error
// Remove deletes member from membersDB.
Remove(ctx context.Context, id uuid.UUID) error
// GetByEmail will return member with specified email.
GetByEmail(ctx context.Context, email string) (Member, error)
// GetByID will return member with specified id.
GetByID(ctx context.Context, id uuid.UUID) (Member, error)
}
// ErrNoMember is a special error type that indicates about absence of member in MembersDB.
var ErrNoMember = errs.Class("no such member")
// Member represents some person that is invited to the MND by node owner.
// Member will have configurable access privileges that will define which functions and which nodes are available for him.
type Member struct {
ID uuid.UUID
Email string
Name string
PasswordHash []byte
}

64
multinode/console/members_test.go Normal file
View File

@ -0,0 +1,64 @@
// Copyright (C) 2020 Storj Labs, Inc.
// See LICENSE for copying information.
package console_test
import (
"testing"
"github.com/stretchr/testify/require"
"github.com/zeebo/assert"
"storj.io/common/testcontext"
"storj.io/common/uuid"
"storj.io/storj/multinode"
"storj.io/storj/multinode/console"
"storj.io/storj/multinode/multinodedb/multinodedbtest"
)
func TestMembersDB(t *testing.T) {
multinodedbtest.Run(t, func(ctx *testcontext.Context, t *testing.T, db multinode.DB) {
members := db.Members()
memberID, err := uuid.New()
require.NoError(t, err)
memberBob := console.Member{
ID: memberID,
Email: "mail@example.com",
Name: "Bob",
PasswordHash: []byte{0},
}
err = members.Invite(ctx, memberBob)
assert.NoError(t, err)
memberToCheck, err := members.GetByEmail(ctx, memberBob.Email)
assert.NoError(t, err)
assert.Equal(t, memberToCheck.Email, memberBob.Email)
assert.Equal(t, memberToCheck.Name, memberBob.Name)
assert.Equal(t, memberToCheck.Email, memberBob.Email)
memberBob.Name = "Alice"
err = members.Update(ctx, memberBob)
assert.NoError(t, err)
memberAlice, err := members.GetByID(ctx, memberToCheck.ID)
assert.NoError(t, err)
assert.Equal(t, memberToCheck.Email, memberAlice.Email)
assert.Equal(t, memberToCheck.Name, memberAlice.Name)
assert.Equal(t, memberToCheck.Email, memberAlice.Email)
assert.Equal(t, memberToCheck.ID, memberAlice.ID)
err = members.Remove(ctx, memberAlice.ID)
assert.NoError(t, err)
_, err = members.GetByID(ctx, memberToCheck.ID)
assert.Error(t, err)
assert.Equal(t, true, console.ErrNoMember.Has(err))
_, err = members.GetByEmail(ctx, memberToCheck.Email)
assert.Error(t, err)
assert.Equal(t, true, console.ErrNoMember.Has(err))
})
}

0
multinode/console/node.go → multinode/console/nodes.go
View File

40
multinode/mutlinodedb/database.go → multinode/multinodedb/database.go
View File

@ -1,16 +1,18 @@
// Copyright (C) 2020 Storj Labs, Inc. // Copyright (C) 2020 Storj Labs, Inc.
// See LICENSE for copying information. // See LICENSE for copying information.
package mutlinodedb package multinodedb
import ( import (
"context"
"github.com/spacemonkeygo/monkit/v3" "github.com/spacemonkeygo/monkit/v3"
"github.com/zeebo/errs" "github.com/zeebo/errs"
"go.uber.org/zap" "go.uber.org/zap"
"storj.io/storj/multinode" "storj.io/storj/multinode"
"storj.io/storj/multinode/console" "storj.io/storj/multinode/console"
"storj.io/storj/multinode/mutlinodedb/dbx" "storj.io/storj/multinode/multinodedb/dbx"
"storj.io/storj/private/dbutil" "storj.io/storj/private/dbutil"
"storj.io/storj/private/dbutil/pgutil" "storj.io/storj/private/dbutil/pgutil"
) )
@ -39,8 +41,8 @@ type multinodeDB struct {
source string source string
} }
// New creates instance of database supports postgres. // Open creates instance of database supports postgres.
func New(log *zap.Logger, databaseURL string) (multinode.DB, error) { func Open(ctx context.Context, log *zap.Logger, databaseURL string) (multinode.DB, error) {
driver, source, implementation, err := dbutil.SplitConnStr(databaseURL) driver, source, implementation, err := dbutil.SplitConnStr(databaseURL)
if err != nil { if err != nil {
return nil, err return nil, err
@ -52,17 +54,17 @@ func New(log *zap.Logger, databaseURL string) (multinode.DB, error) {
source = pgutil.CheckApplicationName(source) source = pgutil.CheckApplicationName(source)
// dbxDB, err := dbx.Open(driver, source) dbxDB, err := dbx.Open(driver, source)
// if err != nil { if err != nil {
// return nil, Error.New("failed opening database via DBX at %q: %v", return nil, Error.New("failed opening database via DBX at %q: %v",
// source, err) source, err)
// } }
// log.Debug("Connected to:", zap.String("db source", source)) log.Debug("Connected to:", zap.String("db source", source))
// dbutil.Configure(dbxDB.DB, "multinodedb", mon) dbutil.Configure(ctx, dbxDB.DB, "multinodedb", mon)
core := &multinodeDB{ core := &multinodeDB{
// DB: dbxDB, DB: dbxDB,
log: log, log: log,
driver: driver, driver: driver,
@ -77,6 +79,18 @@ func New(log *zap.Logger, databaseURL string) (multinode.DB, error) {
func (db *multinodeDB) Nodes() console.Nodes { func (db *multinodeDB) Nodes() console.Nodes {
return &nodes{ return &nodes{
methods: db, methods: db,
db: db,
} }
} }
// Members returns members database.
func (db *multinodeDB) Members() console.Members {
return &members{
methods: db,
}
}
// CreateSchema creates schema.
func (db *multinodeDB) CreateSchema(ctx context.Context) error {
_, err := db.ExecContext(ctx, db.DB.Schema())
return err
}

24
multinode/multinodedb/dbx/gen.go Normal file
View File

@ -0,0 +1,24 @@
// Copyright (C) 2020 Storj Labs, Inc.
// See LICENSE for copying information.
package dbx
import (
"github.com/spacemonkeygo/monkit/v3"
"github.com/zeebo/errs"
)
//go:generate sh gen.sh
var mon = monkit.Package()
func init() {
// catch dbx errors
class := errs.Class("multinodedb dbx error")
WrapErr = func(e *Error) error {
if e.Code == ErrorCode_NoRows {
return e.Err
}
return class.Wrap(e)
}
}

0
multinode/mutlinodedb/dbx/gen.sh → multinode/multinodedb/dbx/gen.sh
View File

45
multinode/multinodedb/dbx/multinodedb.dbx Normal file
View File

@ -0,0 +1,45 @@
// dbx.v1 golang multinodedb.dbx .
model node (
key id
field id blob
field name text ( updatable )
field tag text ( updatable )
field public_address text
field api_secret blob
field logo blob ( updatable )
)
create node ( )
delete node ( where node.id = ? )
read one (
select node
where node.id = ?
)
model member (
key id
field id blob
field email text ( updatable )
field name text ( updatable )
field password_hash blob ( updatable )
field created_at timestamp ( autoinsert )
)
create member ( )
delete member ( where member.id = ? )
update member ( where member.id = ? )
read one (
select member
where member.email = ?
)
read one (
select member
where member.id = ?
)

386
multinode/mutlinodedb/dbx/multinodedb.dbx.go → multinode/multinodedb/dbx/multinodedb.dbx.go
View File

@ -269,7 +269,15 @@ func newpgx(db *DB) *pgxDB {
} }
func (obj *pgxDB) Schema() string { func (obj *pgxDB) Schema() string {
return `CREATE TABLE nodes ( return `CREATE TABLE members (
id bytea NOT NULL,
email text NOT NULL,
name text NOT NULL,
password_hash bytea NOT NULL,
created_at timestamp with time zone NOT NULL,
PRIMARY KEY ( id )
);
CREATE TABLE nodes (
id bytea NOT NULL, id bytea NOT NULL,
name text NOT NULL, name text NOT NULL,
tag text NOT NULL, tag text NOT NULL,
@ -340,6 +348,117 @@ nextval:
fmt.Fprint(f, "]") fmt.Fprint(f, "]")
} }
type Member struct {
Id []byte
Email string
Name string
PasswordHash []byte
CreatedAt time.Time
}
func (Member) _Table() string { return "members" }
type Member_Update_Fields struct {
Email Member_Email_Field
Name Member_Name_Field
PasswordHash Member_PasswordHash_Field
}
type Member_Id_Field struct {
_set bool
_null bool
_value []byte
}
func Member_Id(v []byte) Member_Id_Field {
return Member_Id_Field{_set: true, _value: v}
}
func (f Member_Id_Field) value() interface{} {
if !f._set || f._null {
return nil
}
return f._value
}
func (Member_Id_Field) _Column() string { return "id" }
type Member_Email_Field struct {
_set bool
_null bool
_value string
}
func Member_Email(v string) Member_Email_Field {
return Member_Email_Field{_set: true, _value: v}
}
func (f Member_Email_Field) value() interface{} {
if !f._set || f._null {
return nil
}
return f._value
}
func (Member_Email_Field) _Column() string { return "email" }
type Member_Name_Field struct {
_set bool
_null bool
_value string
}
func Member_Name(v string) Member_Name_Field {
return Member_Name_Field{_set: true, _value: v}
}
func (f Member_Name_Field) value() interface{} {
if !f._set || f._null {
return nil
}
return f._value
}
func (Member_Name_Field) _Column() string { return "name" }
type Member_PasswordHash_Field struct {
_set bool
_null bool
_value []byte
}
func Member_PasswordHash(v []byte) Member_PasswordHash_Field {
return Member_PasswordHash_Field{_set: true, _value: v}
}
func (f Member_PasswordHash_Field) value() interface{} {
if !f._set || f._null {
return nil
}
return f._value
}
func (Member_PasswordHash_Field) _Column() string { return "password_hash" }
type Member_CreatedAt_Field struct {
_set bool
_null bool
_value time.Time
}
func Member_CreatedAt(v time.Time) Member_CreatedAt_Field {
return Member_CreatedAt_Field{_set: true, _value: v}
}
func (f Member_CreatedAt_Field) value() interface{} {
if !f._set || f._null {
return nil
}
return f._value
}
func (Member_CreatedAt_Field) _Column() string { return "created_at" }
type Node struct { type Node struct {
Id []byte Id []byte
Name string Name string
@ -924,6 +1043,38 @@ func (obj *pgxImpl) Create_Node(ctx context.Context,
} }
func (obj *pgxImpl) Create_Member(ctx context.Context,
member_id Member_Id_Field,
member_email Member_Email_Field,
member_name Member_Name_Field,
member_password_hash Member_PasswordHash_Field) (
member *Member, err error) {
defer mon.Task()(&ctx)(&err)
__now := obj.db.Hooks.Now().UTC()
__id_val := member_id.value()
__email_val := member_email.value()
__name_val := member_name.value()
__password_hash_val := member_password_hash.value()
__created_at_val := __now
var __embed_stmt = __sqlbundle_Literal("INSERT INTO members ( id, email, name, password_hash, created_at ) VALUES ( ?, ?, ?, ?, ? ) RETURNING members.id, members.email, members.name, members.password_hash, members.created_at")
var __values []interface{}
__values = append(__values, __id_val, __email_val, __name_val, __password_hash_val, __created_at_val)
var __stmt = __sqlbundle_Render(obj.dialect, __embed_stmt)
obj.logStmt(__stmt, __values...)
member = &Member{}
err = obj.driver.QueryRowContext(ctx, __stmt, __values...).Scan(&member.Id, &member.Email, &member.Name, &member.PasswordHash, &member.CreatedAt)
if err != nil {
return nil, obj.makeErr(err)
}
return member, nil
}
func (obj *pgxImpl) Get_Node_By_Id(ctx context.Context, func (obj *pgxImpl) Get_Node_By_Id(ctx context.Context,
node_id Node_Id_Field) ( node_id Node_Id_Field) (
node *Node, err error) { node *Node, err error) {
@ -946,6 +1097,123 @@ func (obj *pgxImpl) Get_Node_By_Id(ctx context.Context,
} }
func (obj *pgxImpl) Get_Member_By_Email(ctx context.Context,
member_email Member_Email_Field) (
member *Member, err error) {
defer mon.Task()(&ctx)(&err)
var __embed_stmt = __sqlbundle_Literal("SELECT members.id, members.email, members.name, members.password_hash, members.created_at FROM members WHERE members.email = ? LIMIT 2")
var __values []interface{}
__values = append(__values, member_email.value())
var __stmt = __sqlbundle_Render(obj.dialect, __embed_stmt)
obj.logStmt(__stmt, __values...)
__rows, err := obj.driver.QueryContext(ctx, __stmt, __values...)
if err != nil {
return nil, obj.makeErr(err)
}
defer __rows.Close()
if !__rows.Next() {
if err := __rows.Err(); err != nil {
return nil, obj.makeErr(err)
}
return nil, makeErr(sql.ErrNoRows)
}
member = &Member{}
err = __rows.Scan(&member.Id, &member.Email, &member.Name, &member.PasswordHash, &member.CreatedAt)
if err != nil {
return nil, obj.makeErr(err)
}
if __rows.Next() {
return nil, tooManyRows("Member_By_Email")
}
if err := __rows.Err(); err != nil {
return nil, obj.makeErr(err)
}
return member, nil
}
func (obj *pgxImpl) Get_Member_By_Id(ctx context.Context,
member_id Member_Id_Field) (
member *Member, err error) {
defer mon.Task()(&ctx)(&err)
var __embed_stmt = __sqlbundle_Literal("SELECT members.id, members.email, members.name, members.password_hash, members.created_at FROM members WHERE members.id = ?")
var __values []interface{}
__values = append(__values, member_id.value())
var __stmt = __sqlbundle_Render(obj.dialect, __embed_stmt)
obj.logStmt(__stmt, __values...)
member = &Member{}
err = obj.driver.QueryRowContext(ctx, __stmt, __values...).Scan(&member.Id, &member.Email, &member.Name, &member.PasswordHash, &member.CreatedAt)
if err != nil {
return (*Member)(nil), obj.makeErr(err)
}
return member, nil
}
func (obj *pgxImpl) Update_Member_By_Id(ctx context.Context,
member_id Member_Id_Field,
update Member_Update_Fields) (
member *Member, err error) {
defer mon.Task()(&ctx)(&err)
var __sets = &__sqlbundle_Hole{}
var __embed_stmt = __sqlbundle_Literals{Join: "", SQLs: []__sqlbundle_SQL{__sqlbundle_Literal("UPDATE members SET "), __sets, __sqlbundle_Literal(" WHERE members.id = ? RETURNING members.id, members.email, members.name, members.password_hash, members.created_at")}}
__sets_sql := __sqlbundle_Literals{Join: ", "}
var __values []interface{}
var __args []interface{}
if update.Email._set {
__values = append(__values, update.Email.value())
__sets_sql.SQLs = append(__sets_sql.SQLs, __sqlbundle_Literal("email = ?"))
}
if update.Name._set {
__values = append(__values, update.Name.value())
__sets_sql.SQLs = append(__sets_sql.SQLs, __sqlbundle_Literal("name = ?"))
}
if update.PasswordHash._set {
__values = append(__values, update.PasswordHash.value())
__sets_sql.SQLs = append(__sets_sql.SQLs, __sqlbundle_Literal("password_hash = ?"))
}
if len(__sets_sql.SQLs) == 0 {
return nil, emptyUpdate()
}
__args = append(__args, member_id.value())
__values = append(__values, __args...)
__sets.SQL = __sets_sql
var __stmt = __sqlbundle_Render(obj.dialect, __embed_stmt)
obj.logStmt(__stmt, __values...)
member = &Member{}
err = obj.driver.QueryRowContext(ctx, __stmt, __values...).Scan(&member.Id, &member.Email, &member.Name, &member.PasswordHash, &member.CreatedAt)
if err == sql.ErrNoRows {
return nil, nil
}
if err != nil {
return nil, obj.makeErr(err)
}
return member, nil
}
func (obj *pgxImpl) Delete_Node_By_Id(ctx context.Context, func (obj *pgxImpl) Delete_Node_By_Id(ctx context.Context,
node_id Node_Id_Field) ( node_id Node_Id_Field) (
deleted bool, err error) { deleted bool, err error) {
@ -973,6 +1241,33 @@ func (obj *pgxImpl) Delete_Node_By_Id(ctx context.Context,
} }
func (obj *pgxImpl) Delete_Member_By_Id(ctx context.Context,
member_id Member_Id_Field) (
deleted bool, err error) {
defer mon.Task()(&ctx)(&err)
var __embed_stmt = __sqlbundle_Literal("DELETE FROM members WHERE members.id = ?")
var __values []interface{}
__values = append(__values, member_id.value())
var __stmt = __sqlbundle_Render(obj.dialect, __embed_stmt)
obj.logStmt(__stmt, __values...)
__res, err := obj.driver.ExecContext(ctx, __stmt, __values...)
if err != nil {
return false, obj.makeErr(err)
}
__count, err := __res.RowsAffected()
if err != nil {
return false, obj.makeErr(err)
}
return __count > 0, nil
}
func (impl pgxImpl) isConstraintError(err error) ( func (impl pgxImpl) isConstraintError(err error) (
constraint string, ok bool) { constraint string, ok bool) {
if e, ok := err.(*pgconn.PgError); ok { if e, ok := err.(*pgconn.PgError); ok {
@ -992,6 +1287,16 @@ func (obj *pgxImpl) deleteAll(ctx context.Context) (count int64, err error) {
return 0, obj.makeErr(err) return 0, obj.makeErr(err)
} }
__count, err = __res.RowsAffected()
if err != nil {
return 0, obj.makeErr(err)
}
count += __count
__res, err = obj.driver.ExecContext(ctx, "DELETE FROM members;")
if err != nil {
return 0, obj.makeErr(err)
}
__count, err = __res.RowsAffected() __count, err = __res.RowsAffected()
if err != nil { if err != nil {
return 0, obj.makeErr(err) return 0, obj.makeErr(err)
@ -1044,6 +1349,20 @@ func (rx *Rx) Rollback() (err error) {
return err return err
} }
func (rx *Rx) Create_Member(ctx context.Context,
member_id Member_Id_Field,
member_email Member_Email_Field,
member_name Member_Name_Field,
member_password_hash Member_PasswordHash_Field) (
member *Member, err error) {
var tx *Tx
if tx, err = rx.getTx(ctx); err != nil {
return
}
return tx.Create_Member(ctx, member_id, member_email, member_name, member_password_hash)
}
func (rx *Rx) Create_Node(ctx context.Context, func (rx *Rx) Create_Node(ctx context.Context,
node_id Node_Id_Field, node_id Node_Id_Field,
node_name Node_Name_Field, node_name Node_Name_Field,
@ -1060,6 +1379,16 @@ func (rx *Rx) Create_Node(ctx context.Context,
} }
func (rx *Rx) Delete_Member_By_Id(ctx context.Context,
member_id Member_Id_Field) (
deleted bool, err error) {
var tx *Tx
if tx, err = rx.getTx(ctx); err != nil {
return
}
return tx.Delete_Member_By_Id(ctx, member_id)
}
func (rx *Rx) Delete_Node_By_Id(ctx context.Context, func (rx *Rx) Delete_Node_By_Id(ctx context.Context,
node_id Node_Id_Field) ( node_id Node_Id_Field) (
deleted bool, err error) { deleted bool, err error) {
@ -1070,6 +1399,26 @@ func (rx *Rx) Delete_Node_By_Id(ctx context.Context,
return tx.Delete_Node_By_Id(ctx, node_id) return tx.Delete_Node_By_Id(ctx, node_id)
} }
func (rx *Rx) Get_Member_By_Email(ctx context.Context,
member_email Member_Email_Field) (
member *Member, err error) {
var tx *Tx
if tx, err = rx.getTx(ctx); err != nil {
return
}
return tx.Get_Member_By_Email(ctx, member_email)
}
func (rx *Rx) Get_Member_By_Id(ctx context.Context,
member_id Member_Id_Field) (
member *Member, err error) {
var tx *Tx
if tx, err = rx.getTx(ctx); err != nil {
return
}
return tx.Get_Member_By_Id(ctx, member_id)
}
func (rx *Rx) Get_Node_By_Id(ctx context.Context, func (rx *Rx) Get_Node_By_Id(ctx context.Context,
node_id Node_Id_Field) ( node_id Node_Id_Field) (
node *Node, err error) { node *Node, err error) {
@ -1080,7 +1429,25 @@ func (rx *Rx) Get_Node_By_Id(ctx context.Context,
return tx.Get_Node_By_Id(ctx, node_id) return tx.Get_Node_By_Id(ctx, node_id)
} }
func (rx *Rx) Update_Member_By_Id(ctx context.Context,
member_id Member_Id_Field,
update Member_Update_Fields) (
member *Member, err error) {
var tx *Tx
if tx, err = rx.getTx(ctx); err != nil {
return
}
return tx.Update_Member_By_Id(ctx, member_id, update)
}
type Methods interface { type Methods interface {
Create_Member(ctx context.Context,
member_id Member_Id_Field,
member_email Member_Email_Field,
member_name Member_Name_Field,
member_password_hash Member_PasswordHash_Field) (
member *Member, err error)
Create_Node(ctx context.Context, Create_Node(ctx context.Context,
node_id Node_Id_Field, node_id Node_Id_Field,
node_name Node_Name_Field, node_name Node_Name_Field,
@ -1090,13 +1457,30 @@ type Methods interface {
node_logo Node_Logo_Field) ( node_logo Node_Logo_Field) (
node *Node, err error) node *Node, err error)
Delete_Member_By_Id(ctx context.Context,
member_id Member_Id_Field) (
deleted bool, err error)
Delete_Node_By_Id(ctx context.Context, Delete_Node_By_Id(ctx context.Context,
node_id Node_Id_Field) ( node_id Node_Id_Field) (
deleted bool, err error) deleted bool, err error)
Get_Member_By_Email(ctx context.Context,
member_email Member_Email_Field) (
member *Member, err error)
Get_Member_By_Id(ctx context.Context,
member_id Member_Id_Field) (
member *Member, err error)
Get_Node_By_Id(ctx context.Context, Get_Node_By_Id(ctx context.Context,
node_id Node_Id_Field) ( node_id Node_Id_Field) (
node *Node, err error) node *Node, err error)
Update_Member_By_Id(ctx context.Context,
member_id Member_Id_Field,
update Member_Update_Fields) (
member *Member, err error)
} }
type TxMethods interface { type TxMethods interface {

8
multinode/mutlinodedb/dbx/multinodedb.dbx.pgx.sql → multinode/multinodedb/dbx/multinodedb.dbx.pgx.sql
View File

@ -1,5 +1,13 @@
-- AUTOGENERATED BY storj.io/dbx -- AUTOGENERATED BY storj.io/dbx
-- DO NOT EDIT -- DO NOT EDIT
CREATE TABLE members (
id bytea NOT NULL,
email text NOT NULL,
name text NOT NULL,
password_hash bytea NOT NULL,
created_at timestamp with time zone NOT NULL,
PRIMARY KEY ( id )
);
CREATE TABLE nodes ( CREATE TABLE nodes (
id bytea NOT NULL, id bytea NOT NULL,
name text NOT NULL, name text NOT NULL,

0
multinode/mutlinodedb/dbx/templates/golang.decl.tmpl → multinode/multinodedb/dbx/templates/golang.decl.tmpl
View File

117
multinode/multinodedb/members.go Normal file
View File

@ -0,0 +1,117 @@
// Copyright (C) 2020 Storj Labs, Inc.
// See LICENSE for copying information.
package multinodedb
import (
"context"
"database/sql"
"errors"
"github.com/zeebo/errs"
"storj.io/common/uuid"
"storj.io/storj/multinode/console"
"storj.io/storj/multinode/multinodedb/dbx"
)
// MembersDBError indicates about internal MembersDB error.
var MembersDBError = errs.Class("MembersDB error")
// ensures that members implements console.Members.
var _ console.Members = (*members)(nil)
// members exposes needed by MND MembersDB functionality.
// dbx implementation of console.Members.
//
// architecture: Database
type members struct {
methods dbx.Methods
}
// Invite will create empty row in membersDB.
func (m *members) Invite(ctx context.Context, member console.Member) (err error) {
defer mon.Task()(&ctx)(&err)
id, err := uuid.New()
if err != nil {
return MembersDBError.Wrap(err)
}
_, err = m.methods.Create_Member(ctx, dbx.Member_Id(id[:]), dbx.Member_Email(member.Email), dbx.Member_Name(member.Name), dbx.Member_PasswordHash(member.PasswordHash))
return MembersDBError.Wrap(err)
}
// Update updates all updatable fields of member.
func (m *members) Update(ctx context.Context, member console.Member) (err error) {
defer mon.Task()(&ctx)(&err)
_, err = m.methods.Update_Member_By_Id(ctx, dbx.Member_Id(member.ID[:]), dbx.Member_Update_Fields{
Email: dbx.Member_Email(member.Email),
Name: dbx.Member_Name(member.Name),
PasswordHash: dbx.Member_PasswordHash(member.PasswordHash),
})
return MembersDBError.Wrap(err)
}
// Remove deletes member from membersDB.
func (m *members) Remove(ctx context.Context, id uuid.UUID) (err error) {
defer mon.Task()(&ctx)(&err)
_, err = m.methods.Delete_Member_By_Id(ctx, dbx.Member_Id(id[:]))
return MembersDBError.Wrap(err)
}
// GetByEmail will return member with specified email.
func (m *members) GetByEmail(ctx context.Context, email string) (_ console.Member, err error) {
defer mon.Task()(&ctx)(&err)
memberDbx, err := m.methods.Get_Member_By_Email(ctx, dbx.Member_Email(email))
if err != nil {
if errors.Is(err, sql.ErrNoRows) {
return console.Member{}, console.ErrNoMember.Wrap(err)
}
return console.Member{}, MembersDBError.Wrap(err)
}
member, err := fromDBXMember(memberDbx)
return member, MembersDBError.Wrap(err)
}
// GetByID will return member with specified id.
func (m *members) GetByID(ctx context.Context, id uuid.UUID) (_ console.Member, err error) {
defer mon.Task()(&ctx)(&err)
memberDbx, err := m.methods.Get_Member_By_Id(ctx, dbx.Member_Id(id[:]))
if err != nil {
if errors.Is(err, sql.ErrNoRows) {
return console.Member{}, console.ErrNoMember.Wrap(err)
}
return console.Member{}, MembersDBError.Wrap(err)
}
member, err := fromDBXMember(memberDbx)
return member, MembersDBError.Wrap(err)
}
// fromDBXMember converts dbx.Member to console.Member.
func fromDBXMember(member *dbx.Member) (_ console.Member, err error) {
id, err := uuid.FromBytes(member.Id)
if err != nil {
return console.Member{}, err
}
result := console.Member{
ID: id,
Email: member.Email,
Name: member.Name,
PasswordHash: member.PasswordHash,
}
return result, nil
}

141
multinode/multinodedb/multinodedbtest/run.go Normal file
View File

@ -0,0 +1,141 @@
// Copyright (C) 2020 Storj Labs, Inc.
// See LICENSE for copying information.
package multinodedbtest
import (
"context"
"fmt"
"strconv"
"strings"
"testing"
"github.com/zeebo/errs"
"go.uber.org/zap"
"go.uber.org/zap/zaptest"
"storj.io/common/testcontext"
"storj.io/storj/multinode"
"storj.io/storj/multinode/multinodedb"
"storj.io/storj/multinode/multinodedb/dbx"
"storj.io/storj/private/dbutil"
"storj.io/storj/private/dbutil/pgtest"
"storj.io/storj/private/dbutil/pgutil"
"storj.io/storj/private/dbutil/tempdb"
)
// Database describes a test database.
type Database struct {
Name string
URL string
Message string
}
// tempMasterDB is a multinode.DB-implementing type that cleans up after itself when closed.
type tempMasterDB struct {
multinode.DB
tempDB *dbutil.TempDatabase
}
// Close closes a tempMasterDB and cleans it up afterward.
func (db *tempMasterDB) Close() error {
return errs.Combine(db.DB.Close(), db.tempDB.Close())
}
// TestDBAccess provides a somewhat regularized access to the underlying DB.
func (db *tempMasterDB) TestDBAccess() *dbx.DB {
return db.DB.(interface{ TestDBAccess() *dbx.DB }).TestDBAccess()
}
type ignoreSkip struct{}
func (ignoreSkip) Skip(...interface{}) {}
// SchemaSuffix returns a suffix for schemas.
func SchemaSuffix() string {
return pgutil.CreateRandomTestingSchemaName(6)
}
// SchemaName returns a properly formatted schema string.
func SchemaName(testname, category string, index int, schemaSuffix string) string {
// postgres has a maximum schema length of 64
// we need additional 6 bytes for the random suffix
// and 4 bytes for the index "/S0/""
indexStr := strconv.Itoa(index)
var maxTestNameLen = 64 - len(category) - len(indexStr) - len(schemaSuffix) - 2
if len(testname) > maxTestNameLen {
testname = testname[:maxTestNameLen]
}
if schemaSuffix == "" {
return strings.ToLower(testname + "/" + category + indexStr)
}
return strings.ToLower(testname + "/" + schemaSuffix + "/" + category + indexStr)
}
// CreateMasterDB creates a new satellite database for testing.
func CreateMasterDB(ctx context.Context, log *zap.Logger, name string, category string, index int, dbInfo Database) (db multinode.DB, err error) {
if dbInfo.URL == "" {
return nil, fmt.Errorf("database %s connection string not provided. %s", dbInfo.Name, dbInfo.Message)
}
schemaSuffix := SchemaSuffix()
log.Debug("creating", zap.String("suffix", schemaSuffix))
schema := SchemaName(name, category, index, schemaSuffix)
tempDB, err := tempdb.OpenUnique(ctx, dbInfo.URL, schema)
if err != nil {
return nil, err
}
return CreateMasterDBOnTopOf(ctx, log, tempDB)
}
// CreateMasterDBOnTopOf creates a new satellite database on top of an already existing
// temporary database.
func CreateMasterDBOnTopOf(ctx context.Context, log *zap.Logger, tempDB *dbutil.TempDatabase) (db multinode.DB, err error) {
masterDB, err := multinodedb.Open(ctx, log, tempDB.ConnStr)
return &tempMasterDB{DB: masterDB, tempDB: tempDB}, err
}
// Run method will iterate over all supported databases. Will establish
// connection and will create tables for each DB.
func Run(t *testing.T, test func(ctx *testcontext.Context, t *testing.T, db multinode.DB)) {
masterDB := Database{
Name: "Postgres",
URL: pgtest.PickPostgres(ignoreSkip{}),
Message: "Postgres flag missing, example: -postgres-test-db=" + pgtest.DefaultPostgres + " or use STORJ_TEST_POSTGRES environment variable.",
}
t.Run(masterDB.Name, func(t *testing.T) {
t.Parallel()
ctx := testcontext.New(t)
defer ctx.Cleanup()
if masterDB.URL == "" {
t.Skipf("Database %s connection string not provided. %s", masterDB.Name, masterDB.Message)
}
db, err := CreateMasterDB(ctx, zaptest.NewLogger(t), t.Name(), "T", 0, masterDB)
if err != nil {
t.Fatal(err)
}
defer func() {
err := db.Close()
if err != nil {
t.Fatal(err)
}
}()
err = db.CreateSchema(ctx)
if err != nil {
t.Fatal(err)
}
test(ctx, t, db)
})
}

5
multinode/mutlinodedb/nodes.go → multinode/multinodedb/nodes.go
View File

@ -1,7 +1,7 @@
// Copyright (C) 2020 Storj Labs, Inc. // Copyright (C) 2020 Storj Labs, Inc.
// See LICENSE for copying information. // See LICENSE for copying information.
package mutlinodedb package multinodedb
import ( import (
"context" "context"
@ -10,7 +10,7 @@ import (
"storj.io/common/storj" "storj.io/common/storj"
"storj.io/storj/multinode/console" "storj.io/storj/multinode/console"
"storj.io/storj/multinode/mutlinodedb/dbx" "storj.io/storj/multinode/multinodedb/dbx"
) )
// NodesDBError indicates about internal NodesDB error. // NodesDBError indicates about internal NodesDB error.
@ -25,7 +25,6 @@ var _ console.Nodes = (*nodes)(nil)
// architecture: Database // architecture: Database
type nodes struct { type nodes struct {
methods dbx.Methods methods dbx.Methods
db *multinodeDB
} }
// Add creates new node in NodesDB. // Add creates new node in NodesDB.

12
multinode/mutlinodedb/dbx/gen.go
View File

@ -1,12 +0,0 @@
// Copyright (C) 2020 Storj Labs, Inc.
// See LICENSE for copying information.
package dbx
import (
"github.com/spacemonkeygo/monkit/v3"
)
//go:generate sh gen.sh
var mon = monkit.Package()

20
multinode/mutlinodedb/dbx/multinodedb.dbx
View File

@ -1,20 +0,0 @@
// dbx.v1 golang multinodedb.dbx .
model node (
key id
field id blob
field name text ( updatable )
field tag text ( updatable )
field public_address text
field api_secret blob
field logo blob ( updatable )
)
create node ( )
delete node ( where node.id = ? )
read one (
select node
where node.id = ?
)

4
multinode/peer.go
View File

@ -27,9 +27,13 @@ var (
type DB interface { type DB interface {
// Nodes returns nodes database. // Nodes returns nodes database.
Nodes() console.Nodes Nodes() console.Nodes
// Members returns members database.
Members() console.Members
// Close closes the database. // Close closes the database.
Close() error Close() error
// CreateSchema creates schema.
CreateSchema(ctx context.Context) error
} }
// Config is all the configuration parameters for a Multinode Dashboard. // Config is all the configuration parameters for a Multinode Dashboard.

20
private/dbutil/pgtest/flag.go
View File

@ -5,9 +5,9 @@ package pgtest
import ( import (
"flag" "flag"
"math/rand"
"os" "os"
"strings" "strings"
"sync/atomic"
"testing" "testing"
"storj.io/common/testcontext" "storj.io/common/testcontext"
@ -77,20 +77,20 @@ func Run(t *testing.T, test func(ctx *testcontext.Context, t *testing.T, connstr
} }
} }
// PickPostgres picks a random postgres database from flag. // PickPostgres picks one postgres database from flag.
func PickPostgres(t TB) string { func PickPostgres(t TB) string {
if *postgres == "" || strings.EqualFold(*postgres, "omit") { if *postgres == "" || strings.EqualFold(*postgres, "omit") {
t.Skip("Postgres flag missing, example: -postgres-test-db=" + DefaultPostgres) t.Skip("Postgres flag missing, example: -postgres-test-db=" + DefaultPostgres)
} }
return pickRandom(*postgres) return pickNext(*postgres, &pickPostgres)
} }
// PickCockroach picks a random cockroach database from flag. // PickCockroach picks one cockroach database from flag.
func PickCockroach(t TB) string { func PickCockroach(t TB) string {
if *cockroach == "" || strings.EqualFold(*cockroach, "omit") { if *cockroach == "" || strings.EqualFold(*cockroach, "omit") {
t.Skip("Cockroach flag missing, example: -cockroach-test-db=" + DefaultCockroach) t.Skip("Cockroach flag missing, example: -cockroach-test-db=" + DefaultCockroach)
} }
return pickRandom(*cockroach) return pickNext(*cockroach, &pickCockroach)
} }
// PickCockroachAlt picks an alternate cockroach database from flag. // PickCockroachAlt picks an alternate cockroach database from flag.
@ -104,13 +104,17 @@ func PickCockroachAlt(t TB) string {
t.Skip("Cockroach alt flag omitted.") t.Skip("Cockroach alt flag omitted.")
} }
return pickRandom(*cockroachAlt) return pickNext(*cockroachAlt, &pickCockroach)
} }
func pickRandom(dbstr string) string { var pickPostgres uint64
var pickCockroach uint64
func pickNext(dbstr string, counter *uint64) string {
values := strings.Split(dbstr, ";") values := strings.Split(dbstr, ";")
if len(values) <= 1 { if len(values) <= 1 {
return dbstr return dbstr
} }
return values[rand.Intn(len(values))] v := atomic.AddUint64(counter, 1)
return values[v%uint64(len(values))]
} }

25
private/dbutil/sqliteutil/query.go
View File

@ -73,13 +73,20 @@ func QuerySchema(ctx context.Context, db dbschema.Queryer) (*dbschema.Schema, er
func discoverTables(ctx context.Context, db dbschema.Queryer, schema *dbschema.Schema, tableDefinitions []*definition) (err error) { func discoverTables(ctx context.Context, db dbschema.Queryer, schema *dbschema.Schema, tableDefinitions []*definition) (err error) {
for _, definition := range tableDefinitions { for _, definition := range tableDefinitions {
if err := discoverTable(ctx, db, schema, definition); err != nil {
return err
}
}
return errs.Wrap(err)
}
func discoverTable(ctx context.Context, db dbschema.Queryer, schema *dbschema.Schema, definition *definition) (err error) {
table := schema.EnsureTable(definition.name) table := schema.EnsureTable(definition.name)
tableRows, err := db.QueryContext(ctx, `PRAGMA table_info(`+definition.name+`)`) tableRows, err := db.QueryContext(ctx, `PRAGMA table_info(`+definition.name+`)`)
if err != nil { if err != nil {
return errs.Wrap(err) return errs.Wrap(err)
} }
defer func() { err = errs.Combine(err, tableRows.Close()) }()
for tableRows.Next() { for tableRows.Next() {
var defaultValue sql.NullString var defaultValue sql.NullString
@ -88,7 +95,7 @@ func discoverTables(ctx context.Context, db dbschema.Queryer, schema *dbschema.S
var notNull bool var notNull bool
err := tableRows.Scan(&index, &name, &columnType, &notNull, &defaultValue, &pk) err := tableRows.Scan(&index, &name, &columnType, &notNull, &defaultValue, &pk)
if err != nil { if err != nil {
return errs.Wrap(err) return errs.Wrap(errs.Combine(tableRows.Err(), tableRows.Close(), err))
} }
column := &dbschema.Column{ column := &dbschema.Column{
@ -103,7 +110,10 @@ func discoverTables(ctx context.Context, db dbschema.Queryer, schema *dbschema.S
} }
table.PrimaryKey = append(table.PrimaryKey, name) table.PrimaryKey = append(table.PrimaryKey, name)
} }
}
err = errs.Combine(tableRows.Err(), tableRows.Close())
if err != nil {
return errs.Wrap(err)
} }
matches := rxUnique.FindAllStringSubmatch(definition.sql, -1) matches := rxUnique.FindAllStringSubmatch(definition.sql, -1)
@ -121,14 +131,13 @@ func discoverTables(ctx context.Context, db dbschema.Queryer, schema *dbschema.S
if err != nil { if err != nil {
return errs.Wrap(err) return errs.Wrap(err)
} }
defer func() { err = errs.Combine(err, keysRows.Close()) }()
for keysRows.Next() { for keysRows.Next() {
var id, sec int var id, sec int
var tableName, from, to, onUpdate, onDelete, match string var tableName, from, to, onUpdate, onDelete, match string
err := keysRows.Scan(&id, &sec, &tableName, &from, &to, &onUpdate, &onDelete, &match) err := keysRows.Scan(&id, &sec, &tableName, &from, &to, &onUpdate, &onDelete, &match)
if err != nil { if err != nil {
return errs.Wrap(err) return errs.Wrap(errs.Combine(keysRows.Err(), keysRows.Close(), err))
} }
column, found := table.FindColumn(from) column, found := table.FindColumn(from)
@ -147,10 +156,14 @@ func discoverTables(ctx context.Context, db dbschema.Queryer, schema *dbschema.S
} }
} }
} }
} err = errs.Combine(keysRows.Err(), keysRows.Close())
if err != nil {
return errs.Wrap(err) return errs.Wrap(err)
} }
return nil
}
func discoverIndexes(ctx context.Context, db dbschema.Queryer, schema *dbschema.Schema, indexDefinitions []*definition) (err error) { func discoverIndexes(ctx context.Context, db dbschema.Queryer, schema *dbschema.Schema, indexDefinitions []*definition) (err error) {
// TODO improve indexes discovery // TODO improve indexes discovery
for _, definition := range indexDefinitions { for _, definition := range indexDefinitions {

21
private/lifecycle/group.go
View File

@ -8,6 +8,7 @@ import (
"context" "context"
"errors" "errors"
"runtime/pprof" "runtime/pprof"
"time"
"github.com/spacemonkeygo/monkit/v3" "github.com/spacemonkeygo/monkit/v3"
"github.com/zeebo/errs" "github.com/zeebo/errs"
@ -54,7 +55,27 @@ func (group *Group) Run(ctx context.Context, g *errgroup.Group) {
if item.Run == nil { if item.Run == nil {
continue continue
} }
shutdownCtx, shutdownFinished := context.WithCancel(context.Background())
go func() {
select {
case <-ctx.Done():
case <-shutdownCtx.Done():
return
}
shutdownDeadline := time.NewTimer(15 * time.Second)
defer shutdownDeadline.Stop()
select {
case <-shutdownDeadline.C:
group.log.Warn("service takes long to shutdown", zap.String("name", item.Name))
case <-shutdownCtx.Done():
}
}()
g.Go(func() error { g.Go(func() error {
defer shutdownFinished()
var err error var err error
pprof.Do(ctx, pprof.Labels("name", item.Name), func(ctx context.Context) { pprof.Do(ctx, pprof.Labels("name", item.Name), func(ctx context.Context) {
err = item.Run(ctx) err = item.Run(ctx)

116
private/testblobs/bad.go
View File

@ -5,6 +5,7 @@ package testblobs
import ( import (
"context" "context"
"sync"
"time" "time"
"go.uber.org/zap" "go.uber.org/zap"
@ -43,11 +44,30 @@ func (bad *BadDB) SetError(err error) {
// BadBlobs implements a bad blob store. // BadBlobs implements a bad blob store.
type BadBlobs struct { type BadBlobs struct {
err error err lockedErr
blobs storage.Blobs blobs storage.Blobs
log *zap.Logger log *zap.Logger
} }
type lockedErr struct {
mu sync.Mutex
err error
}
// Err returns the error.
func (m *lockedErr) Err() error {
m.mu.Lock()
defer m.mu.Unlock()
return m.err
}
// Set sets the error.
func (m *lockedErr) Set(err error) {
m.mu.Lock()
defer m.mu.Unlock()
m.err = err
}
// newBadBlobs creates a new bad blob store wrapping the provided blobs. // newBadBlobs creates a new bad blob store wrapping the provided blobs.
// Use SetError to manually configure the error returned by all operations. // Use SetError to manually configure the error returned by all operations.
func newBadBlobs(log *zap.Logger, blobs storage.Blobs) *BadBlobs { func newBadBlobs(log *zap.Logger, blobs storage.Blobs) *BadBlobs {
@ -57,27 +77,32 @@ func newBadBlobs(log *zap.Logger, blobs storage.Blobs) *BadBlobs {
} }
} }
// SetError configures the blob store to return a specific error for all operations.
func (bad *BadBlobs) SetError(err error) {
bad.err.Set(err)
}
// Create creates a new blob that can be written optionally takes a size // Create creates a new blob that can be written optionally takes a size
// argument for performance improvements, -1 is unknown size. // argument for performance improvements, -1 is unknown size.
func (bad *BadBlobs) Create(ctx context.Context, ref storage.BlobRef, size int64) (storage.BlobWriter, error) { func (bad *BadBlobs) Create(ctx context.Context, ref storage.BlobRef, size int64) (storage.BlobWriter, error) {
if bad.err != nil { if err := bad.err.Err(); err != nil {
return nil, bad.err return nil, err
} }
return bad.blobs.Create(ctx, ref, size) return bad.blobs.Create(ctx, ref, size)
} }
// Close closes the blob store and any resources associated with it. // Close closes the blob store and any resources associated with it.
func (bad *BadBlobs) Close() error { func (bad *BadBlobs) Close() error {
if bad.err != nil { if err := bad.err.Err(); err != nil {
return bad.err return err
} }
return bad.blobs.Close() return bad.blobs.Close()
} }
// Open opens a reader with the specified namespace and key. // Open opens a reader with the specified namespace and key.
func (bad *BadBlobs) Open(ctx context.Context, ref storage.BlobRef) (storage.BlobReader, error) { func (bad *BadBlobs) Open(ctx context.Context, ref storage.BlobRef) (storage.BlobReader, error) {
if bad.err != nil { if err := bad.err.Err(); err != nil {
return nil, bad.err return nil, err
} }
return bad.blobs.Open(ctx, ref) return bad.blobs.Open(ctx, ref)
} }
@ -85,64 +110,64 @@ func (bad *BadBlobs) Open(ctx context.Context, ref storage.BlobRef) (storage.Blo
// OpenWithStorageFormat opens a reader for the already-located blob, avoiding the potential need // OpenWithStorageFormat opens a reader for the already-located blob, avoiding the potential need
// to check multiple storage formats to find the blob. // to check multiple storage formats to find the blob.
func (bad *BadBlobs) OpenWithStorageFormat(ctx context.Context, ref storage.BlobRef, formatVer storage.FormatVersion) (storage.BlobReader, error) { func (bad *BadBlobs) OpenWithStorageFormat(ctx context.Context, ref storage.BlobRef, formatVer storage.FormatVersion) (storage.BlobReader, error) {
if bad.err != nil { if err := bad.err.Err(); err != nil {
return nil, bad.err return nil, err
} }
return bad.blobs.OpenWithStorageFormat(ctx, ref, formatVer) return bad.blobs.OpenWithStorageFormat(ctx, ref, formatVer)
} }
// Trash deletes the blob with the namespace and key. // Trash deletes the blob with the namespace and key.
func (bad *BadBlobs) Trash(ctx context.Context, ref storage.BlobRef) error { func (bad *BadBlobs) Trash(ctx context.Context, ref storage.BlobRef) error {
if bad.err != nil { if err := bad.err.Err(); err != nil {
return bad.err return err
} }
return bad.blobs.Trash(ctx, ref) return bad.blobs.Trash(ctx, ref)
} }
// RestoreTrash restores all files in the trash. // RestoreTrash restores all files in the trash.
func (bad *BadBlobs) RestoreTrash(ctx context.Context, namespace []byte) ([][]byte, error) { func (bad *BadBlobs) RestoreTrash(ctx context.Context, namespace []byte) ([][]byte, error) {
if bad.err != nil { if err := bad.err.Err(); err != nil {
return nil, bad.err return nil, err
} }
return bad.blobs.RestoreTrash(ctx, namespace) return bad.blobs.RestoreTrash(ctx, namespace)
} }
// EmptyTrash empties the trash. // EmptyTrash empties the trash.
func (bad *BadBlobs) EmptyTrash(ctx context.Context, namespace []byte, trashedBefore time.Time) (int64, [][]byte, error) { func (bad *BadBlobs) EmptyTrash(ctx context.Context, namespace []byte, trashedBefore time.Time) (int64, [][]byte, error) {
if bad.err != nil { if err := bad.err.Err(); err != nil {
return 0, nil, bad.err return 0, nil, err
} }
return bad.blobs.EmptyTrash(ctx, namespace, trashedBefore) return bad.blobs.EmptyTrash(ctx, namespace, trashedBefore)
} }
// Delete deletes the blob with the namespace and key. // Delete deletes the blob with the namespace and key.
func (bad *BadBlobs) Delete(ctx context.Context, ref storage.BlobRef) error { func (bad *BadBlobs) Delete(ctx context.Context, ref storage.BlobRef) error {
if bad.err != nil { if err := bad.err.Err(); err != nil {
return bad.err return err
} }
return bad.blobs.Delete(ctx, ref) return bad.blobs.Delete(ctx, ref)
} }
// DeleteWithStorageFormat deletes the blob with the namespace, key, and format version. // DeleteWithStorageFormat deletes the blob with the namespace, key, and format version.
func (bad *BadBlobs) DeleteWithStorageFormat(ctx context.Context, ref storage.BlobRef, formatVer storage.FormatVersion) error { func (bad *BadBlobs) DeleteWithStorageFormat(ctx context.Context, ref storage.BlobRef, formatVer storage.FormatVersion) error {
if bad.err != nil { if err := bad.err.Err(); err != nil {
return bad.err return err
} }
return bad.blobs.DeleteWithStorageFormat(ctx, ref, formatVer) return bad.blobs.DeleteWithStorageFormat(ctx, ref, formatVer)
} }
// DeleteNamespace deletes blobs of specific satellite, used after successful GE only. // DeleteNamespace deletes blobs of specific satellite, used after successful GE only.
func (bad *BadBlobs) DeleteNamespace(ctx context.Context, ref []byte) (err error) { func (bad *BadBlobs) DeleteNamespace(ctx context.Context, ref []byte) (err error) {
if bad.err != nil { if err := bad.err.Err(); err != nil {
return bad.err return err
} }
return bad.blobs.DeleteNamespace(ctx, ref) return bad.blobs.DeleteNamespace(ctx, ref)
} }
// Stat looks up disk metadata on the blob file. // Stat looks up disk metadata on the blob file.
func (bad *BadBlobs) Stat(ctx context.Context, ref storage.BlobRef) (storage.BlobInfo, error) { func (bad *BadBlobs) Stat(ctx context.Context, ref storage.BlobRef) (storage.BlobInfo, error) {
if bad.err != nil { if err := bad.err.Err(); err != nil {
return nil, bad.err return nil, err
} }
return bad.blobs.Stat(ctx, ref) return bad.blobs.Stat(ctx, ref)
} }
@ -151,8 +176,8 @@ func (bad *BadBlobs) Stat(ctx context.Context, ref storage.BlobRef) (storage.Blo
// version. This avoids the potential need to check multiple storage formats for the blob // version. This avoids the potential need to check multiple storage formats for the blob
// when the format is already known. // when the format is already known.
func (bad *BadBlobs) StatWithStorageFormat(ctx context.Context, ref storage.BlobRef, formatVer storage.FormatVersion) (storage.BlobInfo, error) { func (bad *BadBlobs) StatWithStorageFormat(ctx context.Context, ref storage.BlobRef, formatVer storage.FormatVersion) (storage.BlobInfo, error) {
if bad.err != nil { if err := bad.err.Err(); err != nil {
return nil, bad.err return nil, err
} }
return bad.blobs.StatWithStorageFormat(ctx, ref, formatVer) return bad.blobs.StatWithStorageFormat(ctx, ref, formatVer)
} }
@ -161,64 +186,64 @@ func (bad *BadBlobs) StatWithStorageFormat(ctx context.Context, ref storage.Blob
// If walkFunc returns a non-nil error, WalkNamespace will stop iterating and return the // If walkFunc returns a non-nil error, WalkNamespace will stop iterating and return the
// error immediately. // error immediately.
func (bad *BadBlobs) WalkNamespace(ctx context.Context, namespace []byte, walkFunc func(storage.BlobInfo) error) error { func (bad *BadBlobs) WalkNamespace(ctx context.Context, namespace []byte, walkFunc func(storage.BlobInfo) error) error {
if bad.err != nil { if err := bad.err.Err(); err != nil {
return bad.err return err
} }
return bad.blobs.WalkNamespace(ctx, namespace, walkFunc) return bad.blobs.WalkNamespace(ctx, namespace, walkFunc)
} }
// ListNamespaces returns all namespaces that might be storing data. // ListNamespaces returns all namespaces that might be storing data.
func (bad *BadBlobs) ListNamespaces(ctx context.Context) ([][]byte, error) { func (bad *BadBlobs) ListNamespaces(ctx context.Context) ([][]byte, error) {
if bad.err != nil { if err := bad.err.Err(); err != nil {
return make([][]byte, 0), bad.err return make([][]byte, 0), err
} }
return bad.blobs.ListNamespaces(ctx) return bad.blobs.ListNamespaces(ctx)
} }
// FreeSpace return how much free space left for writing. // FreeSpace return how much free space left for writing.
func (bad *BadBlobs) FreeSpace() (int64, error) { func (bad *BadBlobs) FreeSpace() (int64, error) {
if bad.err != nil { if err := bad.err.Err(); err != nil {
return 0, bad.err return 0, err
} }
return bad.blobs.FreeSpace() return bad.blobs.FreeSpace()
} }
// CheckWritability tests writability of the storage directory by creating and deleting a file. // CheckWritability tests writability of the storage directory by creating and deleting a file.
func (bad *BadBlobs) CheckWritability() error { func (bad *BadBlobs) CheckWritability() error {
if bad.err != nil { if err := bad.err.Err(); err != nil {
return bad.err return err
} }
return bad.blobs.CheckWritability() return bad.blobs.CheckWritability()
} }
// SpaceUsedForBlobs adds up how much is used in all namespaces. // SpaceUsedForBlobs adds up how much is used in all namespaces.
func (bad *BadBlobs) SpaceUsedForBlobs(ctx context.Context) (int64, error) { func (bad *BadBlobs) SpaceUsedForBlobs(ctx context.Context) (int64, error) {
if bad.err != nil { if err := bad.err.Err(); err != nil {
return 0, bad.err return 0, err
} }
return bad.blobs.SpaceUsedForBlobs(ctx) return bad.blobs.SpaceUsedForBlobs(ctx)
} }
// SpaceUsedForBlobsInNamespace adds up how much is used in the given namespace. // SpaceUsedForBlobsInNamespace adds up how much is used in the given namespace.
func (bad *BadBlobs) SpaceUsedForBlobsInNamespace(ctx context.Context, namespace []byte) (int64, error) { func (bad *BadBlobs) SpaceUsedForBlobsInNamespace(ctx context.Context, namespace []byte) (int64, error) {
if bad.err != nil { if err := bad.err.Err(); err != nil {
return 0, bad.err return 0, err
} }
return bad.blobs.SpaceUsedForBlobsInNamespace(ctx, namespace) return bad.blobs.SpaceUsedForBlobsInNamespace(ctx, namespace)
} }
// SpaceUsedForTrash adds up how much is used in all namespaces. // SpaceUsedForTrash adds up how much is used in all namespaces.
func (bad *BadBlobs) SpaceUsedForTrash(ctx context.Context) (int64, error) { func (bad *BadBlobs) SpaceUsedForTrash(ctx context.Context) (int64, error) {
if bad.err != nil { if err := bad.err.Err(); err != nil {
return 0, bad.err return 0, err
} }
return bad.blobs.SpaceUsedForTrash(ctx) return bad.blobs.SpaceUsedForTrash(ctx)
} }
// CreateVerificationFile creates a file to be used for storage directory verification. // CreateVerificationFile creates a file to be used for storage directory verification.
func (bad *BadBlobs) CreateVerificationFile(id storj.NodeID) error { func (bad *BadBlobs) CreateVerificationFile(id storj.NodeID) error {
if bad.err != nil { if err := bad.err.Err(); err != nil {
return bad.err return err
} }
return bad.blobs.CreateVerificationFile(id) return bad.blobs.CreateVerificationFile(id)
} }
@ -226,13 +251,8 @@ func (bad *BadBlobs) CreateVerificationFile(id storj.NodeID) error {
// VerifyStorageDir verifies that the storage directory is correct by checking for the existence and validity // VerifyStorageDir verifies that the storage directory is correct by checking for the existence and validity
// of the verification file. // of the verification file.
func (bad *BadBlobs) VerifyStorageDir(id storj.NodeID) error { func (bad *BadBlobs) VerifyStorageDir(id storj.NodeID) error {
if bad.err != nil { if err := bad.err.Err(); err != nil {
return bad.err return err
} }
return bad.blobs.VerifyStorageDir(id) return bad.blobs.VerifyStorageDir(id)
} }
// SetError configures the blob store to return a specific error for all operations.
func (bad *BadBlobs) SetError(err error) {
bad.err = err
}

69
private/testplanet/log.go Normal file
View File

@ -0,0 +1,69 @@
// Copyright (C) 2020 Storj Labs, Inc.
// See LICENSE for copying information
package testplanet
import (
"bytes"
"fmt"
"os"
"testing"
"time"
"go.uber.org/zap"
"go.uber.org/zap/zapcore"
"go.uber.org/zap/zaptest"
)
var useAbsTime = os.Getenv("STORJ_TESTPLANET_ABSTIME")
func newLogger(t *testing.T) *zap.Logger {
if useAbsTime != "" {
return zaptest.NewLogger(t)
}
start := time.Now()
cfg := zap.NewDevelopmentEncoderConfig()
cfg.EncodeTime = func(t time.Time, enc zapcore.PrimitiveArrayEncoder) {
var nanos, seconds, minutes int64
nanos = t.Sub(start).Nanoseconds()
seconds, nanos = nanos/1e9, nanos%1e9
minutes, seconds = seconds/60, seconds%60
enc.AppendString(fmt.Sprintf("%02d:%02d.%03d", minutes, seconds, nanos/1e6))
}
enc := zapcore.NewConsoleEncoder(cfg)
writer := newTestingWriter(t)
return zap.New(
zapcore.NewCore(enc, writer, zapcore.DebugLevel),
zap.ErrorOutput(writer.WithMarkFailed(true)),
)
}
type testingWriter struct {
t *testing.T
markFailed bool
}
func newTestingWriter(t *testing.T) testingWriter {
return testingWriter{t: t}
}
func (w testingWriter) WithMarkFailed(v bool) testingWriter {
w.markFailed = v
return w
}
func (w testingWriter) Write(p []byte) (n int, err error) {
n = len(p)
p = bytes.TrimRight(p, "\n")
w.t.Logf("%s", p)
if w.markFailed {
w.t.Fail()
}
return n, nil
}
func (w testingWriter) Sync() error { return nil }

2
private/testplanet/planet_test.go
View File

@ -34,6 +34,7 @@ func TestBasic(t *testing.T) {
for _, sat := range planet.Satellites { for _, sat := range planet.Satellites {
for _, sn := range planet.StorageNodes { for _, sn := range planet.StorageNodes {
func() {
node := sn.Contact.Service.Local() node := sn.Contact.Service.Local()
conn, err := sn.Dialer.DialNodeURL(ctx, sat.NodeURL()) conn, err := sn.Dialer.DialNodeURL(ctx, sat.NodeURL())
@ -46,6 +47,7 @@ func TestBasic(t *testing.T) {
Operator: &node.Operator, Operator: &node.Operator,
}) })
require.NoError(t, err) require.NoError(t, err)
}()
} }
} }
// wait a bit to see whether some failures occur // wait a bit to see whether some failures occur

8
private/testplanet/reconfigure.go
View File

@ -67,10 +67,10 @@ var Combine = func(elements ...func(log *zap.Logger, index int, config *satellit
// ReconfigureRS returns function to change satellite redundancy scheme values. // ReconfigureRS returns function to change satellite redundancy scheme values.
var ReconfigureRS = func(minThreshold, repairThreshold, successThreshold, totalThreshold int) func(log *zap.Logger, index int, config *satellite.Config) { var ReconfigureRS = func(minThreshold, repairThreshold, successThreshold, totalThreshold int) func(log *zap.Logger, index int, config *satellite.Config) {
return func(log *zap.Logger, index int, config *satellite.Config) { return func(log *zap.Logger, index int, config *satellite.Config) {
config.Metainfo.RS.MinThreshold = minThreshold config.Metainfo.RS.Min = minThreshold
config.Metainfo.RS.RepairThreshold = repairThreshold config.Metainfo.RS.Repair = repairThreshold
config.Metainfo.RS.SuccessThreshold = successThreshold config.Metainfo.RS.Success = successThreshold
config.Metainfo.RS.TotalThreshold = totalThreshold config.Metainfo.RS.Total = totalThreshold
} }
} }

4
private/testplanet/run.go
View File

@ -9,8 +9,6 @@ import (
"strings" "strings"
"testing" "testing"
"go.uber.org/zap/zaptest"
"storj.io/common/testcontext" "storj.io/common/testcontext"
"storj.io/storj/private/dbutil/pgtest" "storj.io/storj/private/dbutil/pgtest"
"storj.io/storj/satellite/satellitedb/satellitedbtest" "storj.io/storj/satellite/satellitedb/satellitedbtest"
@ -54,7 +52,7 @@ func Run(t *testing.T, config Config, test func(t *testing.T, ctx *testcontext.C
} }
pprof.Do(ctx, pprof.Labels("planet", planetConfig.Name), func(namedctx context.Context) { pprof.Do(ctx, pprof.Labels("planet", planetConfig.Name), func(namedctx context.Context) {
planet, err := NewCustom(namedctx, zaptest.NewLogger(t), planetConfig, satelliteDB) planet, err := NewCustom(namedctx, newLogger(t), planetConfig, satelliteDB)
if err != nil { if err != nil {
t.Fatalf("%+v", err) t.Fatalf("%+v", err)
} }

13
private/testplanet/satellite.go
View File

@ -468,16 +468,11 @@ func (planet *Planet) newSatellite(ctx context.Context, prefix string, index int
MaxCommitInterval: 1 * time.Hour, MaxCommitInterval: 1 * time.Hour,
Overlay: true, Overlay: true,
RS: metainfo.RSConfig{ RS: metainfo.RSConfig{
MaxBufferMem: memory.Size(256),
ErasureShareSize: memory.Size(256), ErasureShareSize: memory.Size(256),
MinThreshold: atLeastOne(planet.config.StorageNodeCount * 1 / 5), Min: atLeastOne(planet.config.StorageNodeCount * 1 / 5),
RepairThreshold: atLeastOne(planet.config.StorageNodeCount * 2 / 5), Repair: atLeastOne(planet.config.StorageNodeCount * 2 / 5),
SuccessThreshold: atLeastOne(planet.config.StorageNodeCount * 3 / 5), Success: atLeastOne(planet.config.StorageNodeCount * 3 / 5),
TotalThreshold: atLeastOne(planet.config.StorageNodeCount * 4 / 5), Total: atLeastOne(planet.config.StorageNodeCount * 4 / 5),
MinTotalThreshold: (planet.config.StorageNodeCount * 4 / 5),
MaxTotalThreshold: (planet.config.StorageNodeCount * 4 / 5),
Validate: false,
}, },
Loop: metainfo.LoopConfig{ Loop: metainfo.LoopConfig{
CoalesceDuration: 1 * time.Second, CoalesceDuration: 1 * time.Second,

2
satellite/accounting/live/doc.go
View File

@ -4,6 +4,6 @@
/* /*
Package live provides live accounting functionality. That is, it keeps track Package live provides live accounting functionality. That is, it keeps track
of deltas in the amount of storage used by each project relative to the last of deltas in the amount of storage used by each project relative to the last
tally operation (see pkg/accounting/tally). tally operation (see satellite/accounting/tally).
*/ */
package live package live

1
satellite/accounting/rollup/rollup_test.go
View File

@ -217,7 +217,6 @@ func dqNodes(ctx *testcontext.Context, planet *testplanet.Planet) (map[storj.Nod
} }
updateRequests = append(updateRequests, &overlay.UpdateRequest{ updateRequests = append(updateRequests, &overlay.UpdateRequest{
NodeID: n.ID(), NodeID: n.ID(),
IsUp: true,
AuditOutcome: overlay.AuditFailure, AuditOutcome: overlay.AuditFailure,
}) })
} }

21
satellite/accounting/tally/tally.go
View File

@ -84,6 +84,24 @@ func (service *Service) SetNow(now func() time.Time) {
} }
// Tally calculates data-at-rest usage once. // Tally calculates data-at-rest usage once.
//
// How live accounting is calculated:
//
// At the beginning of the tally iteration, we get a map containing the current
// project totals from the cache- initialLiveTotals (our current estimation of
// the project totals). At the end of the tally iteration, we have the totals
// from what we saw during the metainfo loop.
//
// However, data which was uploaded during the loop may or may not have been
// seen in the metainfo loop. For this reason, we also read the live accounting
// totals again at the end of the tally iteration- latestLiveTotals.
//
// The difference between latest and initial indicates how much data was
// uploaded during the metainfo loop and is assigned to delta. However, again,
// we aren't certain how much of the delta is accounted for in the metainfo
// totals. For the reason we make an assumption that 50% of the data is
// accounted for. So to calculate the new live accounting totals, we sum the
// metainfo totals and 50% of the deltas.
func (service *Service) Tally(ctx context.Context) (err error) { func (service *Service) Tally(ctx context.Context) (err error) {
defer mon.Task()(&ctx)(&err) defer mon.Task()(&ctx)(&err)
@ -151,6 +169,9 @@ func (service *Service) Tally(ctx context.Context) (err error) {
if delta < 0 { if delta < 0 {
delta = 0 delta = 0
} }
// read the method documentation why the increase passed to this method
// is calculated in this way
err = service.liveAccounting.AddProjectStorageUsage(ctx, projectID, -latestLiveTotals[projectID]+tallyTotal+(delta/2)) err = service.liveAccounting.AddProjectStorageUsage(ctx, projectID, -latestLiveTotals[projectID]+tallyTotal+(delta/2))
if err != nil { if err != nil {
return Error.Wrap(err) return Error.Wrap(err)

20
satellite/accounting/tally/tally_test.go
View File

@ -143,7 +143,7 @@ func TestCalculateNodeAtRestData(t *testing.T) {
require.NoError(t, err) require.NoError(t, err)
// Confirm the correct number of shares were stored // Confirm the correct number of shares were stored
rs := satelliteRS(planet.Satellites[0]) rs := satelliteRS(t, planet.Satellites[0])
if !correctRedundencyScheme(len(obs.Node), rs) { if !correctRedundencyScheme(len(obs.Node), rs) {
t.Fatalf("expected between: %d and %d, actual: %d", rs.RepairShares, rs.TotalShares, len(obs.Node)) t.Fatalf("expected between: %d and %d, actual: %d", rs.RepairShares, rs.TotalShares, len(obs.Node))
} }
@ -175,7 +175,7 @@ func TestCalculateBucketAtRestData(t *testing.T) {
SatelliteCount: 1, StorageNodeCount: 6, UplinkCount: 1, SatelliteCount: 1, StorageNodeCount: 6, UplinkCount: 1,
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) { }, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
satellitePeer := planet.Satellites[0] satellitePeer := planet.Satellites[0]
redundancyScheme := satelliteRS(satellitePeer) redundancyScheme := satelliteRS(t, satellitePeer)
expectedBucketTallies := make(map[metabase.BucketLocation]*accounting.BucketTally) expectedBucketTallies := make(map[metabase.BucketLocation]*accounting.BucketTally)
for _, tt := range testCases { for _, tt := range testCases {
tt := tt // avoid scopelint error tt := tt // avoid scopelint error
@ -221,7 +221,7 @@ func TestTallyIgnoresExpiredPointers(t *testing.T) {
SatelliteCount: 1, StorageNodeCount: 6, UplinkCount: 1, SatelliteCount: 1, StorageNodeCount: 6, UplinkCount: 1,
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) { }, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
satellitePeer := planet.Satellites[0] satellitePeer := planet.Satellites[0]
redundancyScheme := satelliteRS(satellitePeer) redundancyScheme := satelliteRS(t, satellitePeer)
projectID, err := uuid.FromString("9656af6e-2d9c-42fa-91f2-bfd516a722d7") projectID, err := uuid.FromString("9656af6e-2d9c-42fa-91f2-bfd516a722d7")
require.NoError(t, err) require.NoError(t, err)
@ -423,12 +423,14 @@ func correctRedundencyScheme(shareCount int, uplinkRS storj.RedundancyScheme) bo
return int(uplinkRS.RepairShares) <= shareCount && shareCount <= int(uplinkRS.TotalShares) return int(uplinkRS.RepairShares) <= shareCount && shareCount <= int(uplinkRS.TotalShares)
} }
func satelliteRS(satellite *testplanet.Satellite) storj.RedundancyScheme { func satelliteRS(t *testing.T, satellite *testplanet.Satellite) storj.RedundancyScheme {
rs := satellite.Config.Metainfo.RS
return storj.RedundancyScheme{ return storj.RedundancyScheme{
RequiredShares: int16(satellite.Config.Metainfo.RS.MinThreshold), RequiredShares: int16(rs.Min),
RepairShares: int16(satellite.Config.Metainfo.RS.RepairThreshold), RepairShares: int16(rs.Repair),
OptimalShares: int16(satellite.Config.Metainfo.RS.SuccessThreshold), OptimalShares: int16(rs.Success),
TotalShares: int16(satellite.Config.Metainfo.RS.TotalThreshold), TotalShares: int16(rs.Total),
ShareSize: satellite.Config.Metainfo.RS.ErasureShareSize.Int32(), ShareSize: rs.ErasureShareSize.Int32(),
} }
} }

1
satellite/audit/disqualification_test.go
View File

@ -214,7 +214,6 @@ func TestDisqualifiedNodeRemainsDisqualified(t *testing.T) {
_, err = satellitePeer.Overlay.Service.BatchUpdateStats(ctx, []*overlay.UpdateRequest{{ _, err = satellitePeer.Overlay.Service.BatchUpdateStats(ctx, []*overlay.UpdateRequest{{
NodeID: disqualifiedNode.ID(), NodeID: disqualifiedNode.ID(),
IsUp: true,
AuditOutcome: overlay.AuditSuccess, AuditOutcome: overlay.AuditSuccess,
AuditLambda: 0, // forget about history AuditLambda: 0, // forget about history
AuditWeight: 1, AuditWeight: 1,

5
satellite/audit/reporter.go
View File

@ -128,7 +128,6 @@ func (reporter *Reporter) recordAuditFailStatus(ctx context.Context, failedAudit
for i, nodeID := range failedAuditNodeIDs { for i, nodeID := range failedAuditNodeIDs {
updateRequests[i] = &overlay.UpdateRequest{ updateRequests[i] = &overlay.UpdateRequest{
NodeID: nodeID, NodeID: nodeID,
IsUp: true,
AuditOutcome: overlay.AuditFailure, AuditOutcome: overlay.AuditFailure,
} }
} }
@ -148,7 +147,6 @@ func (reporter *Reporter) recordAuditUnknownStatus(ctx context.Context, unknownA
for i, nodeID := range unknownAuditNodeIDs { for i, nodeID := range unknownAuditNodeIDs {
updateRequests[i] = &overlay.UpdateRequest{ updateRequests[i] = &overlay.UpdateRequest{
NodeID: nodeID, NodeID: nodeID,
IsUp: true,
AuditOutcome: overlay.AuditUnknown, AuditOutcome: overlay.AuditUnknown,
} }
} }
@ -169,7 +167,6 @@ func (reporter *Reporter) recordOfflineStatus(ctx context.Context, offlineNodeID
for i, nodeID := range offlineNodeIDs { for i, nodeID := range offlineNodeIDs {
updateRequests[i] = &overlay.UpdateRequest{ updateRequests[i] = &overlay.UpdateRequest{
NodeID: nodeID, NodeID: nodeID,
IsUp: false,
AuditOutcome: overlay.AuditOffline, AuditOutcome: overlay.AuditOffline,
} }
} }
@ -191,7 +188,6 @@ func (reporter *Reporter) recordAuditSuccessStatus(ctx context.Context, successN
for i, nodeID := range successNodeIDs { for i, nodeID := range successNodeIDs {
updateRequests[i] = &overlay.UpdateRequest{ updateRequests[i] = &overlay.UpdateRequest{
NodeID: nodeID, NodeID: nodeID,
IsUp: true,
AuditOutcome: overlay.AuditSuccess, AuditOutcome: overlay.AuditSuccess,
} }
} }
@ -221,7 +217,6 @@ func (reporter *Reporter) recordPendingAudits(ctx context.Context, pendingAudits
// record failure -- max reverify count reached // record failure -- max reverify count reached
updateRequests = append(updateRequests, &overlay.UpdateRequest{ updateRequests = append(updateRequests, &overlay.UpdateRequest{
NodeID: pendingAudit.NodeID, NodeID: pendingAudit.NodeID,
IsUp: true,
AuditOutcome: overlay.AuditFailure, AuditOutcome: overlay.AuditFailure,
}) })
} }

10
satellite/audit/reverify_test.go
View File

@ -1074,16 +1074,14 @@ func TestReverifySlowDownload(t *testing.T) {
StorageNodeDB: func(index int, db storagenode.DB, log *zap.Logger) (storagenode.DB, error) { StorageNodeDB: func(index int, db storagenode.DB, log *zap.Logger) (storagenode.DB, error) {
return testblobs.NewSlowDB(log.Named("slowdb"), db), nil return testblobs.NewSlowDB(log.Named("slowdb"), db), nil
}, },
Satellite: func(log *zap.Logger, index int, config *satellite.Config) { Satellite: testplanet.Combine(
func(log *zap.Logger, index int, config *satellite.Config) {
// These config values are chosen to force the slow node to time out without timing out on the three normal nodes // These config values are chosen to force the slow node to time out without timing out on the three normal nodes
config.Audit.MinBytesPerSecond = 100 * memory.KiB config.Audit.MinBytesPerSecond = 100 * memory.KiB
config.Audit.MinDownloadTimeout = 1 * time.Second config.Audit.MinDownloadTimeout = 1 * time.Second
config.Metainfo.RS.MinThreshold = 2
config.Metainfo.RS.RepairThreshold = 2
config.Metainfo.RS.SuccessThreshold = 4
config.Metainfo.RS.TotalThreshold = 4
}, },
testplanet.ReconfigureRS(2, 2, 4, 4),
),
}, },
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) { }, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
satellite := planet.Satellites[0] satellite := planet.Satellites[0]

10
satellite/audit/verifier_test.go
View File

@ -776,16 +776,14 @@ func TestVerifierSlowDownload(t *testing.T) {
StorageNodeDB: func(index int, db storagenode.DB, log *zap.Logger) (storagenode.DB, error) { StorageNodeDB: func(index int, db storagenode.DB, log *zap.Logger) (storagenode.DB, error) {
return testblobs.NewSlowDB(log.Named("slowdb"), db), nil return testblobs.NewSlowDB(log.Named("slowdb"), db), nil
}, },
Satellite: func(log *zap.Logger, index int, config *satellite.Config) { Satellite: testplanet.Combine(
func(log *zap.Logger, index int, config *satellite.Config) {
// These config values are chosen to force the slow node to time out without timing out on the three normal nodes // These config values are chosen to force the slow node to time out without timing out on the three normal nodes
config.Audit.MinBytesPerSecond = 100 * memory.KiB config.Audit.MinBytesPerSecond = 100 * memory.KiB
config.Audit.MinDownloadTimeout = 950 * time.Millisecond config.Audit.MinDownloadTimeout = 950 * time.Millisecond
config.Metainfo.RS.MinThreshold = 2
config.Metainfo.RS.RepairThreshold = 2
config.Metainfo.RS.SuccessThreshold = 4
config.Metainfo.RS.TotalThreshold = 4
}, },
testplanet.ReconfigureRS(2, 2, 4, 4),
),
}, },
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) { }, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
satellite := planet.Satellites[0] satellite := planet.Satellites[0]

23
satellite/console/consoleweb/consoleapi/auth.go
View File

@ -253,6 +253,29 @@ func (a *Auth) DeleteAccount(w http.ResponseWriter, r *http.Request) {
a.serveJSONError(w, errNotImplemented) a.serveJSONError(w, errNotImplemented)
} }
// ChangeEmail auth user, changes users email for a new one.
func (a *Auth) ChangeEmail(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
var err error
defer mon.Task()(&ctx)(&err)
var emailChange struct {
NewEmail string `json:"newEmail"`
}
err = json.NewDecoder(r.Body).Decode(&emailChange)
if err != nil {
a.serveJSONError(w, err)
return
}
err = a.service.ChangeEmail(ctx, emailChange.NewEmail)
if err != nil {
a.serveJSONError(w, err)
return
}
}
// ChangePassword auth user, changes users password for a new one. // ChangePassword auth user, changes users password for a new one.
func (a *Auth) ChangePassword(w http.ResponseWriter, r *http.Request) { func (a *Auth) ChangePassword(w http.ResponseWriter, r *http.Request) {
ctx := r.Context() ctx := r.Context()

3
satellite/console/consoleweb/consoleapi/auth_test.go
View File

@ -39,7 +39,6 @@ func TestAuth_Register(t *testing.T) {
}, },
}, },
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) { }, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
for i, test := range []struct { for i, test := range []struct {
Partner string Partner string
ValidPartner bool ValidPartner bool
@ -50,6 +49,7 @@ func TestAuth_Register(t *testing.T) {
{Partner: "Raiden nEtwork", ValidPartner: true}, {Partner: "Raiden nEtwork", ValidPartner: true},
{Partner: "invalid-name", ValidPartner: false}, {Partner: "invalid-name", ValidPartner: false},
} { } {
func() {
registerData := struct { registerData := struct {
FullName string `json:"fullName"` FullName string `json:"fullName"`
ShortName string `json:"shortName"` ShortName string `json:"shortName"`
@ -96,6 +96,7 @@ func TestAuth_Register(t *testing.T) {
} else { } else {
require.Equal(t, uuid.UUID{}, user.PartnerID) require.Equal(t, uuid.UUID{}, user.PartnerID)
} }
}()
} }
}) })
} }

1
satellite/console/consoleweb/server.go
View File

@ -173,6 +173,7 @@ func NewServer(logger *zap.Logger, config Config, service *console.Service, mail
authRouter := router.PathPrefix("/api/v0/auth").Subrouter() authRouter := router.PathPrefix("/api/v0/auth").Subrouter()
authRouter.Handle("/account", server.withAuth(http.HandlerFunc(authController.GetAccount))).Methods(http.MethodGet) authRouter.Handle("/account", server.withAuth(http.HandlerFunc(authController.GetAccount))).Methods(http.MethodGet)
authRouter.Handle("/account", server.withAuth(http.HandlerFunc(authController.UpdateAccount))).Methods(http.MethodPatch) authRouter.Handle("/account", server.withAuth(http.HandlerFunc(authController.UpdateAccount))).Methods(http.MethodPatch)
authRouter.Handle("/account/change-email", server.withAuth(http.HandlerFunc(authController.ChangeEmail))).Methods(http.MethodPost)
authRouter.Handle("/account/change-password", server.withAuth(http.HandlerFunc(authController.ChangePassword))).Methods(http.MethodPost) authRouter.Handle("/account/change-password", server.withAuth(http.HandlerFunc(authController.ChangePassword))).Methods(http.MethodPost)
authRouter.Handle("/account/delete", server.withAuth(http.HandlerFunc(authController.DeleteAccount))).Methods(http.MethodPost) authRouter.Handle("/account/delete", server.withAuth(http.HandlerFunc(authController.DeleteAccount))).Methods(http.MethodPost)
authRouter.HandleFunc("/logout", authController.Logout).Methods(http.MethodPost) authRouter.HandleFunc("/logout", authController.Logout).Methods(http.MethodPost)

27
satellite/console/service.go
View File

@ -9,6 +9,7 @@ import (
"database/sql" "database/sql"
"errors" "errors"
"fmt" "fmt"
"net/mail"
"sort" "sort"
"time" "time"
@ -867,6 +868,32 @@ func (s *Service) UpdateAccount(ctx context.Context, fullName string, shortName
return nil return nil
} }
// ChangeEmail updates email for a given user.
func (s *Service) ChangeEmail(ctx context.Context, newEmail string) (err error) {
defer mon.Task()(&ctx)(&err)
auth, err := s.getAuthAndAuditLog(ctx, "change email")
if err != nil {
return Error.Wrap(err)
}
if _, err := mail.ParseAddress(newEmail); err != nil {
return ErrValidation.Wrap(err)
}
_, err = s.store.Users().GetByEmail(ctx, newEmail)
if err == nil {
return ErrEmailUsed.New(emailUsedErrMsg)
}
auth.User.Email = newEmail
err = s.store.Users().Update(ctx, &auth.User)
if err != nil {
return Error.Wrap(err)
}
return nil
}
// ChangePassword updates password for a given user. // ChangePassword updates password for a given user.
func (s *Service) ChangePassword(ctx context.Context, pass, newPass string) (err error) { func (s *Service) ChangePassword(ctx context.Context, pass, newPass string) (err error) {
defer mon.Task()(&ctx)(&err) defer mon.Task()(&ctx)(&err)

14
satellite/console/service_test.go
View File

@ -130,5 +130,19 @@ func TestService(t *testing.T) {
require.Error(t, err) require.Error(t, err)
require.Equal(t, "service error: project usage error: some buckets still exist", err.Error()) require.Equal(t, "service error: project usage error: some buckets still exist", err.Error())
}) })
t.Run("TestChangeEmail", func(t *testing.T) {
const newEmail = "newEmail@example.com"
err = service.ChangeEmail(authCtx2, newEmail)
require.NoError(t, err)
userWithUpdatedEmail, err := service.GetUserByEmail(authCtx2, newEmail)
require.NoError(t, err)
require.Equal(t, newEmail, userWithUpdatedEmail.Email)
err = service.ChangeEmail(authCtx2, newEmail)
require.Error(t, err)
})
}) })
} }

35
satellite/console/wasm/README.md Normal file
View File

@ -0,0 +1,35 @@
# Using WebAssembly in Storj
In order to use the uplink library from the browser, we can compile the uplink library to WebAssembly (wasm).
### Setup
To generate wasm code that can create access grants in the web browser, run the following from the storj/wasm directory:
```
$ GOOS=js GOARCH=wasm go build -o access.wasm access.go
```
The `access.wasm` code can then be loaded into the browser in a script tag in an html page. Also needed is a JavaScript support file which ships with golang.
To copy the JavaScript support file, run:
```
$ cp "$(go env GOROOT)/misc/wasm/wasm_exec.js" .
```
Ref: [Golang WebAssembly docs](https://github.com/golang/go/wiki/WebAssembly)
The HTML file should include the following:
```
<script type="text/javascript" src="/path/to/wasm_exec.js"></script>
<script>
const go = new Go();
WebAssembly.instantiateStreaming(
fetch("/path/to/access.wasm"), go.importObject).then(
(result) => {
go.run(result.instance);
});
</script>
```
Additionally, the HTTP `Content-Security-Policy (CSP) script-src` directive will need to be modified to allow wasm code to be executed.
See: [WebAssembly Content Security Policy docs](https://github.com/WebAssembly/content-security-policy/blob/master/proposals/CSP.md)

64
satellite/console/wasm/access.go Normal file
View File

@ -0,0 +1,64 @@
// +build js,wasm
// Copyright (C) 2020 Storj Labs, Inc.
// See LICENSE for copying information.
package main
import (
"fmt"
"syscall/js"
"storj.io/common/encryption"
"storj.io/common/macaroon"
"storj.io/common/storj"
"storj.io/uplink/private/access2"
)
func main() {
js.Global().Set("generateAccessGrant", generateAccessGrant())
<-make(chan bool)
}
func generateAccessGrant() js.Func {
return js.FuncOf(func(this js.Value, args []js.Value) interface{} {
if len(args) < 4 {
return fmt.Sprintf("Error not enough arguments. Need 4, but only %d supplied. The order of arguments are: satellite Node URL, API key, encryption passphrase, and project salt.", len(args))
}
satelliteNodeURL := args[0].String()
apiKey := args[1].String()
encryptionPassphrase := args[2].String()
projectSalt := args[3].String()
return genAccessGrant(satelliteNodeURL,
apiKey,
encryptionPassphrase,
projectSalt,
)
})
}
func genAccessGrant(satelliteNodeURL, apiKey, encryptionPassphrase, projectSalt string) string {
parsedAPIKey, err := macaroon.ParseAPIKey(apiKey)
if err != nil {
return err.Error()
}
const concurrency = 8
key, err := encryption.DeriveRootKey([]byte(encryptionPassphrase), []byte(projectSalt), "", concurrency)
if err != nil {
return err.Error()
}
encAccess := access2.NewEncryptionAccessWithDefaultKey(key)
encAccess.SetDefaultPathCipher(storj.EncAESGCM)
a := &access2.Access{
SatelliteAddress: satelliteNodeURL,
APIKey: parsedAPIKey,
EncAccess: encAccess,
}
accessString, err := a.Serialize()
if err != nil {
return err.Error()
}
return accessString
}

20
satellite/gracefulexit/chore_test.go
View File

@ -32,14 +32,12 @@ func TestChore(t *testing.T) {
StorageNodeCount: 8, StorageNodeCount: 8,
UplinkCount: 1, UplinkCount: 1,
Reconfigure: testplanet.Reconfigure{ Reconfigure: testplanet.Reconfigure{
Satellite: func(log *zap.Logger, index int, config *satellite.Config) { Satellite: testplanet.Combine(
func(log *zap.Logger, index int, config *satellite.Config) {
config.GracefulExit.MaxInactiveTimeFrame = maximumInactiveTimeFrame config.GracefulExit.MaxInactiveTimeFrame = maximumInactiveTimeFrame
config.Metainfo.RS.MinThreshold = 4
config.Metainfo.RS.RepairThreshold = 6
config.Metainfo.RS.SuccessThreshold = 8
config.Metainfo.RS.TotalThreshold = 8
}, },
testplanet.ReconfigureRS(4, 6, 8, 8),
),
}, },
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) { }, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
uplinkPeer := planet.Uplinks[0] uplinkPeer := planet.Uplinks[0]
@ -136,14 +134,12 @@ func TestDurabilityRatio(t *testing.T) {
StorageNodeCount: 4, StorageNodeCount: 4,
UplinkCount: 1, UplinkCount: 1,
Reconfigure: testplanet.Reconfigure{ Reconfigure: testplanet.Reconfigure{
Satellite: func(log *zap.Logger, index int, config *satellite.Config) { Satellite: testplanet.Combine(
func(log *zap.Logger, index int, config *satellite.Config) {
config.GracefulExit.MaxInactiveTimeFrame = maximumInactiveTimeFrame config.GracefulExit.MaxInactiveTimeFrame = maximumInactiveTimeFrame
config.Metainfo.RS.MinThreshold = 2
config.Metainfo.RS.RepairThreshold = 3
config.Metainfo.RS.SuccessThreshold = successThreshold
config.Metainfo.RS.TotalThreshold = 4
}, },
testplanet.ReconfigureRS(2, 3, successThreshold, 4),
),
}, },
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) { }, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
uplinkPeer := planet.Uplinks[0] uplinkPeer := planet.Uplinks[0]

30
satellite/gracefulexit/endpoint_test.go
View File

@ -255,16 +255,14 @@ func TestRecvTimeout(t *testing.T) {
StorageNodeDB: func(index int, db storagenode.DB, log *zap.Logger) (storagenode.DB, error) { StorageNodeDB: func(index int, db storagenode.DB, log *zap.Logger) (storagenode.DB, error) {
return testblobs.NewSlowDB(log.Named("slowdb"), db), nil return testblobs.NewSlowDB(log.Named("slowdb"), db), nil
}, },
Satellite: func(logger *zap.Logger, index int, config *satellite.Config) { Satellite: testplanet.Combine(
func(log *zap.Logger, index int, config *satellite.Config) {
// This config value will create a very short timeframe allowed for receiving // This config value will create a very short timeframe allowed for receiving
// data from storage nodes. This will cause context to cancel with timeout. // data from storage nodes. This will cause context to cancel with timeout.
config.GracefulExit.RecvTimeout = 10 * time.Millisecond config.GracefulExit.RecvTimeout = 10 * time.Millisecond
config.Metainfo.RS.MinThreshold = 2
config.Metainfo.RS.RepairThreshold = 3
config.Metainfo.RS.SuccessThreshold = successThreshold
config.Metainfo.RS.TotalThreshold = successThreshold
}, },
testplanet.ReconfigureRS(2, 3, successThreshold, successThreshold),
),
StorageNode: func(index int, config *storagenode.Config) { StorageNode: func(index int, config *storagenode.Config) {
config.GracefulExit = gracefulexit.Config{ config.GracefulExit = gracefulexit.Config{
ChoreInterval: 2 * time.Minute, ChoreInterval: 2 * time.Minute,
@ -1240,17 +1238,15 @@ func TestFailureStorageNodeIgnoresTransferMessages(t *testing.T) {
StorageNodeCount: 5, StorageNodeCount: 5,
UplinkCount: 1, UplinkCount: 1,
Reconfigure: testplanet.Reconfigure{ Reconfigure: testplanet.Reconfigure{
Satellite: func(logger *zap.Logger, index int, config *satellite.Config) { Satellite: testplanet.Combine(
func(log *zap.Logger, index int, config *satellite.Config) {
// We don't care whether a node gracefully exits or not in this test, // We don't care whether a node gracefully exits or not in this test,
// so we set the max failures percentage extra high. // so we set the max failures percentage extra high.
config.GracefulExit.OverallMaxFailuresPercentage = 101 config.GracefulExit.OverallMaxFailuresPercentage = 101
config.GracefulExit.MaxOrderLimitSendCount = maxOrderLimitSendCount config.GracefulExit.MaxOrderLimitSendCount = maxOrderLimitSendCount
config.Metainfo.RS.MinThreshold = 2
config.Metainfo.RS.RepairThreshold = 3
config.Metainfo.RS.SuccessThreshold = 4
config.Metainfo.RS.TotalThreshold = 4
}, },
testplanet.ReconfigureRS(2, 3, 4, 4),
),
}, },
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) { }, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
uplinkPeer := planet.Uplinks[0] uplinkPeer := planet.Uplinks[0]
@ -1370,15 +1366,13 @@ func TestIneligibleNodeAge(t *testing.T) {
StorageNodeCount: 5, StorageNodeCount: 5,
UplinkCount: 1, UplinkCount: 1,
Reconfigure: testplanet.Reconfigure{ Reconfigure: testplanet.Reconfigure{
Satellite: func(logger *zap.Logger, index int, config *satellite.Config) { Satellite: testplanet.Combine(
func(log *zap.Logger, index int, config *satellite.Config) {
// Set the required node age to 1 month. // Set the required node age to 1 month.
config.GracefulExit.NodeMinAgeInMonths = 1 config.GracefulExit.NodeMinAgeInMonths = 1
config.Metainfo.RS.MinThreshold = 2
config.Metainfo.RS.RepairThreshold = 3
config.Metainfo.RS.SuccessThreshold = 4
config.Metainfo.RS.TotalThreshold = 4
}, },
testplanet.ReconfigureRS(2, 3, 4, 4),
),
}, },
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) { }, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
uplinkPeer := planet.Uplinks[0] uplinkPeer := planet.Uplinks[0]

78
satellite/metainfo/config.go
View File

@ -5,7 +5,10 @@ package metainfo
import ( import (
"context" "context"
"fmt"
"io" "io"
"strconv"
"strings"
"time" "time"
"go.uber.org/zap" "go.uber.org/zap"
@ -26,18 +29,73 @@ const (
// RSConfig is a configuration struct that keeps details about default // RSConfig is a configuration struct that keeps details about default
// redundancy strategy information. // redundancy strategy information.
//
// Can be used as a flag.
type RSConfig struct { type RSConfig struct {
MaxBufferMem memory.Size `help:"maximum buffer memory to be allocated for read buffers" default:"4.00MiB"` ErasureShareSize memory.Size
ErasureShareSize memory.Size `help:"the size of each new erasure share in bytes" default:"256B"` Min int
MinThreshold int `help:"the minimum pieces required to recover a segment. k." releaseDefault:"29" devDefault:"4"` Repair int
RepairThreshold int `help:"the minimum safe pieces before a repair is triggered. m." releaseDefault:"35" devDefault:"6"` Success int
SuccessThreshold int `help:"the desired total pieces for a segment. o." releaseDefault:"80" devDefault:"8"` Total int
TotalThreshold int `help:"the largest amount of pieces to encode to. n." releaseDefault:"110" devDefault:"10"` }
// TODO left for validation until we will remove CreateSegmentOld // Type implements pflag.Value.
MinTotalThreshold int `help:"the largest amount of pieces to encode to. n (lower bound for validation)." releaseDefault:"95" devDefault:"10"` func (RSConfig) Type() string { return "metainfo.RSConfig" }
MaxTotalThreshold int `help:"the largest amount of pieces to encode to. n (upper bound for validation)." releaseDefault:"130" devDefault:"10"`
Validate bool `help:"validate redundancy scheme configuration" default:"true"` // String is required for pflag.Value.
func (rs *RSConfig) String() string {
return fmt.Sprintf("%d/%d/%d/%d-%s",
rs.Min,
rs.Repair,
rs.Success,
rs.Total,
rs.ErasureShareSize.String())
}
// Set sets the value from a string in the format k/m/o/n-size (min/repair/optimal/total-erasuresharesize).
func (rs *RSConfig) Set(s string) error {
// Split on dash. Expect two items. First item is RS numbers. Second item is memory.Size.
info := strings.Split(s, "-")
if len(info) != 2 {
return Error.New("Invalid default RS config (expect format k/m/o/n-ShareSize, got %s)", s)
}
rsNumbersString := info[0]
shareSizeString := info[1]
// Attempt to parse "-size" part of config.
shareSizeInt, err := memory.ParseString(shareSizeString)
if err != nil {
return Error.New("Invalid share size in RS config: '%s', %w", shareSizeString, err)
}
shareSize := memory.Size(shareSizeInt)
// Split on forward slash. Expect exactly four positive non-decreasing integers.
rsNumbers := strings.Split(rsNumbersString, "/")
if len(rsNumbers) != 4 {
return Error.New("Invalid default RS numbers (wrong size, expect 4): %s", rsNumbersString)
}
minValue := 1
values := []int{}
for _, nextValueString := range rsNumbers {
nextValue, err := strconv.Atoi(nextValueString)
if err != nil {
return Error.New("Invalid default RS numbers (should all be valid integers): %s, %w", rsNumbersString, err)
}
if nextValue < minValue {
return Error.New("Invalid default RS numbers (should be non-decreasing): %s", rsNumbersString)
}
values = append(values, nextValue)
minValue = nextValue
}
rs.ErasureShareSize = shareSize
rs.Min = values[0]
rs.Repair = values[1]
rs.Success = values[2]
rs.Total = values[3]
return nil
} }
// RateLimiterConfig is a configuration struct for endpoint rate limiting. // RateLimiterConfig is a configuration struct for endpoint rate limiting.

98
satellite/metainfo/config_test.go Normal file
View File

@ -0,0 +1,98 @@
// Copyright (C) 2020 Storj Labs, Inc.
// See LICENSE for copying information.
package metainfo_test
import (
"testing"
"github.com/stretchr/testify/require"
"storj.io/common/memory"
"storj.io/storj/satellite/metainfo"
)
func TestRSConfigValidation(t *testing.T) {
tests := []struct {
description string
configString string
expectedConfig metainfo.RSConfig
expectError bool
}{
{
description: "valid rs config",
configString: "4/8/10/20-256B",
expectedConfig: metainfo.RSConfig{
ErasureShareSize: 256 * memory.B, Min: 4, Repair: 8, Success: 10, Total: 20,
},
expectError: false,
},
{
description: "invalid rs config - numbers decrease",
configString: "4/8/5/20-256B",
expectError: true,
},
{
description: "invalid rs config - starts at 0",
configString: "0/2/4/6-256B",
expectError: true,
},
{
description: "invalid rs config - strings",
configString: "4/a/b/20-256B",
expectError: true,
},
{
description: "invalid rs config - floating-point numbers",
configString: "4/5.2/7/20-256B",
expectError: true,
},
{
description: "invalid rs config - not enough items",
configString: "4/5/20-256B",
expectError: true,
},
{
description: "invalid rs config - too many items",
configString: "4/5/20/30/50-256B",
expectError: true,
},
{
description: "invalid rs config - empty numbers",
configString: "-256B",
expectError: true,
},
{
description: "invalid rs config - empty size",
configString: "1/2/3/4-",
expectError: true,
},
{
description: "invalid rs config - empty",
configString: "",
expectError: true,
},
{
description: "invalid valid rs config - invalid share size",
configString: "4/8/10/20-256A",
expectError: true,
},
}
for _, tt := range tests {
t.Log(tt.description)
rsConfig := metainfo.RSConfig{}
err := rsConfig.Set(tt.configString)
if tt.expectError {
require.Error(t, err)
} else {
require.NoError(t, err)
require.EqualValues(t, tt.expectedConfig.ErasureShareSize, rsConfig.ErasureShareSize)
require.EqualValues(t, tt.expectedConfig.Min, rsConfig.Min)
require.EqualValues(t, tt.expectedConfig.Repair, rsConfig.Repair)
require.EqualValues(t, tt.expectedConfig.Success, rsConfig.Success)
require.EqualValues(t, tt.expectedConfig.Total, rsConfig.Total)
}
}
}

31
satellite/metainfo/metainfo.go
View File

@ -79,6 +79,7 @@ type Endpoint struct {
limiterCache *lrucache.ExpiringLRU limiterCache *lrucache.ExpiringLRU
encInlineSegmentSize int64 // max inline segment size + encryption overhead encInlineSegmentSize int64 // max inline segment size + encryption overhead
revocations revocation.DB revocations revocation.DB
defaultRS *pb.RedundancyScheme
config Config config Config
} }
@ -97,6 +98,16 @@ func NewEndpoint(log *zap.Logger, metainfo *Service, deletePieces *piecedeletion
if err != nil { if err != nil {
return nil, err return nil, err
} }
defaultRSScheme := &pb.RedundancyScheme{
Type: pb.RedundancyScheme_RS,
MinReq: int32(config.RS.Min),
RepairThreshold: int32(config.RS.Repair),
SuccessThreshold: int32(config.RS.Success),
Total: int32(config.RS.Total),
ErasureShareSize: config.RS.ErasureShareSize.Int32(),
}
return &Endpoint{ return &Endpoint{
log: log, log: log,
metainfo: metainfo, metainfo: metainfo,
@ -116,6 +127,7 @@ func NewEndpoint(log *zap.Logger, metainfo *Service, deletePieces *piecedeletion
}), }),
encInlineSegmentSize: encInlineSegmentSize, encInlineSegmentSize: encInlineSegmentSize,
revocations: revocations, revocations: revocations,
defaultRS: defaultRSScheme,
config: config, config: config,
}, nil }, nil
} }
@ -241,7 +253,7 @@ func (endpoint *Endpoint) GetBucket(ctx context.Context, req *pb.BucketGetReques
} }
// override RS to fit satellite settings // override RS to fit satellite settings
convBucket, err := convertBucketToProto(bucket, endpoint.redundancyScheme()) convBucket, err := convertBucketToProto(bucket, endpoint.defaultRS)
if err != nil { if err != nil {
return resp, err return resp, err
} }
@ -316,7 +328,7 @@ func (endpoint *Endpoint) CreateBucket(ctx context.Context, req *pb.BucketCreate
} }
// override RS to fit satellite settings // override RS to fit satellite settings
convBucket, err := convertBucketToProto(bucket, endpoint.redundancyScheme()) convBucket, err := convertBucketToProto(bucket, endpoint.defaultRS)
if err != nil { if err != nil {
endpoint.log.Error("error while converting bucket to proto", zap.String("bucketName", bucket.Name), zap.Error(err)) endpoint.log.Error("error while converting bucket to proto", zap.String("bucketName", bucket.Name), zap.Error(err))
return nil, rpcstatus.Error(rpcstatus.Internal, "unable to create bucket") return nil, rpcstatus.Error(rpcstatus.Internal, "unable to create bucket")
@ -375,7 +387,7 @@ func (endpoint *Endpoint) DeleteBucket(ctx context.Context, req *pb.BucketDelete
return nil, err return nil, err
} }
convBucket, err = convertBucketToProto(bucket, endpoint.redundancyScheme()) convBucket, err = convertBucketToProto(bucket, endpoint.defaultRS)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@ -657,7 +669,7 @@ func (endpoint *Endpoint) BeginObject(ctx context.Context, req *pb.ObjectBeginRe
} }
// use only satellite values for Redundancy Scheme // use only satellite values for Redundancy Scheme
pbRS := endpoint.redundancyScheme() pbRS := endpoint.defaultRS
streamID, err := endpoint.packStreamID(ctx, &internalpb.StreamID{ streamID, err := endpoint.packStreamID(ctx, &internalpb.StreamID{
Bucket: req.Bucket, Bucket: req.Bucket,
@ -1825,17 +1837,6 @@ func groupPiecesByNodeID(segments []metabase.DeletedSegmentInfo) map[storj.NodeI
return piecesToDelete return piecesToDelete
} }
func (endpoint *Endpoint) redundancyScheme() *pb.RedundancyScheme {
return &pb.RedundancyScheme{
Type: pb.RedundancyScheme_RS,
MinReq: int32(endpoint.config.RS.MinThreshold),
RepairThreshold: int32(endpoint.config.RS.RepairThreshold),
SuccessThreshold: int32(endpoint.config.RS.SuccessThreshold),
Total: int32(endpoint.config.RS.TotalThreshold),
ErasureShareSize: endpoint.config.RS.ErasureShareSize.Int32(),
}
}
// RevokeAPIKey handles requests to revoke an api key. // RevokeAPIKey handles requests to revoke an api key.
func (endpoint *Endpoint) RevokeAPIKey(ctx context.Context, req *pb.RevokeAPIKeyRequest) (resp *pb.RevokeAPIKeyResponse, err error) { func (endpoint *Endpoint) RevokeAPIKey(ctx context.Context, req *pb.RevokeAPIKeyRequest) (resp *pb.RevokeAPIKeyResponse, err error) {
defer mon.Task()(&ctx)(&err) defer mon.Task()(&ctx)(&err)

2
satellite/metainfo/metainfo_test.go
View File

@ -215,6 +215,7 @@ func TestInvalidAPIKey(t *testing.T) {
require.NoError(t, err) require.NoError(t, err)
for _, invalidAPIKey := range []string{"", "invalid", "testKey"} { for _, invalidAPIKey := range []string{"", "invalid", "testKey"} {
func() {
client, err := planet.Uplinks[0].DialMetainfo(ctx, planet.Satellites[0], throwawayKey) client, err := planet.Uplinks[0].DialMetainfo(ctx, planet.Satellites[0], throwawayKey)
require.NoError(t, err) require.NoError(t, err)
defer ctx.Check(client.Close) defer ctx.Check(client.Close)
@ -294,6 +295,7 @@ func TestInvalidAPIKey(t *testing.T) {
err = client.CommitSegment(ctx, metainfo.CommitSegmentParams{SegmentID: segmentID}) err = client.CommitSegment(ctx, metainfo.CommitSegmentParams{SegmentID: segmentID})
assertInvalidArgument(t, err, false) assertInvalidArgument(t, err, false)
}()
} }
}) })
} }

9
satellite/metainfo/piecedeletion/service_test.go
View File

@ -346,14 +346,13 @@ func TestService_DeletePieces_Timeout(t *testing.T) {
StorageNodeDB: func(index int, db storagenode.DB, log *zap.Logger) (storagenode.DB, error) { StorageNodeDB: func(index int, db storagenode.DB, log *zap.Logger) (storagenode.DB, error) {
return testblobs.NewSlowDB(log.Named("slowdb"), db), nil return testblobs.NewSlowDB(log.Named("slowdb"), db), nil
}, },
Satellite: func(log *zap.Logger, index int, config *satellite.Config) { Satellite: testplanet.Combine(
func(log *zap.Logger, index int, config *satellite.Config) {
config.Metainfo.PieceDeletion.RequestTimeout = 200 * time.Millisecond config.Metainfo.PieceDeletion.RequestTimeout = 200 * time.Millisecond
config.Metainfo.RS.MinThreshold = 2
config.Metainfo.RS.RepairThreshold = 2
config.Metainfo.RS.SuccessThreshold = 4
config.Metainfo.RS.TotalThreshold = 4
config.Metainfo.MaxSegmentSize = 15 * memory.KiB config.Metainfo.MaxSegmentSize = 15 * memory.KiB
}, },
testplanet.ReconfigureRS(2, 2, 4, 4),
),
}, },
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) { }, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
uplnk := planet.Uplinks[0] uplnk := planet.Uplinks[0]

4
satellite/orders/endpoint_test.go
View File

@ -76,6 +76,7 @@ func TestSettlementWithWindowEndpointManyOrders(t *testing.T) {
} }
for _, tt := range testCases { for _, tt := range testCases {
func() {
// create serial number to use in test. must be unique for each run. // create serial number to use in test. must be unique for each run.
serialNumber1 := testrand.SerialNumber() serialNumber1 := testrand.SerialNumber()
err = ordersDB.CreateSerialInfo(ctx, serialNumber1, []byte(bucketID), now.AddDate(1, 0, 10)) err = ordersDB.CreateSerialInfo(ctx, serialNumber1, []byte(bucketID), now.AddDate(1, 0, 10))
@ -176,6 +177,7 @@ func TestSettlementWithWindowEndpointManyOrders(t *testing.T) {
newBbw, err := ordersDB.GetBucketBandwidth(ctx, projectID, []byte(bucketname), time.Time{}, tt.orderCreation) newBbw, err := ordersDB.GetBucketBandwidth(ctx, projectID, []byte(bucketname), time.Time{}, tt.orderCreation)
require.NoError(t, err) require.NoError(t, err)
require.EqualValues(t, tt.settledAmt, newBbw) require.EqualValues(t, tt.settledAmt, newBbw)
}()
} }
}) })
} }
@ -223,6 +225,7 @@ func TestSettlementWithWindowEndpointSingleOrder(t *testing.T) {
} }
for _, tt := range testCases { for _, tt := range testCases {
func() {
// create signed orderlimit or order to test with // create signed orderlimit or order to test with
limit := &pb.OrderLimit{ limit := &pb.OrderLimit{
SerialNumber: serialNumber, SerialNumber: serialNumber,
@ -289,6 +292,7 @@ func TestSettlementWithWindowEndpointSingleOrder(t *testing.T) {
newBbw, err := ordersDB.GetBucketBandwidth(ctx, projectID, []byte(bucketname), time.Time{}, now) newBbw, err := ordersDB.GetBucketBandwidth(ctx, projectID, []byte(bucketname), time.Time{}, now)
require.NoError(t, err) require.NoError(t, err)
require.Equal(t, dataAmount, newBbw) require.Equal(t, dataAmount, newBbw)
}()
} }
}) })
} }

6
satellite/orders/service.go
View File

@ -566,7 +566,11 @@ func (service *Service) CreateGracefulExitPutOrderLimit(ctx context.Context, buc
return nil, storj.PiecePrivateKey{}, Error.Wrap(err) return nil, storj.PiecePrivateKey{}, Error.Wrap(err)
} }
nodeURL := storj.NodeURL{ID: nodeID, Address: node.Address.Address} address := node.Address.Address
if node.LastIPPort != "" {
address = node.LastIPPort
}
nodeURL := storj.NodeURL{ID: nodeID, Address: address}
limit, err = signer.Sign(ctx, nodeURL, pieceNum) limit, err = signer.Sign(ctx, nodeURL, pieceNum)
if err != nil { if err != nil {
return nil, storj.PiecePrivateKey{}, Error.Wrap(err) return nil, storj.PiecePrivateKey{}, Error.Wrap(err)

101
satellite/overlay/benchmark_test.go
View File

@ -89,35 +89,106 @@ func BenchmarkOverlay(b *testing.B) {
} }
}) })
b.Run("UpdateStats", func(b *testing.B) { b.Run("UpdateStatsSuccess", func(b *testing.B) {
for i := 0; i < b.N; i++ { for i := 0; i < b.N; i++ {
id := all[i%len(all)] id := all[i%len(all)]
outcome := overlay.AuditFailure
if i&1 == 0 {
outcome = overlay.AuditSuccess
}
_, err := overlaydb.UpdateStats(ctx, &overlay.UpdateRequest{ _, err := overlaydb.UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: id, NodeID: id,
AuditOutcome: outcome, AuditOutcome: overlay.AuditSuccess,
IsUp: i&2 == 0,
AuditHistory: testAuditHistoryConfig(), AuditHistory: testAuditHistoryConfig(),
}, time.Now()) }, time.Now())
require.NoError(b, err) require.NoError(b, err)
} }
}) })
b.Run("BatchUpdateStats", func(b *testing.B) { b.Run("UpdateStatsFailure", func(b *testing.B) {
for i := 0; i < b.N; i++ {
id := all[i%len(all)]
_, err := overlaydb.UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: id,
AuditOutcome: overlay.AuditFailure,
AuditHistory: testAuditHistoryConfig(),
}, time.Now())
require.NoError(b, err)
}
})
b.Run("UpdateStatsUnknown", func(b *testing.B) {
for i := 0; i < b.N; i++ {
id := all[i%len(all)]
_, err := overlaydb.UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: id,
AuditOutcome: overlay.AuditUnknown,
AuditHistory: testAuditHistoryConfig(),
}, time.Now())
require.NoError(b, err)
}
})
b.Run("UpdateStatsOffline", func(b *testing.B) {
for i := 0; i < b.N; i++ {
id := all[i%len(all)]
_, err := overlaydb.UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: id,
AuditOutcome: overlay.AuditOffline,
AuditHistory: testAuditHistoryConfig(),
}, time.Now())
require.NoError(b, err)
}
})
b.Run("BatchUpdateStatsSuccess", func(b *testing.B) {
var updateRequests []*overlay.UpdateRequest var updateRequests []*overlay.UpdateRequest
for i := 0; i < b.N; i++ { for i := 0; i < b.N; i++ {
id := all[i%len(all)] id := all[i%len(all)]
outcome := overlay.AuditFailure
if i&1 == 0 {
outcome = overlay.AuditSuccess
}
updateRequests = append(updateRequests, &overlay.UpdateRequest{ updateRequests = append(updateRequests, &overlay.UpdateRequest{
NodeID: id, NodeID: id,
AuditOutcome: outcome, AuditOutcome: overlay.AuditSuccess,
IsUp: i&2 == 0, AuditHistory: testAuditHistoryConfig(),
})
}
_, err := overlaydb.BatchUpdateStats(ctx, updateRequests, 100, time.Now())
require.NoError(b, err)
})
b.Run("BatchUpdateStatsFailure", func(b *testing.B) {
var updateRequests []*overlay.UpdateRequest
for i := 0; i < b.N; i++ {
id := all[i%len(all)]
updateRequests = append(updateRequests, &overlay.UpdateRequest{
NodeID: id,
AuditOutcome: overlay.AuditFailure,
AuditHistory: testAuditHistoryConfig(),
})
}
_, err := overlaydb.BatchUpdateStats(ctx, updateRequests, 100, time.Now())
require.NoError(b, err)
})
b.Run("BatchUpdateStatsUnknown", func(b *testing.B) {
var updateRequests []*overlay.UpdateRequest
for i := 0; i < b.N; i++ {
id := all[i%len(all)]
updateRequests = append(updateRequests, &overlay.UpdateRequest{
NodeID: id,
AuditOutcome: overlay.AuditUnknown,
AuditHistory: testAuditHistoryConfig(),
})
}
_, err := overlaydb.BatchUpdateStats(ctx, updateRequests, 100, time.Now())
require.NoError(b, err)
})
b.Run("BatchUpdateStatsOffline", func(b *testing.B) {
var updateRequests []*overlay.UpdateRequest
for i := 0; i < b.N; i++ {
id := all[i%len(all)]
updateRequests = append(updateRequests, &overlay.UpdateRequest{
NodeID: id,
AuditOutcome: overlay.AuditOffline,
AuditHistory: testAuditHistoryConfig(), AuditHistory: testAuditHistoryConfig(),
}) })
@ -272,7 +343,6 @@ func BenchmarkNodeSelection(b *testing.B) {
if i%2 == 0 { // make half of nodes "new" and half "vetted" if i%2 == 0 { // make half of nodes "new" and half "vetted"
_, err = overlaydb.UpdateStats(ctx, &overlay.UpdateRequest{ _, err = overlaydb.UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: nodeID, NodeID: nodeID,
IsUp: true,
AuditOutcome: overlay.AuditSuccess, AuditOutcome: overlay.AuditSuccess,
AuditLambda: 1, AuditLambda: 1,
AuditWeight: 1, AuditWeight: 1,
@ -293,7 +363,6 @@ func BenchmarkNodeSelection(b *testing.B) {
case 2: case 2:
err := overlaydb.UpdateCheckIn(ctx, overlay.NodeCheckInInfo{ err := overlaydb.UpdateCheckIn(ctx, overlay.NodeCheckInInfo{
NodeID: nodeID, NodeID: nodeID,
IsUp: true,
Address: &pb.NodeAddress{ Address: &pb.NodeAddress{
Address: address, Address: address,
}, },

1
satellite/overlay/nodeselectioncache_test.go
View File

@ -104,7 +104,6 @@ func addNodesToNodesTable(ctx context.Context, t *testing.T, db overlay.DB, coun
if i < makeReputable { if i < makeReputable {
stats, err := db.UpdateStats(ctx, &overlay.UpdateRequest{ stats, err := db.UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: storj.NodeID{byte(i)}, NodeID: storj.NodeID{byte(i)},
IsUp: true,
AuditOutcome: overlay.AuditSuccess, AuditOutcome: overlay.AuditSuccess,
AuditLambda: 1, AuditWeight: 1, AuditDQ: 0.5, AuditLambda: 1, AuditWeight: 1, AuditDQ: 0.5,
AuditHistory: testAuditHistoryConfig(), AuditHistory: testAuditHistoryConfig(),

5
satellite/overlay/selection_test.go
View File

@ -188,7 +188,6 @@ func TestEnsureMinimumRequested(t *testing.T) {
reputable[node.ID()] = true reputable[node.ID()] = true
_, err := satellite.DB.OverlayCache().UpdateStats(ctx, &overlay.UpdateRequest{ _, err := satellite.DB.OverlayCache().UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: node.ID(), NodeID: node.ID(),
IsUp: true,
AuditOutcome: overlay.AuditSuccess, AuditOutcome: overlay.AuditSuccess,
AuditLambda: 1, AuditWeight: 1, AuditDQ: 0.5, AuditLambda: 1, AuditWeight: 1, AuditDQ: 0.5,
AuditHistory: testAuditHistoryConfig(), AuditHistory: testAuditHistoryConfig(),
@ -232,7 +231,6 @@ func TestEnsureMinimumRequested(t *testing.T) {
reputable[node.ID()] = true reputable[node.ID()] = true
_, err := satellite.DB.OverlayCache().UpdateStats(ctx, &overlay.UpdateRequest{ _, err := satellite.DB.OverlayCache().UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: node.ID(), NodeID: node.ID(),
IsUp: true,
AuditOutcome: overlay.AuditSuccess, AuditOutcome: overlay.AuditSuccess,
AuditLambda: 1, AuditWeight: 1, AuditDQ: 0.5, AuditLambda: 1, AuditWeight: 1, AuditDQ: 0.5,
AuditHistory: testAuditHistoryConfig(), AuditHistory: testAuditHistoryConfig(),
@ -397,7 +395,6 @@ func TestNodeSelectionGracefulExit(t *testing.T) {
for k := 0; k < i; k++ { for k := 0; k < i; k++ {
_, err := satellite.DB.OverlayCache().UpdateStats(ctx, &overlay.UpdateRequest{ _, err := satellite.DB.OverlayCache().UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: node.ID(), NodeID: node.ID(),
IsUp: true,
AuditOutcome: overlay.AuditSuccess, AuditOutcome: overlay.AuditSuccess,
AuditLambda: 1, AuditWeight: 1, AuditDQ: 0.5, AuditLambda: 1, AuditWeight: 1, AuditDQ: 0.5,
AuditHistory: testAuditHistoryConfig(), AuditHistory: testAuditHistoryConfig(),
@ -628,7 +625,6 @@ func TestDistinctIPs(t *testing.T) {
for i := 9; i > 7; i-- { for i := 9; i > 7; i-- {
_, err := satellite.DB.OverlayCache().UpdateStats(ctx, &overlay.UpdateRequest{ _, err := satellite.DB.OverlayCache().UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: planet.StorageNodes[i].ID(), NodeID: planet.StorageNodes[i].ID(),
IsUp: true,
AuditOutcome: overlay.AuditSuccess, AuditOutcome: overlay.AuditSuccess,
AuditLambda: 1, AuditLambda: 1,
AuditWeight: 1, AuditWeight: 1,
@ -660,7 +656,6 @@ func TestDistinctIPsWithBatch(t *testing.T) {
// These are done individually b/c the previous stat data is important // These are done individually b/c the previous stat data is important
_, err := satellite.Overlay.Service.BatchUpdateStats(ctx, []*overlay.UpdateRequest{{ _, err := satellite.Overlay.Service.BatchUpdateStats(ctx, []*overlay.UpdateRequest{{
NodeID: planet.StorageNodes[i].ID(), NodeID: planet.StorageNodes[i].ID(),
IsUp: true,
AuditOutcome: overlay.AuditSuccess, AuditOutcome: overlay.AuditSuccess,
AuditLambda: 1, AuditLambda: 1,
AuditWeight: 1, AuditWeight: 1,

1
satellite/overlay/service.go
View File

@ -164,7 +164,6 @@ const (
type UpdateRequest struct { type UpdateRequest struct {
NodeID storj.NodeID NodeID storj.NodeID
AuditOutcome AuditType AuditOutcome AuditType
IsUp bool
// n.b. these are set values from the satellite. // n.b. these are set values from the satellite.
// They are part of the UpdateRequest struct in order to be // They are part of the UpdateRequest struct in order to be
// more easily accessible in satellite/satellitedb/overlaycache.go. // more easily accessible in satellite/satellitedb/overlaycache.go.

7
satellite/overlay/service_test.go
View File

@ -131,7 +131,6 @@ func testCache(ctx context.Context, t *testing.T, store overlay.DB) {
stats, err := service.UpdateStats(ctx, &overlay.UpdateRequest{ stats, err := service.UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: valid1ID, NodeID: valid1ID,
IsUp: true,
AuditOutcome: overlay.AuditFailure, AuditOutcome: overlay.AuditFailure,
}) })
require.NoError(t, err) require.NoError(t, err)
@ -148,7 +147,6 @@ func testCache(ctx context.Context, t *testing.T, store overlay.DB) {
// should not update once already disqualified // should not update once already disqualified
_, err = service.BatchUpdateStats(ctx, []*overlay.UpdateRequest{{ _, err = service.BatchUpdateStats(ctx, []*overlay.UpdateRequest{{
NodeID: valid2ID, NodeID: valid2ID,
IsUp: false,
AuditOutcome: overlay.AuditSuccess, AuditOutcome: overlay.AuditSuccess,
}}) }})
require.NoError(t, err) require.NoError(t, err)
@ -197,7 +195,6 @@ func TestRandomizedSelection(t *testing.T) {
if i%2 == 0 { // make half of nodes "new" and half "vetted" if i%2 == 0 { // make half of nodes "new" and half "vetted"
_, err = cache.UpdateStats(ctx, &overlay.UpdateRequest{ _, err = cache.UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: newID, NodeID: newID,
IsUp: true,
AuditOutcome: overlay.AuditSuccess, AuditOutcome: overlay.AuditSuccess,
AuditLambda: 1, AuditLambda: 1,
AuditWeight: 1, AuditWeight: 1,
@ -316,7 +313,6 @@ func TestRandomizedSelectionCache(t *testing.T) {
if i%2 == 0 { // make half of nodes "new" and half "vetted" if i%2 == 0 { // make half of nodes "new" and half "vetted"
_, err = overlaydb.UpdateStats(ctx, &overlay.UpdateRequest{ _, err = overlaydb.UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: newID, NodeID: newID,
IsUp: true,
AuditOutcome: overlay.AuditSuccess, AuditOutcome: overlay.AuditSuccess,
AuditLambda: 1, AuditLambda: 1,
AuditWeight: 1, AuditWeight: 1,
@ -775,7 +771,6 @@ func TestSuspendedSelection(t *testing.T) {
if i%2 == 0 { // make half of nodes "new" and half "vetted" if i%2 == 0 { // make half of nodes "new" and half "vetted"
_, err = cache.UpdateStats(ctx, &overlay.UpdateRequest{ _, err = cache.UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: newID, NodeID: newID,
IsUp: true,
AuditOutcome: overlay.AuditSuccess, AuditOutcome: overlay.AuditSuccess,
AuditLambda: 1, AuditLambda: 1,
AuditWeight: 1, AuditWeight: 1,
@ -835,7 +830,6 @@ func TestConcurrentAudit(t *testing.T) {
_, err := planet.Satellites[0].Overlay.Service.UpdateStats(ctx, &overlay.UpdateRequest{ _, err := planet.Satellites[0].Overlay.Service.UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: planet.StorageNodes[0].ID(), NodeID: planet.StorageNodes[0].ID(),
AuditOutcome: overlay.AuditSuccess, AuditOutcome: overlay.AuditSuccess,
IsUp: true,
}) })
return err return err
}) })
@ -853,7 +847,6 @@ func TestConcurrentAudit(t *testing.T) {
{ {
NodeID: planet.StorageNodes[0].ID(), NodeID: planet.StorageNodes[0].ID(),
AuditOutcome: overlay.AuditSuccess, AuditOutcome: overlay.AuditSuccess,
IsUp: true,
}, },
}) })
return err return err

2
satellite/overlay/statdb_test.go
View File

@ -169,7 +169,6 @@ func testDatabase(ctx context.Context, t *testing.T, cache overlay.DB) {
updateReq := &overlay.UpdateRequest{ updateReq := &overlay.UpdateRequest{
NodeID: nodeID, NodeID: nodeID,
AuditOutcome: overlay.AuditSuccess, AuditOutcome: overlay.AuditSuccess,
IsUp: true,
AuditLambda: 0.123, AuditWeight: 0.456, AuditLambda: 0.123, AuditWeight: 0.456,
AuditDQ: 0, // don't disqualify for any reason AuditDQ: 0, // don't disqualify for any reason
AuditHistory: testAuditHistoryConfig(), AuditHistory: testAuditHistoryConfig(),
@ -186,7 +185,6 @@ func testDatabase(ctx context.Context, t *testing.T, cache overlay.DB) {
auditBeta = expectedAuditBeta auditBeta = expectedAuditBeta
updateReq.AuditOutcome = overlay.AuditFailure updateReq.AuditOutcome = overlay.AuditFailure
updateReq.IsUp = false
stats, err = cache.UpdateStats(ctx, updateReq, time.Now()) stats, err = cache.UpdateStats(ctx, updateReq, time.Now())
require.NoError(t, err) require.NoError(t, err)

14
satellite/overlay/suspension_test.go
View File

@ -66,7 +66,6 @@ func TestAuditSuspendWithUpdateStats(t *testing.T) {
_, err = oc.UpdateStats(ctx, &overlay.UpdateRequest{ _, err = oc.UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: nodeID, NodeID: nodeID,
AuditOutcome: overlay.AuditUnknown, AuditOutcome: overlay.AuditUnknown,
IsUp: true,
AuditLambda: 1, AuditLambda: 1,
AuditWeight: 1, AuditWeight: 1,
AuditDQ: 0.6, AuditDQ: 0.6,
@ -90,7 +89,6 @@ func TestAuditSuspendWithUpdateStats(t *testing.T) {
_, err = oc.UpdateStats(ctx, &overlay.UpdateRequest{ _, err = oc.UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: nodeID, NodeID: nodeID,
AuditOutcome: overlay.AuditSuccess, AuditOutcome: overlay.AuditSuccess,
IsUp: true,
AuditLambda: 1, AuditLambda: 1,
AuditWeight: 1, AuditWeight: 1,
AuditDQ: 0.6, AuditDQ: 0.6,
@ -121,7 +119,6 @@ func TestAuditSuspendFailedAudit(t *testing.T) {
_, err = oc.UpdateStats(ctx, &overlay.UpdateRequest{ _, err = oc.UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: nodeID, NodeID: nodeID,
AuditOutcome: overlay.AuditFailure, AuditOutcome: overlay.AuditFailure,
IsUp: true,
AuditLambda: 1, AuditLambda: 1,
AuditWeight: 1, AuditWeight: 1,
AuditDQ: 0.6, AuditDQ: 0.6,
@ -277,7 +274,6 @@ func TestAuditSuspendBatchUpdateStats(t *testing.T) {
nodeUpdateReq := &overlay.UpdateRequest{ nodeUpdateReq := &overlay.UpdateRequest{
NodeID: nodeID, NodeID: nodeID,
AuditOutcome: overlay.AuditSuccess, AuditOutcome: overlay.AuditSuccess,
IsUp: true,
AuditLambda: 1, AuditLambda: 1,
AuditWeight: 1, AuditWeight: 1,
AuditDQ: 0.6, AuditDQ: 0.6,
@ -343,7 +339,6 @@ func TestOfflineSuspend(t *testing.T) {
updateReq := &overlay.UpdateRequest{ updateReq := &overlay.UpdateRequest{
NodeID: nodeID, NodeID: nodeID,
AuditOutcome: overlay.AuditOffline, AuditOutcome: overlay.AuditOffline,
IsUp: false,
AuditHistory: overlay.AuditHistoryConfig{ AuditHistory: overlay.AuditHistoryConfig{
WindowSize: time.Hour, WindowSize: time.Hour,
TrackingPeriod: 2 * time.Hour, TrackingPeriod: 2 * time.Hour,
@ -472,12 +467,11 @@ func setOnlineScore(ctx context.Context, reqPtr *overlay.UpdateRequest, desiredS
for window := 0; window < windowsPerTrackingPeriod+1; window++ { for window := 0; window < windowsPerTrackingPeriod+1; window++ {
updateReqs := []*overlay.UpdateRequest{} updateReqs := []*overlay.UpdateRequest{}
for i := 0; i < totalAudits; i++ { for i := 0; i < totalAudits; i++ {
isUp := true
if i >= onlineAudits {
isUp = false
}
updateReq := *reqPtr updateReq := *reqPtr
updateReq.IsUp = isUp updateReq.AuditOutcome = overlay.AuditSuccess
if i >= onlineAudits {
updateReq.AuditOutcome = overlay.AuditOffline
}
updateReq.AuditHistory.GracePeriod = gracePeriod updateReq.AuditHistory.GracePeriod = gracePeriod
updateReqs = append(updateReqs, &updateReq) updateReqs = append(updateReqs, &updateReq)

2
satellite/repair/checker/checker.go
View File

@ -391,7 +391,7 @@ func (obs *checkerObserver) InlineSegment(ctx context.Context, segment *metainfo
func (checker *Checker) IrreparableProcess(ctx context.Context) (err error) { func (checker *Checker) IrreparableProcess(ctx context.Context) (err error) {
defer mon.Task()(&ctx)(&err) defer mon.Task()(&ctx)(&err)
const limit = 1000 const limit = 1000
var lastSeenSegmentKey metabase.SegmentKey lastSeenSegmentKey := metabase.SegmentKey{}
for { for {
segments, err := checker.irrdb.GetLimited(ctx, limit, lastSeenSegmentKey) segments, err := checker.irrdb.GetLimited(ctx, limit, lastSeenSegmentKey)

87
satellite/repair/irreparable/irreparable_test.go
View File

@ -13,8 +13,10 @@ import (
"storj.io/common/pb" "storj.io/common/pb"
"storj.io/common/testcontext" "storj.io/common/testcontext"
"storj.io/storj/private/testplanet"
"storj.io/storj/satellite" "storj.io/storj/satellite"
"storj.io/storj/satellite/internalpb" "storj.io/storj/satellite/internalpb"
"storj.io/storj/satellite/metainfo/metabase"
"storj.io/storj/satellite/satellitedb/satellitedbtest" "storj.io/storj/satellite/satellitedb/satellitedbtest"
) )
@ -103,3 +105,88 @@ func TestIrreparable(t *testing.T) {
} }
}) })
} }
func TestIrreparableProcess(t *testing.T) {
testplanet.Run(t, testplanet.Config{
SatelliteCount: 1, StorageNodeCount: 3, UplinkCount: 0,
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
checker := planet.Satellites[0].Repair.Checker
checker.Loop.Stop()
checker.IrreparableLoop.Stop()
irreparabledb := planet.Satellites[0].DB.Irreparable()
queue := planet.Satellites[0].DB.RepairQueue()
seg := &internalpb.IrreparableSegment{
Path: []byte{1},
SegmentDetail: &pb.Pointer{
Type: pb.Pointer_REMOTE,
CreationDate: time.Now(),
Remote: &pb.RemoteSegment{
Redundancy: &pb.RedundancyScheme{
MinReq: 1,
RepairThreshold: 2,
SuccessThreshold: 3,
Total: 4,
},
RemotePieces: []*pb.RemotePiece{
{
NodeId: planet.StorageNodes[0].ID(),
},
{
NodeId: planet.StorageNodes[1].ID(),
},
{
NodeId: planet.StorageNodes[2].ID(),
},
},
},
},
LostPieces: int32(4),
LastRepairAttempt: time.Now().Unix(),
RepairAttemptCount: int64(10),
}
require.NoError(t, irreparabledb.IncrementRepairAttempts(ctx, seg))
result, err := irreparabledb.Get(ctx, metabase.SegmentKey(seg.GetPath()))
require.NoError(t, err)
require.NotNil(t, result)
// test healthy segment is removed from irreparable DB
require.NoError(t, checker.IrreparableProcess(ctx))
result, err = irreparabledb.Get(ctx, metabase.SegmentKey(seg.GetPath()))
require.Error(t, err)
require.Nil(t, result)
// test unhealthy repairable segment is removed from irreparable DB and inserted into repair queue
seg.SegmentDetail.Remote.RemotePieces[0] = &pb.RemotePiece{}
seg.SegmentDetail.Remote.RemotePieces[1] = &pb.RemotePiece{}
require.NoError(t, irreparabledb.IncrementRepairAttempts(ctx, seg))
require.NoError(t, checker.IrreparableProcess(ctx))
result, err = irreparabledb.Get(ctx, metabase.SegmentKey(seg.GetPath()))
require.Error(t, err)
require.Nil(t, result)
injured, err := queue.Select(ctx)
require.NoError(t, err)
require.Equal(t, seg.GetPath(), injured.GetPath())
n, err := queue.Clean(ctx, time.Now())
require.NoError(t, err)
require.EqualValues(t, 1, n)
// test irreparable segment remains in irreparable DB and repair_attempt_count is incremented
seg.SegmentDetail.Remote.RemotePieces[2] = &pb.RemotePiece{}
require.NoError(t, irreparabledb.IncrementRepairAttempts(ctx, seg))
require.NoError(t, checker.IrreparableProcess(ctx))
result, err = irreparabledb.Get(ctx, metabase.SegmentKey(seg.GetPath()))
require.NoError(t, err)
require.Equal(t, seg.GetPath(), result.Path)
require.Equal(t, seg.RepairAttemptCount+1, result.RepairAttemptCount)
})
}

84
satellite/repair/repair_test.go
View File

@ -53,15 +53,13 @@ func testDataRepair(t *testing.T, inMemoryRepair bool) {
StorageNodeCount: 14, StorageNodeCount: 14,
UplinkCount: 1, UplinkCount: 1,
Reconfigure: testplanet.Reconfigure{ Reconfigure: testplanet.Reconfigure{
Satellite: func(log *zap.Logger, index int, config *satellite.Config) { Satellite: testplanet.Combine(
func(log *zap.Logger, index int, config *satellite.Config) {
config.Repairer.MaxExcessRateOptimalThreshold = RepairMaxExcessRateOptimalThreshold config.Repairer.MaxExcessRateOptimalThreshold = RepairMaxExcessRateOptimalThreshold
config.Repairer.InMemoryRepair = inMemoryRepair config.Repairer.InMemoryRepair = inMemoryRepair
config.Metainfo.RS.MinThreshold = minThreshold
config.Metainfo.RS.RepairThreshold = 5
config.Metainfo.RS.SuccessThreshold = successThreshold
config.Metainfo.RS.TotalThreshold = 9
}, },
testplanet.ReconfigureRS(minThreshold, 5, successThreshold, 9),
),
}, },
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) { }, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
@ -188,15 +186,13 @@ func testCorruptDataRepairFailed(t *testing.T, inMemoryRepair bool) {
StorageNodeCount: 14, StorageNodeCount: 14,
UplinkCount: 1, UplinkCount: 1,
Reconfigure: testplanet.Reconfigure{ Reconfigure: testplanet.Reconfigure{
Satellite: func(log *zap.Logger, index int, config *satellite.Config) { Satellite: testplanet.Combine(
func(log *zap.Logger, index int, config *satellite.Config) {
config.Repairer.MaxExcessRateOptimalThreshold = RepairMaxExcessRateOptimalThreshold config.Repairer.MaxExcessRateOptimalThreshold = RepairMaxExcessRateOptimalThreshold
config.Repairer.InMemoryRepair = inMemoryRepair config.Repairer.InMemoryRepair = inMemoryRepair
config.Metainfo.RS.MinThreshold = 3
config.Metainfo.RS.RepairThreshold = 5
config.Metainfo.RS.SuccessThreshold = 7
config.Metainfo.RS.TotalThreshold = 9
}, },
testplanet.ReconfigureRS(3, 5, 7, 9),
),
}, },
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) { }, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
uplinkPeer := planet.Uplinks[0] uplinkPeer := planet.Uplinks[0]
@ -305,15 +301,13 @@ func testCorruptDataRepairSucceed(t *testing.T, inMemoryRepair bool) {
StorageNodeCount: 14, StorageNodeCount: 14,
UplinkCount: 1, UplinkCount: 1,
Reconfigure: testplanet.Reconfigure{ Reconfigure: testplanet.Reconfigure{
Satellite: func(log *zap.Logger, index int, config *satellite.Config) { Satellite: testplanet.Combine(
func(log *zap.Logger, index int, config *satellite.Config) {
config.Repairer.MaxExcessRateOptimalThreshold = RepairMaxExcessRateOptimalThreshold config.Repairer.MaxExcessRateOptimalThreshold = RepairMaxExcessRateOptimalThreshold
config.Repairer.InMemoryRepair = inMemoryRepair config.Repairer.InMemoryRepair = inMemoryRepair
config.Metainfo.RS.MinThreshold = 3
config.Metainfo.RS.RepairThreshold = 5
config.Metainfo.RS.SuccessThreshold = 7
config.Metainfo.RS.TotalThreshold = 9
}, },
testplanet.ReconfigureRS(3, 5, 7, 9),
),
}, },
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) { }, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
uplinkPeer := planet.Uplinks[0] uplinkPeer := planet.Uplinks[0]
@ -799,14 +793,12 @@ func testRepairMultipleDisqualifiedAndSuspended(t *testing.T, inMemoryRepair boo
StorageNodeCount: 12, StorageNodeCount: 12,
UplinkCount: 1, UplinkCount: 1,
Reconfigure: testplanet.Reconfigure{ Reconfigure: testplanet.Reconfigure{
Satellite: func(log *zap.Logger, index int, config *satellite.Config) { Satellite: testplanet.Combine(
func(log *zap.Logger, index int, config *satellite.Config) {
config.Repairer.InMemoryRepair = inMemoryRepair config.Repairer.InMemoryRepair = inMemoryRepair
config.Metainfo.RS.MinThreshold = 3
config.Metainfo.RS.RepairThreshold = 5
config.Metainfo.RS.SuccessThreshold = 7
config.Metainfo.RS.TotalThreshold = 7
}, },
testplanet.ReconfigureRS(3, 5, 7, 7),
),
}, },
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) { }, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
// first, upload some remote data // first, upload some remote data
@ -920,15 +912,13 @@ func testDataRepairOverrideHigherLimit(t *testing.T, inMemoryRepair bool) {
StorageNodeCount: 14, StorageNodeCount: 14,
UplinkCount: 1, UplinkCount: 1,
Reconfigure: testplanet.Reconfigure{ Reconfigure: testplanet.Reconfigure{
Satellite: func(log *zap.Logger, index int, config *satellite.Config) { Satellite: testplanet.Combine(
config.Checker.RepairOverride = repairOverride func(log *zap.Logger, index int, config *satellite.Config) {
config.Repairer.InMemoryRepair = inMemoryRepair config.Repairer.InMemoryRepair = inMemoryRepair
config.Checker.RepairOverride = repairOverride
config.Metainfo.RS.MinThreshold = 3
config.Metainfo.RS.RepairThreshold = 4
config.Metainfo.RS.SuccessThreshold = 9
config.Metainfo.RS.TotalThreshold = 9
}, },
testplanet.ReconfigureRS(3, 4, 9, 9),
),
}, },
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) { }, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
uplinkPeer := planet.Uplinks[0] uplinkPeer := planet.Uplinks[0]
@ -1014,15 +1004,13 @@ func testDataRepairOverrideLowerLimit(t *testing.T, inMemoryRepair bool) {
StorageNodeCount: 14, StorageNodeCount: 14,
UplinkCount: 1, UplinkCount: 1,
Reconfigure: testplanet.Reconfigure{ Reconfigure: testplanet.Reconfigure{
Satellite: func(log *zap.Logger, index int, config *satellite.Config) { Satellite: testplanet.Combine(
config.Checker.RepairOverride = repairOverride func(log *zap.Logger, index int, config *satellite.Config) {
config.Repairer.InMemoryRepair = inMemoryRepair config.Repairer.InMemoryRepair = inMemoryRepair
config.Checker.RepairOverride = repairOverride
config.Metainfo.RS.MinThreshold = 3
config.Metainfo.RS.RepairThreshold = 6
config.Metainfo.RS.SuccessThreshold = 9
config.Metainfo.RS.TotalThreshold = 9
}, },
testplanet.ReconfigureRS(3, 6, 9, 9),
),
}, },
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) { }, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
uplinkPeer := planet.Uplinks[0] uplinkPeer := planet.Uplinks[0]
@ -1141,15 +1129,13 @@ func testDataRepairUploadLimit(t *testing.T, inMemoryRepair bool) {
StorageNodeCount: 13, StorageNodeCount: 13,
UplinkCount: 1, UplinkCount: 1,
Reconfigure: testplanet.Reconfigure{ Reconfigure: testplanet.Reconfigure{
Satellite: func(log *zap.Logger, index int, config *satellite.Config) { Satellite: testplanet.Combine(
func(log *zap.Logger, index int, config *satellite.Config) {
config.Repairer.MaxExcessRateOptimalThreshold = RepairMaxExcessRateOptimalThreshold config.Repairer.MaxExcessRateOptimalThreshold = RepairMaxExcessRateOptimalThreshold
config.Repairer.InMemoryRepair = inMemoryRepair config.Repairer.InMemoryRepair = inMemoryRepair
config.Metainfo.RS.MinThreshold = 3
config.Metainfo.RS.RepairThreshold = repairThreshold
config.Metainfo.RS.SuccessThreshold = successThreshold
config.Metainfo.RS.TotalThreshold = maxThreshold
}, },
testplanet.ReconfigureRS(3, repairThreshold, successThreshold, maxThreshold),
),
}, },
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) { }, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
satellite := planet.Satellites[0] satellite := planet.Satellites[0]
@ -1266,14 +1252,12 @@ func testRepairGracefullyExited(t *testing.T, inMemoryRepair bool) {
StorageNodeCount: 12, StorageNodeCount: 12,
UplinkCount: 1, UplinkCount: 1,
Reconfigure: testplanet.Reconfigure{ Reconfigure: testplanet.Reconfigure{
Satellite: func(log *zap.Logger, index int, config *satellite.Config) { Satellite: testplanet.Combine(
func(log *zap.Logger, index int, config *satellite.Config) {
config.Repairer.InMemoryRepair = inMemoryRepair config.Repairer.InMemoryRepair = inMemoryRepair
config.Metainfo.RS.MinThreshold = 3
config.Metainfo.RS.RepairThreshold = 5
config.Metainfo.RS.SuccessThreshold = 7
config.Metainfo.RS.TotalThreshold = 7
}, },
testplanet.ReconfigureRS(3, 5, 7, 7),
),
}, },
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) { }, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
// first, upload some remote data // first, upload some remote data

1
satellite/repair/repairer/segments.go
View File

@ -346,7 +346,6 @@ func (repairer *SegmentRepairer) updateAuditFailStatus(ctx context.Context, fail
for i, nodeID := range failedAuditNodeIDs { for i, nodeID := range failedAuditNodeIDs {
updateRequests[i] = &overlay.UpdateRequest{ updateRequests[i] = &overlay.UpdateRequest{
NodeID: nodeID, NodeID: nodeID,
IsUp: true,
AuditOutcome: overlay.AuditFailure, AuditOutcome: overlay.AuditFailure,
} }
} }

11
satellite/satellitedb/overlaycache.go
View File

@ -366,7 +366,8 @@ func (cache *overlaycache) BatchUpdateStats(ctx context.Context, updateRequests
continue continue
} }
auditHistory, err := cache.updateAuditHistoryWithTx(ctx, tx, updateReq.NodeID, now, updateReq.IsUp, updateReq.AuditHistory) isUp := updateReq.AuditOutcome != overlay.AuditOffline
auditHistory, err := cache.updateAuditHistoryWithTx(ctx, tx, updateReq.NodeID, now, isUp, updateReq.AuditHistory)
if err != nil { if err != nil {
doAppendAll = false doAppendAll = false
return err return err
@ -444,7 +445,8 @@ func (cache *overlaycache) UpdateStats(ctx context.Context, updateReq *overlay.U
return nil return nil
} }
auditHistory, err := cache.updateAuditHistoryWithTx(ctx, tx, updateReq.NodeID, now, updateReq.IsUp, updateReq.AuditHistory) isUp := updateReq.AuditOutcome != overlay.AuditOffline
auditHistory, err := cache.updateAuditHistoryWithTx(ctx, tx, updateReq.NodeID, now, isUp, updateReq.AuditHistory)
if err != nil { if err != nil {
return err return err
} }
@ -1259,7 +1261,8 @@ func (cache *overlaycache) populateUpdateNodeStats(dbNode *dbx.Node, updateReq *
mon.FloatVal("audit_online_score").Observe(auditOnlineScore) //mon:locked mon.FloatVal("audit_online_score").Observe(auditOnlineScore) //mon:locked
totalUptimeCount := dbNode.TotalUptimeCount totalUptimeCount := dbNode.TotalUptimeCount
if updateReq.IsUp { isUp := updateReq.AuditOutcome != overlay.AuditOffline
if isUp {
totalUptimeCount++ totalUptimeCount++
} }
@ -1317,7 +1320,7 @@ func (cache *overlaycache) populateUpdateNodeStats(dbNode *dbx.Node, updateReq *
updateFields.UnknownAuditSuspended = timeField{set: true, isNil: true} updateFields.UnknownAuditSuspended = timeField{set: true, isNil: true}
} }
if updateReq.IsUp { if isUp {
updateFields.UptimeSuccessCount = int64Field{set: true, value: dbNode.UptimeSuccessCount + 1} updateFields.UptimeSuccessCount = int64Field{set: true, value: dbNode.UptimeSuccessCount + 1}
updateFields.LastContactSuccess = timeField{set: true, value: now} updateFields.LastContactSuccess = timeField{set: true, value: now}
} else { } else {

30
satellite/satellitedb/overlaycache_test.go
View File

@ -28,11 +28,11 @@ func TestUpdateStats(t *testing.T) {
numAudits := int64(2) numAudits := int64(2)
numUptimes := int64(3) numUptimes := int64(3)
// nodes automatically start with 2 uptimes from testplanet startup
// nodeA: 1 audit, 2 uptime -> unvetted // nodeA: 1 audit, 2 uptime -> unvetted
updateReq := &overlay.UpdateRequest{ updateReq := &overlay.UpdateRequest{
NodeID: nodeA.ID(), NodeID: nodeA.ID(),
AuditOutcome: overlay.AuditFailure, AuditOutcome: overlay.AuditOffline,
IsUp: false,
AuditsRequiredForVetting: numAudits, AuditsRequiredForVetting: numAudits,
UptimesRequiredForVetting: numUptimes, UptimesRequiredForVetting: numUptimes,
AuditHistory: testAuditHistoryConfig(), AuditHistory: testAuditHistoryConfig(),
@ -45,8 +45,7 @@ func TestUpdateStats(t *testing.T) {
// nodeA: 2 audits, 2 uptimes -> unvetted // nodeA: 2 audits, 2 uptimes -> unvetted
updateReq.NodeID = nodeA.ID() updateReq.NodeID = nodeA.ID()
updateReq.AuditOutcome = overlay.AuditFailure updateReq.AuditOutcome = overlay.AuditOffline
updateReq.IsUp = false
nodeStats, err = cache.UpdateStats(ctx, updateReq, time.Now()) nodeStats, err = cache.UpdateStats(ctx, updateReq, time.Now())
require.NoError(t, err) require.NoError(t, err)
assert.Nil(t, nodeStats.VettedAt) assert.Nil(t, nodeStats.VettedAt)
@ -56,7 +55,6 @@ func TestUpdateStats(t *testing.T) {
// nodeA: 3 audits, 3 uptimes -> vetted // nodeA: 3 audits, 3 uptimes -> vetted
updateReq.NodeID = nodeA.ID() updateReq.NodeID = nodeA.ID()
updateReq.AuditOutcome = overlay.AuditSuccess updateReq.AuditOutcome = overlay.AuditSuccess
updateReq.IsUp = true
nodeStats, err = cache.UpdateStats(ctx, updateReq, time.Now()) nodeStats, err = cache.UpdateStats(ctx, updateReq, time.Now())
require.NoError(t, err) require.NoError(t, err)
assert.NotNil(t, nodeStats.VettedAt) assert.NotNil(t, nodeStats.VettedAt)
@ -66,7 +64,6 @@ func TestUpdateStats(t *testing.T) {
// nodeB: 1 audit, 3 uptimes -> unvetted // nodeB: 1 audit, 3 uptimes -> unvetted
updateReq.NodeID = nodeB.ID() updateReq.NodeID = nodeB.ID()
updateReq.AuditOutcome = overlay.AuditSuccess updateReq.AuditOutcome = overlay.AuditSuccess
updateReq.IsUp = true
nodeStats, err = cache.UpdateStats(ctx, updateReq, time.Now()) nodeStats, err = cache.UpdateStats(ctx, updateReq, time.Now())
require.NoError(t, err) require.NoError(t, err)
assert.Nil(t, nodeStats.VettedAt) assert.Nil(t, nodeStats.VettedAt)
@ -75,8 +72,7 @@ func TestUpdateStats(t *testing.T) {
// nodeB: 2 audits, 3 uptimes -> vetted // nodeB: 2 audits, 3 uptimes -> vetted
updateReq.NodeID = nodeB.ID() updateReq.NodeID = nodeB.ID()
updateReq.AuditOutcome = overlay.AuditFailure updateReq.AuditOutcome = overlay.AuditOffline
updateReq.IsUp = false
nodeStats, err = cache.UpdateStats(ctx, updateReq, time.Now()) nodeStats, err = cache.UpdateStats(ctx, updateReq, time.Now())
require.NoError(t, err) require.NoError(t, err)
assert.NotNil(t, nodeStats.VettedAt) assert.NotNil(t, nodeStats.VettedAt)
@ -86,7 +82,6 @@ func TestUpdateStats(t *testing.T) {
// Don't overwrite node b's vetted_at timestamp // Don't overwrite node b's vetted_at timestamp
updateReq.NodeID = nodeB.ID() updateReq.NodeID = nodeB.ID()
updateReq.AuditOutcome = overlay.AuditSuccess updateReq.AuditOutcome = overlay.AuditSuccess
updateReq.IsUp = true
nodeStats2, err := cache.UpdateStats(ctx, updateReq, time.Now()) nodeStats2, err := cache.UpdateStats(ctx, updateReq, time.Now())
require.NoError(t, err) require.NoError(t, err)
assert.NotNil(t, nodeStats2.VettedAt) assert.NotNil(t, nodeStats2.VettedAt)
@ -110,9 +105,10 @@ func TestBatchUpdateStats(t *testing.T) {
numUptimes := int64(3) numUptimes := int64(3)
batchSize := 2 batchSize := 2
// nodes automatically start with 2 uptimes from testplanet startup
// both nodeA and nodeB unvetted // both nodeA and nodeB unvetted
updateReqA := &overlay.UpdateRequest{NodeID: nodeA.ID(), AuditOutcome: overlay.AuditFailure, IsUp: false, AuditsRequiredForVetting: numAudits, UptimesRequiredForVetting: numUptimes, AuditHistory: testAuditHistoryConfig()} updateReqA := &overlay.UpdateRequest{NodeID: nodeA.ID(), AuditOutcome: overlay.AuditOffline, AuditsRequiredForVetting: numAudits, UptimesRequiredForVetting: numUptimes, AuditHistory: testAuditHistoryConfig()}
updateReqB := &overlay.UpdateRequest{NodeID: nodeB.ID(), AuditOutcome: overlay.AuditSuccess, IsUp: true, AuditsRequiredForVetting: numAudits, UptimesRequiredForVetting: numUptimes, AuditHistory: testAuditHistoryConfig()} updateReqB := &overlay.UpdateRequest{NodeID: nodeB.ID(), AuditOutcome: overlay.AuditSuccess, AuditsRequiredForVetting: numAudits, UptimesRequiredForVetting: numUptimes, AuditHistory: testAuditHistoryConfig()}
updateReqs := []*overlay.UpdateRequest{updateReqA, updateReqB} updateReqs := []*overlay.UpdateRequest{updateReqA, updateReqB}
failed, err := cache.BatchUpdateStats(ctx, updateReqs, batchSize, time.Now()) failed, err := cache.BatchUpdateStats(ctx, updateReqs, batchSize, time.Now())
require.NoError(t, err) require.NoError(t, err)
@ -131,8 +127,8 @@ func TestBatchUpdateStats(t *testing.T) {
assert.EqualValues(t, 3, nB.Reputation.UptimeCount) assert.EqualValues(t, 3, nB.Reputation.UptimeCount)
// nodeA unvetted, nodeB vetted // nodeA unvetted, nodeB vetted
updateReqA = &overlay.UpdateRequest{NodeID: nodeA.ID(), AuditOutcome: overlay.AuditFailure, IsUp: false, AuditsRequiredForVetting: numAudits, UptimesRequiredForVetting: numUptimes, AuditHistory: testAuditHistoryConfig()} updateReqA = &overlay.UpdateRequest{NodeID: nodeA.ID(), AuditOutcome: overlay.AuditOffline, AuditsRequiredForVetting: numAudits, UptimesRequiredForVetting: numUptimes, AuditHistory: testAuditHistoryConfig()}
updateReqB = &overlay.UpdateRequest{NodeID: nodeB.ID(), AuditOutcome: overlay.AuditFailure, IsUp: false, AuditsRequiredForVetting: numAudits, UptimesRequiredForVetting: numUptimes, AuditHistory: testAuditHistoryConfig()} updateReqB = &overlay.UpdateRequest{NodeID: nodeB.ID(), AuditOutcome: overlay.AuditFailure, AuditsRequiredForVetting: numAudits, UptimesRequiredForVetting: numUptimes, AuditHistory: testAuditHistoryConfig()}
updateReqs = []*overlay.UpdateRequest{updateReqA, updateReqB} updateReqs = []*overlay.UpdateRequest{updateReqA, updateReqB}
failed, err = cache.BatchUpdateStats(ctx, updateReqs, batchSize, time.Now()) failed, err = cache.BatchUpdateStats(ctx, updateReqs, batchSize, time.Now())
require.NoError(t, err) require.NoError(t, err)
@ -148,11 +144,11 @@ func TestBatchUpdateStats(t *testing.T) {
require.NoError(t, err) require.NoError(t, err)
assert.NotNil(t, nB.Reputation.VettedAt) assert.NotNil(t, nB.Reputation.VettedAt)
assert.EqualValues(t, 2, nB.Reputation.AuditCount) assert.EqualValues(t, 2, nB.Reputation.AuditCount)
assert.EqualValues(t, 3, nB.Reputation.UptimeCount) assert.EqualValues(t, 4, nB.Reputation.UptimeCount)
// both nodeA and nodeB vetted (don't overwrite timestamp) // both nodeA and nodeB vetted (don't overwrite timestamp)
updateReqA = &overlay.UpdateRequest{NodeID: nodeA.ID(), AuditOutcome: overlay.AuditSuccess, IsUp: true, AuditsRequiredForVetting: numAudits, UptimesRequiredForVetting: numUptimes, AuditHistory: testAuditHistoryConfig()} updateReqA = &overlay.UpdateRequest{NodeID: nodeA.ID(), AuditOutcome: overlay.AuditSuccess, AuditsRequiredForVetting: numAudits, UptimesRequiredForVetting: numUptimes, AuditHistory: testAuditHistoryConfig()}
updateReqB = &overlay.UpdateRequest{NodeID: nodeB.ID(), AuditOutcome: overlay.AuditSuccess, IsUp: true, AuditsRequiredForVetting: numAudits, UptimesRequiredForVetting: numUptimes, AuditHistory: testAuditHistoryConfig()} updateReqB = &overlay.UpdateRequest{NodeID: nodeB.ID(), AuditOutcome: overlay.AuditSuccess, AuditsRequiredForVetting: numAudits, UptimesRequiredForVetting: numUptimes, AuditHistory: testAuditHistoryConfig()}
updateReqs = []*overlay.UpdateRequest{updateReqA, updateReqB} updateReqs = []*overlay.UpdateRequest{updateReqA, updateReqB}
failed, err = cache.BatchUpdateStats(ctx, updateReqs, batchSize, time.Now()) failed, err = cache.BatchUpdateStats(ctx, updateReqs, batchSize, time.Now())
require.NoError(t, err) require.NoError(t, err)
@ -169,7 +165,7 @@ func TestBatchUpdateStats(t *testing.T) {
assert.NotNil(t, nB2.Reputation.VettedAt) assert.NotNil(t, nB2.Reputation.VettedAt)
assert.Equal(t, nB.Reputation.VettedAt, nB2.Reputation.VettedAt) assert.Equal(t, nB.Reputation.VettedAt, nB2.Reputation.VettedAt)
assert.EqualValues(t, 3, nB2.Reputation.AuditCount) assert.EqualValues(t, 3, nB2.Reputation.AuditCount)
assert.EqualValues(t, 4, nB2.Reputation.UptimeCount) assert.EqualValues(t, 5, nB2.Reputation.UptimeCount)
}) })
} }

14
satellite/satellitedb/projectaccounting.go
View File

@ -358,6 +358,7 @@ func (db *ProjectAccounting) GetBucketUsageRollups(ctx context.Context, projectI
var bucketUsageRollups []accounting.BucketUsageRollup var bucketUsageRollups []accounting.BucketUsageRollup
for _, bucket := range buckets { for _, bucket := range buckets {
err := func() error {
bucketRollup := accounting.BucketUsageRollup{ bucketRollup := accounting.BucketUsageRollup{
ProjectID: projectID, ProjectID: projectID,
BucketName: []byte(bucket), BucketName: []byte(bucket),
@ -368,7 +369,7 @@ func (db *ProjectAccounting) GetBucketUsageRollups(ctx context.Context, projectI
// get bucket_bandwidth_rollups // get bucket_bandwidth_rollups
rollupsRows, err := db.db.QueryContext(ctx, roullupsQuery, projectID[:], []byte(bucket), since, before) rollupsRows, err := db.db.QueryContext(ctx, roullupsQuery, projectID[:], []byte(bucket), since, before)
if err != nil { if err != nil {
return nil, err return err
} }
defer func() { err = errs.Combine(err, rollupsRows.Close()) }() defer func() { err = errs.Combine(err, rollupsRows.Close()) }()
@ -379,7 +380,7 @@ func (db *ProjectAccounting) GetBucketUsageRollups(ctx context.Context, projectI
err = rollupsRows.Scan(&settled, &inline, &action) err = rollupsRows.Scan(&settled, &inline, &action)
if err != nil { if err != nil {
return nil, err return err
} }
switch action { switch action {
@ -394,7 +395,7 @@ func (db *ProjectAccounting) GetBucketUsageRollups(ctx context.Context, projectI
} }
} }
if err := rollupsRows.Err(); err != nil { if err := rollupsRows.Err(); err != nil {
return nil, err return err
} }
bucketStorageTallies, err := storageQuery(ctx, bucketStorageTallies, err := storageQuery(ctx,
@ -404,7 +405,7 @@ func (db *ProjectAccounting) GetBucketUsageRollups(ctx context.Context, projectI
dbx.BucketStorageTally_IntervalStart(before)) dbx.BucketStorageTally_IntervalStart(before))
if err != nil { if err != nil {
return nil, err return err
} }
// fill metadata, objects and stored data // fill metadata, objects and stored data
@ -424,6 +425,11 @@ func (db *ProjectAccounting) GetBucketUsageRollups(ctx context.Context, projectI
} }
bucketUsageRollups = append(bucketUsageRollups, bucketRollup) bucketUsageRollups = append(bucketUsageRollups, bucketRollup)
return nil
}()
if err != nil {
return nil, err
}
} }
return bucketUsageRollups, nil return bucketUsageRollups, nil

28
scripts/testdata/satellite-config.yaml.lock vendored
View File

@ -400,32 +400,8 @@ identity.key-path: /root/.local/share/storj/identity/satellite/identity.key
# request rate per project per second. # request rate per project per second.
# metainfo.rate-limiter.rate: 1000 # metainfo.rate-limiter.rate: 1000
# the size of each new erasure share in bytes # redundancy scheme configuration in the format k/m/o/n-sharesize
# metainfo.rs.erasure-share-size: 256 B # metainfo.rs: 29/35/80/110-256 B
# maximum buffer memory to be allocated for read buffers
# metainfo.rs.max-buffer-mem: 4.0 MiB
# the largest amount of pieces to encode to. n (upper bound for validation).
# metainfo.rs.max-total-threshold: 130
# the minimum pieces required to recover a segment. k.
# metainfo.rs.min-threshold: 29
# the largest amount of pieces to encode to. n (lower bound for validation).
# metainfo.rs.min-total-threshold: 95
# the minimum safe pieces before a repair is triggered. m.
# metainfo.rs.repair-threshold: 35
# the desired total pieces for a segment. o.
# metainfo.rs.success-threshold: 80
# the largest amount of pieces to encode to. n.
# metainfo.rs.total-threshold: 110
# validate redundancy scheme configuration
# metainfo.rs.validate: true
# address(es) to send telemetry to (comma-separated) # address(es) to send telemetry to (comma-separated)
# metrics.addr: collectora.storj.io:9000 # metrics.addr: collectora.storj.io:9000

2
storage/testsuite/test.go
View File

@ -163,6 +163,7 @@ func testConstraints(t *testing.T, ctx *testcontext.Context, store storage.KeyVa
{storage.Value("old-value"), nil}, {storage.Value("old-value"), nil},
{storage.Value("old-value"), storage.Value("new-value")}, {storage.Value("old-value"), storage.Value("new-value")},
} { } {
func() {
errTag := fmt.Sprintf("%d. %+v", i, tt) errTag := fmt.Sprintf("%d. %+v", i, tt)
key := storage.Key("test-key") key := storage.Key("test-key")
val := storage.Value("test-value") val := storage.Value("test-value")
@ -173,6 +174,7 @@ func testConstraints(t *testing.T, ctx *testcontext.Context, store storage.KeyVa
err = store.CompareAndSwap(ctx, key, tt.old, tt.new) err = store.CompareAndSwap(ctx, key, tt.old, tt.new)
assert.True(t, storage.ErrValueChanged.Has(err), "%s: unexpected error: %+v", errTag, err) assert.True(t, storage.ErrValueChanged.Has(err), "%s: unexpected error: %+v", errTag, err)
}()
} }
}) })

9
storagenode/console/consoleapi/payout_test.go
View File

@ -376,8 +376,15 @@ func TestHeldAmountApi(t *testing.T) {
JoinedAt: date, JoinedAt: date,
} }
stefanID, err := storj.NodeIDFromString("118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW")
require.NoError(t, err)
held2 := payout.SatelliteHeldHistory{
SatelliteID: stefanID,
}
var periods []payout.SatelliteHeldHistory var periods []payout.SatelliteHeldHistory
periods = append(periods, held) periods = append(periods, held, held2)
expected, err := json.Marshal(periods) expected, err := json.Marshal(periods)
require.NoError(t, err) require.NoError(t, err)

11
storagenode/gracefulexit/worker_test.go
View File

@ -177,15 +177,14 @@ func TestWorkerFailure_IneligibleNodeAge(t *testing.T) {
StorageNodeCount: 5, StorageNodeCount: 5,
UplinkCount: 1, UplinkCount: 1,
Reconfigure: testplanet.Reconfigure{ Reconfigure: testplanet.Reconfigure{
Satellite: func(logger *zap.Logger, index int, config *satellite.Config) { Satellite: testplanet.Combine(
func(log *zap.Logger, index int, config *satellite.Config) {
// Set the required node age to 1 month. // Set the required node age to 1 month.
config.GracefulExit.NodeMinAgeInMonths = 1 config.GracefulExit.NodeMinAgeInMonths = 1
config.Metainfo.RS.MinThreshold = 2
config.Metainfo.RS.RepairThreshold = 3
config.Metainfo.RS.SuccessThreshold = successThreshold
config.Metainfo.RS.TotalThreshold = successThreshold
}, },
testplanet.ReconfigureRS(2, 3, successThreshold, successThreshold),
),
StorageNode: func(index int, config *storagenode.Config) { StorageNode: func(index int, config *storagenode.Config) {
config.GracefulExit.NumWorkers = 2 config.GracefulExit.NumWorkers = 2
config.GracefulExit.NumConcurrentTransfers = 2 config.GracefulExit.NumConcurrentTransfers = 2

2
storagenode/orders/service.go
View File

@ -398,7 +398,6 @@ func (service *Service) sendOrdersFromFileStore(ctx context.Context, now time.Ti
var group errgroup.Group var group errgroup.Group
attemptedSatellites := 0 attemptedSatellites := 0
ctx, cancel := context.WithTimeout(ctx, service.config.SenderTimeout) ctx, cancel := context.WithTimeout(ctx, service.config.SenderTimeout)
defer cancel()
for satelliteID, unsentInfo := range ordersBySatellite { for satelliteID, unsentInfo := range ordersBySatellite {
satelliteID, unsentInfo := satelliteID, unsentInfo satelliteID, unsentInfo := satelliteID, unsentInfo
@ -430,6 +429,7 @@ func (service *Service) sendOrdersFromFileStore(ctx context.Context, now time.Ti
} }
_ = group.Wait() // doesn't return errors _ = group.Wait() // doesn't return errors
cancel()
// if all satellites that orders need to be sent to are offline, exit and try again later. // if all satellites that orders need to be sent to are offline, exit and try again later.
if attemptedSatellites == 0 { if attemptedSatellites == 0 {

6
storagenode/payout/db_test.go
View File

@ -211,7 +211,8 @@ func TestSatellitePayStubPeriodCached(t *testing.T) {
heldAmountDB := db.Payout() heldAmountDB := db.Payout()
reputationDB := db.Reputation() reputationDB := db.Reputation()
satellitesDB := db.Satellites() satellitesDB := db.Satellites()
service := payout.NewService(nil, heldAmountDB, reputationDB, satellitesDB, nil) service, err := payout.NewService(nil, heldAmountDB, reputationDB, satellitesDB, nil)
require.NoError(t, err)
payStub := payout.PayStub{ payStub := payout.PayStub{
SatelliteID: storj.NodeID{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1}, SatelliteID: storj.NodeID{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1},
@ -261,7 +262,8 @@ func TestAllPayStubPeriodCached(t *testing.T) {
heldAmountDB := db.Payout() heldAmountDB := db.Payout()
reputationDB := db.Reputation() reputationDB := db.Reputation()
satellitesDB := db.Satellites() satellitesDB := db.Satellites()
service := payout.NewService(nil, heldAmountDB, reputationDB, satellitesDB, nil) service, err := payout.NewService(nil, heldAmountDB, reputationDB, satellitesDB, nil)
require.NoError(t, err)
payStub := payout.PayStub{ payStub := payout.PayStub{
SatelliteID: storj.NodeID{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1}, SatelliteID: storj.NodeID{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1},

48
storagenode/payout/service.go
View File

@ -39,6 +39,8 @@ var (
type Service struct { type Service struct {
log *zap.Logger log *zap.Logger
stefanSatellite storj.NodeID
db DB db DB
reputationDB reputation.DB reputationDB reputation.DB
satellitesDB satellites.DB satellitesDB satellites.DB
@ -46,14 +48,20 @@ type Service struct {
} }
// NewService creates new instance of service. // NewService creates new instance of service.
func NewService(log *zap.Logger, db DB, reputationDB reputation.DB, satelliteDB satellites.DB, trust *trust.Pool) *Service { func NewService(log *zap.Logger, db DB, reputationDB reputation.DB, satelliteDB satellites.DB, trust *trust.Pool) (_ *Service, err error) {
id, err := storj.NodeIDFromString("118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW")
if err != nil {
return &Service{}, err
}
return &Service{ return &Service{
log: log, log: log,
stefanSatellite: id,
db: db, db: db,
reputationDB: reputationDB, reputationDB: reputationDB,
satellitesDB: satelliteDB, satellitesDB: satelliteDB,
trust: trust, trust: trust,
} }, nil
} }
// SatellitePayStubMonthly retrieves held amount for particular satellite for selected month from storagenode database. // SatellitePayStubMonthly retrieves held amount for particular satellite for selected month from storagenode database.
@ -161,17 +169,18 @@ func (service *Service) AllPeriods(ctx context.Context) (_ []string, err error)
// AllHeldbackHistory retrieves heldback history for all satellites from storagenode database. // AllHeldbackHistory retrieves heldback history for all satellites from storagenode database.
func (service *Service) AllHeldbackHistory(ctx context.Context) (result []SatelliteHeldHistory, err error) { func (service *Service) AllHeldbackHistory(ctx context.Context) (result []SatelliteHeldHistory, err error) {
defer mon.Task()(&ctx)(&err) defer mon.Task()(&ctx)(&err)
satellitesIDs := service.trust.GetSatellites(ctx)
satellites := service.trust.GetSatellites(ctx) satellitesIDs = append(satellitesIDs, service.stefanSatellite)
for i := 0; i < len(satellites); i++ { for i := 0; i < len(satellitesIDs); i++ {
var history SatelliteHeldHistory var history SatelliteHeldHistory
helds, err := service.db.SatellitesHeldbackHistory(ctx, satellites[i]) helds, err := service.db.SatellitesHeldbackHistory(ctx, satellitesIDs[i])
if err != nil { if err != nil {
return nil, ErrPayoutService.Wrap(err) return nil, ErrPayoutService.Wrap(err)
} }
disposed, err := service.db.SatellitesDisposedHistory(ctx, satellites[i]) disposed, err := service.db.SatellitesDisposedHistory(ctx, satellitesIDs[i])
if err != nil { if err != nil {
return nil, ErrPayoutService.Wrap(err) return nil, ErrPayoutService.Wrap(err)
} }
@ -192,20 +201,22 @@ func (service *Service) AllHeldbackHistory(ctx context.Context) (result []Satell
} }
history.TotalDisposed = disposed history.TotalDisposed = disposed
history.SatelliteID = satellites[i] history.SatelliteID = satellitesIDs[i]
url, err := service.trust.GetNodeURL(ctx, satellites[i])
if satellitesIDs[i] != service.stefanSatellite {
url, err := service.trust.GetNodeURL(ctx, satellitesIDs[i])
if err != nil { if err != nil {
return nil, ErrPayoutService.Wrap(err) return nil, ErrPayoutService.Wrap(err)
} }
stats, err := service.reputationDB.Get(ctx, satellites[i])
if err != nil {
return nil, ErrPayoutService.Wrap(err)
}
history.SatelliteName = url.Address history.SatelliteName = url.Address
history.JoinedAt = stats.JoinedAt }
stats, err := service.reputationDB.Get(ctx, satellitesIDs[i])
if err != nil {
return nil, ErrPayoutService.Wrap(err)
}
history.JoinedAt = stats.JoinedAt
result = append(result, history) result = append(result, history)
} }
@ -217,6 +228,8 @@ func (service *Service) AllSatellitesPayoutPeriod(ctx context.Context, period st
defer mon.Task()(&ctx)(&err) defer mon.Task()(&ctx)(&err)
satelliteIDs := service.trust.GetSatellites(ctx) satelliteIDs := service.trust.GetSatellites(ctx)
satelliteIDs = append(satelliteIDs, service.stefanSatellite)
for i := 0; i < len(satelliteIDs); i++ { for i := 0; i < len(satelliteIDs); i++ {
var payoutForPeriod SatellitePayoutForPeriod var payoutForPeriod SatellitePayoutForPeriod
paystub, err := service.db.GetPayStub(ctx, satelliteIDs[i], period) paystub, err := service.db.GetPayStub(ctx, satelliteIDs[i], period)
@ -249,11 +262,15 @@ func (service *Service) AllSatellitesPayoutPeriod(ctx context.Context, period st
return nil, ErrPayoutService.Wrap(err) return nil, ErrPayoutService.Wrap(err)
} }
if satelliteIDs[i] != service.stefanSatellite {
url, err := service.trust.GetNodeURL(ctx, satelliteIDs[i]) url, err := service.trust.GetNodeURL(ctx, satelliteIDs[i])
if err != nil { if err != nil {
return nil, ErrPayoutService.Wrap(err) return nil, ErrPayoutService.Wrap(err)
} }
payoutForPeriod.SatelliteURL = url.Address
}
if satellite.Status == satellites.ExitSucceeded { if satellite.Status == satellites.ExitSucceeded {
payoutForPeriod.IsExitComplete = true payoutForPeriod.IsExitComplete = true
} }
@ -281,7 +298,6 @@ func (service *Service) AllSatellitesPayoutPeriod(ctx context.Context, period st
payoutForPeriod.Earned = earned payoutForPeriod.Earned = earned
payoutForPeriod.SatelliteID = satelliteIDs[i].String() payoutForPeriod.SatelliteID = satelliteIDs[i].String()
payoutForPeriod.SurgePercent = paystub.SurgePercent payoutForPeriod.SurgePercent = paystub.SurgePercent
payoutForPeriod.SatelliteURL = url.Address
payoutForPeriod.Paid = paystub.Paid payoutForPeriod.Paid = paystub.Paid
payoutForPeriod.HeldPercent = heldPercent payoutForPeriod.HeldPercent = heldPercent

6
storagenode/peer.go
View File

@ -555,13 +555,17 @@ func New(log *zap.Logger, full *identity.FullIdentity, db DB, revocationDB exten
} }
{ // setup payout service. { // setup payout service.
peer.Payout.Service = payout.NewService( service, err := payout.NewService(
peer.Log.Named("payout:service"), peer.Log.Named("payout:service"),
peer.DB.Payout(), peer.DB.Payout(),
peer.DB.Reputation(), peer.DB.Reputation(),
peer.DB.Satellites(), peer.DB.Satellites(),
peer.Storage2.Trust, peer.Storage2.Trust,
) )
if err != nil {
return nil, errs.Combine(err, peer.Close())
}
peer.Payout.Service = service
peer.Payout.Endpoint = payout.NewEndpoint( peer.Payout.Endpoint = payout.NewEndpoint(
peer.Log.Named("payout:endpoint"), peer.Log.Named("payout:endpoint"),
peer.Dialer, peer.Dialer,

4
storagenode/piecestore/endpoint_test.go
View File

@ -50,6 +50,7 @@ func TestUploadAndPartialDownload(t *testing.T) {
{1513, 1584}, {1513, 1584},
{13581, 4783}, {13581, 4783},
} { } {
func() {
if piecestore.DefaultConfig.InitialStep < tt.size { if piecestore.DefaultConfig.InitialStep < tt.size {
t.Fatal("test expects initial step to be larger than size to download") t.Fatal("test expects initial step to be larger than size to download")
} }
@ -67,6 +68,7 @@ func TestUploadAndPartialDownload(t *testing.T) {
assert.Equal(t, expectedData[tt.offset:tt.offset+tt.size], data) assert.Equal(t, expectedData[tt.offset:tt.offset+tt.size], data)
require.NoError(t, download.Close()) require.NoError(t, download.Close())
}()
} }
var totalBandwidthUsage bandwidth.Usage var totalBandwidthUsage bandwidth.Usage
@ -528,8 +530,6 @@ func TestDeletePieces(t *testing.T) {
} }
func TestTooManyRequests(t *testing.T) { func TestTooManyRequests(t *testing.T) {
t.Skip("flaky, because of EOF issues")
const uplinkCount = 6 const uplinkCount = 6
const maxConcurrent = 3 const maxConcurrent = 3
const expectedFailures = uplinkCount - maxConcurrent const expectedFailures = uplinkCount - maxConcurrent

2
storagenode/piecestore/verification_test.go
View File

@ -226,6 +226,7 @@ func TestOrderLimitGetValidation(t *testing.T) {
err: "expected get or get repair or audit action got PUT", err: "expected get or get repair or audit action got PUT",
}, },
} { } {
func() {
client, err := planet.Uplinks[0].DialPiecestore(ctx, planet.StorageNodes[0]) client, err := planet.Uplinks[0].DialPiecestore(ctx, planet.StorageNodes[0])
require.NoError(t, err) require.NoError(t, err)
defer ctx.Check(client.Close) defer ctx.Check(client.Close)
@ -265,6 +266,7 @@ func TestOrderLimitGetValidation(t *testing.T) {
} else { } else {
require.NoError(t, err) require.NoError(t, err)
} }
}()
} }
}) })
} }

10
storagenode/storagenodedb/database.go
View File

@ -349,6 +349,14 @@ func (db *DB) MigrateToLatest(ctx context.Context) error {
// Preflight conducts a pre-flight check to ensure correct schemas and minimal read+write functionality of the database tables. // Preflight conducts a pre-flight check to ensure correct schemas and minimal read+write functionality of the database tables.
func (db *DB) Preflight(ctx context.Context) (err error) { func (db *DB) Preflight(ctx context.Context) (err error) {
for dbName, dbContainer := range db.SQLDBs { for dbName, dbContainer := range db.SQLDBs {
if err := db.preflight(ctx, dbName, dbContainer); err != nil {
return err
}
}
return nil
}
func (db *DB) preflight(ctx context.Context, dbName string, dbContainer DBContainer) error {
nextDB := dbContainer.GetDB() nextDB := dbContainer.GetDB()
// Preflight stage 1: test schema correctness // Preflight stage 1: test schema correctness
schema, err := sqliteutil.QuerySchema(ctx, nextDB) schema, err := sqliteutil.QuerySchema(ctx, nextDB)
@ -442,7 +450,7 @@ func (db *DB) Preflight(ctx context.Context) (err error) {
if err != nil { if err != nil {
return ErrPreflight.New("database %q: failed drop test_table %w", dbName, err) return ErrPreflight.New("database %q: failed drop test_table %w", dbName, err)
} }
}
return nil return nil
} }

2
web/satellite/README.md
View File

@ -67,7 +67,7 @@ docker run -p 8080:8080 storjlabs/satellite-ui:latest
- [unit](./unit "unit") folder: contains project unit tests. - [unit](./unit "unit") folder: contains project unit tests.
### Configuration files ### Configuration files
- **.env**: file for environment level variables. - **.env**: file for environment level variables.
- **.gitignore**: folders, files and extentions which are ignored for git. - **.gitignore**: folders, files and extensions which are ignored for git.
- **babel.config.js**: [babel](https://babeljs.io/) configuration for javascript transcompilation. - **babel.config.js**: [babel](https://babeljs.io/) configuration for javascript transcompilation.
- **index.html**: DOM entry point. - **index.html**: DOM entry point.
- **jestSetup.ts**: [jest](https://jestjs.io/) configuration for unit testing. - **jestSetup.ts**: [jest](https://jestjs.io/) configuration for unit testing.

14
web/storagenode/src/app/components/SNOHeader.vue
View File

@ -68,7 +68,6 @@ import { APPSTATE_ACTIONS } from '@/app/store/modules/appState';
import { NODE_ACTIONS } from '@/app/store/modules/node'; import { NODE_ACTIONS } from '@/app/store/modules/node';
import { NOTIFICATIONS_ACTIONS } from '@/app/store/modules/notifications'; import { NOTIFICATIONS_ACTIONS } from '@/app/store/modules/notifications';
import { PAYOUT_ACTIONS } from '@/app/store/modules/payout'; import { PAYOUT_ACTIONS } from '@/app/store/modules/payout';
import { NotificationsCursor } from '@/app/types/notifications';
const { const {
GET_NODE_INFO, GET_NODE_INFO,
@ -90,18 +89,23 @@ const {
export default class SNOHeader extends Vue { export default class SNOHeader extends Vue {
public isNotificationPopupShown: boolean = false; public isNotificationPopupShown: boolean = false;
public isOptionsShown: boolean = false; public isOptionsShown: boolean = false;
private readonly FIRST_PAGE: number = 1;
/** /**
* Lifecycle hook before render. * Lifecycle hook before render.
* Fetches first page of notifications. * Fetches first page of notifications.
*/ */
public beforeMount(): void { public async beforeMount(): Promise<void> {
await this.$store.dispatch(APPSTATE_ACTIONS.SET_LOADING, true);
try { try {
this.$store.dispatch(NODE_ACTIONS.GET_NODE_INFO); await this.$store.dispatch(NODE_ACTIONS.GET_NODE_INFO);
this.$store.dispatch(NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS, new NotificationsCursor(1)); await this.$store.dispatch(NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS, this.FIRST_PAGE);
} catch (error) { } catch (error) {
console.error(error.message); console.error(error.message);
} }
await this.$store.dispatch(APPSTATE_ACTIONS.SET_LOADING, false);
} }
public get nodeId(): string { public get nodeId(): string {
@ -192,7 +196,7 @@ export default class SNOHeader extends Vue {
} }
try { try {
await this.$store.dispatch(NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS, new NotificationsCursor(1)); await this.$store.dispatch(NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS, this.FIRST_PAGE);
} catch (error) { } catch (error) {
console.error(error.message); console.error(error.message);
} }

9
web/storagenode/src/app/components/notifications/NotificationsPopup.vue
View File

@ -12,10 +12,10 @@
<div <div
class="notification-popup-container__content" class="notification-popup-container__content"
:class="{'collapsed': isCollapsed}" :class="{'collapsed': isCollapsed}"
v-if="latestNotifications.length" v-if="latest.length"
> >
<SNONotification <SNONotification
v-for="notification in latestNotifications" v-for="notification in latest"
:key="notification.id" :key="notification.id"
is-small="true" is-small="true"
:notification="notification" :notification="notification"
@ -34,6 +34,7 @@ import { Component, Vue } from 'vue-property-decorator';
import SNONotification from '@/app/components/notifications/SNONotification.vue'; import SNONotification from '@/app/components/notifications/SNONotification.vue';
import { RouteConfig } from '@/app/router'; import { RouteConfig } from '@/app/router';
import { UINotification } from '@/app/types/notifications';
@Component({ @Component({
components: { components: {
@ -49,7 +50,7 @@ export default class NotificationsPopup extends Vue {
/** /**
* Represents first page of notifications. * Represents first page of notifications.
*/ */
public get latestNotifications(): Notification[] { public get latest(): UINotification[] {
return this.$store.state.notificationsModule.latestNotifications; return this.$store.state.notificationsModule.latestNotifications;
} }
@ -57,7 +58,7 @@ export default class NotificationsPopup extends Vue {
* Indicates if popup is smaller than with scroll. * Indicates if popup is smaller than with scroll.
*/ */
public get isCollapsed(): boolean { public get isCollapsed(): boolean {
return this.latestNotifications.length < 4; return this.latest.length < 4;
} }
} }
</script> </script>

6
web/storagenode/src/app/components/notifications/SNONotification.vue
View File

@ -30,12 +30,12 @@
import { Component, Prop, Vue } from 'vue-property-decorator'; import { Component, Prop, Vue } from 'vue-property-decorator';
import { NOTIFICATIONS_ACTIONS } from '@/app/store/modules/notifications'; import { NOTIFICATIONS_ACTIONS } from '@/app/store/modules/notifications';
import { Notification } from '@/app/types/notifications'; import { UINotification } from '@/app/types/notifications';
@Component @Component
export default class SNONotification extends Vue { export default class SNONotification extends Vue {
@Prop({default: () => new Notification()}) @Prop({default: () => new UINotification()})
public readonly notification: Notification; public readonly notification: UINotification;
/** /**
* isSmall props indicates if component used in popup. * isSmall props indicates if component used in popup.

2
web/storagenode/src/app/components/payments/HeldProgress.vue
View File

@ -89,7 +89,7 @@ export default class HeldProgress extends Vue {
), ),
new HeldStep( new HeldStep(
'+50%', '+50%',
'Month 15', 'Month 16',
this.monthsOnNetwork > 15, this.monthsOnNetwork > 15,
this.monthsOnNetwork < 15, this.monthsOnNetwork < 15,
), ),

6
web/storagenode/src/app/store/index.ts
View File

@ -4,17 +4,19 @@
import Vue from 'vue'; import Vue from 'vue';
import Vuex from 'vuex'; import Vuex from 'vuex';
import { makeNotificationsModule } from '@/app/store/modules/notifications'; import { newNotificationsModule } from '@/app/store/modules/notifications';
import { makePayoutModule } from '@/app/store/modules/payout'; import { makePayoutModule } from '@/app/store/modules/payout';
import { NotificationsHttpApi } from '@/storagenode/api/notifications'; import { NotificationsHttpApi } from '@/storagenode/api/notifications';
import { PayoutHttpApi } from '@/storagenode/api/payout'; import { PayoutHttpApi } from '@/storagenode/api/payout';
import { SNOApi } from '@/storagenode/api/storagenode'; import { SNOApi } from '@/storagenode/api/storagenode';
import { NotificationsService } from '@/storagenode/notifications/service';
import { PayoutService } from '@/storagenode/payouts/service'; import { PayoutService } from '@/storagenode/payouts/service';
import { appStateModule } from './modules/appState'; import { appStateModule } from './modules/appState';
import { makeNodeModule } from './modules/node'; import { makeNodeModule } from './modules/node';
const notificationsApi = new NotificationsHttpApi(); const notificationsApi = new NotificationsHttpApi();
const notificationsService = new NotificationsService(notificationsApi);
const payoutApi = new PayoutHttpApi(); const payoutApi = new PayoutHttpApi();
const payoutService = new PayoutService(payoutApi); const payoutService = new PayoutService(payoutApi);
const nodeApi = new SNOApi(); const nodeApi = new SNOApi();
@ -28,7 +30,7 @@ export const store = new Vuex.Store({
modules: { modules: {
node: makeNodeModule(nodeApi), node: makeNodeModule(nodeApi),
appStateModule, appStateModule,
notificationsModule: makeNotificationsModule(notificationsApi), notificationsModule: newNotificationsModule(notificationsService),
payoutModule: makePayoutModule(payoutApi, payoutService), payoutModule: makePayoutModule(payoutApi, payoutService),
}, },
}); });

52
web/storagenode/src/app/store/modules/notifications.ts
View File

@ -1,12 +1,8 @@
// Copyright (C) 2019 Storj Labs, Inc. // Copyright (C) 2019 Storj Labs, Inc.
// See LICENSE for copying information. // See LICENSE for copying information.
import { import { NotificationsState, UINotification } from '@/app/types/notifications';
Notification, import { NotificationsService } from '@/storagenode/notifications/service';
NotificationsApi,
NotificationsCursor,
NotificationsState,
} from '@/app/types/notifications';
export const NOTIFICATIONS_MUTATIONS = { export const NOTIFICATIONS_MUTATIONS = {
SET_NOTIFICATIONS: 'SET_NOTIFICATIONS', SET_NOTIFICATIONS: 'SET_NOTIFICATIONS',
@ -24,22 +20,22 @@ export const NOTIFICATIONS_ACTIONS = {
/** /**
* creates notifications module with all dependencies * creates notifications module with all dependencies
* *
* @param api - payments api * @param service - payments service
*/ */
export function makeNotificationsModule(api: NotificationsApi) { export function newNotificationsModule(service: NotificationsService) {
return { return {
state: new NotificationsState(), state: new NotificationsState(),
mutations: { mutations: {
[NOTIFICATIONS_MUTATIONS.SET_NOTIFICATIONS](state: NotificationsState, notificationsResponse: NotificationsState): void { [NOTIFICATIONS_MUTATIONS.SET_NOTIFICATIONS](state: NotificationsState, notificationsState: NotificationsState): void {
state.notifications = notificationsResponse.notifications; state.notifications = notificationsState.notifications;
state.pageCount = notificationsResponse.pageCount; state.pageCount = notificationsState.pageCount;
state.unreadCount = notificationsResponse.unreadCount; state.unreadCount = notificationsState.unreadCount;
}, },
[NOTIFICATIONS_MUTATIONS.SET_LATEST](state: NotificationsState, notificationsResponse: NotificationsState): void { [NOTIFICATIONS_MUTATIONS.SET_LATEST](state: NotificationsState, notificationsState: NotificationsState): void {
state.latestNotifications = notificationsResponse.notifications; state.latestNotifications = notificationsState.notifications;
}, },
[NOTIFICATIONS_MUTATIONS.MARK_AS_READ](state: NotificationsState, id: string): void { [NOTIFICATIONS_MUTATIONS.MARK_AS_READ](state: NotificationsState, id: string): void {
state.notifications = state.notifications.map((notification: Notification) => { state.notifications = state.notifications.map((notification: UINotification) => {
if (notification.id === id) { if (notification.id === id) {
notification.markAsRead(); notification.markAsRead();
} }
@ -48,7 +44,7 @@ export function makeNotificationsModule(api: NotificationsApi) {
}); });
}, },
[NOTIFICATIONS_MUTATIONS.READ_ALL](state: NotificationsState): void { [NOTIFICATIONS_MUTATIONS.READ_ALL](state: NotificationsState): void {
state.notifications = state.notifications.map((notification: Notification) => { state.notifications = state.notifications.map((notification: UINotification) => {
notification.markAsRead(); notification.markAsRead();
return notification; return notification;
@ -58,24 +54,26 @@ export function makeNotificationsModule(api: NotificationsApi) {
}, },
}, },
actions: { actions: {
[NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS]: async function ({commit}: any, cursor: NotificationsCursor): Promise<NotificationsState> { [NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS]: async function ({commit}: any, pageIndex: number): Promise<void> {
const notificationsResponse = await api.get(cursor); const notificationsResponse = await service.notifications(pageIndex);
commit(NOTIFICATIONS_MUTATIONS.SET_NOTIFICATIONS, notificationsResponse); const notifications = notificationsResponse.page.notifications.map(notification => new UINotification(notification));
if (cursor.page === 1) { const notificationState = new NotificationsState(notifications, notificationsResponse.page.pageCount, notificationsResponse.unreadCount);
commit(NOTIFICATIONS_MUTATIONS.SET_LATEST, notificationsResponse);
commit(NOTIFICATIONS_MUTATIONS.SET_NOTIFICATIONS, notificationState);
if (pageIndex === 1) {
commit(NOTIFICATIONS_MUTATIONS.SET_LATEST, notificationState);
} }
return notificationsResponse;
}, },
[NOTIFICATIONS_ACTIONS.MARK_AS_READ]: async function ({commit}: any, id: string): Promise<any> { [NOTIFICATIONS_ACTIONS.MARK_AS_READ]: async function ({commit}: any, id: string): Promise<void> {
await api.read(id); await service.readSingeNotification(id);
commit(NOTIFICATIONS_MUTATIONS.MARK_AS_READ, id); commit(NOTIFICATIONS_MUTATIONS.MARK_AS_READ, id);
}, },
[NOTIFICATIONS_ACTIONS.READ_ALL]: async function ({commit}: any): Promise<any> { [NOTIFICATIONS_ACTIONS.READ_ALL]: async function ({commit}: any): Promise<void> {
await api.readAll(); await service.readAllNotifications();
commit(NOTIFICATIONS_MUTATIONS.READ_ALL); commit(NOTIFICATIONS_MUTATIONS.READ_ALL);
}, },

93
web/storagenode/src/app/types/notifications.ts
View File

@ -2,23 +2,39 @@
// See LICENSE for copying information. // See LICENSE for copying information.
import { NotificationIcon } from '@/app/utils/notificationIcons'; import { NotificationIcon } from '@/app/utils/notificationIcons';
import { Notification, NotificationTypes } from '@/storagenode/notifications/notifications';
/**
* Holds all notifications module state.
*/
export class NotificationsState {
public latestNotifications: UINotification[] = [];
public constructor(
public notifications: UINotification[] = [],
public pageCount: number = 0,
public unreadCount: number = 0,
) { }
}
/** /**
* Describes notification entity. * Describes notification entity.
*/ */
export class Notification { export class UINotification {
public icon: NotificationIcon; public icon: NotificationIcon;
public isRead: boolean;
public id: string;
public senderId: string;
public type: NotificationTypes;
public title: string;
public message: string;
public readAt: Date | null;
public createdAt: Date;
public constructor( public constructor(notification: Partial<UINotification> = new Notification()) {
public id: string = '', Object.assign(this, notification);
public senderId: string = '',
public type: NotificationTypes = NotificationTypes.Custom,
public title: string = '',
public message: string = '',
public isRead: boolean = false,
public createdAt: Date = new Date(),
) {
this.setIcon(); this.setIcon();
this.isRead = !!this.readAt;
} }
/** /**
@ -70,60 +86,3 @@ export class Notification {
} }
} }
} }
/**
* Describes all current notifications types.
*/
export enum NotificationTypes {
Custom = 0,
AuditCheckFailure = 1,
UptimeCheckFailure = 2,
Disqualification = 3,
Suspension = 4,
}
/**
* Describes page offset for pagination.
*/
export class NotificationsCursor {
public constructor(
public page: number = 0,
public limit: number = 7,
) { }
}
/**
* Holds all notifications module state.
*/
export class NotificationsState {
public latestNotifications: Notification[] = [];
public constructor(
public notifications: Notification[] = [],
public pageCount: number = 0,
public unreadCount: number = 0,
) { }
}
/**
* Exposes all notifications-related functionality.
*/
export interface NotificationsApi {
/**
* Fetches notifications.
* @throws Error
*/
get(cursor: NotificationsCursor): Promise<NotificationsState>;
/**
* Marks single notification as read.
* @throws Error
*/
read(id: string): Promise<void>;
/**
* Marks all notification as read.
* @throws Error
*/
readAll(): Promise<void>;
}

3
web/storagenode/src/app/views/DashboardArea.vue
View File

@ -20,7 +20,6 @@ import { APPSTATE_ACTIONS } from '@/app/store/modules/appState';
import { NODE_ACTIONS } from '@/app/store/modules/node'; import { NODE_ACTIONS } from '@/app/store/modules/node';
import { NOTIFICATIONS_ACTIONS } from '@/app/store/modules/notifications'; import { NOTIFICATIONS_ACTIONS } from '@/app/store/modules/notifications';
import { PAYOUT_ACTIONS } from '@/app/store/modules/payout'; import { PAYOUT_ACTIONS } from '@/app/store/modules/payout';
import { NotificationsCursor } from '@/app/types/notifications';
@Component ({ @Component ({
components: { components: {
@ -43,7 +42,7 @@ export default class Dashboard extends Vue {
} }
try { try {
await this.$store.dispatch(NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS, new NotificationsCursor(1)); await this.$store.dispatch(NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS, 1);
} catch (error) { } catch (error) {
console.error(error); console.error(error);
} }

6
web/storagenode/src/app/views/NotificationsArea.vue
View File

@ -54,7 +54,7 @@ import VPagination from '@/app/components/VPagination.vue';
import BackArrowIcon from '@/../static/images/notifications/backArrow.svg'; import BackArrowIcon from '@/../static/images/notifications/backArrow.svg';
import { NOTIFICATIONS_ACTIONS } from '@/app/store/modules/notifications'; import { NOTIFICATIONS_ACTIONS } from '@/app/store/modules/notifications';
import { Notification, NotificationsCursor } from '@/app/types/notifications'; import { UINotification } from '@/app/types/notifications';
@Component ({ @Component ({
components: { components: {
@ -67,7 +67,7 @@ export default class NotificationsArea extends Vue {
/** /**
* Returns notification of current page. * Returns notification of current page.
*/ */
public get notifications(): Notification[] { public get notifications(): UINotification[] {
return this.$store.state.notificationsModule.notifications; return this.$store.state.notificationsModule.notifications;
} }
@ -92,7 +92,7 @@ export default class NotificationsArea extends Vue {
*/ */
public async onPageClick(index: number): Promise<void> { public async onPageClick(index: number): Promise<void> {
try { try {
await this.$store.dispatch(NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS, new NotificationsCursor(index)); await this.$store.dispatch(NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS, index);
} catch (error) { } catch (error) {
console.error(error.message); console.error(error.message);
} }

3
web/storagenode/src/app/views/PayoutArea.vue
View File

@ -57,7 +57,6 @@ import { APPSTATE_ACTIONS } from '@/app/store/modules/appState';
import { NODE_ACTIONS } from '@/app/store/modules/node'; import { NODE_ACTIONS } from '@/app/store/modules/node';
import { NOTIFICATIONS_ACTIONS } from '@/app/store/modules/notifications'; import { NOTIFICATIONS_ACTIONS } from '@/app/store/modules/notifications';
import { PAYOUT_ACTIONS } from '@/app/store/modules/payout'; import { PAYOUT_ACTIONS } from '@/app/store/modules/payout';
import { NotificationsCursor } from '@/app/types/notifications';
import { PayoutPeriod, TotalHeldAndPaid } from '@/storagenode/payouts/payouts'; import { PayoutPeriod, TotalHeldAndPaid } from '@/storagenode/payouts/payouts';
@Component ({ @Component ({
@ -88,7 +87,7 @@ export default class PayoutArea extends Vue {
} }
try { try {
await this.$store.dispatch(NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS, new NotificationsCursor(1)); await this.$store.dispatch(NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS, 1);
} catch (error) { } catch (error) {
console.error(error); console.error(error);
} }

35
web/storagenode/src/storagenode/api/notifications.ts
View File

@ -1,7 +1,12 @@
// Copyright (C) 2019 Storj Labs, Inc. // Copyright (C) 2019 Storj Labs, Inc.
// See LICENSE for copying information. // See LICENSE for copying information.
import { Notification, NotificationsApi, NotificationsCursor, NotificationsState } from '@/app/types/notifications'; import {
NotificationsApi,
NotificationsCursor,
NotificationsPage,
NotificationsResponse,
} from '@/storagenode/notifications/notifications';
import { HttpClient } from '@/storagenode/utils/httpClient'; import { HttpClient } from '@/storagenode/utils/httpClient';
/** /**
@ -15,10 +20,10 @@ export class NotificationsHttpApi implements NotificationsApi {
/** /**
* Fetch notifications. * Fetch notifications.
* *
* @returns notifications state * @returns notifications response.
* @throws Error * @throws Error
*/ */
public async get(cursor: NotificationsCursor): Promise<NotificationsState> { public async get(cursor: NotificationsCursor): Promise<NotificationsResponse> {
const path = `${this.ROOT_PATH}/list?page=${cursor.page}&limit=${cursor.limit}`; const path = `${this.ROOT_PATH}/list?page=${cursor.page}&limit=${cursor.limit}`;
const response = await this.client.get(path); const response = await this.client.get(path);
@ -27,28 +32,12 @@ export class NotificationsHttpApi implements NotificationsApi {
} }
const notificationResponse = await response.json(); const notificationResponse = await response.json();
let notifications: Notification[] = [];
let pageCount: number = 0;
let unreadCount: number = 0;
if (notificationResponse) { return new NotificationsResponse(
notifications = notificationResponse.page.notifications.map(item => new NotificationsPage(notificationResponse.page.notifications, notificationResponse.page.pageCount),
new Notification( notificationResponse.unreadCount,
item.id, notificationResponse.totalCount,
item.senderId,
item.type,
item.title,
item.message,
!!item.readAt,
new Date(item.createdAt),
),
); );
pageCount = notificationResponse.page.pageCount;
unreadCount = notificationResponse.unreadCount;
}
return new NotificationsState(notifications, pageCount, unreadCount);
} }
/** /**

88
web/storagenode/src/storagenode/notifications/notifications.ts Normal file
View File

@ -0,0 +1,88 @@
// Copyright (C) 2020 Storj Labs, Inc.
// See LICENSE for copying information.
/**
* Exposes all notifications-related functionality.
*/
export interface NotificationsApi {
/**
* Fetches notifications.
* @throws Error
*/
get(cursor: NotificationsCursor): Promise<NotificationsResponse>;
/**
* Marks single notification as read.
* @throws Error
*/
read(id: string): Promise<void>;
/**
* Marks all notification as read.
* @throws Error
*/
readAll(): Promise<void>;
}
/**
* Describes notification entity.
*/
export class Notification {
public constructor(
public id: string = '',
public senderId: string = '',
public type: NotificationTypes = NotificationTypes.Custom,
public title: string = '',
public message: string = '',
public readAt: Date | null = null,
public createdAt: Date = new Date(),
) {}
}
/**
* Describes all current notifications types.
*/
export enum NotificationTypes {
Custom = 0,
AuditCheckFailure = 1,
UptimeCheckFailure = 2,
Disqualification = 3,
Suspension = 4,
}
/**
* Describes page offset for pagination.
*/
export class NotificationsCursor {
private DEFAULT_LIMIT: number = 7;
public constructor(
public page: number = 0,
public limit: number = 0,
) {
if (!this.limit) {
this.limit = this.DEFAULT_LIMIT;
}
}
}
/**
* Describes response object from server.
*/
export class NotificationsResponse {
public constructor(
public page: NotificationsPage = new NotificationsPage(),
public unreadCount: number = 0,
public totalCount: number = 0,
) {}
}
/**
* Describes page related notification information.
*/
export class NotificationsPage {
public constructor(
public notifications: Notification[] = [],
public pageCount: number = 0,
) {}
}

43
web/storagenode/src/storagenode/notifications/service.ts Normal file
View File

@ -0,0 +1,43 @@
// Copyright (C) 2020 Storj Labs, Inc.
// See LICENSE for copying information.
import { NotificationsApi, NotificationsCursor, NotificationsResponse } from '@/storagenode/notifications/notifications';
/**
* PayoutService is used to store and handle node paystub information.
* PayoutService exposes a business logic related to payouts.
*/
export class NotificationsService {
private readonly api: NotificationsApi;
public constructor(api: NotificationsApi) {
this.api = api;
}
/**
* Fetch notifications.
*
* @returns notifications response.
* @throws Error
*/
public async notifications(index: number, limit?: number): Promise<NotificationsResponse> {
const cursor = new NotificationsCursor(index, limit);
return await this.api.get(cursor);
}
/**
* Marks single notification as read on server.
* @param id
*/
public async readSingeNotification(id: string): Promise<void> {
await this.api.read(id);
}
/**
* Marks all notifications as read on server.
*/
public async readAllNotifications(): Promise<void> {
await this.api.readAll();
}
}

147
web/storagenode/tests/unit/store/notifications.spec.ts Normal file
View File

@ -0,0 +1,147 @@
// Copyright (C) 2020 Storj Labs, Inc.
// See LICENSE for copying information.
import Vuex from 'vuex';
import { newNotificationsModule, NOTIFICATIONS_ACTIONS, NOTIFICATIONS_MUTATIONS } from '@/app/store/modules/notifications';
import { NotificationsState, UINotification } from '@/app/types/notifications';
import { NotificationsHttpApi } from '@/storagenode/api/notifications';
import {
Notification,
NotificationsPage,
NotificationsResponse,
NotificationTypes,
} from '@/storagenode/notifications/notifications';
import { NotificationsService } from '@/storagenode/notifications/service';
import { createLocalVue } from '@vue/test-utils';
const Vue = createLocalVue();
const notificationsApi = new NotificationsHttpApi();
const notificationsService = new NotificationsService(notificationsApi);
const notificationsModule = newNotificationsModule(notificationsService);
Vue.use(Vuex);
const store = new Vuex.Store({ modules: { notificationsModule } });
const state = store.state as any;
let notifications;
describe('mutations', () => {
beforeEach(() => {
createLocalVue().use(Vuex);
notifications = [
new UINotification(new Notification('1', '1', NotificationTypes.Disqualification, 'title1', 'message1', null)),
new UINotification(new Notification('2', '1', NotificationTypes.UptimeCheckFailure, 'title2', 'message2', null)),
];
});
it('sets notification state', (): void => {
const notificationsState = new NotificationsState(notifications, 2, 1);
store.commit(NOTIFICATIONS_MUTATIONS.SET_NOTIFICATIONS, notificationsState);
expect(state.notificationsModule.notifications.length).toBe(notifications.length);
expect(state.notificationsModule.pageCount).toBe(2);
expect(state.notificationsModule.unreadCount).toBe(1);
store.commit(NOTIFICATIONS_MUTATIONS.SET_LATEST, notificationsState);
expect(state.notificationsModule.latestNotifications.length).toBe(notifications.length);
});
it('sets single notification as read', (): void => {
store.commit(NOTIFICATIONS_MUTATIONS.MARK_AS_READ, '1');
const unreadNotificationsCount = state.notificationsModule.notifications.filter(e => !e.isRead).length;
expect(unreadNotificationsCount).toBe(1);
});
it('sets all notification as read', (): void => {
store.commit(NOTIFICATIONS_MUTATIONS.READ_ALL);
const unreadNotificationsCount = state.notificationsModule.notifications.filter(e => !e.isRead).length;
expect(unreadNotificationsCount).toBe(0);
});
});
describe('actions', () => {
beforeEach(() => {
jest.resetAllMocks();
notifications = [
new UINotification(new Notification('1', '1', NotificationTypes.Disqualification, 'title1', 'message1', null)),
new UINotification(new Notification('2', '1', NotificationTypes.UptimeCheckFailure, 'title2', 'message2', null)),
];
});
it('throws error on failed notifications fetch', async (): Promise<void> => {
jest.spyOn(notificationsApi, 'get').mockImplementation(() => { throw new Error(); });
try {
await store.dispatch(NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS, 1);
expect(true).toBe(false);
} catch (error) {
expect(state.notificationsModule.latestNotifications.length).toBe(notifications.length);
}
});
it('success fetches notifications', async (): Promise<void> => {
jest.spyOn(notificationsService, 'notifications')
.mockReturnValue(Promise.resolve(new NotificationsResponse(new NotificationsPage(notifications, 1), 2, 1)));
await store.dispatch(NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS, 1);
expect(state.notificationsModule.latestNotifications.length).toBe(notifications.length);
});
it('throws error on failed single notification read', async (): Promise<void> => {
jest.spyOn(notificationsApi, 'read').mockImplementation(() => { throw new Error(); });
try {
await store.dispatch(NOTIFICATIONS_ACTIONS.MARK_AS_READ, '1');
expect(true).toBe(false);
} catch (error) {
const unreadNotificationsCount = state.notificationsModule.notifications.filter(e => !e.isRead).length;
expect(unreadNotificationsCount).toBe(notifications.length);
}
});
it('success marks single notification as read', async (): Promise<void> => {
jest.spyOn(notificationsService, 'notifications')
.mockReturnValue(Promise.resolve(new NotificationsResponse(new NotificationsPage(notifications, 1), 2, 1)));
await store.dispatch(NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS, 1);
await store.dispatch(NOTIFICATIONS_ACTIONS.MARK_AS_READ, '1');
const unreadNotificationsCount = state.notificationsModule.notifications.filter(e => !e.isRead).length;
expect(unreadNotificationsCount).toBe(1);
});
it('throws error on failed all notifications read', async (): Promise<void> => {
jest.spyOn(notificationsApi, 'readAll').mockImplementation(() => { throw new Error(); });
try {
await store.dispatch(NOTIFICATIONS_ACTIONS.READ_ALL);
expect(true).toBe(false);
} catch (error) {
const unreadNotificationsCount = state.notificationsModule.notifications.filter(e => !e.isRead).length;
expect(unreadNotificationsCount).toBe(1);
}
});
it('success marks all notifications as read', async (): Promise<void> => {
await store.dispatch(NOTIFICATIONS_ACTIONS.READ_ALL);
const unreadNotificationsCount = state.notificationsModule.notifications.filter(e => !e.isRead).length;
expect(unreadNotificationsCount).toBe(0);
});
});