JakeHillion/storj
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge 'master' branch

Browse Source 
Change-Id: I6070089128a150a4dd501bbc62a1f8b394aa643e
This commit is contained in:
Michal Niewrzal 2020-11-10 12:56:30 +01:00
parent 92f9251074
commit 7dde184cb5
101 changed files with 2948 additions and 1177 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
.clabot
View File

@ -1,4 +1,5 @@
{
"message": "Thank you for your pull request and welcome to our community. We require contributors to sign our [Contributor License Agreement](https://docs.google.com/forms/d/e/1FAIpQLSdVzD5W8rx-J_jLaPuG31nbOzS8yhNIIu4yHvzonji6NeZ4ig/viewform), and we don't seem to have the users {{usersWithoutCLA}} on file. Once you have signed the CLA, please let us know, so we can manually review and add you to the approved contributors list.",
"contributors": [
"aleitner",
"aligeti",
@ -62,6 +63,7 @@
"montyanderson",
"sixcorners",
"alexottoboni",
"dominickmarino"
"dominickmarino",
"hectorj2f"
]
}

4
docs/blueprints/access-revocation.md
View File

@ -39,7 +39,7 @@ presents challenges when revoking a macaroon.
For example, if I hold the API key for Project A, I can create Macaroon A with a
caveat that it can only read and write files within Bucket A. I can
then share this macaroon with my own customer, Customer A. Customer A may then,
if they wish, create Macaroon B which is futher caveated -- for example,
if they wish, create Macaroon B which is further caveated -- for example,
restricted to read-only access in Bucket A -- and share Macaroon B with someone
else. This can occur without my knowledge.
@ -94,7 +94,7 @@ This approach was deemed best because it:
- Creates very little load on the database.
- Is backwards compatible, and allows us to revoke existing macaroons.
- Allows us to revoke an entire "macaroon tree" while maintaining the
distributive properies of macaroons.
distributive properties of macaroons.
Disadvantages to this approach:

2
docs/blueprints/cache-node-selection.md
View File

@ -30,7 +30,7 @@ When selecting nodes to store files to, the following criteria must be met:
- the node is not disqualified
- the node is not suspended
- the node has not exited
- the node has sufficent free disk space
- the node has sufficient free disk space
- the node has been contacted recently
- the node has participated in a sufficient number of audit
- the nodes has sufficient uptime counts

2
docs/blueprints/linux-setup.md
View File

@ -79,7 +79,7 @@ The process for building a package is as follows:
- make a source package
- compile it to get binary packages.
Only the binary package is used by the user for installation. It is not a recommended pratice to directly integrate binaries.
Only the binary package is used by the user for installation. It is not a recommended practice to directly integrate binaries.
Building the source package is the most difficult part. But once it is done, we can use tools such as [fpm](https://github.com/jordansissel/fpm/wiki) to convert it to other package formats.

2
docs/blueprints/password-key-derivation.md
View File

@ -15,7 +15,7 @@ That leaves open the question for how a root key is created. Some requirements o
These requirements allow users to be in full control of their encryption, and don't require users safely transporting high entropy (hard to remember) secrets to bootstrap new uplinks.
This design accomodates more requirements that allow for additional features:
This design accommodates more requirements that allow for additional features:
3. A root key can be created for any encrypted path in a bucket, not just the bucket.
4. A table of root keys for low entropy passwords should not be possible. In other words, an attacker with knowledge of the algorithm should not be able to use a dictionary of common passwords and pre-compute what keys to check in the event of a data breach.

2
docs/blueprints/repair-with-hashes.md
View File

@ -41,7 +41,7 @@ Downloading for repair is significantly different enough from streaming as to wa
Using only the minimum number of pieces means that Reed-Solomon does not act as a check during repair. Hence hashing is used instead. While an uplink could potentially send signed bogus data to a storage node, the storage node would not be penalized by these actions. This requires that Audit implements a similar piece hash check instead of relying solely on Reed-Solomon encoding.
The size of all piece hashes downloaded should be roughly equal to a default maximum segment size : 64MiB. It seems preferable to keep this in memory over dealing with persistance to disk.
The size of all piece hashes downloaded should be roughly equal to a default maximum segment size : 64MiB. It seems preferable to keep this in memory over dealing with persistence to disk.
## Implementation

6
docs/blueprints/satellite-billing-system.md
View File

@ -51,7 +51,7 @@ type Accounts interface {
```
# Customer setup
Every satellite user has a corresponding customer entity on stripe which holds credit cards, balance which reflects the ammount of STORJ tokens, and is used for invoicing. Every time a user visits billing page on the satellite UI we try to create a customer for him if one doesn't exists.
Every satellite user has a corresponding customer entity on stripe which holds credit cards, balance which reflects the amount of STORJ tokens, and is used for invoicing. Every time a user visits billing page on the satellite UI we try to create a customer for him if one doesn't exists.
```go
// Setup creates a payment account for the user.
// If account is already set up it will return nil.
@ -257,7 +257,7 @@ type StorjTokens interface {
```
# Making a deposit
STORJ cryptocurrency processing is done via coinpayments API. Every time a user wants to deposit some amount of STORJ token to his account balacne, new coinpayments transaction is created. Transaction amount is set in USD, and conversion rates is beeing locked(saved) after transaction is created and stored in the db.
STORJ cryptocurrency processing is done via coinpayments API. Every time a user wants to deposit some amount of STORJ token to his account balacne, new coinpayments transaction is created. Transaction amount is set in USD, and conversion rates is being locked(saved) after transaction is created and stored in the db.
```go
// Deposit creates new deposit transaction with the given amount returning
// ETH wallet address where funds should be sent. There is one
@ -403,7 +403,7 @@ func (tokens *storjTokens) ListTransactionInfos(ctx context.Context, userID uuid
```
# Transaction update cycle
There is a cycle that iterates over all `pending`(`pending` and `paid` statuses of coinpayments transaction respectively) transactions, list it's infos and updates tx status and received amount. If updated is status is set to `cancelled` or `completed`, that transactions won't take part in the next update cycle. When there is a status transation to `completed` along with the update `apply_balance_transaction_intent` is created. Transaction with status `completed` and present `apply_balance_transaction_intent` with state `unapplied` defines as `UnappliedTransaction` which is later processed in update balance cycle. If the received amount is greater that 50$ a promotional coupon for 55$ is created.
There is a cycle that iterates over all `pending`(`pending` and `paid` statuses of coinpayments transaction respectively) transactions, list it's infos and updates tx status and received amount. If updated is status is set to `cancelled` or `completed`, that transactions won't take part in the next update cycle. When there is a status transaction to `completed` along with the update `apply_balance_transaction_intent` is created. Transaction with status `completed` and present `apply_balance_transaction_intent` with state `unapplied` defines as `UnappliedTransaction` which is later processed in update balance cycle. If the received amount is greater that 50$ a promotional coupon for 55$ is created.
```go
// updateTransactions updates statuses and received amount for given transactions.
func (service *Service) updateTransactions(ctx context.Context, ids TransactionAndUserList) (err error) {

4
docs/blueprints/storage-node-satellite-selection.md
View File

@ -274,7 +274,7 @@ and are left with the following:
The list of trusted Satellite URLs should be recalculated daily (with some jitter).
### Backwards Compatability
### Backwards Compatibility
The old piecestore configuration (i.e. `piecestore.OldConfig`) currently contains a
comma separated list of trusted Satellite URLs (`WhitelistedSatellites`). It
@ -296,7 +296,7 @@ a fixed set of trusted Satellite URLs.
* Implement a `trust.ListConfig` configuration struct which:
* Contains the list of entries (with a release default of a single list containing `https://www.tardigrade.io/trusted-satellites`)
* Contains a refresh interval
* Maintains backwards compatability with `WhitelistedSatellites` in `piecestore.OldConfig`
* Maintains backwards compatibility with `WhitelistedSatellites` in `piecestore.OldConfig`
* Implement `storj.io/storj/storagenode/trust.List` that:
* Consumes `trust.ListConfig` for configuration
* Performs the initial fetching and building of trusted Satellite URLs

4
docs/blueprints/storagenode-graceful-exit/protocol.md
View File

@ -8,7 +8,7 @@ This document describes how storage node transfers its pieces during Graceful Ex
## Background
During Graceful Exit a storage node needs to transfer pieces to other nodes. During transfering the storage node or satellite may crash, hence it needs to be able to continue after a restart.
During Graceful Exit a storage node needs to transfer pieces to other nodes. During transferring the storage node or satellite may crash, hence it needs to be able to continue after a restart.
Satellite gathers transferred pieces list asynchronously, which is described in [Gathering Pieces Document](pieces.md). This may consume a significant amount of time.
@ -28,7 +28,7 @@ The `worker` should continue to poll the satellite at a configurable interval un
The satellite should return pieces to transfer from the transfer queue if piece durability <= optimal. If durability > optimal, we remove the exiting node from the segment / pointer.
The storage node should concurrently transfer pieces returned by the satellite. The storage node should send a `TransferSucceeded` message as pieces are successfuly transfered. The Storage node should send a `TransferFailed`, with reason, on failure.
The storage node should concurrently transfer pieces returned by the satellite. The storage node should send a `TransferSucceeded` message as pieces are successfully transferred. The Storage node should send a `TransferFailed`, with reason, on failure.
The satellites should set the `finished_at` on success, and respond with a `DeletePiece` message. Otherwise increment `failed_count` and set the `last_failed_at` and `last_failed_code` for reprocessing.

7
go.mod
View File

@ -38,14 +38,13 @@ require (
go.etcd.io/bbolt v1.3.5
go.uber.org/zap v1.16.0
golang.org/x/crypto v0.0.0-20200820211705-5c72a883971a
golang.org/x/net v0.0.0-20200226121028-0de0cce0169b // indirect
golang.org/x/sync v0.0.0-20200625203802-6e8e738ad208
golang.org/x/sys v0.0.0-20200929083018-4d22bbb62b3c
golang.org/x/time v0.0.0-20200630173020-3af7569d3a1e
google.golang.org/api v0.20.0 // indirect
storj.io/common v0.0.0-20201030120157-90ae6720d87e
storj.io/drpc v0.0.14
storj.io/common v0.0.0-20201106104920-372a344bdd45
storj.io/drpc v0.0.16
storj.io/monkit-jaeger v0.0.0-20200518165323-80778fc3f91b
storj.io/private v0.0.0-20201026143115-bc926bfa3bca
storj.io/uplink v1.3.2-0.20201028181609-f6efc8fcf771
storj.io/uplink v1.3.2-0.20201106105834-ec8e5cc29f7e
)

154
go.sum
View File

@ -1,5 +1,7 @@
cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
cloud.google.com/go v0.31.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
cloud.google.com/go v0.34.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
cloud.google.com/go v0.37.0/go.mod h1:TS1dMSSfndXH133OKGwekG838Om/cQT0BUHV3HcBgoo=
cloud.google.com/go v0.37.4/go.mod h1:NHPJ89PdicEuT9hdPXMROBD91xc5uRDxsMtSB16k7hw=
cloud.google.com/go v0.38.0/go.mod h1:990N+gfupTy94rShfmMCWGDn0LpTmnzTp2qbd1dvSRU=
cloud.google.com/go v0.44.1/go.mod h1:iSa0KzasP4Uvy3f1mN/7PiObzGgflwredwwASm/v6AU=
@ -17,7 +19,12 @@ cloud.google.com/go/pubsub v1.0.1 h1:W9tAK3E57P75u0XLLR82LZyw8VpAnhmyTOxW9qzmyj8
cloud.google.com/go/pubsub v1.0.1/go.mod h1:R0Gpsv3s54REJCy4fxDixWD93lHJMoZTyQ2kNxGRt3I=
cloud.google.com/go/storage v1.0.0 h1:VV2nUM3wwLLGh9lSABFgZMjInyUbJeaRSE64WuAIQ+4=
cloud.google.com/go/storage v1.0.0/go.mod h1:IhtSnM/ZTZV8YYJWCY8RULGVqBDmpoyjwiyrjsg+URw=
dmitri.shuralyov.com/app/changes v0.0.0-20180602232624-0a106ad413e3/go.mod h1:Yl+fi1br7+Rr3LqpNJf1/uxUdtRUV+Tnj0o93V2B9MU=
dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU=
dmitri.shuralyov.com/html/belt v0.0.0-20180602232347-f7d459c86be0/go.mod h1:JLBrvjyP0v+ecvNYvCpyZgu5/xkfAUhi6wJj28eUfSU=
dmitri.shuralyov.com/service/change v0.0.0-20181023043359-a85b471d5412/go.mod h1:a1inKt/atXimZ4Mv927x+r7UpyzRUf4emIoiiSC2TN4=
dmitri.shuralyov.com/state v0.0.0-20180228185332-28bcc343414c/go.mod h1:0PRwlb0D6DFvNNtx+9ybjezNCa8XF0xaYcETyp6rHWU=
git.apache.org/thrift.git v0.0.0-20180902110319-2566ecd5d999/go.mod h1:fPE2ZNJGynbRyZ4dJvy6G277gSllfV2HJqblrnkyeyg=
github.com/Azure/go-ansiterm v0.0.0-20170929234023-d6e3b3328b78 h1:w+iIsaOQNcT7OZ575w+acHgRric5iCyQh+xv+KJ4HB8=
github.com/Azure/go-ansiterm v0.0.0-20170929234023-d6e3b3328b78/go.mod h1:LmzpDX56iTiv29bbRTIsUNlaFfuhWRQBWjQdVyAevI8=
github.com/BurntSushi/toml v0.3.1 h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ=
@ -40,6 +47,7 @@ github.com/alicebob/gopher-json v0.0.0-20200520072559-a9ecdc9d1d3a h1:HbKu58rmZp
github.com/alicebob/gopher-json v0.0.0-20200520072559-a9ecdc9d1d3a/go.mod h1:SGnFV6hVsYE877CKEZ6tDNTjaSXYUk6QqoIK6PrAtcc=
github.com/alicebob/miniredis/v2 v2.13.3 h1:kohgdtN58KW/r9ZDVmMJE3MrfbumwsDQStd0LPAGmmw=
github.com/alicebob/miniredis/v2 v2.13.3/go.mod h1:uS970Sw5Gs9/iK3yBg0l9Uj9s25wXxSpQUE9EaJ/Blg=
github.com/anmitsu/go-shlex v0.0.0-20161002113705-648efa622239/go.mod h1:2FmKhYUyUczH0OGQWaF5ceTx0UBShxjsH6f8oGKYe2c=
github.com/apache/thrift v0.12.0 h1:pODnxUFNcjP9UTLZGTdeh+j16A8lJbRvD3rOtrk/7bs=
github.com/apache/thrift v0.12.0/go.mod h1:cp2SuWMxlEZw2r+iP2GNCdIi4C1qmUzdZFSVb+bacwQ=
github.com/armon/circbuf v0.0.0-20150827004946-bbbad097214e/go.mod h1:3U/XgcO3hCbHZ8TKRvWD2dDTCfh9M9ya+I9JpbB7O8o=
@ -55,6 +63,7 @@ github.com/bketelsen/crypt v0.0.3-0.20200106085610-5cbc8cc4026c/go.mod h1:MKsuJm
github.com/blang/semver v3.5.1+incompatible h1:cQNTCjp13qL8KC3Nbxr/y2Bqb63oX6wdnnjpJbkM4JQ=
github.com/blang/semver v3.5.1+incompatible/go.mod h1:kRBLl5iJ+tD4TcOOxsy/0fnwebNt5EWlYSAyrTnjyyk=
github.com/bmizerany/assert v0.0.0-20160611221934-b7ed37b82869/go.mod h1:Ekp36dRnpXw/yCqJaO+ZrUyxD+3VXMFFr56k5XYrpB4=
github.com/bradfitz/go-smtpd v0.0.0-20170404230938-deb6d6237625/go.mod h1:HYsPBTaaSFSlLx/70C2HPIMNZpVV8+vt/A+FMnYP11g=
github.com/btcsuite/btcd v0.20.1-beta/go.mod h1:wVuoA8VJLEcwgqHBwHmzLRazpKxTv13Px/pDuV7OomQ=
github.com/btcsuite/btclog v0.0.0-20170628155309-84c8d2346e9f/go.mod h1:TdznJufoqS23FtqVCzL0ZqgP5MqXbb4fg/WgDys70nA=
github.com/btcsuite/btcutil v0.0.0-20190425235716-9e5f4b9a998d/go.mod h1:+5NJ2+qvTyV9exUAL/rxXi3DcLg2Ts+ymUAY5y4NvMg=
@ -66,12 +75,15 @@ github.com/btcsuite/goleveldb v0.0.0-20160330041536-7834afc9e8cd/go.mod h1:F+uVa
github.com/btcsuite/snappy-go v0.0.0-20151229074030-0bdef8d06723/go.mod h1:8woku9dyThutzjeg+3xrA5iCpBRH8XEEg3lh6TiUghc=
github.com/btcsuite/websocket v0.0.0-20150119174127-31079b680792/go.mod h1:ghJtEyQwv5/p4Mg4C0fgbePVuGr935/5ddU9Z3TmDRY=
github.com/btcsuite/winsvc v1.0.0/go.mod h1:jsenWakMcC0zFBFurPLEAyrnc/teJEM1O46fmI40EZs=
github.com/buger/jsonparser v0.0.0-20181115193947-bf1c66bbce23/go.mod h1:bbYlZJ7hK1yFx9hf58LP0zeX7UjIGs20ufpu3evjr+s=
github.com/calebcase/tmpfile v1.0.1/go.mod h1:iErLeG/iqJr8LaQ/gYRv4GXdqssi3jg4iSzvrA06/lw=
github.com/calebcase/tmpfile v1.0.2-0.20200602150926-3af473ef8439/go.mod h1:iErLeG/iqJr8LaQ/gYRv4GXdqssi3jg4iSzvrA06/lw=
github.com/calebcase/tmpfile v1.0.2 h1:1AGuhKiUu4J6wxz6lxuF6ck3f8G2kaV6KSEny0RGCig=
github.com/calebcase/tmpfile v1.0.2/go.mod h1:iErLeG/iqJr8LaQ/gYRv4GXdqssi3jg4iSzvrA06/lw=
github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc=
github.com/cheekybits/genny v1.0.0 h1:uGGa4nei+j20rOSeDeP5Of12XVm7TGUd4dJA9RDitfE=
github.com/cheekybits/genny v1.0.0/go.mod h1:+tQajlRqAUrPI7DOSpB0XAqZYtQakVtB7wXkRAgjxjQ=
github.com/cheggaaa/pb/v3 v3.0.5 h1:lmZOti7CraK9RSjzExsY53+WWfub9Qv13B5m4ptEoPE=
github.com/cheggaaa/pb/v3 v3.0.5/go.mod h1:X1L61/+36nz9bjIsrDU52qHKOQukUQe2Ge+YvGuquCw=
github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI=
@ -90,6 +102,7 @@ github.com/coreos/etcd v3.3.10+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc
github.com/coreos/etcd v3.3.13+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE=
github.com/coreos/go-semver v0.2.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk=
github.com/coreos/go-semver v0.3.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk=
github.com/coreos/go-systemd v0.0.0-20181012123002-c6f51f82210d/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4=
github.com/coreos/go-systemd v0.0.0-20190321100706-95778dfbb74e/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4=
github.com/coreos/go-systemd v0.0.0-20190719114852-fd7a80b32e1f/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4=
github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f/go.mod h1:E3G3o1h8I7cfcXa63jLwjI0eiQQMgzzUDFVpN/nH/eA=
@ -123,6 +136,7 @@ github.com/docker/go-connections v0.4.0 h1:El9xVISelRB7BuFusrZozjnkIM5YnzCViNKoh
github.com/docker/go-connections v0.4.0/go.mod h1:Gbd7IOopHjR8Iph03tsViu4nIes5XhDvyHbTtUxmeec=
github.com/docker/go-units v0.3.3 h1:Xk8S3Xj5sLGlG5g67hJmYMmUgXv5N4PhkjJHHqrwnTk=
github.com/docker/go-units v0.3.3/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk=
github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk=
github.com/eapache/go-resiliency v1.1.0/go.mod h1:kFI+JgMyC7bLPUVY133qvEBtVayf5mFgVsvEsIPBvNs=
github.com/eapache/go-xerial-snappy v0.0.0-20180814174437-776d5712da21/go.mod h1:+020luEh2TKB4/GOp8oxxtq0Daoen/Cii55CzbTV6DU=
github.com/eapache/queue v1.1.0/go.mod h1:6eCeP0CKFpHLu8blIFXhExK/dRa7WDZfr6jVFPTqq+I=
@ -132,10 +146,16 @@ github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7
github.com/fatih/color v1.7.0/go.mod h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5KwzbycvMj4=
github.com/fatih/color v1.9.0 h1:8xPHl4/q1VyqGIPif1F+1V3Y3lSmrq01EabUW3CoW5s=
github.com/fatih/color v1.9.0/go.mod h1:eQcE1qtQxscV5RaZvpXrrb8Drkc3/DdQ+uUYCNjL+zU=
github.com/flynn/go-shlex v0.0.0-20150515145356-3f9db97f8568/go.mod h1:xEzjJPgXI435gkrCt3MPfRiAkVrwSbHsst4LCFVfpJc=
github.com/francoispqt/gojay v1.2.13/go.mod h1:ehT5mTG4ua4581f1++1WLG0vPdaA9HaiDsoyrBGkyDY=
github.com/fsnotify/fsnotify v1.4.7 h1:IXs+QLmnXW2CcXuY+8Mzv/fWEsPGWxqefPtCP5CnV9I=
github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
github.com/fsnotify/fsnotify v1.4.9 h1:hsms1Qyu0jgnwNXIxa+/V/PDsU6CfLf6CNO8H7IWoS4=
github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ=
github.com/fsouza/fake-gcs-server v1.7.0/go.mod h1:5XIRs4YvwNbNoz+1JF8j6KLAyDh7RHGAyAK3EP2EsNk=
github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
github.com/gliderlabs/ssh v0.1.1/go.mod h1:U7qILu1NlMHj9FlMhZLlkCdDnU1DBEAqr0aevW3Awn0=
github.com/go-errors/errors v1.0.1/go.mod h1:f4zRHt4oKfwPJE5k8C9vpYG+aDHdBFUsgrm6/TyX73Q=
github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9AVAgeJqvqgH9Q5CA+iKCZ2gyEVpxRU=
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
@ -159,16 +179,28 @@ github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b h1:VKtxabqXZkF25pY9ekf
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
github.com/golang/groupcache v0.0.0-20190129154638-5b532d6fd5ef/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/groupcache v0.0.0-20191027212112-611e8accdfc9/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7 h1:5ZkaAPbicIKTF2I64qf5Fh8Aa83Q/dnOafMYV0OMwjA=
github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/lint v0.0.0-20180702182130-06c8688daad7/go.mod h1:tluoj9z5200jBnyusfRPU2LqT6J+DAorxEvtC7LHB+E=
github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
github.com/golang/mock v1.2.0/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
github.com/golang/mock v1.3.1 h1:qGJ6qTW+x6xX/my+8YUVl4WNpX9B7+/l2tRsHGZ7f2s=
github.com/golang/mock v1.3.1/go.mod h1:sBzyDLLjw3U8JLTeZvSv8jJB+tU5PVekmnlKIyFUx0Y=
github.com/golang/mock v1.4.0/go.mod h1:UOMv5ysSaYNkG+OFQykRIcU/QvvxJf3p21QfJ2Bt3cw=
github.com/golang/mock v1.4.4 h1:l75CXGRSwbaYNpl/Z2X1XIIAMSCquvXgpVZDhwEIJsc=
github.com/golang/mock v1.4.4/go.mod h1:l3mdAwkq5BuhzHwde/uurv3sEJeZMXNpwsxVWU71h+4=
github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.2 h1:6nsPYzhq5kReh6QImI3k5qWzO4PEbvbIW2cwSfR/6xs=
github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.4.0-rc.1/go.mod h1:ceaxUfeHdC40wWswd/P6IGgMaK3YpKi5j83Wpe3EHw8=
github.com/golang/protobuf v1.4.0-rc.1.0.20200221234624-67d41d38c208/go.mod h1:xKAWHe0F5eneWXFV3EuXVDTCmh+JuBKY0li0aMyXATA=
github.com/golang/protobuf v1.4.0-rc.2/go.mod h1:LlEzMj4AhA7rCAGe4KMBDvJI+AwstrUpVNzEA03Pprs=
github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0/go.mod h1:WU3c8KckQ9AFe+yFwt9sWVRKCVIyN9cPHBJSNnbL67w=
github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvqG2KuDX0=
github.com/golang/protobuf v1.4.2 h1:+Z5KGCizgyZCbGh1KZqA0fcLLkwbsjIzS4aV2v7wJX0=
github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI=
github.com/golang/snappy v0.0.0-20170215233205-553a64147049/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
github.com/golang/snappy v0.0.1/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
@ -176,6 +208,7 @@ github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c/go.mod h1:lNA+9X1NB3Z
github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M=
github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.2 h1:X2ev0eStA3AbceY54o37/0PQ/UWqKEiiO2dKL5OPaFM=
github.com/google/go-cmp v0.5.2/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
@ -189,6 +222,9 @@ github.com/google/pprof v0.0.0-20191218002539-d4f498aebedc/go.mod h1:ZgVRPoUq/hf
github.com/google/pprof v0.0.0-20200229191704-1ebb73c60ed3 h1:SRgJV+IoxM5MKyFdlSUeNy6/ycRUF2yBAKdAQswoHUk=
github.com/google/pprof v0.0.0-20200229191704-1ebb73c60ed3/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI=
github.com/googleapis/gax-go v2.0.0+incompatible h1:j0GKcs05QVmm7yesiZq2+9cxHkNK9YM6zKx4D2qucQU=
github.com/googleapis/gax-go v2.0.0+incompatible/go.mod h1:SFVmujtThgffbyetf+mdk2eWhX2bMyUtNHzFKcPA9HY=
github.com/googleapis/gax-go/v2 v2.0.3/go.mod h1:LLvjysVCY1JZeum8Z6l8qUty8fiNwE08qbEPm1M08qg=
github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg=
github.com/googleapis/gax-go/v2 v2.0.5 h1:sjZBwGj9Jlw33ImPtvFviGYvseOtDM7hkSKB7+Tv3SM=
github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk=
@ -205,8 +241,10 @@ github.com/gorilla/websocket v1.4.0/go.mod h1:E7qHFY5m1UJ88s3WnNqhKjPHQ0heANvMoA
github.com/gorilla/websocket v1.4.2/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
github.com/graphql-go/graphql v0.7.9 h1:5Va/Rt4l5g3YjwDnid3vFfn43faaQBq7rMcIZ0VnV34=
github.com/graphql-go/graphql v0.7.9/go.mod h1:k6yrAYQaSP59DC5UVxbgxESlmVyojThKdORUqGDGmrI=
github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7/go.mod h1:FecbI9+v66THATjSRHfNgh1IVFe/9kFxbXtjV0ctIMA=
github.com/grpc-ecosystem/go-grpc-middleware v1.0.0/go.mod h1:FiyG127CGDf3tlThmgyCl78X/SZQqEOJBCDaAfeWzPs=
github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0/go.mod h1:8NvIoxWQoOIhqOTXgfV/d3M/q6VIi02HzZEHgUlZvzk=
github.com/grpc-ecosystem/grpc-gateway v1.5.0/go.mod h1:RSKVYQBd5MCa4OVpNdGskqpgL2+G+NZTnrVHpWWfpdw=
github.com/grpc-ecosystem/grpc-gateway v1.9.0/go.mod h1:vNeuVxBJEsws4ogUvrchl83t/GYV9WGTSLVdBhOQFDY=
github.com/hailocab/go-hostpool v0.0.0-20160125115350-e80d13ce29ed/go.mod h1:tMWxXQ9wFIaZeTI9F+hmhFiGpFmhOHzyShyFUhRm0H4=
github.com/hashicorp/consul/api v1.1.0/go.mod h1:VmuI/Lkw1nC05EYQWNKwWGbkg+FbDBtguAZLlVdkD9Q=
@ -294,6 +332,7 @@ github.com/jackc/puddle v0.0.0-20190608224051-11cab39313c9/go.mod h1:m4B5Dj62Y0f
github.com/jackc/puddle v1.1.0/go.mod h1:m4B5Dj62Y0fbyuIc15OsIqK0+JU8nkqQjsgx7dvjSWk=
github.com/jackc/puddle v1.1.1/go.mod h1:m4B5Dj62Y0fbyuIc15OsIqK0+JU8nkqQjsgx7dvjSWk=
github.com/jackc/puddle v1.1.2/go.mod h1:m4B5Dj62Y0fbyuIc15OsIqK0+JU8nkqQjsgx7dvjSWk=
github.com/jellevandenhooff/dkim v0.0.0-20150330215556-f50fe3d243e1/go.mod h1:E0B/fFc00Y+Rasa88328GlI/XbtyysCtTHZS8h7IrBU=
github.com/jessevdk/go-flags v0.0.0-20141203071132-1679536dcc89/go.mod h1:4FA24M0QyGHXBuZZK/XkWh8h0e1EYbRYJSGM75WSRxI=
github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af/go.mod h1:Nht3zPeWKUH0NzdCt2Blrr5ys8VGpn0CEB0cQHVjt7k=
github.com/jonboulle/clockwork v0.1.0/go.mod h1:Ii8DK3G1RaLaWxj9trq07+26W01tbo22gdxWY5EU2bo=
@ -321,6 +360,7 @@ github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515/go.mod h1:+0opPa2QZZtGFB
github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI=
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
github.com/kr/pty v1.1.3/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
github.com/kr/pty v1.1.8/go.mod h1:O1sed60cT9XZ5uDucP5qwvh+TE3NnUj51EiZO/lmSfw=
github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
@ -330,9 +370,18 @@ github.com/lib/pq v1.1.0/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo=
github.com/lib/pq v1.2.0/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo=
github.com/lib/pq v1.3.0 h1:/qkRGz8zljWiDcFvgpwUpwIAPu3r07TDvs3Rws+o/pU=
github.com/lib/pq v1.3.0/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo=
github.com/lucas-clemente/quic-go v0.7.1-0.20201102053916-272229abf044 h1:M6zB4Rs4SJDk9IBIvC3ozl23+b0d1Q7NOlHnbxuc3AY=
github.com/lucas-clemente/quic-go v0.7.1-0.20201102053916-272229abf044/go.mod h1:ZUygOqIoai0ASXXLJ92LTnKdbqh9MHCLTX6Nr1jUrK0=
github.com/lunixbochs/vtclean v1.0.0/go.mod h1:pHhQNgMf3btfWnGBVipUOjRYhoOsdGqdm/+2c2E2WMI=
github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
github.com/magiconair/properties v1.8.1 h1:ZC2Vc7/ZFkGmsVC9KvOjumD+G5lXy2RtTKyzRKO2BQ4=
github.com/magiconair/properties v1.8.1/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
github.com/mailru/easyjson v0.0.0-20190312143242-1de009706dbe/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc=
github.com/marten-seemann/qpack v0.2.1/go.mod h1:F7Gl5L1jIgN1D11ucXefiuJS9UMVP2opoCp2jDKb7wc=
github.com/marten-seemann/qtls v0.10.0 h1:ECsuYUKalRL240rRD4Ri33ISb7kAQ3qGDlrrl55b2pc=
github.com/marten-seemann/qtls v0.10.0/go.mod h1:UvMd1oaYDACI99/oZUYLzMCkBXQVT0aGm99sJhbT8hs=
github.com/marten-seemann/qtls-go1-15 v0.1.1 h1:LIH6K34bPVttyXnUWixk0bzH6/N07VxbSabxn5A5gZQ=
github.com/marten-seemann/qtls-go1-15 v0.1.1/go.mod h1:GyFwywLKkRt+6mfU99csTEY1joMZz5vmB1WNZH3P81I=
github.com/mattn/go-colorable v0.0.9/go.mod h1:9vuHe8Xs5qXnSaW/c/ABM9alt+Vo+STaOChaDxuIBZU=
github.com/mattn/go-colorable v0.1.1/go.mod h1:FuOcm+DKB9mbwrcAfNl7/TZVBZ6rcnceauSikq3lYCQ=
github.com/mattn/go-colorable v0.1.2/go.mod h1:U0ppj6V5qS13XJ6of8GYAs25YV2eR4EVcfRqFIhoBtE=
@ -353,6 +402,7 @@ github.com/mattn/go-sqlite3 v1.10.0/go.mod h1:FPy6KqzDD04eiIsT53CuJW3U88zkxoIYsO
github.com/mattn/go-sqlite3 v2.0.3+incompatible h1:gXHsfypPkaMZrKbD5209QV9jbUTJKjyR5WD3HYQSd+U=
github.com/mattn/go-sqlite3 v2.0.3+incompatible/go.mod h1:FPy6KqzDD04eiIsT53CuJW3U88zkxoIYsOqkbpncsNc=
github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0=
github.com/microcosm-cc/bluemonday v1.0.1/go.mod h1:hsXNsILzKxV+sX77C5b8FSuKF00vh2OMYv+xgHpAMF4=
github.com/miekg/dns v1.0.14/go.mod h1:W1PPwlIAgtquWBMBEV9nkV9Cazfe8ScdGz/Lj7v3Nrg=
github.com/minio/sha256-simd v0.0.0-20190328051042-05b4dd3047e5/go.mod h1:2FMWW+8GMoPweT6+pI63m9YE3Lmw4J71hV56Chs1E/U=
github.com/mitchellh/cli v1.0.0/go.mod h1:hNIlj7HEI86fIcpObd7a0FcrxTWetlwJDGcceTlRvqc=
@ -370,22 +420,32 @@ github.com/morikuni/aec v0.0.0-20170113033406-39771216ff4c h1:nXxl5PrvVm2L/wCy8d
github.com/morikuni/aec v0.0.0-20170113033406-39771216ff4c/go.mod h1:BbKIizmSmc5MMPqRYbxO4ZU0S0+P200+tUnFx7PXmsc=
github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
github.com/nakagami/firebirdsql v0.0.0-20190310045651-3c02a58cfed8/go.mod h1:86wM1zFnC6/uDBfZGNwB65O+pR2OFi5q/YQaEUid1qA=
github.com/neelance/astrewrite v0.0.0-20160511093645-99348263ae86/go.mod h1:kHJEU3ofeGjhHklVoIGuVj85JJwZ6kWPaJwCIxgnFmo=
github.com/neelance/sourcemap v0.0.0-20151028013722-8c68805598ab/go.mod h1:Qr6/a/Q4r9LP1IltGz7tA7iOK1WonHEYhu1HRBA7ZiM=
github.com/nsf/jsondiff v0.0.0-20200515183724-f29ed568f4ce h1:RPclfga2SEJmgMmz2k+Mg7cowZ8yv4Trqw9UsJby758=
github.com/nsf/jsondiff v0.0.0-20200515183724-f29ed568f4ce/go.mod h1:uFMI8w+ref4v2r9jz+c9i1IfIttS/OkmLfrk1jne5hs=
github.com/nsf/termbox-go v0.0.0-20200418040025-38ba6e5628f1 h1:lh3PyZvY+B9nFliSGTn5uFuqQQJGuNrD0MLCokv09ag=
github.com/nsf/termbox-go v0.0.0-20200418040025-38ba6e5628f1/go.mod h1:IuKpRQcYE1Tfu+oAQqaLisqDeXgjyyltCfsaoYN18NQ=
github.com/nxadm/tail v1.4.4 h1:DQuhQpB1tVlglWS2hLQ5OV6B5r8aGxSrPc5Qo6uTN78=
github.com/nxadm/tail v1.4.4/go.mod h1:kenIhsEOeOJmVchQTgglprH7qJGnHDVpk1VPCcaMI8A=
github.com/oklog/ulid v1.3.1/go.mod h1:CirwcVhetQ6Lv90oh/F+FBtV6XMibvdAFo93nm5qn4U=
github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
github.com/onsi/ginkgo v1.7.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
github.com/onsi/ginkgo v1.10.3 h1:OoxbjfXVZyod1fmWYhI7SEyaD8B00ynP3T+D5GiyHOY=
github.com/onsi/ginkgo v1.10.3/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
github.com/onsi/ginkgo v1.12.1/go.mod h1:zj2OWP4+oCPe1qIXoGWkgMRwljMUYCdkwsT2108oapk=
github.com/onsi/ginkgo v1.14.0 h1:2mOpI4JVVPBN+WQRa0WKH2eXR+Ey+uK4n7Zj0aYpIQA=
github.com/onsi/ginkgo v1.14.0/go.mod h1:iSB4RoI2tjJc9BBv4NKIKWKya62Rps+oPG/Lv9klQyY=
github.com/onsi/gomega v1.4.3/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY=
github.com/onsi/gomega v1.7.1 h1:K0jcRCwNQM3vFGh1ppMtDh/+7ApJrjldlX8fA0jDTLQ=
github.com/onsi/gomega v1.7.1/go.mod h1:XdKZgCCFLUoM/7CFJVPcG8C1xQ1AJ0vpAezJrB7JYyY=
github.com/onsi/gomega v1.10.1 h1:o0+MgICZLuZ7xjH7Vx6zS/zcu93/BEp1VwkIW1mEXCE=
github.com/onsi/gomega v1.10.1/go.mod h1:iN09h71vgCQne3DLsj+A5owkum+a2tYe+TOCB1ybHNo=
github.com/opencontainers/go-digest v1.0.0-rc1 h1:WzifXhOVOEOuFYOJAW6aQqW0TooG2iki3E3Ii+WN7gQ=
github.com/opencontainers/go-digest v1.0.0-rc1/go.mod h1:cMLVZDEM3+U2I4VmLI6N8jQYUd2OVphdqWwCJHrFt2s=
github.com/opencontainers/image-spec v1.0.1 h1:JMemWkRwHx4Zj+fVxWoMCFm/8sYGGrUVojFA6h/TRcI=
github.com/opencontainers/image-spec v1.0.1/go.mod h1:BtxoFyWECRxE4U/7sNtV5W15zMzWCbyJoFRP3s7yZA0=
github.com/openzipkin/zipkin-go v0.1.1/go.mod h1:NtoC/o8u3JlF1lSlyPNswIbeQH9bJTmOf0Erfk+hxe8=
github.com/openzipkin/zipkin-go v0.1.6/go.mod h1:QgAqvLzwWbR/WpD4A3cGpPtJrZXNIiJc5AZX7/PBEpw=
github.com/pascaldekloe/goe v0.0.0-20180627143212-57f6aae5913c/go.mod h1:lzWF7FIEvWOWxwDKqyGYQf6ZUaNfKdP144TG7ZOy1lc=
github.com/pelletier/go-toml v1.2.0 h1:T5zMGML61Wp+FlcbWjRDT7yAxhJNAiPPLOFECq181zc=
@ -398,6 +458,7 @@ github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINE
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/posener/complete v1.1.1/go.mod h1:em0nMJCgc9GFtwrmVmEMR/ZL6WyhyjMBndrE9hABlRI=
github.com/prometheus/client_golang v0.8.0/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXPKyh/dDVn+NZz0KFw=
github.com/prometheus/client_golang v0.9.1/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXPKyh/dDVn+NZz0KFw=
github.com/prometheus/client_golang v0.9.3-0.20190127221311-3c4408c8b829/go.mod h1:p2iRAGwDERtqlqzRXnrOVns+ignqQo//hLXqYxZYVNs=
github.com/prometheus/client_golang v0.9.3/go.mod h1:/TN21ttK/J9q6uSwhBd54HahCDft0ttaMvbicHlPoso=
@ -405,9 +466,11 @@ github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910/go.mod h1:
github.com/prometheus/client_model v0.0.0-20190115171406-56726106282f/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo=
github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
github.com/prometheus/common v0.0.0-20180801064454-c7de2306084e/go.mod h1:daVV7qP5qjZbuso7PdcryaAu0sAZbrN9i7WWcTMWvro=
github.com/prometheus/common v0.0.0-20181113130724-41aa239b4cce/go.mod h1:daVV7qP5qjZbuso7PdcryaAu0sAZbrN9i7WWcTMWvro=
github.com/prometheus/common v0.2.0/go.mod h1:TNfzLD0ON7rHzMJeJkieUDPYmFC7Snx/y86RQel1bk4=
github.com/prometheus/common v0.4.0/go.mod h1:TNfzLD0ON7rHzMJeJkieUDPYmFC7Snx/y86RQel1bk4=
github.com/prometheus/procfs v0.0.0-20180725123919-05ee40e3a273/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
github.com/prometheus/procfs v0.0.0-20181005140218-185b4288413d/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
github.com/prometheus/procfs v0.0.0-20190117184657-bf6a532e95b1/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
github.com/prometheus/procfs v0.0.0-20190507164030-5867b95ac084/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsTZCD3I8kEA=
@ -418,15 +481,39 @@ github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFR
github.com/rs/xid v1.2.1/go.mod h1:+uKXf+4Djp6Md1KODXJxgGQPKngRmWyn10oCKFzNHOQ=
github.com/rs/zerolog v1.13.0/go.mod h1:YbFCdg8HfsridGWAh22vktObvhZbQsZXe4/zB0OKkWU=
github.com/rs/zerolog v1.15.0/go.mod h1:xYTKnLHcpfU2225ny5qZjxnj9NvkumZYjJHlAThCjNc=
github.com/russross/blackfriday v1.5.2/go.mod h1:JO/DiYxRf+HjHt06OyowR9PTA263kcR/rfWxYHBV53g=
github.com/russross/blackfriday/v2 v2.0.1/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
github.com/ryanuber/columnize v0.0.0-20160712163229-9b3edd62028f/go.mod h1:sm1tb6uqfes/u+d4ooFouqFdy9/2g9QGwK3SQygK0Ts=
github.com/satori/go.uuid v1.2.0/go.mod h1:dA0hQrYB0VpLJoorglMZABFdXlWrHn1NEOzdhQKdks0=
github.com/sean-/seed v0.0.0-20170313163322-e2103e2c3529/go.mod h1:DxrIzT+xaE7yg65j358z/aeFdxmN0P9QXhEzd20vsDc=
github.com/sergi/go-diff v1.0.0/go.mod h1:0CfEIISq7TuYL3j771MWULgwwjU+GofnZX9QAmXWZgo=
github.com/shopspring/decimal v0.0.0-20180709203117-cd690d0c9e24/go.mod h1:M+9NzErvs504Cn4c5DxATwIqPbtswREoFCre64PpcG4=
github.com/shopspring/decimal v0.0.0-20200227202807-02e2044944cc/go.mod h1:DKyhrW/HYNuLGql+MJL6WCR6knT2jwCFRcu2hWCYk4o=
github.com/shopspring/decimal v1.2.0 h1:abSATXmQEYyShuxI4/vyW3tV1MrKAJzCZ/0zLUXYbsQ=
github.com/shopspring/decimal v1.2.0/go.mod h1:DKyhrW/HYNuLGql+MJL6WCR6knT2jwCFRcu2hWCYk4o=
github.com/shurcooL/component v0.0.0-20170202220835-f88ec8f54cc4/go.mod h1:XhFIlyj5a1fBNx5aJTbKoIq0mNaPvOagO+HjB3EtxrY=
github.com/shurcooL/events v0.0.0-20181021180414-410e4ca65f48/go.mod h1:5u70Mqkb5O5cxEA8nxTsgrgLehJeAw6Oc4Ab1c/P1HM=
github.com/shurcooL/github_flavored_markdown v0.0.0-20181002035957-2122de532470/go.mod h1:2dOwnU2uBioM+SGy2aZoq1f/Sd1l9OkAeAUvjSyvgU0=
github.com/shurcooL/go v0.0.0-20180423040247-9e1955d9fb6e/go.mod h1:TDJrrUr11Vxrven61rcy3hJMUqaf/CLWYhHNPmT14Lk=
github.com/shurcooL/go-goon v0.0.0-20170922171312-37c2f522c041/go.mod h1:N5mDOmsrJOB+vfqUK+7DmDyjhSLIIBnXo9lvZJj3MWQ=
github.com/shurcooL/gofontwoff v0.0.0-20180329035133-29b52fc0a18d/go.mod h1:05UtEgK5zq39gLST6uB0cf3NEHjETfB4Fgr3Gx5R9Vw=
github.com/shurcooL/gopherjslib v0.0.0-20160914041154-feb6d3990c2c/go.mod h1:8d3azKNyqcHP1GaQE/c6dDgjkgSx2BZ4IoEi4F1reUI=
github.com/shurcooL/highlight_diff v0.0.0-20170515013008-09bb4053de1b/go.mod h1:ZpfEhSmds4ytuByIcDnOLkTHGUI6KNqRNPDLHDk+mUU=
github.com/shurcooL/highlight_go v0.0.0-20181028180052-98c3abbbae20/go.mod h1:UDKB5a1T23gOMUJrI+uSuH0VRDStOiUVSjBTRDVBVag=
github.com/shurcooL/home v0.0.0-20181020052607-80b7ffcb30f9/go.mod h1:+rgNQw2P9ARFAs37qieuu7ohDNQ3gds9msbT2yn85sg=
github.com/shurcooL/htmlg v0.0.0-20170918183704-d01228ac9e50/go.mod h1:zPn1wHpTIePGnXSHpsVPWEktKXHr6+SS6x/IKRb7cpw=
github.com/shurcooL/httperror v0.0.0-20170206035902-86b7830d14cc/go.mod h1:aYMfkZ6DWSJPJ6c4Wwz3QtW22G7mf/PEgaB9k/ik5+Y=
github.com/shurcooL/httpfs v0.0.0-20171119174359-809beceb2371/go.mod h1:ZY1cvUeJuFPAdZ/B6v7RHavJWZn2YPVFQ1OSXhCGOkg=
github.com/shurcooL/httpgzip v0.0.0-20180522190206-b1c53ac65af9/go.mod h1:919LwcH0M7/W4fcZ0/jy0qGght1GIhqyS/EgWGH2j5Q=
github.com/shurcooL/issues v0.0.0-20181008053335-6292fdc1e191/go.mod h1:e2qWDig5bLteJ4fwvDAc2NHzqFEthkqn7aOZAOpj+PQ=
github.com/shurcooL/issuesapp v0.0.0-20180602232740-048589ce2241/go.mod h1:NPpHK2TI7iSaM0buivtFUc9offApnI0Alt/K8hcHy0I=
github.com/shurcooL/notifications v0.0.0-20181007000457-627ab5aea122/go.mod h1:b5uSkrEVM1jQUspwbixRBhaIjIzL2xazXp6kntxYle0=
github.com/shurcooL/octicon v0.0.0-20181028054416-fa4f57f9efb2/go.mod h1:eWdoE5JD4R5UVWDucdOPg1g2fqQRq78IQa9zlOV1vpQ=
github.com/shurcooL/reactions v0.0.0-20181006231557-f2e0b4ca5b82/go.mod h1:TCR1lToEk4d2s07G3XGfz2QrgHXg4RJBvjrOozvoWfk=
github.com/shurcooL/sanitized_anchor_name v0.0.0-20170918181015-86672fcb3f95/go.mod h1:1NzhyTcUVG4SuEtjjoZeVRXNmyL/1OwPU0+IJeTBvfc=
github.com/shurcooL/sanitized_anchor_name v1.0.0/go.mod h1:1NzhyTcUVG4SuEtjjoZeVRXNmyL/1OwPU0+IJeTBvfc=
github.com/shurcooL/users v0.0.0-20180125191416-49c67e49c537/go.mod h1:QJTqeLYEDaXHZDBsXlPCDqdhQuJkuw4NOtaxYe3xii4=
github.com/shurcooL/webdavfs v0.0.0-20170829043945-18c3829fa133/go.mod h1:hKmq5kWdCj2z2KEozexVbfEZIWiTjhE0+UjmZgPqehw=
github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
github.com/sirupsen/logrus v1.4.1/go.mod h1:ni0Sbl8bgC9z8RoU9G6nDWqqs/fq4eDPysMBDgk/93Q=
github.com/sirupsen/logrus v1.4.2 h1:SPIRibHv4MatM3XXNO2BJeFLZwZ2LvZgfQ5+UNI2im4=
@ -436,6 +523,8 @@ github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d/go.mod h1
github.com/smartystreets/goconvey v1.6.4 h1:fv0U8FUIMPNf1L9lnHLvLhgicrIVChEkdzIKYqbNC9s=
github.com/smartystreets/goconvey v1.6.4/go.mod h1:syvi0/a8iFYH4r/RixwvyeAJjdLS9QV7WQ/tjFTllLA=
github.com/soheilhy/cmux v0.1.4/go.mod h1:IM3LyeVVIOuxMH7sFAkER9+bJ4dT7Ms6E4xg4kGIyLM=
github.com/sourcegraph/annotate v0.0.0-20160123013949-f4cad6c6324d/go.mod h1:UdhH50NIW0fCiwBSr0co2m7BnFLdv4fQTgdqdJTHFeE=
github.com/sourcegraph/syntaxhighlight v0.0.0-20170531221838-bd320f5d308e/go.mod h1:HuIsMU8RRBOtsCgI77wP899iHVBQpCmg4ErYMZB+2IA=
github.com/spacemonkeygo/monkit/v3 v3.0.0-20191108235033-eacca33b3037/go.mod h1:JcK1pCbReQsOsMKF/POFSZCq7drXFybgGmbc27tuwes=
github.com/spacemonkeygo/monkit/v3 v3.0.4/go.mod h1:JcK1pCbReQsOsMKF/POFSZCq7drXFybgGmbc27tuwes=
github.com/spacemonkeygo/monkit/v3 v3.0.5/go.mod h1:JcK1pCbReQsOsMKF/POFSZCq7drXFybgGmbc27tuwes=
@ -475,9 +564,12 @@ github.com/stripe/stripe-go v70.15.0+incompatible h1:hNML7M1zx8RgtepEMlxyu/FpVPr
github.com/stripe/stripe-go v70.15.0+incompatible/go.mod h1:A1dQZmO/QypXmsL0T8axYZkSN/uA/T/A64pfKdBAMiY=
github.com/subosito/gotenv v1.2.0 h1:Slr1R9HxAlEKefgq5jn9U+DnETlIUa6HfgEzj0g5d7s=
github.com/subosito/gotenv v1.2.0/go.mod h1:N0PQaV/YGNqwC0u51sEeR/aUtSLEXKX9iv69rRypqCw=
github.com/tarm/serial v0.0.0-20180830185346-98f6abe2eb07/go.mod h1:kDXzergiv9cbyO7IOYJZWg1U88JhDg3PB6klq9Hg2pA=
github.com/tidwall/pretty v0.0.0-20180105212114-65a9db5fad51/go.mod h1:XNkn88O1ChpSDQmQeStsy+sBenx6DDtFZJxhVysOjyk=
github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U=
github.com/ugorji/go v1.1.4/go.mod h1:uQMGLiO92mf5W77hV/PUCpI3pbzQx3CRekS0kk+RGrc=
github.com/viant/assertly v0.4.8/go.mod h1:aGifi++jvCrUaklKEKT0BU95igDNaqkvz+49uaYMPRU=
github.com/viant/toolbox v0.24.0/go.mod h1:OxMCG57V0PXuIP2HNQrtJf2CjqdmbrOx5EkMILuUhzM=
github.com/vivint/infectious v0.0.0-20200605153912-25a574ae18a3 h1:zMsHhfK9+Wdl1F7sIKLyx3wrOFofpb3rWFbA4HgcK5k=
github.com/vivint/infectious v0.0.0-20200605153912-25a574ae18a3/go.mod h1:R0Gbuw7ElaGSLOZUSwBm/GgVwMd30jWxBDdAyMOeTuc=
github.com/xanzy/go-gitlab v0.15.0/go.mod h1:8zdQa/ri1dfn8eS3Ir1SyfvOKlw7WBJ8DVThkpGiXrs=
@ -508,6 +600,7 @@ go.etcd.io/bbolt v1.3.2/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU=
go.etcd.io/bbolt v1.3.5 h1:XAzx9gjCb0Rxj7EoqcClPD1d5ZBxZJk0jbuoPHenBt0=
go.etcd.io/bbolt v1.3.5/go.mod h1:G5EMThwa9y8QZGBClrRx5EY+Yw9kAhnjy3bSjsnlVTQ=
go.mongodb.org/mongo-driver v1.1.0/go.mod h1:u7ryQJ+DOzQmeO7zB6MHyr8jkEQvC8vH7qLUO4lqsUM=
go.opencensus.io v0.18.0/go.mod h1:vKdFvxhtzZ9onBp9VKHK8z/sRpBMnKAsufL7wlDrCOA=
go.opencensus.io v0.20.1/go.mod h1:6WKK9ahsWS3RSO+PY9ZHZUfv2irvY6gN279GOPZjmmk=
go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU=
go.opencensus.io v0.22.0/go.mod h1:+kGneAE2xo2IficOXnaByMWTGM9T73dGwxeWcUqIpI8=
@ -531,10 +624,14 @@ go.uber.org/zap v1.10.0/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q=
go.uber.org/zap v1.14.1/go.mod h1:Mb2vm2krFEG5DV0W9qcHBYFtp/Wku1cvYaqPsS/WYfc=
go.uber.org/zap v1.16.0 h1:uFRZXykJGK9lLY4HtgSw44DnIcAM+kRBP7x5m+NpAOM=
go.uber.org/zap v1.16.0/go.mod h1:MA8QOfq0BHJwdXa996Y4dYkAqRKB8/1K1QMMZVaNZjQ=
go4.org v0.0.0-20180809161055-417644f6feb5/go.mod h1:MkTOUMDaeVYJUOUsaDXIhWPZYa1yOyC1qaOBpL57BhE=
golang.org/x/build v0.0.0-20190111050920-041ab4dc3f9d/go.mod h1:OWs+y06UdEOHN4y+MfF/py+xQ/tYqIWW03b70/CG9Rw=
golang.org/x/crypto v0.0.0-20170930174604-9419663f5a44/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
golang.org/x/crypto v0.0.0-20181029021203-45a5f77698d3/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
golang.org/x/crypto v0.0.0-20181030102418-4d3f4d9ffa16/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20190313024323-a1f597ede03a/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20190325154230-a5d413f7728c/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20190411191339-88737f569e3a/go.mod h1:WFFai1msRO1wXaEeE5yQxYXgSfI8pQAWXbQop6sCtWE=
golang.org/x/crypto v0.0.0-20190426145343-a29dc8fdc734/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
@ -545,6 +642,7 @@ golang.org/x/crypto v0.0.0-20190911031432-227b76d455e7/go.mod h1:yigFU9vqHzYiE8U
golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20200115085410-6d4e4cb37c7d/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20200220183623-bac4c82f6975/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20200221231518-2aa609cf4a9d/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20200323165209-0ec3e9974c59/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20200604202706-70a84ac30bf9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
@ -559,6 +657,7 @@ golang.org/x/exp v0.0.0-20191227195350-da58074b4299 h1:zQpM52jfKHG6II1ISZY1Zcpyg
golang.org/x/exp v0.0.0-20191227195350-da58074b4299/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4=
golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js=
golang.org/x/image v0.0.0-20190802002840-cff245a6509b/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0=
golang.org/x/lint v0.0.0-20180702182130-06c8688daad7/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU=
golang.org/x/lint v0.0.0-20190301231843-5614ed5bae6f/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
@ -577,6 +676,8 @@ golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73r
golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20181023162649-9b4f9f5ad519/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20181029044818-c44066c5c816/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20181106065722-10aee1819953/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20181108082009-03003ca0c849/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20181201002055-351d144fa1fc/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
@ -585,6 +686,7 @@ golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73r
golang.org/x/net v0.0.0-20190125091013-d26f9f9a57f3/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190313220215-9f648a60d977/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190424112056-4829fb13d2c6/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190501004415-9ce7a6920f09/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
@ -595,15 +697,19 @@ golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLL
golang.org/x/net v0.0.0-20190813141303-74dc4d7220e7/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20190923162816-aa69164e4478/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200114155413-6afb5195e5aa/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200226121028-0de0cce0169b h1:0mm1VjtFUOIlE1SbDlwjYaDxZVDP2S5ou6y0gSgXHu8=
golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200520004742-59133d7f0dd7/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
golang.org/x/net v0.0.0-20200707034311-ab3426394381 h1:VXak5I6aEWmAXeQjA+QSZzlgNrpq9mjcfDemuexIKsU=
golang.org/x/net v0.0.0-20200707034311-ab3426394381/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20181017192945-9dcd33a902f4/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20181106182150-f42d05182288/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20181203162652-d668ce993890/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20190402181905-9f3314589c9a/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d h1:TzXSXBo42m9gQenoE3b9BGiEpg5IG2JkU5FkPIawgtw=
golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/perf v0.0.0-20180704124530-6e6d33e29852/go.mod h1:JLpeXjPJfIyPr5TlbXLkXWLhP8nz10XfvxElABhCtcw=
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
@ -618,6 +724,7 @@ golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5h
golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20181026203630-95b1ffbd15a5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20181029174526-d69651ed3497/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20181107165924-66b7b1311ac8/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20181122145206-62eef0e2fa9b/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
@ -626,6 +733,7 @@ golang.org/x/sys v0.0.0-20190204203706-41f3e6584952/go.mod h1:STP8DvDyc/dI5b8T5h
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190312061237-fead79001313/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190316082340-a2f829d7f35f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190403152447-81d4e9dc473e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190422165155-953cdadca894/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
@ -636,7 +744,10 @@ golang.org/x/sys v0.0.0-20190606165138-5da285871e9c/go.mod h1:h1NjWce9XRLGQEsW7w
golang.org/x/sys v0.0.0-20190624142023-c5567b49c5d0/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190813064441-fde4db37ae7a/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190826190057-c7b8b68b1456/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190904154756-749cb33beabd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191005200804-aed5e4c7ecf9/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191026070338-33540a1f6037/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191120155948-bd437916bb0e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191204072324-ce4227a45e2e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191210023423-ac6580df4449/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200107144601-ef85f5a75ddf/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
@ -645,14 +756,17 @@ golang.org/x/sys v0.0.0-20200116001909-b77594299b42/go.mod h1:h1NjWce9XRLGQEsW7w
golang.org/x/sys v0.0.0-20200202164722-d101bd2416d5/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200519105757-fe76b779f299/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200610111108-226ff32320da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200929083018-4d22bbb62b3c h1:/h0vtH0PyU0xAoZJVcRw1k0Ng+U0JAy3QDiFmppIlIE=
golang.org/x/sys v0.0.0-20200929083018-4d22bbb62b3c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
golang.org/x/text v0.3.3 h1:cokOdA+Jmi5PJGXLlLllQSgYigAEfHXJAERHVMaCc2k=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/time v0.0.0-20180412165947-fbb02b2291d2/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20200630173020-3af7569d3a1e h1:EHBhcS0mlXEAVwNyO2dLfjToGsyY4j24pTs2ScHnX7s=
@ -660,6 +774,7 @@ golang.org/x/time v0.0.0-20200630173020-3af7569d3a1e/go.mod h1:tRJNPiyCQ0inRvYxb
golang.org/x/tools v0.0.0-20180221164845-07fd8470d635/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20180828015842-6cd1fcedba52/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20181030000716-a0a13e073c7b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20181030221726-6c7e314b6563/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY=
@ -691,6 +806,9 @@ golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8T
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543 h1:E7g+9GITq07hpfrRu66IVDexMakfv52eLZ2CXBWiKr4=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
google.golang.org/api v0.0.0-20180910000450-7ca32eb868bf/go.mod h1:4mhQ8q/RsB7i+udVvVy5NUi08OU8ZlA0gRVgrF7VFY0=
google.golang.org/api v0.0.0-20181030000543-1d582fd0359e/go.mod h1:4mhQ8q/RsB7i+udVvVy5NUi08OU8ZlA0gRVgrF7VFY0=
google.golang.org/api v0.1.0/go.mod h1:UGEZY7KEX120AnNLIHFMKIo4obdJhkp2tPbaPlQx13Y=
google.golang.org/api v0.3.1/go.mod h1:6wY9I6uQWHQ8EM57III9mq/AjF+i8G65rmVagqKMtkk=
google.golang.org/api v0.3.2/go.mod h1:6wY9I6uQWHQ8EM57III9mq/AjF+i8G65rmVagqKMtkk=
google.golang.org/api v0.4.0/go.mod h1:8k5glujaEP+g9n7WNsDg8QP6cUVNI86fCNMcbazEtwE=
@ -702,6 +820,7 @@ google.golang.org/api v0.15.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsb
google.golang.org/api v0.20.0 h1:jz2KixHX7EcCPiQrySzPdnYT7DbINAypCqKZ1Z7GM40=
google.golang.org/api v0.20.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE=
google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
google.golang.org/appengine v1.2.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
google.golang.org/appengine v1.3.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
google.golang.org/appengine v1.5.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
@ -709,6 +828,10 @@ google.golang.org/appengine v1.6.1/go.mod h1:i06prIuMbXzDqacNJfV5OdTW448YApPu5ww
google.golang.org/appengine v1.6.5 h1:tycE03LOZYQNhDpS27tcQdAzLCVMaj7QT2SXxebnpCM=
google.golang.org/appengine v1.6.5/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc=
google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
google.golang.org/genproto v0.0.0-20180831171423-11092d34479b/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
google.golang.org/genproto v0.0.0-20181029155118-b69ba1387ce2/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
google.golang.org/genproto v0.0.0-20181202183823-bd91e49a0898/go.mod h1:7Ep/1NZk928CDR8SjdVbjWNpdIf6nzjE3BTgJDr2Atg=
google.golang.org/genproto v0.0.0-20190306203927-b5d61aea6440/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
google.golang.org/genproto v0.0.0-20190307195333-5fe7a883aa19/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
google.golang.org/genproto v0.0.0-20190404172233-64821d5d2107/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
google.golang.org/genproto v0.0.0-20190418145605-e7d98fc518a7/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
@ -720,6 +843,8 @@ google.golang.org/genproto v0.0.0-20190911173649-1774047e7e51/go.mod h1:IbNlFCBr
google.golang.org/genproto v0.0.0-20191108220845-16a3f7862a1a/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20200115191322-ca5a22157cba h1:pRj9OXZbwNtbtZtOB4dLwfK4u+EVRMvP+e9zKkg2grM=
google.golang.org/genproto v0.0.0-20200115191322-ca5a22157cba/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/grpc v1.14.0/go.mod h1:yo6s7OP7yaDglbqo1J04qKzAhqBH6lvTonzMVmEdcZw=
google.golang.org/grpc v1.16.0/go.mod h1:0JHn/cJsOMiMfNA9+DeHDlAU7KAAB5GDlYFpa9MZMio=
google.golang.org/grpc v1.17.0/go.mod h1:6QZJwpn2B+Zp71q/5VxRsJ6NXXVCE5NRUHRo+f3cWCs=
google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
google.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiqXj38=
@ -730,6 +855,13 @@ google.golang.org/grpc v1.26.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8
google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
google.golang.org/grpc v1.27.1 h1:zvIju4sqAGvwKspUQOhwnpcqSbzi7/H6QomNNjTL4sk=
google.golang.org/grpc v1.27.1/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8=
google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0=
google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM=
google.golang.org/protobuf v1.20.1-0.20200309200217-e05f789c0967/go.mod h1:A+miEFZTKqfCUM6K7xSMQL9OKL/b6hQv+e19PK+JZNE=
google.golang.org/protobuf v1.21.0/go.mod h1:47Nbq4nVaFHyn7ilMalzfO3qCViNmqZ2kzikPIcrTAo=
google.golang.org/protobuf v1.23.0 h1:4MY060fB1DLGMB/7MBTLnwQUY6+F09GEiz6SsrNqyzM=
google.golang.org/protobuf v1.23.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 h1:qIbj1fsPNlZgppZ+VLlY7N33q108Sa+fhmuc+sWQYwY=
@ -749,10 +881,13 @@ gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.4 h1:/eiJrUcujPVeJ3xlSWaiNi3uSVmDGBK1pDHUHAnao1I=
gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.3.0 h1:clyUAQHOM3G0M3f5vQj7LuJrETvjVot3Z5el9nffUtU=
gopkg.in/yaml.v2 v2.3.0/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c h1:dUUwHk2QECo/6vqA44rthZ8ie2QXMNeKRTHCNY2nXvo=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gotest.tools v2.2.0+incompatible h1:VsBPFP1AI068pPrMxtb/S8Zkgf9xEmTLJjfM+P5UIEo=
gotest.tools v2.2.0+incompatible/go.mod h1:DsYFclhRJ6vuDpmuTbkuFWG+y2sxOXAzmJt81HFBacw=
grpc.go4.org v0.0.0-20170609214715-11d0a25b4919/go.mod h1:77eQGdRu53HpSqPFJFmuJdjuHRquDANNeA4x7B8WQ9o=
honnef.co/go/tools v0.0.0-20180728063816-88497007e858/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.0-20190106161140-3f1c8253044a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
@ -761,18 +896,23 @@ honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWh
honnef.co/go/tools v0.0.1-2019.2.3 h1:3JgtbtFHMiCmsznwGVTUWbgGov+pVqnlf1dEJTNAXeM=
honnef.co/go/tools v0.0.1-2019.2.3/go.mod h1:a3bituU0lyd329TUQxRnasdCoJDkEUEAqEt0JzvZhAg=
rsc.io/binaryregexp v0.2.0/go.mod h1:qTv7/COck+e2FymRvadv62gMdZztPaShugOCi3I+8D8=
rsc.io/quote/v3 v3.1.0/go.mod h1:yEA65RcK8LyAZtP9Kv3t0HmxON59tX3rD+tICJqUlj0=
rsc.io/sampler v1.3.0/go.mod h1:T1hPZKmBbMNahiBKFy5HrXp6adAjACjK9JXDnKaTXpA=
sourcegraph.com/sourcegraph/go-diff v0.5.0/go.mod h1:kuch7UrkMzY0X+p9CRK03kfuPQ2zzQcaEFbx8wA8rck=
sourcegraph.com/sqs/pbtypes v0.0.0-20180604144634-d3ebe8f20ae4/go.mod h1:ketZ/q3QxT9HOBeFhu6RdvsftgpsbFHBF5Cas6cDKZ0=
storj.io/common v0.0.0-20200424175742-65ac59022f4f/go.mod h1:pZyXiIE7bGETIRXtfs0nICqMwp7PM8HqnDuyUeldNA0=
storj.io/common v0.0.0-20201026135900-1aaeec90670b/go.mod h1:GqdmNf3fLm2UZX/7Zr0BLFCJ4gFjgm6eHrk/fnmr5jQ=
storj.io/common v0.0.0-20201027143432-3718579e12bf/go.mod h1:9iobNl9eI6C2M23FS/b37yFYOdHpoeJ8BFFcxsmv538=
storj.io/common v0.0.0-20201030120157-90ae6720d87e h1:6baDicBbR0/2XgcQ068KN+B4dF6akkdh2vemmXka1ns=
storj.io/common v0.0.0-20201030120157-90ae6720d87e/go.mod h1:9iobNl9eI6C2M23FS/b37yFYOdHpoeJ8BFFcxsmv538=
storj.io/common v0.0.0-20201106104920-372a344bdd45 h1:pv552R7MiRA8VLQC4qXczLjbl2Qb/MNyus2E9NBSXgI=
storj.io/common v0.0.0-20201106104920-372a344bdd45/go.mod h1:ZkQZup2jpFZvvTgz+yPc7K4Vr4bBHM8AA66P57MZkjk=
storj.io/drpc v0.0.11/go.mod h1:TiFc2obNjL9/3isMW1Rpxjy8V9uE0B2HMeMFGiiI7Iw=
storj.io/drpc v0.0.11/go.mod h1:TiFc2obNjL9/3isMW1Rpxjy8V9uE0B2HMeMFGiiI7Iw=
storj.io/drpc v0.0.14 h1:GCBdymTt1BRw4oHmmUZZlxYXLVRxxYj6x3Ivide2J+I=
storj.io/drpc v0.0.14/go.mod h1:82nfl+6YwRwF6UG31cEWWUqv/FaKvP5SGqUvoqTxCMA=
storj.io/drpc v0.0.16 h1:9sxypc5lKi/0D69cR21BR0S21+IvXfON8L5nXMVNTwQ=
storj.io/drpc v0.0.16/go.mod h1:zdmQ93nx4Z35u11pQ+GAnBy4DGOK3HJCSOfeh2RryTo=
storj.io/monkit-jaeger v0.0.0-20200518165323-80778fc3f91b h1:Bbg9JCtY6l3HrDxs3BXzT2UYnYCBLqNi6i84Y8QIPUs=
storj.io/monkit-jaeger v0.0.0-20200518165323-80778fc3f91b/go.mod h1:gj4vuCeyCRjRmH8LIrgoyU9Dc9uR6H+/GcDUXmTbf80=
storj.io/private v0.0.0-20201026143115-bc926bfa3bca h1:ekR7vtUYC5+cDyim0ZJaSZeXidyzQqDYsnFPYXgTozc=
storj.io/private v0.0.0-20201026143115-bc926bfa3bca/go.mod h1:EaLnIyNyqWQUJB+7+KWVez0In9czl0nHHlm2WobebuA=
storj.io/uplink v1.3.2-0.20201028181609-f6efc8fcf771 h1:jPbw74xt8bvv8nOfBaM4g9Ts4moX8mqfD4N/B8vEJrA=
storj.io/uplink v1.3.2-0.20201028181609-f6efc8fcf771/go.mod h1:5do8jvbs4ao4tLdIZKzNFJPVKOH1oDfvVf8OIsR5Z9E=
storj.io/uplink v1.3.2-0.20201106105834-ec8e5cc29f7e h1:a58hzcYciTvBtWST+Byoj76NWuYdnPcz2GK8ynyEyfA=
storj.io/uplink v1.3.2-0.20201106105834-ec8e5cc29f7e/go.mod h1:mrdt4I4EhPRC7cnvCD5490IBm423pgKrVoUiC9a5Srg=

40
multinode/console/members.go Normal file
View File

@ -0,0 +1,40 @@
// Copyright (C) 2020 Storj Labs, Inc.
// See LICENSE for copying information.
package console
import (
"context"
"github.com/zeebo/errs"
"storj.io/common/uuid"
)
// Members exposes needed by MND MembersDB functionality.
//
// architecture: Database
type Members interface {
// Invite will create empty row in membersDB.
Invite(ctx context.Context, member Member) error
// Update updates all updatable fields of member.
Update(ctx context.Context, member Member) error
// Remove deletes member from membersDB.
Remove(ctx context.Context, id uuid.UUID) error
// GetByEmail will return member with specified email.
GetByEmail(ctx context.Context, email string) (Member, error)
// GetByID will return member with specified id.
GetByID(ctx context.Context, id uuid.UUID) (Member, error)
}
// ErrNoMember is a special error type that indicates about absence of member in MembersDB.
var ErrNoMember = errs.Class("no such member")
// Member represents some person that is invited to the MND by node owner.
// Member will have configurable access privileges that will define which functions and which nodes are available for him.
type Member struct {
ID uuid.UUID
Email string
Name string
PasswordHash []byte
}

64
multinode/console/members_test.go Normal file
View File

@ -0,0 +1,64 @@
// Copyright (C) 2020 Storj Labs, Inc.
// See LICENSE for copying information.
package console_test
import (
"testing"
"github.com/stretchr/testify/require"
"github.com/zeebo/assert"
"storj.io/common/testcontext"
"storj.io/common/uuid"
"storj.io/storj/multinode"
"storj.io/storj/multinode/console"
"storj.io/storj/multinode/multinodedb/multinodedbtest"
)
func TestMembersDB(t *testing.T) {
multinodedbtest.Run(t, func(ctx *testcontext.Context, t *testing.T, db multinode.DB) {
members := db.Members()
memberID, err := uuid.New()
require.NoError(t, err)
memberBob := console.Member{
ID: memberID,
Email: "mail@example.com",
Name: "Bob",
PasswordHash: []byte{0},
}
err = members.Invite(ctx, memberBob)
assert.NoError(t, err)
memberToCheck, err := members.GetByEmail(ctx, memberBob.Email)
assert.NoError(t, err)
assert.Equal(t, memberToCheck.Email, memberBob.Email)
assert.Equal(t, memberToCheck.Name, memberBob.Name)
assert.Equal(t, memberToCheck.Email, memberBob.Email)
memberBob.Name = "Alice"
err = members.Update(ctx, memberBob)
assert.NoError(t, err)
memberAlice, err := members.GetByID(ctx, memberToCheck.ID)
assert.NoError(t, err)
assert.Equal(t, memberToCheck.Email, memberAlice.Email)
assert.Equal(t, memberToCheck.Name, memberAlice.Name)
assert.Equal(t, memberToCheck.Email, memberAlice.Email)
assert.Equal(t, memberToCheck.ID, memberAlice.ID)
err = members.Remove(ctx, memberAlice.ID)
assert.NoError(t, err)
_, err = members.GetByID(ctx, memberToCheck.ID)
assert.Error(t, err)
assert.Equal(t, true, console.ErrNoMember.Has(err))
_, err = members.GetByEmail(ctx, memberToCheck.Email)
assert.Error(t, err)
assert.Equal(t, true, console.ErrNoMember.Has(err))
})
}

0
multinode/console/node.go → multinode/console/nodes.go
View File

40
multinode/mutlinodedb/database.go → multinode/multinodedb/database.go
View File

@ -1,16 +1,18 @@
// Copyright (C) 2020 Storj Labs, Inc.
// See LICENSE for copying information.
package mutlinodedb
package multinodedb
import (
"context"
"github.com/spacemonkeygo/monkit/v3"
"github.com/zeebo/errs"
"go.uber.org/zap"
"storj.io/storj/multinode"
"storj.io/storj/multinode/console"
"storj.io/storj/multinode/mutlinodedb/dbx"
"storj.io/storj/multinode/multinodedb/dbx"
"storj.io/storj/private/dbutil"
"storj.io/storj/private/dbutil/pgutil"
)
@ -39,8 +41,8 @@ type multinodeDB struct {
source string
}
// New creates instance of database supports postgres.
func New(log *zap.Logger, databaseURL string) (multinode.DB, error) {
// Open creates instance of database supports postgres.
func Open(ctx context.Context, log *zap.Logger, databaseURL string) (multinode.DB, error) {
driver, source, implementation, err := dbutil.SplitConnStr(databaseURL)
if err != nil {
return nil, err
@ -52,17 +54,17 @@ func New(log *zap.Logger, databaseURL string) (multinode.DB, error) {
source = pgutil.CheckApplicationName(source)
// dbxDB, err := dbx.Open(driver, source)
// if err != nil {
// return nil, Error.New("failed opening database via DBX at %q: %v",
// source, err)
// }
// log.Debug("Connected to:", zap.String("db source", source))
dbxDB, err := dbx.Open(driver, source)
if err != nil {
return nil, Error.New("failed opening database via DBX at %q: %v",
source, err)
}
log.Debug("Connected to:", zap.String("db source", source))
// dbutil.Configure(dbxDB.DB, "multinodedb", mon)
dbutil.Configure(ctx, dbxDB.DB, "multinodedb", mon)
core := &multinodeDB{
// DB: dbxDB,
DB: dbxDB,
log: log,
driver: driver,
@ -77,6 +79,18 @@ func New(log *zap.Logger, databaseURL string) (multinode.DB, error) {
func (db *multinodeDB) Nodes() console.Nodes {
return &nodes{
methods: db,
db: db,
}
}
// Members returns members database.
func (db *multinodeDB) Members() console.Members {
return &members{
methods: db,
}
}
// CreateSchema creates schema.
func (db *multinodeDB) CreateSchema(ctx context.Context) error {
_, err := db.ExecContext(ctx, db.DB.Schema())
return err
}

24
multinode/multinodedb/dbx/gen.go Normal file
View File

@ -0,0 +1,24 @@
// Copyright (C) 2020 Storj Labs, Inc.
// See LICENSE for copying information.
package dbx
import (
"github.com/spacemonkeygo/monkit/v3"
"github.com/zeebo/errs"
)
//go:generate sh gen.sh
var mon = monkit.Package()
func init() {
// catch dbx errors
class := errs.Class("multinodedb dbx error")
WrapErr = func(e *Error) error {
if e.Code == ErrorCode_NoRows {
return e.Err
}
return class.Wrap(e)
}
}

0
multinode/mutlinodedb/dbx/gen.sh → multinode/multinodedb/dbx/gen.sh
View File

45
multinode/multinodedb/dbx/multinodedb.dbx Normal file
View File

@ -0,0 +1,45 @@
// dbx.v1 golang multinodedb.dbx .
model node (
key id
field id blob
field name text ( updatable )
field tag text ( updatable )
field public_address text
field api_secret blob
field logo blob ( updatable )
)
create node ( )
delete node ( where node.id = ? )
read one (
select node
where node.id = ?
)
model member (
key id
field id blob
field email text ( updatable )
field name text ( updatable )
field password_hash blob ( updatable )
field created_at timestamp ( autoinsert )
)
create member ( )
delete member ( where member.id = ? )
update member ( where member.id = ? )
read one (
select member
where member.email = ?
)
read one (
select member
where member.id = ?
)

386
multinode/mutlinodedb/dbx/multinodedb.dbx.go → multinode/multinodedb/dbx/multinodedb.dbx.go
View File

@ -269,7 +269,15 @@ func newpgx(db *DB) *pgxDB {
}
func (obj *pgxDB) Schema() string {
return `CREATE TABLE nodes (
return `CREATE TABLE members (
id bytea NOT NULL,
email text NOT NULL,
name text NOT NULL,
password_hash bytea NOT NULL,
created_at timestamp with time zone NOT NULL,
PRIMARY KEY ( id )
);
CREATE TABLE nodes (
id bytea NOT NULL,
name text NOT NULL,
tag text NOT NULL,
@ -340,6 +348,117 @@ nextval:
fmt.Fprint(f, "]")
}
type Member struct {
Id []byte
Email string
Name string
PasswordHash []byte
CreatedAt time.Time
}
func (Member) _Table() string { return "members" }
type Member_Update_Fields struct {
Email Member_Email_Field
Name Member_Name_Field
PasswordHash Member_PasswordHash_Field
}
type Member_Id_Field struct {
_set bool
_null bool
_value []byte
}
func Member_Id(v []byte) Member_Id_Field {
return Member_Id_Field{_set: true, _value: v}
}
func (f Member_Id_Field) value() interface{} {
if !f._set || f._null {
return nil
}
return f._value
}
func (Member_Id_Field) _Column() string { return "id" }
type Member_Email_Field struct {
_set bool
_null bool
_value string
}
func Member_Email(v string) Member_Email_Field {
return Member_Email_Field{_set: true, _value: v}
}
func (f Member_Email_Field) value() interface{} {
if !f._set || f._null {
return nil
}
return f._value
}
func (Member_Email_Field) _Column() string { return "email" }
type Member_Name_Field struct {
_set bool
_null bool
_value string
}
func Member_Name(v string) Member_Name_Field {
return Member_Name_Field{_set: true, _value: v}
}
func (f Member_Name_Field) value() interface{} {
if !f._set || f._null {
return nil
}
return f._value
}
func (Member_Name_Field) _Column() string { return "name" }
type Member_PasswordHash_Field struct {
_set bool
_null bool
_value []byte
}
func Member_PasswordHash(v []byte) Member_PasswordHash_Field {
return Member_PasswordHash_Field{_set: true, _value: v}
}
func (f Member_PasswordHash_Field) value() interface{} {
if !f._set || f._null {
return nil
}
return f._value
}
func (Member_PasswordHash_Field) _Column() string { return "password_hash" }
type Member_CreatedAt_Field struct {
_set bool
_null bool
_value time.Time
}
func Member_CreatedAt(v time.Time) Member_CreatedAt_Field {
return Member_CreatedAt_Field{_set: true, _value: v}
}
func (f Member_CreatedAt_Field) value() interface{} {
if !f._set || f._null {
return nil
}
return f._value
}
func (Member_CreatedAt_Field) _Column() string { return "created_at" }
type Node struct {
Id []byte
Name string
@ -924,6 +1043,38 @@ func (obj *pgxImpl) Create_Node(ctx context.Context,
}
func (obj *pgxImpl) Create_Member(ctx context.Context,
member_id Member_Id_Field,
member_email Member_Email_Field,
member_name Member_Name_Field,
member_password_hash Member_PasswordHash_Field) (
member *Member, err error) {
defer mon.Task()(&ctx)(&err)
__now := obj.db.Hooks.Now().UTC()
__id_val := member_id.value()
__email_val := member_email.value()
__name_val := member_name.value()
__password_hash_val := member_password_hash.value()
__created_at_val := __now
var __embed_stmt = __sqlbundle_Literal("INSERT INTO members ( id, email, name, password_hash, created_at ) VALUES ( ?, ?, ?, ?, ? ) RETURNING members.id, members.email, members.name, members.password_hash, members.created_at")
var __values []interface{}
__values = append(__values, __id_val, __email_val, __name_val, __password_hash_val, __created_at_val)
var __stmt = __sqlbundle_Render(obj.dialect, __embed_stmt)
obj.logStmt(__stmt, __values...)
member = &Member{}
err = obj.driver.QueryRowContext(ctx, __stmt, __values...).Scan(&member.Id, &member.Email, &member.Name, &member.PasswordHash, &member.CreatedAt)
if err != nil {
return nil, obj.makeErr(err)
}
return member, nil
}
func (obj *pgxImpl) Get_Node_By_Id(ctx context.Context,
node_id Node_Id_Field) (
node *Node, err error) {
@ -946,6 +1097,123 @@ func (obj *pgxImpl) Get_Node_By_Id(ctx context.Context,
}
func (obj *pgxImpl) Get_Member_By_Email(ctx context.Context,
member_email Member_Email_Field) (
member *Member, err error) {
defer mon.Task()(&ctx)(&err)
var __embed_stmt = __sqlbundle_Literal("SELECT members.id, members.email, members.name, members.password_hash, members.created_at FROM members WHERE members.email = ? LIMIT 2")
var __values []interface{}
__values = append(__values, member_email.value())
var __stmt = __sqlbundle_Render(obj.dialect, __embed_stmt)
obj.logStmt(__stmt, __values...)
__rows, err := obj.driver.QueryContext(ctx, __stmt, __values...)
if err != nil {
return nil, obj.makeErr(err)
}
defer __rows.Close()
if !__rows.Next() {
if err := __rows.Err(); err != nil {
return nil, obj.makeErr(err)
}
return nil, makeErr(sql.ErrNoRows)
}
member = &Member{}
err = __rows.Scan(&member.Id, &member.Email, &member.Name, &member.PasswordHash, &member.CreatedAt)
if err != nil {
return nil, obj.makeErr(err)
}
if __rows.Next() {
return nil, tooManyRows("Member_By_Email")
}
if err := __rows.Err(); err != nil {
return nil, obj.makeErr(err)
}
return member, nil
}
func (obj *pgxImpl) Get_Member_By_Id(ctx context.Context,
member_id Member_Id_Field) (
member *Member, err error) {
defer mon.Task()(&ctx)(&err)
var __embed_stmt = __sqlbundle_Literal("SELECT members.id, members.email, members.name, members.password_hash, members.created_at FROM members WHERE members.id = ?")
var __values []interface{}
__values = append(__values, member_id.value())
var __stmt = __sqlbundle_Render(obj.dialect, __embed_stmt)
obj.logStmt(__stmt, __values...)
member = &Member{}
err = obj.driver.QueryRowContext(ctx, __stmt, __values...).Scan(&member.Id, &member.Email, &member.Name, &member.PasswordHash, &member.CreatedAt)
if err != nil {
return (*Member)(nil), obj.makeErr(err)
}
return member, nil
}
func (obj *pgxImpl) Update_Member_By_Id(ctx context.Context,
member_id Member_Id_Field,
update Member_Update_Fields) (
member *Member, err error) {
defer mon.Task()(&ctx)(&err)
var __sets = &__sqlbundle_Hole{}
var __embed_stmt = __sqlbundle_Literals{Join: "", SQLs: []__sqlbundle_SQL{__sqlbundle_Literal("UPDATE members SET "), __sets, __sqlbundle_Literal(" WHERE members.id = ? RETURNING members.id, members.email, members.name, members.password_hash, members.created_at")}}
__sets_sql := __sqlbundle_Literals{Join: ", "}
var __values []interface{}
var __args []interface{}
if update.Email._set {
__values = append(__values, update.Email.value())
__sets_sql.SQLs = append(__sets_sql.SQLs, __sqlbundle_Literal("email = ?"))
}
if update.Name._set {
__values = append(__values, update.Name.value())
__sets_sql.SQLs = append(__sets_sql.SQLs, __sqlbundle_Literal("name = ?"))
}
if update.PasswordHash._set {
__values = append(__values, update.PasswordHash.value())
__sets_sql.SQLs = append(__sets_sql.SQLs, __sqlbundle_Literal("password_hash = ?"))
}
if len(__sets_sql.SQLs) == 0 {
return nil, emptyUpdate()
}
__args = append(__args, member_id.value())
__values = append(__values, __args...)
__sets.SQL = __sets_sql
var __stmt = __sqlbundle_Render(obj.dialect, __embed_stmt)
obj.logStmt(__stmt, __values...)
member = &Member{}
err = obj.driver.QueryRowContext(ctx, __stmt, __values...).Scan(&member.Id, &member.Email, &member.Name, &member.PasswordHash, &member.CreatedAt)
if err == sql.ErrNoRows {
return nil, nil
}
if err != nil {
return nil, obj.makeErr(err)
}
return member, nil
}
func (obj *pgxImpl) Delete_Node_By_Id(ctx context.Context,
node_id Node_Id_Field) (
deleted bool, err error) {
@ -973,6 +1241,33 @@ func (obj *pgxImpl) Delete_Node_By_Id(ctx context.Context,
}
func (obj *pgxImpl) Delete_Member_By_Id(ctx context.Context,
member_id Member_Id_Field) (
deleted bool, err error) {
defer mon.Task()(&ctx)(&err)
var __embed_stmt = __sqlbundle_Literal("DELETE FROM members WHERE members.id = ?")
var __values []interface{}
__values = append(__values, member_id.value())
var __stmt = __sqlbundle_Render(obj.dialect, __embed_stmt)
obj.logStmt(__stmt, __values...)
__res, err := obj.driver.ExecContext(ctx, __stmt, __values...)
if err != nil {
return false, obj.makeErr(err)
}
__count, err := __res.RowsAffected()
if err != nil {
return false, obj.makeErr(err)
}
return __count > 0, nil
}
func (impl pgxImpl) isConstraintError(err error) (
constraint string, ok bool) {
if e, ok := err.(*pgconn.PgError); ok {
@ -992,6 +1287,16 @@ func (obj *pgxImpl) deleteAll(ctx context.Context) (count int64, err error) {
return 0, obj.makeErr(err)
}
__count, err = __res.RowsAffected()
if err != nil {
return 0, obj.makeErr(err)
}
count += __count
__res, err = obj.driver.ExecContext(ctx, "DELETE FROM members;")
if err != nil {
return 0, obj.makeErr(err)
}
__count, err = __res.RowsAffected()
if err != nil {
return 0, obj.makeErr(err)
@ -1044,6 +1349,20 @@ func (rx *Rx) Rollback() (err error) {
return err
}
func (rx *Rx) Create_Member(ctx context.Context,
member_id Member_Id_Field,
member_email Member_Email_Field,
member_name Member_Name_Field,
member_password_hash Member_PasswordHash_Field) (
member *Member, err error) {
var tx *Tx
if tx, err = rx.getTx(ctx); err != nil {
return
}
return tx.Create_Member(ctx, member_id, member_email, member_name, member_password_hash)
}
func (rx *Rx) Create_Node(ctx context.Context,
node_id Node_Id_Field,
node_name Node_Name_Field,
@ -1060,6 +1379,16 @@ func (rx *Rx) Create_Node(ctx context.Context,
}
func (rx *Rx) Delete_Member_By_Id(ctx context.Context,
member_id Member_Id_Field) (
deleted bool, err error) {
var tx *Tx
if tx, err = rx.getTx(ctx); err != nil {
return
}
return tx.Delete_Member_By_Id(ctx, member_id)
}
func (rx *Rx) Delete_Node_By_Id(ctx context.Context,
node_id Node_Id_Field) (
deleted bool, err error) {
@ -1070,6 +1399,26 @@ func (rx *Rx) Delete_Node_By_Id(ctx context.Context,
return tx.Delete_Node_By_Id(ctx, node_id)
}
func (rx *Rx) Get_Member_By_Email(ctx context.Context,
member_email Member_Email_Field) (
member *Member, err error) {
var tx *Tx
if tx, err = rx.getTx(ctx); err != nil {
return
}
return tx.Get_Member_By_Email(ctx, member_email)
}
func (rx *Rx) Get_Member_By_Id(ctx context.Context,
member_id Member_Id_Field) (
member *Member, err error) {
var tx *Tx
if tx, err = rx.getTx(ctx); err != nil {
return
}
return tx.Get_Member_By_Id(ctx, member_id)
}
func (rx *Rx) Get_Node_By_Id(ctx context.Context,
node_id Node_Id_Field) (
node *Node, err error) {
@ -1080,7 +1429,25 @@ func (rx *Rx) Get_Node_By_Id(ctx context.Context,
return tx.Get_Node_By_Id(ctx, node_id)
}
func (rx *Rx) Update_Member_By_Id(ctx context.Context,
member_id Member_Id_Field,
update Member_Update_Fields) (
member *Member, err error) {
var tx *Tx
if tx, err = rx.getTx(ctx); err != nil {
return
}
return tx.Update_Member_By_Id(ctx, member_id, update)
}
type Methods interface {
Create_Member(ctx context.Context,
member_id Member_Id_Field,
member_email Member_Email_Field,
member_name Member_Name_Field,
member_password_hash Member_PasswordHash_Field) (
member *Member, err error)
Create_Node(ctx context.Context,
node_id Node_Id_Field,
node_name Node_Name_Field,
@ -1090,13 +1457,30 @@ type Methods interface {
node_logo Node_Logo_Field) (
node *Node, err error)
Delete_Member_By_Id(ctx context.Context,
member_id Member_Id_Field) (
deleted bool, err error)
Delete_Node_By_Id(ctx context.Context,
node_id Node_Id_Field) (
deleted bool, err error)
Get_Member_By_Email(ctx context.Context,
member_email Member_Email_Field) (
member *Member, err error)
Get_Member_By_Id(ctx context.Context,
member_id Member_Id_Field) (
member *Member, err error)
Get_Node_By_Id(ctx context.Context,
node_id Node_Id_Field) (
node *Node, err error)
Update_Member_By_Id(ctx context.Context,
member_id Member_Id_Field,
update Member_Update_Fields) (
member *Member, err error)
}
type TxMethods interface {

8
multinode/mutlinodedb/dbx/multinodedb.dbx.pgx.sql → multinode/multinodedb/dbx/multinodedb.dbx.pgx.sql
View File

@ -1,5 +1,13 @@
-- AUTOGENERATED BY storj.io/dbx
-- DO NOT EDIT
CREATE TABLE members (
id bytea NOT NULL,
email text NOT NULL,
name text NOT NULL,
password_hash bytea NOT NULL,
created_at timestamp with time zone NOT NULL,
PRIMARY KEY ( id )
);
CREATE TABLE nodes (
id bytea NOT NULL,
name text NOT NULL,

0
multinode/mutlinodedb/dbx/templates/golang.decl.tmpl → multinode/multinodedb/dbx/templates/golang.decl.tmpl
View File

117
multinode/multinodedb/members.go Normal file
View File

@ -0,0 +1,117 @@
// Copyright (C) 2020 Storj Labs, Inc.
// See LICENSE for copying information.
package multinodedb
import (
"context"
"database/sql"
"errors"
"github.com/zeebo/errs"
"storj.io/common/uuid"
"storj.io/storj/multinode/console"
"storj.io/storj/multinode/multinodedb/dbx"
)
// MembersDBError indicates about internal MembersDB error.
var MembersDBError = errs.Class("MembersDB error")
// ensures that members implements console.Members.
var _ console.Members = (*members)(nil)
// members exposes needed by MND MembersDB functionality.
// dbx implementation of console.Members.
//
// architecture: Database
type members struct {
methods dbx.Methods
}
// Invite will create empty row in membersDB.
func (m *members) Invite(ctx context.Context, member console.Member) (err error) {
defer mon.Task()(&ctx)(&err)
id, err := uuid.New()
if err != nil {
return MembersDBError.Wrap(err)
}
_, err = m.methods.Create_Member(ctx, dbx.Member_Id(id[:]), dbx.Member_Email(member.Email), dbx.Member_Name(member.Name), dbx.Member_PasswordHash(member.PasswordHash))
return MembersDBError.Wrap(err)
}
// Update updates all updatable fields of member.
func (m *members) Update(ctx context.Context, member console.Member) (err error) {
defer mon.Task()(&ctx)(&err)
_, err = m.methods.Update_Member_By_Id(ctx, dbx.Member_Id(member.ID[:]), dbx.Member_Update_Fields{
Email: dbx.Member_Email(member.Email),
Name: dbx.Member_Name(member.Name),
PasswordHash: dbx.Member_PasswordHash(member.PasswordHash),
})
return MembersDBError.Wrap(err)
}
// Remove deletes member from membersDB.
func (m *members) Remove(ctx context.Context, id uuid.UUID) (err error) {
defer mon.Task()(&ctx)(&err)
_, err = m.methods.Delete_Member_By_Id(ctx, dbx.Member_Id(id[:]))
return MembersDBError.Wrap(err)
}
// GetByEmail will return member with specified email.
func (m *members) GetByEmail(ctx context.Context, email string) (_ console.Member, err error) {
defer mon.Task()(&ctx)(&err)
memberDbx, err := m.methods.Get_Member_By_Email(ctx, dbx.Member_Email(email))
if err != nil {
if errors.Is(err, sql.ErrNoRows) {
return console.Member{}, console.ErrNoMember.Wrap(err)
}
return console.Member{}, MembersDBError.Wrap(err)
}
member, err := fromDBXMember(memberDbx)
return member, MembersDBError.Wrap(err)
}
// GetByID will return member with specified id.
func (m *members) GetByID(ctx context.Context, id uuid.UUID) (_ console.Member, err error) {
defer mon.Task()(&ctx)(&err)
memberDbx, err := m.methods.Get_Member_By_Id(ctx, dbx.Member_Id(id[:]))
if err != nil {
if errors.Is(err, sql.ErrNoRows) {
return console.Member{}, console.ErrNoMember.Wrap(err)
}
return console.Member{}, MembersDBError.Wrap(err)
}
member, err := fromDBXMember(memberDbx)
return member, MembersDBError.Wrap(err)
}
// fromDBXMember converts dbx.Member to console.Member.
func fromDBXMember(member *dbx.Member) (_ console.Member, err error) {
id, err := uuid.FromBytes(member.Id)
if err != nil {
return console.Member{}, err
}
result := console.Member{
ID: id,
Email: member.Email,
Name: member.Name,
PasswordHash: member.PasswordHash,
}
return result, nil
}

141
multinode/multinodedb/multinodedbtest/run.go Normal file
View File

@ -0,0 +1,141 @@
// Copyright (C) 2020 Storj Labs, Inc.
// See LICENSE for copying information.
package multinodedbtest
import (
"context"
"fmt"
"strconv"
"strings"
"testing"
"github.com/zeebo/errs"
"go.uber.org/zap"
"go.uber.org/zap/zaptest"
"storj.io/common/testcontext"
"storj.io/storj/multinode"
"storj.io/storj/multinode/multinodedb"
"storj.io/storj/multinode/multinodedb/dbx"
"storj.io/storj/private/dbutil"
"storj.io/storj/private/dbutil/pgtest"
"storj.io/storj/private/dbutil/pgutil"
"storj.io/storj/private/dbutil/tempdb"
)
// Database describes a test database.
type Database struct {
Name string
URL string
Message string
}
// tempMasterDB is a multinode.DB-implementing type that cleans up after itself when closed.
type tempMasterDB struct {
multinode.DB
tempDB *dbutil.TempDatabase
}
// Close closes a tempMasterDB and cleans it up afterward.
func (db *tempMasterDB) Close() error {
return errs.Combine(db.DB.Close(), db.tempDB.Close())
}
// TestDBAccess provides a somewhat regularized access to the underlying DB.
func (db *tempMasterDB) TestDBAccess() *dbx.DB {
return db.DB.(interface{ TestDBAccess() *dbx.DB }).TestDBAccess()
}
type ignoreSkip struct{}
func (ignoreSkip) Skip(...interface{}) {}
// SchemaSuffix returns a suffix for schemas.
func SchemaSuffix() string {
return pgutil.CreateRandomTestingSchemaName(6)
}
// SchemaName returns a properly formatted schema string.
func SchemaName(testname, category string, index int, schemaSuffix string) string {
// postgres has a maximum schema length of 64
// we need additional 6 bytes for the random suffix
// and 4 bytes for the index "/S0/""
indexStr := strconv.Itoa(index)
var maxTestNameLen = 64 - len(category) - len(indexStr) - len(schemaSuffix) - 2
if len(testname) > maxTestNameLen {
testname = testname[:maxTestNameLen]
}
if schemaSuffix == "" {
return strings.ToLower(testname + "/" + category + indexStr)
}
return strings.ToLower(testname + "/" + schemaSuffix + "/" + category + indexStr)
}
// CreateMasterDB creates a new satellite database for testing.
func CreateMasterDB(ctx context.Context, log *zap.Logger, name string, category string, index int, dbInfo Database) (db multinode.DB, err error) {
if dbInfo.URL == "" {
return nil, fmt.Errorf("database %s connection string not provided. %s", dbInfo.Name, dbInfo.Message)
}
schemaSuffix := SchemaSuffix()
log.Debug("creating", zap.String("suffix", schemaSuffix))
schema := SchemaName(name, category, index, schemaSuffix)
tempDB, err := tempdb.OpenUnique(ctx, dbInfo.URL, schema)
if err != nil {
return nil, err
}
return CreateMasterDBOnTopOf(ctx, log, tempDB)
}
// CreateMasterDBOnTopOf creates a new satellite database on top of an already existing
// temporary database.
func CreateMasterDBOnTopOf(ctx context.Context, log *zap.Logger, tempDB *dbutil.TempDatabase) (db multinode.DB, err error) {
masterDB, err := multinodedb.Open(ctx, log, tempDB.ConnStr)
return &tempMasterDB{DB: masterDB, tempDB: tempDB}, err
}
// Run method will iterate over all supported databases. Will establish
// connection and will create tables for each DB.
func Run(t *testing.T, test func(ctx *testcontext.Context, t *testing.T, db multinode.DB)) {
masterDB := Database{
Name: "Postgres",
URL: pgtest.PickPostgres(ignoreSkip{}),
Message: "Postgres flag missing, example: -postgres-test-db=" + pgtest.DefaultPostgres + " or use STORJ_TEST_POSTGRES environment variable.",
}
t.Run(masterDB.Name, func(t *testing.T) {
t.Parallel()
ctx := testcontext.New(t)
defer ctx.Cleanup()
if masterDB.URL == "" {
t.Skipf("Database %s connection string not provided. %s", masterDB.Name, masterDB.Message)
}
db, err := CreateMasterDB(ctx, zaptest.NewLogger(t), t.Name(), "T", 0, masterDB)
if err != nil {
t.Fatal(err)
}
defer func() {
err := db.Close()
if err != nil {
t.Fatal(err)
}
}()
err = db.CreateSchema(ctx)
if err != nil {
t.Fatal(err)
}
test(ctx, t, db)
})
}

5
multinode/mutlinodedb/nodes.go → multinode/multinodedb/nodes.go
View File

@ -1,7 +1,7 @@
// Copyright (C) 2020 Storj Labs, Inc.
// See LICENSE for copying information.
package mutlinodedb
package multinodedb
import (
"context"
@ -10,7 +10,7 @@ import (
"storj.io/common/storj"
"storj.io/storj/multinode/console"
"storj.io/storj/multinode/mutlinodedb/dbx"
"storj.io/storj/multinode/multinodedb/dbx"
)
// NodesDBError indicates about internal NodesDB error.
@ -25,7 +25,6 @@ var _ console.Nodes = (*nodes)(nil)
// architecture: Database
type nodes struct {
methods dbx.Methods
db *multinodeDB
}
// Add creates new node in NodesDB.

12
multinode/mutlinodedb/dbx/gen.go
View File

@ -1,12 +0,0 @@
// Copyright (C) 2020 Storj Labs, Inc.
// See LICENSE for copying information.
package dbx
import (
"github.com/spacemonkeygo/monkit/v3"
)
//go:generate sh gen.sh
var mon = monkit.Package()

20
multinode/mutlinodedb/dbx/multinodedb.dbx
View File

@ -1,20 +0,0 @@
// dbx.v1 golang multinodedb.dbx .
model node (
key id
field id blob
field name text ( updatable )
field tag text ( updatable )
field public_address text
field api_secret blob
field logo blob ( updatable )
)
create node ( )
delete node ( where node.id = ? )
read one (
select node
where node.id = ?
)

4
multinode/peer.go
View File

@ -27,9 +27,13 @@ var (
type DB interface {
// Nodes returns nodes database.
Nodes() console.Nodes
// Members returns members database.
Members() console.Members
// Close closes the database.
Close() error
// CreateSchema creates schema.
CreateSchema(ctx context.Context) error
}
// Config is all the configuration parameters for a Multinode Dashboard.

20
private/dbutil/pgtest/flag.go
View File

@ -5,9 +5,9 @@ package pgtest
import (
"flag"
"math/rand"
"os"
"strings"
"sync/atomic"
"testing"
"storj.io/common/testcontext"
@ -77,20 +77,20 @@ func Run(t *testing.T, test func(ctx *testcontext.Context, t *testing.T, connstr
}
}
// PickPostgres picks a random postgres database from flag.
// PickPostgres picks one postgres database from flag.
func PickPostgres(t TB) string {
if *postgres == "" || strings.EqualFold(*postgres, "omit") {
t.Skip("Postgres flag missing, example: -postgres-test-db=" + DefaultPostgres)
}
return pickRandom(*postgres)
return pickNext(*postgres, &pickPostgres)
}
// PickCockroach picks a random cockroach database from flag.
// PickCockroach picks one cockroach database from flag.
func PickCockroach(t TB) string {
if *cockroach == "" || strings.EqualFold(*cockroach, "omit") {
t.Skip("Cockroach flag missing, example: -cockroach-test-db=" + DefaultCockroach)
}
return pickRandom(*cockroach)
return pickNext(*cockroach, &pickCockroach)
}
// PickCockroachAlt picks an alternate cockroach database from flag.
@ -104,13 +104,17 @@ func PickCockroachAlt(t TB) string {
t.Skip("Cockroach alt flag omitted.")
}
return pickRandom(*cockroachAlt)
return pickNext(*cockroachAlt, &pickCockroach)
}
func pickRandom(dbstr string) string {
var pickPostgres uint64
var pickCockroach uint64
func pickNext(dbstr string, counter *uint64) string {
values := strings.Split(dbstr, ";")
if len(values) <= 1 {
return dbstr
}
return values[rand.Intn(len(values))]
v := atomic.AddUint64(counter, 1)
return values[v%uint64(len(values))]
}

25
private/dbutil/sqliteutil/query.go
View File

@ -73,13 +73,20 @@ func QuerySchema(ctx context.Context, db dbschema.Queryer) (*dbschema.Schema, er
func discoverTables(ctx context.Context, db dbschema.Queryer, schema *dbschema.Schema, tableDefinitions []*definition) (err error) {
for _, definition := range tableDefinitions {
if err := discoverTable(ctx, db, schema, definition); err != nil {
return err
}
}
return errs.Wrap(err)
}
func discoverTable(ctx context.Context, db dbschema.Queryer, schema *dbschema.Schema, definition *definition) (err error) {
table := schema.EnsureTable(definition.name)
tableRows, err := db.QueryContext(ctx, `PRAGMA table_info(`+definition.name+`)`)
if err != nil {
return errs.Wrap(err)
}
defer func() { err = errs.Combine(err, tableRows.Close()) }()
for tableRows.Next() {
var defaultValue sql.NullString
@ -88,7 +95,7 @@ func discoverTables(ctx context.Context, db dbschema.Queryer, schema *dbschema.S
var notNull bool
err := tableRows.Scan(&index, &name, &columnType, &notNull, &defaultValue, &pk)
if err != nil {
return errs.Wrap(err)
return errs.Wrap(errs.Combine(tableRows.Err(), tableRows.Close(), err))
}
column := &dbschema.Column{
@ -103,7 +110,10 @@ func discoverTables(ctx context.Context, db dbschema.Queryer, schema *dbschema.S
}
table.PrimaryKey = append(table.PrimaryKey, name)
}
}
err = errs.Combine(tableRows.Err(), tableRows.Close())
if err != nil {
return errs.Wrap(err)
}
matches := rxUnique.FindAllStringSubmatch(definition.sql, -1)
@ -121,14 +131,13 @@ func discoverTables(ctx context.Context, db dbschema.Queryer, schema *dbschema.S
if err != nil {
return errs.Wrap(err)
}
defer func() { err = errs.Combine(err, keysRows.Close()) }()
for keysRows.Next() {
var id, sec int
var tableName, from, to, onUpdate, onDelete, match string
err := keysRows.Scan(&id, &sec, &tableName, &from, &to, &onUpdate, &onDelete, &match)
if err != nil {
return errs.Wrap(err)
return errs.Wrap(errs.Combine(keysRows.Err(), keysRows.Close(), err))
}
column, found := table.FindColumn(from)
@ -147,10 +156,14 @@ func discoverTables(ctx context.Context, db dbschema.Queryer, schema *dbschema.S
}
}
}
}
err = errs.Combine(keysRows.Err(), keysRows.Close())
if err != nil {
return errs.Wrap(err)
}
return nil
}
func discoverIndexes(ctx context.Context, db dbschema.Queryer, schema *dbschema.Schema, indexDefinitions []*definition) (err error) {
// TODO improve indexes discovery
for _, definition := range indexDefinitions {

21
private/lifecycle/group.go
View File

@ -8,6 +8,7 @@ import (
"context"
"errors"
"runtime/pprof"
"time"
"github.com/spacemonkeygo/monkit/v3"
"github.com/zeebo/errs"
@ -54,7 +55,27 @@ func (group *Group) Run(ctx context.Context, g *errgroup.Group) {
if item.Run == nil {
continue
}
shutdownCtx, shutdownFinished := context.WithCancel(context.Background())
go func() {
select {
case <-ctx.Done():
case <-shutdownCtx.Done():
return
}
shutdownDeadline := time.NewTimer(15 * time.Second)
defer shutdownDeadline.Stop()
select {
case <-shutdownDeadline.C:
group.log.Warn("service takes long to shutdown", zap.String("name", item.Name))
case <-shutdownCtx.Done():
}
}()
g.Go(func() error {
defer shutdownFinished()
var err error
pprof.Do(ctx, pprof.Labels("name", item.Name), func(ctx context.Context) {
err = item.Run(ctx)

116
private/testblobs/bad.go
View File

@ -5,6 +5,7 @@ package testblobs
import (
"context"
"sync"
"time"
"go.uber.org/zap"
@ -43,11 +44,30 @@ func (bad *BadDB) SetError(err error) {
// BadBlobs implements a bad blob store.
type BadBlobs struct {
err error
err lockedErr
blobs storage.Blobs
log *zap.Logger
}
type lockedErr struct {
mu sync.Mutex
err error
}
// Err returns the error.
func (m *lockedErr) Err() error {
m.mu.Lock()
defer m.mu.Unlock()
return m.err
}
// Set sets the error.
func (m *lockedErr) Set(err error) {
m.mu.Lock()
defer m.mu.Unlock()
m.err = err
}
// newBadBlobs creates a new bad blob store wrapping the provided blobs.
// Use SetError to manually configure the error returned by all operations.
func newBadBlobs(log *zap.Logger, blobs storage.Blobs) *BadBlobs {
@ -57,27 +77,32 @@ func newBadBlobs(log *zap.Logger, blobs storage.Blobs) *BadBlobs {
}
}
// SetError configures the blob store to return a specific error for all operations.
func (bad *BadBlobs) SetError(err error) {
bad.err.Set(err)
}
// Create creates a new blob that can be written optionally takes a size
// argument for performance improvements, -1 is unknown size.
func (bad *BadBlobs) Create(ctx context.Context, ref storage.BlobRef, size int64) (storage.BlobWriter, error) {
if bad.err != nil {
return nil, bad.err
if err := bad.err.Err(); err != nil {
return nil, err
}
return bad.blobs.Create(ctx, ref, size)
}
// Close closes the blob store and any resources associated with it.
func (bad *BadBlobs) Close() error {
if bad.err != nil {
return bad.err
if err := bad.err.Err(); err != nil {
return err
}
return bad.blobs.Close()
}
// Open opens a reader with the specified namespace and key.
func (bad *BadBlobs) Open(ctx context.Context, ref storage.BlobRef) (storage.BlobReader, error) {
if bad.err != nil {
return nil, bad.err
if err := bad.err.Err(); err != nil {
return nil, err
}
return bad.blobs.Open(ctx, ref)
}
@ -85,64 +110,64 @@ func (bad *BadBlobs) Open(ctx context.Context, ref storage.BlobRef) (storage.Blo
// OpenWithStorageFormat opens a reader for the already-located blob, avoiding the potential need
// to check multiple storage formats to find the blob.
func (bad *BadBlobs) OpenWithStorageFormat(ctx context.Context, ref storage.BlobRef, formatVer storage.FormatVersion) (storage.BlobReader, error) {
if bad.err != nil {
return nil, bad.err
if err := bad.err.Err(); err != nil {
return nil, err
}
return bad.blobs.OpenWithStorageFormat(ctx, ref, formatVer)
}
// Trash deletes the blob with the namespace and key.
func (bad *BadBlobs) Trash(ctx context.Context, ref storage.BlobRef) error {
if bad.err != nil {
return bad.err
if err := bad.err.Err(); err != nil {
return err
}
return bad.blobs.Trash(ctx, ref)
}
// RestoreTrash restores all files in the trash.
func (bad *BadBlobs) RestoreTrash(ctx context.Context, namespace []byte) ([][]byte, error) {
if bad.err != nil {
return nil, bad.err
if err := bad.err.Err(); err != nil {
return nil, err
}
return bad.blobs.RestoreTrash(ctx, namespace)
}
// EmptyTrash empties the trash.
func (bad *BadBlobs) EmptyTrash(ctx context.Context, namespace []byte, trashedBefore time.Time) (int64, [][]byte, error) {
if bad.err != nil {
return 0, nil, bad.err
if err := bad.err.Err(); err != nil {
return 0, nil, err
}
return bad.blobs.EmptyTrash(ctx, namespace, trashedBefore)
}
// Delete deletes the blob with the namespace and key.
func (bad *BadBlobs) Delete(ctx context.Context, ref storage.BlobRef) error {
if bad.err != nil {
return bad.err
if err := bad.err.Err(); err != nil {
return err
}
return bad.blobs.Delete(ctx, ref)
}
// DeleteWithStorageFormat deletes the blob with the namespace, key, and format version.
func (bad *BadBlobs) DeleteWithStorageFormat(ctx context.Context, ref storage.BlobRef, formatVer storage.FormatVersion) error {
if bad.err != nil {
return bad.err
if err := bad.err.Err(); err != nil {
return err
}
return bad.blobs.DeleteWithStorageFormat(ctx, ref, formatVer)
}
// DeleteNamespace deletes blobs of specific satellite, used after successful GE only.
func (bad *BadBlobs) DeleteNamespace(ctx context.Context, ref []byte) (err error) {
if bad.err != nil {
return bad.err
if err := bad.err.Err(); err != nil {
return err
}
return bad.blobs.DeleteNamespace(ctx, ref)
}
// Stat looks up disk metadata on the blob file.
func (bad *BadBlobs) Stat(ctx context.Context, ref storage.BlobRef) (storage.BlobInfo, error) {
if bad.err != nil {
return nil, bad.err
if err := bad.err.Err(); err != nil {
return nil, err
}
return bad.blobs.Stat(ctx, ref)
}
@ -151,8 +176,8 @@ func (bad *BadBlobs) Stat(ctx context.Context, ref storage.BlobRef) (storage.Blo
// version. This avoids the potential need to check multiple storage formats for the blob
// when the format is already known.
func (bad *BadBlobs) StatWithStorageFormat(ctx context.Context, ref storage.BlobRef, formatVer storage.FormatVersion) (storage.BlobInfo, error) {
if bad.err != nil {
return nil, bad.err
if err := bad.err.Err(); err != nil {
return nil, err
}
return bad.blobs.StatWithStorageFormat(ctx, ref, formatVer)
}
@ -161,64 +186,64 @@ func (bad *BadBlobs) StatWithStorageFormat(ctx context.Context, ref storage.Blob
// If walkFunc returns a non-nil error, WalkNamespace will stop iterating and return the
// error immediately.
func (bad *BadBlobs) WalkNamespace(ctx context.Context, namespace []byte, walkFunc func(storage.BlobInfo) error) error {
if bad.err != nil {
return bad.err
if err := bad.err.Err(); err != nil {
return err
}
return bad.blobs.WalkNamespace(ctx, namespace, walkFunc)
}
// ListNamespaces returns all namespaces that might be storing data.
func (bad *BadBlobs) ListNamespaces(ctx context.Context) ([][]byte, error) {
if bad.err != nil {
return make([][]byte, 0), bad.err
if err := bad.err.Err(); err != nil {
return make([][]byte, 0), err
}
return bad.blobs.ListNamespaces(ctx)
}
// FreeSpace return how much free space left for writing.
func (bad *BadBlobs) FreeSpace() (int64, error) {
if bad.err != nil {
return 0, bad.err
if err := bad.err.Err(); err != nil {
return 0, err
}
return bad.blobs.FreeSpace()
}
// CheckWritability tests writability of the storage directory by creating and deleting a file.
func (bad *BadBlobs) CheckWritability() error {
if bad.err != nil {
return bad.err
if err := bad.err.Err(); err != nil {
return err
}
return bad.blobs.CheckWritability()
}
// SpaceUsedForBlobs adds up how much is used in all namespaces.
func (bad *BadBlobs) SpaceUsedForBlobs(ctx context.Context) (int64, error) {
if bad.err != nil {
return 0, bad.err
if err := bad.err.Err(); err != nil {
return 0, err
}
return bad.blobs.SpaceUsedForBlobs(ctx)
}
// SpaceUsedForBlobsInNamespace adds up how much is used in the given namespace.
func (bad *BadBlobs) SpaceUsedForBlobsInNamespace(ctx context.Context, namespace []byte) (int64, error) {
if bad.err != nil {
return 0, bad.err
if err := bad.err.Err(); err != nil {
return 0, err
}
return bad.blobs.SpaceUsedForBlobsInNamespace(ctx, namespace)
}
// SpaceUsedForTrash adds up how much is used in all namespaces.
func (bad *BadBlobs) SpaceUsedForTrash(ctx context.Context) (int64, error) {
if bad.err != nil {
return 0, bad.err
if err := bad.err.Err(); err != nil {
return 0, err
}
return bad.blobs.SpaceUsedForTrash(ctx)
}
// CreateVerificationFile creates a file to be used for storage directory verification.
func (bad *BadBlobs) CreateVerificationFile(id storj.NodeID) error {
if bad.err != nil {
return bad.err
if err := bad.err.Err(); err != nil {
return err
}
return bad.blobs.CreateVerificationFile(id)
}
@ -226,13 +251,8 @@ func (bad *BadBlobs) CreateVerificationFile(id storj.NodeID) error {
// VerifyStorageDir verifies that the storage directory is correct by checking for the existence and validity
// of the verification file.
func (bad *BadBlobs) VerifyStorageDir(id storj.NodeID) error {
if bad.err != nil {
return bad.err
if err := bad.err.Err(); err != nil {
return err
}
return bad.blobs.VerifyStorageDir(id)
}
// SetError configures the blob store to return a specific error for all operations.
func (bad *BadBlobs) SetError(err error) {
bad.err = err
}

69
private/testplanet/log.go Normal file
View File

@ -0,0 +1,69 @@
// Copyright (C) 2020 Storj Labs, Inc.
// See LICENSE for copying information
package testplanet
import (
"bytes"
"fmt"
"os"
"testing"
"time"
"go.uber.org/zap"
"go.uber.org/zap/zapcore"
"go.uber.org/zap/zaptest"
)
var useAbsTime = os.Getenv("STORJ_TESTPLANET_ABSTIME")
func newLogger(t *testing.T) *zap.Logger {
if useAbsTime != "" {
return zaptest.NewLogger(t)
}
start := time.Now()
cfg := zap.NewDevelopmentEncoderConfig()
cfg.EncodeTime = func(t time.Time, enc zapcore.PrimitiveArrayEncoder) {
var nanos, seconds, minutes int64
nanos = t.Sub(start).Nanoseconds()
seconds, nanos = nanos/1e9, nanos%1e9
minutes, seconds = seconds/60, seconds%60
enc.AppendString(fmt.Sprintf("%02d:%02d.%03d", minutes, seconds, nanos/1e6))
}
enc := zapcore.NewConsoleEncoder(cfg)
writer := newTestingWriter(t)
return zap.New(
zapcore.NewCore(enc, writer, zapcore.DebugLevel),
zap.ErrorOutput(writer.WithMarkFailed(true)),
)
}
type testingWriter struct {
t *testing.T
markFailed bool
}
func newTestingWriter(t *testing.T) testingWriter {
return testingWriter{t: t}
}
func (w testingWriter) WithMarkFailed(v bool) testingWriter {
w.markFailed = v
return w
}
func (w testingWriter) Write(p []byte) (n int, err error) {
n = len(p)
p = bytes.TrimRight(p, "\n")
w.t.Logf("%s", p)
if w.markFailed {
w.t.Fail()
}
return n, nil
}
func (w testingWriter) Sync() error { return nil }

2
private/testplanet/planet_test.go
View File

@ -34,6 +34,7 @@ func TestBasic(t *testing.T) {
for _, sat := range planet.Satellites {
for _, sn := range planet.StorageNodes {
func() {
node := sn.Contact.Service.Local()
conn, err := sn.Dialer.DialNodeURL(ctx, sat.NodeURL())
@ -46,6 +47,7 @@ func TestBasic(t *testing.T) {
Operator: &node.Operator,
})
require.NoError(t, err)
}()
}
}
// wait a bit to see whether some failures occur

8
private/testplanet/reconfigure.go
View File

@ -67,10 +67,10 @@ var Combine = func(elements ...func(log *zap.Logger, index int, config *satellit
// ReconfigureRS returns function to change satellite redundancy scheme values.
var ReconfigureRS = func(minThreshold, repairThreshold, successThreshold, totalThreshold int) func(log *zap.Logger, index int, config *satellite.Config) {
return func(log *zap.Logger, index int, config *satellite.Config) {
config.Metainfo.RS.MinThreshold = minThreshold
config.Metainfo.RS.RepairThreshold = repairThreshold
config.Metainfo.RS.SuccessThreshold = successThreshold
config.Metainfo.RS.TotalThreshold = totalThreshold
config.Metainfo.RS.Min = minThreshold
config.Metainfo.RS.Repair = repairThreshold
config.Metainfo.RS.Success = successThreshold
config.Metainfo.RS.Total = totalThreshold
}
}

4
private/testplanet/run.go
View File

@ -9,8 +9,6 @@ import (
"strings"
"testing"
"go.uber.org/zap/zaptest"
"storj.io/common/testcontext"
"storj.io/storj/private/dbutil/pgtest"
"storj.io/storj/satellite/satellitedb/satellitedbtest"
@ -54,7 +52,7 @@ func Run(t *testing.T, config Config, test func(t *testing.T, ctx *testcontext.C
}
pprof.Do(ctx, pprof.Labels("planet", planetConfig.Name), func(namedctx context.Context) {
planet, err := NewCustom(namedctx, zaptest.NewLogger(t), planetConfig, satelliteDB)
planet, err := NewCustom(namedctx, newLogger(t), planetConfig, satelliteDB)
if err != nil {
t.Fatalf("%+v", err)
}

13
private/testplanet/satellite.go
View File

@ -468,16 +468,11 @@ func (planet *Planet) newSatellite(ctx context.Context, prefix string, index int
MaxCommitInterval: 1 * time.Hour,
Overlay: true,
RS: metainfo.RSConfig{
MaxBufferMem: memory.Size(256),
ErasureShareSize: memory.Size(256),
MinThreshold: atLeastOne(planet.config.StorageNodeCount * 1 / 5),
RepairThreshold: atLeastOne(planet.config.StorageNodeCount * 2 / 5),
SuccessThreshold: atLeastOne(planet.config.StorageNodeCount * 3 / 5),
TotalThreshold: atLeastOne(planet.config.StorageNodeCount * 4 / 5),
MinTotalThreshold: (planet.config.StorageNodeCount * 4 / 5),
MaxTotalThreshold: (planet.config.StorageNodeCount * 4 / 5),
Validate: false,
Min: atLeastOne(planet.config.StorageNodeCount * 1 / 5),
Repair: atLeastOne(planet.config.StorageNodeCount * 2 / 5),
Success: atLeastOne(planet.config.StorageNodeCount * 3 / 5),
Total: atLeastOne(planet.config.StorageNodeCount * 4 / 5),
},
Loop: metainfo.LoopConfig{
CoalesceDuration: 1 * time.Second,

2
satellite/accounting/live/doc.go
View File

@ -4,6 +4,6 @@
/*
Package live provides live accounting functionality. That is, it keeps track
of deltas in the amount of storage used by each project relative to the last
tally operation (see pkg/accounting/tally).
tally operation (see satellite/accounting/tally).
*/
package live

1
satellite/accounting/rollup/rollup_test.go
View File

@ -217,7 +217,6 @@ func dqNodes(ctx *testcontext.Context, planet *testplanet.Planet) (map[storj.Nod
}
updateRequests = append(updateRequests, &overlay.UpdateRequest{
NodeID: n.ID(),
IsUp: true,
AuditOutcome: overlay.AuditFailure,
})
}

21
satellite/accounting/tally/tally.go
View File

@ -84,6 +84,24 @@ func (service *Service) SetNow(now func() time.Time) {
}
// Tally calculates data-at-rest usage once.
//
// How live accounting is calculated:
//
// At the beginning of the tally iteration, we get a map containing the current
// project totals from the cache- initialLiveTotals (our current estimation of
// the project totals). At the end of the tally iteration, we have the totals
// from what we saw during the metainfo loop.
//
// However, data which was uploaded during the loop may or may not have been
// seen in the metainfo loop. For this reason, we also read the live accounting
// totals again at the end of the tally iteration- latestLiveTotals.
//
// The difference between latest and initial indicates how much data was
// uploaded during the metainfo loop and is assigned to delta. However, again,
// we aren't certain how much of the delta is accounted for in the metainfo
// totals. For the reason we make an assumption that 50% of the data is
// accounted for. So to calculate the new live accounting totals, we sum the
// metainfo totals and 50% of the deltas.
func (service *Service) Tally(ctx context.Context) (err error) {
defer mon.Task()(&ctx)(&err)
@ -151,6 +169,9 @@ func (service *Service) Tally(ctx context.Context) (err error) {
if delta < 0 {
delta = 0
}
// read the method documentation why the increase passed to this method
// is calculated in this way
err = service.liveAccounting.AddProjectStorageUsage(ctx, projectID, -latestLiveTotals[projectID]+tallyTotal+(delta/2))
if err != nil {
return Error.Wrap(err)

20
satellite/accounting/tally/tally_test.go
View File

@ -143,7 +143,7 @@ func TestCalculateNodeAtRestData(t *testing.T) {
require.NoError(t, err)
// Confirm the correct number of shares were stored
rs := satelliteRS(planet.Satellites[0])
rs := satelliteRS(t, planet.Satellites[0])
if !correctRedundencyScheme(len(obs.Node), rs) {
t.Fatalf("expected between: %d and %d, actual: %d", rs.RepairShares, rs.TotalShares, len(obs.Node))
}
@ -175,7 +175,7 @@ func TestCalculateBucketAtRestData(t *testing.T) {
SatelliteCount: 1, StorageNodeCount: 6, UplinkCount: 1,
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
satellitePeer := planet.Satellites[0]
redundancyScheme := satelliteRS(satellitePeer)
redundancyScheme := satelliteRS(t, satellitePeer)
expectedBucketTallies := make(map[metabase.BucketLocation]*accounting.BucketTally)
for _, tt := range testCases {
tt := tt // avoid scopelint error
@ -221,7 +221,7 @@ func TestTallyIgnoresExpiredPointers(t *testing.T) {
SatelliteCount: 1, StorageNodeCount: 6, UplinkCount: 1,
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
satellitePeer := planet.Satellites[0]
redundancyScheme := satelliteRS(satellitePeer)
redundancyScheme := satelliteRS(t, satellitePeer)
projectID, err := uuid.FromString("9656af6e-2d9c-42fa-91f2-bfd516a722d7")
require.NoError(t, err)
@ -423,12 +423,14 @@ func correctRedundencyScheme(shareCount int, uplinkRS storj.RedundancyScheme) bo
return int(uplinkRS.RepairShares) <= shareCount && shareCount <= int(uplinkRS.TotalShares)
}
func satelliteRS(satellite *testplanet.Satellite) storj.RedundancyScheme {
func satelliteRS(t *testing.T, satellite *testplanet.Satellite) storj.RedundancyScheme {
rs := satellite.Config.Metainfo.RS
return storj.RedundancyScheme{
RequiredShares: int16(satellite.Config.Metainfo.RS.MinThreshold),
RepairShares: int16(satellite.Config.Metainfo.RS.RepairThreshold),
OptimalShares: int16(satellite.Config.Metainfo.RS.SuccessThreshold),
TotalShares: int16(satellite.Config.Metainfo.RS.TotalThreshold),
ShareSize: satellite.Config.Metainfo.RS.ErasureShareSize.Int32(),
RequiredShares: int16(rs.Min),
RepairShares: int16(rs.Repair),
OptimalShares: int16(rs.Success),
TotalShares: int16(rs.Total),
ShareSize: rs.ErasureShareSize.Int32(),
}
}

1
satellite/audit/disqualification_test.go
View File

@ -214,7 +214,6 @@ func TestDisqualifiedNodeRemainsDisqualified(t *testing.T) {
_, err = satellitePeer.Overlay.Service.BatchUpdateStats(ctx, []*overlay.UpdateRequest{{
NodeID: disqualifiedNode.ID(),
IsUp: true,
AuditOutcome: overlay.AuditSuccess,
AuditLambda: 0, // forget about history
AuditWeight: 1,

5
satellite/audit/reporter.go
View File

@ -128,7 +128,6 @@ func (reporter *Reporter) recordAuditFailStatus(ctx context.Context, failedAudit
for i, nodeID := range failedAuditNodeIDs {
updateRequests[i] = &overlay.UpdateRequest{
NodeID: nodeID,
IsUp: true,
AuditOutcome: overlay.AuditFailure,
}
}
@ -148,7 +147,6 @@ func (reporter *Reporter) recordAuditUnknownStatus(ctx context.Context, unknownA
for i, nodeID := range unknownAuditNodeIDs {
updateRequests[i] = &overlay.UpdateRequest{
NodeID: nodeID,
IsUp: true,
AuditOutcome: overlay.AuditUnknown,
}
}
@ -169,7 +167,6 @@ func (reporter *Reporter) recordOfflineStatus(ctx context.Context, offlineNodeID
for i, nodeID := range offlineNodeIDs {
updateRequests[i] = &overlay.UpdateRequest{
NodeID: nodeID,
IsUp: false,
AuditOutcome: overlay.AuditOffline,
}
}
@ -191,7 +188,6 @@ func (reporter *Reporter) recordAuditSuccessStatus(ctx context.Context, successN
for i, nodeID := range successNodeIDs {
updateRequests[i] = &overlay.UpdateRequest{
NodeID: nodeID,
IsUp: true,
AuditOutcome: overlay.AuditSuccess,
}
}
@ -221,7 +217,6 @@ func (reporter *Reporter) recordPendingAudits(ctx context.Context, pendingAudits
// record failure -- max reverify count reached
updateRequests = append(updateRequests, &overlay.UpdateRequest{
NodeID: pendingAudit.NodeID,
IsUp: true,
AuditOutcome: overlay.AuditFailure,
})
}

10
satellite/audit/reverify_test.go
View File

@ -1074,16 +1074,14 @@ func TestReverifySlowDownload(t *testing.T) {
StorageNodeDB: func(index int, db storagenode.DB, log *zap.Logger) (storagenode.DB, error) {
return testblobs.NewSlowDB(log.Named("slowdb"), db), nil
},
Satellite: func(log *zap.Logger, index int, config *satellite.Config) {
Satellite: testplanet.Combine(
func(log *zap.Logger, index int, config *satellite.Config) {
// These config values are chosen to force the slow node to time out without timing out on the three normal nodes
config.Audit.MinBytesPerSecond = 100 * memory.KiB
config.Audit.MinDownloadTimeout = 1 * time.Second
config.Metainfo.RS.MinThreshold = 2
config.Metainfo.RS.RepairThreshold = 2
config.Metainfo.RS.SuccessThreshold = 4
config.Metainfo.RS.TotalThreshold = 4
},
testplanet.ReconfigureRS(2, 2, 4, 4),
),
},
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
satellite := planet.Satellites[0]

10
satellite/audit/verifier_test.go
View File

@ -776,16 +776,14 @@ func TestVerifierSlowDownload(t *testing.T) {
StorageNodeDB: func(index int, db storagenode.DB, log *zap.Logger) (storagenode.DB, error) {
return testblobs.NewSlowDB(log.Named("slowdb"), db), nil
},
Satellite: func(log *zap.Logger, index int, config *satellite.Config) {
Satellite: testplanet.Combine(
func(log *zap.Logger, index int, config *satellite.Config) {
// These config values are chosen to force the slow node to time out without timing out on the three normal nodes
config.Audit.MinBytesPerSecond = 100 * memory.KiB
config.Audit.MinDownloadTimeout = 950 * time.Millisecond
config.Metainfo.RS.MinThreshold = 2
config.Metainfo.RS.RepairThreshold = 2
config.Metainfo.RS.SuccessThreshold = 4
config.Metainfo.RS.TotalThreshold = 4
},
testplanet.ReconfigureRS(2, 2, 4, 4),
),
},
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
satellite := planet.Satellites[0]

23
satellite/console/consoleweb/consoleapi/auth.go
View File

@ -253,6 +253,29 @@ func (a *Auth) DeleteAccount(w http.ResponseWriter, r *http.Request) {
a.serveJSONError(w, errNotImplemented)
}
// ChangeEmail auth user, changes users email for a new one.
func (a *Auth) ChangeEmail(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
var err error
defer mon.Task()(&ctx)(&err)
var emailChange struct {
NewEmail string `json:"newEmail"`
}
err = json.NewDecoder(r.Body).Decode(&emailChange)
if err != nil {
a.serveJSONError(w, err)
return
}
err = a.service.ChangeEmail(ctx, emailChange.NewEmail)
if err != nil {
a.serveJSONError(w, err)
return
}
}
// ChangePassword auth user, changes users password for a new one.
func (a *Auth) ChangePassword(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()

3
satellite/console/consoleweb/consoleapi/auth_test.go
View File

@ -39,7 +39,6 @@ func TestAuth_Register(t *testing.T) {
},
},
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
for i, test := range []struct {
Partner string
ValidPartner bool
@ -50,6 +49,7 @@ func TestAuth_Register(t *testing.T) {
{Partner: "Raiden nEtwork", ValidPartner: true},
{Partner: "invalid-name", ValidPartner: false},
} {
func() {
registerData := struct {
FullName string `json:"fullName"`
ShortName string `json:"shortName"`
@ -96,6 +96,7 @@ func TestAuth_Register(t *testing.T) {
} else {
require.Equal(t, uuid.UUID{}, user.PartnerID)
}
}()
}
})
}

1
satellite/console/consoleweb/server.go
View File

@ -173,6 +173,7 @@ func NewServer(logger *zap.Logger, config Config, service *console.Service, mail
authRouter := router.PathPrefix("/api/v0/auth").Subrouter()
authRouter.Handle("/account", server.withAuth(http.HandlerFunc(authController.GetAccount))).Methods(http.MethodGet)
authRouter.Handle("/account", server.withAuth(http.HandlerFunc(authController.UpdateAccount))).Methods(http.MethodPatch)
authRouter.Handle("/account/change-email", server.withAuth(http.HandlerFunc(authController.ChangeEmail))).Methods(http.MethodPost)
authRouter.Handle("/account/change-password", server.withAuth(http.HandlerFunc(authController.ChangePassword))).Methods(http.MethodPost)
authRouter.Handle("/account/delete", server.withAuth(http.HandlerFunc(authController.DeleteAccount))).Methods(http.MethodPost)
authRouter.HandleFunc("/logout", authController.Logout).Methods(http.MethodPost)

27
satellite/console/service.go
View File

@ -9,6 +9,7 @@ import (
"database/sql"
"errors"
"fmt"
"net/mail"
"sort"
"time"
@ -867,6 +868,32 @@ func (s *Service) UpdateAccount(ctx context.Context, fullName string, shortName
return nil
}
// ChangeEmail updates email for a given user.
func (s *Service) ChangeEmail(ctx context.Context, newEmail string) (err error) {
defer mon.Task()(&ctx)(&err)
auth, err := s.getAuthAndAuditLog(ctx, "change email")
if err != nil {
return Error.Wrap(err)
}
if _, err := mail.ParseAddress(newEmail); err != nil {
return ErrValidation.Wrap(err)
}
_, err = s.store.Users().GetByEmail(ctx, newEmail)
if err == nil {
return ErrEmailUsed.New(emailUsedErrMsg)
}
auth.User.Email = newEmail
err = s.store.Users().Update(ctx, &auth.User)
if err != nil {
return Error.Wrap(err)
}
return nil
}
// ChangePassword updates password for a given user.
func (s *Service) ChangePassword(ctx context.Context, pass, newPass string) (err error) {
defer mon.Task()(&ctx)(&err)

14
satellite/console/service_test.go
View File

@ -130,5 +130,19 @@ func TestService(t *testing.T) {
require.Error(t, err)
require.Equal(t, "service error: project usage error: some buckets still exist", err.Error())
})
t.Run("TestChangeEmail", func(t *testing.T) {
const newEmail = "newEmail@example.com"
err = service.ChangeEmail(authCtx2, newEmail)
require.NoError(t, err)
userWithUpdatedEmail, err := service.GetUserByEmail(authCtx2, newEmail)
require.NoError(t, err)
require.Equal(t, newEmail, userWithUpdatedEmail.Email)
err = service.ChangeEmail(authCtx2, newEmail)
require.Error(t, err)
})
})
}

35
satellite/console/wasm/README.md Normal file
View File

@ -0,0 +1,35 @@
# Using WebAssembly in Storj
In order to use the uplink library from the browser, we can compile the uplink library to WebAssembly (wasm).
### Setup
To generate wasm code that can create access grants in the web browser, run the following from the storj/wasm directory:
```
$ GOOS=js GOARCH=wasm go build -o access.wasm access.go
```
The `access.wasm` code can then be loaded into the browser in a script tag in an html page. Also needed is a JavaScript support file which ships with golang.
To copy the JavaScript support file, run:
```
$ cp "$(go env GOROOT)/misc/wasm/wasm_exec.js" .
```
Ref: [Golang WebAssembly docs](https://github.com/golang/go/wiki/WebAssembly)
The HTML file should include the following:
```
<script type="text/javascript" src="/path/to/wasm_exec.js"></script>
<script>
const go = new Go();
WebAssembly.instantiateStreaming(
fetch("/path/to/access.wasm"), go.importObject).then(
(result) => {
go.run(result.instance);
});
</script>
```
Additionally, the HTTP `Content-Security-Policy (CSP) script-src` directive will need to be modified to allow wasm code to be executed.
See: [WebAssembly Content Security Policy docs](https://github.com/WebAssembly/content-security-policy/blob/master/proposals/CSP.md)

64
satellite/console/wasm/access.go Normal file
View File

@ -0,0 +1,64 @@
// +build js,wasm
// Copyright (C) 2020 Storj Labs, Inc.
// See LICENSE for copying information.
package main
import (
"fmt"
"syscall/js"
"storj.io/common/encryption"
"storj.io/common/macaroon"
"storj.io/common/storj"
"storj.io/uplink/private/access2"
)
func main() {
js.Global().Set("generateAccessGrant", generateAccessGrant())
<-make(chan bool)
}
func generateAccessGrant() js.Func {
return js.FuncOf(func(this js.Value, args []js.Value) interface{} {
if len(args) < 4 {
return fmt.Sprintf("Error not enough arguments. Need 4, but only %d supplied. The order of arguments are: satellite Node URL, API key, encryption passphrase, and project salt.", len(args))
}
satelliteNodeURL := args[0].String()
apiKey := args[1].String()
encryptionPassphrase := args[2].String()
projectSalt := args[3].String()
return genAccessGrant(satelliteNodeURL,
apiKey,
encryptionPassphrase,
projectSalt,
)
})
}
func genAccessGrant(satelliteNodeURL, apiKey, encryptionPassphrase, projectSalt string) string {
parsedAPIKey, err := macaroon.ParseAPIKey(apiKey)
if err != nil {
return err.Error()
}
const concurrency = 8
key, err := encryption.DeriveRootKey([]byte(encryptionPassphrase), []byte(projectSalt), "", concurrency)
if err != nil {
return err.Error()
}
encAccess := access2.NewEncryptionAccessWithDefaultKey(key)
encAccess.SetDefaultPathCipher(storj.EncAESGCM)
a := &access2.Access{
SatelliteAddress: satelliteNodeURL,
APIKey: parsedAPIKey,
EncAccess: encAccess,
}
accessString, err := a.Serialize()
if err != nil {
return err.Error()
}
return accessString
}

20
satellite/gracefulexit/chore_test.go
View File

@ -32,14 +32,12 @@ func TestChore(t *testing.T) {
StorageNodeCount: 8,
UplinkCount: 1,
Reconfigure: testplanet.Reconfigure{
Satellite: func(log *zap.Logger, index int, config *satellite.Config) {
Satellite: testplanet.Combine(
func(log *zap.Logger, index int, config *satellite.Config) {
config.GracefulExit.MaxInactiveTimeFrame = maximumInactiveTimeFrame
config.Metainfo.RS.MinThreshold = 4
config.Metainfo.RS.RepairThreshold = 6
config.Metainfo.RS.SuccessThreshold = 8
config.Metainfo.RS.TotalThreshold = 8
},
testplanet.ReconfigureRS(4, 6, 8, 8),
),
},
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
uplinkPeer := planet.Uplinks[0]
@ -136,14 +134,12 @@ func TestDurabilityRatio(t *testing.T) {
StorageNodeCount: 4,
UplinkCount: 1,
Reconfigure: testplanet.Reconfigure{
Satellite: func(log *zap.Logger, index int, config *satellite.Config) {
Satellite: testplanet.Combine(
func(log *zap.Logger, index int, config *satellite.Config) {
config.GracefulExit.MaxInactiveTimeFrame = maximumInactiveTimeFrame
config.Metainfo.RS.MinThreshold = 2
config.Metainfo.RS.RepairThreshold = 3
config.Metainfo.RS.SuccessThreshold = successThreshold
config.Metainfo.RS.TotalThreshold = 4
},
testplanet.ReconfigureRS(2, 3, successThreshold, 4),
),
},
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
uplinkPeer := planet.Uplinks[0]

30
satellite/gracefulexit/endpoint_test.go
View File

@ -255,16 +255,14 @@ func TestRecvTimeout(t *testing.T) {
StorageNodeDB: func(index int, db storagenode.DB, log *zap.Logger) (storagenode.DB, error) {
return testblobs.NewSlowDB(log.Named("slowdb"), db), nil
},
Satellite: func(logger *zap.Logger, index int, config *satellite.Config) {
Satellite: testplanet.Combine(
func(log *zap.Logger, index int, config *satellite.Config) {
// This config value will create a very short timeframe allowed for receiving
// data from storage nodes. This will cause context to cancel with timeout.
config.GracefulExit.RecvTimeout = 10 * time.Millisecond
config.Metainfo.RS.MinThreshold = 2
config.Metainfo.RS.RepairThreshold = 3
config.Metainfo.RS.SuccessThreshold = successThreshold
config.Metainfo.RS.TotalThreshold = successThreshold
},
testplanet.ReconfigureRS(2, 3, successThreshold, successThreshold),
),
StorageNode: func(index int, config *storagenode.Config) {
config.GracefulExit = gracefulexit.Config{
ChoreInterval: 2 * time.Minute,
@ -1240,17 +1238,15 @@ func TestFailureStorageNodeIgnoresTransferMessages(t *testing.T) {
StorageNodeCount: 5,
UplinkCount: 1,
Reconfigure: testplanet.Reconfigure{
Satellite: func(logger *zap.Logger, index int, config *satellite.Config) {
Satellite: testplanet.Combine(
func(log *zap.Logger, index int, config *satellite.Config) {
// We don't care whether a node gracefully exits or not in this test,
// so we set the max failures percentage extra high.
config.GracefulExit.OverallMaxFailuresPercentage = 101
config.GracefulExit.MaxOrderLimitSendCount = maxOrderLimitSendCount
config.Metainfo.RS.MinThreshold = 2
config.Metainfo.RS.RepairThreshold = 3
config.Metainfo.RS.SuccessThreshold = 4
config.Metainfo.RS.TotalThreshold = 4
},
testplanet.ReconfigureRS(2, 3, 4, 4),
),
},
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
uplinkPeer := planet.Uplinks[0]
@ -1370,15 +1366,13 @@ func TestIneligibleNodeAge(t *testing.T) {
StorageNodeCount: 5,
UplinkCount: 1,
Reconfigure: testplanet.Reconfigure{
Satellite: func(logger *zap.Logger, index int, config *satellite.Config) {
Satellite: testplanet.Combine(
func(log *zap.Logger, index int, config *satellite.Config) {
// Set the required node age to 1 month.
config.GracefulExit.NodeMinAgeInMonths = 1
config.Metainfo.RS.MinThreshold = 2
config.Metainfo.RS.RepairThreshold = 3
config.Metainfo.RS.SuccessThreshold = 4
config.Metainfo.RS.TotalThreshold = 4
},
testplanet.ReconfigureRS(2, 3, 4, 4),
),
},
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
uplinkPeer := planet.Uplinks[0]

78
satellite/metainfo/config.go
View File

@ -5,7 +5,10 @@ package metainfo
import (
"context"
"fmt"
"io"
"strconv"
"strings"
"time"
"go.uber.org/zap"
@ -26,18 +29,73 @@ const (
// RSConfig is a configuration struct that keeps details about default
// redundancy strategy information.
//
// Can be used as a flag.
type RSConfig struct {
MaxBufferMem memory.Size `help:"maximum buffer memory to be allocated for read buffers" default:"4.00MiB"`
ErasureShareSize memory.Size `help:"the size of each new erasure share in bytes" default:"256B"`
MinThreshold int `help:"the minimum pieces required to recover a segment. k." releaseDefault:"29" devDefault:"4"`
RepairThreshold int `help:"the minimum safe pieces before a repair is triggered. m." releaseDefault:"35" devDefault:"6"`
SuccessThreshold int `help:"the desired total pieces for a segment. o." releaseDefault:"80" devDefault:"8"`
TotalThreshold int `help:"the largest amount of pieces to encode to. n." releaseDefault:"110" devDefault:"10"`
ErasureShareSize memory.Size
Min int
Repair int
Success int
Total int
}
// TODO left for validation until we will remove CreateSegmentOld
MinTotalThreshold int `help:"the largest amount of pieces to encode to. n (lower bound for validation)." releaseDefault:"95" devDefault:"10"`
MaxTotalThreshold int `help:"the largest amount of pieces to encode to. n (upper bound for validation)." releaseDefault:"130" devDefault:"10"`
Validate bool `help:"validate redundancy scheme configuration" default:"true"`
// Type implements pflag.Value.
func (RSConfig) Type() string { return "metainfo.RSConfig" }
// String is required for pflag.Value.
func (rs *RSConfig) String() string {
return fmt.Sprintf("%d/%d/%d/%d-%s",
rs.Min,
rs.Repair,
rs.Success,
rs.Total,
rs.ErasureShareSize.String())
}
// Set sets the value from a string in the format k/m/o/n-size (min/repair/optimal/total-erasuresharesize).
func (rs *RSConfig) Set(s string) error {
// Split on dash. Expect two items. First item is RS numbers. Second item is memory.Size.
info := strings.Split(s, "-")
if len(info) != 2 {
return Error.New("Invalid default RS config (expect format k/m/o/n-ShareSize, got %s)", s)
}
rsNumbersString := info[0]
shareSizeString := info[1]
// Attempt to parse "-size" part of config.
shareSizeInt, err := memory.ParseString(shareSizeString)
if err != nil {
return Error.New("Invalid share size in RS config: '%s', %w", shareSizeString, err)
}
shareSize := memory.Size(shareSizeInt)
// Split on forward slash. Expect exactly four positive non-decreasing integers.
rsNumbers := strings.Split(rsNumbersString, "/")
if len(rsNumbers) != 4 {
return Error.New("Invalid default RS numbers (wrong size, expect 4): %s", rsNumbersString)
}
minValue := 1
values := []int{}
for _, nextValueString := range rsNumbers {
nextValue, err := strconv.Atoi(nextValueString)
if err != nil {
return Error.New("Invalid default RS numbers (should all be valid integers): %s, %w", rsNumbersString, err)
}
if nextValue < minValue {
return Error.New("Invalid default RS numbers (should be non-decreasing): %s", rsNumbersString)
}
values = append(values, nextValue)
minValue = nextValue
}
rs.ErasureShareSize = shareSize
rs.Min = values[0]
rs.Repair = values[1]
rs.Success = values[2]
rs.Total = values[3]
return nil
}
// RateLimiterConfig is a configuration struct for endpoint rate limiting.

98
satellite/metainfo/config_test.go Normal file
View File

@ -0,0 +1,98 @@
// Copyright (C) 2020 Storj Labs, Inc.
// See LICENSE for copying information.
package metainfo_test
import (
"testing"
"github.com/stretchr/testify/require"
"storj.io/common/memory"
"storj.io/storj/satellite/metainfo"
)
func TestRSConfigValidation(t *testing.T) {
tests := []struct {
description string
configString string
expectedConfig metainfo.RSConfig
expectError bool
}{
{
description: "valid rs config",
configString: "4/8/10/20-256B",
expectedConfig: metainfo.RSConfig{
ErasureShareSize: 256 * memory.B, Min: 4, Repair: 8, Success: 10, Total: 20,
},
expectError: false,
},
{
description: "invalid rs config - numbers decrease",
configString: "4/8/5/20-256B",
expectError: true,
},
{
description: "invalid rs config - starts at 0",
configString: "0/2/4/6-256B",
expectError: true,
},
{
description: "invalid rs config - strings",
configString: "4/a/b/20-256B",
expectError: true,
},
{
description: "invalid rs config - floating-point numbers",
configString: "4/5.2/7/20-256B",
expectError: true,
},
{
description: "invalid rs config - not enough items",
configString: "4/5/20-256B",
expectError: true,
},
{
description: "invalid rs config - too many items",
configString: "4/5/20/30/50-256B",
expectError: true,
},
{
description: "invalid rs config - empty numbers",
configString: "-256B",
expectError: true,
},
{
description: "invalid rs config - empty size",
configString: "1/2/3/4-",
expectError: true,
},
{
description: "invalid rs config - empty",
configString: "",
expectError: true,
},
{
description: "invalid valid rs config - invalid share size",
configString: "4/8/10/20-256A",
expectError: true,
},
}
for _, tt := range tests {
t.Log(tt.description)
rsConfig := metainfo.RSConfig{}
err := rsConfig.Set(tt.configString)
if tt.expectError {
require.Error(t, err)
} else {
require.NoError(t, err)
require.EqualValues(t, tt.expectedConfig.ErasureShareSize, rsConfig.ErasureShareSize)
require.EqualValues(t, tt.expectedConfig.Min, rsConfig.Min)
require.EqualValues(t, tt.expectedConfig.Repair, rsConfig.Repair)
require.EqualValues(t, tt.expectedConfig.Success, rsConfig.Success)
require.EqualValues(t, tt.expectedConfig.Total, rsConfig.Total)
}
}
}

31
satellite/metainfo/metainfo.go
View File

@ -79,6 +79,7 @@ type Endpoint struct {
limiterCache *lrucache.ExpiringLRU
encInlineSegmentSize int64 // max inline segment size + encryption overhead
revocations revocation.DB
defaultRS *pb.RedundancyScheme
config Config
}
@ -97,6 +98,16 @@ func NewEndpoint(log *zap.Logger, metainfo *Service, deletePieces *piecedeletion
if err != nil {
return nil, err
}
defaultRSScheme := &pb.RedundancyScheme{
Type: pb.RedundancyScheme_RS,
MinReq: int32(config.RS.Min),
RepairThreshold: int32(config.RS.Repair),
SuccessThreshold: int32(config.RS.Success),
Total: int32(config.RS.Total),
ErasureShareSize: config.RS.ErasureShareSize.Int32(),
}
return &Endpoint{
log: log,
metainfo: metainfo,
@ -116,6 +127,7 @@ func NewEndpoint(log *zap.Logger, metainfo *Service, deletePieces *piecedeletion
}),
encInlineSegmentSize: encInlineSegmentSize,
revocations: revocations,
defaultRS: defaultRSScheme,
config: config,
}, nil
}
@ -241,7 +253,7 @@ func (endpoint *Endpoint) GetBucket(ctx context.Context, req *pb.BucketGetReques
}
// override RS to fit satellite settings
convBucket, err := convertBucketToProto(bucket, endpoint.redundancyScheme())
convBucket, err := convertBucketToProto(bucket, endpoint.defaultRS)
if err != nil {
return resp, err
}
@ -316,7 +328,7 @@ func (endpoint *Endpoint) CreateBucket(ctx context.Context, req *pb.BucketCreate
}
// override RS to fit satellite settings
convBucket, err := convertBucketToProto(bucket, endpoint.redundancyScheme())
convBucket, err := convertBucketToProto(bucket, endpoint.defaultRS)
if err != nil {
endpoint.log.Error("error while converting bucket to proto", zap.String("bucketName", bucket.Name), zap.Error(err))
return nil, rpcstatus.Error(rpcstatus.Internal, "unable to create bucket")
@ -375,7 +387,7 @@ func (endpoint *Endpoint) DeleteBucket(ctx context.Context, req *pb.BucketDelete
return nil, err
}
convBucket, err = convertBucketToProto(bucket, endpoint.redundancyScheme())
convBucket, err = convertBucketToProto(bucket, endpoint.defaultRS)
if err != nil {
return nil, err
}
@ -657,7 +669,7 @@ func (endpoint *Endpoint) BeginObject(ctx context.Context, req *pb.ObjectBeginRe
}
// use only satellite values for Redundancy Scheme
pbRS := endpoint.redundancyScheme()
pbRS := endpoint.defaultRS
streamID, err := endpoint.packStreamID(ctx, &internalpb.StreamID{
Bucket: req.Bucket,
@ -1825,17 +1837,6 @@ func groupPiecesByNodeID(segments []metabase.DeletedSegmentInfo) map[storj.NodeI
return piecesToDelete
}
func (endpoint *Endpoint) redundancyScheme() *pb.RedundancyScheme {
return &pb.RedundancyScheme{
Type: pb.RedundancyScheme_RS,
MinReq: int32(endpoint.config.RS.MinThreshold),
RepairThreshold: int32(endpoint.config.RS.RepairThreshold),
SuccessThreshold: int32(endpoint.config.RS.SuccessThreshold),
Total: int32(endpoint.config.RS.TotalThreshold),
ErasureShareSize: endpoint.config.RS.ErasureShareSize.Int32(),
}
}
// RevokeAPIKey handles requests to revoke an api key.
func (endpoint *Endpoint) RevokeAPIKey(ctx context.Context, req *pb.RevokeAPIKeyRequest) (resp *pb.RevokeAPIKeyResponse, err error) {
defer mon.Task()(&ctx)(&err)

2
satellite/metainfo/metainfo_test.go
View File

@ -215,6 +215,7 @@ func TestInvalidAPIKey(t *testing.T) {
require.NoError(t, err)
for _, invalidAPIKey := range []string{"", "invalid", "testKey"} {
func() {
client, err := planet.Uplinks[0].DialMetainfo(ctx, planet.Satellites[0], throwawayKey)
require.NoError(t, err)
defer ctx.Check(client.Close)
@ -294,6 +295,7 @@ func TestInvalidAPIKey(t *testing.T) {
err = client.CommitSegment(ctx, metainfo.CommitSegmentParams{SegmentID: segmentID})
assertInvalidArgument(t, err, false)
}()
}
})
}

9
satellite/metainfo/piecedeletion/service_test.go
View File

@ -346,14 +346,13 @@ func TestService_DeletePieces_Timeout(t *testing.T) {
StorageNodeDB: func(index int, db storagenode.DB, log *zap.Logger) (storagenode.DB, error) {
return testblobs.NewSlowDB(log.Named("slowdb"), db), nil
},
Satellite: func(log *zap.Logger, index int, config *satellite.Config) {
Satellite: testplanet.Combine(
func(log *zap.Logger, index int, config *satellite.Config) {
config.Metainfo.PieceDeletion.RequestTimeout = 200 * time.Millisecond
config.Metainfo.RS.MinThreshold = 2
config.Metainfo.RS.RepairThreshold = 2
config.Metainfo.RS.SuccessThreshold = 4
config.Metainfo.RS.TotalThreshold = 4
config.Metainfo.MaxSegmentSize = 15 * memory.KiB
},
testplanet.ReconfigureRS(2, 2, 4, 4),
),
},
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
uplnk := planet.Uplinks[0]

4
satellite/orders/endpoint_test.go
View File

@ -76,6 +76,7 @@ func TestSettlementWithWindowEndpointManyOrders(t *testing.T) {
}
for _, tt := range testCases {
func() {
// create serial number to use in test. must be unique for each run.
serialNumber1 := testrand.SerialNumber()
err = ordersDB.CreateSerialInfo(ctx, serialNumber1, []byte(bucketID), now.AddDate(1, 0, 10))
@ -176,6 +177,7 @@ func TestSettlementWithWindowEndpointManyOrders(t *testing.T) {
newBbw, err := ordersDB.GetBucketBandwidth(ctx, projectID, []byte(bucketname), time.Time{}, tt.orderCreation)
require.NoError(t, err)
require.EqualValues(t, tt.settledAmt, newBbw)
}()
}
})
}
@ -223,6 +225,7 @@ func TestSettlementWithWindowEndpointSingleOrder(t *testing.T) {
}
for _, tt := range testCases {
func() {
// create signed orderlimit or order to test with
limit := &pb.OrderLimit{
SerialNumber: serialNumber,
@ -289,6 +292,7 @@ func TestSettlementWithWindowEndpointSingleOrder(t *testing.T) {
newBbw, err := ordersDB.GetBucketBandwidth(ctx, projectID, []byte(bucketname), time.Time{}, now)
require.NoError(t, err)
require.Equal(t, dataAmount, newBbw)
}()
}
})
}

6
satellite/orders/service.go
View File

@ -566,7 +566,11 @@ func (service *Service) CreateGracefulExitPutOrderLimit(ctx context.Context, buc
return nil, storj.PiecePrivateKey{}, Error.Wrap(err)
}
nodeURL := storj.NodeURL{ID: nodeID, Address: node.Address.Address}
address := node.Address.Address
if node.LastIPPort != "" {
address = node.LastIPPort
}
nodeURL := storj.NodeURL{ID: nodeID, Address: address}
limit, err = signer.Sign(ctx, nodeURL, pieceNum)
if err != nil {
return nil, storj.PiecePrivateKey{}, Error.Wrap(err)

101
satellite/overlay/benchmark_test.go
View File

@ -89,35 +89,106 @@ func BenchmarkOverlay(b *testing.B) {
}
})
b.Run("UpdateStats", func(b *testing.B) {
b.Run("UpdateStatsSuccess", func(b *testing.B) {
for i := 0; i < b.N; i++ {
id := all[i%len(all)]
outcome := overlay.AuditFailure
if i&1 == 0 {
outcome = overlay.AuditSuccess
}
_, err := overlaydb.UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: id,
AuditOutcome: outcome,
IsUp: i&2 == 0,
AuditOutcome: overlay.AuditSuccess,
AuditHistory: testAuditHistoryConfig(),
}, time.Now())
require.NoError(b, err)
}
})
b.Run("BatchUpdateStats", func(b *testing.B) {
b.Run("UpdateStatsFailure", func(b *testing.B) {
for i := 0; i < b.N; i++ {
id := all[i%len(all)]
_, err := overlaydb.UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: id,
AuditOutcome: overlay.AuditFailure,
AuditHistory: testAuditHistoryConfig(),
}, time.Now())
require.NoError(b, err)
}
})
b.Run("UpdateStatsUnknown", func(b *testing.B) {
for i := 0; i < b.N; i++ {
id := all[i%len(all)]
_, err := overlaydb.UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: id,
AuditOutcome: overlay.AuditUnknown,
AuditHistory: testAuditHistoryConfig(),
}, time.Now())
require.NoError(b, err)
}
})
b.Run("UpdateStatsOffline", func(b *testing.B) {
for i := 0; i < b.N; i++ {
id := all[i%len(all)]
_, err := overlaydb.UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: id,
AuditOutcome: overlay.AuditOffline,
AuditHistory: testAuditHistoryConfig(),
}, time.Now())
require.NoError(b, err)
}
})
b.Run("BatchUpdateStatsSuccess", func(b *testing.B) {
var updateRequests []*overlay.UpdateRequest
for i := 0; i < b.N; i++ {
id := all[i%len(all)]
outcome := overlay.AuditFailure
if i&1 == 0 {
outcome = overlay.AuditSuccess
}
updateRequests = append(updateRequests, &overlay.UpdateRequest{
NodeID: id,
AuditOutcome: outcome,
IsUp: i&2 == 0,
AuditOutcome: overlay.AuditSuccess,
AuditHistory: testAuditHistoryConfig(),
})
}
_, err := overlaydb.BatchUpdateStats(ctx, updateRequests, 100, time.Now())
require.NoError(b, err)
})
b.Run("BatchUpdateStatsFailure", func(b *testing.B) {
var updateRequests []*overlay.UpdateRequest
for i := 0; i < b.N; i++ {
id := all[i%len(all)]
updateRequests = append(updateRequests, &overlay.UpdateRequest{
NodeID: id,
AuditOutcome: overlay.AuditFailure,
AuditHistory: testAuditHistoryConfig(),
})
}
_, err := overlaydb.BatchUpdateStats(ctx, updateRequests, 100, time.Now())
require.NoError(b, err)
})
b.Run("BatchUpdateStatsUnknown", func(b *testing.B) {
var updateRequests []*overlay.UpdateRequest
for i := 0; i < b.N; i++ {
id := all[i%len(all)]
updateRequests = append(updateRequests, &overlay.UpdateRequest{
NodeID: id,
AuditOutcome: overlay.AuditUnknown,
AuditHistory: testAuditHistoryConfig(),
})
}
_, err := overlaydb.BatchUpdateStats(ctx, updateRequests, 100, time.Now())
require.NoError(b, err)
})
b.Run("BatchUpdateStatsOffline", func(b *testing.B) {
var updateRequests []*overlay.UpdateRequest
for i := 0; i < b.N; i++ {
id := all[i%len(all)]
updateRequests = append(updateRequests, &overlay.UpdateRequest{
NodeID: id,
AuditOutcome: overlay.AuditOffline,
AuditHistory: testAuditHistoryConfig(),
})
@ -272,7 +343,6 @@ func BenchmarkNodeSelection(b *testing.B) {
if i%2 == 0 { // make half of nodes "new" and half "vetted"
_, err = overlaydb.UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: nodeID,
IsUp: true,
AuditOutcome: overlay.AuditSuccess,
AuditLambda: 1,
AuditWeight: 1,
@ -293,7 +363,6 @@ func BenchmarkNodeSelection(b *testing.B) {
case 2:
err := overlaydb.UpdateCheckIn(ctx, overlay.NodeCheckInInfo{
NodeID: nodeID,
IsUp: true,
Address: &pb.NodeAddress{
Address: address,
},

1
satellite/overlay/nodeselectioncache_test.go
View File

@ -104,7 +104,6 @@ func addNodesToNodesTable(ctx context.Context, t *testing.T, db overlay.DB, coun
if i < makeReputable {
stats, err := db.UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: storj.NodeID{byte(i)},
IsUp: true,
AuditOutcome: overlay.AuditSuccess,
AuditLambda: 1, AuditWeight: 1, AuditDQ: 0.5,
AuditHistory: testAuditHistoryConfig(),

5
satellite/overlay/selection_test.go
View File

@ -188,7 +188,6 @@ func TestEnsureMinimumRequested(t *testing.T) {
reputable[node.ID()] = true
_, err := satellite.DB.OverlayCache().UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: node.ID(),
IsUp: true,
AuditOutcome: overlay.AuditSuccess,
AuditLambda: 1, AuditWeight: 1, AuditDQ: 0.5,
AuditHistory: testAuditHistoryConfig(),
@ -232,7 +231,6 @@ func TestEnsureMinimumRequested(t *testing.T) {
reputable[node.ID()] = true
_, err := satellite.DB.OverlayCache().UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: node.ID(),
IsUp: true,
AuditOutcome: overlay.AuditSuccess,
AuditLambda: 1, AuditWeight: 1, AuditDQ: 0.5,
AuditHistory: testAuditHistoryConfig(),
@ -397,7 +395,6 @@ func TestNodeSelectionGracefulExit(t *testing.T) {
for k := 0; k < i; k++ {
_, err := satellite.DB.OverlayCache().UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: node.ID(),
IsUp: true,
AuditOutcome: overlay.AuditSuccess,
AuditLambda: 1, AuditWeight: 1, AuditDQ: 0.5,
AuditHistory: testAuditHistoryConfig(),
@ -628,7 +625,6 @@ func TestDistinctIPs(t *testing.T) {
for i := 9; i > 7; i-- {
_, err := satellite.DB.OverlayCache().UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: planet.StorageNodes[i].ID(),
IsUp: true,
AuditOutcome: overlay.AuditSuccess,
AuditLambda: 1,
AuditWeight: 1,
@ -660,7 +656,6 @@ func TestDistinctIPsWithBatch(t *testing.T) {
// These are done individually b/c the previous stat data is important
_, err := satellite.Overlay.Service.BatchUpdateStats(ctx, []*overlay.UpdateRequest{{
NodeID: planet.StorageNodes[i].ID(),
IsUp: true,
AuditOutcome: overlay.AuditSuccess,
AuditLambda: 1,
AuditWeight: 1,

1
satellite/overlay/service.go
View File

@ -164,7 +164,6 @@ const (
type UpdateRequest struct {
NodeID storj.NodeID
AuditOutcome AuditType
IsUp bool
// n.b. these are set values from the satellite.
// They are part of the UpdateRequest struct in order to be
// more easily accessible in satellite/satellitedb/overlaycache.go.

7
satellite/overlay/service_test.go
View File

@ -131,7 +131,6 @@ func testCache(ctx context.Context, t *testing.T, store overlay.DB) {
stats, err := service.UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: valid1ID,
IsUp: true,
AuditOutcome: overlay.AuditFailure,
})
require.NoError(t, err)
@ -148,7 +147,6 @@ func testCache(ctx context.Context, t *testing.T, store overlay.DB) {
// should not update once already disqualified
_, err = service.BatchUpdateStats(ctx, []*overlay.UpdateRequest{{
NodeID: valid2ID,
IsUp: false,
AuditOutcome: overlay.AuditSuccess,
}})
require.NoError(t, err)
@ -197,7 +195,6 @@ func TestRandomizedSelection(t *testing.T) {
if i%2 == 0 { // make half of nodes "new" and half "vetted"
_, err = cache.UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: newID,
IsUp: true,
AuditOutcome: overlay.AuditSuccess,
AuditLambda: 1,
AuditWeight: 1,
@ -316,7 +313,6 @@ func TestRandomizedSelectionCache(t *testing.T) {
if i%2 == 0 { // make half of nodes "new" and half "vetted"
_, err = overlaydb.UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: newID,
IsUp: true,
AuditOutcome: overlay.AuditSuccess,
AuditLambda: 1,
AuditWeight: 1,
@ -775,7 +771,6 @@ func TestSuspendedSelection(t *testing.T) {
if i%2 == 0 { // make half of nodes "new" and half "vetted"
_, err = cache.UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: newID,
IsUp: true,
AuditOutcome: overlay.AuditSuccess,
AuditLambda: 1,
AuditWeight: 1,
@ -835,7 +830,6 @@ func TestConcurrentAudit(t *testing.T) {
_, err := planet.Satellites[0].Overlay.Service.UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: planet.StorageNodes[0].ID(),
AuditOutcome: overlay.AuditSuccess,
IsUp: true,
})
return err
})
@ -853,7 +847,6 @@ func TestConcurrentAudit(t *testing.T) {
{
NodeID: planet.StorageNodes[0].ID(),
AuditOutcome: overlay.AuditSuccess,
IsUp: true,
},
})
return err

2
satellite/overlay/statdb_test.go
View File

@ -169,7 +169,6 @@ func testDatabase(ctx context.Context, t *testing.T, cache overlay.DB) {
updateReq := &overlay.UpdateRequest{
NodeID: nodeID,
AuditOutcome: overlay.AuditSuccess,
IsUp: true,
AuditLambda: 0.123, AuditWeight: 0.456,
AuditDQ: 0, // don't disqualify for any reason
AuditHistory: testAuditHistoryConfig(),
@ -186,7 +185,6 @@ func testDatabase(ctx context.Context, t *testing.T, cache overlay.DB) {
auditBeta = expectedAuditBeta
updateReq.AuditOutcome = overlay.AuditFailure
updateReq.IsUp = false
stats, err = cache.UpdateStats(ctx, updateReq, time.Now())
require.NoError(t, err)

14
satellite/overlay/suspension_test.go
View File

@ -66,7 +66,6 @@ func TestAuditSuspendWithUpdateStats(t *testing.T) {
_, err = oc.UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: nodeID,
AuditOutcome: overlay.AuditUnknown,
IsUp: true,
AuditLambda: 1,
AuditWeight: 1,
AuditDQ: 0.6,
@ -90,7 +89,6 @@ func TestAuditSuspendWithUpdateStats(t *testing.T) {
_, err = oc.UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: nodeID,
AuditOutcome: overlay.AuditSuccess,
IsUp: true,
AuditLambda: 1,
AuditWeight: 1,
AuditDQ: 0.6,
@ -121,7 +119,6 @@ func TestAuditSuspendFailedAudit(t *testing.T) {
_, err = oc.UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: nodeID,
AuditOutcome: overlay.AuditFailure,
IsUp: true,
AuditLambda: 1,
AuditWeight: 1,
AuditDQ: 0.6,
@ -277,7 +274,6 @@ func TestAuditSuspendBatchUpdateStats(t *testing.T) {
nodeUpdateReq := &overlay.UpdateRequest{
NodeID: nodeID,
AuditOutcome: overlay.AuditSuccess,
IsUp: true,
AuditLambda: 1,
AuditWeight: 1,
AuditDQ: 0.6,
@ -343,7 +339,6 @@ func TestOfflineSuspend(t *testing.T) {
updateReq := &overlay.UpdateRequest{
NodeID: nodeID,
AuditOutcome: overlay.AuditOffline,
IsUp: false,
AuditHistory: overlay.AuditHistoryConfig{
WindowSize: time.Hour,
TrackingPeriod: 2 * time.Hour,
@ -472,12 +467,11 @@ func setOnlineScore(ctx context.Context, reqPtr *overlay.UpdateRequest, desiredS
for window := 0; window < windowsPerTrackingPeriod+1; window++ {
updateReqs := []*overlay.UpdateRequest{}
for i := 0; i < totalAudits; i++ {
isUp := true
if i >= onlineAudits {
isUp = false
}
updateReq := *reqPtr
updateReq.IsUp = isUp
updateReq.AuditOutcome = overlay.AuditSuccess
if i >= onlineAudits {
updateReq.AuditOutcome = overlay.AuditOffline
}
updateReq.AuditHistory.GracePeriod = gracePeriod
updateReqs = append(updateReqs, &updateReq)

2
satellite/repair/checker/checker.go
View File

@ -391,7 +391,7 @@ func (obs *checkerObserver) InlineSegment(ctx context.Context, segment *metainfo
func (checker *Checker) IrreparableProcess(ctx context.Context) (err error) {
defer mon.Task()(&ctx)(&err)
const limit = 1000
var lastSeenSegmentKey metabase.SegmentKey
lastSeenSegmentKey := metabase.SegmentKey{}
for {
segments, err := checker.irrdb.GetLimited(ctx, limit, lastSeenSegmentKey)

87
satellite/repair/irreparable/irreparable_test.go
View File

@ -13,8 +13,10 @@ import (
"storj.io/common/pb"
"storj.io/common/testcontext"
"storj.io/storj/private/testplanet"
"storj.io/storj/satellite"
"storj.io/storj/satellite/internalpb"
"storj.io/storj/satellite/metainfo/metabase"
"storj.io/storj/satellite/satellitedb/satellitedbtest"
)
@ -103,3 +105,88 @@ func TestIrreparable(t *testing.T) {
}
})
}
func TestIrreparableProcess(t *testing.T) {
testplanet.Run(t, testplanet.Config{
SatelliteCount: 1, StorageNodeCount: 3, UplinkCount: 0,
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
checker := planet.Satellites[0].Repair.Checker
checker.Loop.Stop()
checker.IrreparableLoop.Stop()
irreparabledb := planet.Satellites[0].DB.Irreparable()
queue := planet.Satellites[0].DB.RepairQueue()
seg := &internalpb.IrreparableSegment{
Path: []byte{1},
SegmentDetail: &pb.Pointer{
Type: pb.Pointer_REMOTE,
CreationDate: time.Now(),
Remote: &pb.RemoteSegment{
Redundancy: &pb.RedundancyScheme{
MinReq: 1,
RepairThreshold: 2,
SuccessThreshold: 3,
Total: 4,
},
RemotePieces: []*pb.RemotePiece{
{
NodeId: planet.StorageNodes[0].ID(),
},
{
NodeId: planet.StorageNodes[1].ID(),
},
{
NodeId: planet.StorageNodes[2].ID(),
},
},
},
},
LostPieces: int32(4),
LastRepairAttempt: time.Now().Unix(),
RepairAttemptCount: int64(10),
}
require.NoError(t, irreparabledb.IncrementRepairAttempts(ctx, seg))
result, err := irreparabledb.Get(ctx, metabase.SegmentKey(seg.GetPath()))
require.NoError(t, err)
require.NotNil(t, result)
// test healthy segment is removed from irreparable DB
require.NoError(t, checker.IrreparableProcess(ctx))
result, err = irreparabledb.Get(ctx, metabase.SegmentKey(seg.GetPath()))
require.Error(t, err)
require.Nil(t, result)
// test unhealthy repairable segment is removed from irreparable DB and inserted into repair queue
seg.SegmentDetail.Remote.RemotePieces[0] = &pb.RemotePiece{}
seg.SegmentDetail.Remote.RemotePieces[1] = &pb.RemotePiece{}
require.NoError(t, irreparabledb.IncrementRepairAttempts(ctx, seg))
require.NoError(t, checker.IrreparableProcess(ctx))
result, err = irreparabledb.Get(ctx, metabase.SegmentKey(seg.GetPath()))
require.Error(t, err)
require.Nil(t, result)
injured, err := queue.Select(ctx)
require.NoError(t, err)
require.Equal(t, seg.GetPath(), injured.GetPath())
n, err := queue.Clean(ctx, time.Now())
require.NoError(t, err)
require.EqualValues(t, 1, n)
// test irreparable segment remains in irreparable DB and repair_attempt_count is incremented
seg.SegmentDetail.Remote.RemotePieces[2] = &pb.RemotePiece{}
require.NoError(t, irreparabledb.IncrementRepairAttempts(ctx, seg))
require.NoError(t, checker.IrreparableProcess(ctx))
result, err = irreparabledb.Get(ctx, metabase.SegmentKey(seg.GetPath()))
require.NoError(t, err)
require.Equal(t, seg.GetPath(), result.Path)
require.Equal(t, seg.RepairAttemptCount+1, result.RepairAttemptCount)
})
}

84
satellite/repair/repair_test.go
View File

@ -53,15 +53,13 @@ func testDataRepair(t *testing.T, inMemoryRepair bool) {
StorageNodeCount: 14,
UplinkCount: 1,
Reconfigure: testplanet.Reconfigure{
Satellite: func(log *zap.Logger, index int, config *satellite.Config) {
Satellite: testplanet.Combine(
func(log *zap.Logger, index int, config *satellite.Config) {
config.Repairer.MaxExcessRateOptimalThreshold = RepairMaxExcessRateOptimalThreshold
config.Repairer.InMemoryRepair = inMemoryRepair
config.Metainfo.RS.MinThreshold = minThreshold
config.Metainfo.RS.RepairThreshold = 5
config.Metainfo.RS.SuccessThreshold = successThreshold
config.Metainfo.RS.TotalThreshold = 9
},
testplanet.ReconfigureRS(minThreshold, 5, successThreshold, 9),
),
},
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
@ -188,15 +186,13 @@ func testCorruptDataRepairFailed(t *testing.T, inMemoryRepair bool) {
StorageNodeCount: 14,
UplinkCount: 1,
Reconfigure: testplanet.Reconfigure{
Satellite: func(log *zap.Logger, index int, config *satellite.Config) {
Satellite: testplanet.Combine(
func(log *zap.Logger, index int, config *satellite.Config) {
config.Repairer.MaxExcessRateOptimalThreshold = RepairMaxExcessRateOptimalThreshold
config.Repairer.InMemoryRepair = inMemoryRepair
config.Metainfo.RS.MinThreshold = 3
config.Metainfo.RS.RepairThreshold = 5
config.Metainfo.RS.SuccessThreshold = 7
config.Metainfo.RS.TotalThreshold = 9
},
testplanet.ReconfigureRS(3, 5, 7, 9),
),
},
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
uplinkPeer := planet.Uplinks[0]
@ -305,15 +301,13 @@ func testCorruptDataRepairSucceed(t *testing.T, inMemoryRepair bool) {
StorageNodeCount: 14,
UplinkCount: 1,
Reconfigure: testplanet.Reconfigure{
Satellite: func(log *zap.Logger, index int, config *satellite.Config) {
Satellite: testplanet.Combine(
func(log *zap.Logger, index int, config *satellite.Config) {
config.Repairer.MaxExcessRateOptimalThreshold = RepairMaxExcessRateOptimalThreshold
config.Repairer.InMemoryRepair = inMemoryRepair
config.Metainfo.RS.MinThreshold = 3
config.Metainfo.RS.RepairThreshold = 5
config.Metainfo.RS.SuccessThreshold = 7
config.Metainfo.RS.TotalThreshold = 9
},
testplanet.ReconfigureRS(3, 5, 7, 9),
),
},
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
uplinkPeer := planet.Uplinks[0]
@ -799,14 +793,12 @@ func testRepairMultipleDisqualifiedAndSuspended(t *testing.T, inMemoryRepair boo
StorageNodeCount: 12,
UplinkCount: 1,
Reconfigure: testplanet.Reconfigure{
Satellite: func(log *zap.Logger, index int, config *satellite.Config) {
Satellite: testplanet.Combine(
func(log *zap.Logger, index int, config *satellite.Config) {
config.Repairer.InMemoryRepair = inMemoryRepair
config.Metainfo.RS.MinThreshold = 3
config.Metainfo.RS.RepairThreshold = 5
config.Metainfo.RS.SuccessThreshold = 7
config.Metainfo.RS.TotalThreshold = 7
},
testplanet.ReconfigureRS(3, 5, 7, 7),
),
},
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
// first, upload some remote data
@ -920,15 +912,13 @@ func testDataRepairOverrideHigherLimit(t *testing.T, inMemoryRepair bool) {
StorageNodeCount: 14,
UplinkCount: 1,
Reconfigure: testplanet.Reconfigure{
Satellite: func(log *zap.Logger, index int, config *satellite.Config) {
config.Checker.RepairOverride = repairOverride
Satellite: testplanet.Combine(
func(log *zap.Logger, index int, config *satellite.Config) {
config.Repairer.InMemoryRepair = inMemoryRepair
config.Metainfo.RS.MinThreshold = 3
config.Metainfo.RS.RepairThreshold = 4
config.Metainfo.RS.SuccessThreshold = 9
config.Metainfo.RS.TotalThreshold = 9
config.Checker.RepairOverride = repairOverride
},
testplanet.ReconfigureRS(3, 4, 9, 9),
),
},
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
uplinkPeer := planet.Uplinks[0]
@ -1014,15 +1004,13 @@ func testDataRepairOverrideLowerLimit(t *testing.T, inMemoryRepair bool) {
StorageNodeCount: 14,
UplinkCount: 1,
Reconfigure: testplanet.Reconfigure{
Satellite: func(log *zap.Logger, index int, config *satellite.Config) {
config.Checker.RepairOverride = repairOverride
Satellite: testplanet.Combine(
func(log *zap.Logger, index int, config *satellite.Config) {
config.Repairer.InMemoryRepair = inMemoryRepair
config.Metainfo.RS.MinThreshold = 3
config.Metainfo.RS.RepairThreshold = 6
config.Metainfo.RS.SuccessThreshold = 9
config.Metainfo.RS.TotalThreshold = 9
config.Checker.RepairOverride = repairOverride
},
testplanet.ReconfigureRS(3, 6, 9, 9),
),
},
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
uplinkPeer := planet.Uplinks[0]
@ -1141,15 +1129,13 @@ func testDataRepairUploadLimit(t *testing.T, inMemoryRepair bool) {
StorageNodeCount: 13,
UplinkCount: 1,
Reconfigure: testplanet.Reconfigure{
Satellite: func(log *zap.Logger, index int, config *satellite.Config) {
Satellite: testplanet.Combine(
func(log *zap.Logger, index int, config *satellite.Config) {
config.Repairer.MaxExcessRateOptimalThreshold = RepairMaxExcessRateOptimalThreshold
config.Repairer.InMemoryRepair = inMemoryRepair
config.Metainfo.RS.MinThreshold = 3
config.Metainfo.RS.RepairThreshold = repairThreshold
config.Metainfo.RS.SuccessThreshold = successThreshold
config.Metainfo.RS.TotalThreshold = maxThreshold
},
testplanet.ReconfigureRS(3, repairThreshold, successThreshold, maxThreshold),
),
},
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
satellite := planet.Satellites[0]
@ -1266,14 +1252,12 @@ func testRepairGracefullyExited(t *testing.T, inMemoryRepair bool) {
StorageNodeCount: 12,
UplinkCount: 1,
Reconfigure: testplanet.Reconfigure{
Satellite: func(log *zap.Logger, index int, config *satellite.Config) {
Satellite: testplanet.Combine(
func(log *zap.Logger, index int, config *satellite.Config) {
config.Repairer.InMemoryRepair = inMemoryRepair
config.Metainfo.RS.MinThreshold = 3
config.Metainfo.RS.RepairThreshold = 5
config.Metainfo.RS.SuccessThreshold = 7
config.Metainfo.RS.TotalThreshold = 7
},
testplanet.ReconfigureRS(3, 5, 7, 7),
),
},
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
// first, upload some remote data

1
satellite/repair/repairer/segments.go
View File

@ -346,7 +346,6 @@ func (repairer *SegmentRepairer) updateAuditFailStatus(ctx context.Context, fail
for i, nodeID := range failedAuditNodeIDs {
updateRequests[i] = &overlay.UpdateRequest{
NodeID: nodeID,
IsUp: true,
AuditOutcome: overlay.AuditFailure,
}
}

11
satellite/satellitedb/overlaycache.go
View File

@ -366,7 +366,8 @@ func (cache *overlaycache) BatchUpdateStats(ctx context.Context, updateRequests
continue
}
auditHistory, err := cache.updateAuditHistoryWithTx(ctx, tx, updateReq.NodeID, now, updateReq.IsUp, updateReq.AuditHistory)
isUp := updateReq.AuditOutcome != overlay.AuditOffline
auditHistory, err := cache.updateAuditHistoryWithTx(ctx, tx, updateReq.NodeID, now, isUp, updateReq.AuditHistory)
if err != nil {
doAppendAll = false
return err
@ -444,7 +445,8 @@ func (cache *overlaycache) UpdateStats(ctx context.Context, updateReq *overlay.U
return nil
}
auditHistory, err := cache.updateAuditHistoryWithTx(ctx, tx, updateReq.NodeID, now, updateReq.IsUp, updateReq.AuditHistory)
isUp := updateReq.AuditOutcome != overlay.AuditOffline
auditHistory, err := cache.updateAuditHistoryWithTx(ctx, tx, updateReq.NodeID, now, isUp, updateReq.AuditHistory)
if err != nil {
return err
}
@ -1259,7 +1261,8 @@ func (cache *overlaycache) populateUpdateNodeStats(dbNode *dbx.Node, updateReq *
mon.FloatVal("audit_online_score").Observe(auditOnlineScore) //mon:locked
totalUptimeCount := dbNode.TotalUptimeCount
if updateReq.IsUp {
isUp := updateReq.AuditOutcome != overlay.AuditOffline
if isUp {
totalUptimeCount++
}
@ -1317,7 +1320,7 @@ func (cache *overlaycache) populateUpdateNodeStats(dbNode *dbx.Node, updateReq *
updateFields.UnknownAuditSuspended = timeField{set: true, isNil: true}
}
if updateReq.IsUp {
if isUp {
updateFields.UptimeSuccessCount = int64Field{set: true, value: dbNode.UptimeSuccessCount + 1}
updateFields.LastContactSuccess = timeField{set: true, value: now}
} else {

30
satellite/satellitedb/overlaycache_test.go
View File

@ -28,11 +28,11 @@ func TestUpdateStats(t *testing.T) {
numAudits := int64(2)
numUptimes := int64(3)
// nodes automatically start with 2 uptimes from testplanet startup
// nodeA: 1 audit, 2 uptime -> unvetted
updateReq := &overlay.UpdateRequest{
NodeID: nodeA.ID(),
AuditOutcome: overlay.AuditFailure,
IsUp: false,
AuditOutcome: overlay.AuditOffline,
AuditsRequiredForVetting: numAudits,
UptimesRequiredForVetting: numUptimes,
AuditHistory: testAuditHistoryConfig(),
@ -45,8 +45,7 @@ func TestUpdateStats(t *testing.T) {
// nodeA: 2 audits, 2 uptimes -> unvetted
updateReq.NodeID = nodeA.ID()
updateReq.AuditOutcome = overlay.AuditFailure
updateReq.IsUp = false
updateReq.AuditOutcome = overlay.AuditOffline
nodeStats, err = cache.UpdateStats(ctx, updateReq, time.Now())
require.NoError(t, err)
assert.Nil(t, nodeStats.VettedAt)
@ -56,7 +55,6 @@ func TestUpdateStats(t *testing.T) {
// nodeA: 3 audits, 3 uptimes -> vetted
updateReq.NodeID = nodeA.ID()
updateReq.AuditOutcome = overlay.AuditSuccess
updateReq.IsUp = true
nodeStats, err = cache.UpdateStats(ctx, updateReq, time.Now())
require.NoError(t, err)
assert.NotNil(t, nodeStats.VettedAt)
@ -66,7 +64,6 @@ func TestUpdateStats(t *testing.T) {
// nodeB: 1 audit, 3 uptimes -> unvetted
updateReq.NodeID = nodeB.ID()
updateReq.AuditOutcome = overlay.AuditSuccess
updateReq.IsUp = true
nodeStats, err = cache.UpdateStats(ctx, updateReq, time.Now())
require.NoError(t, err)
assert.Nil(t, nodeStats.VettedAt)
@ -75,8 +72,7 @@ func TestUpdateStats(t *testing.T) {
// nodeB: 2 audits, 3 uptimes -> vetted
updateReq.NodeID = nodeB.ID()
updateReq.AuditOutcome = overlay.AuditFailure
updateReq.IsUp = false
updateReq.AuditOutcome = overlay.AuditOffline
nodeStats, err = cache.UpdateStats(ctx, updateReq, time.Now())
require.NoError(t, err)
assert.NotNil(t, nodeStats.VettedAt)
@ -86,7 +82,6 @@ func TestUpdateStats(t *testing.T) {
// Don't overwrite node b's vetted_at timestamp
updateReq.NodeID = nodeB.ID()
updateReq.AuditOutcome = overlay.AuditSuccess
updateReq.IsUp = true
nodeStats2, err := cache.UpdateStats(ctx, updateReq, time.Now())
require.NoError(t, err)
assert.NotNil(t, nodeStats2.VettedAt)
@ -110,9 +105,10 @@ func TestBatchUpdateStats(t *testing.T) {
numUptimes := int64(3)
batchSize := 2
// nodes automatically start with 2 uptimes from testplanet startup
// both nodeA and nodeB unvetted
updateReqA := &overlay.UpdateRequest{NodeID: nodeA.ID(), AuditOutcome: overlay.AuditFailure, IsUp: false, AuditsRequiredForVetting: numAudits, UptimesRequiredForVetting: numUptimes, AuditHistory: testAuditHistoryConfig()}
updateReqB := &overlay.UpdateRequest{NodeID: nodeB.ID(), AuditOutcome: overlay.AuditSuccess, IsUp: true, AuditsRequiredForVetting: numAudits, UptimesRequiredForVetting: numUptimes, AuditHistory: testAuditHistoryConfig()}
updateReqA := &overlay.UpdateRequest{NodeID: nodeA.ID(), AuditOutcome: overlay.AuditOffline, AuditsRequiredForVetting: numAudits, UptimesRequiredForVetting: numUptimes, AuditHistory: testAuditHistoryConfig()}
updateReqB := &overlay.UpdateRequest{NodeID: nodeB.ID(), AuditOutcome: overlay.AuditSuccess, AuditsRequiredForVetting: numAudits, UptimesRequiredForVetting: numUptimes, AuditHistory: testAuditHistoryConfig()}
updateReqs := []*overlay.UpdateRequest{updateReqA, updateReqB}
failed, err := cache.BatchUpdateStats(ctx, updateReqs, batchSize, time.Now())
require.NoError(t, err)
@ -131,8 +127,8 @@ func TestBatchUpdateStats(t *testing.T) {
assert.EqualValues(t, 3, nB.Reputation.UptimeCount)
// nodeA unvetted, nodeB vetted
updateReqA = &overlay.UpdateRequest{NodeID: nodeA.ID(), AuditOutcome: overlay.AuditFailure, IsUp: false, AuditsRequiredForVetting: numAudits, UptimesRequiredForVetting: numUptimes, AuditHistory: testAuditHistoryConfig()}
updateReqB = &overlay.UpdateRequest{NodeID: nodeB.ID(), AuditOutcome: overlay.AuditFailure, IsUp: false, AuditsRequiredForVetting: numAudits, UptimesRequiredForVetting: numUptimes, AuditHistory: testAuditHistoryConfig()}
updateReqA = &overlay.UpdateRequest{NodeID: nodeA.ID(), AuditOutcome: overlay.AuditOffline, AuditsRequiredForVetting: numAudits, UptimesRequiredForVetting: numUptimes, AuditHistory: testAuditHistoryConfig()}
updateReqB = &overlay.UpdateRequest{NodeID: nodeB.ID(), AuditOutcome: overlay.AuditFailure, AuditsRequiredForVetting: numAudits, UptimesRequiredForVetting: numUptimes, AuditHistory: testAuditHistoryConfig()}
updateReqs = []*overlay.UpdateRequest{updateReqA, updateReqB}
failed, err = cache.BatchUpdateStats(ctx, updateReqs, batchSize, time.Now())
require.NoError(t, err)
@ -148,11 +144,11 @@ func TestBatchUpdateStats(t *testing.T) {
require.NoError(t, err)
assert.NotNil(t, nB.Reputation.VettedAt)
assert.EqualValues(t, 2, nB.Reputation.AuditCount)
assert.EqualValues(t, 3, nB.Reputation.UptimeCount)
assert.EqualValues(t, 4, nB.Reputation.UptimeCount)
// both nodeA and nodeB vetted (don't overwrite timestamp)
updateReqA = &overlay.UpdateRequest{NodeID: nodeA.ID(), AuditOutcome: overlay.AuditSuccess, IsUp: true, AuditsRequiredForVetting: numAudits, UptimesRequiredForVetting: numUptimes, AuditHistory: testAuditHistoryConfig()}
updateReqB = &overlay.UpdateRequest{NodeID: nodeB.ID(), AuditOutcome: overlay.AuditSuccess, IsUp: true, AuditsRequiredForVetting: numAudits, UptimesRequiredForVetting: numUptimes, AuditHistory: testAuditHistoryConfig()}
updateReqA = &overlay.UpdateRequest{NodeID: nodeA.ID(), AuditOutcome: overlay.AuditSuccess, AuditsRequiredForVetting: numAudits, UptimesRequiredForVetting: numUptimes, AuditHistory: testAuditHistoryConfig()}
updateReqB = &overlay.UpdateRequest{NodeID: nodeB.ID(), AuditOutcome: overlay.AuditSuccess, AuditsRequiredForVetting: numAudits, UptimesRequiredForVetting: numUptimes, AuditHistory: testAuditHistoryConfig()}
updateReqs = []*overlay.UpdateRequest{updateReqA, updateReqB}
failed, err = cache.BatchUpdateStats(ctx, updateReqs, batchSize, time.Now())
require.NoError(t, err)
@ -169,7 +165,7 @@ func TestBatchUpdateStats(t *testing.T) {
assert.NotNil(t, nB2.Reputation.VettedAt)
assert.Equal(t, nB.Reputation.VettedAt, nB2.Reputation.VettedAt)
assert.EqualValues(t, 3, nB2.Reputation.AuditCount)
assert.EqualValues(t, 4, nB2.Reputation.UptimeCount)
assert.EqualValues(t, 5, nB2.Reputation.UptimeCount)
})
}

14
satellite/satellitedb/projectaccounting.go
View File

@ -358,6 +358,7 @@ func (db *ProjectAccounting) GetBucketUsageRollups(ctx context.Context, projectI
var bucketUsageRollups []accounting.BucketUsageRollup
for _, bucket := range buckets {
err := func() error {
bucketRollup := accounting.BucketUsageRollup{
ProjectID: projectID,
BucketName: []byte(bucket),
@ -368,7 +369,7 @@ func (db *ProjectAccounting) GetBucketUsageRollups(ctx context.Context, projectI
// get bucket_bandwidth_rollups
rollupsRows, err := db.db.QueryContext(ctx, roullupsQuery, projectID[:], []byte(bucket), since, before)
if err != nil {
return nil, err
return err
}
defer func() { err = errs.Combine(err, rollupsRows.Close()) }()
@ -379,7 +380,7 @@ func (db *ProjectAccounting) GetBucketUsageRollups(ctx context.Context, projectI
err = rollupsRows.Scan(&settled, &inline, &action)
if err != nil {
return nil, err
return err
}
switch action {
@ -394,7 +395,7 @@ func (db *ProjectAccounting) GetBucketUsageRollups(ctx context.Context, projectI
}
}
if err := rollupsRows.Err(); err != nil {
return nil, err
return err
}
bucketStorageTallies, err := storageQuery(ctx,
@ -404,7 +405,7 @@ func (db *ProjectAccounting) GetBucketUsageRollups(ctx context.Context, projectI
dbx.BucketStorageTally_IntervalStart(before))
if err != nil {
return nil, err
return err
}
// fill metadata, objects and stored data
@ -424,6 +425,11 @@ func (db *ProjectAccounting) GetBucketUsageRollups(ctx context.Context, projectI
}
bucketUsageRollups = append(bucketUsageRollups, bucketRollup)
return nil
}()
if err != nil {
return nil, err
}
}
return bucketUsageRollups, nil

28
scripts/testdata/satellite-config.yaml.lock vendored
View File

@ -400,32 +400,8 @@ identity.key-path: /root/.local/share/storj/identity/satellite/identity.key
# request rate per project per second.
# metainfo.rate-limiter.rate: 1000
# the size of each new erasure share in bytes
# metainfo.rs.erasure-share-size: 256 B
# maximum buffer memory to be allocated for read buffers
# metainfo.rs.max-buffer-mem: 4.0 MiB
# the largest amount of pieces to encode to. n (upper bound for validation).
# metainfo.rs.max-total-threshold: 130
# the minimum pieces required to recover a segment. k.
# metainfo.rs.min-threshold: 29
# the largest amount of pieces to encode to. n (lower bound for validation).
# metainfo.rs.min-total-threshold: 95
# the minimum safe pieces before a repair is triggered. m.
# metainfo.rs.repair-threshold: 35
# the desired total pieces for a segment. o.
# metainfo.rs.success-threshold: 80
# the largest amount of pieces to encode to. n.
# metainfo.rs.total-threshold: 110
# validate redundancy scheme configuration
# metainfo.rs.validate: true
# redundancy scheme configuration in the format k/m/o/n-sharesize
# metainfo.rs: 29/35/80/110-256 B
# address(es) to send telemetry to (comma-separated)
# metrics.addr: collectora.storj.io:9000

2
storage/testsuite/test.go
View File

@ -163,6 +163,7 @@ func testConstraints(t *testing.T, ctx *testcontext.Context, store storage.KeyVa
{storage.Value("old-value"), nil},
{storage.Value("old-value"), storage.Value("new-value")},
} {
func() {
errTag := fmt.Sprintf("%d. %+v", i, tt)
key := storage.Key("test-key")
val := storage.Value("test-value")
@ -173,6 +174,7 @@ func testConstraints(t *testing.T, ctx *testcontext.Context, store storage.KeyVa
err = store.CompareAndSwap(ctx, key, tt.old, tt.new)
assert.True(t, storage.ErrValueChanged.Has(err), "%s: unexpected error: %+v", errTag, err)
}()
}
})

9
storagenode/console/consoleapi/payout_test.go
View File

@ -376,8 +376,15 @@ func TestHeldAmountApi(t *testing.T) {
JoinedAt: date,
}
stefanID, err := storj.NodeIDFromString("118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW")
require.NoError(t, err)
held2 := payout.SatelliteHeldHistory{
SatelliteID: stefanID,
}
var periods []payout.SatelliteHeldHistory
periods = append(periods, held)
periods = append(periods, held, held2)
expected, err := json.Marshal(periods)
require.NoError(t, err)

11
storagenode/gracefulexit/worker_test.go
View File

@ -177,15 +177,14 @@ func TestWorkerFailure_IneligibleNodeAge(t *testing.T) {
StorageNodeCount: 5,
UplinkCount: 1,
Reconfigure: testplanet.Reconfigure{
Satellite: func(logger *zap.Logger, index int, config *satellite.Config) {
Satellite: testplanet.Combine(
func(log *zap.Logger, index int, config *satellite.Config) {
// Set the required node age to 1 month.
config.GracefulExit.NodeMinAgeInMonths = 1
config.Metainfo.RS.MinThreshold = 2
config.Metainfo.RS.RepairThreshold = 3
config.Metainfo.RS.SuccessThreshold = successThreshold
config.Metainfo.RS.TotalThreshold = successThreshold
},
testplanet.ReconfigureRS(2, 3, successThreshold, successThreshold),
),
StorageNode: func(index int, config *storagenode.Config) {
config.GracefulExit.NumWorkers = 2
config.GracefulExit.NumConcurrentTransfers = 2

2
storagenode/orders/service.go
View File

@ -398,7 +398,6 @@ func (service *Service) sendOrdersFromFileStore(ctx context.Context, now time.Ti
var group errgroup.Group
attemptedSatellites := 0
ctx, cancel := context.WithTimeout(ctx, service.config.SenderTimeout)
defer cancel()
for satelliteID, unsentInfo := range ordersBySatellite {
satelliteID, unsentInfo := satelliteID, unsentInfo
@ -430,6 +429,7 @@ func (service *Service) sendOrdersFromFileStore(ctx context.Context, now time.Ti
}
_ = group.Wait() // doesn't return errors
cancel()
// if all satellites that orders need to be sent to are offline, exit and try again later.
if attemptedSatellites == 0 {

6
storagenode/payout/db_test.go
View File

@ -211,7 +211,8 @@ func TestSatellitePayStubPeriodCached(t *testing.T) {
heldAmountDB := db.Payout()
reputationDB := db.Reputation()
satellitesDB := db.Satellites()
service := payout.NewService(nil, heldAmountDB, reputationDB, satellitesDB, nil)
service, err := payout.NewService(nil, heldAmountDB, reputationDB, satellitesDB, nil)
require.NoError(t, err)
payStub := payout.PayStub{
SatelliteID: storj.NodeID{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1},
@ -261,7 +262,8 @@ func TestAllPayStubPeriodCached(t *testing.T) {
heldAmountDB := db.Payout()
reputationDB := db.Reputation()
satellitesDB := db.Satellites()
service := payout.NewService(nil, heldAmountDB, reputationDB, satellitesDB, nil)
service, err := payout.NewService(nil, heldAmountDB, reputationDB, satellitesDB, nil)
require.NoError(t, err)
payStub := payout.PayStub{
SatelliteID: storj.NodeID{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1},

48
storagenode/payout/service.go
View File

@ -39,6 +39,8 @@ var (
type Service struct {
log *zap.Logger
stefanSatellite storj.NodeID
db DB
reputationDB reputation.DB
satellitesDB satellites.DB
@ -46,14 +48,20 @@ type Service struct {
}
// NewService creates new instance of service.
func NewService(log *zap.Logger, db DB, reputationDB reputation.DB, satelliteDB satellites.DB, trust *trust.Pool) *Service {
func NewService(log *zap.Logger, db DB, reputationDB reputation.DB, satelliteDB satellites.DB, trust *trust.Pool) (_ *Service, err error) {
id, err := storj.NodeIDFromString("118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW")
if err != nil {
return &Service{}, err
}
return &Service{
log: log,
stefanSatellite: id,
db: db,
reputationDB: reputationDB,
satellitesDB: satelliteDB,
trust: trust,
}
}, nil
}
// SatellitePayStubMonthly retrieves held amount for particular satellite for selected month from storagenode database.
@ -161,17 +169,18 @@ func (service *Service) AllPeriods(ctx context.Context) (_ []string, err error)
// AllHeldbackHistory retrieves heldback history for all satellites from storagenode database.
func (service *Service) AllHeldbackHistory(ctx context.Context) (result []SatelliteHeldHistory, err error) {
defer mon.Task()(&ctx)(&err)
satellitesIDs := service.trust.GetSatellites(ctx)
satellites := service.trust.GetSatellites(ctx)
for i := 0; i < len(satellites); i++ {
satellitesIDs = append(satellitesIDs, service.stefanSatellite)
for i := 0; i < len(satellitesIDs); i++ {
var history SatelliteHeldHistory
helds, err := service.db.SatellitesHeldbackHistory(ctx, satellites[i])
helds, err := service.db.SatellitesHeldbackHistory(ctx, satellitesIDs[i])
if err != nil {
return nil, ErrPayoutService.Wrap(err)
}
disposed, err := service.db.SatellitesDisposedHistory(ctx, satellites[i])
disposed, err := service.db.SatellitesDisposedHistory(ctx, satellitesIDs[i])
if err != nil {
return nil, ErrPayoutService.Wrap(err)
}
@ -192,20 +201,22 @@ func (service *Service) AllHeldbackHistory(ctx context.Context) (result []Satell
}
history.TotalDisposed = disposed
history.SatelliteID = satellites[i]
url, err := service.trust.GetNodeURL(ctx, satellites[i])
history.SatelliteID = satellitesIDs[i]
if satellitesIDs[i] != service.stefanSatellite {
url, err := service.trust.GetNodeURL(ctx, satellitesIDs[i])
if err != nil {
return nil, ErrPayoutService.Wrap(err)
}
stats, err := service.reputationDB.Get(ctx, satellites[i])
if err != nil {
return nil, ErrPayoutService.Wrap(err)
}
history.SatelliteName = url.Address
history.JoinedAt = stats.JoinedAt
}
stats, err := service.reputationDB.Get(ctx, satellitesIDs[i])
if err != nil {
return nil, ErrPayoutService.Wrap(err)
}
history.JoinedAt = stats.JoinedAt
result = append(result, history)
}
@ -217,6 +228,8 @@ func (service *Service) AllSatellitesPayoutPeriod(ctx context.Context, period st
defer mon.Task()(&ctx)(&err)
satelliteIDs := service.trust.GetSatellites(ctx)
satelliteIDs = append(satelliteIDs, service.stefanSatellite)
for i := 0; i < len(satelliteIDs); i++ {
var payoutForPeriod SatellitePayoutForPeriod
paystub, err := service.db.GetPayStub(ctx, satelliteIDs[i], period)
@ -249,11 +262,15 @@ func (service *Service) AllSatellitesPayoutPeriod(ctx context.Context, period st
return nil, ErrPayoutService.Wrap(err)
}
if satelliteIDs[i] != service.stefanSatellite {
url, err := service.trust.GetNodeURL(ctx, satelliteIDs[i])
if err != nil {
return nil, ErrPayoutService.Wrap(err)
}
payoutForPeriod.SatelliteURL = url.Address
}
if satellite.Status == satellites.ExitSucceeded {
payoutForPeriod.IsExitComplete = true
}
@ -281,7 +298,6 @@ func (service *Service) AllSatellitesPayoutPeriod(ctx context.Context, period st
payoutForPeriod.Earned = earned
payoutForPeriod.SatelliteID = satelliteIDs[i].String()
payoutForPeriod.SurgePercent = paystub.SurgePercent
payoutForPeriod.SatelliteURL = url.Address
payoutForPeriod.Paid = paystub.Paid
payoutForPeriod.HeldPercent = heldPercent

6
storagenode/peer.go
View File

@ -555,13 +555,17 @@ func New(log *zap.Logger, full *identity.FullIdentity, db DB, revocationDB exten
}
{ // setup payout service.
peer.Payout.Service = payout.NewService(
service, err := payout.NewService(
peer.Log.Named("payout:service"),
peer.DB.Payout(),
peer.DB.Reputation(),
peer.DB.Satellites(),
peer.Storage2.Trust,
)
if err != nil {
return nil, errs.Combine(err, peer.Close())
}
peer.Payout.Service = service
peer.Payout.Endpoint = payout.NewEndpoint(
peer.Log.Named("payout:endpoint"),
peer.Dialer,

4
storagenode/piecestore/endpoint_test.go
View File

@ -50,6 +50,7 @@ func TestUploadAndPartialDownload(t *testing.T) {
{1513, 1584},
{13581, 4783},
} {
func() {
if piecestore.DefaultConfig.InitialStep < tt.size {
t.Fatal("test expects initial step to be larger than size to download")
}
@ -67,6 +68,7 @@ func TestUploadAndPartialDownload(t *testing.T) {
assert.Equal(t, expectedData[tt.offset:tt.offset+tt.size], data)
require.NoError(t, download.Close())
}()
}
var totalBandwidthUsage bandwidth.Usage
@ -528,8 +530,6 @@ func TestDeletePieces(t *testing.T) {
}
func TestTooManyRequests(t *testing.T) {
t.Skip("flaky, because of EOF issues")
const uplinkCount = 6
const maxConcurrent = 3
const expectedFailures = uplinkCount - maxConcurrent

2
storagenode/piecestore/verification_test.go
View File

@ -226,6 +226,7 @@ func TestOrderLimitGetValidation(t *testing.T) {
err: "expected get or get repair or audit action got PUT",
},
} {
func() {
client, err := planet.Uplinks[0].DialPiecestore(ctx, planet.StorageNodes[0])
require.NoError(t, err)
defer ctx.Check(client.Close)
@ -265,6 +266,7 @@ func TestOrderLimitGetValidation(t *testing.T) {
} else {
require.NoError(t, err)
}
}()
}
})
}

10
storagenode/storagenodedb/database.go
View File

@ -349,6 +349,14 @@ func (db *DB) MigrateToLatest(ctx context.Context) error {
// Preflight conducts a pre-flight check to ensure correct schemas and minimal read+write functionality of the database tables.
func (db *DB) Preflight(ctx context.Context) (err error) {
for dbName, dbContainer := range db.SQLDBs {
if err := db.preflight(ctx, dbName, dbContainer); err != nil {
return err
}
}
return nil
}
func (db *DB) preflight(ctx context.Context, dbName string, dbContainer DBContainer) error {
nextDB := dbContainer.GetDB()
// Preflight stage 1: test schema correctness
schema, err := sqliteutil.QuerySchema(ctx, nextDB)
@ -442,7 +450,7 @@ func (db *DB) Preflight(ctx context.Context) (err error) {
if err != nil {
return ErrPreflight.New("database %q: failed drop test_table %w", dbName, err)
}
}
return nil
}

2
web/satellite/README.md
View File

@ -67,7 +67,7 @@ docker run -p 8080:8080 storjlabs/satellite-ui:latest
- [unit](./unit "unit") folder: contains project unit tests.
### Configuration files
- **.env**: file for environment level variables.
- **.gitignore**: folders, files and extentions which are ignored for git.
- **.gitignore**: folders, files and extensions which are ignored for git.
- **babel.config.js**: [babel](https://babeljs.io/) configuration for javascript transcompilation.
- **index.html**: DOM entry point.
- **jestSetup.ts**: [jest](https://jestjs.io/) configuration for unit testing.

14
web/storagenode/src/app/components/SNOHeader.vue
View File

@ -68,7 +68,6 @@ import { APPSTATE_ACTIONS } from '@/app/store/modules/appState';
import { NODE_ACTIONS } from '@/app/store/modules/node';
import { NOTIFICATIONS_ACTIONS } from '@/app/store/modules/notifications';
import { PAYOUT_ACTIONS } from '@/app/store/modules/payout';
import { NotificationsCursor } from '@/app/types/notifications';
const {
GET_NODE_INFO,
@ -90,18 +89,23 @@ const {
export default class SNOHeader extends Vue {
public isNotificationPopupShown: boolean = false;
public isOptionsShown: boolean = false;
private readonly FIRST_PAGE: number = 1;
/**
* Lifecycle hook before render.
* Fetches first page of notifications.
*/
public beforeMount(): void {
public async beforeMount(): Promise<void> {
await this.$store.dispatch(APPSTATE_ACTIONS.SET_LOADING, true);
try {
this.$store.dispatch(NODE_ACTIONS.GET_NODE_INFO);
this.$store.dispatch(NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS, new NotificationsCursor(1));
await this.$store.dispatch(NODE_ACTIONS.GET_NODE_INFO);
await this.$store.dispatch(NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS, this.FIRST_PAGE);
} catch (error) {
console.error(error.message);
}
await this.$store.dispatch(APPSTATE_ACTIONS.SET_LOADING, false);
}
public get nodeId(): string {
@ -192,7 +196,7 @@ export default class SNOHeader extends Vue {
}
try {
await this.$store.dispatch(NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS, new NotificationsCursor(1));
await this.$store.dispatch(NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS, this.FIRST_PAGE);
} catch (error) {
console.error(error.message);
}

9
web/storagenode/src/app/components/notifications/NotificationsPopup.vue
View File

@ -12,10 +12,10 @@
<div
class="notification-popup-container__content"
:class="{'collapsed': isCollapsed}"
v-if="latestNotifications.length"
v-if="latest.length"
>
<SNONotification
v-for="notification in latestNotifications"
v-for="notification in latest"
:key="notification.id"
is-small="true"
:notification="notification"
@ -34,6 +34,7 @@ import { Component, Vue } from 'vue-property-decorator';
import SNONotification from '@/app/components/notifications/SNONotification.vue';
import { RouteConfig } from '@/app/router';
import { UINotification } from '@/app/types/notifications';
@Component({
components: {
@ -49,7 +50,7 @@ export default class NotificationsPopup extends Vue {
/**
* Represents first page of notifications.
*/
public get latestNotifications(): Notification[] {
public get latest(): UINotification[] {
return this.$store.state.notificationsModule.latestNotifications;
}
@ -57,7 +58,7 @@ export default class NotificationsPopup extends Vue {
* Indicates if popup is smaller than with scroll.
*/
public get isCollapsed(): boolean {
return this.latestNotifications.length < 4;
return this.latest.length < 4;
}
}
</script>

6
web/storagenode/src/app/components/notifications/SNONotification.vue
View File

@ -30,12 +30,12 @@
import { Component, Prop, Vue } from 'vue-property-decorator';
import { NOTIFICATIONS_ACTIONS } from '@/app/store/modules/notifications';
import { Notification } from '@/app/types/notifications';
import { UINotification } from '@/app/types/notifications';
@Component
export default class SNONotification extends Vue {
@Prop({default: () => new Notification()})
public readonly notification: Notification;
@Prop({default: () => new UINotification()})
public readonly notification: UINotification;
/**
* isSmall props indicates if component used in popup.

2
web/storagenode/src/app/components/payments/HeldProgress.vue
View File

@ -89,7 +89,7 @@ export default class HeldProgress extends Vue {
),
new HeldStep(
'+50%',
'Month 15',
'Month 16',
this.monthsOnNetwork > 15,
this.monthsOnNetwork < 15,
),

6
web/storagenode/src/app/store/index.ts
View File

@ -4,17 +4,19 @@
import Vue from 'vue';
import Vuex from 'vuex';
import { makeNotificationsModule } from '@/app/store/modules/notifications';
import { newNotificationsModule } from '@/app/store/modules/notifications';
import { makePayoutModule } from '@/app/store/modules/payout';
import { NotificationsHttpApi } from '@/storagenode/api/notifications';
import { PayoutHttpApi } from '@/storagenode/api/payout';
import { SNOApi } from '@/storagenode/api/storagenode';
import { NotificationsService } from '@/storagenode/notifications/service';
import { PayoutService } from '@/storagenode/payouts/service';
import { appStateModule } from './modules/appState';
import { makeNodeModule } from './modules/node';
const notificationsApi = new NotificationsHttpApi();
const notificationsService = new NotificationsService(notificationsApi);
const payoutApi = new PayoutHttpApi();
const payoutService = new PayoutService(payoutApi);
const nodeApi = new SNOApi();
@ -28,7 +30,7 @@ export const store = new Vuex.Store({
modules: {
node: makeNodeModule(nodeApi),
appStateModule,
notificationsModule: makeNotificationsModule(notificationsApi),
notificationsModule: newNotificationsModule(notificationsService),
payoutModule: makePayoutModule(payoutApi, payoutService),
},
});

52
web/storagenode/src/app/store/modules/notifications.ts
View File

@ -1,12 +1,8 @@
// Copyright (C) 2019 Storj Labs, Inc.
// See LICENSE for copying information.
import {
Notification,
NotificationsApi,
NotificationsCursor,
NotificationsState,
} from '@/app/types/notifications';
import { NotificationsState, UINotification } from '@/app/types/notifications';
import { NotificationsService } from '@/storagenode/notifications/service';
export const NOTIFICATIONS_MUTATIONS = {
SET_NOTIFICATIONS: 'SET_NOTIFICATIONS',
@ -24,22 +20,22 @@ export const NOTIFICATIONS_ACTIONS = {
/**
* creates notifications module with all dependencies
*
* @param api - payments api
* @param service - payments service
*/
export function makeNotificationsModule(api: NotificationsApi) {
export function newNotificationsModule(service: NotificationsService) {
return {
state: new NotificationsState(),
mutations: {
[NOTIFICATIONS_MUTATIONS.SET_NOTIFICATIONS](state: NotificationsState, notificationsResponse: NotificationsState): void {
state.notifications = notificationsResponse.notifications;
state.pageCount = notificationsResponse.pageCount;
state.unreadCount = notificationsResponse.unreadCount;
[NOTIFICATIONS_MUTATIONS.SET_NOTIFICATIONS](state: NotificationsState, notificationsState: NotificationsState): void {
state.notifications = notificationsState.notifications;
state.pageCount = notificationsState.pageCount;
state.unreadCount = notificationsState.unreadCount;
},
[NOTIFICATIONS_MUTATIONS.SET_LATEST](state: NotificationsState, notificationsResponse: NotificationsState): void {
state.latestNotifications = notificationsResponse.notifications;
[NOTIFICATIONS_MUTATIONS.SET_LATEST](state: NotificationsState, notificationsState: NotificationsState): void {
state.latestNotifications = notificationsState.notifications;
},
[NOTIFICATIONS_MUTATIONS.MARK_AS_READ](state: NotificationsState, id: string): void {
state.notifications = state.notifications.map((notification: Notification) => {
state.notifications = state.notifications.map((notification: UINotification) => {
if (notification.id === id) {
notification.markAsRead();
}
@ -48,7 +44,7 @@ export function makeNotificationsModule(api: NotificationsApi) {
});
},
[NOTIFICATIONS_MUTATIONS.READ_ALL](state: NotificationsState): void {
state.notifications = state.notifications.map((notification: Notification) => {
state.notifications = state.notifications.map((notification: UINotification) => {
notification.markAsRead();
return notification;
@ -58,24 +54,26 @@ export function makeNotificationsModule(api: NotificationsApi) {
},
},
actions: {
[NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS]: async function ({commit}: any, cursor: NotificationsCursor): Promise<NotificationsState> {
const notificationsResponse = await api.get(cursor);
[NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS]: async function ({commit}: any, pageIndex: number): Promise<void> {
const notificationsResponse = await service.notifications(pageIndex);
commit(NOTIFICATIONS_MUTATIONS.SET_NOTIFICATIONS, notificationsResponse);
const notifications = notificationsResponse.page.notifications.map(notification => new UINotification(notification));
if (cursor.page === 1) {
commit(NOTIFICATIONS_MUTATIONS.SET_LATEST, notificationsResponse);
const notificationState = new NotificationsState(notifications, notificationsResponse.page.pageCount, notificationsResponse.unreadCount);
commit(NOTIFICATIONS_MUTATIONS.SET_NOTIFICATIONS, notificationState);
if (pageIndex === 1) {
commit(NOTIFICATIONS_MUTATIONS.SET_LATEST, notificationState);
}
return notificationsResponse;
},
[NOTIFICATIONS_ACTIONS.MARK_AS_READ]: async function ({commit}: any, id: string): Promise<any> {
await api.read(id);
[NOTIFICATIONS_ACTIONS.MARK_AS_READ]: async function ({commit}: any, id: string): Promise<void> {
await service.readSingeNotification(id);
commit(NOTIFICATIONS_MUTATIONS.MARK_AS_READ, id);
},
[NOTIFICATIONS_ACTIONS.READ_ALL]: async function ({commit}: any): Promise<any> {
await api.readAll();
[NOTIFICATIONS_ACTIONS.READ_ALL]: async function ({commit}: any): Promise<void> {
await service.readAllNotifications();
commit(NOTIFICATIONS_MUTATIONS.READ_ALL);
},

93
web/storagenode/src/app/types/notifications.ts
View File

@ -2,23 +2,39 @@
// See LICENSE for copying information.
import { NotificationIcon } from '@/app/utils/notificationIcons';
import { Notification, NotificationTypes } from '@/storagenode/notifications/notifications';
/**
* Holds all notifications module state.
*/
export class NotificationsState {
public latestNotifications: UINotification[] = [];
public constructor(
public notifications: UINotification[] = [],
public pageCount: number = 0,
public unreadCount: number = 0,
) { }
}
/**
* Describes notification entity.
*/
export class Notification {
export class UINotification {
public icon: NotificationIcon;
public isRead: boolean;
public id: string;
public senderId: string;
public type: NotificationTypes;
public title: string;
public message: string;
public readAt: Date | null;
public createdAt: Date;
public constructor(
public id: string = '',
public senderId: string = '',
public type: NotificationTypes = NotificationTypes.Custom,
public title: string = '',
public message: string = '',
public isRead: boolean = false,
public createdAt: Date = new Date(),
) {
public constructor(notification: Partial<UINotification> = new Notification()) {
Object.assign(this, notification);
this.setIcon();
this.isRead = !!this.readAt;
}
/**
@ -70,60 +86,3 @@ export class Notification {
}
}
}
/**
* Describes all current notifications types.
*/
export enum NotificationTypes {
Custom = 0,
AuditCheckFailure = 1,
UptimeCheckFailure = 2,
Disqualification = 3,
Suspension = 4,
}
/**
* Describes page offset for pagination.
*/
export class NotificationsCursor {
public constructor(
public page: number = 0,
public limit: number = 7,
) { }
}
/**
* Holds all notifications module state.
*/
export class NotificationsState {
public latestNotifications: Notification[] = [];
public constructor(
public notifications: Notification[] = [],
public pageCount: number = 0,
public unreadCount: number = 0,
) { }
}
/**
* Exposes all notifications-related functionality.
*/
export interface NotificationsApi {
/**
* Fetches notifications.
* @throws Error
*/
get(cursor: NotificationsCursor): Promise<NotificationsState>;
/**
* Marks single notification as read.
* @throws Error
*/
read(id: string): Promise<void>;
/**
* Marks all notification as read.
* @throws Error
*/
readAll(): Promise<void>;
}

3
web/storagenode/src/app/views/DashboardArea.vue
View File

@ -20,7 +20,6 @@ import { APPSTATE_ACTIONS } from '@/app/store/modules/appState';
import { NODE_ACTIONS } from '@/app/store/modules/node';
import { NOTIFICATIONS_ACTIONS } from '@/app/store/modules/notifications';
import { PAYOUT_ACTIONS } from '@/app/store/modules/payout';
import { NotificationsCursor } from '@/app/types/notifications';
@Component ({
components: {
@ -43,7 +42,7 @@ export default class Dashboard extends Vue {
}
try {
await this.$store.dispatch(NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS, new NotificationsCursor(1));
await this.$store.dispatch(NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS, 1);
} catch (error) {
console.error(error);
}

6
web/storagenode/src/app/views/NotificationsArea.vue
View File

@ -54,7 +54,7 @@ import VPagination from '@/app/components/VPagination.vue';
import BackArrowIcon from '@/../static/images/notifications/backArrow.svg';
import { NOTIFICATIONS_ACTIONS } from '@/app/store/modules/notifications';
import { Notification, NotificationsCursor } from '@/app/types/notifications';
import { UINotification } from '@/app/types/notifications';
@Component ({
components: {
@ -67,7 +67,7 @@ export default class NotificationsArea extends Vue {
/**
* Returns notification of current page.
*/
public get notifications(): Notification[] {
public get notifications(): UINotification[] {
return this.$store.state.notificationsModule.notifications;
}
@ -92,7 +92,7 @@ export default class NotificationsArea extends Vue {
*/
public async onPageClick(index: number): Promise<void> {
try {
await this.$store.dispatch(NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS, new NotificationsCursor(index));
await this.$store.dispatch(NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS, index);
} catch (error) {
console.error(error.message);
}

3
web/storagenode/src/app/views/PayoutArea.vue
View File

@ -57,7 +57,6 @@ import { APPSTATE_ACTIONS } from '@/app/store/modules/appState';
import { NODE_ACTIONS } from '@/app/store/modules/node';
import { NOTIFICATIONS_ACTIONS } from '@/app/store/modules/notifications';
import { PAYOUT_ACTIONS } from '@/app/store/modules/payout';
import { NotificationsCursor } from '@/app/types/notifications';
import { PayoutPeriod, TotalHeldAndPaid } from '@/storagenode/payouts/payouts';
@Component ({
@ -88,7 +87,7 @@ export default class PayoutArea extends Vue {
}
try {
await this.$store.dispatch(NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS, new NotificationsCursor(1));
await this.$store.dispatch(NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS, 1);
} catch (error) {
console.error(error);
}

35
web/storagenode/src/storagenode/api/notifications.ts
View File

@ -1,7 +1,12 @@
// Copyright (C) 2019 Storj Labs, Inc.
// See LICENSE for copying information.
import { Notification, NotificationsApi, NotificationsCursor, NotificationsState } from '@/app/types/notifications';
import {
NotificationsApi,
NotificationsCursor,
NotificationsPage,
NotificationsResponse,
} from '@/storagenode/notifications/notifications';
import { HttpClient } from '@/storagenode/utils/httpClient';
/**
@ -15,10 +20,10 @@ export class NotificationsHttpApi implements NotificationsApi {
/**
* Fetch notifications.
*
* @returns notifications state
* @returns notifications response.
* @throws Error
*/
public async get(cursor: NotificationsCursor): Promise<NotificationsState> {
public async get(cursor: NotificationsCursor): Promise<NotificationsResponse> {
const path = `${this.ROOT_PATH}/list?page=${cursor.page}&limit=${cursor.limit}`;
const response = await this.client.get(path);
@ -27,28 +32,12 @@ export class NotificationsHttpApi implements NotificationsApi {
}
const notificationResponse = await response.json();
let notifications: Notification[] = [];
let pageCount: number = 0;
let unreadCount: number = 0;
if (notificationResponse) {
notifications = notificationResponse.page.notifications.map(item =>
new Notification(
item.id,
item.senderId,
item.type,
item.title,
item.message,
!!item.readAt,
new Date(item.createdAt),
),
return new NotificationsResponse(
new NotificationsPage(notificationResponse.page.notifications, notificationResponse.page.pageCount),
notificationResponse.unreadCount,
notificationResponse.totalCount,
);
pageCount = notificationResponse.page.pageCount;
unreadCount = notificationResponse.unreadCount;
}
return new NotificationsState(notifications, pageCount, unreadCount);
}
/**

88
web/storagenode/src/storagenode/notifications/notifications.ts Normal file
View File

@ -0,0 +1,88 @@
// Copyright (C) 2020 Storj Labs, Inc.
// See LICENSE for copying information.
/**
* Exposes all notifications-related functionality.
*/
export interface NotificationsApi {
/**
* Fetches notifications.
* @throws Error
*/
get(cursor: NotificationsCursor): Promise<NotificationsResponse>;
/**
* Marks single notification as read.
* @throws Error
*/
read(id: string): Promise<void>;
/**
* Marks all notification as read.
* @throws Error
*/
readAll(): Promise<void>;
}
/**
* Describes notification entity.
*/
export class Notification {
public constructor(
public id: string = '',
public senderId: string = '',
public type: NotificationTypes = NotificationTypes.Custom,
public title: string = '',
public message: string = '',
public readAt: Date | null = null,
public createdAt: Date = new Date(),
) {}
}
/**
* Describes all current notifications types.
*/
export enum NotificationTypes {
Custom = 0,
AuditCheckFailure = 1,
UptimeCheckFailure = 2,
Disqualification = 3,
Suspension = 4,
}
/**
* Describes page offset for pagination.
*/
export class NotificationsCursor {
private DEFAULT_LIMIT: number = 7;
public constructor(
public page: number = 0,
public limit: number = 0,
) {
if (!this.limit) {
this.limit = this.DEFAULT_LIMIT;
}
}
}
/**
* Describes response object from server.
*/
export class NotificationsResponse {
public constructor(
public page: NotificationsPage = new NotificationsPage(),
public unreadCount: number = 0,
public totalCount: number = 0,
) {}
}
/**
* Describes page related notification information.
*/
export class NotificationsPage {
public constructor(
public notifications: Notification[] = [],
public pageCount: number = 0,
) {}
}

43
web/storagenode/src/storagenode/notifications/service.ts Normal file
View File

@ -0,0 +1,43 @@
// Copyright (C) 2020 Storj Labs, Inc.
// See LICENSE for copying information.
import { NotificationsApi, NotificationsCursor, NotificationsResponse } from '@/storagenode/notifications/notifications';
/**
* PayoutService is used to store and handle node paystub information.
* PayoutService exposes a business logic related to payouts.
*/
export class NotificationsService {
private readonly api: NotificationsApi;
public constructor(api: NotificationsApi) {
this.api = api;
}
/**
* Fetch notifications.
*
* @returns notifications response.
* @throws Error
*/
public async notifications(index: number, limit?: number): Promise<NotificationsResponse> {
const cursor = new NotificationsCursor(index, limit);
return await this.api.get(cursor);
}
/**
* Marks single notification as read on server.
* @param id
*/
public async readSingeNotification(id: string): Promise<void> {
await this.api.read(id);
}
/**
* Marks all notifications as read on server.
*/
public async readAllNotifications(): Promise<void> {
await this.api.readAll();
}
}

147
web/storagenode/tests/unit/store/notifications.spec.ts Normal file
View File

@ -0,0 +1,147 @@
// Copyright (C) 2020 Storj Labs, Inc.
// See LICENSE for copying information.
import Vuex from 'vuex';
import { newNotificationsModule, NOTIFICATIONS_ACTIONS, NOTIFICATIONS_MUTATIONS } from '@/app/store/modules/notifications';
import { NotificationsState, UINotification } from '@/app/types/notifications';
import { NotificationsHttpApi } from '@/storagenode/api/notifications';
import {
Notification,
NotificationsPage,
NotificationsResponse,
NotificationTypes,
} from '@/storagenode/notifications/notifications';
import { NotificationsService } from '@/storagenode/notifications/service';
import { createLocalVue } from '@vue/test-utils';
const Vue = createLocalVue();
const notificationsApi = new NotificationsHttpApi();
const notificationsService = new NotificationsService(notificationsApi);
const notificationsModule = newNotificationsModule(notificationsService);
Vue.use(Vuex);
const store = new Vuex.Store({ modules: { notificationsModule } });
const state = store.state as any;
let notifications;
describe('mutations', () => {
beforeEach(() => {
createLocalVue().use(Vuex);
notifications = [
new UINotification(new Notification('1', '1', NotificationTypes.Disqualification, 'title1', 'message1', null)),
new UINotification(new Notification('2', '1', NotificationTypes.UptimeCheckFailure, 'title2', 'message2', null)),
];
});
it('sets notification state', (): void => {
const notificationsState = new NotificationsState(notifications, 2, 1);
store.commit(NOTIFICATIONS_MUTATIONS.SET_NOTIFICATIONS, notificationsState);
expect(state.notificationsModule.notifications.length).toBe(notifications.length);
expect(state.notificationsModule.pageCount).toBe(2);
expect(state.notificationsModule.unreadCount).toBe(1);
store.commit(NOTIFICATIONS_MUTATIONS.SET_LATEST, notificationsState);
expect(state.notificationsModule.latestNotifications.length).toBe(notifications.length);
});
it('sets single notification as read', (): void => {
store.commit(NOTIFICATIONS_MUTATIONS.MARK_AS_READ, '1');
const unreadNotificationsCount = state.notificationsModule.notifications.filter(e => !e.isRead).length;
expect(unreadNotificationsCount).toBe(1);
});
it('sets all notification as read', (): void => {
store.commit(NOTIFICATIONS_MUTATIONS.READ_ALL);
const unreadNotificationsCount = state.notificationsModule.notifications.filter(e => !e.isRead).length;
expect(unreadNotificationsCount).toBe(0);
});
});
describe('actions', () => {
beforeEach(() => {
jest.resetAllMocks();
notifications = [
new UINotification(new Notification('1', '1', NotificationTypes.Disqualification, 'title1', 'message1', null)),
new UINotification(new Notification('2', '1', NotificationTypes.UptimeCheckFailure, 'title2', 'message2', null)),
];
});
it('throws error on failed notifications fetch', async (): Promise<void> => {
jest.spyOn(notificationsApi, 'get').mockImplementation(() => { throw new Error(); });
try {
await store.dispatch(NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS, 1);
expect(true).toBe(false);
} catch (error) {
expect(state.notificationsModule.latestNotifications.length).toBe(notifications.length);
}
});
it('success fetches notifications', async (): Promise<void> => {
jest.spyOn(notificationsService, 'notifications')
.mockReturnValue(Promise.resolve(new NotificationsResponse(new NotificationsPage(notifications, 1), 2, 1)));
await store.dispatch(NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS, 1);
expect(state.notificationsModule.latestNotifications.length).toBe(notifications.length);
});
it('throws error on failed single notification read', async (): Promise<void> => {
jest.spyOn(notificationsApi, 'read').mockImplementation(() => { throw new Error(); });
try {
await store.dispatch(NOTIFICATIONS_ACTIONS.MARK_AS_READ, '1');
expect(true).toBe(false);
} catch (error) {
const unreadNotificationsCount = state.notificationsModule.notifications.filter(e => !e.isRead).length;
expect(unreadNotificationsCount).toBe(notifications.length);
}
});
it('success marks single notification as read', async (): Promise<void> => {
jest.spyOn(notificationsService, 'notifications')
.mockReturnValue(Promise.resolve(new NotificationsResponse(new NotificationsPage(notifications, 1), 2, 1)));
await store.dispatch(NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS, 1);
await store.dispatch(NOTIFICATIONS_ACTIONS.MARK_AS_READ, '1');
const unreadNotificationsCount = state.notificationsModule.notifications.filter(e => !e.isRead).length;
expect(unreadNotificationsCount).toBe(1);
});
it('throws error on failed all notifications read', async (): Promise<void> => {
jest.spyOn(notificationsApi, 'readAll').mockImplementation(() => { throw new Error(); });
try {
await store.dispatch(NOTIFICATIONS_ACTIONS.READ_ALL);
expect(true).toBe(false);
} catch (error) {
const unreadNotificationsCount = state.notificationsModule.notifications.filter(e => !e.isRead).length;
expect(unreadNotificationsCount).toBe(1);
}
});
it('success marks all notifications as read', async (): Promise<void> => {
await store.dispatch(NOTIFICATIONS_ACTIONS.READ_ALL);
const unreadNotificationsCount = state.notificationsModule.notifications.filter(e => !e.isRead).length;
expect(unreadNotificationsCount).toBe(0);
});
});