Merge 'master' branch

Change-Id: I6070089128a150a4dd501bbc62a1f8b394aa643e
This commit is contained in:
Michal Niewrzal 2020-11-10 12:56:30 +01:00
parent 92f9251074
commit 7dde184cb5
101 changed files with 2948 additions and 1177 deletions

View File

@ -1,4 +1,5 @@
{
"message": "Thank you for your pull request and welcome to our community. We require contributors to sign our [Contributor License Agreement](https://docs.google.com/forms/d/e/1FAIpQLSdVzD5W8rx-J_jLaPuG31nbOzS8yhNIIu4yHvzonji6NeZ4ig/viewform), and we don't seem to have the users {{usersWithoutCLA}} on file. Once you have signed the CLA, please let us know, so we can manually review and add you to the approved contributors list.",
"contributors": [
"aleitner",
"aligeti",
@ -62,6 +63,7 @@
"montyanderson",
"sixcorners",
"alexottoboni",
"dominickmarino"
"dominickmarino",
"hectorj2f"
]
}

View File

@ -39,7 +39,7 @@ presents challenges when revoking a macaroon.
For example, if I hold the API key for Project A, I can create Macaroon A with a
caveat that it can only read and write files within Bucket A. I can
then share this macaroon with my own customer, Customer A. Customer A may then,
if they wish, create Macaroon B which is futher caveated -- for example,
if they wish, create Macaroon B which is further caveated -- for example,
restricted to read-only access in Bucket A -- and share Macaroon B with someone
else. This can occur without my knowledge.
@ -94,7 +94,7 @@ This approach was deemed best because it:
- Creates very little load on the database.
- Is backwards compatible, and allows us to revoke existing macaroons.
- Allows us to revoke an entire "macaroon tree" while maintaining the
distributive properies of macaroons.
distributive properties of macaroons.
Disadvantages to this approach:

View File

@ -30,7 +30,7 @@ When selecting nodes to store files to, the following criteria must be met:
- the node is not disqualified
- the node is not suspended
- the node has not exited
- the node has sufficent free disk space
- the node has sufficient free disk space
- the node has been contacted recently
- the node has participated in a sufficient number of audit
- the nodes has sufficient uptime counts
@ -102,7 +102,7 @@ For now, lets try out using an in-memory cache for the performanace gains. If we
3) Using a postgres materialized view for cached node data
Using a postgres materialized view to store all the vetted and unvetted nodes would allow us to implement this cache in the database layer instead of the application layer. This would require the application code to handle the refresh of the materialized view which could occur when one of the events from the #Update section happened.
Using a postgres materialized view to store all the vetted and unvetted nodes would allow us to implement this cache in the database layer instead of the application layer. This would require the application code to handle the refresh of the materialized view which could occur when one of the events from the #Update section happened.
Pro:
- allows the db to handle the logic instead of adding a cache at the application layer

View File

@ -13,7 +13,7 @@ The idea is to have two main components, like for Windows: the storage node bina
The parts we need:
- storagenode as a service
- a system for updating the storagenode binary aka. the updater (with rollout versioning support)
- a system for updating the updater
- a system for updating the updater
- a way to collect the configuration data from the user during the installation
- packaging to ship the above
@ -36,20 +36,20 @@ The installer will be a debian package. We choose to auto-update the binary, eve
- Email
- External address/port
- Advertised storage
- Identity directory
- Identity directory
- Storage directory
- Generate `config.yaml` file with the user configuration.
The default value for these directories can be defined using the [XDG Base Directory](https://wiki.archlinux.org/index.php/XDG_Base_Directory).
We choose to reuse the storagenode-updater and the recovery mechanism used in windows. They will be daemonized using systemd. The storagenode updater will auto-update. A recovery will be triggered if the updated updater service fails to restart.
We will use debconf to retrieve user data.
We will use debconf to retrieve user data.
The debian package will NOT contain the storagenode and storagenode-updater binaries. They will be downloaded as part of the post-installation script. A separate git repository will be created for holding the debian package.
Once we get a fully working debian package, we can convert it to the RPM format using the fpm tool. There are no debconf-like for RPMs, we will need to implement a post-install script to gather user inputs.
The debian package will be available by direct download and on a APT repository that users can add to their package manager source list. The repository will be managed using reprepro. Each time the repository is modified, it commits the static content to a dedicated git repository.
The debian package will be available by direct download and on a APT repository that users can add to their package manager source list. The repository will be managed using reprepro. Each time the repository is modified, it commits the static content to a dedicated git repository.
## Rationale
@ -62,7 +62,7 @@ Hence, we should use systemd for building our storagenode service.
Packaging in its simplest form would be tar.gz with an installation binary. This solution would be simple for us, but represents an annoyance for the user as our application would not be managed by their package manager.
#### Packages
A package is an archive file containing the application and metadata for indicating to the package manager how to install it.
A package is an archive file containing the application and metadata for indicating to the package manager how to install it.
Its format depends on the used package manager.
The most common formats are:
- deb for debian-based distributions
@ -79,7 +79,7 @@ The process for building a package is as follows:
- make a source package
- compile it to get binary packages.
Only the binary package is used by the user for installation. It is not a recommended pratice to directly integrate binaries.
Only the binary package is used by the user for installation. It is not a recommended practice to directly integrate binaries.
Building the source package is the most difficult part. But once it is done, we can use tools such as [fpm](https://github.com/jordansissel/fpm/wiki) to convert it to other package formats.
@ -100,7 +100,7 @@ There are [3 major agnostic packaging system](https://www.ostechnix.com/linux-pa
##### Snap
[Snaps](https://snapcraft.io/first-snap#go) are containerised software packages. They auto-update daily and work on a variety of Linux distributions. They also revert to the previous version if an update fails. This feature would make it necessary to find out how to implement the rollout versioning.
From the [snap documentation](https://snapcraft.io/docs/go-applications), it seems pretty straightforward to package an application. Snaps are defined in a yaml file. Running an application as a service is done only by specifying "daemon: simple" in the application description.
From the [snap documentation](https://snapcraft.io/docs/go-applications), it seems pretty straightforward to package an application. Snaps are defined in a yaml file. Running an application as a service is done only by specifying "daemon: simple" in the application description.
This would make us save the work of building a storage node service.
Snaps can then be published in the snapcraft [app store](https://snapcraft.io/). In the store, we would able to monitor the number of installed snaps. It is possible to [host our own store](https://ubuntu.com/blog/howto-host-your-own-snap-store) but that the snap daemon only handles one repository. Therefore, the use of Canonical's store seems mandatory. Snaps integrate well with [github](https://snapcraft.io/build).
@ -137,7 +137,7 @@ We are thinking of using native packaging for the following reasons:
- some linux users are reluctant to use snap
- covering deb and rpm packaging would make us cover most used distributions
- with proper packaging, we could directly be included in the distributions
## Implementation
### Debian package
- create a storj debian git
@ -184,4 +184,4 @@ We still need to support docker images. The Docker image we provide should make
## Wrapup
- As a first step and as part of the PoC, the git repository and the debian package skeleton will be created.
- The PoC will create the user and the directories, download a binary (will not check for the latest) and install a basic storagenode systemd service.
- The PoC will also contain first Dockerfile for the reprepro repository.
- The PoC will also contain first Dockerfile for the reprepro repository.

View File

@ -15,7 +15,7 @@ That leaves open the question for how a root key is created. Some requirements o
These requirements allow users to be in full control of their encryption, and don't require users safely transporting high entropy (hard to remember) secrets to bootstrap new uplinks.
This design accomodates more requirements that allow for additional features:
This design accommodates more requirements that allow for additional features:
3. A root key can be created for any encrypted path in a bucket, not just the bucket.
4. A table of root keys for low entropy passwords should not be possible. In other words, an attacker with knowledge of the algorithm should not be able to use a dictionary of common passwords and pre-compute what keys to check in the event of a data breach.

View File

@ -9,9 +9,9 @@ The satellite should repair files also using piece hashes to minimize CPU and ba
The white-paper states:
> Data repair is an ongoing, costly operation that will use significant bandwidth, memory, and processing power, often impacting a single operator. As a result, repair resource usage should be aggressively minimized as much as possible.
>
>
> For repairing a segment to be effective at minimizing bandwidth usage, only as few pieces as needed for reconstruction should be downloaded. Unfortunately, Reed-Solomon is insufficient on its own for correcting errors when only a few redundant pieces are provided. Instead, piece hashes provide a better way to be confident that were repairing the data correctly.
>
>
> To solve this problem, hashes of every piece will be stored alongside each piece on each storage node. A validation hash that the set of hashes is correct will be stored in the pointer. During repair, the hashes of every piece can be retrieved and validated for correctness against the pointer, thus allowing each piece to be validated in its entirety. This allows the repair system to correctly assess whether or not repair has been completed successfully without using extra redundancy for the same task.
Hash verification on the satellite requires understanding the current piece signing and verification workflow:
@ -41,7 +41,7 @@ Downloading for repair is significantly different enough from streaming as to wa
Using only the minimum number of pieces means that Reed-Solomon does not act as a check during repair. Hence hashing is used instead. While an uplink could potentially send signed bogus data to a storage node, the storage node would not be penalized by these actions. This requires that Audit implements a similar piece hash check instead of relying solely on Reed-Solomon encoding.
The size of all piece hashes downloaded should be roughly equal to a default maximum segment size : 64MiB. It seems preferable to keep this in memory over dealing with persistance to disk.
The size of all piece hashes downloaded should be roughly equal to a default maximum segment size : 64MiB. It seems preferable to keep this in memory over dealing with persistence to disk.
## Implementation

View File

@ -2,7 +2,7 @@
Satellite billing system combines stripe and coinpayments API for credit card and cryptocurrency processing. It uses `satellite/accounting` pkg for project accounting. Billing is set on account but that is the subject for future changes as we want billing to be on a project level. That requires decoupling stripe dependency to the level where we utilize only credit card processing and maintain all other stuff such as customer balances and invoicing internally. Every satellite should have separate stripe and coinpayments account to prevent collision of customer related data such as uuid and email.
# Stripe customer
Stripe operates on a basis of customers. Where customer is, from stripe doc: Customer objects allow you to perform recurring charges, and to track multiple charges, that are associated with the same customer. The API allows you to create, delete, and update your customers. You can retrieve individual customers as well as a list of all your customers. Satellite billing doesn't uses `customer` concern with public API, so it is treated as implementation detail. Stripe customer balance is automatically applied to invoice total before charging a credit card.
Stripe operates on a basis of customers. Where customer is, from stripe doc: Customer objects allow you to perform recurring charges, and to track multiple charges, that are associated with the same customer. The API allows you to create, delete, and update your customers. You can retrieve individual customers as well as a list of all your customers. Satellite billing doesn't uses `customer` concern with public API, so it is treated as implementation detail. Stripe customer balance is automatically applied to invoice total before charging a credit card.
Stripe billing system implementation stores a customer reference for every user:
```
@ -51,7 +51,7 @@ type Accounts interface {
```
# Customer setup
Every satellite user has a corresponding customer entity on stripe which holds credit cards, balance which reflects the ammount of STORJ tokens, and is used for invoicing. Every time a user visits billing page on the satellite UI we try to create a customer for him if one doesn't exists.
Every satellite user has a corresponding customer entity on stripe which holds credit cards, balance which reflects the amount of STORJ tokens, and is used for invoicing. Every time a user visits billing page on the satellite UI we try to create a customer for him if one doesn't exists.
```go
// Setup creates a payment account for the user.
// If account is already set up it will return nil.
@ -241,7 +241,7 @@ type Chore struct {
```
# STORJ tokens processsing
Unlike with credit cards billing system uses deposit model for STORJ tokens, user has to deposit some amount prior using satellite services.
Unlike with credit cards billing system uses deposit model for STORJ tokens, user has to deposit some amount prior using satellite services.
Public API of token related billing:
```go
@ -257,7 +257,7 @@ type StorjTokens interface {
```
# Making a deposit
STORJ cryptocurrency processing is done via coinpayments API. Every time a user wants to deposit some amount of STORJ token to his account balacne, new coinpayments transaction is created. Transaction amount is set in USD, and conversion rates is beeing locked(saved) after transaction is created and stored in the db.
STORJ cryptocurrency processing is done via coinpayments API. Every time a user wants to deposit some amount of STORJ token to his account balacne, new coinpayments transaction is created. Transaction amount is set in USD, and conversion rates is being locked(saved) after transaction is created and stored in the db.
```go
// Deposit creates new deposit transaction with the given amount returning
// ETH wallet address where funds should be sent. There is one
@ -403,7 +403,7 @@ func (tokens *storjTokens) ListTransactionInfos(ctx context.Context, userID uuid
```
# Transaction update cycle
There is a cycle that iterates over all `pending`(`pending` and `paid` statuses of coinpayments transaction respectively) transactions, list it's infos and updates tx status and received amount. If updated is status is set to `cancelled` or `completed`, that transactions won't take part in the next update cycle. When there is a status transation to `completed` along with the update `apply_balance_transaction_intent` is created. Transaction with status `completed` and present `apply_balance_transaction_intent` with state `unapplied` defines as `UnappliedTransaction` which is later processed in update balance cycle. If the received amount is greater that 50$ a promotional coupon for 55$ is created.
There is a cycle that iterates over all `pending`(`pending` and `paid` statuses of coinpayments transaction respectively) transactions, list it's infos and updates tx status and received amount. If updated is status is set to `cancelled` or `completed`, that transactions won't take part in the next update cycle. When there is a status transaction to `completed` along with the update `apply_balance_transaction_intent` is created. Transaction with status `completed` and present `apply_balance_transaction_intent` with state `unapplied` defines as `UnappliedTransaction` which is later processed in update balance cycle. If the received amount is greater that 50$ a promotional coupon for 55$ is created.
```go
// updateTransactions updates statuses and received amount for given transactions.
func (service *Service) updateTransactions(ctx context.Context, ids TransactionAndUserList) (err error) {
@ -501,7 +501,7 @@ func (service *Service) applyTransactionBalance(ctx context.Context, tx Transact
```
# Invoices
Invoices are statements of amounts owed by a customer, and are generated one-off.
Invoices are statements of amounts owed by a customer, and are generated one-off.
```go
// Invoice holds all public information about invoice.
type Invoice struct {
@ -526,7 +526,7 @@ type Invoices interface {
```
# Invoice creation
Invoice include project usage cost as well as any discounts applied. Coupons and credits applied as separate invoice line items, therefore it reduce total due amount. Next applied STORJ token amount which is repesented as credits on custmer balance if any. If invoice total amount is greater than zero after bonuses and STORJ tokens, default credit card at the moment of invoice creation will be charged. If total amount is less than 1$, then stripe won't try to charge credit card but increase debt on customer balance.
Invoice include project usage cost as well as any discounts applied. Coupons and credits applied as separate invoice line items, therefore it reduce total due amount. Next applied STORJ token amount which is repesented as credits on custmer balance if any. If invoice total amount is greater than zero after bonuses and STORJ tokens, default credit card at the moment of invoice creation will be charged. If total amount is less than 1$, then stripe won't try to charge credit card but increase debt on customer balance.
Invoice creation consist of few steps. First invoice project records have to be created. Each record consist of project id, usage and timestamps of the start and end of billing period. This way we ensure that usage is the same during all invoice creation steps and there won't be two or more invoices created for the same period(actually only invoice line items for certain billing period and project are ensured not to be created more than once). Coupon usages are also created during this step, which are later used to create coupon invoice line items.
@ -548,7 +548,7 @@ prepare-invoice-records Prepares invoice project records that will be used durin
```bash
inspector payments prepare-invoice-records [mm/yyyy]
```
Create project records for all projects for specified billing period. Billing period defined as `[0th nanosecond of the first day of the month; 0th nanosecond of the first day of the following month)`.
Create project records for all projects for specified billing period. Billing period defined as `[0th nanosecond of the first day of the month; 0th nanosecond of the first day of the following month)`.
Project record contains project usage for some billing period. Therefore, it is impossible to create project record for the same project and billing period.
```go
// ProjectRecord holds project usage particular for billing period.
@ -608,7 +608,7 @@ func (service *Service) PrepareInvoiceProjectRecords(ctx context.Context, period
return nil
}
```
If a project record already exists, project is skipped.
If a project record already exists, project is skipped.
```go
// createProjectRecords creates invoice project record if none exists.
func (service *Service) createProjectRecords(ctx context.Context, projects []console.Project, start, end time.Time) (err error) {
@ -696,31 +696,31 @@ Iterate over all project records, calculating price and creating invoice line it
// applyProjectRecords applies invoice intents as invoice line items to stripe customer.
func (service *Service) applyProjectRecords(ctx context.Context, records []ProjectRecord) (err error) {
defer mon.Task()(&ctx)(&err)
for _, record := range records {
if err = ctx.Err(); err != nil {
return err
}
proj, err := service.projectsDB.Get(ctx, record.ProjectID)
if err != nil {
return err
}
cusID, err := service.db.Customers().GetCustomerID(ctx, proj.OwnerID)
if err != nil {
if err == ErrNoCustomer {
continue
}
return err
}
if err = service.createInvoiceItems(ctx, cusID, proj.Name, record); err != nil {
return err
}
}
return nil
}
```
@ -763,46 +763,46 @@ Iterate over all customers and create invoice for each.
// CreateInvoices lists through all customers and creates invoices.
func (service *Service) CreateInvoices(ctx context.Context) (err error) {
defer mon.Task()(&ctx)(&err)
const limit = 25
before := time.Now()
cusPage, err := service.db.Customers().List(ctx, 0, limit, before)
if err != nil {
return Error.Wrap(err)
}
for _, cus := range cusPage.Customers {
if err = ctx.Err(); err != nil {
return Error.Wrap(err)
}
if err = service.createInvoice(ctx, cus.ID); err != nil {
return Error.Wrap(err)
}
}
for cusPage.Next {
if err = ctx.Err(); err != nil {
return Error.Wrap(err)
}
cusPage, err = service.db.Customers().List(ctx, cusPage.NextOffset, limit, before)
if err != nil {
return Error.Wrap(err)
}
for _, cus := range cusPage.Customers {
if err = ctx.Err(); err != nil {
return Error.Wrap(err)
}
if err = service.createInvoice(ctx, cus.ID); err != nil {
return Error.Wrap(err)
}
}
}
return nil
}
```
@ -833,4 +833,4 @@ func (service *Service) createInvoice(ctx context.Context, cusID string) (err er
return nil
}
```
```

View File

@ -274,7 +274,7 @@ and are left with the following:
The list of trusted Satellite URLs should be recalculated daily (with some jitter).
### Backwards Compatability
### Backwards Compatibility
The old piecestore configuration (i.e. `piecestore.OldConfig`) currently contains a
comma separated list of trusted Satellite URLs (`WhitelistedSatellites`). It
@ -296,7 +296,7 @@ a fixed set of trusted Satellite URLs.
* Implement a `trust.ListConfig` configuration struct which:
* Contains the list of entries (with a release default of a single list containing `https://www.tardigrade.io/trusted-satellites`)
* Contains a refresh interval
* Maintains backwards compatability with `WhitelistedSatellites` in `piecestore.OldConfig`
* Maintains backwards compatibility with `WhitelistedSatellites` in `piecestore.OldConfig`
* Implement `storj.io/storj/storagenode/trust.List` that:
* Consumes `trust.ListConfig` for configuration
* Performs the initial fetching and building of trusted Satellite URLs

View File

@ -87,7 +87,7 @@ Create `satellites_exit_progress` tables:
```
model satellite_exit_progress (
fk satellite_id
fk satellite_id
field initiated_at timestamp ( updateable )
field finished_at timestamp ( updateable )

View File

@ -8,7 +8,7 @@ This document describes how storage node transfers its pieces during Graceful Ex
## Background
During Graceful Exit a storage node needs to transfer pieces to other nodes. During transfering the storage node or satellite may crash, hence it needs to be able to continue after a restart.
During Graceful Exit a storage node needs to transfer pieces to other nodes. During transferring the storage node or satellite may crash, hence it needs to be able to continue after a restart.
Satellite gathers transferred pieces list asynchronously, which is described in [Gathering Pieces Document](pieces.md). This may consume a significant amount of time.
@ -28,11 +28,11 @@ The `worker` should continue to poll the satellite at a configurable interval un
The satellite should return pieces to transfer from the transfer queue if piece durability <= optimal. If durability > optimal, we remove the exiting node from the segment / pointer.
The storage node should concurrently transfer pieces returned by the satellite. The storage node should send a `TransferSucceeded` message as pieces are successfuly transfered. The Storage node should send a `TransferFailed`, with reason, on failure.
The storage node should concurrently transfer pieces returned by the satellite. The storage node should send a `TransferSucceeded` message as pieces are successfully transferred. The Storage node should send a `TransferFailed`, with reason, on failure.
The satellites should set the `finished_at` on success, and respond with a `DeletePiece` message. Otherwise increment `failed_count` and set the `last_failed_at` and `last_failed_code` for reprocessing.
The satellite should respond with an `ExitCompleted` message when all pieces have finished processing.
The satellite should respond with an `ExitCompleted` message when all pieces have finished processing.
If the storage node has failed too many transfers overall, failed the same piece over a certain threshold, or has sent incorrect data, the satellite will send an `ExitFailed` message. This indicates that the process has ended ungracefully.
@ -75,7 +75,7 @@ We could have a separate initiate graceful exit RPC, however this would complica
## Implementation
1. Add protobuf definitions.
2. Update node selection to ignore exiting nodes for repairs and uploads.
2. Update node selection to ignore exiting nodes for repairs and uploads.
3. Update repairer to repair segments for nodes that failed an exit.
4. Implement verifying a transfer on the satellite.
5. Implement transferring a single piece on storage node.
@ -143,7 +143,7 @@ when storage node prematurely exits
go func() {
for {
ensure we have only up to N inprogress at the same time
list transferred piece that is not in progress
if no pieces {
morepieces = false

7
go.mod
View File

@ -38,14 +38,13 @@ require (
go.etcd.io/bbolt v1.3.5
go.uber.org/zap v1.16.0
golang.org/x/crypto v0.0.0-20200820211705-5c72a883971a
golang.org/x/net v0.0.0-20200226121028-0de0cce0169b // indirect
golang.org/x/sync v0.0.0-20200625203802-6e8e738ad208
golang.org/x/sys v0.0.0-20200929083018-4d22bbb62b3c
golang.org/x/time v0.0.0-20200630173020-3af7569d3a1e
google.golang.org/api v0.20.0 // indirect
storj.io/common v0.0.0-20201030120157-90ae6720d87e
storj.io/drpc v0.0.14
storj.io/common v0.0.0-20201106104920-372a344bdd45
storj.io/drpc v0.0.16
storj.io/monkit-jaeger v0.0.0-20200518165323-80778fc3f91b
storj.io/private v0.0.0-20201026143115-bc926bfa3bca
storj.io/uplink v1.3.2-0.20201028181609-f6efc8fcf771
storj.io/uplink v1.3.2-0.20201106105834-ec8e5cc29f7e
)

154
go.sum
View File

@ -1,5 +1,7 @@
cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
cloud.google.com/go v0.31.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
cloud.google.com/go v0.34.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
cloud.google.com/go v0.37.0/go.mod h1:TS1dMSSfndXH133OKGwekG838Om/cQT0BUHV3HcBgoo=
cloud.google.com/go v0.37.4/go.mod h1:NHPJ89PdicEuT9hdPXMROBD91xc5uRDxsMtSB16k7hw=
cloud.google.com/go v0.38.0/go.mod h1:990N+gfupTy94rShfmMCWGDn0LpTmnzTp2qbd1dvSRU=
cloud.google.com/go v0.44.1/go.mod h1:iSa0KzasP4Uvy3f1mN/7PiObzGgflwredwwASm/v6AU=
@ -17,7 +19,12 @@ cloud.google.com/go/pubsub v1.0.1 h1:W9tAK3E57P75u0XLLR82LZyw8VpAnhmyTOxW9qzmyj8
cloud.google.com/go/pubsub v1.0.1/go.mod h1:R0Gpsv3s54REJCy4fxDixWD93lHJMoZTyQ2kNxGRt3I=
cloud.google.com/go/storage v1.0.0 h1:VV2nUM3wwLLGh9lSABFgZMjInyUbJeaRSE64WuAIQ+4=
cloud.google.com/go/storage v1.0.0/go.mod h1:IhtSnM/ZTZV8YYJWCY8RULGVqBDmpoyjwiyrjsg+URw=
dmitri.shuralyov.com/app/changes v0.0.0-20180602232624-0a106ad413e3/go.mod h1:Yl+fi1br7+Rr3LqpNJf1/uxUdtRUV+Tnj0o93V2B9MU=
dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU=
dmitri.shuralyov.com/html/belt v0.0.0-20180602232347-f7d459c86be0/go.mod h1:JLBrvjyP0v+ecvNYvCpyZgu5/xkfAUhi6wJj28eUfSU=
dmitri.shuralyov.com/service/change v0.0.0-20181023043359-a85b471d5412/go.mod h1:a1inKt/atXimZ4Mv927x+r7UpyzRUf4emIoiiSC2TN4=
dmitri.shuralyov.com/state v0.0.0-20180228185332-28bcc343414c/go.mod h1:0PRwlb0D6DFvNNtx+9ybjezNCa8XF0xaYcETyp6rHWU=
git.apache.org/thrift.git v0.0.0-20180902110319-2566ecd5d999/go.mod h1:fPE2ZNJGynbRyZ4dJvy6G277gSllfV2HJqblrnkyeyg=
github.com/Azure/go-ansiterm v0.0.0-20170929234023-d6e3b3328b78 h1:w+iIsaOQNcT7OZ575w+acHgRric5iCyQh+xv+KJ4HB8=
github.com/Azure/go-ansiterm v0.0.0-20170929234023-d6e3b3328b78/go.mod h1:LmzpDX56iTiv29bbRTIsUNlaFfuhWRQBWjQdVyAevI8=
github.com/BurntSushi/toml v0.3.1 h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ=
@ -40,6 +47,7 @@ github.com/alicebob/gopher-json v0.0.0-20200520072559-a9ecdc9d1d3a h1:HbKu58rmZp
github.com/alicebob/gopher-json v0.0.0-20200520072559-a9ecdc9d1d3a/go.mod h1:SGnFV6hVsYE877CKEZ6tDNTjaSXYUk6QqoIK6PrAtcc=
github.com/alicebob/miniredis/v2 v2.13.3 h1:kohgdtN58KW/r9ZDVmMJE3MrfbumwsDQStd0LPAGmmw=
github.com/alicebob/miniredis/v2 v2.13.3/go.mod h1:uS970Sw5Gs9/iK3yBg0l9Uj9s25wXxSpQUE9EaJ/Blg=
github.com/anmitsu/go-shlex v0.0.0-20161002113705-648efa622239/go.mod h1:2FmKhYUyUczH0OGQWaF5ceTx0UBShxjsH6f8oGKYe2c=
github.com/apache/thrift v0.12.0 h1:pODnxUFNcjP9UTLZGTdeh+j16A8lJbRvD3rOtrk/7bs=
github.com/apache/thrift v0.12.0/go.mod h1:cp2SuWMxlEZw2r+iP2GNCdIi4C1qmUzdZFSVb+bacwQ=
github.com/armon/circbuf v0.0.0-20150827004946-bbbad097214e/go.mod h1:3U/XgcO3hCbHZ8TKRvWD2dDTCfh9M9ya+I9JpbB7O8o=
@ -55,6 +63,7 @@ github.com/bketelsen/crypt v0.0.3-0.20200106085610-5cbc8cc4026c/go.mod h1:MKsuJm
github.com/blang/semver v3.5.1+incompatible h1:cQNTCjp13qL8KC3Nbxr/y2Bqb63oX6wdnnjpJbkM4JQ=
github.com/blang/semver v3.5.1+incompatible/go.mod h1:kRBLl5iJ+tD4TcOOxsy/0fnwebNt5EWlYSAyrTnjyyk=
github.com/bmizerany/assert v0.0.0-20160611221934-b7ed37b82869/go.mod h1:Ekp36dRnpXw/yCqJaO+ZrUyxD+3VXMFFr56k5XYrpB4=
github.com/bradfitz/go-smtpd v0.0.0-20170404230938-deb6d6237625/go.mod h1:HYsPBTaaSFSlLx/70C2HPIMNZpVV8+vt/A+FMnYP11g=
github.com/btcsuite/btcd v0.20.1-beta/go.mod h1:wVuoA8VJLEcwgqHBwHmzLRazpKxTv13Px/pDuV7OomQ=
github.com/btcsuite/btclog v0.0.0-20170628155309-84c8d2346e9f/go.mod h1:TdznJufoqS23FtqVCzL0ZqgP5MqXbb4fg/WgDys70nA=
github.com/btcsuite/btcutil v0.0.0-20190425235716-9e5f4b9a998d/go.mod h1:+5NJ2+qvTyV9exUAL/rxXi3DcLg2Ts+ymUAY5y4NvMg=
@ -66,12 +75,15 @@ github.com/btcsuite/goleveldb v0.0.0-20160330041536-7834afc9e8cd/go.mod h1:F+uVa
github.com/btcsuite/snappy-go v0.0.0-20151229074030-0bdef8d06723/go.mod h1:8woku9dyThutzjeg+3xrA5iCpBRH8XEEg3lh6TiUghc=
github.com/btcsuite/websocket v0.0.0-20150119174127-31079b680792/go.mod h1:ghJtEyQwv5/p4Mg4C0fgbePVuGr935/5ddU9Z3TmDRY=
github.com/btcsuite/winsvc v1.0.0/go.mod h1:jsenWakMcC0zFBFurPLEAyrnc/teJEM1O46fmI40EZs=
github.com/buger/jsonparser v0.0.0-20181115193947-bf1c66bbce23/go.mod h1:bbYlZJ7hK1yFx9hf58LP0zeX7UjIGs20ufpu3evjr+s=
github.com/calebcase/tmpfile v1.0.1/go.mod h1:iErLeG/iqJr8LaQ/gYRv4GXdqssi3jg4iSzvrA06/lw=
github.com/calebcase/tmpfile v1.0.2-0.20200602150926-3af473ef8439/go.mod h1:iErLeG/iqJr8LaQ/gYRv4GXdqssi3jg4iSzvrA06/lw=
github.com/calebcase/tmpfile v1.0.2 h1:1AGuhKiUu4J6wxz6lxuF6ck3f8G2kaV6KSEny0RGCig=
github.com/calebcase/tmpfile v1.0.2/go.mod h1:iErLeG/iqJr8LaQ/gYRv4GXdqssi3jg4iSzvrA06/lw=
github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc=
github.com/cheekybits/genny v1.0.0 h1:uGGa4nei+j20rOSeDeP5Of12XVm7TGUd4dJA9RDitfE=
github.com/cheekybits/genny v1.0.0/go.mod h1:+tQajlRqAUrPI7DOSpB0XAqZYtQakVtB7wXkRAgjxjQ=
github.com/cheggaaa/pb/v3 v3.0.5 h1:lmZOti7CraK9RSjzExsY53+WWfub9Qv13B5m4ptEoPE=
github.com/cheggaaa/pb/v3 v3.0.5/go.mod h1:X1L61/+36nz9bjIsrDU52qHKOQukUQe2Ge+YvGuquCw=
github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI=
@ -90,6 +102,7 @@ github.com/coreos/etcd v3.3.10+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc
github.com/coreos/etcd v3.3.13+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE=
github.com/coreos/go-semver v0.2.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk=
github.com/coreos/go-semver v0.3.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk=
github.com/coreos/go-systemd v0.0.0-20181012123002-c6f51f82210d/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4=
github.com/coreos/go-systemd v0.0.0-20190321100706-95778dfbb74e/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4=
github.com/coreos/go-systemd v0.0.0-20190719114852-fd7a80b32e1f/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4=
github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f/go.mod h1:E3G3o1h8I7cfcXa63jLwjI0eiQQMgzzUDFVpN/nH/eA=
@ -123,6 +136,7 @@ github.com/docker/go-connections v0.4.0 h1:El9xVISelRB7BuFusrZozjnkIM5YnzCViNKoh
github.com/docker/go-connections v0.4.0/go.mod h1:Gbd7IOopHjR8Iph03tsViu4nIes5XhDvyHbTtUxmeec=
github.com/docker/go-units v0.3.3 h1:Xk8S3Xj5sLGlG5g67hJmYMmUgXv5N4PhkjJHHqrwnTk=
github.com/docker/go-units v0.3.3/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk=
github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk=
github.com/eapache/go-resiliency v1.1.0/go.mod h1:kFI+JgMyC7bLPUVY133qvEBtVayf5mFgVsvEsIPBvNs=
github.com/eapache/go-xerial-snappy v0.0.0-20180814174437-776d5712da21/go.mod h1:+020luEh2TKB4/GOp8oxxtq0Daoen/Cii55CzbTV6DU=
github.com/eapache/queue v1.1.0/go.mod h1:6eCeP0CKFpHLu8blIFXhExK/dRa7WDZfr6jVFPTqq+I=
@ -132,10 +146,16 @@ github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7
github.com/fatih/color v1.7.0/go.mod h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5KwzbycvMj4=
github.com/fatih/color v1.9.0 h1:8xPHl4/q1VyqGIPif1F+1V3Y3lSmrq01EabUW3CoW5s=
github.com/fatih/color v1.9.0/go.mod h1:eQcE1qtQxscV5RaZvpXrrb8Drkc3/DdQ+uUYCNjL+zU=
github.com/flynn/go-shlex v0.0.0-20150515145356-3f9db97f8568/go.mod h1:xEzjJPgXI435gkrCt3MPfRiAkVrwSbHsst4LCFVfpJc=
github.com/francoispqt/gojay v1.2.13/go.mod h1:ehT5mTG4ua4581f1++1WLG0vPdaA9HaiDsoyrBGkyDY=
github.com/fsnotify/fsnotify v1.4.7 h1:IXs+QLmnXW2CcXuY+8Mzv/fWEsPGWxqefPtCP5CnV9I=
github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
github.com/fsnotify/fsnotify v1.4.9 h1:hsms1Qyu0jgnwNXIxa+/V/PDsU6CfLf6CNO8H7IWoS4=
github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ=
github.com/fsouza/fake-gcs-server v1.7.0/go.mod h1:5XIRs4YvwNbNoz+1JF8j6KLAyDh7RHGAyAK3EP2EsNk=
github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
github.com/gliderlabs/ssh v0.1.1/go.mod h1:U7qILu1NlMHj9FlMhZLlkCdDnU1DBEAqr0aevW3Awn0=
github.com/go-errors/errors v1.0.1/go.mod h1:f4zRHt4oKfwPJE5k8C9vpYG+aDHdBFUsgrm6/TyX73Q=
github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9AVAgeJqvqgH9Q5CA+iKCZ2gyEVpxRU=
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
@ -159,16 +179,28 @@ github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b h1:VKtxabqXZkF25pY9ekf
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
github.com/golang/groupcache v0.0.0-20190129154638-5b532d6fd5ef/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/groupcache v0.0.0-20191027212112-611e8accdfc9/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7 h1:5ZkaAPbicIKTF2I64qf5Fh8Aa83Q/dnOafMYV0OMwjA=
github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/lint v0.0.0-20180702182130-06c8688daad7/go.mod h1:tluoj9z5200jBnyusfRPU2LqT6J+DAorxEvtC7LHB+E=
github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
github.com/golang/mock v1.2.0/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
github.com/golang/mock v1.3.1 h1:qGJ6qTW+x6xX/my+8YUVl4WNpX9B7+/l2tRsHGZ7f2s=
github.com/golang/mock v1.3.1/go.mod h1:sBzyDLLjw3U8JLTeZvSv8jJB+tU5PVekmnlKIyFUx0Y=
github.com/golang/mock v1.4.0/go.mod h1:UOMv5ysSaYNkG+OFQykRIcU/QvvxJf3p21QfJ2Bt3cw=
github.com/golang/mock v1.4.4 h1:l75CXGRSwbaYNpl/Z2X1XIIAMSCquvXgpVZDhwEIJsc=
github.com/golang/mock v1.4.4/go.mod h1:l3mdAwkq5BuhzHwde/uurv3sEJeZMXNpwsxVWU71h+4=
github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.2 h1:6nsPYzhq5kReh6QImI3k5qWzO4PEbvbIW2cwSfR/6xs=
github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.4.0-rc.1/go.mod h1:ceaxUfeHdC40wWswd/P6IGgMaK3YpKi5j83Wpe3EHw8=
github.com/golang/protobuf v1.4.0-rc.1.0.20200221234624-67d41d38c208/go.mod h1:xKAWHe0F5eneWXFV3EuXVDTCmh+JuBKY0li0aMyXATA=
github.com/golang/protobuf v1.4.0-rc.2/go.mod h1:LlEzMj4AhA7rCAGe4KMBDvJI+AwstrUpVNzEA03Pprs=
github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0/go.mod h1:WU3c8KckQ9AFe+yFwt9sWVRKCVIyN9cPHBJSNnbL67w=
github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvqG2KuDX0=
github.com/golang/protobuf v1.4.2 h1:+Z5KGCizgyZCbGh1KZqA0fcLLkwbsjIzS4aV2v7wJX0=
github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI=
github.com/golang/snappy v0.0.0-20170215233205-553a64147049/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
github.com/golang/snappy v0.0.1/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
@ -176,6 +208,7 @@ github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c/go.mod h1:lNA+9X1NB3Z
github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M=
github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.2 h1:X2ev0eStA3AbceY54o37/0PQ/UWqKEiiO2dKL5OPaFM=
github.com/google/go-cmp v0.5.2/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
@ -189,6 +222,9 @@ github.com/google/pprof v0.0.0-20191218002539-d4f498aebedc/go.mod h1:ZgVRPoUq/hf
github.com/google/pprof v0.0.0-20200229191704-1ebb73c60ed3 h1:SRgJV+IoxM5MKyFdlSUeNy6/ycRUF2yBAKdAQswoHUk=
github.com/google/pprof v0.0.0-20200229191704-1ebb73c60ed3/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI=
github.com/googleapis/gax-go v2.0.0+incompatible h1:j0GKcs05QVmm7yesiZq2+9cxHkNK9YM6zKx4D2qucQU=
github.com/googleapis/gax-go v2.0.0+incompatible/go.mod h1:SFVmujtThgffbyetf+mdk2eWhX2bMyUtNHzFKcPA9HY=
github.com/googleapis/gax-go/v2 v2.0.3/go.mod h1:LLvjysVCY1JZeum8Z6l8qUty8fiNwE08qbEPm1M08qg=
github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg=
github.com/googleapis/gax-go/v2 v2.0.5 h1:sjZBwGj9Jlw33ImPtvFviGYvseOtDM7hkSKB7+Tv3SM=
github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk=
@ -205,8 +241,10 @@ github.com/gorilla/websocket v1.4.0/go.mod h1:E7qHFY5m1UJ88s3WnNqhKjPHQ0heANvMoA
github.com/gorilla/websocket v1.4.2/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
github.com/graphql-go/graphql v0.7.9 h1:5Va/Rt4l5g3YjwDnid3vFfn43faaQBq7rMcIZ0VnV34=
github.com/graphql-go/graphql v0.7.9/go.mod h1:k6yrAYQaSP59DC5UVxbgxESlmVyojThKdORUqGDGmrI=
github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7/go.mod h1:FecbI9+v66THATjSRHfNgh1IVFe/9kFxbXtjV0ctIMA=
github.com/grpc-ecosystem/go-grpc-middleware v1.0.0/go.mod h1:FiyG127CGDf3tlThmgyCl78X/SZQqEOJBCDaAfeWzPs=
github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0/go.mod h1:8NvIoxWQoOIhqOTXgfV/d3M/q6VIi02HzZEHgUlZvzk=
github.com/grpc-ecosystem/grpc-gateway v1.5.0/go.mod h1:RSKVYQBd5MCa4OVpNdGskqpgL2+G+NZTnrVHpWWfpdw=
github.com/grpc-ecosystem/grpc-gateway v1.9.0/go.mod h1:vNeuVxBJEsws4ogUvrchl83t/GYV9WGTSLVdBhOQFDY=
github.com/hailocab/go-hostpool v0.0.0-20160125115350-e80d13ce29ed/go.mod h1:tMWxXQ9wFIaZeTI9F+hmhFiGpFmhOHzyShyFUhRm0H4=
github.com/hashicorp/consul/api v1.1.0/go.mod h1:VmuI/Lkw1nC05EYQWNKwWGbkg+FbDBtguAZLlVdkD9Q=
@ -294,6 +332,7 @@ github.com/jackc/puddle v0.0.0-20190608224051-11cab39313c9/go.mod h1:m4B5Dj62Y0f
github.com/jackc/puddle v1.1.0/go.mod h1:m4B5Dj62Y0fbyuIc15OsIqK0+JU8nkqQjsgx7dvjSWk=
github.com/jackc/puddle v1.1.1/go.mod h1:m4B5Dj62Y0fbyuIc15OsIqK0+JU8nkqQjsgx7dvjSWk=
github.com/jackc/puddle v1.1.2/go.mod h1:m4B5Dj62Y0fbyuIc15OsIqK0+JU8nkqQjsgx7dvjSWk=
github.com/jellevandenhooff/dkim v0.0.0-20150330215556-f50fe3d243e1/go.mod h1:E0B/fFc00Y+Rasa88328GlI/XbtyysCtTHZS8h7IrBU=
github.com/jessevdk/go-flags v0.0.0-20141203071132-1679536dcc89/go.mod h1:4FA24M0QyGHXBuZZK/XkWh8h0e1EYbRYJSGM75WSRxI=
github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af/go.mod h1:Nht3zPeWKUH0NzdCt2Blrr5ys8VGpn0CEB0cQHVjt7k=
github.com/jonboulle/clockwork v0.1.0/go.mod h1:Ii8DK3G1RaLaWxj9trq07+26W01tbo22gdxWY5EU2bo=
@ -321,6 +360,7 @@ github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515/go.mod h1:+0opPa2QZZtGFB
github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI=
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
github.com/kr/pty v1.1.3/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
github.com/kr/pty v1.1.8/go.mod h1:O1sed60cT9XZ5uDucP5qwvh+TE3NnUj51EiZO/lmSfw=
github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
@ -330,9 +370,18 @@ github.com/lib/pq v1.1.0/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo=
github.com/lib/pq v1.2.0/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo=
github.com/lib/pq v1.3.0 h1:/qkRGz8zljWiDcFvgpwUpwIAPu3r07TDvs3Rws+o/pU=
github.com/lib/pq v1.3.0/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo=
github.com/lucas-clemente/quic-go v0.7.1-0.20201102053916-272229abf044 h1:M6zB4Rs4SJDk9IBIvC3ozl23+b0d1Q7NOlHnbxuc3AY=
github.com/lucas-clemente/quic-go v0.7.1-0.20201102053916-272229abf044/go.mod h1:ZUygOqIoai0ASXXLJ92LTnKdbqh9MHCLTX6Nr1jUrK0=
github.com/lunixbochs/vtclean v1.0.0/go.mod h1:pHhQNgMf3btfWnGBVipUOjRYhoOsdGqdm/+2c2E2WMI=
github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
github.com/magiconair/properties v1.8.1 h1:ZC2Vc7/ZFkGmsVC9KvOjumD+G5lXy2RtTKyzRKO2BQ4=
github.com/magiconair/properties v1.8.1/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
github.com/mailru/easyjson v0.0.0-20190312143242-1de009706dbe/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc=
github.com/marten-seemann/qpack v0.2.1/go.mod h1:F7Gl5L1jIgN1D11ucXefiuJS9UMVP2opoCp2jDKb7wc=
github.com/marten-seemann/qtls v0.10.0 h1:ECsuYUKalRL240rRD4Ri33ISb7kAQ3qGDlrrl55b2pc=
github.com/marten-seemann/qtls v0.10.0/go.mod h1:UvMd1oaYDACI99/oZUYLzMCkBXQVT0aGm99sJhbT8hs=
github.com/marten-seemann/qtls-go1-15 v0.1.1 h1:LIH6K34bPVttyXnUWixk0bzH6/N07VxbSabxn5A5gZQ=
github.com/marten-seemann/qtls-go1-15 v0.1.1/go.mod h1:GyFwywLKkRt+6mfU99csTEY1joMZz5vmB1WNZH3P81I=
github.com/mattn/go-colorable v0.0.9/go.mod h1:9vuHe8Xs5qXnSaW/c/ABM9alt+Vo+STaOChaDxuIBZU=
github.com/mattn/go-colorable v0.1.1/go.mod h1:FuOcm+DKB9mbwrcAfNl7/TZVBZ6rcnceauSikq3lYCQ=
github.com/mattn/go-colorable v0.1.2/go.mod h1:U0ppj6V5qS13XJ6of8GYAs25YV2eR4EVcfRqFIhoBtE=
@ -353,6 +402,7 @@ github.com/mattn/go-sqlite3 v1.10.0/go.mod h1:FPy6KqzDD04eiIsT53CuJW3U88zkxoIYsO
github.com/mattn/go-sqlite3 v2.0.3+incompatible h1:gXHsfypPkaMZrKbD5209QV9jbUTJKjyR5WD3HYQSd+U=
github.com/mattn/go-sqlite3 v2.0.3+incompatible/go.mod h1:FPy6KqzDD04eiIsT53CuJW3U88zkxoIYsOqkbpncsNc=
github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0=
github.com/microcosm-cc/bluemonday v1.0.1/go.mod h1:hsXNsILzKxV+sX77C5b8FSuKF00vh2OMYv+xgHpAMF4=
github.com/miekg/dns v1.0.14/go.mod h1:W1PPwlIAgtquWBMBEV9nkV9Cazfe8ScdGz/Lj7v3Nrg=
github.com/minio/sha256-simd v0.0.0-20190328051042-05b4dd3047e5/go.mod h1:2FMWW+8GMoPweT6+pI63m9YE3Lmw4J71hV56Chs1E/U=
github.com/mitchellh/cli v1.0.0/go.mod h1:hNIlj7HEI86fIcpObd7a0FcrxTWetlwJDGcceTlRvqc=
@ -370,22 +420,32 @@ github.com/morikuni/aec v0.0.0-20170113033406-39771216ff4c h1:nXxl5PrvVm2L/wCy8d
github.com/morikuni/aec v0.0.0-20170113033406-39771216ff4c/go.mod h1:BbKIizmSmc5MMPqRYbxO4ZU0S0+P200+tUnFx7PXmsc=
github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
github.com/nakagami/firebirdsql v0.0.0-20190310045651-3c02a58cfed8/go.mod h1:86wM1zFnC6/uDBfZGNwB65O+pR2OFi5q/YQaEUid1qA=
github.com/neelance/astrewrite v0.0.0-20160511093645-99348263ae86/go.mod h1:kHJEU3ofeGjhHklVoIGuVj85JJwZ6kWPaJwCIxgnFmo=
github.com/neelance/sourcemap v0.0.0-20151028013722-8c68805598ab/go.mod h1:Qr6/a/Q4r9LP1IltGz7tA7iOK1WonHEYhu1HRBA7ZiM=
github.com/nsf/jsondiff v0.0.0-20200515183724-f29ed568f4ce h1:RPclfga2SEJmgMmz2k+Mg7cowZ8yv4Trqw9UsJby758=
github.com/nsf/jsondiff v0.0.0-20200515183724-f29ed568f4ce/go.mod h1:uFMI8w+ref4v2r9jz+c9i1IfIttS/OkmLfrk1jne5hs=
github.com/nsf/termbox-go v0.0.0-20200418040025-38ba6e5628f1 h1:lh3PyZvY+B9nFliSGTn5uFuqQQJGuNrD0MLCokv09ag=
github.com/nsf/termbox-go v0.0.0-20200418040025-38ba6e5628f1/go.mod h1:IuKpRQcYE1Tfu+oAQqaLisqDeXgjyyltCfsaoYN18NQ=
github.com/nxadm/tail v1.4.4 h1:DQuhQpB1tVlglWS2hLQ5OV6B5r8aGxSrPc5Qo6uTN78=
github.com/nxadm/tail v1.4.4/go.mod h1:kenIhsEOeOJmVchQTgglprH7qJGnHDVpk1VPCcaMI8A=
github.com/oklog/ulid v1.3.1/go.mod h1:CirwcVhetQ6Lv90oh/F+FBtV6XMibvdAFo93nm5qn4U=
github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
github.com/onsi/ginkgo v1.7.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
github.com/onsi/ginkgo v1.10.3 h1:OoxbjfXVZyod1fmWYhI7SEyaD8B00ynP3T+D5GiyHOY=
github.com/onsi/ginkgo v1.10.3/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
github.com/onsi/ginkgo v1.12.1/go.mod h1:zj2OWP4+oCPe1qIXoGWkgMRwljMUYCdkwsT2108oapk=
github.com/onsi/ginkgo v1.14.0 h1:2mOpI4JVVPBN+WQRa0WKH2eXR+Ey+uK4n7Zj0aYpIQA=
github.com/onsi/ginkgo v1.14.0/go.mod h1:iSB4RoI2tjJc9BBv4NKIKWKya62Rps+oPG/Lv9klQyY=
github.com/onsi/gomega v1.4.3/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY=
github.com/onsi/gomega v1.7.1 h1:K0jcRCwNQM3vFGh1ppMtDh/+7ApJrjldlX8fA0jDTLQ=
github.com/onsi/gomega v1.7.1/go.mod h1:XdKZgCCFLUoM/7CFJVPcG8C1xQ1AJ0vpAezJrB7JYyY=
github.com/onsi/gomega v1.10.1 h1:o0+MgICZLuZ7xjH7Vx6zS/zcu93/BEp1VwkIW1mEXCE=
github.com/onsi/gomega v1.10.1/go.mod h1:iN09h71vgCQne3DLsj+A5owkum+a2tYe+TOCB1ybHNo=
github.com/opencontainers/go-digest v1.0.0-rc1 h1:WzifXhOVOEOuFYOJAW6aQqW0TooG2iki3E3Ii+WN7gQ=
github.com/opencontainers/go-digest v1.0.0-rc1/go.mod h1:cMLVZDEM3+U2I4VmLI6N8jQYUd2OVphdqWwCJHrFt2s=
github.com/opencontainers/image-spec v1.0.1 h1:JMemWkRwHx4Zj+fVxWoMCFm/8sYGGrUVojFA6h/TRcI=
github.com/opencontainers/image-spec v1.0.1/go.mod h1:BtxoFyWECRxE4U/7sNtV5W15zMzWCbyJoFRP3s7yZA0=
github.com/openzipkin/zipkin-go v0.1.1/go.mod h1:NtoC/o8u3JlF1lSlyPNswIbeQH9bJTmOf0Erfk+hxe8=
github.com/openzipkin/zipkin-go v0.1.6/go.mod h1:QgAqvLzwWbR/WpD4A3cGpPtJrZXNIiJc5AZX7/PBEpw=
github.com/pascaldekloe/goe v0.0.0-20180627143212-57f6aae5913c/go.mod h1:lzWF7FIEvWOWxwDKqyGYQf6ZUaNfKdP144TG7ZOy1lc=
github.com/pelletier/go-toml v1.2.0 h1:T5zMGML61Wp+FlcbWjRDT7yAxhJNAiPPLOFECq181zc=
@ -398,6 +458,7 @@ github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINE
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/posener/complete v1.1.1/go.mod h1:em0nMJCgc9GFtwrmVmEMR/ZL6WyhyjMBndrE9hABlRI=
github.com/prometheus/client_golang v0.8.0/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXPKyh/dDVn+NZz0KFw=
github.com/prometheus/client_golang v0.9.1/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXPKyh/dDVn+NZz0KFw=
github.com/prometheus/client_golang v0.9.3-0.20190127221311-3c4408c8b829/go.mod h1:p2iRAGwDERtqlqzRXnrOVns+ignqQo//hLXqYxZYVNs=
github.com/prometheus/client_golang v0.9.3/go.mod h1:/TN21ttK/J9q6uSwhBd54HahCDft0ttaMvbicHlPoso=
@ -405,9 +466,11 @@ github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910/go.mod h1:
github.com/prometheus/client_model v0.0.0-20190115171406-56726106282f/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo=
github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
github.com/prometheus/common v0.0.0-20180801064454-c7de2306084e/go.mod h1:daVV7qP5qjZbuso7PdcryaAu0sAZbrN9i7WWcTMWvro=
github.com/prometheus/common v0.0.0-20181113130724-41aa239b4cce/go.mod h1:daVV7qP5qjZbuso7PdcryaAu0sAZbrN9i7WWcTMWvro=
github.com/prometheus/common v0.2.0/go.mod h1:TNfzLD0ON7rHzMJeJkieUDPYmFC7Snx/y86RQel1bk4=
github.com/prometheus/common v0.4.0/go.mod h1:TNfzLD0ON7rHzMJeJkieUDPYmFC7Snx/y86RQel1bk4=
github.com/prometheus/procfs v0.0.0-20180725123919-05ee40e3a273/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
github.com/prometheus/procfs v0.0.0-20181005140218-185b4288413d/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
github.com/prometheus/procfs v0.0.0-20190117184657-bf6a532e95b1/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
github.com/prometheus/procfs v0.0.0-20190507164030-5867b95ac084/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsTZCD3I8kEA=
@ -418,15 +481,39 @@ github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFR
github.com/rs/xid v1.2.1/go.mod h1:+uKXf+4Djp6Md1KODXJxgGQPKngRmWyn10oCKFzNHOQ=
github.com/rs/zerolog v1.13.0/go.mod h1:YbFCdg8HfsridGWAh22vktObvhZbQsZXe4/zB0OKkWU=
github.com/rs/zerolog v1.15.0/go.mod h1:xYTKnLHcpfU2225ny5qZjxnj9NvkumZYjJHlAThCjNc=
github.com/russross/blackfriday v1.5.2/go.mod h1:JO/DiYxRf+HjHt06OyowR9PTA263kcR/rfWxYHBV53g=
github.com/russross/blackfriday/v2 v2.0.1/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
github.com/ryanuber/columnize v0.0.0-20160712163229-9b3edd62028f/go.mod h1:sm1tb6uqfes/u+d4ooFouqFdy9/2g9QGwK3SQygK0Ts=
github.com/satori/go.uuid v1.2.0/go.mod h1:dA0hQrYB0VpLJoorglMZABFdXlWrHn1NEOzdhQKdks0=
github.com/sean-/seed v0.0.0-20170313163322-e2103e2c3529/go.mod h1:DxrIzT+xaE7yg65j358z/aeFdxmN0P9QXhEzd20vsDc=
github.com/sergi/go-diff v1.0.0/go.mod h1:0CfEIISq7TuYL3j771MWULgwwjU+GofnZX9QAmXWZgo=
github.com/shopspring/decimal v0.0.0-20180709203117-cd690d0c9e24/go.mod h1:M+9NzErvs504Cn4c5DxATwIqPbtswREoFCre64PpcG4=
github.com/shopspring/decimal v0.0.0-20200227202807-02e2044944cc/go.mod h1:DKyhrW/HYNuLGql+MJL6WCR6knT2jwCFRcu2hWCYk4o=
github.com/shopspring/decimal v1.2.0 h1:abSATXmQEYyShuxI4/vyW3tV1MrKAJzCZ/0zLUXYbsQ=
github.com/shopspring/decimal v1.2.0/go.mod h1:DKyhrW/HYNuLGql+MJL6WCR6knT2jwCFRcu2hWCYk4o=
github.com/shurcooL/component v0.0.0-20170202220835-f88ec8f54cc4/go.mod h1:XhFIlyj5a1fBNx5aJTbKoIq0mNaPvOagO+HjB3EtxrY=
github.com/shurcooL/events v0.0.0-20181021180414-410e4ca65f48/go.mod h1:5u70Mqkb5O5cxEA8nxTsgrgLehJeAw6Oc4Ab1c/P1HM=
github.com/shurcooL/github_flavored_markdown v0.0.0-20181002035957-2122de532470/go.mod h1:2dOwnU2uBioM+SGy2aZoq1f/Sd1l9OkAeAUvjSyvgU0=
github.com/shurcooL/go v0.0.0-20180423040247-9e1955d9fb6e/go.mod h1:TDJrrUr11Vxrven61rcy3hJMUqaf/CLWYhHNPmT14Lk=
github.com/shurcooL/go-goon v0.0.0-20170922171312-37c2f522c041/go.mod h1:N5mDOmsrJOB+vfqUK+7DmDyjhSLIIBnXo9lvZJj3MWQ=
github.com/shurcooL/gofontwoff v0.0.0-20180329035133-29b52fc0a18d/go.mod h1:05UtEgK5zq39gLST6uB0cf3NEHjETfB4Fgr3Gx5R9Vw=
github.com/shurcooL/gopherjslib v0.0.0-20160914041154-feb6d3990c2c/go.mod h1:8d3azKNyqcHP1GaQE/c6dDgjkgSx2BZ4IoEi4F1reUI=
github.com/shurcooL/highlight_diff v0.0.0-20170515013008-09bb4053de1b/go.mod h1:ZpfEhSmds4ytuByIcDnOLkTHGUI6KNqRNPDLHDk+mUU=
github.com/shurcooL/highlight_go v0.0.0-20181028180052-98c3abbbae20/go.mod h1:UDKB5a1T23gOMUJrI+uSuH0VRDStOiUVSjBTRDVBVag=
github.com/shurcooL/home v0.0.0-20181020052607-80b7ffcb30f9/go.mod h1:+rgNQw2P9ARFAs37qieuu7ohDNQ3gds9msbT2yn85sg=
github.com/shurcooL/htmlg v0.0.0-20170918183704-d01228ac9e50/go.mod h1:zPn1wHpTIePGnXSHpsVPWEktKXHr6+SS6x/IKRb7cpw=
github.com/shurcooL/httperror v0.0.0-20170206035902-86b7830d14cc/go.mod h1:aYMfkZ6DWSJPJ6c4Wwz3QtW22G7mf/PEgaB9k/ik5+Y=
github.com/shurcooL/httpfs v0.0.0-20171119174359-809beceb2371/go.mod h1:ZY1cvUeJuFPAdZ/B6v7RHavJWZn2YPVFQ1OSXhCGOkg=
github.com/shurcooL/httpgzip v0.0.0-20180522190206-b1c53ac65af9/go.mod h1:919LwcH0M7/W4fcZ0/jy0qGght1GIhqyS/EgWGH2j5Q=
github.com/shurcooL/issues v0.0.0-20181008053335-6292fdc1e191/go.mod h1:e2qWDig5bLteJ4fwvDAc2NHzqFEthkqn7aOZAOpj+PQ=
github.com/shurcooL/issuesapp v0.0.0-20180602232740-048589ce2241/go.mod h1:NPpHK2TI7iSaM0buivtFUc9offApnI0Alt/K8hcHy0I=
github.com/shurcooL/notifications v0.0.0-20181007000457-627ab5aea122/go.mod h1:b5uSkrEVM1jQUspwbixRBhaIjIzL2xazXp6kntxYle0=
github.com/shurcooL/octicon v0.0.0-20181028054416-fa4f57f9efb2/go.mod h1:eWdoE5JD4R5UVWDucdOPg1g2fqQRq78IQa9zlOV1vpQ=
github.com/shurcooL/reactions v0.0.0-20181006231557-f2e0b4ca5b82/go.mod h1:TCR1lToEk4d2s07G3XGfz2QrgHXg4RJBvjrOozvoWfk=
github.com/shurcooL/sanitized_anchor_name v0.0.0-20170918181015-86672fcb3f95/go.mod h1:1NzhyTcUVG4SuEtjjoZeVRXNmyL/1OwPU0+IJeTBvfc=
github.com/shurcooL/sanitized_anchor_name v1.0.0/go.mod h1:1NzhyTcUVG4SuEtjjoZeVRXNmyL/1OwPU0+IJeTBvfc=
github.com/shurcooL/users v0.0.0-20180125191416-49c67e49c537/go.mod h1:QJTqeLYEDaXHZDBsXlPCDqdhQuJkuw4NOtaxYe3xii4=
github.com/shurcooL/webdavfs v0.0.0-20170829043945-18c3829fa133/go.mod h1:hKmq5kWdCj2z2KEozexVbfEZIWiTjhE0+UjmZgPqehw=
github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
github.com/sirupsen/logrus v1.4.1/go.mod h1:ni0Sbl8bgC9z8RoU9G6nDWqqs/fq4eDPysMBDgk/93Q=
github.com/sirupsen/logrus v1.4.2 h1:SPIRibHv4MatM3XXNO2BJeFLZwZ2LvZgfQ5+UNI2im4=
@ -436,6 +523,8 @@ github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d/go.mod h1
github.com/smartystreets/goconvey v1.6.4 h1:fv0U8FUIMPNf1L9lnHLvLhgicrIVChEkdzIKYqbNC9s=
github.com/smartystreets/goconvey v1.6.4/go.mod h1:syvi0/a8iFYH4r/RixwvyeAJjdLS9QV7WQ/tjFTllLA=
github.com/soheilhy/cmux v0.1.4/go.mod h1:IM3LyeVVIOuxMH7sFAkER9+bJ4dT7Ms6E4xg4kGIyLM=
github.com/sourcegraph/annotate v0.0.0-20160123013949-f4cad6c6324d/go.mod h1:UdhH50NIW0fCiwBSr0co2m7BnFLdv4fQTgdqdJTHFeE=
github.com/sourcegraph/syntaxhighlight v0.0.0-20170531221838-bd320f5d308e/go.mod h1:HuIsMU8RRBOtsCgI77wP899iHVBQpCmg4ErYMZB+2IA=
github.com/spacemonkeygo/monkit/v3 v3.0.0-20191108235033-eacca33b3037/go.mod h1:JcK1pCbReQsOsMKF/POFSZCq7drXFybgGmbc27tuwes=
github.com/spacemonkeygo/monkit/v3 v3.0.4/go.mod h1:JcK1pCbReQsOsMKF/POFSZCq7drXFybgGmbc27tuwes=
github.com/spacemonkeygo/monkit/v3 v3.0.5/go.mod h1:JcK1pCbReQsOsMKF/POFSZCq7drXFybgGmbc27tuwes=
@ -475,9 +564,12 @@ github.com/stripe/stripe-go v70.15.0+incompatible h1:hNML7M1zx8RgtepEMlxyu/FpVPr
github.com/stripe/stripe-go v70.15.0+incompatible/go.mod h1:A1dQZmO/QypXmsL0T8axYZkSN/uA/T/A64pfKdBAMiY=
github.com/subosito/gotenv v1.2.0 h1:Slr1R9HxAlEKefgq5jn9U+DnETlIUa6HfgEzj0g5d7s=
github.com/subosito/gotenv v1.2.0/go.mod h1:N0PQaV/YGNqwC0u51sEeR/aUtSLEXKX9iv69rRypqCw=
github.com/tarm/serial v0.0.0-20180830185346-98f6abe2eb07/go.mod h1:kDXzergiv9cbyO7IOYJZWg1U88JhDg3PB6klq9Hg2pA=
github.com/tidwall/pretty v0.0.0-20180105212114-65a9db5fad51/go.mod h1:XNkn88O1ChpSDQmQeStsy+sBenx6DDtFZJxhVysOjyk=
github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U=
github.com/ugorji/go v1.1.4/go.mod h1:uQMGLiO92mf5W77hV/PUCpI3pbzQx3CRekS0kk+RGrc=
github.com/viant/assertly v0.4.8/go.mod h1:aGifi++jvCrUaklKEKT0BU95igDNaqkvz+49uaYMPRU=
github.com/viant/toolbox v0.24.0/go.mod h1:OxMCG57V0PXuIP2HNQrtJf2CjqdmbrOx5EkMILuUhzM=
github.com/vivint/infectious v0.0.0-20200605153912-25a574ae18a3 h1:zMsHhfK9+Wdl1F7sIKLyx3wrOFofpb3rWFbA4HgcK5k=
github.com/vivint/infectious v0.0.0-20200605153912-25a574ae18a3/go.mod h1:R0Gbuw7ElaGSLOZUSwBm/GgVwMd30jWxBDdAyMOeTuc=
github.com/xanzy/go-gitlab v0.15.0/go.mod h1:8zdQa/ri1dfn8eS3Ir1SyfvOKlw7WBJ8DVThkpGiXrs=
@ -508,6 +600,7 @@ go.etcd.io/bbolt v1.3.2/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU=
go.etcd.io/bbolt v1.3.5 h1:XAzx9gjCb0Rxj7EoqcClPD1d5ZBxZJk0jbuoPHenBt0=
go.etcd.io/bbolt v1.3.5/go.mod h1:G5EMThwa9y8QZGBClrRx5EY+Yw9kAhnjy3bSjsnlVTQ=
go.mongodb.org/mongo-driver v1.1.0/go.mod h1:u7ryQJ+DOzQmeO7zB6MHyr8jkEQvC8vH7qLUO4lqsUM=
go.opencensus.io v0.18.0/go.mod h1:vKdFvxhtzZ9onBp9VKHK8z/sRpBMnKAsufL7wlDrCOA=
go.opencensus.io v0.20.1/go.mod h1:6WKK9ahsWS3RSO+PY9ZHZUfv2irvY6gN279GOPZjmmk=
go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU=
go.opencensus.io v0.22.0/go.mod h1:+kGneAE2xo2IficOXnaByMWTGM9T73dGwxeWcUqIpI8=
@ -531,10 +624,14 @@ go.uber.org/zap v1.10.0/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q=
go.uber.org/zap v1.14.1/go.mod h1:Mb2vm2krFEG5DV0W9qcHBYFtp/Wku1cvYaqPsS/WYfc=
go.uber.org/zap v1.16.0 h1:uFRZXykJGK9lLY4HtgSw44DnIcAM+kRBP7x5m+NpAOM=
go.uber.org/zap v1.16.0/go.mod h1:MA8QOfq0BHJwdXa996Y4dYkAqRKB8/1K1QMMZVaNZjQ=
go4.org v0.0.0-20180809161055-417644f6feb5/go.mod h1:MkTOUMDaeVYJUOUsaDXIhWPZYa1yOyC1qaOBpL57BhE=
golang.org/x/build v0.0.0-20190111050920-041ab4dc3f9d/go.mod h1:OWs+y06UdEOHN4y+MfF/py+xQ/tYqIWW03b70/CG9Rw=
golang.org/x/crypto v0.0.0-20170930174604-9419663f5a44/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
golang.org/x/crypto v0.0.0-20181029021203-45a5f77698d3/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
golang.org/x/crypto v0.0.0-20181030102418-4d3f4d9ffa16/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20190313024323-a1f597ede03a/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20190325154230-a5d413f7728c/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20190411191339-88737f569e3a/go.mod h1:WFFai1msRO1wXaEeE5yQxYXgSfI8pQAWXbQop6sCtWE=
golang.org/x/crypto v0.0.0-20190426145343-a29dc8fdc734/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
@ -545,6 +642,7 @@ golang.org/x/crypto v0.0.0-20190911031432-227b76d455e7/go.mod h1:yigFU9vqHzYiE8U
golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20200115085410-6d4e4cb37c7d/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20200220183623-bac4c82f6975/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20200221231518-2aa609cf4a9d/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20200323165209-0ec3e9974c59/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20200604202706-70a84ac30bf9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
@ -559,6 +657,7 @@ golang.org/x/exp v0.0.0-20191227195350-da58074b4299 h1:zQpM52jfKHG6II1ISZY1Zcpyg
golang.org/x/exp v0.0.0-20191227195350-da58074b4299/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4=
golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js=
golang.org/x/image v0.0.0-20190802002840-cff245a6509b/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0=
golang.org/x/lint v0.0.0-20180702182130-06c8688daad7/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU=
golang.org/x/lint v0.0.0-20190301231843-5614ed5bae6f/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
@ -577,6 +676,8 @@ golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73r
golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20181023162649-9b4f9f5ad519/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20181029044818-c44066c5c816/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20181106065722-10aee1819953/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20181108082009-03003ca0c849/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20181201002055-351d144fa1fc/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
@ -585,6 +686,7 @@ golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73r
golang.org/x/net v0.0.0-20190125091013-d26f9f9a57f3/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190313220215-9f648a60d977/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190424112056-4829fb13d2c6/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190501004415-9ce7a6920f09/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
@ -595,15 +697,19 @@ golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLL
golang.org/x/net v0.0.0-20190813141303-74dc4d7220e7/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20190923162816-aa69164e4478/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200114155413-6afb5195e5aa/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200226121028-0de0cce0169b h1:0mm1VjtFUOIlE1SbDlwjYaDxZVDP2S5ou6y0gSgXHu8=
golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200520004742-59133d7f0dd7/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
golang.org/x/net v0.0.0-20200707034311-ab3426394381 h1:VXak5I6aEWmAXeQjA+QSZzlgNrpq9mjcfDemuexIKsU=
golang.org/x/net v0.0.0-20200707034311-ab3426394381/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20181017192945-9dcd33a902f4/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20181106182150-f42d05182288/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20181203162652-d668ce993890/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20190402181905-9f3314589c9a/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d h1:TzXSXBo42m9gQenoE3b9BGiEpg5IG2JkU5FkPIawgtw=
golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/perf v0.0.0-20180704124530-6e6d33e29852/go.mod h1:JLpeXjPJfIyPr5TlbXLkXWLhP8nz10XfvxElABhCtcw=
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
@ -618,6 +724,7 @@ golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5h
golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20181026203630-95b1ffbd15a5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20181029174526-d69651ed3497/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20181107165924-66b7b1311ac8/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20181122145206-62eef0e2fa9b/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
@ -626,6 +733,7 @@ golang.org/x/sys v0.0.0-20190204203706-41f3e6584952/go.mod h1:STP8DvDyc/dI5b8T5h
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190312061237-fead79001313/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190316082340-a2f829d7f35f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190403152447-81d4e9dc473e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190422165155-953cdadca894/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
@ -636,7 +744,10 @@ golang.org/x/sys v0.0.0-20190606165138-5da285871e9c/go.mod h1:h1NjWce9XRLGQEsW7w
golang.org/x/sys v0.0.0-20190624142023-c5567b49c5d0/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190813064441-fde4db37ae7a/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190826190057-c7b8b68b1456/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190904154756-749cb33beabd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191005200804-aed5e4c7ecf9/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191026070338-33540a1f6037/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191120155948-bd437916bb0e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191204072324-ce4227a45e2e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191210023423-ac6580df4449/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200107144601-ef85f5a75ddf/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
@ -645,14 +756,17 @@ golang.org/x/sys v0.0.0-20200116001909-b77594299b42/go.mod h1:h1NjWce9XRLGQEsW7w
golang.org/x/sys v0.0.0-20200202164722-d101bd2416d5/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200519105757-fe76b779f299/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200610111108-226ff32320da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200929083018-4d22bbb62b3c h1:/h0vtH0PyU0xAoZJVcRw1k0Ng+U0JAy3QDiFmppIlIE=
golang.org/x/sys v0.0.0-20200929083018-4d22bbb62b3c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
golang.org/x/text v0.3.3 h1:cokOdA+Jmi5PJGXLlLllQSgYigAEfHXJAERHVMaCc2k=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/time v0.0.0-20180412165947-fbb02b2291d2/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20200630173020-3af7569d3a1e h1:EHBhcS0mlXEAVwNyO2dLfjToGsyY4j24pTs2ScHnX7s=
@ -660,6 +774,7 @@ golang.org/x/time v0.0.0-20200630173020-3af7569d3a1e/go.mod h1:tRJNPiyCQ0inRvYxb
golang.org/x/tools v0.0.0-20180221164845-07fd8470d635/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20180828015842-6cd1fcedba52/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20181030000716-a0a13e073c7b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20181030221726-6c7e314b6563/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY=
@ -691,6 +806,9 @@ golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8T
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543 h1:E7g+9GITq07hpfrRu66IVDexMakfv52eLZ2CXBWiKr4=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
google.golang.org/api v0.0.0-20180910000450-7ca32eb868bf/go.mod h1:4mhQ8q/RsB7i+udVvVy5NUi08OU8ZlA0gRVgrF7VFY0=
google.golang.org/api v0.0.0-20181030000543-1d582fd0359e/go.mod h1:4mhQ8q/RsB7i+udVvVy5NUi08OU8ZlA0gRVgrF7VFY0=
google.golang.org/api v0.1.0/go.mod h1:UGEZY7KEX120AnNLIHFMKIo4obdJhkp2tPbaPlQx13Y=
google.golang.org/api v0.3.1/go.mod h1:6wY9I6uQWHQ8EM57III9mq/AjF+i8G65rmVagqKMtkk=
google.golang.org/api v0.3.2/go.mod h1:6wY9I6uQWHQ8EM57III9mq/AjF+i8G65rmVagqKMtkk=
google.golang.org/api v0.4.0/go.mod h1:8k5glujaEP+g9n7WNsDg8QP6cUVNI86fCNMcbazEtwE=
@ -702,6 +820,7 @@ google.golang.org/api v0.15.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsb
google.golang.org/api v0.20.0 h1:jz2KixHX7EcCPiQrySzPdnYT7DbINAypCqKZ1Z7GM40=
google.golang.org/api v0.20.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE=
google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
google.golang.org/appengine v1.2.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
google.golang.org/appengine v1.3.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
google.golang.org/appengine v1.5.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
@ -709,6 +828,10 @@ google.golang.org/appengine v1.6.1/go.mod h1:i06prIuMbXzDqacNJfV5OdTW448YApPu5ww
google.golang.org/appengine v1.6.5 h1:tycE03LOZYQNhDpS27tcQdAzLCVMaj7QT2SXxebnpCM=
google.golang.org/appengine v1.6.5/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc=
google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
google.golang.org/genproto v0.0.0-20180831171423-11092d34479b/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
google.golang.org/genproto v0.0.0-20181029155118-b69ba1387ce2/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
google.golang.org/genproto v0.0.0-20181202183823-bd91e49a0898/go.mod h1:7Ep/1NZk928CDR8SjdVbjWNpdIf6nzjE3BTgJDr2Atg=
google.golang.org/genproto v0.0.0-20190306203927-b5d61aea6440/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
google.golang.org/genproto v0.0.0-20190307195333-5fe7a883aa19/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
google.golang.org/genproto v0.0.0-20190404172233-64821d5d2107/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
google.golang.org/genproto v0.0.0-20190418145605-e7d98fc518a7/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
@ -720,6 +843,8 @@ google.golang.org/genproto v0.0.0-20190911173649-1774047e7e51/go.mod h1:IbNlFCBr
google.golang.org/genproto v0.0.0-20191108220845-16a3f7862a1a/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20200115191322-ca5a22157cba h1:pRj9OXZbwNtbtZtOB4dLwfK4u+EVRMvP+e9zKkg2grM=
google.golang.org/genproto v0.0.0-20200115191322-ca5a22157cba/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/grpc v1.14.0/go.mod h1:yo6s7OP7yaDglbqo1J04qKzAhqBH6lvTonzMVmEdcZw=
google.golang.org/grpc v1.16.0/go.mod h1:0JHn/cJsOMiMfNA9+DeHDlAU7KAAB5GDlYFpa9MZMio=
google.golang.org/grpc v1.17.0/go.mod h1:6QZJwpn2B+Zp71q/5VxRsJ6NXXVCE5NRUHRo+f3cWCs=
google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
google.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiqXj38=
@ -730,6 +855,13 @@ google.golang.org/grpc v1.26.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8
google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
google.golang.org/grpc v1.27.1 h1:zvIju4sqAGvwKspUQOhwnpcqSbzi7/H6QomNNjTL4sk=
google.golang.org/grpc v1.27.1/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8=
google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0=
google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM=
google.golang.org/protobuf v1.20.1-0.20200309200217-e05f789c0967/go.mod h1:A+miEFZTKqfCUM6K7xSMQL9OKL/b6hQv+e19PK+JZNE=
google.golang.org/protobuf v1.21.0/go.mod h1:47Nbq4nVaFHyn7ilMalzfO3qCViNmqZ2kzikPIcrTAo=
google.golang.org/protobuf v1.23.0 h1:4MY060fB1DLGMB/7MBTLnwQUY6+F09GEiz6SsrNqyzM=
google.golang.org/protobuf v1.23.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 h1:qIbj1fsPNlZgppZ+VLlY7N33q108Sa+fhmuc+sWQYwY=
@ -749,10 +881,13 @@ gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.4 h1:/eiJrUcujPVeJ3xlSWaiNi3uSVmDGBK1pDHUHAnao1I=
gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.3.0 h1:clyUAQHOM3G0M3f5vQj7LuJrETvjVot3Z5el9nffUtU=
gopkg.in/yaml.v2 v2.3.0/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c h1:dUUwHk2QECo/6vqA44rthZ8ie2QXMNeKRTHCNY2nXvo=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gotest.tools v2.2.0+incompatible h1:VsBPFP1AI068pPrMxtb/S8Zkgf9xEmTLJjfM+P5UIEo=
gotest.tools v2.2.0+incompatible/go.mod h1:DsYFclhRJ6vuDpmuTbkuFWG+y2sxOXAzmJt81HFBacw=
grpc.go4.org v0.0.0-20170609214715-11d0a25b4919/go.mod h1:77eQGdRu53HpSqPFJFmuJdjuHRquDANNeA4x7B8WQ9o=
honnef.co/go/tools v0.0.0-20180728063816-88497007e858/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.0-20190106161140-3f1c8253044a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
@ -761,18 +896,23 @@ honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWh
honnef.co/go/tools v0.0.1-2019.2.3 h1:3JgtbtFHMiCmsznwGVTUWbgGov+pVqnlf1dEJTNAXeM=
honnef.co/go/tools v0.0.1-2019.2.3/go.mod h1:a3bituU0lyd329TUQxRnasdCoJDkEUEAqEt0JzvZhAg=
rsc.io/binaryregexp v0.2.0/go.mod h1:qTv7/COck+e2FymRvadv62gMdZztPaShugOCi3I+8D8=
rsc.io/quote/v3 v3.1.0/go.mod h1:yEA65RcK8LyAZtP9Kv3t0HmxON59tX3rD+tICJqUlj0=
rsc.io/sampler v1.3.0/go.mod h1:T1hPZKmBbMNahiBKFy5HrXp6adAjACjK9JXDnKaTXpA=
sourcegraph.com/sourcegraph/go-diff v0.5.0/go.mod h1:kuch7UrkMzY0X+p9CRK03kfuPQ2zzQcaEFbx8wA8rck=
sourcegraph.com/sqs/pbtypes v0.0.0-20180604144634-d3ebe8f20ae4/go.mod h1:ketZ/q3QxT9HOBeFhu6RdvsftgpsbFHBF5Cas6cDKZ0=
storj.io/common v0.0.0-20200424175742-65ac59022f4f/go.mod h1:pZyXiIE7bGETIRXtfs0nICqMwp7PM8HqnDuyUeldNA0=
storj.io/common v0.0.0-20201026135900-1aaeec90670b/go.mod h1:GqdmNf3fLm2UZX/7Zr0BLFCJ4gFjgm6eHrk/fnmr5jQ=
storj.io/common v0.0.0-20201027143432-3718579e12bf/go.mod h1:9iobNl9eI6C2M23FS/b37yFYOdHpoeJ8BFFcxsmv538=
storj.io/common v0.0.0-20201030120157-90ae6720d87e h1:6baDicBbR0/2XgcQ068KN+B4dF6akkdh2vemmXka1ns=
storj.io/common v0.0.0-20201030120157-90ae6720d87e/go.mod h1:9iobNl9eI6C2M23FS/b37yFYOdHpoeJ8BFFcxsmv538=
storj.io/common v0.0.0-20201106104920-372a344bdd45 h1:pv552R7MiRA8VLQC4qXczLjbl2Qb/MNyus2E9NBSXgI=
storj.io/common v0.0.0-20201106104920-372a344bdd45/go.mod h1:ZkQZup2jpFZvvTgz+yPc7K4Vr4bBHM8AA66P57MZkjk=
storj.io/drpc v0.0.11/go.mod h1:TiFc2obNjL9/3isMW1Rpxjy8V9uE0B2HMeMFGiiI7Iw=
storj.io/drpc v0.0.11/go.mod h1:TiFc2obNjL9/3isMW1Rpxjy8V9uE0B2HMeMFGiiI7Iw=
storj.io/drpc v0.0.14 h1:GCBdymTt1BRw4oHmmUZZlxYXLVRxxYj6x3Ivide2J+I=
storj.io/drpc v0.0.14/go.mod h1:82nfl+6YwRwF6UG31cEWWUqv/FaKvP5SGqUvoqTxCMA=
storj.io/drpc v0.0.16 h1:9sxypc5lKi/0D69cR21BR0S21+IvXfON8L5nXMVNTwQ=
storj.io/drpc v0.0.16/go.mod h1:zdmQ93nx4Z35u11pQ+GAnBy4DGOK3HJCSOfeh2RryTo=
storj.io/monkit-jaeger v0.0.0-20200518165323-80778fc3f91b h1:Bbg9JCtY6l3HrDxs3BXzT2UYnYCBLqNi6i84Y8QIPUs=
storj.io/monkit-jaeger v0.0.0-20200518165323-80778fc3f91b/go.mod h1:gj4vuCeyCRjRmH8LIrgoyU9Dc9uR6H+/GcDUXmTbf80=
storj.io/private v0.0.0-20201026143115-bc926bfa3bca h1:ekR7vtUYC5+cDyim0ZJaSZeXidyzQqDYsnFPYXgTozc=
storj.io/private v0.0.0-20201026143115-bc926bfa3bca/go.mod h1:EaLnIyNyqWQUJB+7+KWVez0In9czl0nHHlm2WobebuA=
storj.io/uplink v1.3.2-0.20201028181609-f6efc8fcf771 h1:jPbw74xt8bvv8nOfBaM4g9Ts4moX8mqfD4N/B8vEJrA=
storj.io/uplink v1.3.2-0.20201028181609-f6efc8fcf771/go.mod h1:5do8jvbs4ao4tLdIZKzNFJPVKOH1oDfvVf8OIsR5Z9E=
storj.io/uplink v1.3.2-0.20201106105834-ec8e5cc29f7e h1:a58hzcYciTvBtWST+Byoj76NWuYdnPcz2GK8ynyEyfA=
storj.io/uplink v1.3.2-0.20201106105834-ec8e5cc29f7e/go.mod h1:mrdt4I4EhPRC7cnvCD5490IBm423pgKrVoUiC9a5Srg=

View File

@ -0,0 +1,40 @@
// Copyright (C) 2020 Storj Labs, Inc.
// See LICENSE for copying information.
package console
import (
"context"
"github.com/zeebo/errs"
"storj.io/common/uuid"
)
// Members exposes needed by MND MembersDB functionality.
//
// architecture: Database
type Members interface {
// Invite will create empty row in membersDB.
Invite(ctx context.Context, member Member) error
// Update updates all updatable fields of member.
Update(ctx context.Context, member Member) error
// Remove deletes member from membersDB.
Remove(ctx context.Context, id uuid.UUID) error
// GetByEmail will return member with specified email.
GetByEmail(ctx context.Context, email string) (Member, error)
// GetByID will return member with specified id.
GetByID(ctx context.Context, id uuid.UUID) (Member, error)
}
// ErrNoMember is a special error type that indicates about absence of member in MembersDB.
var ErrNoMember = errs.Class("no such member")
// Member represents some person that is invited to the MND by node owner.
// Member will have configurable access privileges that will define which functions and which nodes are available for him.
type Member struct {
ID uuid.UUID
Email string
Name string
PasswordHash []byte
}

View File

@ -0,0 +1,64 @@
// Copyright (C) 2020 Storj Labs, Inc.
// See LICENSE for copying information.
package console_test
import (
"testing"
"github.com/stretchr/testify/require"
"github.com/zeebo/assert"
"storj.io/common/testcontext"
"storj.io/common/uuid"
"storj.io/storj/multinode"
"storj.io/storj/multinode/console"
"storj.io/storj/multinode/multinodedb/multinodedbtest"
)
func TestMembersDB(t *testing.T) {
multinodedbtest.Run(t, func(ctx *testcontext.Context, t *testing.T, db multinode.DB) {
members := db.Members()
memberID, err := uuid.New()
require.NoError(t, err)
memberBob := console.Member{
ID: memberID,
Email: "mail@example.com",
Name: "Bob",
PasswordHash: []byte{0},
}
err = members.Invite(ctx, memberBob)
assert.NoError(t, err)
memberToCheck, err := members.GetByEmail(ctx, memberBob.Email)
assert.NoError(t, err)
assert.Equal(t, memberToCheck.Email, memberBob.Email)
assert.Equal(t, memberToCheck.Name, memberBob.Name)
assert.Equal(t, memberToCheck.Email, memberBob.Email)
memberBob.Name = "Alice"
err = members.Update(ctx, memberBob)
assert.NoError(t, err)
memberAlice, err := members.GetByID(ctx, memberToCheck.ID)
assert.NoError(t, err)
assert.Equal(t, memberToCheck.Email, memberAlice.Email)
assert.Equal(t, memberToCheck.Name, memberAlice.Name)
assert.Equal(t, memberToCheck.Email, memberAlice.Email)
assert.Equal(t, memberToCheck.ID, memberAlice.ID)
err = members.Remove(ctx, memberAlice.ID)
assert.NoError(t, err)
_, err = members.GetByID(ctx, memberToCheck.ID)
assert.Error(t, err)
assert.Equal(t, true, console.ErrNoMember.Has(err))
_, err = members.GetByEmail(ctx, memberToCheck.Email)
assert.Error(t, err)
assert.Equal(t, true, console.ErrNoMember.Has(err))
})
}

View File

@ -1,16 +1,18 @@
// Copyright (C) 2020 Storj Labs, Inc.
// See LICENSE for copying information.
package mutlinodedb
package multinodedb
import (
"context"
"github.com/spacemonkeygo/monkit/v3"
"github.com/zeebo/errs"
"go.uber.org/zap"
"storj.io/storj/multinode"
"storj.io/storj/multinode/console"
"storj.io/storj/multinode/mutlinodedb/dbx"
"storj.io/storj/multinode/multinodedb/dbx"
"storj.io/storj/private/dbutil"
"storj.io/storj/private/dbutil/pgutil"
)
@ -39,8 +41,8 @@ type multinodeDB struct {
source string
}
// New creates instance of database supports postgres.
func New(log *zap.Logger, databaseURL string) (multinode.DB, error) {
// Open creates instance of database supports postgres.
func Open(ctx context.Context, log *zap.Logger, databaseURL string) (multinode.DB, error) {
driver, source, implementation, err := dbutil.SplitConnStr(databaseURL)
if err != nil {
return nil, err
@ -52,17 +54,17 @@ func New(log *zap.Logger, databaseURL string) (multinode.DB, error) {
source = pgutil.CheckApplicationName(source)
// dbxDB, err := dbx.Open(driver, source)
// if err != nil {
// return nil, Error.New("failed opening database via DBX at %q: %v",
// source, err)
// }
// log.Debug("Connected to:", zap.String("db source", source))
dbxDB, err := dbx.Open(driver, source)
if err != nil {
return nil, Error.New("failed opening database via DBX at %q: %v",
source, err)
}
log.Debug("Connected to:", zap.String("db source", source))
// dbutil.Configure(dbxDB.DB, "multinodedb", mon)
dbutil.Configure(ctx, dbxDB.DB, "multinodedb", mon)
core := &multinodeDB{
// DB: dbxDB,
DB: dbxDB,
log: log,
driver: driver,
@ -77,6 +79,18 @@ func New(log *zap.Logger, databaseURL string) (multinode.DB, error) {
func (db *multinodeDB) Nodes() console.Nodes {
return &nodes{
methods: db,
db: db,
}
}
// Members returns members database.
func (db *multinodeDB) Members() console.Members {
return &members{
methods: db,
}
}
// CreateSchema creates schema.
func (db *multinodeDB) CreateSchema(ctx context.Context) error {
_, err := db.ExecContext(ctx, db.DB.Schema())
return err
}

View File

@ -0,0 +1,24 @@
// Copyright (C) 2020 Storj Labs, Inc.
// See LICENSE for copying information.
package dbx
import (
"github.com/spacemonkeygo/monkit/v3"
"github.com/zeebo/errs"
)
//go:generate sh gen.sh
var mon = monkit.Package()
func init() {
// catch dbx errors
class := errs.Class("multinodedb dbx error")
WrapErr = func(e *Error) error {
if e.Code == ErrorCode_NoRows {
return e.Err
}
return class.Wrap(e)
}
}

View File

@ -0,0 +1,45 @@
// dbx.v1 golang multinodedb.dbx .
model node (
key id
field id blob
field name text ( updatable )
field tag text ( updatable )
field public_address text
field api_secret blob
field logo blob ( updatable )
)
create node ( )
delete node ( where node.id = ? )
read one (
select node
where node.id = ?
)
model member (
key id
field id blob
field email text ( updatable )
field name text ( updatable )
field password_hash blob ( updatable )
field created_at timestamp ( autoinsert )
)
create member ( )
delete member ( where member.id = ? )
update member ( where member.id = ? )
read one (
select member
where member.email = ?
)
read one (
select member
where member.id = ?
)

View File

@ -269,7 +269,15 @@ func newpgx(db *DB) *pgxDB {
}
func (obj *pgxDB) Schema() string {
return `CREATE TABLE nodes (
return `CREATE TABLE members (
id bytea NOT NULL,
email text NOT NULL,
name text NOT NULL,
password_hash bytea NOT NULL,
created_at timestamp with time zone NOT NULL,
PRIMARY KEY ( id )
);
CREATE TABLE nodes (
id bytea NOT NULL,
name text NOT NULL,
tag text NOT NULL,
@ -340,6 +348,117 @@ nextval:
fmt.Fprint(f, "]")
}
type Member struct {
Id []byte
Email string
Name string
PasswordHash []byte
CreatedAt time.Time
}
func (Member) _Table() string { return "members" }
type Member_Update_Fields struct {
Email Member_Email_Field
Name Member_Name_Field
PasswordHash Member_PasswordHash_Field
}
type Member_Id_Field struct {
_set bool
_null bool
_value []byte
}
func Member_Id(v []byte) Member_Id_Field {
return Member_Id_Field{_set: true, _value: v}
}
func (f Member_Id_Field) value() interface{} {
if !f._set || f._null {
return nil
}
return f._value
}
func (Member_Id_Field) _Column() string { return "id" }
type Member_Email_Field struct {
_set bool
_null bool
_value string
}
func Member_Email(v string) Member_Email_Field {
return Member_Email_Field{_set: true, _value: v}
}
func (f Member_Email_Field) value() interface{} {
if !f._set || f._null {
return nil
}
return f._value
}
func (Member_Email_Field) _Column() string { return "email" }
type Member_Name_Field struct {
_set bool
_null bool
_value string
}
func Member_Name(v string) Member_Name_Field {
return Member_Name_Field{_set: true, _value: v}
}
func (f Member_Name_Field) value() interface{} {
if !f._set || f._null {
return nil
}
return f._value
}
func (Member_Name_Field) _Column() string { return "name" }
type Member_PasswordHash_Field struct {
_set bool
_null bool
_value []byte
}
func Member_PasswordHash(v []byte) Member_PasswordHash_Field {
return Member_PasswordHash_Field{_set: true, _value: v}
}
func (f Member_PasswordHash_Field) value() interface{} {
if !f._set || f._null {
return nil
}
return f._value
}
func (Member_PasswordHash_Field) _Column() string { return "password_hash" }
type Member_CreatedAt_Field struct {
_set bool
_null bool
_value time.Time
}
func Member_CreatedAt(v time.Time) Member_CreatedAt_Field {
return Member_CreatedAt_Field{_set: true, _value: v}
}
func (f Member_CreatedAt_Field) value() interface{} {
if !f._set || f._null {
return nil
}
return f._value
}
func (Member_CreatedAt_Field) _Column() string { return "created_at" }
type Node struct {
Id []byte
Name string
@ -924,6 +1043,38 @@ func (obj *pgxImpl) Create_Node(ctx context.Context,
}
func (obj *pgxImpl) Create_Member(ctx context.Context,
member_id Member_Id_Field,
member_email Member_Email_Field,
member_name Member_Name_Field,
member_password_hash Member_PasswordHash_Field) (
member *Member, err error) {
defer mon.Task()(&ctx)(&err)
__now := obj.db.Hooks.Now().UTC()
__id_val := member_id.value()
__email_val := member_email.value()
__name_val := member_name.value()
__password_hash_val := member_password_hash.value()
__created_at_val := __now
var __embed_stmt = __sqlbundle_Literal("INSERT INTO members ( id, email, name, password_hash, created_at ) VALUES ( ?, ?, ?, ?, ? ) RETURNING members.id, members.email, members.name, members.password_hash, members.created_at")
var __values []interface{}
__values = append(__values, __id_val, __email_val, __name_val, __password_hash_val, __created_at_val)
var __stmt = __sqlbundle_Render(obj.dialect, __embed_stmt)
obj.logStmt(__stmt, __values...)
member = &Member{}
err = obj.driver.QueryRowContext(ctx, __stmt, __values...).Scan(&member.Id, &member.Email, &member.Name, &member.PasswordHash, &member.CreatedAt)
if err != nil {
return nil, obj.makeErr(err)
}
return member, nil
}
func (obj *pgxImpl) Get_Node_By_Id(ctx context.Context,
node_id Node_Id_Field) (
node *Node, err error) {
@ -946,6 +1097,123 @@ func (obj *pgxImpl) Get_Node_By_Id(ctx context.Context,
}
func (obj *pgxImpl) Get_Member_By_Email(ctx context.Context,
member_email Member_Email_Field) (
member *Member, err error) {
defer mon.Task()(&ctx)(&err)
var __embed_stmt = __sqlbundle_Literal("SELECT members.id, members.email, members.name, members.password_hash, members.created_at FROM members WHERE members.email = ? LIMIT 2")
var __values []interface{}
__values = append(__values, member_email.value())
var __stmt = __sqlbundle_Render(obj.dialect, __embed_stmt)
obj.logStmt(__stmt, __values...)
__rows, err := obj.driver.QueryContext(ctx, __stmt, __values...)
if err != nil {
return nil, obj.makeErr(err)
}
defer __rows.Close()
if !__rows.Next() {
if err := __rows.Err(); err != nil {
return nil, obj.makeErr(err)
}
return nil, makeErr(sql.ErrNoRows)
}
member = &Member{}
err = __rows.Scan(&member.Id, &member.Email, &member.Name, &member.PasswordHash, &member.CreatedAt)
if err != nil {
return nil, obj.makeErr(err)
}
if __rows.Next() {
return nil, tooManyRows("Member_By_Email")
}
if err := __rows.Err(); err != nil {
return nil, obj.makeErr(err)
}
return member, nil
}
func (obj *pgxImpl) Get_Member_By_Id(ctx context.Context,
member_id Member_Id_Field) (
member *Member, err error) {
defer mon.Task()(&ctx)(&err)
var __embed_stmt = __sqlbundle_Literal("SELECT members.id, members.email, members.name, members.password_hash, members.created_at FROM members WHERE members.id = ?")
var __values []interface{}
__values = append(__values, member_id.value())
var __stmt = __sqlbundle_Render(obj.dialect, __embed_stmt)
obj.logStmt(__stmt, __values...)
member = &Member{}
err = obj.driver.QueryRowContext(ctx, __stmt, __values...).Scan(&member.Id, &member.Email, &member.Name, &member.PasswordHash, &member.CreatedAt)
if err != nil {
return (*Member)(nil), obj.makeErr(err)
}
return member, nil
}
func (obj *pgxImpl) Update_Member_By_Id(ctx context.Context,
member_id Member_Id_Field,
update Member_Update_Fields) (
member *Member, err error) {
defer mon.Task()(&ctx)(&err)
var __sets = &__sqlbundle_Hole{}
var __embed_stmt = __sqlbundle_Literals{Join: "", SQLs: []__sqlbundle_SQL{__sqlbundle_Literal("UPDATE members SET "), __sets, __sqlbundle_Literal(" WHERE members.id = ? RETURNING members.id, members.email, members.name, members.password_hash, members.created_at")}}
__sets_sql := __sqlbundle_Literals{Join: ", "}
var __values []interface{}
var __args []interface{}
if update.Email._set {
__values = append(__values, update.Email.value())
__sets_sql.SQLs = append(__sets_sql.SQLs, __sqlbundle_Literal("email = ?"))
}
if update.Name._set {
__values = append(__values, update.Name.value())
__sets_sql.SQLs = append(__sets_sql.SQLs, __sqlbundle_Literal("name = ?"))
}
if update.PasswordHash._set {
__values = append(__values, update.PasswordHash.value())
__sets_sql.SQLs = append(__sets_sql.SQLs, __sqlbundle_Literal("password_hash = ?"))
}
if len(__sets_sql.SQLs) == 0 {
return nil, emptyUpdate()
}
__args = append(__args, member_id.value())
__values = append(__values, __args...)
__sets.SQL = __sets_sql
var __stmt = __sqlbundle_Render(obj.dialect, __embed_stmt)
obj.logStmt(__stmt, __values...)
member = &Member{}
err = obj.driver.QueryRowContext(ctx, __stmt, __values...).Scan(&member.Id, &member.Email, &member.Name, &member.PasswordHash, &member.CreatedAt)
if err == sql.ErrNoRows {
return nil, nil
}
if err != nil {
return nil, obj.makeErr(err)
}
return member, nil
}
func (obj *pgxImpl) Delete_Node_By_Id(ctx context.Context,
node_id Node_Id_Field) (
deleted bool, err error) {
@ -973,6 +1241,33 @@ func (obj *pgxImpl) Delete_Node_By_Id(ctx context.Context,
}
func (obj *pgxImpl) Delete_Member_By_Id(ctx context.Context,
member_id Member_Id_Field) (
deleted bool, err error) {
defer mon.Task()(&ctx)(&err)
var __embed_stmt = __sqlbundle_Literal("DELETE FROM members WHERE members.id = ?")
var __values []interface{}
__values = append(__values, member_id.value())
var __stmt = __sqlbundle_Render(obj.dialect, __embed_stmt)
obj.logStmt(__stmt, __values...)
__res, err := obj.driver.ExecContext(ctx, __stmt, __values...)
if err != nil {
return false, obj.makeErr(err)
}
__count, err := __res.RowsAffected()
if err != nil {
return false, obj.makeErr(err)
}
return __count > 0, nil
}
func (impl pgxImpl) isConstraintError(err error) (
constraint string, ok bool) {
if e, ok := err.(*pgconn.PgError); ok {
@ -992,6 +1287,16 @@ func (obj *pgxImpl) deleteAll(ctx context.Context) (count int64, err error) {
return 0, obj.makeErr(err)
}
__count, err = __res.RowsAffected()
if err != nil {
return 0, obj.makeErr(err)
}
count += __count
__res, err = obj.driver.ExecContext(ctx, "DELETE FROM members;")
if err != nil {
return 0, obj.makeErr(err)
}
__count, err = __res.RowsAffected()
if err != nil {
return 0, obj.makeErr(err)
@ -1044,6 +1349,20 @@ func (rx *Rx) Rollback() (err error) {
return err
}
func (rx *Rx) Create_Member(ctx context.Context,
member_id Member_Id_Field,
member_email Member_Email_Field,
member_name Member_Name_Field,
member_password_hash Member_PasswordHash_Field) (
member *Member, err error) {
var tx *Tx
if tx, err = rx.getTx(ctx); err != nil {
return
}
return tx.Create_Member(ctx, member_id, member_email, member_name, member_password_hash)
}
func (rx *Rx) Create_Node(ctx context.Context,
node_id Node_Id_Field,
node_name Node_Name_Field,
@ -1060,6 +1379,16 @@ func (rx *Rx) Create_Node(ctx context.Context,
}
func (rx *Rx) Delete_Member_By_Id(ctx context.Context,
member_id Member_Id_Field) (
deleted bool, err error) {
var tx *Tx
if tx, err = rx.getTx(ctx); err != nil {
return
}
return tx.Delete_Member_By_Id(ctx, member_id)
}
func (rx *Rx) Delete_Node_By_Id(ctx context.Context,
node_id Node_Id_Field) (
deleted bool, err error) {
@ -1070,6 +1399,26 @@ func (rx *Rx) Delete_Node_By_Id(ctx context.Context,
return tx.Delete_Node_By_Id(ctx, node_id)
}
func (rx *Rx) Get_Member_By_Email(ctx context.Context,
member_email Member_Email_Field) (
member *Member, err error) {
var tx *Tx
if tx, err = rx.getTx(ctx); err != nil {
return
}
return tx.Get_Member_By_Email(ctx, member_email)
}
func (rx *Rx) Get_Member_By_Id(ctx context.Context,
member_id Member_Id_Field) (
member *Member, err error) {
var tx *Tx
if tx, err = rx.getTx(ctx); err != nil {
return
}
return tx.Get_Member_By_Id(ctx, member_id)
}
func (rx *Rx) Get_Node_By_Id(ctx context.Context,
node_id Node_Id_Field) (
node *Node, err error) {
@ -1080,7 +1429,25 @@ func (rx *Rx) Get_Node_By_Id(ctx context.Context,
return tx.Get_Node_By_Id(ctx, node_id)
}
func (rx *Rx) Update_Member_By_Id(ctx context.Context,
member_id Member_Id_Field,
update Member_Update_Fields) (
member *Member, err error) {
var tx *Tx
if tx, err = rx.getTx(ctx); err != nil {
return
}
return tx.Update_Member_By_Id(ctx, member_id, update)
}
type Methods interface {
Create_Member(ctx context.Context,
member_id Member_Id_Field,
member_email Member_Email_Field,
member_name Member_Name_Field,
member_password_hash Member_PasswordHash_Field) (
member *Member, err error)
Create_Node(ctx context.Context,
node_id Node_Id_Field,
node_name Node_Name_Field,
@ -1090,13 +1457,30 @@ type Methods interface {
node_logo Node_Logo_Field) (
node *Node, err error)
Delete_Member_By_Id(ctx context.Context,
member_id Member_Id_Field) (
deleted bool, err error)
Delete_Node_By_Id(ctx context.Context,
node_id Node_Id_Field) (
deleted bool, err error)
Get_Member_By_Email(ctx context.Context,
member_email Member_Email_Field) (
member *Member, err error)
Get_Member_By_Id(ctx context.Context,
member_id Member_Id_Field) (
member *Member, err error)
Get_Node_By_Id(ctx context.Context,
node_id Node_Id_Field) (
node *Node, err error)
Update_Member_By_Id(ctx context.Context,
member_id Member_Id_Field,
update Member_Update_Fields) (
member *Member, err error)
}
type TxMethods interface {

View File

@ -1,5 +1,13 @@
-- AUTOGENERATED BY storj.io/dbx
-- DO NOT EDIT
CREATE TABLE members (
id bytea NOT NULL,
email text NOT NULL,
name text NOT NULL,
password_hash bytea NOT NULL,
created_at timestamp with time zone NOT NULL,
PRIMARY KEY ( id )
);
CREATE TABLE nodes (
id bytea NOT NULL,
name text NOT NULL,

View File

@ -0,0 +1,117 @@
// Copyright (C) 2020 Storj Labs, Inc.
// See LICENSE for copying information.
package multinodedb
import (
"context"
"database/sql"
"errors"
"github.com/zeebo/errs"
"storj.io/common/uuid"
"storj.io/storj/multinode/console"
"storj.io/storj/multinode/multinodedb/dbx"
)
// MembersDBError indicates about internal MembersDB error.
var MembersDBError = errs.Class("MembersDB error")
// ensures that members implements console.Members.
var _ console.Members = (*members)(nil)
// members exposes needed by MND MembersDB functionality.
// dbx implementation of console.Members.
//
// architecture: Database
type members struct {
methods dbx.Methods
}
// Invite will create empty row in membersDB.
func (m *members) Invite(ctx context.Context, member console.Member) (err error) {
defer mon.Task()(&ctx)(&err)
id, err := uuid.New()
if err != nil {
return MembersDBError.Wrap(err)
}
_, err = m.methods.Create_Member(ctx, dbx.Member_Id(id[:]), dbx.Member_Email(member.Email), dbx.Member_Name(member.Name), dbx.Member_PasswordHash(member.PasswordHash))
return MembersDBError.Wrap(err)
}
// Update updates all updatable fields of member.
func (m *members) Update(ctx context.Context, member console.Member) (err error) {
defer mon.Task()(&ctx)(&err)
_, err = m.methods.Update_Member_By_Id(ctx, dbx.Member_Id(member.ID[:]), dbx.Member_Update_Fields{
Email: dbx.Member_Email(member.Email),
Name: dbx.Member_Name(member.Name),
PasswordHash: dbx.Member_PasswordHash(member.PasswordHash),
})
return MembersDBError.Wrap(err)
}
// Remove deletes member from membersDB.
func (m *members) Remove(ctx context.Context, id uuid.UUID) (err error) {
defer mon.Task()(&ctx)(&err)
_, err = m.methods.Delete_Member_By_Id(ctx, dbx.Member_Id(id[:]))
return MembersDBError.Wrap(err)
}
// GetByEmail will return member with specified email.
func (m *members) GetByEmail(ctx context.Context, email string) (_ console.Member, err error) {
defer mon.Task()(&ctx)(&err)
memberDbx, err := m.methods.Get_Member_By_Email(ctx, dbx.Member_Email(email))
if err != nil {
if errors.Is(err, sql.ErrNoRows) {
return console.Member{}, console.ErrNoMember.Wrap(err)
}
return console.Member{}, MembersDBError.Wrap(err)
}
member, err := fromDBXMember(memberDbx)
return member, MembersDBError.Wrap(err)
}
// GetByID will return member with specified id.
func (m *members) GetByID(ctx context.Context, id uuid.UUID) (_ console.Member, err error) {
defer mon.Task()(&ctx)(&err)
memberDbx, err := m.methods.Get_Member_By_Id(ctx, dbx.Member_Id(id[:]))
if err != nil {
if errors.Is(err, sql.ErrNoRows) {
return console.Member{}, console.ErrNoMember.Wrap(err)
}
return console.Member{}, MembersDBError.Wrap(err)
}
member, err := fromDBXMember(memberDbx)
return member, MembersDBError.Wrap(err)
}
// fromDBXMember converts dbx.Member to console.Member.
func fromDBXMember(member *dbx.Member) (_ console.Member, err error) {
id, err := uuid.FromBytes(member.Id)
if err != nil {
return console.Member{}, err
}
result := console.Member{
ID: id,
Email: member.Email,
Name: member.Name,
PasswordHash: member.PasswordHash,
}
return result, nil
}

View File

@ -0,0 +1,141 @@
// Copyright (C) 2020 Storj Labs, Inc.
// See LICENSE for copying information.
package multinodedbtest
import (
"context"
"fmt"
"strconv"
"strings"
"testing"
"github.com/zeebo/errs"
"go.uber.org/zap"
"go.uber.org/zap/zaptest"
"storj.io/common/testcontext"
"storj.io/storj/multinode"
"storj.io/storj/multinode/multinodedb"
"storj.io/storj/multinode/multinodedb/dbx"
"storj.io/storj/private/dbutil"
"storj.io/storj/private/dbutil/pgtest"
"storj.io/storj/private/dbutil/pgutil"
"storj.io/storj/private/dbutil/tempdb"
)
// Database describes a test database.
type Database struct {
Name string
URL string
Message string
}
// tempMasterDB is a multinode.DB-implementing type that cleans up after itself when closed.
type tempMasterDB struct {
multinode.DB
tempDB *dbutil.TempDatabase
}
// Close closes a tempMasterDB and cleans it up afterward.
func (db *tempMasterDB) Close() error {
return errs.Combine(db.DB.Close(), db.tempDB.Close())
}
// TestDBAccess provides a somewhat regularized access to the underlying DB.
func (db *tempMasterDB) TestDBAccess() *dbx.DB {
return db.DB.(interface{ TestDBAccess() *dbx.DB }).TestDBAccess()
}
type ignoreSkip struct{}
func (ignoreSkip) Skip(...interface{}) {}
// SchemaSuffix returns a suffix for schemas.
func SchemaSuffix() string {
return pgutil.CreateRandomTestingSchemaName(6)
}
// SchemaName returns a properly formatted schema string.
func SchemaName(testname, category string, index int, schemaSuffix string) string {
// postgres has a maximum schema length of 64
// we need additional 6 bytes for the random suffix
// and 4 bytes for the index "/S0/""
indexStr := strconv.Itoa(index)
var maxTestNameLen = 64 - len(category) - len(indexStr) - len(schemaSuffix) - 2
if len(testname) > maxTestNameLen {
testname = testname[:maxTestNameLen]
}
if schemaSuffix == "" {
return strings.ToLower(testname + "/" + category + indexStr)
}
return strings.ToLower(testname + "/" + schemaSuffix + "/" + category + indexStr)
}
// CreateMasterDB creates a new satellite database for testing.
func CreateMasterDB(ctx context.Context, log *zap.Logger, name string, category string, index int, dbInfo Database) (db multinode.DB, err error) {
if dbInfo.URL == "" {
return nil, fmt.Errorf("database %s connection string not provided. %s", dbInfo.Name, dbInfo.Message)
}
schemaSuffix := SchemaSuffix()
log.Debug("creating", zap.String("suffix", schemaSuffix))
schema := SchemaName(name, category, index, schemaSuffix)
tempDB, err := tempdb.OpenUnique(ctx, dbInfo.URL, schema)
if err != nil {
return nil, err
}
return CreateMasterDBOnTopOf(ctx, log, tempDB)
}
// CreateMasterDBOnTopOf creates a new satellite database on top of an already existing
// temporary database.
func CreateMasterDBOnTopOf(ctx context.Context, log *zap.Logger, tempDB *dbutil.TempDatabase) (db multinode.DB, err error) {
masterDB, err := multinodedb.Open(ctx, log, tempDB.ConnStr)
return &tempMasterDB{DB: masterDB, tempDB: tempDB}, err
}
// Run method will iterate over all supported databases. Will establish
// connection and will create tables for each DB.
func Run(t *testing.T, test func(ctx *testcontext.Context, t *testing.T, db multinode.DB)) {
masterDB := Database{
Name: "Postgres",
URL: pgtest.PickPostgres(ignoreSkip{}),
Message: "Postgres flag missing, example: -postgres-test-db=" + pgtest.DefaultPostgres + " or use STORJ_TEST_POSTGRES environment variable.",
}
t.Run(masterDB.Name, func(t *testing.T) {
t.Parallel()
ctx := testcontext.New(t)
defer ctx.Cleanup()
if masterDB.URL == "" {
t.Skipf("Database %s connection string not provided. %s", masterDB.Name, masterDB.Message)
}
db, err := CreateMasterDB(ctx, zaptest.NewLogger(t), t.Name(), "T", 0, masterDB)
if err != nil {
t.Fatal(err)
}
defer func() {
err := db.Close()
if err != nil {
t.Fatal(err)
}
}()
err = db.CreateSchema(ctx)
if err != nil {
t.Fatal(err)
}
test(ctx, t, db)
})
}

View File

@ -1,7 +1,7 @@
// Copyright (C) 2020 Storj Labs, Inc.
// See LICENSE for copying information.
package mutlinodedb
package multinodedb
import (
"context"
@ -10,7 +10,7 @@ import (
"storj.io/common/storj"
"storj.io/storj/multinode/console"
"storj.io/storj/multinode/mutlinodedb/dbx"
"storj.io/storj/multinode/multinodedb/dbx"
)
// NodesDBError indicates about internal NodesDB error.
@ -25,7 +25,6 @@ var _ console.Nodes = (*nodes)(nil)
// architecture: Database
type nodes struct {
methods dbx.Methods
db *multinodeDB
}
// Add creates new node in NodesDB.

View File

@ -1,12 +0,0 @@
// Copyright (C) 2020 Storj Labs, Inc.
// See LICENSE for copying information.
package dbx
import (
"github.com/spacemonkeygo/monkit/v3"
)
//go:generate sh gen.sh
var mon = monkit.Package()

View File

@ -1,20 +0,0 @@
// dbx.v1 golang multinodedb.dbx .
model node (
key id
field id blob
field name text ( updatable )
field tag text ( updatable )
field public_address text
field api_secret blob
field logo blob ( updatable )
)
create node ( )
delete node ( where node.id = ? )
read one (
select node
where node.id = ?
)

View File

@ -27,9 +27,13 @@ var (
type DB interface {
// Nodes returns nodes database.
Nodes() console.Nodes
// Members returns members database.
Members() console.Members
// Close closes the database.
Close() error
// CreateSchema creates schema.
CreateSchema(ctx context.Context) error
}
// Config is all the configuration parameters for a Multinode Dashboard.

View File

@ -5,9 +5,9 @@ package pgtest
import (
"flag"
"math/rand"
"os"
"strings"
"sync/atomic"
"testing"
"storj.io/common/testcontext"
@ -77,20 +77,20 @@ func Run(t *testing.T, test func(ctx *testcontext.Context, t *testing.T, connstr
}
}
// PickPostgres picks a random postgres database from flag.
// PickPostgres picks one postgres database from flag.
func PickPostgres(t TB) string {
if *postgres == "" || strings.EqualFold(*postgres, "omit") {
t.Skip("Postgres flag missing, example: -postgres-test-db=" + DefaultPostgres)
}
return pickRandom(*postgres)
return pickNext(*postgres, &pickPostgres)
}
// PickCockroach picks a random cockroach database from flag.
// PickCockroach picks one cockroach database from flag.
func PickCockroach(t TB) string {
if *cockroach == "" || strings.EqualFold(*cockroach, "omit") {
t.Skip("Cockroach flag missing, example: -cockroach-test-db=" + DefaultCockroach)
}
return pickRandom(*cockroach)
return pickNext(*cockroach, &pickCockroach)
}
// PickCockroachAlt picks an alternate cockroach database from flag.
@ -104,13 +104,17 @@ func PickCockroachAlt(t TB) string {
t.Skip("Cockroach alt flag omitted.")
}
return pickRandom(*cockroachAlt)
return pickNext(*cockroachAlt, &pickCockroach)
}
func pickRandom(dbstr string) string {
var pickPostgres uint64
var pickCockroach uint64
func pickNext(dbstr string, counter *uint64) string {
values := strings.Split(dbstr, ";")
if len(values) <= 1 {
return dbstr
}
return values[rand.Intn(len(values))]
v := atomic.AddUint64(counter, 1)
return values[v%uint64(len(values))]
}

View File

@ -73,84 +73,97 @@ func QuerySchema(ctx context.Context, db dbschema.Queryer) (*dbschema.Schema, er
func discoverTables(ctx context.Context, db dbschema.Queryer, schema *dbschema.Schema, tableDefinitions []*definition) (err error) {
for _, definition := range tableDefinitions {
table := schema.EnsureTable(definition.name)
tableRows, err := db.QueryContext(ctx, `PRAGMA table_info(`+definition.name+`)`)
if err != nil {
return errs.Wrap(err)
}
defer func() { err = errs.Combine(err, tableRows.Close()) }()
for tableRows.Next() {
var defaultValue sql.NullString
var index, name, columnType string
var pk int
var notNull bool
err := tableRows.Scan(&index, &name, &columnType, &notNull, &defaultValue, &pk)
if err != nil {
return errs.Wrap(err)
}
column := &dbschema.Column{
Name: name,
Type: columnType,
IsNullable: !notNull && pk == 0,
}
table.AddColumn(column)
if pk > 0 {
if table.PrimaryKey == nil {
table.PrimaryKey = make([]string, 0)
}
table.PrimaryKey = append(table.PrimaryKey, name)
}
}
matches := rxUnique.FindAllStringSubmatch(definition.sql, -1)
for _, match := range matches {
// TODO feel this can be done easier
var columns []string
for _, name := range strings.Split(match[1], ",") {
columns = append(columns, strings.TrimSpace(name))
}
table.Unique = append(table.Unique, columns)
}
keysRows, err := db.QueryContext(ctx, `PRAGMA foreign_key_list(`+definition.name+`)`)
if err != nil {
return errs.Wrap(err)
}
defer func() { err = errs.Combine(err, keysRows.Close()) }()
for keysRows.Next() {
var id, sec int
var tableName, from, to, onUpdate, onDelete, match string
err := keysRows.Scan(&id, &sec, &tableName, &from, &to, &onUpdate, &onDelete, &match)
if err != nil {
return errs.Wrap(err)
}
column, found := table.FindColumn(from)
if found {
if onDelete == "NO ACTION" {
onDelete = ""
}
if onUpdate == "NO ACTION" {
onUpdate = ""
}
column.Reference = &dbschema.Reference{
Table: tableName,
Column: to,
OnUpdate: onUpdate,
OnDelete: onDelete,
}
}
if err := discoverTable(ctx, db, schema, definition); err != nil {
return err
}
}
return errs.Wrap(err)
}
func discoverTable(ctx context.Context, db dbschema.Queryer, schema *dbschema.Schema, definition *definition) (err error) {
table := schema.EnsureTable(definition.name)
tableRows, err := db.QueryContext(ctx, `PRAGMA table_info(`+definition.name+`)`)
if err != nil {
return errs.Wrap(err)
}
for tableRows.Next() {
var defaultValue sql.NullString
var index, name, columnType string
var pk int
var notNull bool
err := tableRows.Scan(&index, &name, &columnType, &notNull, &defaultValue, &pk)
if err != nil {
return errs.Wrap(errs.Combine(tableRows.Err(), tableRows.Close(), err))
}
column := &dbschema.Column{
Name: name,
Type: columnType,
IsNullable: !notNull && pk == 0,
}
table.AddColumn(column)
if pk > 0 {
if table.PrimaryKey == nil {
table.PrimaryKey = make([]string, 0)
}
table.PrimaryKey = append(table.PrimaryKey, name)
}
}
err = errs.Combine(tableRows.Err(), tableRows.Close())
if err != nil {
return errs.Wrap(err)
}
matches := rxUnique.FindAllStringSubmatch(definition.sql, -1)
for _, match := range matches {
// TODO feel this can be done easier
var columns []string
for _, name := range strings.Split(match[1], ",") {
columns = append(columns, strings.TrimSpace(name))
}
table.Unique = append(table.Unique, columns)
}
keysRows, err := db.QueryContext(ctx, `PRAGMA foreign_key_list(`+definition.name+`)`)
if err != nil {
return errs.Wrap(err)
}
for keysRows.Next() {
var id, sec int
var tableName, from, to, onUpdate, onDelete, match string
err := keysRows.Scan(&id, &sec, &tableName, &from, &to, &onUpdate, &onDelete, &match)
if err != nil {
return errs.Wrap(errs.Combine(keysRows.Err(), keysRows.Close(), err))
}
column, found := table.FindColumn(from)
if found {
if onDelete == "NO ACTION" {
onDelete = ""
}
if onUpdate == "NO ACTION" {
onUpdate = ""
}
column.Reference = &dbschema.Reference{
Table: tableName,
Column: to,
OnUpdate: onUpdate,
OnDelete: onDelete,
}
}
}
err = errs.Combine(keysRows.Err(), keysRows.Close())
if err != nil {
return errs.Wrap(err)
}
return nil
}
func discoverIndexes(ctx context.Context, db dbschema.Queryer, schema *dbschema.Schema, indexDefinitions []*definition) (err error) {
// TODO improve indexes discovery
for _, definition := range indexDefinitions {

View File

@ -8,6 +8,7 @@ import (
"context"
"errors"
"runtime/pprof"
"time"
"github.com/spacemonkeygo/monkit/v3"
"github.com/zeebo/errs"
@ -54,7 +55,27 @@ func (group *Group) Run(ctx context.Context, g *errgroup.Group) {
if item.Run == nil {
continue
}
shutdownCtx, shutdownFinished := context.WithCancel(context.Background())
go func() {
select {
case <-ctx.Done():
case <-shutdownCtx.Done():
return
}
shutdownDeadline := time.NewTimer(15 * time.Second)
defer shutdownDeadline.Stop()
select {
case <-shutdownDeadline.C:
group.log.Warn("service takes long to shutdown", zap.String("name", item.Name))
case <-shutdownCtx.Done():
}
}()
g.Go(func() error {
defer shutdownFinished()
var err error
pprof.Do(ctx, pprof.Labels("name", item.Name), func(ctx context.Context) {
err = item.Run(ctx)

View File

@ -5,6 +5,7 @@ package testblobs
import (
"context"
"sync"
"time"
"go.uber.org/zap"
@ -43,11 +44,30 @@ func (bad *BadDB) SetError(err error) {
// BadBlobs implements a bad blob store.
type BadBlobs struct {
err error
err lockedErr
blobs storage.Blobs
log *zap.Logger
}
type lockedErr struct {
mu sync.Mutex
err error
}
// Err returns the error.
func (m *lockedErr) Err() error {
m.mu.Lock()
defer m.mu.Unlock()
return m.err
}
// Set sets the error.
func (m *lockedErr) Set(err error) {
m.mu.Lock()
defer m.mu.Unlock()
m.err = err
}
// newBadBlobs creates a new bad blob store wrapping the provided blobs.
// Use SetError to manually configure the error returned by all operations.
func newBadBlobs(log *zap.Logger, blobs storage.Blobs) *BadBlobs {
@ -57,27 +77,32 @@ func newBadBlobs(log *zap.Logger, blobs storage.Blobs) *BadBlobs {
}
}
// SetError configures the blob store to return a specific error for all operations.
func (bad *BadBlobs) SetError(err error) {
bad.err.Set(err)
}
// Create creates a new blob that can be written optionally takes a size
// argument for performance improvements, -1 is unknown size.
func (bad *BadBlobs) Create(ctx context.Context, ref storage.BlobRef, size int64) (storage.BlobWriter, error) {
if bad.err != nil {
return nil, bad.err
if err := bad.err.Err(); err != nil {
return nil, err
}
return bad.blobs.Create(ctx, ref, size)
}
// Close closes the blob store and any resources associated with it.
func (bad *BadBlobs) Close() error {
if bad.err != nil {
return bad.err
if err := bad.err.Err(); err != nil {
return err
}
return bad.blobs.Close()
}
// Open opens a reader with the specified namespace and key.
func (bad *BadBlobs) Open(ctx context.Context, ref storage.BlobRef) (storage.BlobReader, error) {
if bad.err != nil {
return nil, bad.err
if err := bad.err.Err(); err != nil {
return nil, err
}
return bad.blobs.Open(ctx, ref)
}
@ -85,64 +110,64 @@ func (bad *BadBlobs) Open(ctx context.Context, ref storage.BlobRef) (storage.Blo
// OpenWithStorageFormat opens a reader for the already-located blob, avoiding the potential need
// to check multiple storage formats to find the blob.
func (bad *BadBlobs) OpenWithStorageFormat(ctx context.Context, ref storage.BlobRef, formatVer storage.FormatVersion) (storage.BlobReader, error) {
if bad.err != nil {
return nil, bad.err
if err := bad.err.Err(); err != nil {
return nil, err
}
return bad.blobs.OpenWithStorageFormat(ctx, ref, formatVer)
}
// Trash deletes the blob with the namespace and key.
func (bad *BadBlobs) Trash(ctx context.Context, ref storage.BlobRef) error {
if bad.err != nil {
return bad.err
if err := bad.err.Err(); err != nil {
return err
}
return bad.blobs.Trash(ctx, ref)
}
// RestoreTrash restores all files in the trash.
func (bad *BadBlobs) RestoreTrash(ctx context.Context, namespace []byte) ([][]byte, error) {
if bad.err != nil {
return nil, bad.err
if err := bad.err.Err(); err != nil {
return nil, err
}
return bad.blobs.RestoreTrash(ctx, namespace)
}
// EmptyTrash empties the trash.
func (bad *BadBlobs) EmptyTrash(ctx context.Context, namespace []byte, trashedBefore time.Time) (int64, [][]byte, error) {
if bad.err != nil {
return 0, nil, bad.err
if err := bad.err.Err(); err != nil {
return 0, nil, err
}
return bad.blobs.EmptyTrash(ctx, namespace, trashedBefore)
}
// Delete deletes the blob with the namespace and key.
func (bad *BadBlobs) Delete(ctx context.Context, ref storage.BlobRef) error {
if bad.err != nil {
return bad.err
if err := bad.err.Err(); err != nil {
return err
}
return bad.blobs.Delete(ctx, ref)
}
// DeleteWithStorageFormat deletes the blob with the namespace, key, and format version.
func (bad *BadBlobs) DeleteWithStorageFormat(ctx context.Context, ref storage.BlobRef, formatVer storage.FormatVersion) error {
if bad.err != nil {
return bad.err
if err := bad.err.Err(); err != nil {
return err
}
return bad.blobs.DeleteWithStorageFormat(ctx, ref, formatVer)
}
// DeleteNamespace deletes blobs of specific satellite, used after successful GE only.
func (bad *BadBlobs) DeleteNamespace(ctx context.Context, ref []byte) (err error) {
if bad.err != nil {
return bad.err
if err := bad.err.Err(); err != nil {
return err
}
return bad.blobs.DeleteNamespace(ctx, ref)
}
// Stat looks up disk metadata on the blob file.
func (bad *BadBlobs) Stat(ctx context.Context, ref storage.BlobRef) (storage.BlobInfo, error) {
if bad.err != nil {
return nil, bad.err
if err := bad.err.Err(); err != nil {
return nil, err
}
return bad.blobs.Stat(ctx, ref)
}
@ -151,8 +176,8 @@ func (bad *BadBlobs) Stat(ctx context.Context, ref storage.BlobRef) (storage.Blo
// version. This avoids the potential need to check multiple storage formats for the blob
// when the format is already known.
func (bad *BadBlobs) StatWithStorageFormat(ctx context.Context, ref storage.BlobRef, formatVer storage.FormatVersion) (storage.BlobInfo, error) {
if bad.err != nil {
return nil, bad.err
if err := bad.err.Err(); err != nil {
return nil, err
}
return bad.blobs.StatWithStorageFormat(ctx, ref, formatVer)
}
@ -161,64 +186,64 @@ func (bad *BadBlobs) StatWithStorageFormat(ctx context.Context, ref storage.Blob
// If walkFunc returns a non-nil error, WalkNamespace will stop iterating and return the
// error immediately.
func (bad *BadBlobs) WalkNamespace(ctx context.Context, namespace []byte, walkFunc func(storage.BlobInfo) error) error {
if bad.err != nil {
return bad.err
if err := bad.err.Err(); err != nil {
return err
}
return bad.blobs.WalkNamespace(ctx, namespace, walkFunc)
}
// ListNamespaces returns all namespaces that might be storing data.
func (bad *BadBlobs) ListNamespaces(ctx context.Context) ([][]byte, error) {
if bad.err != nil {
return make([][]byte, 0), bad.err
if err := bad.err.Err(); err != nil {
return make([][]byte, 0), err
}
return bad.blobs.ListNamespaces(ctx)
}
// FreeSpace return how much free space left for writing.
func (bad *BadBlobs) FreeSpace() (int64, error) {
if bad.err != nil {
return 0, bad.err
if err := bad.err.Err(); err != nil {
return 0, err
}
return bad.blobs.FreeSpace()
}
// CheckWritability tests writability of the storage directory by creating and deleting a file.
func (bad *BadBlobs) CheckWritability() error {
if bad.err != nil {
return bad.err
if err := bad.err.Err(); err != nil {
return err
}
return bad.blobs.CheckWritability()
}
// SpaceUsedForBlobs adds up how much is used in all namespaces.
func (bad *BadBlobs) SpaceUsedForBlobs(ctx context.Context) (int64, error) {
if bad.err != nil {
return 0, bad.err
if err := bad.err.Err(); err != nil {
return 0, err
}
return bad.blobs.SpaceUsedForBlobs(ctx)
}
// SpaceUsedForBlobsInNamespace adds up how much is used in the given namespace.
func (bad *BadBlobs) SpaceUsedForBlobsInNamespace(ctx context.Context, namespace []byte) (int64, error) {
if bad.err != nil {
return 0, bad.err
if err := bad.err.Err(); err != nil {
return 0, err
}
return bad.blobs.SpaceUsedForBlobsInNamespace(ctx, namespace)
}
// SpaceUsedForTrash adds up how much is used in all namespaces.
func (bad *BadBlobs) SpaceUsedForTrash(ctx context.Context) (int64, error) {
if bad.err != nil {
return 0, bad.err
if err := bad.err.Err(); err != nil {
return 0, err
}
return bad.blobs.SpaceUsedForTrash(ctx)
}
// CreateVerificationFile creates a file to be used for storage directory verification.
func (bad *BadBlobs) CreateVerificationFile(id storj.NodeID) error {
if bad.err != nil {
return bad.err
if err := bad.err.Err(); err != nil {
return err
}
return bad.blobs.CreateVerificationFile(id)
}
@ -226,13 +251,8 @@ func (bad *BadBlobs) CreateVerificationFile(id storj.NodeID) error {
// VerifyStorageDir verifies that the storage directory is correct by checking for the existence and validity
// of the verification file.
func (bad *BadBlobs) VerifyStorageDir(id storj.NodeID) error {
if bad.err != nil {
return bad.err
if err := bad.err.Err(); err != nil {
return err
}
return bad.blobs.VerifyStorageDir(id)
}
// SetError configures the blob store to return a specific error for all operations.
func (bad *BadBlobs) SetError(err error) {
bad.err = err
}

69
private/testplanet/log.go Normal file
View File

@ -0,0 +1,69 @@
// Copyright (C) 2020 Storj Labs, Inc.
// See LICENSE for copying information
package testplanet
import (
"bytes"
"fmt"
"os"
"testing"
"time"
"go.uber.org/zap"
"go.uber.org/zap/zapcore"
"go.uber.org/zap/zaptest"
)
var useAbsTime = os.Getenv("STORJ_TESTPLANET_ABSTIME")
func newLogger(t *testing.T) *zap.Logger {
if useAbsTime != "" {
return zaptest.NewLogger(t)
}
start := time.Now()
cfg := zap.NewDevelopmentEncoderConfig()
cfg.EncodeTime = func(t time.Time, enc zapcore.PrimitiveArrayEncoder) {
var nanos, seconds, minutes int64
nanos = t.Sub(start).Nanoseconds()
seconds, nanos = nanos/1e9, nanos%1e9
minutes, seconds = seconds/60, seconds%60
enc.AppendString(fmt.Sprintf("%02d:%02d.%03d", minutes, seconds, nanos/1e6))
}
enc := zapcore.NewConsoleEncoder(cfg)
writer := newTestingWriter(t)
return zap.New(
zapcore.NewCore(enc, writer, zapcore.DebugLevel),
zap.ErrorOutput(writer.WithMarkFailed(true)),
)
}
type testingWriter struct {
t *testing.T
markFailed bool
}
func newTestingWriter(t *testing.T) testingWriter {
return testingWriter{t: t}
}
func (w testingWriter) WithMarkFailed(v bool) testingWriter {
w.markFailed = v
return w
}
func (w testingWriter) Write(p []byte) (n int, err error) {
n = len(p)
p = bytes.TrimRight(p, "\n")
w.t.Logf("%s", p)
if w.markFailed {
w.t.Fail()
}
return n, nil
}
func (w testingWriter) Sync() error { return nil }

View File

@ -34,18 +34,20 @@ func TestBasic(t *testing.T) {
for _, sat := range planet.Satellites {
for _, sn := range planet.StorageNodes {
node := sn.Contact.Service.Local()
conn, err := sn.Dialer.DialNodeURL(ctx, sat.NodeURL())
func() {
node := sn.Contact.Service.Local()
conn, err := sn.Dialer.DialNodeURL(ctx, sat.NodeURL())
require.NoError(t, err)
defer ctx.Check(conn.Close)
_, err = pb.NewDRPCNodeClient(conn).CheckIn(ctx, &pb.CheckInRequest{
Address: node.Address,
Version: &node.Version,
Capacity: &node.Capacity,
Operator: &node.Operator,
})
require.NoError(t, err)
require.NoError(t, err)
defer ctx.Check(conn.Close)
_, err = pb.NewDRPCNodeClient(conn).CheckIn(ctx, &pb.CheckInRequest{
Address: node.Address,
Version: &node.Version,
Capacity: &node.Capacity,
Operator: &node.Operator,
})
require.NoError(t, err)
}()
}
}
// wait a bit to see whether some failures occur

View File

@ -67,10 +67,10 @@ var Combine = func(elements ...func(log *zap.Logger, index int, config *satellit
// ReconfigureRS returns function to change satellite redundancy scheme values.
var ReconfigureRS = func(minThreshold, repairThreshold, successThreshold, totalThreshold int) func(log *zap.Logger, index int, config *satellite.Config) {
return func(log *zap.Logger, index int, config *satellite.Config) {
config.Metainfo.RS.MinThreshold = minThreshold
config.Metainfo.RS.RepairThreshold = repairThreshold
config.Metainfo.RS.SuccessThreshold = successThreshold
config.Metainfo.RS.TotalThreshold = totalThreshold
config.Metainfo.RS.Min = minThreshold
config.Metainfo.RS.Repair = repairThreshold
config.Metainfo.RS.Success = successThreshold
config.Metainfo.RS.Total = totalThreshold
}
}

View File

@ -9,8 +9,6 @@ import (
"strings"
"testing"
"go.uber.org/zap/zaptest"
"storj.io/common/testcontext"
"storj.io/storj/private/dbutil/pgtest"
"storj.io/storj/satellite/satellitedb/satellitedbtest"
@ -54,7 +52,7 @@ func Run(t *testing.T, config Config, test func(t *testing.T, ctx *testcontext.C
}
pprof.Do(ctx, pprof.Labels("planet", planetConfig.Name), func(namedctx context.Context) {
planet, err := NewCustom(namedctx, zaptest.NewLogger(t), planetConfig, satelliteDB)
planet, err := NewCustom(namedctx, newLogger(t), planetConfig, satelliteDB)
if err != nil {
t.Fatalf("%+v", err)
}

View File

@ -468,16 +468,11 @@ func (planet *Planet) newSatellite(ctx context.Context, prefix string, index int
MaxCommitInterval: 1 * time.Hour,
Overlay: true,
RS: metainfo.RSConfig{
MaxBufferMem: memory.Size(256),
ErasureShareSize: memory.Size(256),
MinThreshold: atLeastOne(planet.config.StorageNodeCount * 1 / 5),
RepairThreshold: atLeastOne(planet.config.StorageNodeCount * 2 / 5),
SuccessThreshold: atLeastOne(planet.config.StorageNodeCount * 3 / 5),
TotalThreshold: atLeastOne(planet.config.StorageNodeCount * 4 / 5),
MinTotalThreshold: (planet.config.StorageNodeCount * 4 / 5),
MaxTotalThreshold: (planet.config.StorageNodeCount * 4 / 5),
Validate: false,
Min: atLeastOne(planet.config.StorageNodeCount * 1 / 5),
Repair: atLeastOne(planet.config.StorageNodeCount * 2 / 5),
Success: atLeastOne(planet.config.StorageNodeCount * 3 / 5),
Total: atLeastOne(planet.config.StorageNodeCount * 4 / 5),
},
Loop: metainfo.LoopConfig{
CoalesceDuration: 1 * time.Second,

View File

@ -4,6 +4,6 @@
/*
Package live provides live accounting functionality. That is, it keeps track
of deltas in the amount of storage used by each project relative to the last
tally operation (see pkg/accounting/tally).
tally operation (see satellite/accounting/tally).
*/
package live

View File

@ -217,7 +217,6 @@ func dqNodes(ctx *testcontext.Context, planet *testplanet.Planet) (map[storj.Nod
}
updateRequests = append(updateRequests, &overlay.UpdateRequest{
NodeID: n.ID(),
IsUp: true,
AuditOutcome: overlay.AuditFailure,
})
}

View File

@ -84,6 +84,24 @@ func (service *Service) SetNow(now func() time.Time) {
}
// Tally calculates data-at-rest usage once.
//
// How live accounting is calculated:
//
// At the beginning of the tally iteration, we get a map containing the current
// project totals from the cache- initialLiveTotals (our current estimation of
// the project totals). At the end of the tally iteration, we have the totals
// from what we saw during the metainfo loop.
//
// However, data which was uploaded during the loop may or may not have been
// seen in the metainfo loop. For this reason, we also read the live accounting
// totals again at the end of the tally iteration- latestLiveTotals.
//
// The difference between latest and initial indicates how much data was
// uploaded during the metainfo loop and is assigned to delta. However, again,
// we aren't certain how much of the delta is accounted for in the metainfo
// totals. For the reason we make an assumption that 50% of the data is
// accounted for. So to calculate the new live accounting totals, we sum the
// metainfo totals and 50% of the deltas.
func (service *Service) Tally(ctx context.Context) (err error) {
defer mon.Task()(&ctx)(&err)
@ -151,6 +169,9 @@ func (service *Service) Tally(ctx context.Context) (err error) {
if delta < 0 {
delta = 0
}
// read the method documentation why the increase passed to this method
// is calculated in this way
err = service.liveAccounting.AddProjectStorageUsage(ctx, projectID, -latestLiveTotals[projectID]+tallyTotal+(delta/2))
if err != nil {
return Error.Wrap(err)

View File

@ -143,7 +143,7 @@ func TestCalculateNodeAtRestData(t *testing.T) {
require.NoError(t, err)
// Confirm the correct number of shares were stored
rs := satelliteRS(planet.Satellites[0])
rs := satelliteRS(t, planet.Satellites[0])
if !correctRedundencyScheme(len(obs.Node), rs) {
t.Fatalf("expected between: %d and %d, actual: %d", rs.RepairShares, rs.TotalShares, len(obs.Node))
}
@ -175,7 +175,7 @@ func TestCalculateBucketAtRestData(t *testing.T) {
SatelliteCount: 1, StorageNodeCount: 6, UplinkCount: 1,
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
satellitePeer := planet.Satellites[0]
redundancyScheme := satelliteRS(satellitePeer)
redundancyScheme := satelliteRS(t, satellitePeer)
expectedBucketTallies := make(map[metabase.BucketLocation]*accounting.BucketTally)
for _, tt := range testCases {
tt := tt // avoid scopelint error
@ -221,7 +221,7 @@ func TestTallyIgnoresExpiredPointers(t *testing.T) {
SatelliteCount: 1, StorageNodeCount: 6, UplinkCount: 1,
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
satellitePeer := planet.Satellites[0]
redundancyScheme := satelliteRS(satellitePeer)
redundancyScheme := satelliteRS(t, satellitePeer)
projectID, err := uuid.FromString("9656af6e-2d9c-42fa-91f2-bfd516a722d7")
require.NoError(t, err)
@ -423,12 +423,14 @@ func correctRedundencyScheme(shareCount int, uplinkRS storj.RedundancyScheme) bo
return int(uplinkRS.RepairShares) <= shareCount && shareCount <= int(uplinkRS.TotalShares)
}
func satelliteRS(satellite *testplanet.Satellite) storj.RedundancyScheme {
func satelliteRS(t *testing.T, satellite *testplanet.Satellite) storj.RedundancyScheme {
rs := satellite.Config.Metainfo.RS
return storj.RedundancyScheme{
RequiredShares: int16(satellite.Config.Metainfo.RS.MinThreshold),
RepairShares: int16(satellite.Config.Metainfo.RS.RepairThreshold),
OptimalShares: int16(satellite.Config.Metainfo.RS.SuccessThreshold),
TotalShares: int16(satellite.Config.Metainfo.RS.TotalThreshold),
ShareSize: satellite.Config.Metainfo.RS.ErasureShareSize.Int32(),
RequiredShares: int16(rs.Min),
RepairShares: int16(rs.Repair),
OptimalShares: int16(rs.Success),
TotalShares: int16(rs.Total),
ShareSize: rs.ErasureShareSize.Int32(),
}
}

View File

@ -214,7 +214,6 @@ func TestDisqualifiedNodeRemainsDisqualified(t *testing.T) {
_, err = satellitePeer.Overlay.Service.BatchUpdateStats(ctx, []*overlay.UpdateRequest{{
NodeID: disqualifiedNode.ID(),
IsUp: true,
AuditOutcome: overlay.AuditSuccess,
AuditLambda: 0, // forget about history
AuditWeight: 1,

View File

@ -128,7 +128,6 @@ func (reporter *Reporter) recordAuditFailStatus(ctx context.Context, failedAudit
for i, nodeID := range failedAuditNodeIDs {
updateRequests[i] = &overlay.UpdateRequest{
NodeID: nodeID,
IsUp: true,
AuditOutcome: overlay.AuditFailure,
}
}
@ -148,7 +147,6 @@ func (reporter *Reporter) recordAuditUnknownStatus(ctx context.Context, unknownA
for i, nodeID := range unknownAuditNodeIDs {
updateRequests[i] = &overlay.UpdateRequest{
NodeID: nodeID,
IsUp: true,
AuditOutcome: overlay.AuditUnknown,
}
}
@ -169,7 +167,6 @@ func (reporter *Reporter) recordOfflineStatus(ctx context.Context, offlineNodeID
for i, nodeID := range offlineNodeIDs {
updateRequests[i] = &overlay.UpdateRequest{
NodeID: nodeID,
IsUp: false,
AuditOutcome: overlay.AuditOffline,
}
}
@ -191,7 +188,6 @@ func (reporter *Reporter) recordAuditSuccessStatus(ctx context.Context, successN
for i, nodeID := range successNodeIDs {
updateRequests[i] = &overlay.UpdateRequest{
NodeID: nodeID,
IsUp: true,
AuditOutcome: overlay.AuditSuccess,
}
}
@ -221,7 +217,6 @@ func (reporter *Reporter) recordPendingAudits(ctx context.Context, pendingAudits
// record failure -- max reverify count reached
updateRequests = append(updateRequests, &overlay.UpdateRequest{
NodeID: pendingAudit.NodeID,
IsUp: true,
AuditOutcome: overlay.AuditFailure,
})
}

View File

@ -1074,16 +1074,14 @@ func TestReverifySlowDownload(t *testing.T) {
StorageNodeDB: func(index int, db storagenode.DB, log *zap.Logger) (storagenode.DB, error) {
return testblobs.NewSlowDB(log.Named("slowdb"), db), nil
},
Satellite: func(log *zap.Logger, index int, config *satellite.Config) {
// These config values are chosen to force the slow node to time out without timing out on the three normal nodes
config.Audit.MinBytesPerSecond = 100 * memory.KiB
config.Audit.MinDownloadTimeout = 1 * time.Second
config.Metainfo.RS.MinThreshold = 2
config.Metainfo.RS.RepairThreshold = 2
config.Metainfo.RS.SuccessThreshold = 4
config.Metainfo.RS.TotalThreshold = 4
},
Satellite: testplanet.Combine(
func(log *zap.Logger, index int, config *satellite.Config) {
// These config values are chosen to force the slow node to time out without timing out on the three normal nodes
config.Audit.MinBytesPerSecond = 100 * memory.KiB
config.Audit.MinDownloadTimeout = 1 * time.Second
},
testplanet.ReconfigureRS(2, 2, 4, 4),
),
},
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
satellite := planet.Satellites[0]

View File

@ -776,16 +776,14 @@ func TestVerifierSlowDownload(t *testing.T) {
StorageNodeDB: func(index int, db storagenode.DB, log *zap.Logger) (storagenode.DB, error) {
return testblobs.NewSlowDB(log.Named("slowdb"), db), nil
},
Satellite: func(log *zap.Logger, index int, config *satellite.Config) {
// These config values are chosen to force the slow node to time out without timing out on the three normal nodes
config.Audit.MinBytesPerSecond = 100 * memory.KiB
config.Audit.MinDownloadTimeout = 950 * time.Millisecond
config.Metainfo.RS.MinThreshold = 2
config.Metainfo.RS.RepairThreshold = 2
config.Metainfo.RS.SuccessThreshold = 4
config.Metainfo.RS.TotalThreshold = 4
},
Satellite: testplanet.Combine(
func(log *zap.Logger, index int, config *satellite.Config) {
// These config values are chosen to force the slow node to time out without timing out on the three normal nodes
config.Audit.MinBytesPerSecond = 100 * memory.KiB
config.Audit.MinDownloadTimeout = 950 * time.Millisecond
},
testplanet.ReconfigureRS(2, 2, 4, 4),
),
},
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
satellite := planet.Satellites[0]

View File

@ -253,6 +253,29 @@ func (a *Auth) DeleteAccount(w http.ResponseWriter, r *http.Request) {
a.serveJSONError(w, errNotImplemented)
}
// ChangeEmail auth user, changes users email for a new one.
func (a *Auth) ChangeEmail(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
var err error
defer mon.Task()(&ctx)(&err)
var emailChange struct {
NewEmail string `json:"newEmail"`
}
err = json.NewDecoder(r.Body).Decode(&emailChange)
if err != nil {
a.serveJSONError(w, err)
return
}
err = a.service.ChangeEmail(ctx, emailChange.NewEmail)
if err != nil {
a.serveJSONError(w, err)
return
}
}
// ChangePassword auth user, changes users password for a new one.
func (a *Auth) ChangePassword(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()

View File

@ -39,7 +39,6 @@ func TestAuth_Register(t *testing.T) {
},
},
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
for i, test := range []struct {
Partner string
ValidPartner bool
@ -50,52 +49,54 @@ func TestAuth_Register(t *testing.T) {
{Partner: "Raiden nEtwork", ValidPartner: true},
{Partner: "invalid-name", ValidPartner: false},
} {
registerData := struct {
FullName string `json:"fullName"`
ShortName string `json:"shortName"`
Email string `json:"email"`
Partner string `json:"partner"`
PartnerID string `json:"partnerId"`
Password string `json:"password"`
SecretInput string `json:"secret"`
ReferrerUserID string `json:"referrerUserId"`
}{
FullName: "testuser" + strconv.Itoa(i),
ShortName: "test",
Email: "user@test" + strconv.Itoa(i),
Partner: test.Partner,
Password: "abc123",
}
func() {
registerData := struct {
FullName string `json:"fullName"`
ShortName string `json:"shortName"`
Email string `json:"email"`
Partner string `json:"partner"`
PartnerID string `json:"partnerId"`
Password string `json:"password"`
SecretInput string `json:"secret"`
ReferrerUserID string `json:"referrerUserId"`
}{
FullName: "testuser" + strconv.Itoa(i),
ShortName: "test",
Email: "user@test" + strconv.Itoa(i),
Partner: test.Partner,
Password: "abc123",
}
jsonBody, err := json.Marshal(registerData)
require.NoError(t, err)
result, err := http.Post("http://"+planet.Satellites[0].API.Console.Listener.Addr().String()+"/api/v0/auth/register", "application/json", bytes.NewBuffer(jsonBody))
require.NoError(t, err)
require.Equal(t, http.StatusOK, result.StatusCode)
defer func() {
err = result.Body.Close()
jsonBody, err := json.Marshal(registerData)
require.NoError(t, err)
result, err := http.Post("http://"+planet.Satellites[0].API.Console.Listener.Addr().String()+"/api/v0/auth/register", "application/json", bytes.NewBuffer(jsonBody))
require.NoError(t, err)
require.Equal(t, http.StatusOK, result.StatusCode)
defer func() {
err = result.Body.Close()
require.NoError(t, err)
}()
body, err := ioutil.ReadAll(result.Body)
require.NoError(t, err)
var userID uuid.UUID
err = json.Unmarshal(body, &userID)
require.NoError(t, err)
user, err := planet.Satellites[0].API.Console.Service.GetUser(ctx, userID)
require.NoError(t, err)
if test.ValidPartner {
info, err := planet.Satellites[0].API.Marketing.PartnersService.ByName(ctx, test.Partner)
require.NoError(t, err)
require.Equal(t, info.UUID, user.PartnerID)
} else {
require.Equal(t, uuid.UUID{}, user.PartnerID)
}
}()
body, err := ioutil.ReadAll(result.Body)
require.NoError(t, err)
var userID uuid.UUID
err = json.Unmarshal(body, &userID)
require.NoError(t, err)
user, err := planet.Satellites[0].API.Console.Service.GetUser(ctx, userID)
require.NoError(t, err)
if test.ValidPartner {
info, err := planet.Satellites[0].API.Marketing.PartnersService.ByName(ctx, test.Partner)
require.NoError(t, err)
require.Equal(t, info.UUID, user.PartnerID)
} else {
require.Equal(t, uuid.UUID{}, user.PartnerID)
}
}
})
}

View File

@ -173,6 +173,7 @@ func NewServer(logger *zap.Logger, config Config, service *console.Service, mail
authRouter := router.PathPrefix("/api/v0/auth").Subrouter()
authRouter.Handle("/account", server.withAuth(http.HandlerFunc(authController.GetAccount))).Methods(http.MethodGet)
authRouter.Handle("/account", server.withAuth(http.HandlerFunc(authController.UpdateAccount))).Methods(http.MethodPatch)
authRouter.Handle("/account/change-email", server.withAuth(http.HandlerFunc(authController.ChangeEmail))).Methods(http.MethodPost)
authRouter.Handle("/account/change-password", server.withAuth(http.HandlerFunc(authController.ChangePassword))).Methods(http.MethodPost)
authRouter.Handle("/account/delete", server.withAuth(http.HandlerFunc(authController.DeleteAccount))).Methods(http.MethodPost)
authRouter.HandleFunc("/logout", authController.Logout).Methods(http.MethodPost)

View File

@ -9,6 +9,7 @@ import (
"database/sql"
"errors"
"fmt"
"net/mail"
"sort"
"time"
@ -867,6 +868,32 @@ func (s *Service) UpdateAccount(ctx context.Context, fullName string, shortName
return nil
}
// ChangeEmail updates email for a given user.
func (s *Service) ChangeEmail(ctx context.Context, newEmail string) (err error) {
defer mon.Task()(&ctx)(&err)
auth, err := s.getAuthAndAuditLog(ctx, "change email")
if err != nil {
return Error.Wrap(err)
}
if _, err := mail.ParseAddress(newEmail); err != nil {
return ErrValidation.Wrap(err)
}
_, err = s.store.Users().GetByEmail(ctx, newEmail)
if err == nil {
return ErrEmailUsed.New(emailUsedErrMsg)
}
auth.User.Email = newEmail
err = s.store.Users().Update(ctx, &auth.User)
if err != nil {
return Error.Wrap(err)
}
return nil
}
// ChangePassword updates password for a given user.
func (s *Service) ChangePassword(ctx context.Context, pass, newPass string) (err error) {
defer mon.Task()(&ctx)(&err)

View File

@ -130,5 +130,19 @@ func TestService(t *testing.T) {
require.Error(t, err)
require.Equal(t, "service error: project usage error: some buckets still exist", err.Error())
})
t.Run("TestChangeEmail", func(t *testing.T) {
const newEmail = "newEmail@example.com"
err = service.ChangeEmail(authCtx2, newEmail)
require.NoError(t, err)
userWithUpdatedEmail, err := service.GetUserByEmail(authCtx2, newEmail)
require.NoError(t, err)
require.Equal(t, newEmail, userWithUpdatedEmail.Email)
err = service.ChangeEmail(authCtx2, newEmail)
require.Error(t, err)
})
})
}

View File

@ -0,0 +1,35 @@
# Using WebAssembly in Storj
In order to use the uplink library from the browser, we can compile the uplink library to WebAssembly (wasm).
### Setup
To generate wasm code that can create access grants in the web browser, run the following from the storj/wasm directory:
```
$ GOOS=js GOARCH=wasm go build -o access.wasm access.go
```
The `access.wasm` code can then be loaded into the browser in a script tag in an html page. Also needed is a JavaScript support file which ships with golang.
To copy the JavaScript support file, run:
```
$ cp "$(go env GOROOT)/misc/wasm/wasm_exec.js" .
```
Ref: [Golang WebAssembly docs](https://github.com/golang/go/wiki/WebAssembly)
The HTML file should include the following:
```
<script type="text/javascript" src="/path/to/wasm_exec.js"></script>
<script>
const go = new Go();
WebAssembly.instantiateStreaming(
fetch("/path/to/access.wasm"), go.importObject).then(
(result) => {
go.run(result.instance);
});
</script>
```
Additionally, the HTTP `Content-Security-Policy (CSP) script-src` directive will need to be modified to allow wasm code to be executed.
See: [WebAssembly Content Security Policy docs](https://github.com/WebAssembly/content-security-policy/blob/master/proposals/CSP.md)

View File

@ -0,0 +1,64 @@
// +build js,wasm
// Copyright (C) 2020 Storj Labs, Inc.
// See LICENSE for copying information.
package main
import (
"fmt"
"syscall/js"
"storj.io/common/encryption"
"storj.io/common/macaroon"
"storj.io/common/storj"
"storj.io/uplink/private/access2"
)
func main() {
js.Global().Set("generateAccessGrant", generateAccessGrant())
<-make(chan bool)
}
func generateAccessGrant() js.Func {
return js.FuncOf(func(this js.Value, args []js.Value) interface{} {
if len(args) < 4 {
return fmt.Sprintf("Error not enough arguments. Need 4, but only %d supplied. The order of arguments are: satellite Node URL, API key, encryption passphrase, and project salt.", len(args))
}
satelliteNodeURL := args[0].String()
apiKey := args[1].String()
encryptionPassphrase := args[2].String()
projectSalt := args[3].String()
return genAccessGrant(satelliteNodeURL,
apiKey,
encryptionPassphrase,
projectSalt,
)
})
}
func genAccessGrant(satelliteNodeURL, apiKey, encryptionPassphrase, projectSalt string) string {
parsedAPIKey, err := macaroon.ParseAPIKey(apiKey)
if err != nil {
return err.Error()
}
const concurrency = 8
key, err := encryption.DeriveRootKey([]byte(encryptionPassphrase), []byte(projectSalt), "", concurrency)
if err != nil {
return err.Error()
}
encAccess := access2.NewEncryptionAccessWithDefaultKey(key)
encAccess.SetDefaultPathCipher(storj.EncAESGCM)
a := &access2.Access{
SatelliteAddress: satelliteNodeURL,
APIKey: parsedAPIKey,
EncAccess: encAccess,
}
accessString, err := a.Serialize()
if err != nil {
return err.Error()
}
return accessString
}

View File

@ -32,14 +32,12 @@ func TestChore(t *testing.T) {
StorageNodeCount: 8,
UplinkCount: 1,
Reconfigure: testplanet.Reconfigure{
Satellite: func(log *zap.Logger, index int, config *satellite.Config) {
config.GracefulExit.MaxInactiveTimeFrame = maximumInactiveTimeFrame
config.Metainfo.RS.MinThreshold = 4
config.Metainfo.RS.RepairThreshold = 6
config.Metainfo.RS.SuccessThreshold = 8
config.Metainfo.RS.TotalThreshold = 8
},
Satellite: testplanet.Combine(
func(log *zap.Logger, index int, config *satellite.Config) {
config.GracefulExit.MaxInactiveTimeFrame = maximumInactiveTimeFrame
},
testplanet.ReconfigureRS(4, 6, 8, 8),
),
},
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
uplinkPeer := planet.Uplinks[0]
@ -136,14 +134,12 @@ func TestDurabilityRatio(t *testing.T) {
StorageNodeCount: 4,
UplinkCount: 1,
Reconfigure: testplanet.Reconfigure{
Satellite: func(log *zap.Logger, index int, config *satellite.Config) {
config.GracefulExit.MaxInactiveTimeFrame = maximumInactiveTimeFrame
config.Metainfo.RS.MinThreshold = 2
config.Metainfo.RS.RepairThreshold = 3
config.Metainfo.RS.SuccessThreshold = successThreshold
config.Metainfo.RS.TotalThreshold = 4
},
Satellite: testplanet.Combine(
func(log *zap.Logger, index int, config *satellite.Config) {
config.GracefulExit.MaxInactiveTimeFrame = maximumInactiveTimeFrame
},
testplanet.ReconfigureRS(2, 3, successThreshold, 4),
),
},
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
uplinkPeer := planet.Uplinks[0]

View File

@ -255,16 +255,14 @@ func TestRecvTimeout(t *testing.T) {
StorageNodeDB: func(index int, db storagenode.DB, log *zap.Logger) (storagenode.DB, error) {
return testblobs.NewSlowDB(log.Named("slowdb"), db), nil
},
Satellite: func(logger *zap.Logger, index int, config *satellite.Config) {
// This config value will create a very short timeframe allowed for receiving
// data from storage nodes. This will cause context to cancel with timeout.
config.GracefulExit.RecvTimeout = 10 * time.Millisecond
config.Metainfo.RS.MinThreshold = 2
config.Metainfo.RS.RepairThreshold = 3
config.Metainfo.RS.SuccessThreshold = successThreshold
config.Metainfo.RS.TotalThreshold = successThreshold
},
Satellite: testplanet.Combine(
func(log *zap.Logger, index int, config *satellite.Config) {
// This config value will create a very short timeframe allowed for receiving
// data from storage nodes. This will cause context to cancel with timeout.
config.GracefulExit.RecvTimeout = 10 * time.Millisecond
},
testplanet.ReconfigureRS(2, 3, successThreshold, successThreshold),
),
StorageNode: func(index int, config *storagenode.Config) {
config.GracefulExit = gracefulexit.Config{
ChoreInterval: 2 * time.Minute,
@ -1240,17 +1238,15 @@ func TestFailureStorageNodeIgnoresTransferMessages(t *testing.T) {
StorageNodeCount: 5,
UplinkCount: 1,
Reconfigure: testplanet.Reconfigure{
Satellite: func(logger *zap.Logger, index int, config *satellite.Config) {
// We don't care whether a node gracefully exits or not in this test,
// so we set the max failures percentage extra high.
config.GracefulExit.OverallMaxFailuresPercentage = 101
config.GracefulExit.MaxOrderLimitSendCount = maxOrderLimitSendCount
config.Metainfo.RS.MinThreshold = 2
config.Metainfo.RS.RepairThreshold = 3
config.Metainfo.RS.SuccessThreshold = 4
config.Metainfo.RS.TotalThreshold = 4
},
Satellite: testplanet.Combine(
func(log *zap.Logger, index int, config *satellite.Config) {
// We don't care whether a node gracefully exits or not in this test,
// so we set the max failures percentage extra high.
config.GracefulExit.OverallMaxFailuresPercentage = 101
config.GracefulExit.MaxOrderLimitSendCount = maxOrderLimitSendCount
},
testplanet.ReconfigureRS(2, 3, 4, 4),
),
},
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
uplinkPeer := planet.Uplinks[0]
@ -1370,15 +1366,13 @@ func TestIneligibleNodeAge(t *testing.T) {
StorageNodeCount: 5,
UplinkCount: 1,
Reconfigure: testplanet.Reconfigure{
Satellite: func(logger *zap.Logger, index int, config *satellite.Config) {
// Set the required node age to 1 month.
config.GracefulExit.NodeMinAgeInMonths = 1
config.Metainfo.RS.MinThreshold = 2
config.Metainfo.RS.RepairThreshold = 3
config.Metainfo.RS.SuccessThreshold = 4
config.Metainfo.RS.TotalThreshold = 4
},
Satellite: testplanet.Combine(
func(log *zap.Logger, index int, config *satellite.Config) {
// Set the required node age to 1 month.
config.GracefulExit.NodeMinAgeInMonths = 1
},
testplanet.ReconfigureRS(2, 3, 4, 4),
),
},
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
uplinkPeer := planet.Uplinks[0]

View File

@ -5,7 +5,10 @@ package metainfo
import (
"context"
"fmt"
"io"
"strconv"
"strings"
"time"
"go.uber.org/zap"
@ -26,18 +29,73 @@ const (
// RSConfig is a configuration struct that keeps details about default
// redundancy strategy information.
//
// Can be used as a flag.
type RSConfig struct {
MaxBufferMem memory.Size `help:"maximum buffer memory to be allocated for read buffers" default:"4.00MiB"`
ErasureShareSize memory.Size `help:"the size of each new erasure share in bytes" default:"256B"`
MinThreshold int `help:"the minimum pieces required to recover a segment. k." releaseDefault:"29" devDefault:"4"`
RepairThreshold int `help:"the minimum safe pieces before a repair is triggered. m." releaseDefault:"35" devDefault:"6"`
SuccessThreshold int `help:"the desired total pieces for a segment. o." releaseDefault:"80" devDefault:"8"`
TotalThreshold int `help:"the largest amount of pieces to encode to. n." releaseDefault:"110" devDefault:"10"`
ErasureShareSize memory.Size
Min int
Repair int
Success int
Total int
}
// TODO left for validation until we will remove CreateSegmentOld
MinTotalThreshold int `help:"the largest amount of pieces to encode to. n (lower bound for validation)." releaseDefault:"95" devDefault:"10"`
MaxTotalThreshold int `help:"the largest amount of pieces to encode to. n (upper bound for validation)." releaseDefault:"130" devDefault:"10"`
Validate bool `help:"validate redundancy scheme configuration" default:"true"`
// Type implements pflag.Value.
func (RSConfig) Type() string { return "metainfo.RSConfig" }
// String is required for pflag.Value.
func (rs *RSConfig) String() string {
return fmt.Sprintf("%d/%d/%d/%d-%s",
rs.Min,
rs.Repair,
rs.Success,
rs.Total,
rs.ErasureShareSize.String())
}
// Set sets the value from a string in the format k/m/o/n-size (min/repair/optimal/total-erasuresharesize).
func (rs *RSConfig) Set(s string) error {
// Split on dash. Expect two items. First item is RS numbers. Second item is memory.Size.
info := strings.Split(s, "-")
if len(info) != 2 {
return Error.New("Invalid default RS config (expect format k/m/o/n-ShareSize, got %s)", s)
}
rsNumbersString := info[0]
shareSizeString := info[1]
// Attempt to parse "-size" part of config.
shareSizeInt, err := memory.ParseString(shareSizeString)
if err != nil {
return Error.New("Invalid share size in RS config: '%s', %w", shareSizeString, err)
}
shareSize := memory.Size(shareSizeInt)
// Split on forward slash. Expect exactly four positive non-decreasing integers.
rsNumbers := strings.Split(rsNumbersString, "/")
if len(rsNumbers) != 4 {
return Error.New("Invalid default RS numbers (wrong size, expect 4): %s", rsNumbersString)
}
minValue := 1
values := []int{}
for _, nextValueString := range rsNumbers {
nextValue, err := strconv.Atoi(nextValueString)
if err != nil {
return Error.New("Invalid default RS numbers (should all be valid integers): %s, %w", rsNumbersString, err)
}
if nextValue < minValue {
return Error.New("Invalid default RS numbers (should be non-decreasing): %s", rsNumbersString)
}
values = append(values, nextValue)
minValue = nextValue
}
rs.ErasureShareSize = shareSize
rs.Min = values[0]
rs.Repair = values[1]
rs.Success = values[2]
rs.Total = values[3]
return nil
}
// RateLimiterConfig is a configuration struct for endpoint rate limiting.

View File

@ -0,0 +1,98 @@
// Copyright (C) 2020 Storj Labs, Inc.
// See LICENSE for copying information.
package metainfo_test
import (
"testing"
"github.com/stretchr/testify/require"
"storj.io/common/memory"
"storj.io/storj/satellite/metainfo"
)
func TestRSConfigValidation(t *testing.T) {
tests := []struct {
description string
configString string
expectedConfig metainfo.RSConfig
expectError bool
}{
{
description: "valid rs config",
configString: "4/8/10/20-256B",
expectedConfig: metainfo.RSConfig{
ErasureShareSize: 256 * memory.B, Min: 4, Repair: 8, Success: 10, Total: 20,
},
expectError: false,
},
{
description: "invalid rs config - numbers decrease",
configString: "4/8/5/20-256B",
expectError: true,
},
{
description: "invalid rs config - starts at 0",
configString: "0/2/4/6-256B",
expectError: true,
},
{
description: "invalid rs config - strings",
configString: "4/a/b/20-256B",
expectError: true,
},
{
description: "invalid rs config - floating-point numbers",
configString: "4/5.2/7/20-256B",
expectError: true,
},
{
description: "invalid rs config - not enough items",
configString: "4/5/20-256B",
expectError: true,
},
{
description: "invalid rs config - too many items",
configString: "4/5/20/30/50-256B",
expectError: true,
},
{
description: "invalid rs config - empty numbers",
configString: "-256B",
expectError: true,
},
{
description: "invalid rs config - empty size",
configString: "1/2/3/4-",
expectError: true,
},
{
description: "invalid rs config - empty",
configString: "",
expectError: true,
},
{
description: "invalid valid rs config - invalid share size",
configString: "4/8/10/20-256A",
expectError: true,
},
}
for _, tt := range tests {
t.Log(tt.description)
rsConfig := metainfo.RSConfig{}
err := rsConfig.Set(tt.configString)
if tt.expectError {
require.Error(t, err)
} else {
require.NoError(t, err)
require.EqualValues(t, tt.expectedConfig.ErasureShareSize, rsConfig.ErasureShareSize)
require.EqualValues(t, tt.expectedConfig.Min, rsConfig.Min)
require.EqualValues(t, tt.expectedConfig.Repair, rsConfig.Repair)
require.EqualValues(t, tt.expectedConfig.Success, rsConfig.Success)
require.EqualValues(t, tt.expectedConfig.Total, rsConfig.Total)
}
}
}

View File

@ -79,6 +79,7 @@ type Endpoint struct {
limiterCache *lrucache.ExpiringLRU
encInlineSegmentSize int64 // max inline segment size + encryption overhead
revocations revocation.DB
defaultRS *pb.RedundancyScheme
config Config
}
@ -97,6 +98,16 @@ func NewEndpoint(log *zap.Logger, metainfo *Service, deletePieces *piecedeletion
if err != nil {
return nil, err
}
defaultRSScheme := &pb.RedundancyScheme{
Type: pb.RedundancyScheme_RS,
MinReq: int32(config.RS.Min),
RepairThreshold: int32(config.RS.Repair),
SuccessThreshold: int32(config.RS.Success),
Total: int32(config.RS.Total),
ErasureShareSize: config.RS.ErasureShareSize.Int32(),
}
return &Endpoint{
log: log,
metainfo: metainfo,
@ -116,6 +127,7 @@ func NewEndpoint(log *zap.Logger, metainfo *Service, deletePieces *piecedeletion
}),
encInlineSegmentSize: encInlineSegmentSize,
revocations: revocations,
defaultRS: defaultRSScheme,
config: config,
}, nil
}
@ -241,7 +253,7 @@ func (endpoint *Endpoint) GetBucket(ctx context.Context, req *pb.BucketGetReques
}
// override RS to fit satellite settings
convBucket, err := convertBucketToProto(bucket, endpoint.redundancyScheme())
convBucket, err := convertBucketToProto(bucket, endpoint.defaultRS)
if err != nil {
return resp, err
}
@ -316,7 +328,7 @@ func (endpoint *Endpoint) CreateBucket(ctx context.Context, req *pb.BucketCreate
}
// override RS to fit satellite settings
convBucket, err := convertBucketToProto(bucket, endpoint.redundancyScheme())
convBucket, err := convertBucketToProto(bucket, endpoint.defaultRS)
if err != nil {
endpoint.log.Error("error while converting bucket to proto", zap.String("bucketName", bucket.Name), zap.Error(err))
return nil, rpcstatus.Error(rpcstatus.Internal, "unable to create bucket")
@ -375,7 +387,7 @@ func (endpoint *Endpoint) DeleteBucket(ctx context.Context, req *pb.BucketDelete
return nil, err
}
convBucket, err = convertBucketToProto(bucket, endpoint.redundancyScheme())
convBucket, err = convertBucketToProto(bucket, endpoint.defaultRS)
if err != nil {
return nil, err
}
@ -657,7 +669,7 @@ func (endpoint *Endpoint) BeginObject(ctx context.Context, req *pb.ObjectBeginRe
}
// use only satellite values for Redundancy Scheme
pbRS := endpoint.redundancyScheme()
pbRS := endpoint.defaultRS
streamID, err := endpoint.packStreamID(ctx, &internalpb.StreamID{
Bucket: req.Bucket,
@ -1825,17 +1837,6 @@ func groupPiecesByNodeID(segments []metabase.DeletedSegmentInfo) map[storj.NodeI
return piecesToDelete
}
func (endpoint *Endpoint) redundancyScheme() *pb.RedundancyScheme {
return &pb.RedundancyScheme{
Type: pb.RedundancyScheme_RS,
MinReq: int32(endpoint.config.RS.MinThreshold),
RepairThreshold: int32(endpoint.config.RS.RepairThreshold),
SuccessThreshold: int32(endpoint.config.RS.SuccessThreshold),
Total: int32(endpoint.config.RS.TotalThreshold),
ErasureShareSize: endpoint.config.RS.ErasureShareSize.Int32(),
}
}
// RevokeAPIKey handles requests to revoke an api key.
func (endpoint *Endpoint) RevokeAPIKey(ctx context.Context, req *pb.RevokeAPIKeyRequest) (resp *pb.RevokeAPIKeyResponse, err error) {
defer mon.Task()(&ctx)(&err)

View File

@ -215,85 +215,87 @@ func TestInvalidAPIKey(t *testing.T) {
require.NoError(t, err)
for _, invalidAPIKey := range []string{"", "invalid", "testKey"} {
client, err := planet.Uplinks[0].DialMetainfo(ctx, planet.Satellites[0], throwawayKey)
require.NoError(t, err)
defer ctx.Check(client.Close)
func() {
client, err := planet.Uplinks[0].DialMetainfo(ctx, planet.Satellites[0], throwawayKey)
require.NoError(t, err)
defer ctx.Check(client.Close)
client.SetRawAPIKey([]byte(invalidAPIKey))
client.SetRawAPIKey([]byte(invalidAPIKey))
_, err = client.BeginObject(ctx, metainfo.BeginObjectParams{})
assertInvalidArgument(t, err, false)
_, err = client.BeginObject(ctx, metainfo.BeginObjectParams{})
assertInvalidArgument(t, err, false)
_, err = client.BeginDeleteObject(ctx, metainfo.BeginDeleteObjectParams{})
assertInvalidArgument(t, err, false)
_, err = client.BeginDeleteObject(ctx, metainfo.BeginDeleteObjectParams{})
assertInvalidArgument(t, err, false)
_, err = client.ListBuckets(ctx, metainfo.ListBucketsParams{})
assertInvalidArgument(t, err, false)
_, err = client.ListBuckets(ctx, metainfo.ListBucketsParams{})
assertInvalidArgument(t, err, false)
_, _, err = client.ListObjects(ctx, metainfo.ListObjectsParams{})
assertInvalidArgument(t, err, false)
_, _, err = client.ListObjects(ctx, metainfo.ListObjectsParams{})
assertInvalidArgument(t, err, false)
_, err = client.CreateBucket(ctx, metainfo.CreateBucketParams{})
assertInvalidArgument(t, err, false)
_, err = client.CreateBucket(ctx, metainfo.CreateBucketParams{})
assertInvalidArgument(t, err, false)
_, err = client.DeleteBucket(ctx, metainfo.DeleteBucketParams{})
assertInvalidArgument(t, err, false)
_, err = client.DeleteBucket(ctx, metainfo.DeleteBucketParams{})
assertInvalidArgument(t, err, false)
_, err = client.BeginDeleteObject(ctx, metainfo.BeginDeleteObjectParams{})
assertInvalidArgument(t, err, false)
_, err = client.BeginDeleteObject(ctx, metainfo.BeginDeleteObjectParams{})
assertInvalidArgument(t, err, false)
_, err = client.GetBucket(ctx, metainfo.GetBucketParams{})
assertInvalidArgument(t, err, false)
_, err = client.GetBucket(ctx, metainfo.GetBucketParams{})
assertInvalidArgument(t, err, false)
_, err = client.GetObject(ctx, metainfo.GetObjectParams{})
assertInvalidArgument(t, err, false)
_, err = client.GetObject(ctx, metainfo.GetObjectParams{})
assertInvalidArgument(t, err, false)
_, err = client.GetProjectInfo(ctx)
assertInvalidArgument(t, err, false)
_, err = client.GetProjectInfo(ctx)
assertInvalidArgument(t, err, false)
// these methods needs StreamID to do authentication
// these methods needs StreamID to do authentication
signer := signing.SignerFromFullIdentity(planet.Satellites[0].Identity)
satStreamID := &internalpb.StreamID{
CreationDate: time.Now(),
}
signedStreamID, err := satMetainfo.SignStreamID(ctx, signer, satStreamID)
require.NoError(t, err)
signer := signing.SignerFromFullIdentity(planet.Satellites[0].Identity)
satStreamID := &internalpb.StreamID{
CreationDate: time.Now(),
}
signedStreamID, err := satMetainfo.SignStreamID(ctx, signer, satStreamID)
require.NoError(t, err)
encodedStreamID, err := pb.Marshal(signedStreamID)
require.NoError(t, err)
encodedStreamID, err := pb.Marshal(signedStreamID)
require.NoError(t, err)
streamID, err := storj.StreamIDFromBytes(encodedStreamID)
require.NoError(t, err)
streamID, err := storj.StreamIDFromBytes(encodedStreamID)
require.NoError(t, err)
err = client.CommitObject(ctx, metainfo.CommitObjectParams{StreamID: streamID})
assertInvalidArgument(t, err, false)
err = client.CommitObject(ctx, metainfo.CommitObjectParams{StreamID: streamID})
assertInvalidArgument(t, err, false)
_, _, _, err = client.BeginSegment(ctx, metainfo.BeginSegmentParams{StreamID: streamID})
assertInvalidArgument(t, err, false)
_, _, _, err = client.BeginSegment(ctx, metainfo.BeginSegmentParams{StreamID: streamID})
assertInvalidArgument(t, err, false)
err = client.MakeInlineSegment(ctx, metainfo.MakeInlineSegmentParams{StreamID: streamID})
assertInvalidArgument(t, err, false)
err = client.MakeInlineSegment(ctx, metainfo.MakeInlineSegmentParams{StreamID: streamID})
assertInvalidArgument(t, err, false)
_, _, err = client.DownloadSegment(ctx, metainfo.DownloadSegmentParams{StreamID: streamID})
assertInvalidArgument(t, err, false)
_, _, err = client.DownloadSegment(ctx, metainfo.DownloadSegmentParams{StreamID: streamID})
assertInvalidArgument(t, err, false)
// these methods needs SegmentID
// these methods needs SegmentID
signedSegmentID, err := satMetainfo.SignSegmentID(ctx, signer, &internalpb.SegmentID{
StreamId: satStreamID,
CreationDate: time.Now(),
})
require.NoError(t, err)
signedSegmentID, err := satMetainfo.SignSegmentID(ctx, signer, &internalpb.SegmentID{
StreamId: satStreamID,
CreationDate: time.Now(),
})
require.NoError(t, err)
encodedSegmentID, err := pb.Marshal(signedSegmentID)
require.NoError(t, err)
encodedSegmentID, err := pb.Marshal(signedSegmentID)
require.NoError(t, err)
segmentID, err := storj.SegmentIDFromBytes(encodedSegmentID)
require.NoError(t, err)
segmentID, err := storj.SegmentIDFromBytes(encodedSegmentID)
require.NoError(t, err)
err = client.CommitSegment(ctx, metainfo.CommitSegmentParams{SegmentID: segmentID})
assertInvalidArgument(t, err, false)
err = client.CommitSegment(ctx, metainfo.CommitSegmentParams{SegmentID: segmentID})
assertInvalidArgument(t, err, false)
}()
}
})
}

View File

@ -346,14 +346,13 @@ func TestService_DeletePieces_Timeout(t *testing.T) {
StorageNodeDB: func(index int, db storagenode.DB, log *zap.Logger) (storagenode.DB, error) {
return testblobs.NewSlowDB(log.Named("slowdb"), db), nil
},
Satellite: func(log *zap.Logger, index int, config *satellite.Config) {
config.Metainfo.PieceDeletion.RequestTimeout = 200 * time.Millisecond
config.Metainfo.RS.MinThreshold = 2
config.Metainfo.RS.RepairThreshold = 2
config.Metainfo.RS.SuccessThreshold = 4
config.Metainfo.RS.TotalThreshold = 4
config.Metainfo.MaxSegmentSize = 15 * memory.KiB
},
Satellite: testplanet.Combine(
func(log *zap.Logger, index int, config *satellite.Config) {
config.Metainfo.PieceDeletion.RequestTimeout = 200 * time.Millisecond
config.Metainfo.MaxSegmentSize = 15 * memory.KiB
},
testplanet.ReconfigureRS(2, 2, 4, 4),
),
},
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
uplnk := planet.Uplinks[0]

View File

@ -76,106 +76,108 @@ func TestSettlementWithWindowEndpointManyOrders(t *testing.T) {
}
for _, tt := range testCases {
// create serial number to use in test. must be unique for each run.
serialNumber1 := testrand.SerialNumber()
err = ordersDB.CreateSerialInfo(ctx, serialNumber1, []byte(bucketID), now.AddDate(1, 0, 10))
require.NoError(t, err)
func() {
// create serial number to use in test. must be unique for each run.
serialNumber1 := testrand.SerialNumber()
err = ordersDB.CreateSerialInfo(ctx, serialNumber1, []byte(bucketID), now.AddDate(1, 0, 10))
require.NoError(t, err)
serialNumber2 := testrand.SerialNumber()
err = ordersDB.CreateSerialInfo(ctx, serialNumber2, []byte(bucketID), now.AddDate(1, 0, 10))
require.NoError(t, err)
serialNumber2 := testrand.SerialNumber()
err = ordersDB.CreateSerialInfo(ctx, serialNumber2, []byte(bucketID), now.AddDate(1, 0, 10))
require.NoError(t, err)
piecePublicKey, piecePrivateKey, err := storj.NewPieceKey()
require.NoError(t, err)
piecePublicKey, piecePrivateKey, err := storj.NewPieceKey()
require.NoError(t, err)
// create signed orderlimit or order to test with
limit1 := &pb.OrderLimit{
SerialNumber: serialNumber1,
SatelliteId: satellite.ID(),
UplinkPublicKey: piecePublicKey,
StorageNodeId: storagenode.ID(),
PieceId: storj.NewPieceID(),
Action: pb.PieceAction_PUT,
Limit: 1000,
PieceExpiration: time.Time{},
OrderCreation: tt.orderCreation,
OrderExpiration: now.Add(24 * time.Hour),
}
orderLimit1, err := signing.SignOrderLimit(ctx, signing.SignerFromFullIdentity(satellite.Identity), limit1)
require.NoError(t, err)
// create signed orderlimit or order to test with
limit1 := &pb.OrderLimit{
SerialNumber: serialNumber1,
SatelliteId: satellite.ID(),
UplinkPublicKey: piecePublicKey,
StorageNodeId: storagenode.ID(),
PieceId: storj.NewPieceID(),
Action: pb.PieceAction_PUT,
Limit: 1000,
PieceExpiration: time.Time{},
OrderCreation: tt.orderCreation,
OrderExpiration: now.Add(24 * time.Hour),
}
orderLimit1, err := signing.SignOrderLimit(ctx, signing.SignerFromFullIdentity(satellite.Identity), limit1)
require.NoError(t, err)
order1, err := signing.SignUplinkOrder(ctx, piecePrivateKey, &pb.Order{
SerialNumber: serialNumber1,
Amount: tt.dataAmount,
})
require.NoError(t, err)
order1, err := signing.SignUplinkOrder(ctx, piecePrivateKey, &pb.Order{
SerialNumber: serialNumber1,
Amount: tt.dataAmount,
})
require.NoError(t, err)
limit2 := &pb.OrderLimit{
SerialNumber: serialNumber2,
SatelliteId: satellite.ID(),
UplinkPublicKey: piecePublicKey,
StorageNodeId: storagenode.ID(),
PieceId: storj.NewPieceID(),
Action: pb.PieceAction_PUT,
Limit: 1000,
PieceExpiration: time.Time{},
OrderCreation: now,
OrderExpiration: now.Add(24 * time.Hour),
}
orderLimit2, err := signing.SignOrderLimit(ctx, signing.SignerFromFullIdentity(satellite.Identity), limit2)
require.NoError(t, err)
limit2 := &pb.OrderLimit{
SerialNumber: serialNumber2,
SatelliteId: satellite.ID(),
UplinkPublicKey: piecePublicKey,
StorageNodeId: storagenode.ID(),
PieceId: storj.NewPieceID(),
Action: pb.PieceAction_PUT,
Limit: 1000,
PieceExpiration: time.Time{},
OrderCreation: now,
OrderExpiration: now.Add(24 * time.Hour),
}
orderLimit2, err := signing.SignOrderLimit(ctx, signing.SignerFromFullIdentity(satellite.Identity), limit2)
require.NoError(t, err)
order2, err := signing.SignUplinkOrder(ctx, piecePrivateKey, &pb.Order{
SerialNumber: serialNumber2,
Amount: tt.dataAmount,
})
require.NoError(t, err)
order2, err := signing.SignUplinkOrder(ctx, piecePrivateKey, &pb.Order{
SerialNumber: serialNumber2,
Amount: tt.dataAmount,
})
require.NoError(t, err)
// create connection between storagenode and satellite
conn, err := storagenode.Dialer.DialNodeURL(ctx, storj.NodeURL{ID: satellite.ID(), Address: satellite.Addr()})
require.NoError(t, err)
defer ctx.Check(conn.Close)
// create connection between storagenode and satellite
conn, err := storagenode.Dialer.DialNodeURL(ctx, storj.NodeURL{ID: satellite.ID(), Address: satellite.Addr()})
require.NoError(t, err)
defer ctx.Check(conn.Close)
stream, err := pb.NewDRPCOrdersClient(conn).SettlementWithWindow(ctx)
require.NoError(t, err)
defer ctx.Check(stream.Close)
stream, err := pb.NewDRPCOrdersClient(conn).SettlementWithWindow(ctx)
require.NoError(t, err)
defer ctx.Check(stream.Close)
// storagenode settles an order and orderlimit
err = stream.Send(&pb.SettlementRequest{
Limit: orderLimit1,
Order: order1,
})
require.NoError(t, err)
err = stream.Send(&pb.SettlementRequest{
Limit: orderLimit2,
Order: order2,
})
require.NoError(t, err)
resp, err := stream.CloseAndRecv()
require.NoError(t, err)
// storagenode settles an order and orderlimit
err = stream.Send(&pb.SettlementRequest{
Limit: orderLimit1,
Order: order1,
})
require.NoError(t, err)
err = stream.Send(&pb.SettlementRequest{
Limit: orderLimit2,
Order: order2,
})
require.NoError(t, err)
resp, err := stream.CloseAndRecv()
require.NoError(t, err)
// the settled amount is only returned during phase3
var settled map[int32]int64
if satellite.Config.Orders.WindowEndpointRolloutPhase == orders.WindowEndpointRolloutPhase3 {
settled = map[int32]int64{int32(pb.PieceAction_PUT): tt.settledAmt}
}
require.Equal(t, &pb.SettlementWithWindowResponse{
Status: pb.SettlementWithWindowResponse_ACCEPTED,
ActionSettled: settled,
}, resp)
// the settled amount is only returned during phase3
var settled map[int32]int64
if satellite.Config.Orders.WindowEndpointRolloutPhase == orders.WindowEndpointRolloutPhase3 {
settled = map[int32]int64{int32(pb.PieceAction_PUT): tt.settledAmt}
}
require.Equal(t, &pb.SettlementWithWindowResponse{
Status: pb.SettlementWithWindowResponse_ACCEPTED,
ActionSettled: settled,
}, resp)
// trigger and wait for all of the chores necessary to flush the orders
assert.NoError(t, satellite.Accounting.ReportedRollup.RunOnce(ctx, tt.orderCreation))
satellite.Orders.Chore.Loop.TriggerWait()
// trigger and wait for all of the chores necessary to flush the orders
assert.NoError(t, satellite.Accounting.ReportedRollup.RunOnce(ctx, tt.orderCreation))
satellite.Orders.Chore.Loop.TriggerWait()
// assert all the right stuff is in the satellite storagenode and bucket bandwidth tables
snbw, err = ordersDB.GetStorageNodeBandwidth(ctx, storagenode.ID(), time.Time{}, tt.orderCreation)
require.NoError(t, err)
require.EqualValues(t, tt.settledAmt, snbw)
// assert all the right stuff is in the satellite storagenode and bucket bandwidth tables
snbw, err = ordersDB.GetStorageNodeBandwidth(ctx, storagenode.ID(), time.Time{}, tt.orderCreation)
require.NoError(t, err)
require.EqualValues(t, tt.settledAmt, snbw)
newBbw, err := ordersDB.GetBucketBandwidth(ctx, projectID, []byte(bucketname), time.Time{}, tt.orderCreation)
require.NoError(t, err)
require.EqualValues(t, tt.settledAmt, newBbw)
newBbw, err := ordersDB.GetBucketBandwidth(ctx, projectID, []byte(bucketname), time.Time{}, tt.orderCreation)
require.NoError(t, err)
require.EqualValues(t, tt.settledAmt, newBbw)
}()
}
})
}
@ -223,72 +225,74 @@ func TestSettlementWithWindowEndpointSingleOrder(t *testing.T) {
}
for _, tt := range testCases {
// create signed orderlimit or order to test with
limit := &pb.OrderLimit{
SerialNumber: serialNumber,
SatelliteId: satellite.ID(),
UplinkPublicKey: piecePublicKey,
StorageNodeId: storagenode.ID(),
PieceId: storj.NewPieceID(),
Action: pb.PieceAction_PUT,
Limit: 1000,
PieceExpiration: time.Time{},
OrderCreation: now,
OrderExpiration: now.Add(24 * time.Hour),
}
orderLimit, err := signing.SignOrderLimit(ctx, signing.SignerFromFullIdentity(satellite.Identity), limit)
require.NoError(t, err)
func() {
// create signed orderlimit or order to test with
limit := &pb.OrderLimit{
SerialNumber: serialNumber,
SatelliteId: satellite.ID(),
UplinkPublicKey: piecePublicKey,
StorageNodeId: storagenode.ID(),
PieceId: storj.NewPieceID(),
Action: pb.PieceAction_PUT,
Limit: 1000,
PieceExpiration: time.Time{},
OrderCreation: now,
OrderExpiration: now.Add(24 * time.Hour),
}
orderLimit, err := signing.SignOrderLimit(ctx, signing.SignerFromFullIdentity(satellite.Identity), limit)
require.NoError(t, err)
order, err := signing.SignUplinkOrder(ctx, piecePrivateKey, &pb.Order{
SerialNumber: serialNumber,
Amount: tt.dataAmount,
})
require.NoError(t, err)
order, err := signing.SignUplinkOrder(ctx, piecePrivateKey, &pb.Order{
SerialNumber: serialNumber,
Amount: tt.dataAmount,
})
require.NoError(t, err)
// create connection between storagenode and satellite
conn, err := storagenode.Dialer.DialNodeURL(ctx, storj.NodeURL{ID: satellite.ID(), Address: satellite.Addr()})
require.NoError(t, err)
defer ctx.Check(conn.Close)
// create connection between storagenode and satellite
conn, err := storagenode.Dialer.DialNodeURL(ctx, storj.NodeURL{ID: satellite.ID(), Address: satellite.Addr()})
require.NoError(t, err)
defer ctx.Check(conn.Close)
stream, err := pb.NewDRPCOrdersClient(conn).SettlementWithWindow(ctx)
require.NoError(t, err)
defer ctx.Check(stream.Close)
stream, err := pb.NewDRPCOrdersClient(conn).SettlementWithWindow(ctx)
require.NoError(t, err)
defer ctx.Check(stream.Close)
// storagenode settles an order and orderlimit
err = stream.Send(&pb.SettlementRequest{
Limit: orderLimit,
Order: order,
})
require.NoError(t, err)
resp, err := stream.CloseAndRecv()
require.NoError(t, err)
// storagenode settles an order and orderlimit
err = stream.Send(&pb.SettlementRequest{
Limit: orderLimit,
Order: order,
})
require.NoError(t, err)
resp, err := stream.CloseAndRecv()
require.NoError(t, err)
expected := new(pb.SettlementWithWindowResponse)
switch {
case satellite.Config.Orders.WindowEndpointRolloutPhase != orders.WindowEndpointRolloutPhase3:
expected.Status = pb.SettlementWithWindowResponse_ACCEPTED
expected.ActionSettled = nil
case tt.expectedStatus == pb.SettlementWithWindowResponse_ACCEPTED:
expected.Status = pb.SettlementWithWindowResponse_ACCEPTED
expected.ActionSettled = map[int32]int64{int32(pb.PieceAction_PUT): tt.dataAmount}
default:
expected.Status = pb.SettlementWithWindowResponse_REJECTED
expected.ActionSettled = nil
}
require.Equal(t, expected, resp)
expected := new(pb.SettlementWithWindowResponse)
switch {
case satellite.Config.Orders.WindowEndpointRolloutPhase != orders.WindowEndpointRolloutPhase3:
expected.Status = pb.SettlementWithWindowResponse_ACCEPTED
expected.ActionSettled = nil
case tt.expectedStatus == pb.SettlementWithWindowResponse_ACCEPTED:
expected.Status = pb.SettlementWithWindowResponse_ACCEPTED
expected.ActionSettled = map[int32]int64{int32(pb.PieceAction_PUT): tt.dataAmount}
default:
expected.Status = pb.SettlementWithWindowResponse_REJECTED
expected.ActionSettled = nil
}
require.Equal(t, expected, resp)
// flush all the chores
assert.NoError(t, satellite.Accounting.ReportedRollup.RunOnce(ctx, now))
satellite.Orders.Chore.Loop.TriggerWait()
// flush all the chores
assert.NoError(t, satellite.Accounting.ReportedRollup.RunOnce(ctx, now))
satellite.Orders.Chore.Loop.TriggerWait()
// assert all the right stuff is in the satellite storagenode and bucket bandwidth tables
snbw, err = ordersDB.GetStorageNodeBandwidth(ctx, storagenode.ID(), time.Time{}, now)
require.NoError(t, err)
require.Equal(t, dataAmount, snbw)
// assert all the right stuff is in the satellite storagenode and bucket bandwidth tables
snbw, err = ordersDB.GetStorageNodeBandwidth(ctx, storagenode.ID(), time.Time{}, now)
require.NoError(t, err)
require.Equal(t, dataAmount, snbw)
newBbw, err := ordersDB.GetBucketBandwidth(ctx, projectID, []byte(bucketname), time.Time{}, now)
require.NoError(t, err)
require.Equal(t, dataAmount, newBbw)
newBbw, err := ordersDB.GetBucketBandwidth(ctx, projectID, []byte(bucketname), time.Time{}, now)
require.NoError(t, err)
require.Equal(t, dataAmount, newBbw)
}()
}
})
}

View File

@ -566,7 +566,11 @@ func (service *Service) CreateGracefulExitPutOrderLimit(ctx context.Context, buc
return nil, storj.PiecePrivateKey{}, Error.Wrap(err)
}
nodeURL := storj.NodeURL{ID: nodeID, Address: node.Address.Address}
address := node.Address.Address
if node.LastIPPort != "" {
address = node.LastIPPort
}
nodeURL := storj.NodeURL{ID: nodeID, Address: address}
limit, err = signer.Sign(ctx, nodeURL, pieceNum)
if err != nil {
return nil, storj.PiecePrivateKey{}, Error.Wrap(err)

View File

@ -89,35 +89,106 @@ func BenchmarkOverlay(b *testing.B) {
}
})
b.Run("UpdateStats", func(b *testing.B) {
b.Run("UpdateStatsSuccess", func(b *testing.B) {
for i := 0; i < b.N; i++ {
id := all[i%len(all)]
outcome := overlay.AuditFailure
if i&1 == 0 {
outcome = overlay.AuditSuccess
}
_, err := overlaydb.UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: id,
AuditOutcome: outcome,
IsUp: i&2 == 0,
AuditOutcome: overlay.AuditSuccess,
AuditHistory: testAuditHistoryConfig(),
}, time.Now())
require.NoError(b, err)
}
})
b.Run("BatchUpdateStats", func(b *testing.B) {
b.Run("UpdateStatsFailure", func(b *testing.B) {
for i := 0; i < b.N; i++ {
id := all[i%len(all)]
_, err := overlaydb.UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: id,
AuditOutcome: overlay.AuditFailure,
AuditHistory: testAuditHistoryConfig(),
}, time.Now())
require.NoError(b, err)
}
})
b.Run("UpdateStatsUnknown", func(b *testing.B) {
for i := 0; i < b.N; i++ {
id := all[i%len(all)]
_, err := overlaydb.UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: id,
AuditOutcome: overlay.AuditUnknown,
AuditHistory: testAuditHistoryConfig(),
}, time.Now())
require.NoError(b, err)
}
})
b.Run("UpdateStatsOffline", func(b *testing.B) {
for i := 0; i < b.N; i++ {
id := all[i%len(all)]
_, err := overlaydb.UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: id,
AuditOutcome: overlay.AuditOffline,
AuditHistory: testAuditHistoryConfig(),
}, time.Now())
require.NoError(b, err)
}
})
b.Run("BatchUpdateStatsSuccess", func(b *testing.B) {
var updateRequests []*overlay.UpdateRequest
for i := 0; i < b.N; i++ {
id := all[i%len(all)]
outcome := overlay.AuditFailure
if i&1 == 0 {
outcome = overlay.AuditSuccess
}
updateRequests = append(updateRequests, &overlay.UpdateRequest{
NodeID: id,
AuditOutcome: outcome,
IsUp: i&2 == 0,
AuditOutcome: overlay.AuditSuccess,
AuditHistory: testAuditHistoryConfig(),
})
}
_, err := overlaydb.BatchUpdateStats(ctx, updateRequests, 100, time.Now())
require.NoError(b, err)
})
b.Run("BatchUpdateStatsFailure", func(b *testing.B) {
var updateRequests []*overlay.UpdateRequest
for i := 0; i < b.N; i++ {
id := all[i%len(all)]
updateRequests = append(updateRequests, &overlay.UpdateRequest{
NodeID: id,
AuditOutcome: overlay.AuditFailure,
AuditHistory: testAuditHistoryConfig(),
})
}
_, err := overlaydb.BatchUpdateStats(ctx, updateRequests, 100, time.Now())
require.NoError(b, err)
})
b.Run("BatchUpdateStatsUnknown", func(b *testing.B) {
var updateRequests []*overlay.UpdateRequest
for i := 0; i < b.N; i++ {
id := all[i%len(all)]
updateRequests = append(updateRequests, &overlay.UpdateRequest{
NodeID: id,
AuditOutcome: overlay.AuditUnknown,
AuditHistory: testAuditHistoryConfig(),
})
}
_, err := overlaydb.BatchUpdateStats(ctx, updateRequests, 100, time.Now())
require.NoError(b, err)
})
b.Run("BatchUpdateStatsOffline", func(b *testing.B) {
var updateRequests []*overlay.UpdateRequest
for i := 0; i < b.N; i++ {
id := all[i%len(all)]
updateRequests = append(updateRequests, &overlay.UpdateRequest{
NodeID: id,
AuditOutcome: overlay.AuditOffline,
AuditHistory: testAuditHistoryConfig(),
})
@ -272,7 +343,6 @@ func BenchmarkNodeSelection(b *testing.B) {
if i%2 == 0 { // make half of nodes "new" and half "vetted"
_, err = overlaydb.UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: nodeID,
IsUp: true,
AuditOutcome: overlay.AuditSuccess,
AuditLambda: 1,
AuditWeight: 1,
@ -293,7 +363,6 @@ func BenchmarkNodeSelection(b *testing.B) {
case 2:
err := overlaydb.UpdateCheckIn(ctx, overlay.NodeCheckInInfo{
NodeID: nodeID,
IsUp: true,
Address: &pb.NodeAddress{
Address: address,
},

View File

@ -104,7 +104,6 @@ func addNodesToNodesTable(ctx context.Context, t *testing.T, db overlay.DB, coun
if i < makeReputable {
stats, err := db.UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: storj.NodeID{byte(i)},
IsUp: true,
AuditOutcome: overlay.AuditSuccess,
AuditLambda: 1, AuditWeight: 1, AuditDQ: 0.5,
AuditHistory: testAuditHistoryConfig(),

View File

@ -188,7 +188,6 @@ func TestEnsureMinimumRequested(t *testing.T) {
reputable[node.ID()] = true
_, err := satellite.DB.OverlayCache().UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: node.ID(),
IsUp: true,
AuditOutcome: overlay.AuditSuccess,
AuditLambda: 1, AuditWeight: 1, AuditDQ: 0.5,
AuditHistory: testAuditHistoryConfig(),
@ -232,7 +231,6 @@ func TestEnsureMinimumRequested(t *testing.T) {
reputable[node.ID()] = true
_, err := satellite.DB.OverlayCache().UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: node.ID(),
IsUp: true,
AuditOutcome: overlay.AuditSuccess,
AuditLambda: 1, AuditWeight: 1, AuditDQ: 0.5,
AuditHistory: testAuditHistoryConfig(),
@ -397,7 +395,6 @@ func TestNodeSelectionGracefulExit(t *testing.T) {
for k := 0; k < i; k++ {
_, err := satellite.DB.OverlayCache().UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: node.ID(),
IsUp: true,
AuditOutcome: overlay.AuditSuccess,
AuditLambda: 1, AuditWeight: 1, AuditDQ: 0.5,
AuditHistory: testAuditHistoryConfig(),
@ -628,7 +625,6 @@ func TestDistinctIPs(t *testing.T) {
for i := 9; i > 7; i-- {
_, err := satellite.DB.OverlayCache().UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: planet.StorageNodes[i].ID(),
IsUp: true,
AuditOutcome: overlay.AuditSuccess,
AuditLambda: 1,
AuditWeight: 1,
@ -660,7 +656,6 @@ func TestDistinctIPsWithBatch(t *testing.T) {
// These are done individually b/c the previous stat data is important
_, err := satellite.Overlay.Service.BatchUpdateStats(ctx, []*overlay.UpdateRequest{{
NodeID: planet.StorageNodes[i].ID(),
IsUp: true,
AuditOutcome: overlay.AuditSuccess,
AuditLambda: 1,
AuditWeight: 1,

View File

@ -164,7 +164,6 @@ const (
type UpdateRequest struct {
NodeID storj.NodeID
AuditOutcome AuditType
IsUp bool
// n.b. these are set values from the satellite.
// They are part of the UpdateRequest struct in order to be
// more easily accessible in satellite/satellitedb/overlaycache.go.

View File

@ -131,7 +131,6 @@ func testCache(ctx context.Context, t *testing.T, store overlay.DB) {
stats, err := service.UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: valid1ID,
IsUp: true,
AuditOutcome: overlay.AuditFailure,
})
require.NoError(t, err)
@ -148,7 +147,6 @@ func testCache(ctx context.Context, t *testing.T, store overlay.DB) {
// should not update once already disqualified
_, err = service.BatchUpdateStats(ctx, []*overlay.UpdateRequest{{
NodeID: valid2ID,
IsUp: false,
AuditOutcome: overlay.AuditSuccess,
}})
require.NoError(t, err)
@ -197,7 +195,6 @@ func TestRandomizedSelection(t *testing.T) {
if i%2 == 0 { // make half of nodes "new" and half "vetted"
_, err = cache.UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: newID,
IsUp: true,
AuditOutcome: overlay.AuditSuccess,
AuditLambda: 1,
AuditWeight: 1,
@ -316,7 +313,6 @@ func TestRandomizedSelectionCache(t *testing.T) {
if i%2 == 0 { // make half of nodes "new" and half "vetted"
_, err = overlaydb.UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: newID,
IsUp: true,
AuditOutcome: overlay.AuditSuccess,
AuditLambda: 1,
AuditWeight: 1,
@ -775,7 +771,6 @@ func TestSuspendedSelection(t *testing.T) {
if i%2 == 0 { // make half of nodes "new" and half "vetted"
_, err = cache.UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: newID,
IsUp: true,
AuditOutcome: overlay.AuditSuccess,
AuditLambda: 1,
AuditWeight: 1,
@ -835,7 +830,6 @@ func TestConcurrentAudit(t *testing.T) {
_, err := planet.Satellites[0].Overlay.Service.UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: planet.StorageNodes[0].ID(),
AuditOutcome: overlay.AuditSuccess,
IsUp: true,
})
return err
})
@ -853,7 +847,6 @@ func TestConcurrentAudit(t *testing.T) {
{
NodeID: planet.StorageNodes[0].ID(),
AuditOutcome: overlay.AuditSuccess,
IsUp: true,
},
})
return err

View File

@ -169,7 +169,6 @@ func testDatabase(ctx context.Context, t *testing.T, cache overlay.DB) {
updateReq := &overlay.UpdateRequest{
NodeID: nodeID,
AuditOutcome: overlay.AuditSuccess,
IsUp: true,
AuditLambda: 0.123, AuditWeight: 0.456,
AuditDQ: 0, // don't disqualify for any reason
AuditHistory: testAuditHistoryConfig(),
@ -186,7 +185,6 @@ func testDatabase(ctx context.Context, t *testing.T, cache overlay.DB) {
auditBeta = expectedAuditBeta
updateReq.AuditOutcome = overlay.AuditFailure
updateReq.IsUp = false
stats, err = cache.UpdateStats(ctx, updateReq, time.Now())
require.NoError(t, err)

View File

@ -66,7 +66,6 @@ func TestAuditSuspendWithUpdateStats(t *testing.T) {
_, err = oc.UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: nodeID,
AuditOutcome: overlay.AuditUnknown,
IsUp: true,
AuditLambda: 1,
AuditWeight: 1,
AuditDQ: 0.6,
@ -90,7 +89,6 @@ func TestAuditSuspendWithUpdateStats(t *testing.T) {
_, err = oc.UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: nodeID,
AuditOutcome: overlay.AuditSuccess,
IsUp: true,
AuditLambda: 1,
AuditWeight: 1,
AuditDQ: 0.6,
@ -121,7 +119,6 @@ func TestAuditSuspendFailedAudit(t *testing.T) {
_, err = oc.UpdateStats(ctx, &overlay.UpdateRequest{
NodeID: nodeID,
AuditOutcome: overlay.AuditFailure,
IsUp: true,
AuditLambda: 1,
AuditWeight: 1,
AuditDQ: 0.6,
@ -277,7 +274,6 @@ func TestAuditSuspendBatchUpdateStats(t *testing.T) {
nodeUpdateReq := &overlay.UpdateRequest{
NodeID: nodeID,
AuditOutcome: overlay.AuditSuccess,
IsUp: true,
AuditLambda: 1,
AuditWeight: 1,
AuditDQ: 0.6,
@ -343,7 +339,6 @@ func TestOfflineSuspend(t *testing.T) {
updateReq := &overlay.UpdateRequest{
NodeID: nodeID,
AuditOutcome: overlay.AuditOffline,
IsUp: false,
AuditHistory: overlay.AuditHistoryConfig{
WindowSize: time.Hour,
TrackingPeriod: 2 * time.Hour,
@ -472,12 +467,11 @@ func setOnlineScore(ctx context.Context, reqPtr *overlay.UpdateRequest, desiredS
for window := 0; window < windowsPerTrackingPeriod+1; window++ {
updateReqs := []*overlay.UpdateRequest{}
for i := 0; i < totalAudits; i++ {
isUp := true
if i >= onlineAudits {
isUp = false
}
updateReq := *reqPtr
updateReq.IsUp = isUp
updateReq.AuditOutcome = overlay.AuditSuccess
if i >= onlineAudits {
updateReq.AuditOutcome = overlay.AuditOffline
}
updateReq.AuditHistory.GracePeriod = gracePeriod
updateReqs = append(updateReqs, &updateReq)

View File

@ -391,7 +391,7 @@ func (obs *checkerObserver) InlineSegment(ctx context.Context, segment *metainfo
func (checker *Checker) IrreparableProcess(ctx context.Context) (err error) {
defer mon.Task()(&ctx)(&err)
const limit = 1000
var lastSeenSegmentKey metabase.SegmentKey
lastSeenSegmentKey := metabase.SegmentKey{}
for {
segments, err := checker.irrdb.GetLimited(ctx, limit, lastSeenSegmentKey)

View File

@ -13,8 +13,10 @@ import (
"storj.io/common/pb"
"storj.io/common/testcontext"
"storj.io/storj/private/testplanet"
"storj.io/storj/satellite"
"storj.io/storj/satellite/internalpb"
"storj.io/storj/satellite/metainfo/metabase"
"storj.io/storj/satellite/satellitedb/satellitedbtest"
)
@ -103,3 +105,88 @@ func TestIrreparable(t *testing.T) {
}
})
}
func TestIrreparableProcess(t *testing.T) {
testplanet.Run(t, testplanet.Config{
SatelliteCount: 1, StorageNodeCount: 3, UplinkCount: 0,
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
checker := planet.Satellites[0].Repair.Checker
checker.Loop.Stop()
checker.IrreparableLoop.Stop()
irreparabledb := planet.Satellites[0].DB.Irreparable()
queue := planet.Satellites[0].DB.RepairQueue()
seg := &internalpb.IrreparableSegment{
Path: []byte{1},
SegmentDetail: &pb.Pointer{
Type: pb.Pointer_REMOTE,
CreationDate: time.Now(),
Remote: &pb.RemoteSegment{
Redundancy: &pb.RedundancyScheme{
MinReq: 1,
RepairThreshold: 2,
SuccessThreshold: 3,
Total: 4,
},
RemotePieces: []*pb.RemotePiece{
{
NodeId: planet.StorageNodes[0].ID(),
},
{
NodeId: planet.StorageNodes[1].ID(),
},
{
NodeId: planet.StorageNodes[2].ID(),
},
},
},
},
LostPieces: int32(4),
LastRepairAttempt: time.Now().Unix(),
RepairAttemptCount: int64(10),
}
require.NoError(t, irreparabledb.IncrementRepairAttempts(ctx, seg))
result, err := irreparabledb.Get(ctx, metabase.SegmentKey(seg.GetPath()))
require.NoError(t, err)
require.NotNil(t, result)
// test healthy segment is removed from irreparable DB
require.NoError(t, checker.IrreparableProcess(ctx))
result, err = irreparabledb.Get(ctx, metabase.SegmentKey(seg.GetPath()))
require.Error(t, err)
require.Nil(t, result)
// test unhealthy repairable segment is removed from irreparable DB and inserted into repair queue
seg.SegmentDetail.Remote.RemotePieces[0] = &pb.RemotePiece{}
seg.SegmentDetail.Remote.RemotePieces[1] = &pb.RemotePiece{}
require.NoError(t, irreparabledb.IncrementRepairAttempts(ctx, seg))
require.NoError(t, checker.IrreparableProcess(ctx))
result, err = irreparabledb.Get(ctx, metabase.SegmentKey(seg.GetPath()))
require.Error(t, err)
require.Nil(t, result)
injured, err := queue.Select(ctx)
require.NoError(t, err)
require.Equal(t, seg.GetPath(), injured.GetPath())
n, err := queue.Clean(ctx, time.Now())
require.NoError(t, err)
require.EqualValues(t, 1, n)
// test irreparable segment remains in irreparable DB and repair_attempt_count is incremented
seg.SegmentDetail.Remote.RemotePieces[2] = &pb.RemotePiece{}
require.NoError(t, irreparabledb.IncrementRepairAttempts(ctx, seg))
require.NoError(t, checker.IrreparableProcess(ctx))
result, err = irreparabledb.Get(ctx, metabase.SegmentKey(seg.GetPath()))
require.NoError(t, err)
require.Equal(t, seg.GetPath(), result.Path)
require.Equal(t, seg.RepairAttemptCount+1, result.RepairAttemptCount)
})
}

View File

@ -53,15 +53,13 @@ func testDataRepair(t *testing.T, inMemoryRepair bool) {
StorageNodeCount: 14,
UplinkCount: 1,
Reconfigure: testplanet.Reconfigure{
Satellite: func(log *zap.Logger, index int, config *satellite.Config) {
config.Repairer.MaxExcessRateOptimalThreshold = RepairMaxExcessRateOptimalThreshold
config.Repairer.InMemoryRepair = inMemoryRepair
config.Metainfo.RS.MinThreshold = minThreshold
config.Metainfo.RS.RepairThreshold = 5
config.Metainfo.RS.SuccessThreshold = successThreshold
config.Metainfo.RS.TotalThreshold = 9
},
Satellite: testplanet.Combine(
func(log *zap.Logger, index int, config *satellite.Config) {
config.Repairer.MaxExcessRateOptimalThreshold = RepairMaxExcessRateOptimalThreshold
config.Repairer.InMemoryRepair = inMemoryRepair
},
testplanet.ReconfigureRS(minThreshold, 5, successThreshold, 9),
),
},
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
@ -188,15 +186,13 @@ func testCorruptDataRepairFailed(t *testing.T, inMemoryRepair bool) {
StorageNodeCount: 14,
UplinkCount: 1,
Reconfigure: testplanet.Reconfigure{
Satellite: func(log *zap.Logger, index int, config *satellite.Config) {
config.Repairer.MaxExcessRateOptimalThreshold = RepairMaxExcessRateOptimalThreshold
config.Repairer.InMemoryRepair = inMemoryRepair
config.Metainfo.RS.MinThreshold = 3
config.Metainfo.RS.RepairThreshold = 5
config.Metainfo.RS.SuccessThreshold = 7
config.Metainfo.RS.TotalThreshold = 9
},
Satellite: testplanet.Combine(
func(log *zap.Logger, index int, config *satellite.Config) {
config.Repairer.MaxExcessRateOptimalThreshold = RepairMaxExcessRateOptimalThreshold
config.Repairer.InMemoryRepair = inMemoryRepair
},
testplanet.ReconfigureRS(3, 5, 7, 9),
),
},
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
uplinkPeer := planet.Uplinks[0]
@ -305,15 +301,13 @@ func testCorruptDataRepairSucceed(t *testing.T, inMemoryRepair bool) {
StorageNodeCount: 14,
UplinkCount: 1,
Reconfigure: testplanet.Reconfigure{
Satellite: func(log *zap.Logger, index int, config *satellite.Config) {
config.Repairer.MaxExcessRateOptimalThreshold = RepairMaxExcessRateOptimalThreshold
config.Repairer.InMemoryRepair = inMemoryRepair
config.Metainfo.RS.MinThreshold = 3
config.Metainfo.RS.RepairThreshold = 5
config.Metainfo.RS.SuccessThreshold = 7
config.Metainfo.RS.TotalThreshold = 9
},
Satellite: testplanet.Combine(
func(log *zap.Logger, index int, config *satellite.Config) {
config.Repairer.MaxExcessRateOptimalThreshold = RepairMaxExcessRateOptimalThreshold
config.Repairer.InMemoryRepair = inMemoryRepair
},
testplanet.ReconfigureRS(3, 5, 7, 9),
),
},
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
uplinkPeer := planet.Uplinks[0]
@ -799,14 +793,12 @@ func testRepairMultipleDisqualifiedAndSuspended(t *testing.T, inMemoryRepair boo
StorageNodeCount: 12,
UplinkCount: 1,
Reconfigure: testplanet.Reconfigure{
Satellite: func(log *zap.Logger, index int, config *satellite.Config) {
config.Repairer.InMemoryRepair = inMemoryRepair
config.Metainfo.RS.MinThreshold = 3
config.Metainfo.RS.RepairThreshold = 5
config.Metainfo.RS.SuccessThreshold = 7
config.Metainfo.RS.TotalThreshold = 7
},
Satellite: testplanet.Combine(
func(log *zap.Logger, index int, config *satellite.Config) {
config.Repairer.InMemoryRepair = inMemoryRepair
},
testplanet.ReconfigureRS(3, 5, 7, 7),
),
},
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
// first, upload some remote data
@ -920,15 +912,13 @@ func testDataRepairOverrideHigherLimit(t *testing.T, inMemoryRepair bool) {
StorageNodeCount: 14,
UplinkCount: 1,
Reconfigure: testplanet.Reconfigure{
Satellite: func(log *zap.Logger, index int, config *satellite.Config) {
config.Checker.RepairOverride = repairOverride
config.Repairer.InMemoryRepair = inMemoryRepair
config.Metainfo.RS.MinThreshold = 3
config.Metainfo.RS.RepairThreshold = 4
config.Metainfo.RS.SuccessThreshold = 9
config.Metainfo.RS.TotalThreshold = 9
},
Satellite: testplanet.Combine(
func(log *zap.Logger, index int, config *satellite.Config) {
config.Repairer.InMemoryRepair = inMemoryRepair
config.Checker.RepairOverride = repairOverride
},
testplanet.ReconfigureRS(3, 4, 9, 9),
),
},
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
uplinkPeer := planet.Uplinks[0]
@ -1014,15 +1004,13 @@ func testDataRepairOverrideLowerLimit(t *testing.T, inMemoryRepair bool) {
StorageNodeCount: 14,
UplinkCount: 1,
Reconfigure: testplanet.Reconfigure{
Satellite: func(log *zap.Logger, index int, config *satellite.Config) {
config.Checker.RepairOverride = repairOverride
config.Repairer.InMemoryRepair = inMemoryRepair
config.Metainfo.RS.MinThreshold = 3
config.Metainfo.RS.RepairThreshold = 6
config.Metainfo.RS.SuccessThreshold = 9
config.Metainfo.RS.TotalThreshold = 9
},
Satellite: testplanet.Combine(
func(log *zap.Logger, index int, config *satellite.Config) {
config.Repairer.InMemoryRepair = inMemoryRepair
config.Checker.RepairOverride = repairOverride
},
testplanet.ReconfigureRS(3, 6, 9, 9),
),
},
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
uplinkPeer := planet.Uplinks[0]
@ -1141,15 +1129,13 @@ func testDataRepairUploadLimit(t *testing.T, inMemoryRepair bool) {
StorageNodeCount: 13,
UplinkCount: 1,
Reconfigure: testplanet.Reconfigure{
Satellite: func(log *zap.Logger, index int, config *satellite.Config) {
config.Repairer.MaxExcessRateOptimalThreshold = RepairMaxExcessRateOptimalThreshold
config.Repairer.InMemoryRepair = inMemoryRepair
config.Metainfo.RS.MinThreshold = 3
config.Metainfo.RS.RepairThreshold = repairThreshold
config.Metainfo.RS.SuccessThreshold = successThreshold
config.Metainfo.RS.TotalThreshold = maxThreshold
},
Satellite: testplanet.Combine(
func(log *zap.Logger, index int, config *satellite.Config) {
config.Repairer.MaxExcessRateOptimalThreshold = RepairMaxExcessRateOptimalThreshold
config.Repairer.InMemoryRepair = inMemoryRepair
},
testplanet.ReconfigureRS(3, repairThreshold, successThreshold, maxThreshold),
),
},
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
satellite := planet.Satellites[0]
@ -1266,14 +1252,12 @@ func testRepairGracefullyExited(t *testing.T, inMemoryRepair bool) {
StorageNodeCount: 12,
UplinkCount: 1,
Reconfigure: testplanet.Reconfigure{
Satellite: func(log *zap.Logger, index int, config *satellite.Config) {
config.Repairer.InMemoryRepair = inMemoryRepair
config.Metainfo.RS.MinThreshold = 3
config.Metainfo.RS.RepairThreshold = 5
config.Metainfo.RS.SuccessThreshold = 7
config.Metainfo.RS.TotalThreshold = 7
},
Satellite: testplanet.Combine(
func(log *zap.Logger, index int, config *satellite.Config) {
config.Repairer.InMemoryRepair = inMemoryRepair
},
testplanet.ReconfigureRS(3, 5, 7, 7),
),
},
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
// first, upload some remote data

View File

@ -346,7 +346,6 @@ func (repairer *SegmentRepairer) updateAuditFailStatus(ctx context.Context, fail
for i, nodeID := range failedAuditNodeIDs {
updateRequests[i] = &overlay.UpdateRequest{
NodeID: nodeID,
IsUp: true,
AuditOutcome: overlay.AuditFailure,
}
}

View File

@ -366,7 +366,8 @@ func (cache *overlaycache) BatchUpdateStats(ctx context.Context, updateRequests
continue
}
auditHistory, err := cache.updateAuditHistoryWithTx(ctx, tx, updateReq.NodeID, now, updateReq.IsUp, updateReq.AuditHistory)
isUp := updateReq.AuditOutcome != overlay.AuditOffline
auditHistory, err := cache.updateAuditHistoryWithTx(ctx, tx, updateReq.NodeID, now, isUp, updateReq.AuditHistory)
if err != nil {
doAppendAll = false
return err
@ -444,7 +445,8 @@ func (cache *overlaycache) UpdateStats(ctx context.Context, updateReq *overlay.U
return nil
}
auditHistory, err := cache.updateAuditHistoryWithTx(ctx, tx, updateReq.NodeID, now, updateReq.IsUp, updateReq.AuditHistory)
isUp := updateReq.AuditOutcome != overlay.AuditOffline
auditHistory, err := cache.updateAuditHistoryWithTx(ctx, tx, updateReq.NodeID, now, isUp, updateReq.AuditHistory)
if err != nil {
return err
}
@ -1259,7 +1261,8 @@ func (cache *overlaycache) populateUpdateNodeStats(dbNode *dbx.Node, updateReq *
mon.FloatVal("audit_online_score").Observe(auditOnlineScore) //mon:locked
totalUptimeCount := dbNode.TotalUptimeCount
if updateReq.IsUp {
isUp := updateReq.AuditOutcome != overlay.AuditOffline
if isUp {
totalUptimeCount++
}
@ -1317,7 +1320,7 @@ func (cache *overlaycache) populateUpdateNodeStats(dbNode *dbx.Node, updateReq *
updateFields.UnknownAuditSuspended = timeField{set: true, isNil: true}
}
if updateReq.IsUp {
if isUp {
updateFields.UptimeSuccessCount = int64Field{set: true, value: dbNode.UptimeSuccessCount + 1}
updateFields.LastContactSuccess = timeField{set: true, value: now}
} else {

View File

@ -28,11 +28,11 @@ func TestUpdateStats(t *testing.T) {
numAudits := int64(2)
numUptimes := int64(3)
// nodes automatically start with 2 uptimes from testplanet startup
// nodeA: 1 audit, 2 uptime -> unvetted
updateReq := &overlay.UpdateRequest{
NodeID: nodeA.ID(),
AuditOutcome: overlay.AuditFailure,
IsUp: false,
AuditOutcome: overlay.AuditOffline,
AuditsRequiredForVetting: numAudits,
UptimesRequiredForVetting: numUptimes,
AuditHistory: testAuditHistoryConfig(),
@ -45,8 +45,7 @@ func TestUpdateStats(t *testing.T) {
// nodeA: 2 audits, 2 uptimes -> unvetted
updateReq.NodeID = nodeA.ID()
updateReq.AuditOutcome = overlay.AuditFailure
updateReq.IsUp = false
updateReq.AuditOutcome = overlay.AuditOffline
nodeStats, err = cache.UpdateStats(ctx, updateReq, time.Now())
require.NoError(t, err)
assert.Nil(t, nodeStats.VettedAt)
@ -56,7 +55,6 @@ func TestUpdateStats(t *testing.T) {
// nodeA: 3 audits, 3 uptimes -> vetted
updateReq.NodeID = nodeA.ID()
updateReq.AuditOutcome = overlay.AuditSuccess
updateReq.IsUp = true
nodeStats, err = cache.UpdateStats(ctx, updateReq, time.Now())
require.NoError(t, err)
assert.NotNil(t, nodeStats.VettedAt)
@ -66,7 +64,6 @@ func TestUpdateStats(t *testing.T) {
// nodeB: 1 audit, 3 uptimes -> unvetted
updateReq.NodeID = nodeB.ID()
updateReq.AuditOutcome = overlay.AuditSuccess
updateReq.IsUp = true
nodeStats, err = cache.UpdateStats(ctx, updateReq, time.Now())
require.NoError(t, err)
assert.Nil(t, nodeStats.VettedAt)
@ -75,8 +72,7 @@ func TestUpdateStats(t *testing.T) {
// nodeB: 2 audits, 3 uptimes -> vetted
updateReq.NodeID = nodeB.ID()
updateReq.AuditOutcome = overlay.AuditFailure
updateReq.IsUp = false
updateReq.AuditOutcome = overlay.AuditOffline
nodeStats, err = cache.UpdateStats(ctx, updateReq, time.Now())
require.NoError(t, err)
assert.NotNil(t, nodeStats.VettedAt)
@ -86,7 +82,6 @@ func TestUpdateStats(t *testing.T) {
// Don't overwrite node b's vetted_at timestamp
updateReq.NodeID = nodeB.ID()
updateReq.AuditOutcome = overlay.AuditSuccess
updateReq.IsUp = true
nodeStats2, err := cache.UpdateStats(ctx, updateReq, time.Now())
require.NoError(t, err)
assert.NotNil(t, nodeStats2.VettedAt)
@ -110,9 +105,10 @@ func TestBatchUpdateStats(t *testing.T) {
numUptimes := int64(3)
batchSize := 2
// nodes automatically start with 2 uptimes from testplanet startup
// both nodeA and nodeB unvetted
updateReqA := &overlay.UpdateRequest{NodeID: nodeA.ID(), AuditOutcome: overlay.AuditFailure, IsUp: false, AuditsRequiredForVetting: numAudits, UptimesRequiredForVetting: numUptimes, AuditHistory: testAuditHistoryConfig()}
updateReqB := &overlay.UpdateRequest{NodeID: nodeB.ID(), AuditOutcome: overlay.AuditSuccess, IsUp: true, AuditsRequiredForVetting: numAudits, UptimesRequiredForVetting: numUptimes, AuditHistory: testAuditHistoryConfig()}
updateReqA := &overlay.UpdateRequest{NodeID: nodeA.ID(), AuditOutcome: overlay.AuditOffline, AuditsRequiredForVetting: numAudits, UptimesRequiredForVetting: numUptimes, AuditHistory: testAuditHistoryConfig()}
updateReqB := &overlay.UpdateRequest{NodeID: nodeB.ID(), AuditOutcome: overlay.AuditSuccess, AuditsRequiredForVetting: numAudits, UptimesRequiredForVetting: numUptimes, AuditHistory: testAuditHistoryConfig()}
updateReqs := []*overlay.UpdateRequest{updateReqA, updateReqB}
failed, err := cache.BatchUpdateStats(ctx, updateReqs, batchSize, time.Now())
require.NoError(t, err)
@ -131,8 +127,8 @@ func TestBatchUpdateStats(t *testing.T) {
assert.EqualValues(t, 3, nB.Reputation.UptimeCount)
// nodeA unvetted, nodeB vetted
updateReqA = &overlay.UpdateRequest{NodeID: nodeA.ID(), AuditOutcome: overlay.AuditFailure, IsUp: false, AuditsRequiredForVetting: numAudits, UptimesRequiredForVetting: numUptimes, AuditHistory: testAuditHistoryConfig()}
updateReqB = &overlay.UpdateRequest{NodeID: nodeB.ID(), AuditOutcome: overlay.AuditFailure, IsUp: false, AuditsRequiredForVetting: numAudits, UptimesRequiredForVetting: numUptimes, AuditHistory: testAuditHistoryConfig()}
updateReqA = &overlay.UpdateRequest{NodeID: nodeA.ID(), AuditOutcome: overlay.AuditOffline, AuditsRequiredForVetting: numAudits, UptimesRequiredForVetting: numUptimes, AuditHistory: testAuditHistoryConfig()}
updateReqB = &overlay.UpdateRequest{NodeID: nodeB.ID(), AuditOutcome: overlay.AuditFailure, AuditsRequiredForVetting: numAudits, UptimesRequiredForVetting: numUptimes, AuditHistory: testAuditHistoryConfig()}
updateReqs = []*overlay.UpdateRequest{updateReqA, updateReqB}
failed, err = cache.BatchUpdateStats(ctx, updateReqs, batchSize, time.Now())
require.NoError(t, err)
@ -148,11 +144,11 @@ func TestBatchUpdateStats(t *testing.T) {
require.NoError(t, err)
assert.NotNil(t, nB.Reputation.VettedAt)
assert.EqualValues(t, 2, nB.Reputation.AuditCount)
assert.EqualValues(t, 3, nB.Reputation.UptimeCount)
assert.EqualValues(t, 4, nB.Reputation.UptimeCount)
// both nodeA and nodeB vetted (don't overwrite timestamp)
updateReqA = &overlay.UpdateRequest{NodeID: nodeA.ID(), AuditOutcome: overlay.AuditSuccess, IsUp: true, AuditsRequiredForVetting: numAudits, UptimesRequiredForVetting: numUptimes, AuditHistory: testAuditHistoryConfig()}
updateReqB = &overlay.UpdateRequest{NodeID: nodeB.ID(), AuditOutcome: overlay.AuditSuccess, IsUp: true, AuditsRequiredForVetting: numAudits, UptimesRequiredForVetting: numUptimes, AuditHistory: testAuditHistoryConfig()}
updateReqA = &overlay.UpdateRequest{NodeID: nodeA.ID(), AuditOutcome: overlay.AuditSuccess, AuditsRequiredForVetting: numAudits, UptimesRequiredForVetting: numUptimes, AuditHistory: testAuditHistoryConfig()}
updateReqB = &overlay.UpdateRequest{NodeID: nodeB.ID(), AuditOutcome: overlay.AuditSuccess, AuditsRequiredForVetting: numAudits, UptimesRequiredForVetting: numUptimes, AuditHistory: testAuditHistoryConfig()}
updateReqs = []*overlay.UpdateRequest{updateReqA, updateReqB}
failed, err = cache.BatchUpdateStats(ctx, updateReqs, batchSize, time.Now())
require.NoError(t, err)
@ -169,7 +165,7 @@ func TestBatchUpdateStats(t *testing.T) {
assert.NotNil(t, nB2.Reputation.VettedAt)
assert.Equal(t, nB.Reputation.VettedAt, nB2.Reputation.VettedAt)
assert.EqualValues(t, 3, nB2.Reputation.AuditCount)
assert.EqualValues(t, 4, nB2.Reputation.UptimeCount)
assert.EqualValues(t, 5, nB2.Reputation.UptimeCount)
})
}

View File

@ -358,72 +358,78 @@ func (db *ProjectAccounting) GetBucketUsageRollups(ctx context.Context, projectI
var bucketUsageRollups []accounting.BucketUsageRollup
for _, bucket := range buckets {
bucketRollup := accounting.BucketUsageRollup{
ProjectID: projectID,
BucketName: []byte(bucket),
Since: since,
Before: before,
}
err := func() error {
bucketRollup := accounting.BucketUsageRollup{
ProjectID: projectID,
BucketName: []byte(bucket),
Since: since,
Before: before,
}
// get bucket_bandwidth_rollups
rollupsRows, err := db.db.QueryContext(ctx, roullupsQuery, projectID[:], []byte(bucket), since, before)
if err != nil {
return nil, err
}
defer func() { err = errs.Combine(err, rollupsRows.Close()) }()
// fill egress
for rollupsRows.Next() {
var action pb.PieceAction
var settled, inline int64
err = rollupsRows.Scan(&settled, &inline, &action)
// get bucket_bandwidth_rollups
rollupsRows, err := db.db.QueryContext(ctx, roullupsQuery, projectID[:], []byte(bucket), since, before)
if err != nil {
return nil, err
return err
}
defer func() { err = errs.Combine(err, rollupsRows.Close()) }()
// fill egress
for rollupsRows.Next() {
var action pb.PieceAction
var settled, inline int64
err = rollupsRows.Scan(&settled, &inline, &action)
if err != nil {
return err
}
switch action {
case pb.PieceAction_GET:
bucketRollup.GetEgress += memory.Size(settled + inline).GB()
case pb.PieceAction_GET_AUDIT:
bucketRollup.AuditEgress += memory.Size(settled + inline).GB()
case pb.PieceAction_GET_REPAIR:
bucketRollup.RepairEgress += memory.Size(settled + inline).GB()
default:
continue
}
}
if err := rollupsRows.Err(); err != nil {
return err
}
switch action {
case pb.PieceAction_GET:
bucketRollup.GetEgress += memory.Size(settled + inline).GB()
case pb.PieceAction_GET_AUDIT:
bucketRollup.AuditEgress += memory.Size(settled + inline).GB()
case pb.PieceAction_GET_REPAIR:
bucketRollup.RepairEgress += memory.Size(settled + inline).GB()
default:
continue
bucketStorageTallies, err := storageQuery(ctx,
dbx.BucketStorageTally_ProjectId(projectID[:]),
dbx.BucketStorageTally_BucketName([]byte(bucket)),
dbx.BucketStorageTally_IntervalStart(since),
dbx.BucketStorageTally_IntervalStart(before))
if err != nil {
return err
}
}
if err := rollupsRows.Err(); err != nil {
return nil, err
}
bucketStorageTallies, err := storageQuery(ctx,
dbx.BucketStorageTally_ProjectId(projectID[:]),
dbx.BucketStorageTally_BucketName([]byte(bucket)),
dbx.BucketStorageTally_IntervalStart(since),
dbx.BucketStorageTally_IntervalStart(before))
// fill metadata, objects and stored data
// hours calculated from previous tallies,
// so we skip the most recent one
for i := len(bucketStorageTallies) - 1; i > 0; i-- {
current := bucketStorageTallies[i]
hours := bucketStorageTallies[i-1].IntervalStart.Sub(current.IntervalStart).Hours()
bucketRollup.RemoteStoredData += memory.Size(current.Remote).GB() * hours
bucketRollup.InlineStoredData += memory.Size(current.Inline).GB() * hours
bucketRollup.MetadataSize += memory.Size(current.MetadataSize).GB() * hours
bucketRollup.RemoteSegments += float64(current.RemoteSegmentsCount) * hours
bucketRollup.InlineSegments += float64(current.InlineSegmentsCount) * hours
bucketRollup.ObjectCount += float64(current.ObjectCount) * hours
}
bucketUsageRollups = append(bucketUsageRollups, bucketRollup)
return nil
}()
if err != nil {
return nil, err
}
// fill metadata, objects and stored data
// hours calculated from previous tallies,
// so we skip the most recent one
for i := len(bucketStorageTallies) - 1; i > 0; i-- {
current := bucketStorageTallies[i]
hours := bucketStorageTallies[i-1].IntervalStart.Sub(current.IntervalStart).Hours()
bucketRollup.RemoteStoredData += memory.Size(current.Remote).GB() * hours
bucketRollup.InlineStoredData += memory.Size(current.Inline).GB() * hours
bucketRollup.MetadataSize += memory.Size(current.MetadataSize).GB() * hours
bucketRollup.RemoteSegments += float64(current.RemoteSegmentsCount) * hours
bucketRollup.InlineSegments += float64(current.InlineSegmentsCount) * hours
bucketRollup.ObjectCount += float64(current.ObjectCount) * hours
}
bucketUsageRollups = append(bucketUsageRollups, bucketRollup)
}
return bucketUsageRollups, nil

View File

@ -400,32 +400,8 @@ identity.key-path: /root/.local/share/storj/identity/satellite/identity.key
# request rate per project per second.
# metainfo.rate-limiter.rate: 1000
# the size of each new erasure share in bytes
# metainfo.rs.erasure-share-size: 256 B
# maximum buffer memory to be allocated for read buffers
# metainfo.rs.max-buffer-mem: 4.0 MiB
# the largest amount of pieces to encode to. n (upper bound for validation).
# metainfo.rs.max-total-threshold: 130
# the minimum pieces required to recover a segment. k.
# metainfo.rs.min-threshold: 29
# the largest amount of pieces to encode to. n (lower bound for validation).
# metainfo.rs.min-total-threshold: 95
# the minimum safe pieces before a repair is triggered. m.
# metainfo.rs.repair-threshold: 35
# the desired total pieces for a segment. o.
# metainfo.rs.success-threshold: 80
# the largest amount of pieces to encode to. n.
# metainfo.rs.total-threshold: 110
# validate redundancy scheme configuration
# metainfo.rs.validate: true
# redundancy scheme configuration in the format k/m/o/n-sharesize
# metainfo.rs: 29/35/80/110-256 B
# address(es) to send telemetry to (comma-separated)
# metrics.addr: collectora.storj.io:9000

View File

@ -163,16 +163,18 @@ func testConstraints(t *testing.T, ctx *testcontext.Context, store storage.KeyVa
{storage.Value("old-value"), nil},
{storage.Value("old-value"), storage.Value("new-value")},
} {
errTag := fmt.Sprintf("%d. %+v", i, tt)
key := storage.Key("test-key")
val := storage.Value("test-value")
defer func() { _ = store.Delete(ctx, key) }()
func() {
errTag := fmt.Sprintf("%d. %+v", i, tt)
key := storage.Key("test-key")
val := storage.Value("test-value")
defer func() { _ = store.Delete(ctx, key) }()
err := store.Put(ctx, key, val)
require.NoError(t, err, errTag)
err := store.Put(ctx, key, val)
require.NoError(t, err, errTag)
err = store.CompareAndSwap(ctx, key, tt.old, tt.new)
assert.True(t, storage.ErrValueChanged.Has(err), "%s: unexpected error: %+v", errTag, err)
err = store.CompareAndSwap(ctx, key, tt.old, tt.new)
assert.True(t, storage.ErrValueChanged.Has(err), "%s: unexpected error: %+v", errTag, err)
}()
}
})

View File

@ -376,8 +376,15 @@ func TestHeldAmountApi(t *testing.T) {
JoinedAt: date,
}
stefanID, err := storj.NodeIDFromString("118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW")
require.NoError(t, err)
held2 := payout.SatelliteHeldHistory{
SatelliteID: stefanID,
}
var periods []payout.SatelliteHeldHistory
periods = append(periods, held)
periods = append(periods, held, held2)
expected, err := json.Marshal(periods)
require.NoError(t, err)

View File

@ -177,15 +177,14 @@ func TestWorkerFailure_IneligibleNodeAge(t *testing.T) {
StorageNodeCount: 5,
UplinkCount: 1,
Reconfigure: testplanet.Reconfigure{
Satellite: func(logger *zap.Logger, index int, config *satellite.Config) {
// Set the required node age to 1 month.
config.GracefulExit.NodeMinAgeInMonths = 1
Satellite: testplanet.Combine(
func(log *zap.Logger, index int, config *satellite.Config) {
// Set the required node age to 1 month.
config.GracefulExit.NodeMinAgeInMonths = 1
},
testplanet.ReconfigureRS(2, 3, successThreshold, successThreshold),
),
config.Metainfo.RS.MinThreshold = 2
config.Metainfo.RS.RepairThreshold = 3
config.Metainfo.RS.SuccessThreshold = successThreshold
config.Metainfo.RS.TotalThreshold = successThreshold
},
StorageNode: func(index int, config *storagenode.Config) {
config.GracefulExit.NumWorkers = 2
config.GracefulExit.NumConcurrentTransfers = 2

View File

@ -398,7 +398,6 @@ func (service *Service) sendOrdersFromFileStore(ctx context.Context, now time.Ti
var group errgroup.Group
attemptedSatellites := 0
ctx, cancel := context.WithTimeout(ctx, service.config.SenderTimeout)
defer cancel()
for satelliteID, unsentInfo := range ordersBySatellite {
satelliteID, unsentInfo := satelliteID, unsentInfo
@ -430,6 +429,7 @@ func (service *Service) sendOrdersFromFileStore(ctx context.Context, now time.Ti
}
_ = group.Wait() // doesn't return errors
cancel()
// if all satellites that orders need to be sent to are offline, exit and try again later.
if attemptedSatellites == 0 {

View File

@ -211,7 +211,8 @@ func TestSatellitePayStubPeriodCached(t *testing.T) {
heldAmountDB := db.Payout()
reputationDB := db.Reputation()
satellitesDB := db.Satellites()
service := payout.NewService(nil, heldAmountDB, reputationDB, satellitesDB, nil)
service, err := payout.NewService(nil, heldAmountDB, reputationDB, satellitesDB, nil)
require.NoError(t, err)
payStub := payout.PayStub{
SatelliteID: storj.NodeID{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1},
@ -261,7 +262,8 @@ func TestAllPayStubPeriodCached(t *testing.T) {
heldAmountDB := db.Payout()
reputationDB := db.Reputation()
satellitesDB := db.Satellites()
service := payout.NewService(nil, heldAmountDB, reputationDB, satellitesDB, nil)
service, err := payout.NewService(nil, heldAmountDB, reputationDB, satellitesDB, nil)
require.NoError(t, err)
payStub := payout.PayStub{
SatelliteID: storj.NodeID{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1},

View File

@ -39,6 +39,8 @@ var (
type Service struct {
log *zap.Logger
stefanSatellite storj.NodeID
db DB
reputationDB reputation.DB
satellitesDB satellites.DB
@ -46,14 +48,20 @@ type Service struct {
}
// NewService creates new instance of service.
func NewService(log *zap.Logger, db DB, reputationDB reputation.DB, satelliteDB satellites.DB, trust *trust.Pool) *Service {
return &Service{
log: log,
db: db,
reputationDB: reputationDB,
satellitesDB: satelliteDB,
trust: trust,
func NewService(log *zap.Logger, db DB, reputationDB reputation.DB, satelliteDB satellites.DB, trust *trust.Pool) (_ *Service, err error) {
id, err := storj.NodeIDFromString("118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW")
if err != nil {
return &Service{}, err
}
return &Service{
log: log,
stefanSatellite: id,
db: db,
reputationDB: reputationDB,
satellitesDB: satelliteDB,
trust: trust,
}, nil
}
// SatellitePayStubMonthly retrieves held amount for particular satellite for selected month from storagenode database.
@ -161,17 +169,18 @@ func (service *Service) AllPeriods(ctx context.Context) (_ []string, err error)
// AllHeldbackHistory retrieves heldback history for all satellites from storagenode database.
func (service *Service) AllHeldbackHistory(ctx context.Context) (result []SatelliteHeldHistory, err error) {
defer mon.Task()(&ctx)(&err)
satellitesIDs := service.trust.GetSatellites(ctx)
satellites := service.trust.GetSatellites(ctx)
for i := 0; i < len(satellites); i++ {
satellitesIDs = append(satellitesIDs, service.stefanSatellite)
for i := 0; i < len(satellitesIDs); i++ {
var history SatelliteHeldHistory
helds, err := service.db.SatellitesHeldbackHistory(ctx, satellites[i])
helds, err := service.db.SatellitesHeldbackHistory(ctx, satellitesIDs[i])
if err != nil {
return nil, ErrPayoutService.Wrap(err)
}
disposed, err := service.db.SatellitesDisposedHistory(ctx, satellites[i])
disposed, err := service.db.SatellitesDisposedHistory(ctx, satellitesIDs[i])
if err != nil {
return nil, ErrPayoutService.Wrap(err)
}
@ -192,20 +201,22 @@ func (service *Service) AllHeldbackHistory(ctx context.Context) (result []Satell
}
history.TotalDisposed = disposed
history.SatelliteID = satellites[i]
url, err := service.trust.GetNodeURL(ctx, satellites[i])
history.SatelliteID = satellitesIDs[i]
if satellitesIDs[i] != service.stefanSatellite {
url, err := service.trust.GetNodeURL(ctx, satellitesIDs[i])
if err != nil {
return nil, ErrPayoutService.Wrap(err)
}
history.SatelliteName = url.Address
}
stats, err := service.reputationDB.Get(ctx, satellitesIDs[i])
if err != nil {
return nil, ErrPayoutService.Wrap(err)
}
stats, err := service.reputationDB.Get(ctx, satellites[i])
if err != nil {
return nil, ErrPayoutService.Wrap(err)
}
history.SatelliteName = url.Address
history.JoinedAt = stats.JoinedAt
result = append(result, history)
}
@ -217,6 +228,8 @@ func (service *Service) AllSatellitesPayoutPeriod(ctx context.Context, period st
defer mon.Task()(&ctx)(&err)
satelliteIDs := service.trust.GetSatellites(ctx)
satelliteIDs = append(satelliteIDs, service.stefanSatellite)
for i := 0; i < len(satelliteIDs); i++ {
var payoutForPeriod SatellitePayoutForPeriod
paystub, err := service.db.GetPayStub(ctx, satelliteIDs[i], period)
@ -249,9 +262,13 @@ func (service *Service) AllSatellitesPayoutPeriod(ctx context.Context, period st
return nil, ErrPayoutService.Wrap(err)
}
url, err := service.trust.GetNodeURL(ctx, satelliteIDs[i])
if err != nil {
return nil, ErrPayoutService.Wrap(err)
if satelliteIDs[i] != service.stefanSatellite {
url, err := service.trust.GetNodeURL(ctx, satelliteIDs[i])
if err != nil {
return nil, ErrPayoutService.Wrap(err)
}
payoutForPeriod.SatelliteURL = url.Address
}
if satellite.Status == satellites.ExitSucceeded {
@ -281,7 +298,6 @@ func (service *Service) AllSatellitesPayoutPeriod(ctx context.Context, period st
payoutForPeriod.Earned = earned
payoutForPeriod.SatelliteID = satelliteIDs[i].String()
payoutForPeriod.SurgePercent = paystub.SurgePercent
payoutForPeriod.SatelliteURL = url.Address
payoutForPeriod.Paid = paystub.Paid
payoutForPeriod.HeldPercent = heldPercent

View File

@ -555,13 +555,17 @@ func New(log *zap.Logger, full *identity.FullIdentity, db DB, revocationDB exten
}
{ // setup payout service.
peer.Payout.Service = payout.NewService(
service, err := payout.NewService(
peer.Log.Named("payout:service"),
peer.DB.Payout(),
peer.DB.Reputation(),
peer.DB.Satellites(),
peer.Storage2.Trust,
)
if err != nil {
return nil, errs.Combine(err, peer.Close())
}
peer.Payout.Service = service
peer.Payout.Endpoint = payout.NewEndpoint(
peer.Log.Named("payout:endpoint"),
peer.Dialer,

View File

@ -50,23 +50,25 @@ func TestUploadAndPartialDownload(t *testing.T) {
{1513, 1584},
{13581, 4783},
} {
if piecestore.DefaultConfig.InitialStep < tt.size {
t.Fatal("test expects initial step to be larger than size to download")
}
totalDownload += piecestore.DefaultConfig.InitialStep
func() {
if piecestore.DefaultConfig.InitialStep < tt.size {
t.Fatal("test expects initial step to be larger than size to download")
}
totalDownload += piecestore.DefaultConfig.InitialStep
download, cleanup, err := planet.Uplinks[0].DownloadStreamRange(ctx, planet.Satellites[0], "testbucket", "test/path", tt.offset, -1)
require.NoError(t, err)
defer ctx.Check(cleanup)
download, cleanup, err := planet.Uplinks[0].DownloadStreamRange(ctx, planet.Satellites[0], "testbucket", "test/path", tt.offset, -1)
require.NoError(t, err)
defer ctx.Check(cleanup)
data := make([]byte, tt.size)
n, err := io.ReadFull(download, data)
require.NoError(t, err)
assert.Equal(t, int(tt.size), n)
data := make([]byte, tt.size)
n, err := io.ReadFull(download, data)
require.NoError(t, err)
assert.Equal(t, int(tt.size), n)
assert.Equal(t, expectedData[tt.offset:tt.offset+tt.size], data)
assert.Equal(t, expectedData[tt.offset:tt.offset+tt.size], data)
require.NoError(t, download.Close())
require.NoError(t, download.Close())
}()
}
var totalBandwidthUsage bandwidth.Usage
@ -528,8 +530,6 @@ func TestDeletePieces(t *testing.T) {
}
func TestTooManyRequests(t *testing.T) {
t.Skip("flaky, because of EOF issues")
const uplinkCount = 6
const maxConcurrent = 3
const expectedFailures = uplinkCount - maxConcurrent

View File

@ -226,45 +226,47 @@ func TestOrderLimitGetValidation(t *testing.T) {
err: "expected get or get repair or audit action got PUT",
},
} {
client, err := planet.Uplinks[0].DialPiecestore(ctx, planet.StorageNodes[0])
require.NoError(t, err)
defer ctx.Check(client.Close)
signer := signing.SignerFromFullIdentity(planet.Satellites[0].Identity)
satellite := planet.Satellites[0].Identity
if tt.satellite != nil {
signer = signing.SignerFromFullIdentity(tt.satellite)
satellite = tt.satellite
}
orderLimit, piecePrivateKey := GenerateOrderLimit(
t,
satellite.ID,
planet.StorageNodes[0].ID(),
tt.pieceID,
tt.action,
tt.serialNumber,
tt.pieceExpiration,
tt.orderExpiration,
tt.limit,
)
orderLimit, err = signing.SignOrderLimit(ctx, signer, orderLimit)
require.NoError(t, err)
downloader, err := client.Download(ctx, orderLimit, piecePrivateKey, 0, tt.limit)
require.NoError(t, err)
buffer, readErr := ioutil.ReadAll(downloader)
closeErr := downloader.Close()
err = errs.Combine(readErr, closeErr)
if tt.err != "" {
assert.Equal(t, 0, len(buffer))
require.Error(t, err)
require.Contains(t, err.Error(), tt.err)
} else {
func() {
client, err := planet.Uplinks[0].DialPiecestore(ctx, planet.StorageNodes[0])
require.NoError(t, err)
}
defer ctx.Check(client.Close)
signer := signing.SignerFromFullIdentity(planet.Satellites[0].Identity)
satellite := planet.Satellites[0].Identity
if tt.satellite != nil {
signer = signing.SignerFromFullIdentity(tt.satellite)
satellite = tt.satellite
}
orderLimit, piecePrivateKey := GenerateOrderLimit(
t,
satellite.ID,
planet.StorageNodes[0].ID(),
tt.pieceID,
tt.action,
tt.serialNumber,
tt.pieceExpiration,
tt.orderExpiration,
tt.limit,
)
orderLimit, err = signing.SignOrderLimit(ctx, signer, orderLimit)
require.NoError(t, err)
downloader, err := client.Download(ctx, orderLimit, piecePrivateKey, 0, tt.limit)
require.NoError(t, err)
buffer, readErr := ioutil.ReadAll(downloader)
closeErr := downloader.Close()
err = errs.Combine(readErr, closeErr)
if tt.err != "" {
assert.Equal(t, 0, len(buffer))
require.Error(t, err)
require.Contains(t, err.Error(), tt.err)
} else {
require.NoError(t, err)
}
}()
}
})
}

View File

@ -349,103 +349,111 @@ func (db *DB) MigrateToLatest(ctx context.Context) error {
// Preflight conducts a pre-flight check to ensure correct schemas and minimal read+write functionality of the database tables.
func (db *DB) Preflight(ctx context.Context) (err error) {
for dbName, dbContainer := range db.SQLDBs {
nextDB := dbContainer.GetDB()
// Preflight stage 1: test schema correctness
schema, err := sqliteutil.QuerySchema(ctx, nextDB)
if err != nil {
return ErrPreflight.New("database %q: schema check failed: %v", dbName, err)
}
// we don't care about changes in versions table
schema.DropTable("versions")
// if there was a previous pre-flight failure, test_table might still be in the schema
schema.DropTable("test_table")
// If tables and indexes of the schema are empty, set to nil
// to help with comparison to the snapshot.
if len(schema.Tables) == 0 {
schema.Tables = nil
}
if len(schema.Indexes) == 0 {
schema.Indexes = nil
}
// get expected schema
expectedSchema := Schema()[dbName]
// find extra indexes
var extraIdxs []*dbschema.Index
for _, idx := range schema.Indexes {
if _, exists := expectedSchema.FindIndex(idx.Name); exists {
continue
}
extraIdxs = append(extraIdxs, idx)
}
// drop index from schema if it is not unique to not fail preflight
for _, idx := range extraIdxs {
if !idx.Unique {
schema.DropIndex(idx.Name)
}
}
// warn that schema contains unexpected indexes
if len(extraIdxs) > 0 {
db.log.Warn(fmt.Sprintf("database %q: schema contains unexpected indices %v", dbName, extraIdxs))
}
// expect expected schema to match actual schema
if diff := cmp.Diff(expectedSchema, schema); diff != "" {
return ErrPreflight.New("database %q: expected schema does not match actual: %s", dbName, diff)
}
// Preflight stage 2: test basic read/write access
// for each database, create a new table, insert a row into that table, retrieve and validate that row, and drop the table.
// drop test table in case the last preflight check failed before table could be dropped
_, err = nextDB.ExecContext(ctx, "DROP TABLE IF EXISTS test_table")
if err != nil {
return ErrPreflight.New("database %q: failed drop if test_table: %w", dbName, err)
}
_, err = nextDB.ExecContext(ctx, "CREATE TABLE test_table(id int NOT NULL, name varchar(30), PRIMARY KEY (id))")
if err != nil {
return ErrPreflight.New("database %q: failed create test_table: %w", dbName, err)
}
var expectedID, actualID int
var expectedName, actualName string
expectedID = 1
expectedName = "TEST"
_, err = nextDB.ExecContext(ctx, "INSERT INTO test_table VALUES ( ?, ? )", expectedID, expectedName)
if err != nil {
return ErrPreflight.New("database: %q: failed inserting test value: %w", dbName, err)
}
rows, err := nextDB.QueryContext(ctx, "SELECT id, name FROM test_table")
if err != nil {
return ErrPreflight.New("database: %q: failed selecting test value: %w", dbName, err)
}
defer func() { err = errs.Combine(err, rows.Close()) }()
if !rows.Next() {
return ErrPreflight.New("database %q: no rows in test_table", dbName)
}
err = rows.Scan(&actualID, &actualName)
if err != nil {
return ErrPreflight.New("database %q: failed scanning row: %w", dbName, err)
}
if expectedID != actualID || expectedName != actualName {
return ErrPreflight.New("database %q: expected (%d, '%s'), actual (%d, '%s')", dbName, expectedID, expectedName, actualID, actualName)
}
if rows.Next() {
return ErrPreflight.New("database %q: more than one row in test_table", dbName)
}
_, err = nextDB.ExecContext(ctx, "DROP TABLE test_table")
if err != nil {
return ErrPreflight.New("database %q: failed drop test_table %w", dbName, err)
if err := db.preflight(ctx, dbName, dbContainer); err != nil {
return err
}
}
return nil
}
func (db *DB) preflight(ctx context.Context, dbName string, dbContainer DBContainer) error {
nextDB := dbContainer.GetDB()
// Preflight stage 1: test schema correctness
schema, err := sqliteutil.QuerySchema(ctx, nextDB)
if err != nil {
return ErrPreflight.New("database %q: schema check failed: %v", dbName, err)
}
// we don't care about changes in versions table
schema.DropTable("versions")
// if there was a previous pre-flight failure, test_table might still be in the schema
schema.DropTable("test_table")
// If tables and indexes of the schema are empty, set to nil
// to help with comparison to the snapshot.
if len(schema.Tables) == 0 {
schema.Tables = nil
}
if len(schema.Indexes) == 0 {
schema.Indexes = nil
}
// get expected schema
expectedSchema := Schema()[dbName]
// find extra indexes
var extraIdxs []*dbschema.Index
for _, idx := range schema.Indexes {
if _, exists := expectedSchema.FindIndex(idx.Name); exists {
continue
}
extraIdxs = append(extraIdxs, idx)
}
// drop index from schema if it is not unique to not fail preflight
for _, idx := range extraIdxs {
if !idx.Unique {
schema.DropIndex(idx.Name)
}
}
// warn that schema contains unexpected indexes
if len(extraIdxs) > 0 {
db.log.Warn(fmt.Sprintf("database %q: schema contains unexpected indices %v", dbName, extraIdxs))
}
// expect expected schema to match actual schema
if diff := cmp.Diff(expectedSchema, schema); diff != "" {
return ErrPreflight.New("database %q: expected schema does not match actual: %s", dbName, diff)
}
// Preflight stage 2: test basic read/write access
// for each database, create a new table, insert a row into that table, retrieve and validate that row, and drop the table.
// drop test table in case the last preflight check failed before table could be dropped
_, err = nextDB.ExecContext(ctx, "DROP TABLE IF EXISTS test_table")
if err != nil {
return ErrPreflight.New("database %q: failed drop if test_table: %w", dbName, err)
}
_, err = nextDB.ExecContext(ctx, "CREATE TABLE test_table(id int NOT NULL, name varchar(30), PRIMARY KEY (id))")
if err != nil {
return ErrPreflight.New("database %q: failed create test_table: %w", dbName, err)
}
var expectedID, actualID int
var expectedName, actualName string
expectedID = 1
expectedName = "TEST"
_, err = nextDB.ExecContext(ctx, "INSERT INTO test_table VALUES ( ?, ? )", expectedID, expectedName)
if err != nil {
return ErrPreflight.New("database: %q: failed inserting test value: %w", dbName, err)
}
rows, err := nextDB.QueryContext(ctx, "SELECT id, name FROM test_table")
if err != nil {
return ErrPreflight.New("database: %q: failed selecting test value: %w", dbName, err)
}
defer func() { err = errs.Combine(err, rows.Close()) }()
if !rows.Next() {
return ErrPreflight.New("database %q: no rows in test_table", dbName)
}
err = rows.Scan(&actualID, &actualName)
if err != nil {
return ErrPreflight.New("database %q: failed scanning row: %w", dbName, err)
}
if expectedID != actualID || expectedName != actualName {
return ErrPreflight.New("database %q: expected (%d, '%s'), actual (%d, '%s')", dbName, expectedID, expectedName, actualID, actualName)
}
if rows.Next() {
return ErrPreflight.New("database %q: more than one row in test_table", dbName)
}
_, err = nextDB.ExecContext(ctx, "DROP TABLE test_table")
if err != nil {
return ErrPreflight.New("database %q: failed drop test_table %w", dbName, err)
}
return nil
}
// Close closes any resources.
func (db *DB) Close() error {
return db.closeDatabases()

View File

@ -67,11 +67,11 @@ docker run -p 8080:8080 storjlabs/satellite-ui:latest
- [unit](./unit "unit") folder: contains project unit tests.
### Configuration files
- **.env**: file for environment level variables.
- **.gitignore**: folders, files and extentions which are ignored for git.
- **.gitignore**: folders, files and extensions which are ignored for git.
- **babel.config.js**: [babel](https://babeljs.io/) configuration for javascript transcompilation.
- **index.html**: DOM entry point.
- **jestSetup.ts**: [jest](https://jestjs.io/) configuration for unit testing.
- **package.json**: file holds various metadata relevant to the project such as version, dependencies, scripts and configurations.
- **tsconfig.json**: holds [TypeScript](https://www.typescriptlang.org/) configurations.
- **tslint.json**: holds [TypeScript](https://www.typescriptlang.org/) linter configurations.
- **vue.config.js**: holds [Vue](https://vuejs.org/) configurations.
- **vue.config.js**: holds [Vue](https://vuejs.org/) configurations.

View File

@ -68,7 +68,6 @@ import { APPSTATE_ACTIONS } from '@/app/store/modules/appState';
import { NODE_ACTIONS } from '@/app/store/modules/node';
import { NOTIFICATIONS_ACTIONS } from '@/app/store/modules/notifications';
import { PAYOUT_ACTIONS } from '@/app/store/modules/payout';
import { NotificationsCursor } from '@/app/types/notifications';
const {
GET_NODE_INFO,
@ -90,18 +89,23 @@ const {
export default class SNOHeader extends Vue {
public isNotificationPopupShown: boolean = false;
public isOptionsShown: boolean = false;
private readonly FIRST_PAGE: number = 1;
/**
* Lifecycle hook before render.
* Fetches first page of notifications.
*/
public beforeMount(): void {
public async beforeMount(): Promise<void> {
await this.$store.dispatch(APPSTATE_ACTIONS.SET_LOADING, true);
try {
this.$store.dispatch(NODE_ACTIONS.GET_NODE_INFO);
this.$store.dispatch(NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS, new NotificationsCursor(1));
await this.$store.dispatch(NODE_ACTIONS.GET_NODE_INFO);
await this.$store.dispatch(NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS, this.FIRST_PAGE);
} catch (error) {
console.error(error.message);
}
await this.$store.dispatch(APPSTATE_ACTIONS.SET_LOADING, false);
}
public get nodeId(): string {
@ -192,7 +196,7 @@ export default class SNOHeader extends Vue {
}
try {
await this.$store.dispatch(NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS, new NotificationsCursor(1));
await this.$store.dispatch(NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS, this.FIRST_PAGE);
} catch (error) {
console.error(error.message);
}

View File

@ -12,10 +12,10 @@
<div
class="notification-popup-container__content"
:class="{'collapsed': isCollapsed}"
v-if="latestNotifications.length"
v-if="latest.length"
>
<SNONotification
v-for="notification in latestNotifications"
v-for="notification in latest"
:key="notification.id"
is-small="true"
:notification="notification"
@ -34,6 +34,7 @@ import { Component, Vue } from 'vue-property-decorator';
import SNONotification from '@/app/components/notifications/SNONotification.vue';
import { RouteConfig } from '@/app/router';
import { UINotification } from '@/app/types/notifications';
@Component({
components: {
@ -49,7 +50,7 @@ export default class NotificationsPopup extends Vue {
/**
* Represents first page of notifications.
*/
public get latestNotifications(): Notification[] {
public get latest(): UINotification[] {
return this.$store.state.notificationsModule.latestNotifications;
}
@ -57,7 +58,7 @@ export default class NotificationsPopup extends Vue {
* Indicates if popup is smaller than with scroll.
*/
public get isCollapsed(): boolean {
return this.latestNotifications.length < 4;
return this.latest.length < 4;
}
}
</script>

View File

@ -30,12 +30,12 @@
import { Component, Prop, Vue } from 'vue-property-decorator';
import { NOTIFICATIONS_ACTIONS } from '@/app/store/modules/notifications';
import { Notification } from '@/app/types/notifications';
import { UINotification } from '@/app/types/notifications';
@Component
export default class SNONotification extends Vue {
@Prop({default: () => new Notification()})
public readonly notification: Notification;
@Prop({default: () => new UINotification()})
public readonly notification: UINotification;
/**
* isSmall props indicates if component used in popup.

View File

@ -89,7 +89,7 @@ export default class HeldProgress extends Vue {
),
new HeldStep(
'+50%',
'Month 15',
'Month 16',
this.monthsOnNetwork > 15,
this.monthsOnNetwork < 15,
),

View File

@ -4,17 +4,19 @@
import Vue from 'vue';
import Vuex from 'vuex';
import { makeNotificationsModule } from '@/app/store/modules/notifications';
import { newNotificationsModule } from '@/app/store/modules/notifications';
import { makePayoutModule } from '@/app/store/modules/payout';
import { NotificationsHttpApi } from '@/storagenode/api/notifications';
import { PayoutHttpApi } from '@/storagenode/api/payout';
import { SNOApi } from '@/storagenode/api/storagenode';
import { NotificationsService } from '@/storagenode/notifications/service';
import { PayoutService } from '@/storagenode/payouts/service';
import { appStateModule } from './modules/appState';
import { makeNodeModule } from './modules/node';
const notificationsApi = new NotificationsHttpApi();
const notificationsService = new NotificationsService(notificationsApi);
const payoutApi = new PayoutHttpApi();
const payoutService = new PayoutService(payoutApi);
const nodeApi = new SNOApi();
@ -28,7 +30,7 @@ export const store = new Vuex.Store({
modules: {
node: makeNodeModule(nodeApi),
appStateModule,
notificationsModule: makeNotificationsModule(notificationsApi),
notificationsModule: newNotificationsModule(notificationsService),
payoutModule: makePayoutModule(payoutApi, payoutService),
},
});

View File

@ -1,12 +1,8 @@
// Copyright (C) 2019 Storj Labs, Inc.
// See LICENSE for copying information.
import {
Notification,
NotificationsApi,
NotificationsCursor,
NotificationsState,
} from '@/app/types/notifications';
import { NotificationsState, UINotification } from '@/app/types/notifications';
import { NotificationsService } from '@/storagenode/notifications/service';
export const NOTIFICATIONS_MUTATIONS = {
SET_NOTIFICATIONS: 'SET_NOTIFICATIONS',
@ -24,22 +20,22 @@ export const NOTIFICATIONS_ACTIONS = {
/**
* creates notifications module with all dependencies
*
* @param api - payments api
* @param service - payments service
*/
export function makeNotificationsModule(api: NotificationsApi) {
export function newNotificationsModule(service: NotificationsService) {
return {
state: new NotificationsState(),
mutations: {
[NOTIFICATIONS_MUTATIONS.SET_NOTIFICATIONS](state: NotificationsState, notificationsResponse: NotificationsState): void {
state.notifications = notificationsResponse.notifications;
state.pageCount = notificationsResponse.pageCount;
state.unreadCount = notificationsResponse.unreadCount;
[NOTIFICATIONS_MUTATIONS.SET_NOTIFICATIONS](state: NotificationsState, notificationsState: NotificationsState): void {
state.notifications = notificationsState.notifications;
state.pageCount = notificationsState.pageCount;
state.unreadCount = notificationsState.unreadCount;
},
[NOTIFICATIONS_MUTATIONS.SET_LATEST](state: NotificationsState, notificationsResponse: NotificationsState): void {
state.latestNotifications = notificationsResponse.notifications;
[NOTIFICATIONS_MUTATIONS.SET_LATEST](state: NotificationsState, notificationsState: NotificationsState): void {
state.latestNotifications = notificationsState.notifications;
},
[NOTIFICATIONS_MUTATIONS.MARK_AS_READ](state: NotificationsState, id: string): void {
state.notifications = state.notifications.map((notification: Notification) => {
state.notifications = state.notifications.map((notification: UINotification) => {
if (notification.id === id) {
notification.markAsRead();
}
@ -48,7 +44,7 @@ export function makeNotificationsModule(api: NotificationsApi) {
});
},
[NOTIFICATIONS_MUTATIONS.READ_ALL](state: NotificationsState): void {
state.notifications = state.notifications.map((notification: Notification) => {
state.notifications = state.notifications.map((notification: UINotification) => {
notification.markAsRead();
return notification;
@ -58,24 +54,26 @@ export function makeNotificationsModule(api: NotificationsApi) {
},
},
actions: {
[NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS]: async function ({commit}: any, cursor: NotificationsCursor): Promise<NotificationsState> {
const notificationsResponse = await api.get(cursor);
[NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS]: async function ({commit}: any, pageIndex: number): Promise<void> {
const notificationsResponse = await service.notifications(pageIndex);
commit(NOTIFICATIONS_MUTATIONS.SET_NOTIFICATIONS, notificationsResponse);
const notifications = notificationsResponse.page.notifications.map(notification => new UINotification(notification));
if (cursor.page === 1) {
commit(NOTIFICATIONS_MUTATIONS.SET_LATEST, notificationsResponse);
const notificationState = new NotificationsState(notifications, notificationsResponse.page.pageCount, notificationsResponse.unreadCount);
commit(NOTIFICATIONS_MUTATIONS.SET_NOTIFICATIONS, notificationState);
if (pageIndex === 1) {
commit(NOTIFICATIONS_MUTATIONS.SET_LATEST, notificationState);
}
return notificationsResponse;
},
[NOTIFICATIONS_ACTIONS.MARK_AS_READ]: async function ({commit}: any, id: string): Promise<any> {
await api.read(id);
[NOTIFICATIONS_ACTIONS.MARK_AS_READ]: async function ({commit}: any, id: string): Promise<void> {
await service.readSingeNotification(id);
commit(NOTIFICATIONS_MUTATIONS.MARK_AS_READ, id);
},
[NOTIFICATIONS_ACTIONS.READ_ALL]: async function ({commit}: any): Promise<any> {
await api.readAll();
[NOTIFICATIONS_ACTIONS.READ_ALL]: async function ({commit}: any): Promise<void> {
await service.readAllNotifications();
commit(NOTIFICATIONS_MUTATIONS.READ_ALL);
},

View File

@ -2,23 +2,39 @@
// See LICENSE for copying information.
import { NotificationIcon } from '@/app/utils/notificationIcons';
import { Notification, NotificationTypes } from '@/storagenode/notifications/notifications';
/**
* Holds all notifications module state.
*/
export class NotificationsState {
public latestNotifications: UINotification[] = [];
public constructor(
public notifications: UINotification[] = [],
public pageCount: number = 0,
public unreadCount: number = 0,
) { }
}
/**
* Describes notification entity.
*/
export class Notification {
export class UINotification {
public icon: NotificationIcon;
public isRead: boolean;
public id: string;
public senderId: string;
public type: NotificationTypes;
public title: string;
public message: string;
public readAt: Date | null;
public createdAt: Date;
public constructor(
public id: string = '',
public senderId: string = '',
public type: NotificationTypes = NotificationTypes.Custom,
public title: string = '',
public message: string = '',
public isRead: boolean = false,
public createdAt: Date = new Date(),
) {
public constructor(notification: Partial<UINotification> = new Notification()) {
Object.assign(this, notification);
this.setIcon();
this.isRead = !!this.readAt;
}
/**
@ -70,60 +86,3 @@ export class Notification {
}
}
}
/**
* Describes all current notifications types.
*/
export enum NotificationTypes {
Custom = 0,
AuditCheckFailure = 1,
UptimeCheckFailure = 2,
Disqualification = 3,
Suspension = 4,
}
/**
* Describes page offset for pagination.
*/
export class NotificationsCursor {
public constructor(
public page: number = 0,
public limit: number = 7,
) { }
}
/**
* Holds all notifications module state.
*/
export class NotificationsState {
public latestNotifications: Notification[] = [];
public constructor(
public notifications: Notification[] = [],
public pageCount: number = 0,
public unreadCount: number = 0,
) { }
}
/**
* Exposes all notifications-related functionality.
*/
export interface NotificationsApi {
/**
* Fetches notifications.
* @throws Error
*/
get(cursor: NotificationsCursor): Promise<NotificationsState>;
/**
* Marks single notification as read.
* @throws Error
*/
read(id: string): Promise<void>;
/**
* Marks all notification as read.
* @throws Error
*/
readAll(): Promise<void>;
}

View File

@ -20,7 +20,6 @@ import { APPSTATE_ACTIONS } from '@/app/store/modules/appState';
import { NODE_ACTIONS } from '@/app/store/modules/node';
import { NOTIFICATIONS_ACTIONS } from '@/app/store/modules/notifications';
import { PAYOUT_ACTIONS } from '@/app/store/modules/payout';
import { NotificationsCursor } from '@/app/types/notifications';
@Component ({
components: {
@ -43,7 +42,7 @@ export default class Dashboard extends Vue {
}
try {
await this.$store.dispatch(NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS, new NotificationsCursor(1));
await this.$store.dispatch(NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS, 1);
} catch (error) {
console.error(error);
}

View File

@ -54,7 +54,7 @@ import VPagination from '@/app/components/VPagination.vue';
import BackArrowIcon from '@/../static/images/notifications/backArrow.svg';
import { NOTIFICATIONS_ACTIONS } from '@/app/store/modules/notifications';
import { Notification, NotificationsCursor } from '@/app/types/notifications';
import { UINotification } from '@/app/types/notifications';
@Component ({
components: {
@ -67,7 +67,7 @@ export default class NotificationsArea extends Vue {
/**
* Returns notification of current page.
*/
public get notifications(): Notification[] {
public get notifications(): UINotification[] {
return this.$store.state.notificationsModule.notifications;
}
@ -92,7 +92,7 @@ export default class NotificationsArea extends Vue {
*/
public async onPageClick(index: number): Promise<void> {
try {
await this.$store.dispatch(NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS, new NotificationsCursor(index));
await this.$store.dispatch(NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS, index);
} catch (error) {
console.error(error.message);
}

View File

@ -57,7 +57,6 @@ import { APPSTATE_ACTIONS } from '@/app/store/modules/appState';
import { NODE_ACTIONS } from '@/app/store/modules/node';
import { NOTIFICATIONS_ACTIONS } from '@/app/store/modules/notifications';
import { PAYOUT_ACTIONS } from '@/app/store/modules/payout';
import { NotificationsCursor } from '@/app/types/notifications';
import { PayoutPeriod, TotalHeldAndPaid } from '@/storagenode/payouts/payouts';
@Component ({
@ -88,7 +87,7 @@ export default class PayoutArea extends Vue {
}
try {
await this.$store.dispatch(NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS, new NotificationsCursor(1));
await this.$store.dispatch(NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS, 1);
} catch (error) {
console.error(error);
}

View File

@ -1,7 +1,12 @@
// Copyright (C) 2019 Storj Labs, Inc.
// See LICENSE for copying information.
import { Notification, NotificationsApi, NotificationsCursor, NotificationsState } from '@/app/types/notifications';
import {
NotificationsApi,
NotificationsCursor,
NotificationsPage,
NotificationsResponse,
} from '@/storagenode/notifications/notifications';
import { HttpClient } from '@/storagenode/utils/httpClient';
/**
@ -15,10 +20,10 @@ export class NotificationsHttpApi implements NotificationsApi {
/**
* Fetch notifications.
*
* @returns notifications state
* @returns notifications response.
* @throws Error
*/
public async get(cursor: NotificationsCursor): Promise<NotificationsState> {
public async get(cursor: NotificationsCursor): Promise<NotificationsResponse> {
const path = `${this.ROOT_PATH}/list?page=${cursor.page}&limit=${cursor.limit}`;
const response = await this.client.get(path);
@ -27,28 +32,12 @@ export class NotificationsHttpApi implements NotificationsApi {
}
const notificationResponse = await response.json();
let notifications: Notification[] = [];
let pageCount: number = 0;
let unreadCount: number = 0;
if (notificationResponse) {
notifications = notificationResponse.page.notifications.map(item =>
new Notification(
item.id,
item.senderId,
item.type,
item.title,
item.message,
!!item.readAt,
new Date(item.createdAt),
),
);
pageCount = notificationResponse.page.pageCount;
unreadCount = notificationResponse.unreadCount;
}
return new NotificationsState(notifications, pageCount, unreadCount);
return new NotificationsResponse(
new NotificationsPage(notificationResponse.page.notifications, notificationResponse.page.pageCount),
notificationResponse.unreadCount,
notificationResponse.totalCount,
);
}
/**

View File

@ -0,0 +1,88 @@
// Copyright (C) 2020 Storj Labs, Inc.
// See LICENSE for copying information.
/**
* Exposes all notifications-related functionality.
*/
export interface NotificationsApi {
/**
* Fetches notifications.
* @throws Error
*/
get(cursor: NotificationsCursor): Promise<NotificationsResponse>;
/**
* Marks single notification as read.
* @throws Error
*/
read(id: string): Promise<void>;
/**
* Marks all notification as read.
* @throws Error
*/
readAll(): Promise<void>;
}
/**
* Describes notification entity.
*/
export class Notification {
public constructor(
public id: string = '',
public senderId: string = '',
public type: NotificationTypes = NotificationTypes.Custom,
public title: string = '',
public message: string = '',
public readAt: Date | null = null,
public createdAt: Date = new Date(),
) {}
}
/**
* Describes all current notifications types.
*/
export enum NotificationTypes {
Custom = 0,
AuditCheckFailure = 1,
UptimeCheckFailure = 2,
Disqualification = 3,
Suspension = 4,
}
/**
* Describes page offset for pagination.
*/
export class NotificationsCursor {
private DEFAULT_LIMIT: number = 7;
public constructor(
public page: number = 0,
public limit: number = 0,
) {
if (!this.limit) {
this.limit = this.DEFAULT_LIMIT;
}
}
}
/**
* Describes response object from server.
*/
export class NotificationsResponse {
public constructor(
public page: NotificationsPage = new NotificationsPage(),
public unreadCount: number = 0,
public totalCount: number = 0,
) {}
}
/**
* Describes page related notification information.
*/
export class NotificationsPage {
public constructor(
public notifications: Notification[] = [],
public pageCount: number = 0,
) {}
}

View File

@ -0,0 +1,43 @@
// Copyright (C) 2020 Storj Labs, Inc.
// See LICENSE for copying information.
import { NotificationsApi, NotificationsCursor, NotificationsResponse } from '@/storagenode/notifications/notifications';
/**
* PayoutService is used to store and handle node paystub information.
* PayoutService exposes a business logic related to payouts.
*/
export class NotificationsService {
private readonly api: NotificationsApi;
public constructor(api: NotificationsApi) {
this.api = api;
}
/**
* Fetch notifications.
*
* @returns notifications response.
* @throws Error
*/
public async notifications(index: number, limit?: number): Promise<NotificationsResponse> {
const cursor = new NotificationsCursor(index, limit);
return await this.api.get(cursor);
}
/**
* Marks single notification as read on server.
* @param id
*/
public async readSingeNotification(id: string): Promise<void> {
await this.api.read(id);
}
/**
* Marks all notifications as read on server.
*/
public async readAllNotifications(): Promise<void> {
await this.api.readAll();
}
}

Some files were not shown because too many files have changed in this diff Show More