Merge 'master' branch
Change-Id: I6070089128a150a4dd501bbc62a1f8b394aa643e
This commit is contained in:
Michal Niewrzal
parent
92f9251074
commit
7dde184cb5
4
.clabot
4
.clabot
@ -1,4 +1,5 @@
|
||||
{
|
||||
"message": "Thank you for your pull request and welcome to our community. We require contributors to sign our [Contributor License Agreement](https://docs.google.com/forms/d/e/1FAIpQLSdVzD5W8rx-J_jLaPuG31nbOzS8yhNIIu4yHvzonji6NeZ4ig/viewform), and we don't seem to have the users {{usersWithoutCLA}} on file. Once you have signed the CLA, please let us know, so we can manually review and add you to the approved contributors list.",
|
||||
"contributors": [
|
||||
"aleitner",
|
||||
"aligeti",
|
||||
@ -62,6 +63,7 @@
|
||||
"montyanderson",
|
||||
"sixcorners",
|
||||
"alexottoboni",
|
||||
"dominickmarino"
|
||||
"dominickmarino",
|
||||
"hectorj2f"
|
||||
]
|
||||
}
|
||||
|
||||
@ -39,7 +39,7 @@ presents challenges when revoking a macaroon.
|
||||
For example, if I hold the API key for Project A, I can create Macaroon A with a
|
||||
caveat that it can only read and write files within Bucket A. I can
|
||||
then share this macaroon with my own customer, Customer A. Customer A may then,
|
||||
if they wish, create Macaroon B which is futher caveated -- for example,
|
||||
if they wish, create Macaroon B which is further caveated -- for example,
|
||||
restricted to read-only access in Bucket A -- and share Macaroon B with someone
|
||||
else. This can occur without my knowledge.
|
||||
|
||||
@ -94,7 +94,7 @@ This approach was deemed best because it:
|
||||
- Creates very little load on the database.
|
||||
- Is backwards compatible, and allows us to revoke existing macaroons.
|
||||
- Allows us to revoke an entire "macaroon tree" while maintaining the
|
||||
distributive properies of macaroons.
|
||||
distributive properties of macaroons.
|
||||
|
||||
Disadvantages to this approach:
|
||||
|
||||
|
||||
@ -30,7 +30,7 @@ When selecting nodes to store files to, the following criteria must be met:
|
||||
- the node is not disqualified
|
||||
- the node is not suspended
|
||||
- the node has not exited
|
||||
- the node has sufficent free disk space
|
||||
- the node has sufficient free disk space
|
||||
- the node has been contacted recently
|
||||
- the node has participated in a sufficient number of audit
|
||||
- the nodes has sufficient uptime counts
|
||||
@ -102,7 +102,7 @@ For now, lets try out using an in-memory cache for the performanace gains. If we
|
||||
|
||||
3) Using a postgres materialized view for cached node data
|
||||
|
||||
Using a postgres materialized view to store all the vetted and unvetted nodes would allow us to implement this cache in the database layer instead of the application layer. This would require the application code to handle the refresh of the materialized view which could occur when one of the events from the #Update section happened.
|
||||
Using a postgres materialized view to store all the vetted and unvetted nodes would allow us to implement this cache in the database layer instead of the application layer. This would require the application code to handle the refresh of the materialized view which could occur when one of the events from the #Update section happened.
|
||||
|
||||
Pro:
|
||||
- allows the db to handle the logic instead of adding a cache at the application layer
|
||||
|
||||
@ -13,7 +13,7 @@ The idea is to have two main components, like for Windows: the storage node bina
|
||||
The parts we need:
|
||||
- storagenode as a service
|
||||
- a system for updating the storagenode binary aka. the updater (with rollout versioning support)
|
||||
- a system for updating the updater
|
||||
- a system for updating the updater
|
||||
- a way to collect the configuration data from the user during the installation
|
||||
- packaging to ship the above
|
||||
|
||||
@ -36,20 +36,20 @@ The installer will be a debian package. We choose to auto-update the binary, eve
|
||||
- Email
|
||||
- External address/port
|
||||
- Advertised storage
|
||||
- Identity directory
|
||||
- Identity directory
|
||||
- Storage directory
|
||||
- Generate `config.yaml` file with the user configuration.
|
||||
|
||||
The default value for these directories can be defined using the [XDG Base Directory](https://wiki.archlinux.org/index.php/XDG_Base_Directory).
|
||||
|
||||
We choose to reuse the storagenode-updater and the recovery mechanism used in windows. They will be daemonized using systemd. The storagenode updater will auto-update. A recovery will be triggered if the updated updater service fails to restart.
|
||||
We will use debconf to retrieve user data.
|
||||
We will use debconf to retrieve user data.
|
||||
|
||||
The debian package will NOT contain the storagenode and storagenode-updater binaries. They will be downloaded as part of the post-installation script. A separate git repository will be created for holding the debian package.
|
||||
|
||||
Once we get a fully working debian package, we can convert it to the RPM format using the fpm tool. There are no debconf-like for RPMs, we will need to implement a post-install script to gather user inputs.
|
||||
|
||||
The debian package will be available by direct download and on a APT repository that users can add to their package manager source list. The repository will be managed using reprepro. Each time the repository is modified, it commits the static content to a dedicated git repository.
|
||||
The debian package will be available by direct download and on a APT repository that users can add to their package manager source list. The repository will be managed using reprepro. Each time the repository is modified, it commits the static content to a dedicated git repository.
|
||||
|
||||
## Rationale
|
||||
|
||||
@ -62,7 +62,7 @@ Hence, we should use systemd for building our storagenode service.
|
||||
Packaging in its simplest form would be tar.gz with an installation binary. This solution would be simple for us, but represents an annoyance for the user as our application would not be managed by their package manager.
|
||||
|
||||
#### Packages
|
||||
A package is an archive file containing the application and metadata for indicating to the package manager how to install it.
|
||||
A package is an archive file containing the application and metadata for indicating to the package manager how to install it.
|
||||
Its format depends on the used package manager.
|
||||
The most common formats are:
|
||||
- deb for debian-based distributions
|
||||
@ -79,7 +79,7 @@ The process for building a package is as follows:
|
||||
- make a source package
|
||||
- compile it to get binary packages.
|
||||
|
||||
Only the binary package is used by the user for installation. It is not a recommended pratice to directly integrate binaries.
|
||||
Only the binary package is used by the user for installation. It is not a recommended practice to directly integrate binaries.
|
||||
|
||||
Building the source package is the most difficult part. But once it is done, we can use tools such as [fpm](https://github.com/jordansissel/fpm/wiki) to convert it to other package formats.
|
||||
|
||||
@ -100,7 +100,7 @@ There are [3 major agnostic packaging system](https://www.ostechnix.com/linux-pa
|
||||
##### Snap
|
||||
[Snaps](https://snapcraft.io/first-snap#go) are containerised software packages. They auto-update daily and work on a variety of Linux distributions. They also revert to the previous version if an update fails. This feature would make it necessary to find out how to implement the rollout versioning.
|
||||
|
||||
From the [snap documentation](https://snapcraft.io/docs/go-applications), it seems pretty straightforward to package an application. Snaps are defined in a yaml file. Running an application as a service is done only by specifying "daemon: simple" in the application description.
|
||||
From the [snap documentation](https://snapcraft.io/docs/go-applications), it seems pretty straightforward to package an application. Snaps are defined in a yaml file. Running an application as a service is done only by specifying "daemon: simple" in the application description.
|
||||
This would make us save the work of building a storage node service.
|
||||
|
||||
Snaps can then be published in the snapcraft [app store](https://snapcraft.io/). In the store, we would able to monitor the number of installed snaps. It is possible to [host our own store](https://ubuntu.com/blog/howto-host-your-own-snap-store) but that the snap daemon only handles one repository. Therefore, the use of Canonical's store seems mandatory. Snaps integrate well with [github](https://snapcraft.io/build).
|
||||
@ -137,7 +137,7 @@ We are thinking of using native packaging for the following reasons:
|
||||
- some linux users are reluctant to use snap
|
||||
- covering deb and rpm packaging would make us cover most used distributions
|
||||
- with proper packaging, we could directly be included in the distributions
|
||||
|
||||
|
||||
## Implementation
|
||||
### Debian package
|
||||
- create a storj debian git
|
||||
@ -184,4 +184,4 @@ We still need to support docker images. The Docker image we provide should make
|
||||
## Wrapup
|
||||
- As a first step and as part of the PoC, the git repository and the debian package skeleton will be created.
|
||||
- The PoC will create the user and the directories, download a binary (will not check for the latest) and install a basic storagenode systemd service.
|
||||
- The PoC will also contain first Dockerfile for the reprepro repository.
|
||||
- The PoC will also contain first Dockerfile for the reprepro repository.
|
||||
|
||||
@ -15,7 +15,7 @@ That leaves open the question for how a root key is created. Some requirements o
|
||||
|
||||
These requirements allow users to be in full control of their encryption, and don't require users safely transporting high entropy (hard to remember) secrets to bootstrap new uplinks.
|
||||
|
||||
This design accomodates more requirements that allow for additional features:
|
||||
This design accommodates more requirements that allow for additional features:
|
||||
|
||||
3. A root key can be created for any encrypted path in a bucket, not just the bucket.
|
||||
4. A table of root keys for low entropy passwords should not be possible. In other words, an attacker with knowledge of the algorithm should not be able to use a dictionary of common passwords and pre-compute what keys to check in the event of a data breach.
|
||||
|
||||
@ -9,9 +9,9 @@ The satellite should repair files also using piece hashes to minimize CPU and ba
|
||||
The white-paper states:
|
||||
|
||||
> Data repair is an ongoing, costly operation that will use significant bandwidth, memory, and processing power, often impacting a single operator. As a result, repair resource usage should be aggressively minimized as much as possible.
|
||||
>
|
||||
>
|
||||
> For repairing a segment to be effective at minimizing bandwidth usage, only as few pieces as needed for reconstruction should be downloaded. Unfortunately, Reed-Solomon is insufficient on its own for correcting errors when only a few redundant pieces are provided. Instead, piece hashes provide a better way to be confident that we’re repairing the data correctly.
|
||||
>
|
||||
>
|
||||
> To solve this problem, hashes of every piece will be stored alongside each piece on each storage node. A validation hash that the set of hashes is correct will be stored in the pointer. During repair, the hashes of every piece can be retrieved and validated for correctness against the pointer, thus allowing each piece to be validated in its entirety. This allows the repair system to correctly assess whether or not repair has been completed successfully without using extra redundancy for the same task.
|
||||
|
||||
Hash verification on the satellite requires understanding the current piece signing and verification workflow:
|
||||
@ -41,7 +41,7 @@ Downloading for repair is significantly different enough from streaming as to wa
|
||||
|
||||
Using only the minimum number of pieces means that Reed-Solomon does not act as a check during repair. Hence hashing is used instead. While an uplink could potentially send signed bogus data to a storage node, the storage node would not be penalized by these actions. This requires that Audit implements a similar piece hash check instead of relying solely on Reed-Solomon encoding.
|
||||
|
||||
The size of all piece hashes downloaded should be roughly equal to a default maximum segment size : 64MiB. It seems preferable to keep this in memory over dealing with persistance to disk.
|
||||
The size of all piece hashes downloaded should be roughly equal to a default maximum segment size : 64MiB. It seems preferable to keep this in memory over dealing with persistence to disk.
|
||||
|
||||
## Implementation
|
||||
|
||||
|
||||
@ -2,7 +2,7 @@
|
||||
Satellite billing system combines stripe and coinpayments API for credit card and cryptocurrency processing. It uses `satellite/accounting` pkg for project accounting. Billing is set on account but that is the subject for future changes as we want billing to be on a project level. That requires decoupling stripe dependency to the level where we utilize only credit card processing and maintain all other stuff such as customer balances and invoicing internally. Every satellite should have separate stripe and coinpayments account to prevent collision of customer related data such as uuid and email.
|
||||
|
||||
# Stripe customer
|
||||
Stripe operates on a basis of customers. Where customer is, from stripe doc: Customer objects allow you to perform recurring charges, and to track multiple charges, that are associated with the same customer. The API allows you to create, delete, and update your customers. You can retrieve individual customers as well as a list of all your customers. Satellite billing doesn't uses `customer` concern with public API, so it is treated as implementation detail. Stripe customer balance is automatically applied to invoice total before charging a credit card.
|
||||
Stripe operates on a basis of customers. Where customer is, from stripe doc: Customer objects allow you to perform recurring charges, and to track multiple charges, that are associated with the same customer. The API allows you to create, delete, and update your customers. You can retrieve individual customers as well as a list of all your customers. Satellite billing doesn't uses `customer` concern with public API, so it is treated as implementation detail. Stripe customer balance is automatically applied to invoice total before charging a credit card.
|
||||
|
||||
Stripe billing system implementation stores a customer reference for every user:
|
||||
```
|
||||
@ -51,7 +51,7 @@ type Accounts interface {
|
||||
```
|
||||
|
||||
# Customer setup
|
||||
Every satellite user has a corresponding customer entity on stripe which holds credit cards, balance which reflects the ammount of STORJ tokens, and is used for invoicing. Every time a user visits billing page on the satellite UI we try to create a customer for him if one doesn't exists.
|
||||
Every satellite user has a corresponding customer entity on stripe which holds credit cards, balance which reflects the amount of STORJ tokens, and is used for invoicing. Every time a user visits billing page on the satellite UI we try to create a customer for him if one doesn't exists.
|
||||
```go
|
||||
// Setup creates a payment account for the user.
|
||||
// If account is already set up it will return nil.
|
||||
@ -241,7 +241,7 @@ type Chore struct {
|
||||
```
|
||||
|
||||
# STORJ tokens processsing
|
||||
Unlike with credit cards billing system uses deposit model for STORJ tokens, user has to deposit some amount prior using satellite services.
|
||||
Unlike with credit cards billing system uses deposit model for STORJ tokens, user has to deposit some amount prior using satellite services.
|
||||
|
||||
Public API of token related billing:
|
||||
```go
|
||||
@ -257,7 +257,7 @@ type StorjTokens interface {
|
||||
```
|
||||
|
||||
# Making a deposit
|
||||
STORJ cryptocurrency processing is done via coinpayments API. Every time a user wants to deposit some amount of STORJ token to his account balacne, new coinpayments transaction is created. Transaction amount is set in USD, and conversion rates is beeing locked(saved) after transaction is created and stored in the db.
|
||||
STORJ cryptocurrency processing is done via coinpayments API. Every time a user wants to deposit some amount of STORJ token to his account balacne, new coinpayments transaction is created. Transaction amount is set in USD, and conversion rates is being locked(saved) after transaction is created and stored in the db.
|
||||
```go
|
||||
// Deposit creates new deposit transaction with the given amount returning
|
||||
// ETH wallet address where funds should be sent. There is one
|
||||
@ -403,7 +403,7 @@ func (tokens *storjTokens) ListTransactionInfos(ctx context.Context, userID uuid
|
||||
```
|
||||
|
||||
# Transaction update cycle
|
||||
There is a cycle that iterates over all `pending`(`pending` and `paid` statuses of coinpayments transaction respectively) transactions, list it's infos and updates tx status and received amount. If updated is status is set to `cancelled` or `completed`, that transactions won't take part in the next update cycle. When there is a status transation to `completed` along with the update `apply_balance_transaction_intent` is created. Transaction with status `completed` and present `apply_balance_transaction_intent` with state `unapplied` defines as `UnappliedTransaction` which is later processed in update balance cycle. If the received amount is greater that 50$ a promotional coupon for 55$ is created.
|
||||
There is a cycle that iterates over all `pending`(`pending` and `paid` statuses of coinpayments transaction respectively) transactions, list it's infos and updates tx status and received amount. If updated is status is set to `cancelled` or `completed`, that transactions won't take part in the next update cycle. When there is a status transaction to `completed` along with the update `apply_balance_transaction_intent` is created. Transaction with status `completed` and present `apply_balance_transaction_intent` with state `unapplied` defines as `UnappliedTransaction` which is later processed in update balance cycle. If the received amount is greater that 50$ a promotional coupon for 55$ is created.
|
||||
```go
|
||||
// updateTransactions updates statuses and received amount for given transactions.
|
||||
func (service *Service) updateTransactions(ctx context.Context, ids TransactionAndUserList) (err error) {
|
||||
@ -501,7 +501,7 @@ func (service *Service) applyTransactionBalance(ctx context.Context, tx Transact
|
||||
```
|
||||
|
||||
# Invoices
|
||||
Invoices are statements of amounts owed by a customer, and are generated one-off.
|
||||
Invoices are statements of amounts owed by a customer, and are generated one-off.
|
||||
```go
|
||||
// Invoice holds all public information about invoice.
|
||||
type Invoice struct {
|
||||
@ -526,7 +526,7 @@ type Invoices interface {
|
||||
```
|
||||
|
||||
# Invoice creation
|
||||
Invoice include project usage cost as well as any discounts applied. Coupons and credits applied as separate invoice line items, therefore it reduce total due amount. Next applied STORJ token amount which is repesented as credits on custmer balance if any. If invoice total amount is greater than zero after bonuses and STORJ tokens, default credit card at the moment of invoice creation will be charged. If total amount is less than 1$, then stripe won't try to charge credit card but increase debt on customer balance.
|
||||
Invoice include project usage cost as well as any discounts applied. Coupons and credits applied as separate invoice line items, therefore it reduce total due amount. Next applied STORJ token amount which is repesented as credits on custmer balance if any. If invoice total amount is greater than zero after bonuses and STORJ tokens, default credit card at the moment of invoice creation will be charged. If total amount is less than 1$, then stripe won't try to charge credit card but increase debt on customer balance.
|
||||
|
||||
Invoice creation consist of few steps. First invoice project records have to be created. Each record consist of project id, usage and timestamps of the start and end of billing period. This way we ensure that usage is the same during all invoice creation steps and there won't be two or more invoices created for the same period(actually only invoice line items for certain billing period and project are ensured not to be created more than once). Coupon usages are also created during this step, which are later used to create coupon invoice line items.
|
||||
|
||||
@ -548,7 +548,7 @@ prepare-invoice-records Prepares invoice project records that will be used durin
|
||||
```bash
|
||||
inspector payments prepare-invoice-records [mm/yyyy]
|
||||
```
|
||||
Create project records for all projects for specified billing period. Billing period defined as `[0th nanosecond of the first day of the month; 0th nanosecond of the first day of the following month)`.
|
||||
Create project records for all projects for specified billing period. Billing period defined as `[0th nanosecond of the first day of the month; 0th nanosecond of the first day of the following month)`.
|
||||
Project record contains project usage for some billing period. Therefore, it is impossible to create project record for the same project and billing period.
|
||||
```go
|
||||
// ProjectRecord holds project usage particular for billing period.
|
||||
@ -608,7 +608,7 @@ func (service *Service) PrepareInvoiceProjectRecords(ctx context.Context, period
|
||||
return nil
|
||||
}
|
||||
```
|
||||
If a project record already exists, project is skipped.
|
||||
If a project record already exists, project is skipped.
|
||||
```go
|
||||
// createProjectRecords creates invoice project record if none exists.
|
||||
func (service *Service) createProjectRecords(ctx context.Context, projects []console.Project, start, end time.Time) (err error) {
|
||||
@ -696,31 +696,31 @@ Iterate over all project records, calculating price and creating invoice line it
|
||||
// applyProjectRecords applies invoice intents as invoice line items to stripe customer.
|
||||
func (service *Service) applyProjectRecords(ctx context.Context, records []ProjectRecord) (err error) {
|
||||
defer mon.Task()(&ctx)(&err)
|
||||
|
||||
|
||||
for _, record := range records {
|
||||
if err = ctx.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
|
||||
proj, err := service.projectsDB.Get(ctx, record.ProjectID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
|
||||
cusID, err := service.db.Customers().GetCustomerID(ctx, proj.OwnerID)
|
||||
if err != nil {
|
||||
if err == ErrNoCustomer {
|
||||
continue
|
||||
}
|
||||
|
||||
|
||||
return err
|
||||
}
|
||||
|
||||
|
||||
if err = service.createInvoiceItems(ctx, cusID, proj.Name, record); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
return nil
|
||||
}
|
||||
```
|
||||
@ -763,46 +763,46 @@ Iterate over all customers and create invoice for each.
|
||||
// CreateInvoices lists through all customers and creates invoices.
|
||||
func (service *Service) CreateInvoices(ctx context.Context) (err error) {
|
||||
defer mon.Task()(&ctx)(&err)
|
||||
|
||||
|
||||
const limit = 25
|
||||
before := time.Now()
|
||||
|
||||
|
||||
cusPage, err := service.db.Customers().List(ctx, 0, limit, before)
|
||||
if err != nil {
|
||||
return Error.Wrap(err)
|
||||
}
|
||||
|
||||
|
||||
for _, cus := range cusPage.Customers {
|
||||
if err = ctx.Err(); err != nil {
|
||||
return Error.Wrap(err)
|
||||
}
|
||||
|
||||
|
||||
if err = service.createInvoice(ctx, cus.ID); err != nil {
|
||||
return Error.Wrap(err)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
for cusPage.Next {
|
||||
if err = ctx.Err(); err != nil {
|
||||
return Error.Wrap(err)
|
||||
}
|
||||
|
||||
|
||||
cusPage, err = service.db.Customers().List(ctx, cusPage.NextOffset, limit, before)
|
||||
if err != nil {
|
||||
return Error.Wrap(err)
|
||||
}
|
||||
|
||||
|
||||
for _, cus := range cusPage.Customers {
|
||||
if err = ctx.Err(); err != nil {
|
||||
return Error.Wrap(err)
|
||||
}
|
||||
|
||||
|
||||
if err = service.createInvoice(ctx, cus.ID); err != nil {
|
||||
return Error.Wrap(err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
return nil
|
||||
}
|
||||
```
|
||||
@ -833,4 +833,4 @@ func (service *Service) createInvoice(ctx context.Context, cusID string) (err er
|
||||
|
||||
return nil
|
||||
}
|
||||
```
|
||||
```
|
||||
|
||||
@ -274,7 +274,7 @@ and are left with the following:
|
||||
|
||||
The list of trusted Satellite URLs should be recalculated daily (with some jitter).
|
||||
|
||||
### Backwards Compatability
|
||||
### Backwards Compatibility
|
||||
|
||||
The old piecestore configuration (i.e. `piecestore.OldConfig`) currently contains a
|
||||
comma separated list of trusted Satellite URLs (`WhitelistedSatellites`). It
|
||||
@ -296,7 +296,7 @@ a fixed set of trusted Satellite URLs.
|
||||
* Implement a `trust.ListConfig` configuration struct which:
|
||||
* Contains the list of entries (with a release default of a single list containing `https://www.tardigrade.io/trusted-satellites`)
|
||||
* Contains a refresh interval
|
||||
* Maintains backwards compatability with `WhitelistedSatellites` in `piecestore.OldConfig`
|
||||
* Maintains backwards compatibility with `WhitelistedSatellites` in `piecestore.OldConfig`
|
||||
* Implement `storj.io/storj/storagenode/trust.List` that:
|
||||
* Consumes `trust.ListConfig` for configuration
|
||||
* Performs the initial fetching and building of trusted Satellite URLs
|
||||
|
||||
@ -87,7 +87,7 @@ Create `satellites_exit_progress` tables:
|
||||
|
||||
```
|
||||
model satellite_exit_progress (
|
||||
fk satellite_id
|
||||
fk satellite_id
|
||||
|
||||
field initiated_at timestamp ( updateable )
|
||||
field finished_at timestamp ( updateable )
|
||||
|
||||
@ -8,7 +8,7 @@ This document describes how storage node transfers its pieces during Graceful Ex
|
||||
|
||||
## Background
|
||||
|
||||
During Graceful Exit a storage node needs to transfer pieces to other nodes. During transfering the storage node or satellite may crash, hence it needs to be able to continue after a restart.
|
||||
During Graceful Exit a storage node needs to transfer pieces to other nodes. During transferring the storage node or satellite may crash, hence it needs to be able to continue after a restart.
|
||||
|
||||
Satellite gathers transferred pieces list asynchronously, which is described in [Gathering Pieces Document](pieces.md). This may consume a significant amount of time.
|
||||
|
||||
@ -28,11 +28,11 @@ The `worker` should continue to poll the satellite at a configurable interval un
|
||||
|
||||
The satellite should return pieces to transfer from the transfer queue if piece durability <= optimal. If durability > optimal, we remove the exiting node from the segment / pointer.
|
||||
|
||||
The storage node should concurrently transfer pieces returned by the satellite. The storage node should send a `TransferSucceeded` message as pieces are successfuly transfered. The Storage node should send a `TransferFailed`, with reason, on failure.
|
||||
The storage node should concurrently transfer pieces returned by the satellite. The storage node should send a `TransferSucceeded` message as pieces are successfully transferred. The Storage node should send a `TransferFailed`, with reason, on failure.
|
||||
|
||||
The satellites should set the `finished_at` on success, and respond with a `DeletePiece` message. Otherwise increment `failed_count` and set the `last_failed_at` and `last_failed_code` for reprocessing.
|
||||
|
||||
The satellite should respond with an `ExitCompleted` message when all pieces have finished processing.
|
||||
The satellite should respond with an `ExitCompleted` message when all pieces have finished processing.
|
||||
|
||||
If the storage node has failed too many transfers overall, failed the same piece over a certain threshold, or has sent incorrect data, the satellite will send an `ExitFailed` message. This indicates that the process has ended ungracefully.
|
||||
|
||||
@ -75,7 +75,7 @@ We could have a separate initiate graceful exit RPC, however this would complica
|
||||
## Implementation
|
||||
|
||||
1. Add protobuf definitions.
|
||||
2. Update node selection to ignore exiting nodes for repairs and uploads.
|
||||
2. Update node selection to ignore exiting nodes for repairs and uploads.
|
||||
3. Update repairer to repair segments for nodes that failed an exit.
|
||||
4. Implement verifying a transfer on the satellite.
|
||||
5. Implement transferring a single piece on storage node.
|
||||
@ -143,7 +143,7 @@ when storage node prematurely exits
|
||||
go func() {
|
||||
for {
|
||||
ensure we have only up to N inprogress at the same time
|
||||
|
||||
|
||||
list transferred piece that is not in progress
|
||||
if no pieces {
|
||||
morepieces = false
|
||||
|
||||
7
go.mod
7
go.mod
@ -38,14 +38,13 @@ require (
|
||||
go.etcd.io/bbolt v1.3.5
|
||||
go.uber.org/zap v1.16.0
|
||||
golang.org/x/crypto v0.0.0-20200820211705-5c72a883971a
|
||||
golang.org/x/net v0.0.0-20200226121028-0de0cce0169b // indirect
|
||||
golang.org/x/sync v0.0.0-20200625203802-6e8e738ad208
|
||||
golang.org/x/sys v0.0.0-20200929083018-4d22bbb62b3c
|
||||
golang.org/x/time v0.0.0-20200630173020-3af7569d3a1e
|
||||
google.golang.org/api v0.20.0 // indirect
|
||||
storj.io/common v0.0.0-20201030120157-90ae6720d87e
|
||||
storj.io/drpc v0.0.14
|
||||
storj.io/common v0.0.0-20201106104920-372a344bdd45
|
||||
storj.io/drpc v0.0.16
|
||||
storj.io/monkit-jaeger v0.0.0-20200518165323-80778fc3f91b
|
||||
storj.io/private v0.0.0-20201026143115-bc926bfa3bca
|
||||
storj.io/uplink v1.3.2-0.20201028181609-f6efc8fcf771
|
||||
storj.io/uplink v1.3.2-0.20201106105834-ec8e5cc29f7e
|
||||
)
|
||||
|
||||
154
go.sum
154
go.sum
@ -1,5 +1,7 @@
|
||||
cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
|
||||
cloud.google.com/go v0.31.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
|
||||
cloud.google.com/go v0.34.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
|
||||
cloud.google.com/go v0.37.0/go.mod h1:TS1dMSSfndXH133OKGwekG838Om/cQT0BUHV3HcBgoo=
|
||||
cloud.google.com/go v0.37.4/go.mod h1:NHPJ89PdicEuT9hdPXMROBD91xc5uRDxsMtSB16k7hw=
|
||||
cloud.google.com/go v0.38.0/go.mod h1:990N+gfupTy94rShfmMCWGDn0LpTmnzTp2qbd1dvSRU=
|
||||
cloud.google.com/go v0.44.1/go.mod h1:iSa0KzasP4Uvy3f1mN/7PiObzGgflwredwwASm/v6AU=
|
||||
@ -17,7 +19,12 @@ cloud.google.com/go/pubsub v1.0.1 h1:W9tAK3E57P75u0XLLR82LZyw8VpAnhmyTOxW9qzmyj8
|
||||
cloud.google.com/go/pubsub v1.0.1/go.mod h1:R0Gpsv3s54REJCy4fxDixWD93lHJMoZTyQ2kNxGRt3I=
|
||||
cloud.google.com/go/storage v1.0.0 h1:VV2nUM3wwLLGh9lSABFgZMjInyUbJeaRSE64WuAIQ+4=
|
||||
cloud.google.com/go/storage v1.0.0/go.mod h1:IhtSnM/ZTZV8YYJWCY8RULGVqBDmpoyjwiyrjsg+URw=
|
||||
dmitri.shuralyov.com/app/changes v0.0.0-20180602232624-0a106ad413e3/go.mod h1:Yl+fi1br7+Rr3LqpNJf1/uxUdtRUV+Tnj0o93V2B9MU=
|
||||
dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU=
|
||||
dmitri.shuralyov.com/html/belt v0.0.0-20180602232347-f7d459c86be0/go.mod h1:JLBrvjyP0v+ecvNYvCpyZgu5/xkfAUhi6wJj28eUfSU=
|
||||
dmitri.shuralyov.com/service/change v0.0.0-20181023043359-a85b471d5412/go.mod h1:a1inKt/atXimZ4Mv927x+r7UpyzRUf4emIoiiSC2TN4=
|
||||
dmitri.shuralyov.com/state v0.0.0-20180228185332-28bcc343414c/go.mod h1:0PRwlb0D6DFvNNtx+9ybjezNCa8XF0xaYcETyp6rHWU=
|
||||
git.apache.org/thrift.git v0.0.0-20180902110319-2566ecd5d999/go.mod h1:fPE2ZNJGynbRyZ4dJvy6G277gSllfV2HJqblrnkyeyg=
|
||||
github.com/Azure/go-ansiterm v0.0.0-20170929234023-d6e3b3328b78 h1:w+iIsaOQNcT7OZ575w+acHgRric5iCyQh+xv+KJ4HB8=
|
||||
github.com/Azure/go-ansiterm v0.0.0-20170929234023-d6e3b3328b78/go.mod h1:LmzpDX56iTiv29bbRTIsUNlaFfuhWRQBWjQdVyAevI8=
|
||||
github.com/BurntSushi/toml v0.3.1 h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ=
|
||||
@ -40,6 +47,7 @@ github.com/alicebob/gopher-json v0.0.0-20200520072559-a9ecdc9d1d3a h1:HbKu58rmZp
|
||||
github.com/alicebob/gopher-json v0.0.0-20200520072559-a9ecdc9d1d3a/go.mod h1:SGnFV6hVsYE877CKEZ6tDNTjaSXYUk6QqoIK6PrAtcc=
|
||||
github.com/alicebob/miniredis/v2 v2.13.3 h1:kohgdtN58KW/r9ZDVmMJE3MrfbumwsDQStd0LPAGmmw=
|
||||
github.com/alicebob/miniredis/v2 v2.13.3/go.mod h1:uS970Sw5Gs9/iK3yBg0l9Uj9s25wXxSpQUE9EaJ/Blg=
|
||||
github.com/anmitsu/go-shlex v0.0.0-20161002113705-648efa622239/go.mod h1:2FmKhYUyUczH0OGQWaF5ceTx0UBShxjsH6f8oGKYe2c=
|
||||
github.com/apache/thrift v0.12.0 h1:pODnxUFNcjP9UTLZGTdeh+j16A8lJbRvD3rOtrk/7bs=
|
||||
github.com/apache/thrift v0.12.0/go.mod h1:cp2SuWMxlEZw2r+iP2GNCdIi4C1qmUzdZFSVb+bacwQ=
|
||||
github.com/armon/circbuf v0.0.0-20150827004946-bbbad097214e/go.mod h1:3U/XgcO3hCbHZ8TKRvWD2dDTCfh9M9ya+I9JpbB7O8o=
|
||||
@ -55,6 +63,7 @@ github.com/bketelsen/crypt v0.0.3-0.20200106085610-5cbc8cc4026c/go.mod h1:MKsuJm
|
||||
github.com/blang/semver v3.5.1+incompatible h1:cQNTCjp13qL8KC3Nbxr/y2Bqb63oX6wdnnjpJbkM4JQ=
|
||||
github.com/blang/semver v3.5.1+incompatible/go.mod h1:kRBLl5iJ+tD4TcOOxsy/0fnwebNt5EWlYSAyrTnjyyk=
|
||||
github.com/bmizerany/assert v0.0.0-20160611221934-b7ed37b82869/go.mod h1:Ekp36dRnpXw/yCqJaO+ZrUyxD+3VXMFFr56k5XYrpB4=
|
||||
github.com/bradfitz/go-smtpd v0.0.0-20170404230938-deb6d6237625/go.mod h1:HYsPBTaaSFSlLx/70C2HPIMNZpVV8+vt/A+FMnYP11g=
|
||||
github.com/btcsuite/btcd v0.20.1-beta/go.mod h1:wVuoA8VJLEcwgqHBwHmzLRazpKxTv13Px/pDuV7OomQ=
|
||||
github.com/btcsuite/btclog v0.0.0-20170628155309-84c8d2346e9f/go.mod h1:TdznJufoqS23FtqVCzL0ZqgP5MqXbb4fg/WgDys70nA=
|
||||
github.com/btcsuite/btcutil v0.0.0-20190425235716-9e5f4b9a998d/go.mod h1:+5NJ2+qvTyV9exUAL/rxXi3DcLg2Ts+ymUAY5y4NvMg=
|
||||
@ -66,12 +75,15 @@ github.com/btcsuite/goleveldb v0.0.0-20160330041536-7834afc9e8cd/go.mod h1:F+uVa
|
||||
github.com/btcsuite/snappy-go v0.0.0-20151229074030-0bdef8d06723/go.mod h1:8woku9dyThutzjeg+3xrA5iCpBRH8XEEg3lh6TiUghc=
|
||||
github.com/btcsuite/websocket v0.0.0-20150119174127-31079b680792/go.mod h1:ghJtEyQwv5/p4Mg4C0fgbePVuGr935/5ddU9Z3TmDRY=
|
||||
github.com/btcsuite/winsvc v1.0.0/go.mod h1:jsenWakMcC0zFBFurPLEAyrnc/teJEM1O46fmI40EZs=
|
||||
github.com/buger/jsonparser v0.0.0-20181115193947-bf1c66bbce23/go.mod h1:bbYlZJ7hK1yFx9hf58LP0zeX7UjIGs20ufpu3evjr+s=
|
||||
github.com/calebcase/tmpfile v1.0.1/go.mod h1:iErLeG/iqJr8LaQ/gYRv4GXdqssi3jg4iSzvrA06/lw=
|
||||
github.com/calebcase/tmpfile v1.0.2-0.20200602150926-3af473ef8439/go.mod h1:iErLeG/iqJr8LaQ/gYRv4GXdqssi3jg4iSzvrA06/lw=
|
||||
github.com/calebcase/tmpfile v1.0.2 h1:1AGuhKiUu4J6wxz6lxuF6ck3f8G2kaV6KSEny0RGCig=
|
||||
github.com/calebcase/tmpfile v1.0.2/go.mod h1:iErLeG/iqJr8LaQ/gYRv4GXdqssi3jg4iSzvrA06/lw=
|
||||
github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
|
||||
github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc=
|
||||
github.com/cheekybits/genny v1.0.0 h1:uGGa4nei+j20rOSeDeP5Of12XVm7TGUd4dJA9RDitfE=
|
||||
github.com/cheekybits/genny v1.0.0/go.mod h1:+tQajlRqAUrPI7DOSpB0XAqZYtQakVtB7wXkRAgjxjQ=
|
||||
github.com/cheggaaa/pb/v3 v3.0.5 h1:lmZOti7CraK9RSjzExsY53+WWfub9Qv13B5m4ptEoPE=
|
||||
github.com/cheggaaa/pb/v3 v3.0.5/go.mod h1:X1L61/+36nz9bjIsrDU52qHKOQukUQe2Ge+YvGuquCw=
|
||||
github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI=
|
||||
@ -90,6 +102,7 @@ github.com/coreos/etcd v3.3.10+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc
|
||||
github.com/coreos/etcd v3.3.13+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE=
|
||||
github.com/coreos/go-semver v0.2.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk=
|
||||
github.com/coreos/go-semver v0.3.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk=
|
||||
github.com/coreos/go-systemd v0.0.0-20181012123002-c6f51f82210d/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4=
|
||||
github.com/coreos/go-systemd v0.0.0-20190321100706-95778dfbb74e/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4=
|
||||
github.com/coreos/go-systemd v0.0.0-20190719114852-fd7a80b32e1f/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4=
|
||||
github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f/go.mod h1:E3G3o1h8I7cfcXa63jLwjI0eiQQMgzzUDFVpN/nH/eA=
|
||||
@ -123,6 +136,7 @@ github.com/docker/go-connections v0.4.0 h1:El9xVISelRB7BuFusrZozjnkIM5YnzCViNKoh
|
||||
github.com/docker/go-connections v0.4.0/go.mod h1:Gbd7IOopHjR8Iph03tsViu4nIes5XhDvyHbTtUxmeec=
|
||||
github.com/docker/go-units v0.3.3 h1:Xk8S3Xj5sLGlG5g67hJmYMmUgXv5N4PhkjJHHqrwnTk=
|
||||
github.com/docker/go-units v0.3.3/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk=
|
||||
github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk=
|
||||
github.com/eapache/go-resiliency v1.1.0/go.mod h1:kFI+JgMyC7bLPUVY133qvEBtVayf5mFgVsvEsIPBvNs=
|
||||
github.com/eapache/go-xerial-snappy v0.0.0-20180814174437-776d5712da21/go.mod h1:+020luEh2TKB4/GOp8oxxtq0Daoen/Cii55CzbTV6DU=
|
||||
github.com/eapache/queue v1.1.0/go.mod h1:6eCeP0CKFpHLu8blIFXhExK/dRa7WDZfr6jVFPTqq+I=
|
||||
@ -132,10 +146,16 @@ github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7
|
||||
github.com/fatih/color v1.7.0/go.mod h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5KwzbycvMj4=
|
||||
github.com/fatih/color v1.9.0 h1:8xPHl4/q1VyqGIPif1F+1V3Y3lSmrq01EabUW3CoW5s=
|
||||
github.com/fatih/color v1.9.0/go.mod h1:eQcE1qtQxscV5RaZvpXrrb8Drkc3/DdQ+uUYCNjL+zU=
|
||||
github.com/flynn/go-shlex v0.0.0-20150515145356-3f9db97f8568/go.mod h1:xEzjJPgXI435gkrCt3MPfRiAkVrwSbHsst4LCFVfpJc=
|
||||
github.com/francoispqt/gojay v1.2.13/go.mod h1:ehT5mTG4ua4581f1++1WLG0vPdaA9HaiDsoyrBGkyDY=
|
||||
github.com/fsnotify/fsnotify v1.4.7 h1:IXs+QLmnXW2CcXuY+8Mzv/fWEsPGWxqefPtCP5CnV9I=
|
||||
github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
|
||||
github.com/fsnotify/fsnotify v1.4.9 h1:hsms1Qyu0jgnwNXIxa+/V/PDsU6CfLf6CNO8H7IWoS4=
|
||||
github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ=
|
||||
github.com/fsouza/fake-gcs-server v1.7.0/go.mod h1:5XIRs4YvwNbNoz+1JF8j6KLAyDh7RHGAyAK3EP2EsNk=
|
||||
github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
|
||||
github.com/gliderlabs/ssh v0.1.1/go.mod h1:U7qILu1NlMHj9FlMhZLlkCdDnU1DBEAqr0aevW3Awn0=
|
||||
github.com/go-errors/errors v1.0.1/go.mod h1:f4zRHt4oKfwPJE5k8C9vpYG+aDHdBFUsgrm6/TyX73Q=
|
||||
github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9AVAgeJqvqgH9Q5CA+iKCZ2gyEVpxRU=
|
||||
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
|
||||
github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
|
||||
@ -159,16 +179,28 @@ github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b h1:VKtxabqXZkF25pY9ekf
|
||||
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
|
||||
github.com/golang/groupcache v0.0.0-20190129154638-5b532d6fd5ef/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
|
||||
github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
|
||||
github.com/golang/groupcache v0.0.0-20191027212112-611e8accdfc9/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
|
||||
github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7 h1:5ZkaAPbicIKTF2I64qf5Fh8Aa83Q/dnOafMYV0OMwjA=
|
||||
github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
|
||||
github.com/golang/lint v0.0.0-20180702182130-06c8688daad7/go.mod h1:tluoj9z5200jBnyusfRPU2LqT6J+DAorxEvtC7LHB+E=
|
||||
github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
|
||||
github.com/golang/mock v1.2.0/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
|
||||
github.com/golang/mock v1.3.1 h1:qGJ6qTW+x6xX/my+8YUVl4WNpX9B7+/l2tRsHGZ7f2s=
|
||||
github.com/golang/mock v1.3.1/go.mod h1:sBzyDLLjw3U8JLTeZvSv8jJB+tU5PVekmnlKIyFUx0Y=
|
||||
github.com/golang/mock v1.4.0/go.mod h1:UOMv5ysSaYNkG+OFQykRIcU/QvvxJf3p21QfJ2Bt3cw=
|
||||
github.com/golang/mock v1.4.4 h1:l75CXGRSwbaYNpl/Z2X1XIIAMSCquvXgpVZDhwEIJsc=
|
||||
github.com/golang/mock v1.4.4/go.mod h1:l3mdAwkq5BuhzHwde/uurv3sEJeZMXNpwsxVWU71h+4=
|
||||
github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
|
||||
github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
|
||||
github.com/golang/protobuf v1.3.2 h1:6nsPYzhq5kReh6QImI3k5qWzO4PEbvbIW2cwSfR/6xs=
|
||||
github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
|
||||
github.com/golang/protobuf v1.4.0-rc.1/go.mod h1:ceaxUfeHdC40wWswd/P6IGgMaK3YpKi5j83Wpe3EHw8=
|
||||
github.com/golang/protobuf v1.4.0-rc.1.0.20200221234624-67d41d38c208/go.mod h1:xKAWHe0F5eneWXFV3EuXVDTCmh+JuBKY0li0aMyXATA=
|
||||
github.com/golang/protobuf v1.4.0-rc.2/go.mod h1:LlEzMj4AhA7rCAGe4KMBDvJI+AwstrUpVNzEA03Pprs=
|
||||
github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0/go.mod h1:WU3c8KckQ9AFe+yFwt9sWVRKCVIyN9cPHBJSNnbL67w=
|
||||
github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvqG2KuDX0=
|
||||
github.com/golang/protobuf v1.4.2 h1:+Z5KGCizgyZCbGh1KZqA0fcLLkwbsjIzS4aV2v7wJX0=
|
||||
github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI=
|
||||
github.com/golang/snappy v0.0.0-20170215233205-553a64147049/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
|
||||
github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
|
||||
github.com/golang/snappy v0.0.1/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
|
||||
@ -176,6 +208,7 @@ github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c/go.mod h1:lNA+9X1NB3Z
|
||||
github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
|
||||
github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M=
|
||||
github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
|
||||
github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
|
||||
github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
|
||||
github.com/google/go-cmp v0.5.2 h1:X2ev0eStA3AbceY54o37/0PQ/UWqKEiiO2dKL5OPaFM=
|
||||
github.com/google/go-cmp v0.5.2/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
|
||||
@ -189,6 +222,9 @@ github.com/google/pprof v0.0.0-20191218002539-d4f498aebedc/go.mod h1:ZgVRPoUq/hf
|
||||
github.com/google/pprof v0.0.0-20200229191704-1ebb73c60ed3 h1:SRgJV+IoxM5MKyFdlSUeNy6/ycRUF2yBAKdAQswoHUk=
|
||||
github.com/google/pprof v0.0.0-20200229191704-1ebb73c60ed3/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
|
||||
github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI=
|
||||
github.com/googleapis/gax-go v2.0.0+incompatible h1:j0GKcs05QVmm7yesiZq2+9cxHkNK9YM6zKx4D2qucQU=
|
||||
github.com/googleapis/gax-go v2.0.0+incompatible/go.mod h1:SFVmujtThgffbyetf+mdk2eWhX2bMyUtNHzFKcPA9HY=
|
||||
github.com/googleapis/gax-go/v2 v2.0.3/go.mod h1:LLvjysVCY1JZeum8Z6l8qUty8fiNwE08qbEPm1M08qg=
|
||||
github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg=
|
||||
github.com/googleapis/gax-go/v2 v2.0.5 h1:sjZBwGj9Jlw33ImPtvFviGYvseOtDM7hkSKB7+Tv3SM=
|
||||
github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk=
|
||||
@ -205,8 +241,10 @@ github.com/gorilla/websocket v1.4.0/go.mod h1:E7qHFY5m1UJ88s3WnNqhKjPHQ0heANvMoA
|
||||
github.com/gorilla/websocket v1.4.2/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
|
||||
github.com/graphql-go/graphql v0.7.9 h1:5Va/Rt4l5g3YjwDnid3vFfn43faaQBq7rMcIZ0VnV34=
|
||||
github.com/graphql-go/graphql v0.7.9/go.mod h1:k6yrAYQaSP59DC5UVxbgxESlmVyojThKdORUqGDGmrI=
|
||||
github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7/go.mod h1:FecbI9+v66THATjSRHfNgh1IVFe/9kFxbXtjV0ctIMA=
|
||||
github.com/grpc-ecosystem/go-grpc-middleware v1.0.0/go.mod h1:FiyG127CGDf3tlThmgyCl78X/SZQqEOJBCDaAfeWzPs=
|
||||
github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0/go.mod h1:8NvIoxWQoOIhqOTXgfV/d3M/q6VIi02HzZEHgUlZvzk=
|
||||
github.com/grpc-ecosystem/grpc-gateway v1.5.0/go.mod h1:RSKVYQBd5MCa4OVpNdGskqpgL2+G+NZTnrVHpWWfpdw=
|
||||
github.com/grpc-ecosystem/grpc-gateway v1.9.0/go.mod h1:vNeuVxBJEsws4ogUvrchl83t/GYV9WGTSLVdBhOQFDY=
|
||||
github.com/hailocab/go-hostpool v0.0.0-20160125115350-e80d13ce29ed/go.mod h1:tMWxXQ9wFIaZeTI9F+hmhFiGpFmhOHzyShyFUhRm0H4=
|
||||
github.com/hashicorp/consul/api v1.1.0/go.mod h1:VmuI/Lkw1nC05EYQWNKwWGbkg+FbDBtguAZLlVdkD9Q=
|
||||
@ -294,6 +332,7 @@ github.com/jackc/puddle v0.0.0-20190608224051-11cab39313c9/go.mod h1:m4B5Dj62Y0f
|
||||
github.com/jackc/puddle v1.1.0/go.mod h1:m4B5Dj62Y0fbyuIc15OsIqK0+JU8nkqQjsgx7dvjSWk=
|
||||
github.com/jackc/puddle v1.1.1/go.mod h1:m4B5Dj62Y0fbyuIc15OsIqK0+JU8nkqQjsgx7dvjSWk=
|
||||
github.com/jackc/puddle v1.1.2/go.mod h1:m4B5Dj62Y0fbyuIc15OsIqK0+JU8nkqQjsgx7dvjSWk=
|
||||
github.com/jellevandenhooff/dkim v0.0.0-20150330215556-f50fe3d243e1/go.mod h1:E0B/fFc00Y+Rasa88328GlI/XbtyysCtTHZS8h7IrBU=
|
||||
github.com/jessevdk/go-flags v0.0.0-20141203071132-1679536dcc89/go.mod h1:4FA24M0QyGHXBuZZK/XkWh8h0e1EYbRYJSGM75WSRxI=
|
||||
github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af/go.mod h1:Nht3zPeWKUH0NzdCt2Blrr5ys8VGpn0CEB0cQHVjt7k=
|
||||
github.com/jonboulle/clockwork v0.1.0/go.mod h1:Ii8DK3G1RaLaWxj9trq07+26W01tbo22gdxWY5EU2bo=
|
||||
@ -321,6 +360,7 @@ github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515/go.mod h1:+0opPa2QZZtGFB
|
||||
github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI=
|
||||
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
|
||||
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
|
||||
github.com/kr/pty v1.1.3/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
|
||||
github.com/kr/pty v1.1.8/go.mod h1:O1sed60cT9XZ5uDucP5qwvh+TE3NnUj51EiZO/lmSfw=
|
||||
github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=
|
||||
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
|
||||
@ -330,9 +370,18 @@ github.com/lib/pq v1.1.0/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo=
|
||||
github.com/lib/pq v1.2.0/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo=
|
||||
github.com/lib/pq v1.3.0 h1:/qkRGz8zljWiDcFvgpwUpwIAPu3r07TDvs3Rws+o/pU=
|
||||
github.com/lib/pq v1.3.0/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo=
|
||||
github.com/lucas-clemente/quic-go v0.7.1-0.20201102053916-272229abf044 h1:M6zB4Rs4SJDk9IBIvC3ozl23+b0d1Q7NOlHnbxuc3AY=
|
||||
github.com/lucas-clemente/quic-go v0.7.1-0.20201102053916-272229abf044/go.mod h1:ZUygOqIoai0ASXXLJ92LTnKdbqh9MHCLTX6Nr1jUrK0=
|
||||
github.com/lunixbochs/vtclean v1.0.0/go.mod h1:pHhQNgMf3btfWnGBVipUOjRYhoOsdGqdm/+2c2E2WMI=
|
||||
github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
|
||||
github.com/magiconair/properties v1.8.1 h1:ZC2Vc7/ZFkGmsVC9KvOjumD+G5lXy2RtTKyzRKO2BQ4=
|
||||
github.com/magiconair/properties v1.8.1/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
|
||||
github.com/mailru/easyjson v0.0.0-20190312143242-1de009706dbe/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc=
|
||||
github.com/marten-seemann/qpack v0.2.1/go.mod h1:F7Gl5L1jIgN1D11ucXefiuJS9UMVP2opoCp2jDKb7wc=
|
||||
github.com/marten-seemann/qtls v0.10.0 h1:ECsuYUKalRL240rRD4Ri33ISb7kAQ3qGDlrrl55b2pc=
|
||||
github.com/marten-seemann/qtls v0.10.0/go.mod h1:UvMd1oaYDACI99/oZUYLzMCkBXQVT0aGm99sJhbT8hs=
|
||||
github.com/marten-seemann/qtls-go1-15 v0.1.1 h1:LIH6K34bPVttyXnUWixk0bzH6/N07VxbSabxn5A5gZQ=
|
||||
github.com/marten-seemann/qtls-go1-15 v0.1.1/go.mod h1:GyFwywLKkRt+6mfU99csTEY1joMZz5vmB1WNZH3P81I=
|
||||
github.com/mattn/go-colorable v0.0.9/go.mod h1:9vuHe8Xs5qXnSaW/c/ABM9alt+Vo+STaOChaDxuIBZU=
|
||||
github.com/mattn/go-colorable v0.1.1/go.mod h1:FuOcm+DKB9mbwrcAfNl7/TZVBZ6rcnceauSikq3lYCQ=
|
||||
github.com/mattn/go-colorable v0.1.2/go.mod h1:U0ppj6V5qS13XJ6of8GYAs25YV2eR4EVcfRqFIhoBtE=
|
||||
@ -353,6 +402,7 @@ github.com/mattn/go-sqlite3 v1.10.0/go.mod h1:FPy6KqzDD04eiIsT53CuJW3U88zkxoIYsO
|
||||
github.com/mattn/go-sqlite3 v2.0.3+incompatible h1:gXHsfypPkaMZrKbD5209QV9jbUTJKjyR5WD3HYQSd+U=
|
||||
github.com/mattn/go-sqlite3 v2.0.3+incompatible/go.mod h1:FPy6KqzDD04eiIsT53CuJW3U88zkxoIYsOqkbpncsNc=
|
||||
github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0=
|
||||
github.com/microcosm-cc/bluemonday v1.0.1/go.mod h1:hsXNsILzKxV+sX77C5b8FSuKF00vh2OMYv+xgHpAMF4=
|
||||
github.com/miekg/dns v1.0.14/go.mod h1:W1PPwlIAgtquWBMBEV9nkV9Cazfe8ScdGz/Lj7v3Nrg=
|
||||
github.com/minio/sha256-simd v0.0.0-20190328051042-05b4dd3047e5/go.mod h1:2FMWW+8GMoPweT6+pI63m9YE3Lmw4J71hV56Chs1E/U=
|
||||
github.com/mitchellh/cli v1.0.0/go.mod h1:hNIlj7HEI86fIcpObd7a0FcrxTWetlwJDGcceTlRvqc=
|
||||
@ -370,22 +420,32 @@ github.com/morikuni/aec v0.0.0-20170113033406-39771216ff4c h1:nXxl5PrvVm2L/wCy8d
|
||||
github.com/morikuni/aec v0.0.0-20170113033406-39771216ff4c/go.mod h1:BbKIizmSmc5MMPqRYbxO4ZU0S0+P200+tUnFx7PXmsc=
|
||||
github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
|
||||
github.com/nakagami/firebirdsql v0.0.0-20190310045651-3c02a58cfed8/go.mod h1:86wM1zFnC6/uDBfZGNwB65O+pR2OFi5q/YQaEUid1qA=
|
||||
github.com/neelance/astrewrite v0.0.0-20160511093645-99348263ae86/go.mod h1:kHJEU3ofeGjhHklVoIGuVj85JJwZ6kWPaJwCIxgnFmo=
|
||||
github.com/neelance/sourcemap v0.0.0-20151028013722-8c68805598ab/go.mod h1:Qr6/a/Q4r9LP1IltGz7tA7iOK1WonHEYhu1HRBA7ZiM=
|
||||
github.com/nsf/jsondiff v0.0.0-20200515183724-f29ed568f4ce h1:RPclfga2SEJmgMmz2k+Mg7cowZ8yv4Trqw9UsJby758=
|
||||
github.com/nsf/jsondiff v0.0.0-20200515183724-f29ed568f4ce/go.mod h1:uFMI8w+ref4v2r9jz+c9i1IfIttS/OkmLfrk1jne5hs=
|
||||
github.com/nsf/termbox-go v0.0.0-20200418040025-38ba6e5628f1 h1:lh3PyZvY+B9nFliSGTn5uFuqQQJGuNrD0MLCokv09ag=
|
||||
github.com/nsf/termbox-go v0.0.0-20200418040025-38ba6e5628f1/go.mod h1:IuKpRQcYE1Tfu+oAQqaLisqDeXgjyyltCfsaoYN18NQ=
|
||||
github.com/nxadm/tail v1.4.4 h1:DQuhQpB1tVlglWS2hLQ5OV6B5r8aGxSrPc5Qo6uTN78=
|
||||
github.com/nxadm/tail v1.4.4/go.mod h1:kenIhsEOeOJmVchQTgglprH7qJGnHDVpk1VPCcaMI8A=
|
||||
github.com/oklog/ulid v1.3.1/go.mod h1:CirwcVhetQ6Lv90oh/F+FBtV6XMibvdAFo93nm5qn4U=
|
||||
github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
|
||||
github.com/onsi/ginkgo v1.7.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
|
||||
github.com/onsi/ginkgo v1.10.3 h1:OoxbjfXVZyod1fmWYhI7SEyaD8B00ynP3T+D5GiyHOY=
|
||||
github.com/onsi/ginkgo v1.10.3/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
|
||||
github.com/onsi/ginkgo v1.12.1/go.mod h1:zj2OWP4+oCPe1qIXoGWkgMRwljMUYCdkwsT2108oapk=
|
||||
github.com/onsi/ginkgo v1.14.0 h1:2mOpI4JVVPBN+WQRa0WKH2eXR+Ey+uK4n7Zj0aYpIQA=
|
||||
github.com/onsi/ginkgo v1.14.0/go.mod h1:iSB4RoI2tjJc9BBv4NKIKWKya62Rps+oPG/Lv9klQyY=
|
||||
github.com/onsi/gomega v1.4.3/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY=
|
||||
github.com/onsi/gomega v1.7.1 h1:K0jcRCwNQM3vFGh1ppMtDh/+7ApJrjldlX8fA0jDTLQ=
|
||||
github.com/onsi/gomega v1.7.1/go.mod h1:XdKZgCCFLUoM/7CFJVPcG8C1xQ1AJ0vpAezJrB7JYyY=
|
||||
github.com/onsi/gomega v1.10.1 h1:o0+MgICZLuZ7xjH7Vx6zS/zcu93/BEp1VwkIW1mEXCE=
|
||||
github.com/onsi/gomega v1.10.1/go.mod h1:iN09h71vgCQne3DLsj+A5owkum+a2tYe+TOCB1ybHNo=
|
||||
github.com/opencontainers/go-digest v1.0.0-rc1 h1:WzifXhOVOEOuFYOJAW6aQqW0TooG2iki3E3Ii+WN7gQ=
|
||||
github.com/opencontainers/go-digest v1.0.0-rc1/go.mod h1:cMLVZDEM3+U2I4VmLI6N8jQYUd2OVphdqWwCJHrFt2s=
|
||||
github.com/opencontainers/image-spec v1.0.1 h1:JMemWkRwHx4Zj+fVxWoMCFm/8sYGGrUVojFA6h/TRcI=
|
||||
github.com/opencontainers/image-spec v1.0.1/go.mod h1:BtxoFyWECRxE4U/7sNtV5W15zMzWCbyJoFRP3s7yZA0=
|
||||
github.com/openzipkin/zipkin-go v0.1.1/go.mod h1:NtoC/o8u3JlF1lSlyPNswIbeQH9bJTmOf0Erfk+hxe8=
|
||||
github.com/openzipkin/zipkin-go v0.1.6/go.mod h1:QgAqvLzwWbR/WpD4A3cGpPtJrZXNIiJc5AZX7/PBEpw=
|
||||
github.com/pascaldekloe/goe v0.0.0-20180627143212-57f6aae5913c/go.mod h1:lzWF7FIEvWOWxwDKqyGYQf6ZUaNfKdP144TG7ZOy1lc=
|
||||
github.com/pelletier/go-toml v1.2.0 h1:T5zMGML61Wp+FlcbWjRDT7yAxhJNAiPPLOFECq181zc=
|
||||
@ -398,6 +458,7 @@ github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINE
|
||||
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/posener/complete v1.1.1/go.mod h1:em0nMJCgc9GFtwrmVmEMR/ZL6WyhyjMBndrE9hABlRI=
|
||||
github.com/prometheus/client_golang v0.8.0/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXPKyh/dDVn+NZz0KFw=
|
||||
github.com/prometheus/client_golang v0.9.1/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXPKyh/dDVn+NZz0KFw=
|
||||
github.com/prometheus/client_golang v0.9.3-0.20190127221311-3c4408c8b829/go.mod h1:p2iRAGwDERtqlqzRXnrOVns+ignqQo//hLXqYxZYVNs=
|
||||
github.com/prometheus/client_golang v0.9.3/go.mod h1:/TN21ttK/J9q6uSwhBd54HahCDft0ttaMvbicHlPoso=
|
||||
@ -405,9 +466,11 @@ github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910/go.mod h1:
|
||||
github.com/prometheus/client_model v0.0.0-20190115171406-56726106282f/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo=
|
||||
github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
|
||||
github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
|
||||
github.com/prometheus/common v0.0.0-20180801064454-c7de2306084e/go.mod h1:daVV7qP5qjZbuso7PdcryaAu0sAZbrN9i7WWcTMWvro=
|
||||
github.com/prometheus/common v0.0.0-20181113130724-41aa239b4cce/go.mod h1:daVV7qP5qjZbuso7PdcryaAu0sAZbrN9i7WWcTMWvro=
|
||||
github.com/prometheus/common v0.2.0/go.mod h1:TNfzLD0ON7rHzMJeJkieUDPYmFC7Snx/y86RQel1bk4=
|
||||
github.com/prometheus/common v0.4.0/go.mod h1:TNfzLD0ON7rHzMJeJkieUDPYmFC7Snx/y86RQel1bk4=
|
||||
github.com/prometheus/procfs v0.0.0-20180725123919-05ee40e3a273/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
|
||||
github.com/prometheus/procfs v0.0.0-20181005140218-185b4288413d/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
|
||||
github.com/prometheus/procfs v0.0.0-20190117184657-bf6a532e95b1/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
|
||||
github.com/prometheus/procfs v0.0.0-20190507164030-5867b95ac084/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsTZCD3I8kEA=
|
||||
@ -418,15 +481,39 @@ github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFR
|
||||
github.com/rs/xid v1.2.1/go.mod h1:+uKXf+4Djp6Md1KODXJxgGQPKngRmWyn10oCKFzNHOQ=
|
||||
github.com/rs/zerolog v1.13.0/go.mod h1:YbFCdg8HfsridGWAh22vktObvhZbQsZXe4/zB0OKkWU=
|
||||
github.com/rs/zerolog v1.15.0/go.mod h1:xYTKnLHcpfU2225ny5qZjxnj9NvkumZYjJHlAThCjNc=
|
||||
github.com/russross/blackfriday v1.5.2/go.mod h1:JO/DiYxRf+HjHt06OyowR9PTA263kcR/rfWxYHBV53g=
|
||||
github.com/russross/blackfriday/v2 v2.0.1/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
|
||||
github.com/ryanuber/columnize v0.0.0-20160712163229-9b3edd62028f/go.mod h1:sm1tb6uqfes/u+d4ooFouqFdy9/2g9QGwK3SQygK0Ts=
|
||||
github.com/satori/go.uuid v1.2.0/go.mod h1:dA0hQrYB0VpLJoorglMZABFdXlWrHn1NEOzdhQKdks0=
|
||||
github.com/sean-/seed v0.0.0-20170313163322-e2103e2c3529/go.mod h1:DxrIzT+xaE7yg65j358z/aeFdxmN0P9QXhEzd20vsDc=
|
||||
github.com/sergi/go-diff v1.0.0/go.mod h1:0CfEIISq7TuYL3j771MWULgwwjU+GofnZX9QAmXWZgo=
|
||||
github.com/shopspring/decimal v0.0.0-20180709203117-cd690d0c9e24/go.mod h1:M+9NzErvs504Cn4c5DxATwIqPbtswREoFCre64PpcG4=
|
||||
github.com/shopspring/decimal v0.0.0-20200227202807-02e2044944cc/go.mod h1:DKyhrW/HYNuLGql+MJL6WCR6knT2jwCFRcu2hWCYk4o=
|
||||
github.com/shopspring/decimal v1.2.0 h1:abSATXmQEYyShuxI4/vyW3tV1MrKAJzCZ/0zLUXYbsQ=
|
||||
github.com/shopspring/decimal v1.2.0/go.mod h1:DKyhrW/HYNuLGql+MJL6WCR6knT2jwCFRcu2hWCYk4o=
|
||||
github.com/shurcooL/component v0.0.0-20170202220835-f88ec8f54cc4/go.mod h1:XhFIlyj5a1fBNx5aJTbKoIq0mNaPvOagO+HjB3EtxrY=
|
||||
github.com/shurcooL/events v0.0.0-20181021180414-410e4ca65f48/go.mod h1:5u70Mqkb5O5cxEA8nxTsgrgLehJeAw6Oc4Ab1c/P1HM=
|
||||
github.com/shurcooL/github_flavored_markdown v0.0.0-20181002035957-2122de532470/go.mod h1:2dOwnU2uBioM+SGy2aZoq1f/Sd1l9OkAeAUvjSyvgU0=
|
||||
github.com/shurcooL/go v0.0.0-20180423040247-9e1955d9fb6e/go.mod h1:TDJrrUr11Vxrven61rcy3hJMUqaf/CLWYhHNPmT14Lk=
|
||||
github.com/shurcooL/go-goon v0.0.0-20170922171312-37c2f522c041/go.mod h1:N5mDOmsrJOB+vfqUK+7DmDyjhSLIIBnXo9lvZJj3MWQ=
|
||||
github.com/shurcooL/gofontwoff v0.0.0-20180329035133-29b52fc0a18d/go.mod h1:05UtEgK5zq39gLST6uB0cf3NEHjETfB4Fgr3Gx5R9Vw=
|
||||
github.com/shurcooL/gopherjslib v0.0.0-20160914041154-feb6d3990c2c/go.mod h1:8d3azKNyqcHP1GaQE/c6dDgjkgSx2BZ4IoEi4F1reUI=
|
||||
github.com/shurcooL/highlight_diff v0.0.0-20170515013008-09bb4053de1b/go.mod h1:ZpfEhSmds4ytuByIcDnOLkTHGUI6KNqRNPDLHDk+mUU=
|
||||
github.com/shurcooL/highlight_go v0.0.0-20181028180052-98c3abbbae20/go.mod h1:UDKB5a1T23gOMUJrI+uSuH0VRDStOiUVSjBTRDVBVag=
|
||||
github.com/shurcooL/home v0.0.0-20181020052607-80b7ffcb30f9/go.mod h1:+rgNQw2P9ARFAs37qieuu7ohDNQ3gds9msbT2yn85sg=
|
||||
github.com/shurcooL/htmlg v0.0.0-20170918183704-d01228ac9e50/go.mod h1:zPn1wHpTIePGnXSHpsVPWEktKXHr6+SS6x/IKRb7cpw=
|
||||
github.com/shurcooL/httperror v0.0.0-20170206035902-86b7830d14cc/go.mod h1:aYMfkZ6DWSJPJ6c4Wwz3QtW22G7mf/PEgaB9k/ik5+Y=
|
||||
github.com/shurcooL/httpfs v0.0.0-20171119174359-809beceb2371/go.mod h1:ZY1cvUeJuFPAdZ/B6v7RHavJWZn2YPVFQ1OSXhCGOkg=
|
||||
github.com/shurcooL/httpgzip v0.0.0-20180522190206-b1c53ac65af9/go.mod h1:919LwcH0M7/W4fcZ0/jy0qGght1GIhqyS/EgWGH2j5Q=
|
||||
github.com/shurcooL/issues v0.0.0-20181008053335-6292fdc1e191/go.mod h1:e2qWDig5bLteJ4fwvDAc2NHzqFEthkqn7aOZAOpj+PQ=
|
||||
github.com/shurcooL/issuesapp v0.0.0-20180602232740-048589ce2241/go.mod h1:NPpHK2TI7iSaM0buivtFUc9offApnI0Alt/K8hcHy0I=
|
||||
github.com/shurcooL/notifications v0.0.0-20181007000457-627ab5aea122/go.mod h1:b5uSkrEVM1jQUspwbixRBhaIjIzL2xazXp6kntxYle0=
|
||||
github.com/shurcooL/octicon v0.0.0-20181028054416-fa4f57f9efb2/go.mod h1:eWdoE5JD4R5UVWDucdOPg1g2fqQRq78IQa9zlOV1vpQ=
|
||||
github.com/shurcooL/reactions v0.0.0-20181006231557-f2e0b4ca5b82/go.mod h1:TCR1lToEk4d2s07G3XGfz2QrgHXg4RJBvjrOozvoWfk=
|
||||
github.com/shurcooL/sanitized_anchor_name v0.0.0-20170918181015-86672fcb3f95/go.mod h1:1NzhyTcUVG4SuEtjjoZeVRXNmyL/1OwPU0+IJeTBvfc=
|
||||
github.com/shurcooL/sanitized_anchor_name v1.0.0/go.mod h1:1NzhyTcUVG4SuEtjjoZeVRXNmyL/1OwPU0+IJeTBvfc=
|
||||
github.com/shurcooL/users v0.0.0-20180125191416-49c67e49c537/go.mod h1:QJTqeLYEDaXHZDBsXlPCDqdhQuJkuw4NOtaxYe3xii4=
|
||||
github.com/shurcooL/webdavfs v0.0.0-20170829043945-18c3829fa133/go.mod h1:hKmq5kWdCj2z2KEozexVbfEZIWiTjhE0+UjmZgPqehw=
|
||||
github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
|
||||
github.com/sirupsen/logrus v1.4.1/go.mod h1:ni0Sbl8bgC9z8RoU9G6nDWqqs/fq4eDPysMBDgk/93Q=
|
||||
github.com/sirupsen/logrus v1.4.2 h1:SPIRibHv4MatM3XXNO2BJeFLZwZ2LvZgfQ5+UNI2im4=
|
||||
@ -436,6 +523,8 @@ github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d/go.mod h1
|
||||
github.com/smartystreets/goconvey v1.6.4 h1:fv0U8FUIMPNf1L9lnHLvLhgicrIVChEkdzIKYqbNC9s=
|
||||
github.com/smartystreets/goconvey v1.6.4/go.mod h1:syvi0/a8iFYH4r/RixwvyeAJjdLS9QV7WQ/tjFTllLA=
|
||||
github.com/soheilhy/cmux v0.1.4/go.mod h1:IM3LyeVVIOuxMH7sFAkER9+bJ4dT7Ms6E4xg4kGIyLM=
|
||||
github.com/sourcegraph/annotate v0.0.0-20160123013949-f4cad6c6324d/go.mod h1:UdhH50NIW0fCiwBSr0co2m7BnFLdv4fQTgdqdJTHFeE=
|
||||
github.com/sourcegraph/syntaxhighlight v0.0.0-20170531221838-bd320f5d308e/go.mod h1:HuIsMU8RRBOtsCgI77wP899iHVBQpCmg4ErYMZB+2IA=
|
||||
github.com/spacemonkeygo/monkit/v3 v3.0.0-20191108235033-eacca33b3037/go.mod h1:JcK1pCbReQsOsMKF/POFSZCq7drXFybgGmbc27tuwes=
|
||||
github.com/spacemonkeygo/monkit/v3 v3.0.4/go.mod h1:JcK1pCbReQsOsMKF/POFSZCq7drXFybgGmbc27tuwes=
|
||||
github.com/spacemonkeygo/monkit/v3 v3.0.5/go.mod h1:JcK1pCbReQsOsMKF/POFSZCq7drXFybgGmbc27tuwes=
|
||||
@ -475,9 +564,12 @@ github.com/stripe/stripe-go v70.15.0+incompatible h1:hNML7M1zx8RgtepEMlxyu/FpVPr
|
||||
github.com/stripe/stripe-go v70.15.0+incompatible/go.mod h1:A1dQZmO/QypXmsL0T8axYZkSN/uA/T/A64pfKdBAMiY=
|
||||
github.com/subosito/gotenv v1.2.0 h1:Slr1R9HxAlEKefgq5jn9U+DnETlIUa6HfgEzj0g5d7s=
|
||||
github.com/subosito/gotenv v1.2.0/go.mod h1:N0PQaV/YGNqwC0u51sEeR/aUtSLEXKX9iv69rRypqCw=
|
||||
github.com/tarm/serial v0.0.0-20180830185346-98f6abe2eb07/go.mod h1:kDXzergiv9cbyO7IOYJZWg1U88JhDg3PB6klq9Hg2pA=
|
||||
github.com/tidwall/pretty v0.0.0-20180105212114-65a9db5fad51/go.mod h1:XNkn88O1ChpSDQmQeStsy+sBenx6DDtFZJxhVysOjyk=
|
||||
github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U=
|
||||
github.com/ugorji/go v1.1.4/go.mod h1:uQMGLiO92mf5W77hV/PUCpI3pbzQx3CRekS0kk+RGrc=
|
||||
github.com/viant/assertly v0.4.8/go.mod h1:aGifi++jvCrUaklKEKT0BU95igDNaqkvz+49uaYMPRU=
|
||||
github.com/viant/toolbox v0.24.0/go.mod h1:OxMCG57V0PXuIP2HNQrtJf2CjqdmbrOx5EkMILuUhzM=
|
||||
github.com/vivint/infectious v0.0.0-20200605153912-25a574ae18a3 h1:zMsHhfK9+Wdl1F7sIKLyx3wrOFofpb3rWFbA4HgcK5k=
|
||||
github.com/vivint/infectious v0.0.0-20200605153912-25a574ae18a3/go.mod h1:R0Gbuw7ElaGSLOZUSwBm/GgVwMd30jWxBDdAyMOeTuc=
|
||||
github.com/xanzy/go-gitlab v0.15.0/go.mod h1:8zdQa/ri1dfn8eS3Ir1SyfvOKlw7WBJ8DVThkpGiXrs=
|
||||
@ -508,6 +600,7 @@ go.etcd.io/bbolt v1.3.2/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU=
|
||||
go.etcd.io/bbolt v1.3.5 h1:XAzx9gjCb0Rxj7EoqcClPD1d5ZBxZJk0jbuoPHenBt0=
|
||||
go.etcd.io/bbolt v1.3.5/go.mod h1:G5EMThwa9y8QZGBClrRx5EY+Yw9kAhnjy3bSjsnlVTQ=
|
||||
go.mongodb.org/mongo-driver v1.1.0/go.mod h1:u7ryQJ+DOzQmeO7zB6MHyr8jkEQvC8vH7qLUO4lqsUM=
|
||||
go.opencensus.io v0.18.0/go.mod h1:vKdFvxhtzZ9onBp9VKHK8z/sRpBMnKAsufL7wlDrCOA=
|
||||
go.opencensus.io v0.20.1/go.mod h1:6WKK9ahsWS3RSO+PY9ZHZUfv2irvY6gN279GOPZjmmk=
|
||||
go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU=
|
||||
go.opencensus.io v0.22.0/go.mod h1:+kGneAE2xo2IficOXnaByMWTGM9T73dGwxeWcUqIpI8=
|
||||
@ -531,10 +624,14 @@ go.uber.org/zap v1.10.0/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q=
|
||||
go.uber.org/zap v1.14.1/go.mod h1:Mb2vm2krFEG5DV0W9qcHBYFtp/Wku1cvYaqPsS/WYfc=
|
||||
go.uber.org/zap v1.16.0 h1:uFRZXykJGK9lLY4HtgSw44DnIcAM+kRBP7x5m+NpAOM=
|
||||
go.uber.org/zap v1.16.0/go.mod h1:MA8QOfq0BHJwdXa996Y4dYkAqRKB8/1K1QMMZVaNZjQ=
|
||||
go4.org v0.0.0-20180809161055-417644f6feb5/go.mod h1:MkTOUMDaeVYJUOUsaDXIhWPZYa1yOyC1qaOBpL57BhE=
|
||||
golang.org/x/build v0.0.0-20190111050920-041ab4dc3f9d/go.mod h1:OWs+y06UdEOHN4y+MfF/py+xQ/tYqIWW03b70/CG9Rw=
|
||||
golang.org/x/crypto v0.0.0-20170930174604-9419663f5a44/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
|
||||
golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
|
||||
golang.org/x/crypto v0.0.0-20181029021203-45a5f77698d3/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
|
||||
golang.org/x/crypto v0.0.0-20181030102418-4d3f4d9ffa16/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
|
||||
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
|
||||
golang.org/x/crypto v0.0.0-20190313024323-a1f597ede03a/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
|
||||
golang.org/x/crypto v0.0.0-20190325154230-a5d413f7728c/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
|
||||
golang.org/x/crypto v0.0.0-20190411191339-88737f569e3a/go.mod h1:WFFai1msRO1wXaEeE5yQxYXgSfI8pQAWXbQop6sCtWE=
|
||||
golang.org/x/crypto v0.0.0-20190426145343-a29dc8fdc734/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
|
||||
@ -545,6 +642,7 @@ golang.org/x/crypto v0.0.0-20190911031432-227b76d455e7/go.mod h1:yigFU9vqHzYiE8U
|
||||
golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
|
||||
golang.org/x/crypto v0.0.0-20200115085410-6d4e4cb37c7d/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
|
||||
golang.org/x/crypto v0.0.0-20200220183623-bac4c82f6975/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
|
||||
golang.org/x/crypto v0.0.0-20200221231518-2aa609cf4a9d/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
|
||||
golang.org/x/crypto v0.0.0-20200323165209-0ec3e9974c59/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
|
||||
golang.org/x/crypto v0.0.0-20200604202706-70a84ac30bf9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
|
||||
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
|
||||
@ -559,6 +657,7 @@ golang.org/x/exp v0.0.0-20191227195350-da58074b4299 h1:zQpM52jfKHG6II1ISZY1Zcpyg
|
||||
golang.org/x/exp v0.0.0-20191227195350-da58074b4299/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4=
|
||||
golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js=
|
||||
golang.org/x/image v0.0.0-20190802002840-cff245a6509b/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0=
|
||||
golang.org/x/lint v0.0.0-20180702182130-06c8688daad7/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
|
||||
golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
|
||||
golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU=
|
||||
golang.org/x/lint v0.0.0-20190301231843-5614ed5bae6f/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
|
||||
@ -577,6 +676,8 @@ golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73r
|
||||
golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
|
||||
golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
|
||||
golang.org/x/net v0.0.0-20181023162649-9b4f9f5ad519/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
|
||||
golang.org/x/net v0.0.0-20181029044818-c44066c5c816/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
|
||||
golang.org/x/net v0.0.0-20181106065722-10aee1819953/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
|
||||
golang.org/x/net v0.0.0-20181108082009-03003ca0c849/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
|
||||
golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
|
||||
golang.org/x/net v0.0.0-20181201002055-351d144fa1fc/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
|
||||
@ -585,6 +686,7 @@ golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73r
|
||||
golang.org/x/net v0.0.0-20190125091013-d26f9f9a57f3/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
|
||||
golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
|
||||
golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
|
||||
golang.org/x/net v0.0.0-20190313220215-9f648a60d977/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
|
||||
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
|
||||
golang.org/x/net v0.0.0-20190424112056-4829fb13d2c6/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
|
||||
golang.org/x/net v0.0.0-20190501004415-9ce7a6920f09/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
|
||||
@ -595,15 +697,19 @@ golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLL
|
||||
golang.org/x/net v0.0.0-20190813141303-74dc4d7220e7/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
|
||||
golang.org/x/net v0.0.0-20190923162816-aa69164e4478/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
|
||||
golang.org/x/net v0.0.0-20200114155413-6afb5195e5aa/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
|
||||
golang.org/x/net v0.0.0-20200226121028-0de0cce0169b h1:0mm1VjtFUOIlE1SbDlwjYaDxZVDP2S5ou6y0gSgXHu8=
|
||||
golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
|
||||
golang.org/x/net v0.0.0-20200520004742-59133d7f0dd7/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
|
||||
golang.org/x/net v0.0.0-20200707034311-ab3426394381 h1:VXak5I6aEWmAXeQjA+QSZzlgNrpq9mjcfDemuexIKsU=
|
||||
golang.org/x/net v0.0.0-20200707034311-ab3426394381/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
|
||||
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
|
||||
golang.org/x/oauth2 v0.0.0-20181017192945-9dcd33a902f4/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
|
||||
golang.org/x/oauth2 v0.0.0-20181106182150-f42d05182288/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
|
||||
golang.org/x/oauth2 v0.0.0-20181203162652-d668ce993890/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
|
||||
golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
|
||||
golang.org/x/oauth2 v0.0.0-20190402181905-9f3314589c9a/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
|
||||
golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
|
||||
golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d h1:TzXSXBo42m9gQenoE3b9BGiEpg5IG2JkU5FkPIawgtw=
|
||||
golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
|
||||
golang.org/x/perf v0.0.0-20180704124530-6e6d33e29852/go.mod h1:JLpeXjPJfIyPr5TlbXLkXWLhP8nz10XfvxElABhCtcw=
|
||||
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
||||
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
||||
golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
||||
@ -618,6 +724,7 @@ golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5h
|
||||
golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
|
||||
golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
|
||||
golang.org/x/sys v0.0.0-20181026203630-95b1ffbd15a5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
|
||||
golang.org/x/sys v0.0.0-20181029174526-d69651ed3497/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
|
||||
golang.org/x/sys v0.0.0-20181107165924-66b7b1311ac8/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
|
||||
golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
|
||||
golang.org/x/sys v0.0.0-20181122145206-62eef0e2fa9b/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
|
||||
@ -626,6 +733,7 @@ golang.org/x/sys v0.0.0-20190204203706-41f3e6584952/go.mod h1:STP8DvDyc/dI5b8T5h
|
||||
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
|
||||
golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
|
||||
golang.org/x/sys v0.0.0-20190312061237-fead79001313/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/sys v0.0.0-20190316082340-a2f829d7f35f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/sys v0.0.0-20190403152447-81d4e9dc473e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/sys v0.0.0-20190422165155-953cdadca894/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
@ -636,7 +744,10 @@ golang.org/x/sys v0.0.0-20190606165138-5da285871e9c/go.mod h1:h1NjWce9XRLGQEsW7w
|
||||
golang.org/x/sys v0.0.0-20190624142023-c5567b49c5d0/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/sys v0.0.0-20190813064441-fde4db37ae7a/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/sys v0.0.0-20190826190057-c7b8b68b1456/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/sys v0.0.0-20190904154756-749cb33beabd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/sys v0.0.0-20191005200804-aed5e4c7ecf9/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/sys v0.0.0-20191026070338-33540a1f6037/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/sys v0.0.0-20191120155948-bd437916bb0e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/sys v0.0.0-20191204072324-ce4227a45e2e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/sys v0.0.0-20191210023423-ac6580df4449/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/sys v0.0.0-20200107144601-ef85f5a75ddf/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
@ -645,14 +756,17 @@ golang.org/x/sys v0.0.0-20200116001909-b77594299b42/go.mod h1:h1NjWce9XRLGQEsW7w
|
||||
golang.org/x/sys v0.0.0-20200202164722-d101bd2416d5/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/sys v0.0.0-20200519105757-fe76b779f299/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/sys v0.0.0-20200610111108-226ff32320da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/sys v0.0.0-20200929083018-4d22bbb62b3c h1:/h0vtH0PyU0xAoZJVcRw1k0Ng+U0JAy3QDiFmppIlIE=
|
||||
golang.org/x/sys v0.0.0-20200929083018-4d22bbb62b3c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
|
||||
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
|
||||
golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
|
||||
golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
|
||||
golang.org/x/text v0.3.3 h1:cokOdA+Jmi5PJGXLlLllQSgYigAEfHXJAERHVMaCc2k=
|
||||
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
|
||||
golang.org/x/time v0.0.0-20180412165947-fbb02b2291d2/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
|
||||
golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
|
||||
golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
|
||||
golang.org/x/time v0.0.0-20200630173020-3af7569d3a1e h1:EHBhcS0mlXEAVwNyO2dLfjToGsyY4j24pTs2ScHnX7s=
|
||||
@ -660,6 +774,7 @@ golang.org/x/time v0.0.0-20200630173020-3af7569d3a1e/go.mod h1:tRJNPiyCQ0inRvYxb
|
||||
golang.org/x/tools v0.0.0-20180221164845-07fd8470d635/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
|
||||
golang.org/x/tools v0.0.0-20180828015842-6cd1fcedba52/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
|
||||
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
|
||||
golang.org/x/tools v0.0.0-20181030000716-a0a13e073c7b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
|
||||
golang.org/x/tools v0.0.0-20181030221726-6c7e314b6563/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
|
||||
golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
|
||||
golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY=
|
||||
@ -691,6 +806,9 @@ golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8T
|
||||
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
|
||||
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543 h1:E7g+9GITq07hpfrRu66IVDexMakfv52eLZ2CXBWiKr4=
|
||||
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
|
||||
google.golang.org/api v0.0.0-20180910000450-7ca32eb868bf/go.mod h1:4mhQ8q/RsB7i+udVvVy5NUi08OU8ZlA0gRVgrF7VFY0=
|
||||
google.golang.org/api v0.0.0-20181030000543-1d582fd0359e/go.mod h1:4mhQ8q/RsB7i+udVvVy5NUi08OU8ZlA0gRVgrF7VFY0=
|
||||
google.golang.org/api v0.1.0/go.mod h1:UGEZY7KEX120AnNLIHFMKIo4obdJhkp2tPbaPlQx13Y=
|
||||
google.golang.org/api v0.3.1/go.mod h1:6wY9I6uQWHQ8EM57III9mq/AjF+i8G65rmVagqKMtkk=
|
||||
google.golang.org/api v0.3.2/go.mod h1:6wY9I6uQWHQ8EM57III9mq/AjF+i8G65rmVagqKMtkk=
|
||||
google.golang.org/api v0.4.0/go.mod h1:8k5glujaEP+g9n7WNsDg8QP6cUVNI86fCNMcbazEtwE=
|
||||
@ -702,6 +820,7 @@ google.golang.org/api v0.15.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsb
|
||||
google.golang.org/api v0.20.0 h1:jz2KixHX7EcCPiQrySzPdnYT7DbINAypCqKZ1Z7GM40=
|
||||
google.golang.org/api v0.20.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE=
|
||||
google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
|
||||
google.golang.org/appengine v1.2.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
|
||||
google.golang.org/appengine v1.3.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
|
||||
google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
|
||||
google.golang.org/appengine v1.5.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
|
||||
@ -709,6 +828,10 @@ google.golang.org/appengine v1.6.1/go.mod h1:i06prIuMbXzDqacNJfV5OdTW448YApPu5ww
|
||||
google.golang.org/appengine v1.6.5 h1:tycE03LOZYQNhDpS27tcQdAzLCVMaj7QT2SXxebnpCM=
|
||||
google.golang.org/appengine v1.6.5/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc=
|
||||
google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
|
||||
google.golang.org/genproto v0.0.0-20180831171423-11092d34479b/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
|
||||
google.golang.org/genproto v0.0.0-20181029155118-b69ba1387ce2/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
|
||||
google.golang.org/genproto v0.0.0-20181202183823-bd91e49a0898/go.mod h1:7Ep/1NZk928CDR8SjdVbjWNpdIf6nzjE3BTgJDr2Atg=
|
||||
google.golang.org/genproto v0.0.0-20190306203927-b5d61aea6440/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
|
||||
google.golang.org/genproto v0.0.0-20190307195333-5fe7a883aa19/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
|
||||
google.golang.org/genproto v0.0.0-20190404172233-64821d5d2107/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
|
||||
google.golang.org/genproto v0.0.0-20190418145605-e7d98fc518a7/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
|
||||
@ -720,6 +843,8 @@ google.golang.org/genproto v0.0.0-20190911173649-1774047e7e51/go.mod h1:IbNlFCBr
|
||||
google.golang.org/genproto v0.0.0-20191108220845-16a3f7862a1a/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
|
||||
google.golang.org/genproto v0.0.0-20200115191322-ca5a22157cba h1:pRj9OXZbwNtbtZtOB4dLwfK4u+EVRMvP+e9zKkg2grM=
|
||||
google.golang.org/genproto v0.0.0-20200115191322-ca5a22157cba/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
|
||||
google.golang.org/grpc v1.14.0/go.mod h1:yo6s7OP7yaDglbqo1J04qKzAhqBH6lvTonzMVmEdcZw=
|
||||
google.golang.org/grpc v1.16.0/go.mod h1:0JHn/cJsOMiMfNA9+DeHDlAU7KAAB5GDlYFpa9MZMio=
|
||||
google.golang.org/grpc v1.17.0/go.mod h1:6QZJwpn2B+Zp71q/5VxRsJ6NXXVCE5NRUHRo+f3cWCs=
|
||||
google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
|
||||
google.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiqXj38=
|
||||
@ -730,6 +855,13 @@ google.golang.org/grpc v1.26.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8
|
||||
google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
|
||||
google.golang.org/grpc v1.27.1 h1:zvIju4sqAGvwKspUQOhwnpcqSbzi7/H6QomNNjTL4sk=
|
||||
google.golang.org/grpc v1.27.1/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
|
||||
google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8=
|
||||
google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0=
|
||||
google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM=
|
||||
google.golang.org/protobuf v1.20.1-0.20200309200217-e05f789c0967/go.mod h1:A+miEFZTKqfCUM6K7xSMQL9OKL/b6hQv+e19PK+JZNE=
|
||||
google.golang.org/protobuf v1.21.0/go.mod h1:47Nbq4nVaFHyn7ilMalzfO3qCViNmqZ2kzikPIcrTAo=
|
||||
google.golang.org/protobuf v1.23.0 h1:4MY060fB1DLGMB/7MBTLnwQUY6+F09GEiz6SsrNqyzM=
|
||||
google.golang.org/protobuf v1.23.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
|
||||
gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 h1:qIbj1fsPNlZgppZ+VLlY7N33q108Sa+fhmuc+sWQYwY=
|
||||
@ -749,10 +881,13 @@ gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
|
||||
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
|
||||
gopkg.in/yaml.v2 v2.2.4 h1:/eiJrUcujPVeJ3xlSWaiNi3uSVmDGBK1pDHUHAnao1I=
|
||||
gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
|
||||
gopkg.in/yaml.v2 v2.3.0 h1:clyUAQHOM3G0M3f5vQj7LuJrETvjVot3Z5el9nffUtU=
|
||||
gopkg.in/yaml.v2 v2.3.0/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
|
||||
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c h1:dUUwHk2QECo/6vqA44rthZ8ie2QXMNeKRTHCNY2nXvo=
|
||||
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
gotest.tools v2.2.0+incompatible h1:VsBPFP1AI068pPrMxtb/S8Zkgf9xEmTLJjfM+P5UIEo=
|
||||
gotest.tools v2.2.0+incompatible/go.mod h1:DsYFclhRJ6vuDpmuTbkuFWG+y2sxOXAzmJt81HFBacw=
|
||||
grpc.go4.org v0.0.0-20170609214715-11d0a25b4919/go.mod h1:77eQGdRu53HpSqPFJFmuJdjuHRquDANNeA4x7B8WQ9o=
|
||||
honnef.co/go/tools v0.0.0-20180728063816-88497007e858/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
|
||||
honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
|
||||
honnef.co/go/tools v0.0.0-20190106161140-3f1c8253044a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
|
||||
@ -761,18 +896,23 @@ honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWh
|
||||
honnef.co/go/tools v0.0.1-2019.2.3 h1:3JgtbtFHMiCmsznwGVTUWbgGov+pVqnlf1dEJTNAXeM=
|
||||
honnef.co/go/tools v0.0.1-2019.2.3/go.mod h1:a3bituU0lyd329TUQxRnasdCoJDkEUEAqEt0JzvZhAg=
|
||||
rsc.io/binaryregexp v0.2.0/go.mod h1:qTv7/COck+e2FymRvadv62gMdZztPaShugOCi3I+8D8=
|
||||
rsc.io/quote/v3 v3.1.0/go.mod h1:yEA65RcK8LyAZtP9Kv3t0HmxON59tX3rD+tICJqUlj0=
|
||||
rsc.io/sampler v1.3.0/go.mod h1:T1hPZKmBbMNahiBKFy5HrXp6adAjACjK9JXDnKaTXpA=
|
||||
sourcegraph.com/sourcegraph/go-diff v0.5.0/go.mod h1:kuch7UrkMzY0X+p9CRK03kfuPQ2zzQcaEFbx8wA8rck=
|
||||
sourcegraph.com/sqs/pbtypes v0.0.0-20180604144634-d3ebe8f20ae4/go.mod h1:ketZ/q3QxT9HOBeFhu6RdvsftgpsbFHBF5Cas6cDKZ0=
|
||||
storj.io/common v0.0.0-20200424175742-65ac59022f4f/go.mod h1:pZyXiIE7bGETIRXtfs0nICqMwp7PM8HqnDuyUeldNA0=
|
||||
storj.io/common v0.0.0-20201026135900-1aaeec90670b/go.mod h1:GqdmNf3fLm2UZX/7Zr0BLFCJ4gFjgm6eHrk/fnmr5jQ=
|
||||
storj.io/common v0.0.0-20201027143432-3718579e12bf/go.mod h1:9iobNl9eI6C2M23FS/b37yFYOdHpoeJ8BFFcxsmv538=
|
||||
storj.io/common v0.0.0-20201030120157-90ae6720d87e h1:6baDicBbR0/2XgcQ068KN+B4dF6akkdh2vemmXka1ns=
|
||||
storj.io/common v0.0.0-20201030120157-90ae6720d87e/go.mod h1:9iobNl9eI6C2M23FS/b37yFYOdHpoeJ8BFFcxsmv538=
|
||||
storj.io/common v0.0.0-20201106104920-372a344bdd45 h1:pv552R7MiRA8VLQC4qXczLjbl2Qb/MNyus2E9NBSXgI=
|
||||
storj.io/common v0.0.0-20201106104920-372a344bdd45/go.mod h1:ZkQZup2jpFZvvTgz+yPc7K4Vr4bBHM8AA66P57MZkjk=
|
||||
storj.io/drpc v0.0.11/go.mod h1:TiFc2obNjL9/3isMW1Rpxjy8V9uE0B2HMeMFGiiI7Iw=
|
||||
storj.io/drpc v0.0.11/go.mod h1:TiFc2obNjL9/3isMW1Rpxjy8V9uE0B2HMeMFGiiI7Iw=
|
||||
storj.io/drpc v0.0.14 h1:GCBdymTt1BRw4oHmmUZZlxYXLVRxxYj6x3Ivide2J+I=
|
||||
storj.io/drpc v0.0.14/go.mod h1:82nfl+6YwRwF6UG31cEWWUqv/FaKvP5SGqUvoqTxCMA=
|
||||
storj.io/drpc v0.0.16 h1:9sxypc5lKi/0D69cR21BR0S21+IvXfON8L5nXMVNTwQ=
|
||||
storj.io/drpc v0.0.16/go.mod h1:zdmQ93nx4Z35u11pQ+GAnBy4DGOK3HJCSOfeh2RryTo=
|
||||
storj.io/monkit-jaeger v0.0.0-20200518165323-80778fc3f91b h1:Bbg9JCtY6l3HrDxs3BXzT2UYnYCBLqNi6i84Y8QIPUs=
|
||||
storj.io/monkit-jaeger v0.0.0-20200518165323-80778fc3f91b/go.mod h1:gj4vuCeyCRjRmH8LIrgoyU9Dc9uR6H+/GcDUXmTbf80=
|
||||
storj.io/private v0.0.0-20201026143115-bc926bfa3bca h1:ekR7vtUYC5+cDyim0ZJaSZeXidyzQqDYsnFPYXgTozc=
|
||||
storj.io/private v0.0.0-20201026143115-bc926bfa3bca/go.mod h1:EaLnIyNyqWQUJB+7+KWVez0In9czl0nHHlm2WobebuA=
|
||||
storj.io/uplink v1.3.2-0.20201028181609-f6efc8fcf771 h1:jPbw74xt8bvv8nOfBaM4g9Ts4moX8mqfD4N/B8vEJrA=
|
||||
storj.io/uplink v1.3.2-0.20201028181609-f6efc8fcf771/go.mod h1:5do8jvbs4ao4tLdIZKzNFJPVKOH1oDfvVf8OIsR5Z9E=
|
||||
storj.io/uplink v1.3.2-0.20201106105834-ec8e5cc29f7e h1:a58hzcYciTvBtWST+Byoj76NWuYdnPcz2GK8ynyEyfA=
|
||||
storj.io/uplink v1.3.2-0.20201106105834-ec8e5cc29f7e/go.mod h1:mrdt4I4EhPRC7cnvCD5490IBm423pgKrVoUiC9a5Srg=
|
||||
|
||||
40
multinode/console/members.go
Normal file
40
multinode/console/members.go
Normal file
@ -0,0 +1,40 @@
|
||||
// Copyright (C) 2020 Storj Labs, Inc.
|
||||
// See LICENSE for copying information.
|
||||
|
||||
package console
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"github.com/zeebo/errs"
|
||||
|
||||
"storj.io/common/uuid"
|
||||
)
|
||||
|
||||
// Members exposes needed by MND MembersDB functionality.
|
||||
//
|
||||
// architecture: Database
|
||||
type Members interface {
|
||||
// Invite will create empty row in membersDB.
|
||||
Invite(ctx context.Context, member Member) error
|
||||
// Update updates all updatable fields of member.
|
||||
Update(ctx context.Context, member Member) error
|
||||
// Remove deletes member from membersDB.
|
||||
Remove(ctx context.Context, id uuid.UUID) error
|
||||
// GetByEmail will return member with specified email.
|
||||
GetByEmail(ctx context.Context, email string) (Member, error)
|
||||
// GetByID will return member with specified id.
|
||||
GetByID(ctx context.Context, id uuid.UUID) (Member, error)
|
||||
}
|
||||
|
||||
// ErrNoMember is a special error type that indicates about absence of member in MembersDB.
|
||||
var ErrNoMember = errs.Class("no such member")
|
||||
|
||||
// Member represents some person that is invited to the MND by node owner.
|
||||
// Member will have configurable access privileges that will define which functions and which nodes are available for him.
|
||||
type Member struct {
|
||||
ID uuid.UUID
|
||||
Email string
|
||||
Name string
|
||||
PasswordHash []byte
|
||||
}
|
||||
64
multinode/console/members_test.go
Normal file
64
multinode/console/members_test.go
Normal file
@ -0,0 +1,64 @@
|
||||
// Copyright (C) 2020 Storj Labs, Inc.
|
||||
// See LICENSE for copying information.
|
||||
|
||||
package console_test
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/require"
|
||||
"github.com/zeebo/assert"
|
||||
|
||||
"storj.io/common/testcontext"
|
||||
"storj.io/common/uuid"
|
||||
"storj.io/storj/multinode"
|
||||
"storj.io/storj/multinode/console"
|
||||
"storj.io/storj/multinode/multinodedb/multinodedbtest"
|
||||
)
|
||||
|
||||
func TestMembersDB(t *testing.T) {
|
||||
multinodedbtest.Run(t, func(ctx *testcontext.Context, t *testing.T, db multinode.DB) {
|
||||
members := db.Members()
|
||||
|
||||
memberID, err := uuid.New()
|
||||
require.NoError(t, err)
|
||||
|
||||
memberBob := console.Member{
|
||||
ID: memberID,
|
||||
Email: "mail@example.com",
|
||||
Name: "Bob",
|
||||
PasswordHash: []byte{0},
|
||||
}
|
||||
|
||||
err = members.Invite(ctx, memberBob)
|
||||
assert.NoError(t, err)
|
||||
|
||||
memberToCheck, err := members.GetByEmail(ctx, memberBob.Email)
|
||||
assert.NoError(t, err)
|
||||
assert.Equal(t, memberToCheck.Email, memberBob.Email)
|
||||
assert.Equal(t, memberToCheck.Name, memberBob.Name)
|
||||
assert.Equal(t, memberToCheck.Email, memberBob.Email)
|
||||
|
||||
memberBob.Name = "Alice"
|
||||
err = members.Update(ctx, memberBob)
|
||||
assert.NoError(t, err)
|
||||
|
||||
memberAlice, err := members.GetByID(ctx, memberToCheck.ID)
|
||||
assert.NoError(t, err)
|
||||
assert.Equal(t, memberToCheck.Email, memberAlice.Email)
|
||||
assert.Equal(t, memberToCheck.Name, memberAlice.Name)
|
||||
assert.Equal(t, memberToCheck.Email, memberAlice.Email)
|
||||
assert.Equal(t, memberToCheck.ID, memberAlice.ID)
|
||||
|
||||
err = members.Remove(ctx, memberAlice.ID)
|
||||
assert.NoError(t, err)
|
||||
|
||||
_, err = members.GetByID(ctx, memberToCheck.ID)
|
||||
assert.Error(t, err)
|
||||
assert.Equal(t, true, console.ErrNoMember.Has(err))
|
||||
|
||||
_, err = members.GetByEmail(ctx, memberToCheck.Email)
|
||||
assert.Error(t, err)
|
||||
assert.Equal(t, true, console.ErrNoMember.Has(err))
|
||||
})
|
||||
}
|
||||
@ -1,16 +1,18 @@
|
||||
// Copyright (C) 2020 Storj Labs, Inc.
|
||||
// See LICENSE for copying information.
|
||||
|
||||
package mutlinodedb
|
||||
package multinodedb
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"github.com/spacemonkeygo/monkit/v3"
|
||||
"github.com/zeebo/errs"
|
||||
"go.uber.org/zap"
|
||||
|
||||
"storj.io/storj/multinode"
|
||||
"storj.io/storj/multinode/console"
|
||||
"storj.io/storj/multinode/mutlinodedb/dbx"
|
||||
"storj.io/storj/multinode/multinodedb/dbx"
|
||||
"storj.io/storj/private/dbutil"
|
||||
"storj.io/storj/private/dbutil/pgutil"
|
||||
)
|
||||
@ -39,8 +41,8 @@ type multinodeDB struct {
|
||||
source string
|
||||
}
|
||||
|
||||
// New creates instance of database supports postgres.
|
||||
func New(log *zap.Logger, databaseURL string) (multinode.DB, error) {
|
||||
// Open creates instance of database supports postgres.
|
||||
func Open(ctx context.Context, log *zap.Logger, databaseURL string) (multinode.DB, error) {
|
||||
driver, source, implementation, err := dbutil.SplitConnStr(databaseURL)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
@ -52,17 +54,17 @@ func New(log *zap.Logger, databaseURL string) (multinode.DB, error) {
|
||||
|
||||
source = pgutil.CheckApplicationName(source)
|
||||
|
||||
// dbxDB, err := dbx.Open(driver, source)
|
||||
// if err != nil {
|
||||
// return nil, Error.New("failed opening database via DBX at %q: %v",
|
||||
// source, err)
|
||||
// }
|
||||
// log.Debug("Connected to:", zap.String("db source", source))
|
||||
dbxDB, err := dbx.Open(driver, source)
|
||||
if err != nil {
|
||||
return nil, Error.New("failed opening database via DBX at %q: %v",
|
||||
source, err)
|
||||
}
|
||||
log.Debug("Connected to:", zap.String("db source", source))
|
||||
|
||||
// dbutil.Configure(dbxDB.DB, "multinodedb", mon)
|
||||
dbutil.Configure(ctx, dbxDB.DB, "multinodedb", mon)
|
||||
|
||||
core := &multinodeDB{
|
||||
// DB: dbxDB,
|
||||
DB: dbxDB,
|
||||
|
||||
log: log,
|
||||
driver: driver,
|
||||
@ -77,6 +79,18 @@ func New(log *zap.Logger, databaseURL string) (multinode.DB, error) {
|
||||
func (db *multinodeDB) Nodes() console.Nodes {
|
||||
return &nodes{
|
||||
methods: db,
|
||||
db: db,
|
||||
}
|
||||
}
|
||||
|
||||
// Members returns members database.
|
||||
func (db *multinodeDB) Members() console.Members {
|
||||
return &members{
|
||||
methods: db,
|
||||
}
|
||||
}
|
||||
|
||||
// CreateSchema creates schema.
|
||||
func (db *multinodeDB) CreateSchema(ctx context.Context) error {
|
||||
_, err := db.ExecContext(ctx, db.DB.Schema())
|
||||
return err
|
||||
}
|
||||
24
multinode/multinodedb/dbx/gen.go
Normal file
24
multinode/multinodedb/dbx/gen.go
Normal file
@ -0,0 +1,24 @@
|
||||
// Copyright (C) 2020 Storj Labs, Inc.
|
||||
// See LICENSE for copying information.
|
||||
|
||||
package dbx
|
||||
|
||||
import (
|
||||
"github.com/spacemonkeygo/monkit/v3"
|
||||
"github.com/zeebo/errs"
|
||||
)
|
||||
|
||||
//go:generate sh gen.sh
|
||||
|
||||
var mon = monkit.Package()
|
||||
|
||||
func init() {
|
||||
// catch dbx errors
|
||||
class := errs.Class("multinodedb dbx error")
|
||||
WrapErr = func(e *Error) error {
|
||||
if e.Code == ErrorCode_NoRows {
|
||||
return e.Err
|
||||
}
|
||||
return class.Wrap(e)
|
||||
}
|
||||
}
|
||||
45
multinode/multinodedb/dbx/multinodedb.dbx
Normal file
45
multinode/multinodedb/dbx/multinodedb.dbx
Normal file
@ -0,0 +1,45 @@
|
||||
// dbx.v1 golang multinodedb.dbx .
|
||||
|
||||
model node (
|
||||
key id
|
||||
|
||||
field id blob
|
||||
field name text ( updatable )
|
||||
field tag text ( updatable )
|
||||
field public_address text
|
||||
field api_secret blob
|
||||
field logo blob ( updatable )
|
||||
)
|
||||
|
||||
create node ( )
|
||||
delete node ( where node.id = ? )
|
||||
|
||||
read one (
|
||||
select node
|
||||
where node.id = ?
|
||||
)
|
||||
|
||||
model member (
|
||||
key id
|
||||
|
||||
field id blob
|
||||
field email text ( updatable )
|
||||
field name text ( updatable )
|
||||
field password_hash blob ( updatable )
|
||||
|
||||
field created_at timestamp ( autoinsert )
|
||||
)
|
||||
|
||||
create member ( )
|
||||
delete member ( where member.id = ? )
|
||||
|
||||
update member ( where member.id = ? )
|
||||
|
||||
read one (
|
||||
select member
|
||||
where member.email = ?
|
||||
)
|
||||
read one (
|
||||
select member
|
||||
where member.id = ?
|
||||
)
|
||||
@ -269,7 +269,15 @@ func newpgx(db *DB) *pgxDB {
|
||||
}
|
||||
|
||||
func (obj *pgxDB) Schema() string {
|
||||
return `CREATE TABLE nodes (
|
||||
return `CREATE TABLE members (
|
||||
id bytea NOT NULL,
|
||||
email text NOT NULL,
|
||||
name text NOT NULL,
|
||||
password_hash bytea NOT NULL,
|
||||
created_at timestamp with time zone NOT NULL,
|
||||
PRIMARY KEY ( id )
|
||||
);
|
||||
CREATE TABLE nodes (
|
||||
id bytea NOT NULL,
|
||||
name text NOT NULL,
|
||||
tag text NOT NULL,
|
||||
@ -340,6 +348,117 @@ nextval:
|
||||
fmt.Fprint(f, "]")
|
||||
}
|
||||
|
||||
type Member struct {
|
||||
Id []byte
|
||||
Email string
|
||||
Name string
|
||||
PasswordHash []byte
|
||||
CreatedAt time.Time
|
||||
}
|
||||
|
||||
func (Member) _Table() string { return "members" }
|
||||
|
||||
type Member_Update_Fields struct {
|
||||
Email Member_Email_Field
|
||||
Name Member_Name_Field
|
||||
PasswordHash Member_PasswordHash_Field
|
||||
}
|
||||
|
||||
type Member_Id_Field struct {
|
||||
_set bool
|
||||
_null bool
|
||||
_value []byte
|
||||
}
|
||||
|
||||
func Member_Id(v []byte) Member_Id_Field {
|
||||
return Member_Id_Field{_set: true, _value: v}
|
||||
}
|
||||
|
||||
func (f Member_Id_Field) value() interface{} {
|
||||
if !f._set || f._null {
|
||||
return nil
|
||||
}
|
||||
return f._value
|
||||
}
|
||||
|
||||
func (Member_Id_Field) _Column() string { return "id" }
|
||||
|
||||
type Member_Email_Field struct {
|
||||
_set bool
|
||||
_null bool
|
||||
_value string
|
||||
}
|
||||
|
||||
func Member_Email(v string) Member_Email_Field {
|
||||
return Member_Email_Field{_set: true, _value: v}
|
||||
}
|
||||
|
||||
func (f Member_Email_Field) value() interface{} {
|
||||
if !f._set || f._null {
|
||||
return nil
|
||||
}
|
||||
return f._value
|
||||
}
|
||||
|
||||
func (Member_Email_Field) _Column() string { return "email" }
|
||||
|
||||
type Member_Name_Field struct {
|
||||
_set bool
|
||||
_null bool
|
||||
_value string
|
||||
}
|
||||
|
||||
func Member_Name(v string) Member_Name_Field {
|
||||
return Member_Name_Field{_set: true, _value: v}
|
||||
}
|
||||
|
||||
func (f Member_Name_Field) value() interface{} {
|
||||
if !f._set || f._null {
|
||||
return nil
|
||||
}
|
||||
return f._value
|
||||
}
|
||||
|
||||
func (Member_Name_Field) _Column() string { return "name" }
|
||||
|
||||
type Member_PasswordHash_Field struct {
|
||||
_set bool
|
||||
_null bool
|
||||
_value []byte
|
||||
}
|
||||
|
||||
func Member_PasswordHash(v []byte) Member_PasswordHash_Field {
|
||||
return Member_PasswordHash_Field{_set: true, _value: v}
|
||||
}
|
||||
|
||||
func (f Member_PasswordHash_Field) value() interface{} {
|
||||
if !f._set || f._null {
|
||||
return nil
|
||||
}
|
||||
return f._value
|
||||
}
|
||||
|
||||
func (Member_PasswordHash_Field) _Column() string { return "password_hash" }
|
||||
|
||||
type Member_CreatedAt_Field struct {
|
||||
_set bool
|
||||
_null bool
|
||||
_value time.Time
|
||||
}
|
||||
|
||||
func Member_CreatedAt(v time.Time) Member_CreatedAt_Field {
|
||||
return Member_CreatedAt_Field{_set: true, _value: v}
|
||||
}
|
||||
|
||||
func (f Member_CreatedAt_Field) value() interface{} {
|
||||
if !f._set || f._null {
|
||||
return nil
|
||||
}
|
||||
return f._value
|
||||
}
|
||||
|
||||
func (Member_CreatedAt_Field) _Column() string { return "created_at" }
|
||||
|
||||
type Node struct {
|
||||
Id []byte
|
||||
Name string
|
||||
@ -924,6 +1043,38 @@ func (obj *pgxImpl) Create_Node(ctx context.Context,
|
||||
|
||||
}
|
||||
|
||||
func (obj *pgxImpl) Create_Member(ctx context.Context,
|
||||
member_id Member_Id_Field,
|
||||
member_email Member_Email_Field,
|
||||
member_name Member_Name_Field,
|
||||
member_password_hash Member_PasswordHash_Field) (
|
||||
member *Member, err error) {
|
||||
defer mon.Task()(&ctx)(&err)
|
||||
|
||||
__now := obj.db.Hooks.Now().UTC()
|
||||
__id_val := member_id.value()
|
||||
__email_val := member_email.value()
|
||||
__name_val := member_name.value()
|
||||
__password_hash_val := member_password_hash.value()
|
||||
__created_at_val := __now
|
||||
|
||||
var __embed_stmt = __sqlbundle_Literal("INSERT INTO members ( id, email, name, password_hash, created_at ) VALUES ( ?, ?, ?, ?, ? ) RETURNING members.id, members.email, members.name, members.password_hash, members.created_at")
|
||||
|
||||
var __values []interface{}
|
||||
__values = append(__values, __id_val, __email_val, __name_val, __password_hash_val, __created_at_val)
|
||||
|
||||
var __stmt = __sqlbundle_Render(obj.dialect, __embed_stmt)
|
||||
obj.logStmt(__stmt, __values...)
|
||||
|
||||
member = &Member{}
|
||||
err = obj.driver.QueryRowContext(ctx, __stmt, __values...).Scan(&member.Id, &member.Email, &member.Name, &member.PasswordHash, &member.CreatedAt)
|
||||
if err != nil {
|
||||
return nil, obj.makeErr(err)
|
||||
}
|
||||
return member, nil
|
||||
|
||||
}
|
||||
|
||||
func (obj *pgxImpl) Get_Node_By_Id(ctx context.Context,
|
||||
node_id Node_Id_Field) (
|
||||
node *Node, err error) {
|
||||
@ -946,6 +1097,123 @@ func (obj *pgxImpl) Get_Node_By_Id(ctx context.Context,
|
||||
|
||||
}
|
||||
|
||||
func (obj *pgxImpl) Get_Member_By_Email(ctx context.Context,
|
||||
member_email Member_Email_Field) (
|
||||
member *Member, err error) {
|
||||
defer mon.Task()(&ctx)(&err)
|
||||
|
||||
var __embed_stmt = __sqlbundle_Literal("SELECT members.id, members.email, members.name, members.password_hash, members.created_at FROM members WHERE members.email = ? LIMIT 2")
|
||||
|
||||
var __values []interface{}
|
||||
__values = append(__values, member_email.value())
|
||||
|
||||
var __stmt = __sqlbundle_Render(obj.dialect, __embed_stmt)
|
||||
obj.logStmt(__stmt, __values...)
|
||||
|
||||
__rows, err := obj.driver.QueryContext(ctx, __stmt, __values...)
|
||||
if err != nil {
|
||||
return nil, obj.makeErr(err)
|
||||
}
|
||||
defer __rows.Close()
|
||||
|
||||
if !__rows.Next() {
|
||||
if err := __rows.Err(); err != nil {
|
||||
return nil, obj.makeErr(err)
|
||||
}
|
||||
return nil, makeErr(sql.ErrNoRows)
|
||||
}
|
||||
|
||||
member = &Member{}
|
||||
err = __rows.Scan(&member.Id, &member.Email, &member.Name, &member.PasswordHash, &member.CreatedAt)
|
||||
if err != nil {
|
||||
return nil, obj.makeErr(err)
|
||||
}
|
||||
|
||||
if __rows.Next() {
|
||||
return nil, tooManyRows("Member_By_Email")
|
||||
}
|
||||
|
||||
if err := __rows.Err(); err != nil {
|
||||
return nil, obj.makeErr(err)
|
||||
}
|
||||
|
||||
return member, nil
|
||||
|
||||
}
|
||||
|
||||
func (obj *pgxImpl) Get_Member_By_Id(ctx context.Context,
|
||||
member_id Member_Id_Field) (
|
||||
member *Member, err error) {
|
||||
defer mon.Task()(&ctx)(&err)
|
||||
|
||||
var __embed_stmt = __sqlbundle_Literal("SELECT members.id, members.email, members.name, members.password_hash, members.created_at FROM members WHERE members.id = ?")
|
||||
|
||||
var __values []interface{}
|
||||
__values = append(__values, member_id.value())
|
||||
|
||||
var __stmt = __sqlbundle_Render(obj.dialect, __embed_stmt)
|
||||
obj.logStmt(__stmt, __values...)
|
||||
|
||||
member = &Member{}
|
||||
err = obj.driver.QueryRowContext(ctx, __stmt, __values...).Scan(&member.Id, &member.Email, &member.Name, &member.PasswordHash, &member.CreatedAt)
|
||||
if err != nil {
|
||||
return (*Member)(nil), obj.makeErr(err)
|
||||
}
|
||||
return member, nil
|
||||
|
||||
}
|
||||
|
||||
func (obj *pgxImpl) Update_Member_By_Id(ctx context.Context,
|
||||
member_id Member_Id_Field,
|
||||
update Member_Update_Fields) (
|
||||
member *Member, err error) {
|
||||
defer mon.Task()(&ctx)(&err)
|
||||
var __sets = &__sqlbundle_Hole{}
|
||||
|
||||
var __embed_stmt = __sqlbundle_Literals{Join: "", SQLs: []__sqlbundle_SQL{__sqlbundle_Literal("UPDATE members SET "), __sets, __sqlbundle_Literal(" WHERE members.id = ? RETURNING members.id, members.email, members.name, members.password_hash, members.created_at")}}
|
||||
|
||||
__sets_sql := __sqlbundle_Literals{Join: ", "}
|
||||
var __values []interface{}
|
||||
var __args []interface{}
|
||||
|
||||
if update.Email._set {
|
||||
__values = append(__values, update.Email.value())
|
||||
__sets_sql.SQLs = append(__sets_sql.SQLs, __sqlbundle_Literal("email = ?"))
|
||||
}
|
||||
|
||||
if update.Name._set {
|
||||
__values = append(__values, update.Name.value())
|
||||
__sets_sql.SQLs = append(__sets_sql.SQLs, __sqlbundle_Literal("name = ?"))
|
||||
}
|
||||
|
||||
if update.PasswordHash._set {
|
||||
__values = append(__values, update.PasswordHash.value())
|
||||
__sets_sql.SQLs = append(__sets_sql.SQLs, __sqlbundle_Literal("password_hash = ?"))
|
||||
}
|
||||
|
||||
if len(__sets_sql.SQLs) == 0 {
|
||||
return nil, emptyUpdate()
|
||||
}
|
||||
|
||||
__args = append(__args, member_id.value())
|
||||
|
||||
__values = append(__values, __args...)
|
||||
__sets.SQL = __sets_sql
|
||||
|
||||
var __stmt = __sqlbundle_Render(obj.dialect, __embed_stmt)
|
||||
obj.logStmt(__stmt, __values...)
|
||||
|
||||
member = &Member{}
|
||||
err = obj.driver.QueryRowContext(ctx, __stmt, __values...).Scan(&member.Id, &member.Email, &member.Name, &member.PasswordHash, &member.CreatedAt)
|
||||
if err == sql.ErrNoRows {
|
||||
return nil, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, obj.makeErr(err)
|
||||
}
|
||||
return member, nil
|
||||
}
|
||||
|
||||
func (obj *pgxImpl) Delete_Node_By_Id(ctx context.Context,
|
||||
node_id Node_Id_Field) (
|
||||
deleted bool, err error) {
|
||||
@ -973,6 +1241,33 @@ func (obj *pgxImpl) Delete_Node_By_Id(ctx context.Context,
|
||||
|
||||
}
|
||||
|
||||
func (obj *pgxImpl) Delete_Member_By_Id(ctx context.Context,
|
||||
member_id Member_Id_Field) (
|
||||
deleted bool, err error) {
|
||||
defer mon.Task()(&ctx)(&err)
|
||||
|
||||
var __embed_stmt = __sqlbundle_Literal("DELETE FROM members WHERE members.id = ?")
|
||||
|
||||
var __values []interface{}
|
||||
__values = append(__values, member_id.value())
|
||||
|
||||
var __stmt = __sqlbundle_Render(obj.dialect, __embed_stmt)
|
||||
obj.logStmt(__stmt, __values...)
|
||||
|
||||
__res, err := obj.driver.ExecContext(ctx, __stmt, __values...)
|
||||
if err != nil {
|
||||
return false, obj.makeErr(err)
|
||||
}
|
||||
|
||||
__count, err := __res.RowsAffected()
|
||||
if err != nil {
|
||||
return false, obj.makeErr(err)
|
||||
}
|
||||
|
||||
return __count > 0, nil
|
||||
|
||||
}
|
||||
|
||||
func (impl pgxImpl) isConstraintError(err error) (
|
||||
constraint string, ok bool) {
|
||||
if e, ok := err.(*pgconn.PgError); ok {
|
||||
@ -992,6 +1287,16 @@ func (obj *pgxImpl) deleteAll(ctx context.Context) (count int64, err error) {
|
||||
return 0, obj.makeErr(err)
|
||||
}
|
||||
|
||||
__count, err = __res.RowsAffected()
|
||||
if err != nil {
|
||||
return 0, obj.makeErr(err)
|
||||
}
|
||||
count += __count
|
||||
__res, err = obj.driver.ExecContext(ctx, "DELETE FROM members;")
|
||||
if err != nil {
|
||||
return 0, obj.makeErr(err)
|
||||
}
|
||||
|
||||
__count, err = __res.RowsAffected()
|
||||
if err != nil {
|
||||
return 0, obj.makeErr(err)
|
||||
@ -1044,6 +1349,20 @@ func (rx *Rx) Rollback() (err error) {
|
||||
return err
|
||||
}
|
||||
|
||||
func (rx *Rx) Create_Member(ctx context.Context,
|
||||
member_id Member_Id_Field,
|
||||
member_email Member_Email_Field,
|
||||
member_name Member_Name_Field,
|
||||
member_password_hash Member_PasswordHash_Field) (
|
||||
member *Member, err error) {
|
||||
var tx *Tx
|
||||
if tx, err = rx.getTx(ctx); err != nil {
|
||||
return
|
||||
}
|
||||
return tx.Create_Member(ctx, member_id, member_email, member_name, member_password_hash)
|
||||
|
||||
}
|
||||
|
||||
func (rx *Rx) Create_Node(ctx context.Context,
|
||||
node_id Node_Id_Field,
|
||||
node_name Node_Name_Field,
|
||||
@ -1060,6 +1379,16 @@ func (rx *Rx) Create_Node(ctx context.Context,
|
||||
|
||||
}
|
||||
|
||||
func (rx *Rx) Delete_Member_By_Id(ctx context.Context,
|
||||
member_id Member_Id_Field) (
|
||||
deleted bool, err error) {
|
||||
var tx *Tx
|
||||
if tx, err = rx.getTx(ctx); err != nil {
|
||||
return
|
||||
}
|
||||
return tx.Delete_Member_By_Id(ctx, member_id)
|
||||
}
|
||||
|
||||
func (rx *Rx) Delete_Node_By_Id(ctx context.Context,
|
||||
node_id Node_Id_Field) (
|
||||
deleted bool, err error) {
|
||||
@ -1070,6 +1399,26 @@ func (rx *Rx) Delete_Node_By_Id(ctx context.Context,
|
||||
return tx.Delete_Node_By_Id(ctx, node_id)
|
||||
}
|
||||
|
||||
func (rx *Rx) Get_Member_By_Email(ctx context.Context,
|
||||
member_email Member_Email_Field) (
|
||||
member *Member, err error) {
|
||||
var tx *Tx
|
||||
if tx, err = rx.getTx(ctx); err != nil {
|
||||
return
|
||||
}
|
||||
return tx.Get_Member_By_Email(ctx, member_email)
|
||||
}
|
||||
|
||||
func (rx *Rx) Get_Member_By_Id(ctx context.Context,
|
||||
member_id Member_Id_Field) (
|
||||
member *Member, err error) {
|
||||
var tx *Tx
|
||||
if tx, err = rx.getTx(ctx); err != nil {
|
||||
return
|
||||
}
|
||||
return tx.Get_Member_By_Id(ctx, member_id)
|
||||
}
|
||||
|
||||
func (rx *Rx) Get_Node_By_Id(ctx context.Context,
|
||||
node_id Node_Id_Field) (
|
||||
node *Node, err error) {
|
||||
@ -1080,7 +1429,25 @@ func (rx *Rx) Get_Node_By_Id(ctx context.Context,
|
||||
return tx.Get_Node_By_Id(ctx, node_id)
|
||||
}
|
||||
|
||||
func (rx *Rx) Update_Member_By_Id(ctx context.Context,
|
||||
member_id Member_Id_Field,
|
||||
update Member_Update_Fields) (
|
||||
member *Member, err error) {
|
||||
var tx *Tx
|
||||
if tx, err = rx.getTx(ctx); err != nil {
|
||||
return
|
||||
}
|
||||
return tx.Update_Member_By_Id(ctx, member_id, update)
|
||||
}
|
||||
|
||||
type Methods interface {
|
||||
Create_Member(ctx context.Context,
|
||||
member_id Member_Id_Field,
|
||||
member_email Member_Email_Field,
|
||||
member_name Member_Name_Field,
|
||||
member_password_hash Member_PasswordHash_Field) (
|
||||
member *Member, err error)
|
||||
|
||||
Create_Node(ctx context.Context,
|
||||
node_id Node_Id_Field,
|
||||
node_name Node_Name_Field,
|
||||
@ -1090,13 +1457,30 @@ type Methods interface {
|
||||
node_logo Node_Logo_Field) (
|
||||
node *Node, err error)
|
||||
|
||||
Delete_Member_By_Id(ctx context.Context,
|
||||
member_id Member_Id_Field) (
|
||||
deleted bool, err error)
|
||||
|
||||
Delete_Node_By_Id(ctx context.Context,
|
||||
node_id Node_Id_Field) (
|
||||
deleted bool, err error)
|
||||
|
||||
Get_Member_By_Email(ctx context.Context,
|
||||
member_email Member_Email_Field) (
|
||||
member *Member, err error)
|
||||
|
||||
Get_Member_By_Id(ctx context.Context,
|
||||
member_id Member_Id_Field) (
|
||||
member *Member, err error)
|
||||
|
||||
Get_Node_By_Id(ctx context.Context,
|
||||
node_id Node_Id_Field) (
|
||||
node *Node, err error)
|
||||
|
||||
Update_Member_By_Id(ctx context.Context,
|
||||
member_id Member_Id_Field,
|
||||
update Member_Update_Fields) (
|
||||
member *Member, err error)
|
||||
}
|
||||
|
||||
type TxMethods interface {
|
||||
@ -1,5 +1,13 @@
|
||||
-- AUTOGENERATED BY storj.io/dbx
|
||||
-- DO NOT EDIT
|
||||
CREATE TABLE members (
|
||||
id bytea NOT NULL,
|
||||
email text NOT NULL,
|
||||
name text NOT NULL,
|
||||
password_hash bytea NOT NULL,
|
||||
created_at timestamp with time zone NOT NULL,
|
||||
PRIMARY KEY ( id )
|
||||
);
|
||||
CREATE TABLE nodes (
|
||||
id bytea NOT NULL,
|
||||
name text NOT NULL,
|
||||
117
multinode/multinodedb/members.go
Normal file
117
multinode/multinodedb/members.go
Normal file
@ -0,0 +1,117 @@
|
||||
// Copyright (C) 2020 Storj Labs, Inc.
|
||||
// See LICENSE for copying information.
|
||||
|
||||
package multinodedb
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"errors"
|
||||
|
||||
"github.com/zeebo/errs"
|
||||
|
||||
"storj.io/common/uuid"
|
||||
"storj.io/storj/multinode/console"
|
||||
"storj.io/storj/multinode/multinodedb/dbx"
|
||||
)
|
||||
|
||||
// MembersDBError indicates about internal MembersDB error.
|
||||
var MembersDBError = errs.Class("MembersDB error")
|
||||
|
||||
// ensures that members implements console.Members.
|
||||
var _ console.Members = (*members)(nil)
|
||||
|
||||
// members exposes needed by MND MembersDB functionality.
|
||||
// dbx implementation of console.Members.
|
||||
//
|
||||
// architecture: Database
|
||||
type members struct {
|
||||
methods dbx.Methods
|
||||
}
|
||||
|
||||
// Invite will create empty row in membersDB.
|
||||
func (m *members) Invite(ctx context.Context, member console.Member) (err error) {
|
||||
defer mon.Task()(&ctx)(&err)
|
||||
|
||||
id, err := uuid.New()
|
||||
if err != nil {
|
||||
return MembersDBError.Wrap(err)
|
||||
}
|
||||
|
||||
_, err = m.methods.Create_Member(ctx, dbx.Member_Id(id[:]), dbx.Member_Email(member.Email), dbx.Member_Name(member.Name), dbx.Member_PasswordHash(member.PasswordHash))
|
||||
|
||||
return MembersDBError.Wrap(err)
|
||||
}
|
||||
|
||||
// Update updates all updatable fields of member.
|
||||
func (m *members) Update(ctx context.Context, member console.Member) (err error) {
|
||||
defer mon.Task()(&ctx)(&err)
|
||||
|
||||
_, err = m.methods.Update_Member_By_Id(ctx, dbx.Member_Id(member.ID[:]), dbx.Member_Update_Fields{
|
||||
Email: dbx.Member_Email(member.Email),
|
||||
Name: dbx.Member_Name(member.Name),
|
||||
PasswordHash: dbx.Member_PasswordHash(member.PasswordHash),
|
||||
})
|
||||
|
||||
return MembersDBError.Wrap(err)
|
||||
}
|
||||
|
||||
// Remove deletes member from membersDB.
|
||||
func (m *members) Remove(ctx context.Context, id uuid.UUID) (err error) {
|
||||
defer mon.Task()(&ctx)(&err)
|
||||
|
||||
_, err = m.methods.Delete_Member_By_Id(ctx, dbx.Member_Id(id[:]))
|
||||
|
||||
return MembersDBError.Wrap(err)
|
||||
}
|
||||
|
||||
// GetByEmail will return member with specified email.
|
||||
func (m *members) GetByEmail(ctx context.Context, email string) (_ console.Member, err error) {
|
||||
defer mon.Task()(&ctx)(&err)
|
||||
|
||||
memberDbx, err := m.methods.Get_Member_By_Email(ctx, dbx.Member_Email(email))
|
||||
if err != nil {
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return console.Member{}, console.ErrNoMember.Wrap(err)
|
||||
}
|
||||
return console.Member{}, MembersDBError.Wrap(err)
|
||||
}
|
||||
|
||||
member, err := fromDBXMember(memberDbx)
|
||||
|
||||
return member, MembersDBError.Wrap(err)
|
||||
}
|
||||
|
||||
// GetByID will return member with specified id.
|
||||
func (m *members) GetByID(ctx context.Context, id uuid.UUID) (_ console.Member, err error) {
|
||||
defer mon.Task()(&ctx)(&err)
|
||||
|
||||
memberDbx, err := m.methods.Get_Member_By_Id(ctx, dbx.Member_Id(id[:]))
|
||||
if err != nil {
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return console.Member{}, console.ErrNoMember.Wrap(err)
|
||||
}
|
||||
return console.Member{}, MembersDBError.Wrap(err)
|
||||
}
|
||||
|
||||
member, err := fromDBXMember(memberDbx)
|
||||
|
||||
return member, MembersDBError.Wrap(err)
|
||||
}
|
||||
|
||||
// fromDBXMember converts dbx.Member to console.Member.
|
||||
func fromDBXMember(member *dbx.Member) (_ console.Member, err error) {
|
||||
id, err := uuid.FromBytes(member.Id)
|
||||
if err != nil {
|
||||
return console.Member{}, err
|
||||
}
|
||||
|
||||
result := console.Member{
|
||||
ID: id,
|
||||
Email: member.Email,
|
||||
Name: member.Name,
|
||||
PasswordHash: member.PasswordHash,
|
||||
}
|
||||
|
||||
return result, nil
|
||||
}
|
||||
141
multinode/multinodedb/multinodedbtest/run.go
Normal file
141
multinode/multinodedb/multinodedbtest/run.go
Normal file
@ -0,0 +1,141 @@
|
||||
// Copyright (C) 2020 Storj Labs, Inc.
|
||||
// See LICENSE for copying information.
|
||||
|
||||
package multinodedbtest
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/zeebo/errs"
|
||||
"go.uber.org/zap"
|
||||
"go.uber.org/zap/zaptest"
|
||||
|
||||
"storj.io/common/testcontext"
|
||||
"storj.io/storj/multinode"
|
||||
"storj.io/storj/multinode/multinodedb"
|
||||
"storj.io/storj/multinode/multinodedb/dbx"
|
||||
"storj.io/storj/private/dbutil"
|
||||
"storj.io/storj/private/dbutil/pgtest"
|
||||
"storj.io/storj/private/dbutil/pgutil"
|
||||
"storj.io/storj/private/dbutil/tempdb"
|
||||
)
|
||||
|
||||
// Database describes a test database.
|
||||
type Database struct {
|
||||
Name string
|
||||
URL string
|
||||
Message string
|
||||
}
|
||||
|
||||
// tempMasterDB is a multinode.DB-implementing type that cleans up after itself when closed.
|
||||
type tempMasterDB struct {
|
||||
multinode.DB
|
||||
tempDB *dbutil.TempDatabase
|
||||
}
|
||||
|
||||
// Close closes a tempMasterDB and cleans it up afterward.
|
||||
func (db *tempMasterDB) Close() error {
|
||||
return errs.Combine(db.DB.Close(), db.tempDB.Close())
|
||||
}
|
||||
|
||||
// TestDBAccess provides a somewhat regularized access to the underlying DB.
|
||||
func (db *tempMasterDB) TestDBAccess() *dbx.DB {
|
||||
return db.DB.(interface{ TestDBAccess() *dbx.DB }).TestDBAccess()
|
||||
}
|
||||
|
||||
type ignoreSkip struct{}
|
||||
|
||||
func (ignoreSkip) Skip(...interface{}) {}
|
||||
|
||||
// SchemaSuffix returns a suffix for schemas.
|
||||
func SchemaSuffix() string {
|
||||
return pgutil.CreateRandomTestingSchemaName(6)
|
||||
}
|
||||
|
||||
// SchemaName returns a properly formatted schema string.
|
||||
func SchemaName(testname, category string, index int, schemaSuffix string) string {
|
||||
// postgres has a maximum schema length of 64
|
||||
// we need additional 6 bytes for the random suffix
|
||||
// and 4 bytes for the index "/S0/""
|
||||
|
||||
indexStr := strconv.Itoa(index)
|
||||
|
||||
var maxTestNameLen = 64 - len(category) - len(indexStr) - len(schemaSuffix) - 2
|
||||
if len(testname) > maxTestNameLen {
|
||||
testname = testname[:maxTestNameLen]
|
||||
}
|
||||
|
||||
if schemaSuffix == "" {
|
||||
return strings.ToLower(testname + "/" + category + indexStr)
|
||||
}
|
||||
|
||||
return strings.ToLower(testname + "/" + schemaSuffix + "/" + category + indexStr)
|
||||
}
|
||||
|
||||
// CreateMasterDB creates a new satellite database for testing.
|
||||
func CreateMasterDB(ctx context.Context, log *zap.Logger, name string, category string, index int, dbInfo Database) (db multinode.DB, err error) {
|
||||
if dbInfo.URL == "" {
|
||||
return nil, fmt.Errorf("database %s connection string not provided. %s", dbInfo.Name, dbInfo.Message)
|
||||
}
|
||||
|
||||
schemaSuffix := SchemaSuffix()
|
||||
log.Debug("creating", zap.String("suffix", schemaSuffix))
|
||||
schema := SchemaName(name, category, index, schemaSuffix)
|
||||
|
||||
tempDB, err := tempdb.OpenUnique(ctx, dbInfo.URL, schema)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return CreateMasterDBOnTopOf(ctx, log, tempDB)
|
||||
}
|
||||
|
||||
// CreateMasterDBOnTopOf creates a new satellite database on top of an already existing
|
||||
// temporary database.
|
||||
func CreateMasterDBOnTopOf(ctx context.Context, log *zap.Logger, tempDB *dbutil.TempDatabase) (db multinode.DB, err error) {
|
||||
masterDB, err := multinodedb.Open(ctx, log, tempDB.ConnStr)
|
||||
return &tempMasterDB{DB: masterDB, tempDB: tempDB}, err
|
||||
}
|
||||
|
||||
// Run method will iterate over all supported databases. Will establish
|
||||
// connection and will create tables for each DB.
|
||||
func Run(t *testing.T, test func(ctx *testcontext.Context, t *testing.T, db multinode.DB)) {
|
||||
masterDB := Database{
|
||||
Name: "Postgres",
|
||||
URL: pgtest.PickPostgres(ignoreSkip{}),
|
||||
Message: "Postgres flag missing, example: -postgres-test-db=" + pgtest.DefaultPostgres + " or use STORJ_TEST_POSTGRES environment variable.",
|
||||
}
|
||||
|
||||
t.Run(masterDB.Name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
ctx := testcontext.New(t)
|
||||
defer ctx.Cleanup()
|
||||
|
||||
if masterDB.URL == "" {
|
||||
t.Skipf("Database %s connection string not provided. %s", masterDB.Name, masterDB.Message)
|
||||
}
|
||||
|
||||
db, err := CreateMasterDB(ctx, zaptest.NewLogger(t), t.Name(), "T", 0, masterDB)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer func() {
|
||||
err := db.Close()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}()
|
||||
|
||||
err = db.CreateSchema(ctx)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
test(ctx, t, db)
|
||||
})
|
||||
}
|
||||
@ -1,7 +1,7 @@
|
||||
// Copyright (C) 2020 Storj Labs, Inc.
|
||||
// See LICENSE for copying information.
|
||||
|
||||
package mutlinodedb
|
||||
package multinodedb
|
||||
|
||||
import (
|
||||
"context"
|
||||
@ -10,7 +10,7 @@ import (
|
||||
|
||||
"storj.io/common/storj"
|
||||
"storj.io/storj/multinode/console"
|
||||
"storj.io/storj/multinode/mutlinodedb/dbx"
|
||||
"storj.io/storj/multinode/multinodedb/dbx"
|
||||
)
|
||||
|
||||
// NodesDBError indicates about internal NodesDB error.
|
||||
@ -25,7 +25,6 @@ var _ console.Nodes = (*nodes)(nil)
|
||||
// architecture: Database
|
||||
type nodes struct {
|
||||
methods dbx.Methods
|
||||
db *multinodeDB
|
||||
}
|
||||
|
||||
// Add creates new node in NodesDB.
|
||||
@ -1,12 +0,0 @@
|
||||
// Copyright (C) 2020 Storj Labs, Inc.
|
||||
// See LICENSE for copying information.
|
||||
|
||||
package dbx
|
||||
|
||||
import (
|
||||
"github.com/spacemonkeygo/monkit/v3"
|
||||
)
|
||||
|
||||
//go:generate sh gen.sh
|
||||
|
||||
var mon = monkit.Package()
|
||||
@ -1,20 +0,0 @@
|
||||
// dbx.v1 golang multinodedb.dbx .
|
||||
|
||||
model node (
|
||||
key id
|
||||
|
||||
field id blob
|
||||
field name text ( updatable )
|
||||
field tag text ( updatable )
|
||||
field public_address text
|
||||
field api_secret blob
|
||||
field logo blob ( updatable )
|
||||
)
|
||||
|
||||
create node ( )
|
||||
delete node ( where node.id = ? )
|
||||
|
||||
read one (
|
||||
select node
|
||||
where node.id = ?
|
||||
)
|
||||
@ -27,9 +27,13 @@ var (
|
||||
type DB interface {
|
||||
// Nodes returns nodes database.
|
||||
Nodes() console.Nodes
|
||||
// Members returns members database.
|
||||
Members() console.Members
|
||||
|
||||
// Close closes the database.
|
||||
Close() error
|
||||
// CreateSchema creates schema.
|
||||
CreateSchema(ctx context.Context) error
|
||||
}
|
||||
|
||||
// Config is all the configuration parameters for a Multinode Dashboard.
|
||||
|
||||
@ -5,9 +5,9 @@ package pgtest
|
||||
|
||||
import (
|
||||
"flag"
|
||||
"math/rand"
|
||||
"os"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
|
||||
"storj.io/common/testcontext"
|
||||
@ -77,20 +77,20 @@ func Run(t *testing.T, test func(ctx *testcontext.Context, t *testing.T, connstr
|
||||
}
|
||||
}
|
||||
|
||||
// PickPostgres picks a random postgres database from flag.
|
||||
// PickPostgres picks one postgres database from flag.
|
||||
func PickPostgres(t TB) string {
|
||||
if *postgres == "" || strings.EqualFold(*postgres, "omit") {
|
||||
t.Skip("Postgres flag missing, example: -postgres-test-db=" + DefaultPostgres)
|
||||
}
|
||||
return pickRandom(*postgres)
|
||||
return pickNext(*postgres, &pickPostgres)
|
||||
}
|
||||
|
||||
// PickCockroach picks a random cockroach database from flag.
|
||||
// PickCockroach picks one cockroach database from flag.
|
||||
func PickCockroach(t TB) string {
|
||||
if *cockroach == "" || strings.EqualFold(*cockroach, "omit") {
|
||||
t.Skip("Cockroach flag missing, example: -cockroach-test-db=" + DefaultCockroach)
|
||||
}
|
||||
return pickRandom(*cockroach)
|
||||
return pickNext(*cockroach, &pickCockroach)
|
||||
}
|
||||
|
||||
// PickCockroachAlt picks an alternate cockroach database from flag.
|
||||
@ -104,13 +104,17 @@ func PickCockroachAlt(t TB) string {
|
||||
t.Skip("Cockroach alt flag omitted.")
|
||||
}
|
||||
|
||||
return pickRandom(*cockroachAlt)
|
||||
return pickNext(*cockroachAlt, &pickCockroach)
|
||||
}
|
||||
|
||||
func pickRandom(dbstr string) string {
|
||||
var pickPostgres uint64
|
||||
var pickCockroach uint64
|
||||
|
||||
func pickNext(dbstr string, counter *uint64) string {
|
||||
values := strings.Split(dbstr, ";")
|
||||
if len(values) <= 1 {
|
||||
return dbstr
|
||||
}
|
||||
return values[rand.Intn(len(values))]
|
||||
v := atomic.AddUint64(counter, 1)
|
||||
return values[v%uint64(len(values))]
|
||||
}
|
||||
|
||||
@ -73,84 +73,97 @@ func QuerySchema(ctx context.Context, db dbschema.Queryer) (*dbschema.Schema, er
|
||||
|
||||
func discoverTables(ctx context.Context, db dbschema.Queryer, schema *dbschema.Schema, tableDefinitions []*definition) (err error) {
|
||||
for _, definition := range tableDefinitions {
|
||||
table := schema.EnsureTable(definition.name)
|
||||
|
||||
tableRows, err := db.QueryContext(ctx, `PRAGMA table_info(`+definition.name+`)`)
|
||||
if err != nil {
|
||||
return errs.Wrap(err)
|
||||
}
|
||||
defer func() { err = errs.Combine(err, tableRows.Close()) }()
|
||||
|
||||
for tableRows.Next() {
|
||||
var defaultValue sql.NullString
|
||||
var index, name, columnType string
|
||||
var pk int
|
||||
var notNull bool
|
||||
err := tableRows.Scan(&index, &name, &columnType, ¬Null, &defaultValue, &pk)
|
||||
if err != nil {
|
||||
return errs.Wrap(err)
|
||||
}
|
||||
|
||||
column := &dbschema.Column{
|
||||
Name: name,
|
||||
Type: columnType,
|
||||
IsNullable: !notNull && pk == 0,
|
||||
}
|
||||
table.AddColumn(column)
|
||||
if pk > 0 {
|
||||
if table.PrimaryKey == nil {
|
||||
table.PrimaryKey = make([]string, 0)
|
||||
}
|
||||
table.PrimaryKey = append(table.PrimaryKey, name)
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
matches := rxUnique.FindAllStringSubmatch(definition.sql, -1)
|
||||
for _, match := range matches {
|
||||
// TODO feel this can be done easier
|
||||
var columns []string
|
||||
for _, name := range strings.Split(match[1], ",") {
|
||||
columns = append(columns, strings.TrimSpace(name))
|
||||
}
|
||||
|
||||
table.Unique = append(table.Unique, columns)
|
||||
}
|
||||
|
||||
keysRows, err := db.QueryContext(ctx, `PRAGMA foreign_key_list(`+definition.name+`)`)
|
||||
if err != nil {
|
||||
return errs.Wrap(err)
|
||||
}
|
||||
defer func() { err = errs.Combine(err, keysRows.Close()) }()
|
||||
|
||||
for keysRows.Next() {
|
||||
var id, sec int
|
||||
var tableName, from, to, onUpdate, onDelete, match string
|
||||
err := keysRows.Scan(&id, &sec, &tableName, &from, &to, &onUpdate, &onDelete, &match)
|
||||
if err != nil {
|
||||
return errs.Wrap(err)
|
||||
}
|
||||
|
||||
column, found := table.FindColumn(from)
|
||||
if found {
|
||||
if onDelete == "NO ACTION" {
|
||||
onDelete = ""
|
||||
}
|
||||
if onUpdate == "NO ACTION" {
|
||||
onUpdate = ""
|
||||
}
|
||||
column.Reference = &dbschema.Reference{
|
||||
Table: tableName,
|
||||
Column: to,
|
||||
OnUpdate: onUpdate,
|
||||
OnDelete: onDelete,
|
||||
}
|
||||
}
|
||||
if err := discoverTable(ctx, db, schema, definition); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return errs.Wrap(err)
|
||||
}
|
||||
|
||||
func discoverTable(ctx context.Context, db dbschema.Queryer, schema *dbschema.Schema, definition *definition) (err error) {
|
||||
table := schema.EnsureTable(definition.name)
|
||||
|
||||
tableRows, err := db.QueryContext(ctx, `PRAGMA table_info(`+definition.name+`)`)
|
||||
if err != nil {
|
||||
return errs.Wrap(err)
|
||||
}
|
||||
|
||||
for tableRows.Next() {
|
||||
var defaultValue sql.NullString
|
||||
var index, name, columnType string
|
||||
var pk int
|
||||
var notNull bool
|
||||
err := tableRows.Scan(&index, &name, &columnType, ¬Null, &defaultValue, &pk)
|
||||
if err != nil {
|
||||
return errs.Wrap(errs.Combine(tableRows.Err(), tableRows.Close(), err))
|
||||
}
|
||||
|
||||
column := &dbschema.Column{
|
||||
Name: name,
|
||||
Type: columnType,
|
||||
IsNullable: !notNull && pk == 0,
|
||||
}
|
||||
table.AddColumn(column)
|
||||
if pk > 0 {
|
||||
if table.PrimaryKey == nil {
|
||||
table.PrimaryKey = make([]string, 0)
|
||||
}
|
||||
table.PrimaryKey = append(table.PrimaryKey, name)
|
||||
}
|
||||
}
|
||||
err = errs.Combine(tableRows.Err(), tableRows.Close())
|
||||
if err != nil {
|
||||
return errs.Wrap(err)
|
||||
}
|
||||
|
||||
matches := rxUnique.FindAllStringSubmatch(definition.sql, -1)
|
||||
for _, match := range matches {
|
||||
// TODO feel this can be done easier
|
||||
var columns []string
|
||||
for _, name := range strings.Split(match[1], ",") {
|
||||
columns = append(columns, strings.TrimSpace(name))
|
||||
}
|
||||
|
||||
table.Unique = append(table.Unique, columns)
|
||||
}
|
||||
|
||||
keysRows, err := db.QueryContext(ctx, `PRAGMA foreign_key_list(`+definition.name+`)`)
|
||||
if err != nil {
|
||||
return errs.Wrap(err)
|
||||
}
|
||||
|
||||
for keysRows.Next() {
|
||||
var id, sec int
|
||||
var tableName, from, to, onUpdate, onDelete, match string
|
||||
err := keysRows.Scan(&id, &sec, &tableName, &from, &to, &onUpdate, &onDelete, &match)
|
||||
if err != nil {
|
||||
return errs.Wrap(errs.Combine(keysRows.Err(), keysRows.Close(), err))
|
||||
}
|
||||
|
||||
column, found := table.FindColumn(from)
|
||||
if found {
|
||||
if onDelete == "NO ACTION" {
|
||||
onDelete = ""
|
||||
}
|
||||
if onUpdate == "NO ACTION" {
|
||||
onUpdate = ""
|
||||
}
|
||||
column.Reference = &dbschema.Reference{
|
||||
Table: tableName,
|
||||
Column: to,
|
||||
OnUpdate: onUpdate,
|
||||
OnDelete: onDelete,
|
||||
}
|
||||
}
|
||||
}
|
||||
err = errs.Combine(keysRows.Err(), keysRows.Close())
|
||||
if err != nil {
|
||||
return errs.Wrap(err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func discoverIndexes(ctx context.Context, db dbschema.Queryer, schema *dbschema.Schema, indexDefinitions []*definition) (err error) {
|
||||
// TODO improve indexes discovery
|
||||
for _, definition := range indexDefinitions {
|
||||
|
||||
@ -8,6 +8,7 @@ import (
|
||||
"context"
|
||||
"errors"
|
||||
"runtime/pprof"
|
||||
"time"
|
||||
|
||||
"github.com/spacemonkeygo/monkit/v3"
|
||||
"github.com/zeebo/errs"
|
||||
@ -54,7 +55,27 @@ func (group *Group) Run(ctx context.Context, g *errgroup.Group) {
|
||||
if item.Run == nil {
|
||||
continue
|
||||
}
|
||||
|
||||
shutdownCtx, shutdownFinished := context.WithCancel(context.Background())
|
||||
go func() {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
case <-shutdownCtx.Done():
|
||||
return
|
||||
}
|
||||
|
||||
shutdownDeadline := time.NewTimer(15 * time.Second)
|
||||
defer shutdownDeadline.Stop()
|
||||
select {
|
||||
case <-shutdownDeadline.C:
|
||||
group.log.Warn("service takes long to shutdown", zap.String("name", item.Name))
|
||||
case <-shutdownCtx.Done():
|
||||
}
|
||||
}()
|
||||
|
||||
g.Go(func() error {
|
||||
defer shutdownFinished()
|
||||
|
||||
var err error
|
||||
pprof.Do(ctx, pprof.Labels("name", item.Name), func(ctx context.Context) {
|
||||
err = item.Run(ctx)
|
||||
|
||||
@ -5,6 +5,7 @@ package testblobs
|
||||
|
||||
import (
|
||||
"context"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"go.uber.org/zap"
|
||||
@ -43,11 +44,30 @@ func (bad *BadDB) SetError(err error) {
|
||||
|
||||
// BadBlobs implements a bad blob store.
|
||||
type BadBlobs struct {
|
||||
err error
|
||||
err lockedErr
|
||||
blobs storage.Blobs
|
||||
log *zap.Logger
|
||||
}
|
||||
|
||||
type lockedErr struct {
|
||||
mu sync.Mutex
|
||||
err error
|
||||
}
|
||||
|
||||
// Err returns the error.
|
||||
func (m *lockedErr) Err() error {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
return m.err
|
||||
}
|
||||
|
||||
// Set sets the error.
|
||||
func (m *lockedErr) Set(err error) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
m.err = err
|
||||
}
|
||||
|
||||
// newBadBlobs creates a new bad blob store wrapping the provided blobs.
|
||||
// Use SetError to manually configure the error returned by all operations.
|
||||
func newBadBlobs(log *zap.Logger, blobs storage.Blobs) *BadBlobs {
|
||||
@ -57,27 +77,32 @@ func newBadBlobs(log *zap.Logger, blobs storage.Blobs) *BadBlobs {
|
||||
}
|
||||
}
|
||||
|
||||
// SetError configures the blob store to return a specific error for all operations.
|
||||
func (bad *BadBlobs) SetError(err error) {
|
||||
bad.err.Set(err)
|
||||
}
|
||||
|
||||
// Create creates a new blob that can be written optionally takes a size
|
||||
// argument for performance improvements, -1 is unknown size.
|
||||
func (bad *BadBlobs) Create(ctx context.Context, ref storage.BlobRef, size int64) (storage.BlobWriter, error) {
|
||||
if bad.err != nil {
|
||||
return nil, bad.err
|
||||
if err := bad.err.Err(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return bad.blobs.Create(ctx, ref, size)
|
||||
}
|
||||
|
||||
// Close closes the blob store and any resources associated with it.
|
||||
func (bad *BadBlobs) Close() error {
|
||||
if bad.err != nil {
|
||||
return bad.err
|
||||
if err := bad.err.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
return bad.blobs.Close()
|
||||
}
|
||||
|
||||
// Open opens a reader with the specified namespace and key.
|
||||
func (bad *BadBlobs) Open(ctx context.Context, ref storage.BlobRef) (storage.BlobReader, error) {
|
||||
if bad.err != nil {
|
||||
return nil, bad.err
|
||||
if err := bad.err.Err(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return bad.blobs.Open(ctx, ref)
|
||||
}
|
||||
@ -85,64 +110,64 @@ func (bad *BadBlobs) Open(ctx context.Context, ref storage.BlobRef) (storage.Blo
|
||||
// OpenWithStorageFormat opens a reader for the already-located blob, avoiding the potential need
|
||||
// to check multiple storage formats to find the blob.
|
||||
func (bad *BadBlobs) OpenWithStorageFormat(ctx context.Context, ref storage.BlobRef, formatVer storage.FormatVersion) (storage.BlobReader, error) {
|
||||
if bad.err != nil {
|
||||
return nil, bad.err
|
||||
if err := bad.err.Err(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return bad.blobs.OpenWithStorageFormat(ctx, ref, formatVer)
|
||||
}
|
||||
|
||||
// Trash deletes the blob with the namespace and key.
|
||||
func (bad *BadBlobs) Trash(ctx context.Context, ref storage.BlobRef) error {
|
||||
if bad.err != nil {
|
||||
return bad.err
|
||||
if err := bad.err.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
return bad.blobs.Trash(ctx, ref)
|
||||
}
|
||||
|
||||
// RestoreTrash restores all files in the trash.
|
||||
func (bad *BadBlobs) RestoreTrash(ctx context.Context, namespace []byte) ([][]byte, error) {
|
||||
if bad.err != nil {
|
||||
return nil, bad.err
|
||||
if err := bad.err.Err(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return bad.blobs.RestoreTrash(ctx, namespace)
|
||||
}
|
||||
|
||||
// EmptyTrash empties the trash.
|
||||
func (bad *BadBlobs) EmptyTrash(ctx context.Context, namespace []byte, trashedBefore time.Time) (int64, [][]byte, error) {
|
||||
if bad.err != nil {
|
||||
return 0, nil, bad.err
|
||||
if err := bad.err.Err(); err != nil {
|
||||
return 0, nil, err
|
||||
}
|
||||
return bad.blobs.EmptyTrash(ctx, namespace, trashedBefore)
|
||||
}
|
||||
|
||||
// Delete deletes the blob with the namespace and key.
|
||||
func (bad *BadBlobs) Delete(ctx context.Context, ref storage.BlobRef) error {
|
||||
if bad.err != nil {
|
||||
return bad.err
|
||||
if err := bad.err.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
return bad.blobs.Delete(ctx, ref)
|
||||
}
|
||||
|
||||
// DeleteWithStorageFormat deletes the blob with the namespace, key, and format version.
|
||||
func (bad *BadBlobs) DeleteWithStorageFormat(ctx context.Context, ref storage.BlobRef, formatVer storage.FormatVersion) error {
|
||||
if bad.err != nil {
|
||||
return bad.err
|
||||
if err := bad.err.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
return bad.blobs.DeleteWithStorageFormat(ctx, ref, formatVer)
|
||||
}
|
||||
|
||||
// DeleteNamespace deletes blobs of specific satellite, used after successful GE only.
|
||||
func (bad *BadBlobs) DeleteNamespace(ctx context.Context, ref []byte) (err error) {
|
||||
if bad.err != nil {
|
||||
return bad.err
|
||||
if err := bad.err.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
return bad.blobs.DeleteNamespace(ctx, ref)
|
||||
}
|
||||
|
||||
// Stat looks up disk metadata on the blob file.
|
||||
func (bad *BadBlobs) Stat(ctx context.Context, ref storage.BlobRef) (storage.BlobInfo, error) {
|
||||
if bad.err != nil {
|
||||
return nil, bad.err
|
||||
if err := bad.err.Err(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return bad.blobs.Stat(ctx, ref)
|
||||
}
|
||||
@ -151,8 +176,8 @@ func (bad *BadBlobs) Stat(ctx context.Context, ref storage.BlobRef) (storage.Blo
|
||||
// version. This avoids the potential need to check multiple storage formats for the blob
|
||||
// when the format is already known.
|
||||
func (bad *BadBlobs) StatWithStorageFormat(ctx context.Context, ref storage.BlobRef, formatVer storage.FormatVersion) (storage.BlobInfo, error) {
|
||||
if bad.err != nil {
|
||||
return nil, bad.err
|
||||
if err := bad.err.Err(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return bad.blobs.StatWithStorageFormat(ctx, ref, formatVer)
|
||||
}
|
||||
@ -161,64 +186,64 @@ func (bad *BadBlobs) StatWithStorageFormat(ctx context.Context, ref storage.Blob
|
||||
// If walkFunc returns a non-nil error, WalkNamespace will stop iterating and return the
|
||||
// error immediately.
|
||||
func (bad *BadBlobs) WalkNamespace(ctx context.Context, namespace []byte, walkFunc func(storage.BlobInfo) error) error {
|
||||
if bad.err != nil {
|
||||
return bad.err
|
||||
if err := bad.err.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
return bad.blobs.WalkNamespace(ctx, namespace, walkFunc)
|
||||
}
|
||||
|
||||
// ListNamespaces returns all namespaces that might be storing data.
|
||||
func (bad *BadBlobs) ListNamespaces(ctx context.Context) ([][]byte, error) {
|
||||
if bad.err != nil {
|
||||
return make([][]byte, 0), bad.err
|
||||
if err := bad.err.Err(); err != nil {
|
||||
return make([][]byte, 0), err
|
||||
}
|
||||
return bad.blobs.ListNamespaces(ctx)
|
||||
}
|
||||
|
||||
// FreeSpace return how much free space left for writing.
|
||||
func (bad *BadBlobs) FreeSpace() (int64, error) {
|
||||
if bad.err != nil {
|
||||
return 0, bad.err
|
||||
if err := bad.err.Err(); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return bad.blobs.FreeSpace()
|
||||
}
|
||||
|
||||
// CheckWritability tests writability of the storage directory by creating and deleting a file.
|
||||
func (bad *BadBlobs) CheckWritability() error {
|
||||
if bad.err != nil {
|
||||
return bad.err
|
||||
if err := bad.err.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
return bad.blobs.CheckWritability()
|
||||
}
|
||||
|
||||
// SpaceUsedForBlobs adds up how much is used in all namespaces.
|
||||
func (bad *BadBlobs) SpaceUsedForBlobs(ctx context.Context) (int64, error) {
|
||||
if bad.err != nil {
|
||||
return 0, bad.err
|
||||
if err := bad.err.Err(); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return bad.blobs.SpaceUsedForBlobs(ctx)
|
||||
}
|
||||
|
||||
// SpaceUsedForBlobsInNamespace adds up how much is used in the given namespace.
|
||||
func (bad *BadBlobs) SpaceUsedForBlobsInNamespace(ctx context.Context, namespace []byte) (int64, error) {
|
||||
if bad.err != nil {
|
||||
return 0, bad.err
|
||||
if err := bad.err.Err(); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return bad.blobs.SpaceUsedForBlobsInNamespace(ctx, namespace)
|
||||
}
|
||||
|
||||
// SpaceUsedForTrash adds up how much is used in all namespaces.
|
||||
func (bad *BadBlobs) SpaceUsedForTrash(ctx context.Context) (int64, error) {
|
||||
if bad.err != nil {
|
||||
return 0, bad.err
|
||||
if err := bad.err.Err(); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return bad.blobs.SpaceUsedForTrash(ctx)
|
||||
}
|
||||
|
||||
// CreateVerificationFile creates a file to be used for storage directory verification.
|
||||
func (bad *BadBlobs) CreateVerificationFile(id storj.NodeID) error {
|
||||
if bad.err != nil {
|
||||
return bad.err
|
||||
if err := bad.err.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
return bad.blobs.CreateVerificationFile(id)
|
||||
}
|
||||
@ -226,13 +251,8 @@ func (bad *BadBlobs) CreateVerificationFile(id storj.NodeID) error {
|
||||
// VerifyStorageDir verifies that the storage directory is correct by checking for the existence and validity
|
||||
// of the verification file.
|
||||
func (bad *BadBlobs) VerifyStorageDir(id storj.NodeID) error {
|
||||
if bad.err != nil {
|
||||
return bad.err
|
||||
if err := bad.err.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
return bad.blobs.VerifyStorageDir(id)
|
||||
}
|
||||
|
||||
// SetError configures the blob store to return a specific error for all operations.
|
||||
func (bad *BadBlobs) SetError(err error) {
|
||||
bad.err = err
|
||||
}
|
||||
|
||||
69
private/testplanet/log.go
Normal file
69
private/testplanet/log.go
Normal file
@ -0,0 +1,69 @@
|
||||
// Copyright (C) 2020 Storj Labs, Inc.
|
||||
// See LICENSE for copying information
|
||||
|
||||
package testplanet
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"fmt"
|
||||
"os"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"go.uber.org/zap"
|
||||
"go.uber.org/zap/zapcore"
|
||||
"go.uber.org/zap/zaptest"
|
||||
)
|
||||
|
||||
var useAbsTime = os.Getenv("STORJ_TESTPLANET_ABSTIME")
|
||||
|
||||
func newLogger(t *testing.T) *zap.Logger {
|
||||
if useAbsTime != "" {
|
||||
return zaptest.NewLogger(t)
|
||||
}
|
||||
|
||||
start := time.Now()
|
||||
cfg := zap.NewDevelopmentEncoderConfig()
|
||||
|
||||
cfg.EncodeTime = func(t time.Time, enc zapcore.PrimitiveArrayEncoder) {
|
||||
var nanos, seconds, minutes int64
|
||||
nanos = t.Sub(start).Nanoseconds()
|
||||
|
||||
seconds, nanos = nanos/1e9, nanos%1e9
|
||||
minutes, seconds = seconds/60, seconds%60
|
||||
|
||||
enc.AppendString(fmt.Sprintf("%02d:%02d.%03d", minutes, seconds, nanos/1e6))
|
||||
}
|
||||
enc := zapcore.NewConsoleEncoder(cfg)
|
||||
writer := newTestingWriter(t)
|
||||
return zap.New(
|
||||
zapcore.NewCore(enc, writer, zapcore.DebugLevel),
|
||||
zap.ErrorOutput(writer.WithMarkFailed(true)),
|
||||
)
|
||||
}
|
||||
|
||||
type testingWriter struct {
|
||||
t *testing.T
|
||||
markFailed bool
|
||||
}
|
||||
|
||||
func newTestingWriter(t *testing.T) testingWriter {
|
||||
return testingWriter{t: t}
|
||||
}
|
||||
|
||||
func (w testingWriter) WithMarkFailed(v bool) testingWriter {
|
||||
w.markFailed = v
|
||||
return w
|
||||
}
|
||||
|
||||
func (w testingWriter) Write(p []byte) (n int, err error) {
|
||||
n = len(p)
|
||||
p = bytes.TrimRight(p, "\n")
|
||||
w.t.Logf("%s", p)
|
||||
if w.markFailed {
|
||||
w.t.Fail()
|
||||
}
|
||||
return n, nil
|
||||
}
|
||||
|
||||
func (w testingWriter) Sync() error { return nil }
|
||||
@ -34,18 +34,20 @@ func TestBasic(t *testing.T) {
|
||||
|
||||
for _, sat := range planet.Satellites {
|
||||
for _, sn := range planet.StorageNodes {
|
||||
node := sn.Contact.Service.Local()
|
||||
conn, err := sn.Dialer.DialNodeURL(ctx, sat.NodeURL())
|
||||
func() {
|
||||
node := sn.Contact.Service.Local()
|
||||
conn, err := sn.Dialer.DialNodeURL(ctx, sat.NodeURL())
|
||||
|
||||
require.NoError(t, err)
|
||||
defer ctx.Check(conn.Close)
|
||||
_, err = pb.NewDRPCNodeClient(conn).CheckIn(ctx, &pb.CheckInRequest{
|
||||
Address: node.Address,
|
||||
Version: &node.Version,
|
||||
Capacity: &node.Capacity,
|
||||
Operator: &node.Operator,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, err)
|
||||
defer ctx.Check(conn.Close)
|
||||
_, err = pb.NewDRPCNodeClient(conn).CheckIn(ctx, &pb.CheckInRequest{
|
||||
Address: node.Address,
|
||||
Version: &node.Version,
|
||||
Capacity: &node.Capacity,
|
||||
Operator: &node.Operator,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
}()
|
||||
}
|
||||
}
|
||||
// wait a bit to see whether some failures occur
|
||||
|
||||
@ -67,10 +67,10 @@ var Combine = func(elements ...func(log *zap.Logger, index int, config *satellit
|
||||
// ReconfigureRS returns function to change satellite redundancy scheme values.
|
||||
var ReconfigureRS = func(minThreshold, repairThreshold, successThreshold, totalThreshold int) func(log *zap.Logger, index int, config *satellite.Config) {
|
||||
return func(log *zap.Logger, index int, config *satellite.Config) {
|
||||
config.Metainfo.RS.MinThreshold = minThreshold
|
||||
config.Metainfo.RS.RepairThreshold = repairThreshold
|
||||
config.Metainfo.RS.SuccessThreshold = successThreshold
|
||||
config.Metainfo.RS.TotalThreshold = totalThreshold
|
||||
config.Metainfo.RS.Min = minThreshold
|
||||
config.Metainfo.RS.Repair = repairThreshold
|
||||
config.Metainfo.RS.Success = successThreshold
|
||||
config.Metainfo.RS.Total = totalThreshold
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@ -9,8 +9,6 @@ import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"go.uber.org/zap/zaptest"
|
||||
|
||||
"storj.io/common/testcontext"
|
||||
"storj.io/storj/private/dbutil/pgtest"
|
||||
"storj.io/storj/satellite/satellitedb/satellitedbtest"
|
||||
@ -54,7 +52,7 @@ func Run(t *testing.T, config Config, test func(t *testing.T, ctx *testcontext.C
|
||||
}
|
||||
|
||||
pprof.Do(ctx, pprof.Labels("planet", planetConfig.Name), func(namedctx context.Context) {
|
||||
planet, err := NewCustom(namedctx, zaptest.NewLogger(t), planetConfig, satelliteDB)
|
||||
planet, err := NewCustom(namedctx, newLogger(t), planetConfig, satelliteDB)
|
||||
if err != nil {
|
||||
t.Fatalf("%+v", err)
|
||||
}
|
||||
|
||||
@ -468,16 +468,11 @@ func (planet *Planet) newSatellite(ctx context.Context, prefix string, index int
|
||||
MaxCommitInterval: 1 * time.Hour,
|
||||
Overlay: true,
|
||||
RS: metainfo.RSConfig{
|
||||
MaxBufferMem: memory.Size(256),
|
||||
ErasureShareSize: memory.Size(256),
|
||||
MinThreshold: atLeastOne(planet.config.StorageNodeCount * 1 / 5),
|
||||
RepairThreshold: atLeastOne(planet.config.StorageNodeCount * 2 / 5),
|
||||
SuccessThreshold: atLeastOne(planet.config.StorageNodeCount * 3 / 5),
|
||||
TotalThreshold: atLeastOne(planet.config.StorageNodeCount * 4 / 5),
|
||||
|
||||
MinTotalThreshold: (planet.config.StorageNodeCount * 4 / 5),
|
||||
MaxTotalThreshold: (planet.config.StorageNodeCount * 4 / 5),
|
||||
Validate: false,
|
||||
Min: atLeastOne(planet.config.StorageNodeCount * 1 / 5),
|
||||
Repair: atLeastOne(planet.config.StorageNodeCount * 2 / 5),
|
||||
Success: atLeastOne(planet.config.StorageNodeCount * 3 / 5),
|
||||
Total: atLeastOne(planet.config.StorageNodeCount * 4 / 5),
|
||||
},
|
||||
Loop: metainfo.LoopConfig{
|
||||
CoalesceDuration: 1 * time.Second,
|
||||
|
||||
@ -4,6 +4,6 @@
|
||||
/*
|
||||
Package live provides live accounting functionality. That is, it keeps track
|
||||
of deltas in the amount of storage used by each project relative to the last
|
||||
tally operation (see pkg/accounting/tally).
|
||||
tally operation (see satellite/accounting/tally).
|
||||
*/
|
||||
package live
|
||||
|
||||
@ -217,7 +217,6 @@ func dqNodes(ctx *testcontext.Context, planet *testplanet.Planet) (map[storj.Nod
|
||||
}
|
||||
updateRequests = append(updateRequests, &overlay.UpdateRequest{
|
||||
NodeID: n.ID(),
|
||||
IsUp: true,
|
||||
AuditOutcome: overlay.AuditFailure,
|
||||
})
|
||||
}
|
||||
|
||||
@ -84,6 +84,24 @@ func (service *Service) SetNow(now func() time.Time) {
|
||||
}
|
||||
|
||||
// Tally calculates data-at-rest usage once.
|
||||
//
|
||||
// How live accounting is calculated:
|
||||
//
|
||||
// At the beginning of the tally iteration, we get a map containing the current
|
||||
// project totals from the cache- initialLiveTotals (our current estimation of
|
||||
// the project totals). At the end of the tally iteration, we have the totals
|
||||
// from what we saw during the metainfo loop.
|
||||
//
|
||||
// However, data which was uploaded during the loop may or may not have been
|
||||
// seen in the metainfo loop. For this reason, we also read the live accounting
|
||||
// totals again at the end of the tally iteration- latestLiveTotals.
|
||||
//
|
||||
// The difference between latest and initial indicates how much data was
|
||||
// uploaded during the metainfo loop and is assigned to delta. However, again,
|
||||
// we aren't certain how much of the delta is accounted for in the metainfo
|
||||
// totals. For the reason we make an assumption that 50% of the data is
|
||||
// accounted for. So to calculate the new live accounting totals, we sum the
|
||||
// metainfo totals and 50% of the deltas.
|
||||
func (service *Service) Tally(ctx context.Context) (err error) {
|
||||
defer mon.Task()(&ctx)(&err)
|
||||
|
||||
@ -151,6 +169,9 @@ func (service *Service) Tally(ctx context.Context) (err error) {
|
||||
if delta < 0 {
|
||||
delta = 0
|
||||
}
|
||||
|
||||
// read the method documentation why the increase passed to this method
|
||||
// is calculated in this way
|
||||
err = service.liveAccounting.AddProjectStorageUsage(ctx, projectID, -latestLiveTotals[projectID]+tallyTotal+(delta/2))
|
||||
if err != nil {
|
||||
return Error.Wrap(err)
|
||||
|
||||
@ -143,7 +143,7 @@ func TestCalculateNodeAtRestData(t *testing.T) {
|
||||
require.NoError(t, err)
|
||||
|
||||
// Confirm the correct number of shares were stored
|
||||
rs := satelliteRS(planet.Satellites[0])
|
||||
rs := satelliteRS(t, planet.Satellites[0])
|
||||
if !correctRedundencyScheme(len(obs.Node), rs) {
|
||||
t.Fatalf("expected between: %d and %d, actual: %d", rs.RepairShares, rs.TotalShares, len(obs.Node))
|
||||
}
|
||||
@ -175,7 +175,7 @@ func TestCalculateBucketAtRestData(t *testing.T) {
|
||||
SatelliteCount: 1, StorageNodeCount: 6, UplinkCount: 1,
|
||||
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
|
||||
satellitePeer := planet.Satellites[0]
|
||||
redundancyScheme := satelliteRS(satellitePeer)
|
||||
redundancyScheme := satelliteRS(t, satellitePeer)
|
||||
expectedBucketTallies := make(map[metabase.BucketLocation]*accounting.BucketTally)
|
||||
for _, tt := range testCases {
|
||||
tt := tt // avoid scopelint error
|
||||
@ -221,7 +221,7 @@ func TestTallyIgnoresExpiredPointers(t *testing.T) {
|
||||
SatelliteCount: 1, StorageNodeCount: 6, UplinkCount: 1,
|
||||
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
|
||||
satellitePeer := planet.Satellites[0]
|
||||
redundancyScheme := satelliteRS(satellitePeer)
|
||||
redundancyScheme := satelliteRS(t, satellitePeer)
|
||||
|
||||
projectID, err := uuid.FromString("9656af6e-2d9c-42fa-91f2-bfd516a722d7")
|
||||
require.NoError(t, err)
|
||||
@ -423,12 +423,14 @@ func correctRedundencyScheme(shareCount int, uplinkRS storj.RedundancyScheme) bo
|
||||
return int(uplinkRS.RepairShares) <= shareCount && shareCount <= int(uplinkRS.TotalShares)
|
||||
}
|
||||
|
||||
func satelliteRS(satellite *testplanet.Satellite) storj.RedundancyScheme {
|
||||
func satelliteRS(t *testing.T, satellite *testplanet.Satellite) storj.RedundancyScheme {
|
||||
rs := satellite.Config.Metainfo.RS
|
||||
|
||||
return storj.RedundancyScheme{
|
||||
RequiredShares: int16(satellite.Config.Metainfo.RS.MinThreshold),
|
||||
RepairShares: int16(satellite.Config.Metainfo.RS.RepairThreshold),
|
||||
OptimalShares: int16(satellite.Config.Metainfo.RS.SuccessThreshold),
|
||||
TotalShares: int16(satellite.Config.Metainfo.RS.TotalThreshold),
|
||||
ShareSize: satellite.Config.Metainfo.RS.ErasureShareSize.Int32(),
|
||||
RequiredShares: int16(rs.Min),
|
||||
RepairShares: int16(rs.Repair),
|
||||
OptimalShares: int16(rs.Success),
|
||||
TotalShares: int16(rs.Total),
|
||||
ShareSize: rs.ErasureShareSize.Int32(),
|
||||
}
|
||||
}
|
||||
|
||||
@ -214,7 +214,6 @@ func TestDisqualifiedNodeRemainsDisqualified(t *testing.T) {
|
||||
|
||||
_, err = satellitePeer.Overlay.Service.BatchUpdateStats(ctx, []*overlay.UpdateRequest{{
|
||||
NodeID: disqualifiedNode.ID(),
|
||||
IsUp: true,
|
||||
AuditOutcome: overlay.AuditSuccess,
|
||||
AuditLambda: 0, // forget about history
|
||||
AuditWeight: 1,
|
||||
|
||||
@ -128,7 +128,6 @@ func (reporter *Reporter) recordAuditFailStatus(ctx context.Context, failedAudit
|
||||
for i, nodeID := range failedAuditNodeIDs {
|
||||
updateRequests[i] = &overlay.UpdateRequest{
|
||||
NodeID: nodeID,
|
||||
IsUp: true,
|
||||
AuditOutcome: overlay.AuditFailure,
|
||||
}
|
||||
}
|
||||
@ -148,7 +147,6 @@ func (reporter *Reporter) recordAuditUnknownStatus(ctx context.Context, unknownA
|
||||
for i, nodeID := range unknownAuditNodeIDs {
|
||||
updateRequests[i] = &overlay.UpdateRequest{
|
||||
NodeID: nodeID,
|
||||
IsUp: true,
|
||||
AuditOutcome: overlay.AuditUnknown,
|
||||
}
|
||||
}
|
||||
@ -169,7 +167,6 @@ func (reporter *Reporter) recordOfflineStatus(ctx context.Context, offlineNodeID
|
||||
for i, nodeID := range offlineNodeIDs {
|
||||
updateRequests[i] = &overlay.UpdateRequest{
|
||||
NodeID: nodeID,
|
||||
IsUp: false,
|
||||
AuditOutcome: overlay.AuditOffline,
|
||||
}
|
||||
}
|
||||
@ -191,7 +188,6 @@ func (reporter *Reporter) recordAuditSuccessStatus(ctx context.Context, successN
|
||||
for i, nodeID := range successNodeIDs {
|
||||
updateRequests[i] = &overlay.UpdateRequest{
|
||||
NodeID: nodeID,
|
||||
IsUp: true,
|
||||
AuditOutcome: overlay.AuditSuccess,
|
||||
}
|
||||
}
|
||||
@ -221,7 +217,6 @@ func (reporter *Reporter) recordPendingAudits(ctx context.Context, pendingAudits
|
||||
// record failure -- max reverify count reached
|
||||
updateRequests = append(updateRequests, &overlay.UpdateRequest{
|
||||
NodeID: pendingAudit.NodeID,
|
||||
IsUp: true,
|
||||
AuditOutcome: overlay.AuditFailure,
|
||||
})
|
||||
}
|
||||
|
||||
@ -1074,16 +1074,14 @@ func TestReverifySlowDownload(t *testing.T) {
|
||||
StorageNodeDB: func(index int, db storagenode.DB, log *zap.Logger) (storagenode.DB, error) {
|
||||
return testblobs.NewSlowDB(log.Named("slowdb"), db), nil
|
||||
},
|
||||
Satellite: func(log *zap.Logger, index int, config *satellite.Config) {
|
||||
// These config values are chosen to force the slow node to time out without timing out on the three normal nodes
|
||||
config.Audit.MinBytesPerSecond = 100 * memory.KiB
|
||||
config.Audit.MinDownloadTimeout = 1 * time.Second
|
||||
|
||||
config.Metainfo.RS.MinThreshold = 2
|
||||
config.Metainfo.RS.RepairThreshold = 2
|
||||
config.Metainfo.RS.SuccessThreshold = 4
|
||||
config.Metainfo.RS.TotalThreshold = 4
|
||||
},
|
||||
Satellite: testplanet.Combine(
|
||||
func(log *zap.Logger, index int, config *satellite.Config) {
|
||||
// These config values are chosen to force the slow node to time out without timing out on the three normal nodes
|
||||
config.Audit.MinBytesPerSecond = 100 * memory.KiB
|
||||
config.Audit.MinDownloadTimeout = 1 * time.Second
|
||||
},
|
||||
testplanet.ReconfigureRS(2, 2, 4, 4),
|
||||
),
|
||||
},
|
||||
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
|
||||
satellite := planet.Satellites[0]
|
||||
|
||||
@ -776,16 +776,14 @@ func TestVerifierSlowDownload(t *testing.T) {
|
||||
StorageNodeDB: func(index int, db storagenode.DB, log *zap.Logger) (storagenode.DB, error) {
|
||||
return testblobs.NewSlowDB(log.Named("slowdb"), db), nil
|
||||
},
|
||||
Satellite: func(log *zap.Logger, index int, config *satellite.Config) {
|
||||
// These config values are chosen to force the slow node to time out without timing out on the three normal nodes
|
||||
config.Audit.MinBytesPerSecond = 100 * memory.KiB
|
||||
config.Audit.MinDownloadTimeout = 950 * time.Millisecond
|
||||
|
||||
config.Metainfo.RS.MinThreshold = 2
|
||||
config.Metainfo.RS.RepairThreshold = 2
|
||||
config.Metainfo.RS.SuccessThreshold = 4
|
||||
config.Metainfo.RS.TotalThreshold = 4
|
||||
},
|
||||
Satellite: testplanet.Combine(
|
||||
func(log *zap.Logger, index int, config *satellite.Config) {
|
||||
// These config values are chosen to force the slow node to time out without timing out on the three normal nodes
|
||||
config.Audit.MinBytesPerSecond = 100 * memory.KiB
|
||||
config.Audit.MinDownloadTimeout = 950 * time.Millisecond
|
||||
},
|
||||
testplanet.ReconfigureRS(2, 2, 4, 4),
|
||||
),
|
||||
},
|
||||
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
|
||||
satellite := planet.Satellites[0]
|
||||
|
||||
@ -253,6 +253,29 @@ func (a *Auth) DeleteAccount(w http.ResponseWriter, r *http.Request) {
|
||||
a.serveJSONError(w, errNotImplemented)
|
||||
}
|
||||
|
||||
// ChangeEmail auth user, changes users email for a new one.
|
||||
func (a *Auth) ChangeEmail(w http.ResponseWriter, r *http.Request) {
|
||||
ctx := r.Context()
|
||||
var err error
|
||||
defer mon.Task()(&ctx)(&err)
|
||||
|
||||
var emailChange struct {
|
||||
NewEmail string `json:"newEmail"`
|
||||
}
|
||||
|
||||
err = json.NewDecoder(r.Body).Decode(&emailChange)
|
||||
if err != nil {
|
||||
a.serveJSONError(w, err)
|
||||
return
|
||||
}
|
||||
|
||||
err = a.service.ChangeEmail(ctx, emailChange.NewEmail)
|
||||
if err != nil {
|
||||
a.serveJSONError(w, err)
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
// ChangePassword auth user, changes users password for a new one.
|
||||
func (a *Auth) ChangePassword(w http.ResponseWriter, r *http.Request) {
|
||||
ctx := r.Context()
|
||||
|
||||
@ -39,7 +39,6 @@ func TestAuth_Register(t *testing.T) {
|
||||
},
|
||||
},
|
||||
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
|
||||
|
||||
for i, test := range []struct {
|
||||
Partner string
|
||||
ValidPartner bool
|
||||
@ -50,52 +49,54 @@ func TestAuth_Register(t *testing.T) {
|
||||
{Partner: "Raiden nEtwork", ValidPartner: true},
|
||||
{Partner: "invalid-name", ValidPartner: false},
|
||||
} {
|
||||
registerData := struct {
|
||||
FullName string `json:"fullName"`
|
||||
ShortName string `json:"shortName"`
|
||||
Email string `json:"email"`
|
||||
Partner string `json:"partner"`
|
||||
PartnerID string `json:"partnerId"`
|
||||
Password string `json:"password"`
|
||||
SecretInput string `json:"secret"`
|
||||
ReferrerUserID string `json:"referrerUserId"`
|
||||
}{
|
||||
FullName: "testuser" + strconv.Itoa(i),
|
||||
ShortName: "test",
|
||||
Email: "user@test" + strconv.Itoa(i),
|
||||
Partner: test.Partner,
|
||||
Password: "abc123",
|
||||
}
|
||||
func() {
|
||||
registerData := struct {
|
||||
FullName string `json:"fullName"`
|
||||
ShortName string `json:"shortName"`
|
||||
Email string `json:"email"`
|
||||
Partner string `json:"partner"`
|
||||
PartnerID string `json:"partnerId"`
|
||||
Password string `json:"password"`
|
||||
SecretInput string `json:"secret"`
|
||||
ReferrerUserID string `json:"referrerUserId"`
|
||||
}{
|
||||
FullName: "testuser" + strconv.Itoa(i),
|
||||
ShortName: "test",
|
||||
Email: "user@test" + strconv.Itoa(i),
|
||||
Partner: test.Partner,
|
||||
Password: "abc123",
|
||||
}
|
||||
|
||||
jsonBody, err := json.Marshal(registerData)
|
||||
require.NoError(t, err)
|
||||
|
||||
result, err := http.Post("http://"+planet.Satellites[0].API.Console.Listener.Addr().String()+"/api/v0/auth/register", "application/json", bytes.NewBuffer(jsonBody))
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, http.StatusOK, result.StatusCode)
|
||||
|
||||
defer func() {
|
||||
err = result.Body.Close()
|
||||
jsonBody, err := json.Marshal(registerData)
|
||||
require.NoError(t, err)
|
||||
|
||||
result, err := http.Post("http://"+planet.Satellites[0].API.Console.Listener.Addr().String()+"/api/v0/auth/register", "application/json", bytes.NewBuffer(jsonBody))
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, http.StatusOK, result.StatusCode)
|
||||
|
||||
defer func() {
|
||||
err = result.Body.Close()
|
||||
require.NoError(t, err)
|
||||
}()
|
||||
|
||||
body, err := ioutil.ReadAll(result.Body)
|
||||
require.NoError(t, err)
|
||||
|
||||
var userID uuid.UUID
|
||||
err = json.Unmarshal(body, &userID)
|
||||
require.NoError(t, err)
|
||||
|
||||
user, err := planet.Satellites[0].API.Console.Service.GetUser(ctx, userID)
|
||||
require.NoError(t, err)
|
||||
|
||||
if test.ValidPartner {
|
||||
info, err := planet.Satellites[0].API.Marketing.PartnersService.ByName(ctx, test.Partner)
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, info.UUID, user.PartnerID)
|
||||
} else {
|
||||
require.Equal(t, uuid.UUID{}, user.PartnerID)
|
||||
}
|
||||
}()
|
||||
|
||||
body, err := ioutil.ReadAll(result.Body)
|
||||
require.NoError(t, err)
|
||||
|
||||
var userID uuid.UUID
|
||||
err = json.Unmarshal(body, &userID)
|
||||
require.NoError(t, err)
|
||||
|
||||
user, err := planet.Satellites[0].API.Console.Service.GetUser(ctx, userID)
|
||||
require.NoError(t, err)
|
||||
|
||||
if test.ValidPartner {
|
||||
info, err := planet.Satellites[0].API.Marketing.PartnersService.ByName(ctx, test.Partner)
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, info.UUID, user.PartnerID)
|
||||
} else {
|
||||
require.Equal(t, uuid.UUID{}, user.PartnerID)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
@ -173,6 +173,7 @@ func NewServer(logger *zap.Logger, config Config, service *console.Service, mail
|
||||
authRouter := router.PathPrefix("/api/v0/auth").Subrouter()
|
||||
authRouter.Handle("/account", server.withAuth(http.HandlerFunc(authController.GetAccount))).Methods(http.MethodGet)
|
||||
authRouter.Handle("/account", server.withAuth(http.HandlerFunc(authController.UpdateAccount))).Methods(http.MethodPatch)
|
||||
authRouter.Handle("/account/change-email", server.withAuth(http.HandlerFunc(authController.ChangeEmail))).Methods(http.MethodPost)
|
||||
authRouter.Handle("/account/change-password", server.withAuth(http.HandlerFunc(authController.ChangePassword))).Methods(http.MethodPost)
|
||||
authRouter.Handle("/account/delete", server.withAuth(http.HandlerFunc(authController.DeleteAccount))).Methods(http.MethodPost)
|
||||
authRouter.HandleFunc("/logout", authController.Logout).Methods(http.MethodPost)
|
||||
|
||||
@ -9,6 +9,7 @@ import (
|
||||
"database/sql"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/mail"
|
||||
"sort"
|
||||
"time"
|
||||
|
||||
@ -867,6 +868,32 @@ func (s *Service) UpdateAccount(ctx context.Context, fullName string, shortName
|
||||
return nil
|
||||
}
|
||||
|
||||
// ChangeEmail updates email for a given user.
|
||||
func (s *Service) ChangeEmail(ctx context.Context, newEmail string) (err error) {
|
||||
defer mon.Task()(&ctx)(&err)
|
||||
auth, err := s.getAuthAndAuditLog(ctx, "change email")
|
||||
if err != nil {
|
||||
return Error.Wrap(err)
|
||||
}
|
||||
|
||||
if _, err := mail.ParseAddress(newEmail); err != nil {
|
||||
return ErrValidation.Wrap(err)
|
||||
}
|
||||
|
||||
_, err = s.store.Users().GetByEmail(ctx, newEmail)
|
||||
if err == nil {
|
||||
return ErrEmailUsed.New(emailUsedErrMsg)
|
||||
}
|
||||
|
||||
auth.User.Email = newEmail
|
||||
err = s.store.Users().Update(ctx, &auth.User)
|
||||
if err != nil {
|
||||
return Error.Wrap(err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// ChangePassword updates password for a given user.
|
||||
func (s *Service) ChangePassword(ctx context.Context, pass, newPass string) (err error) {
|
||||
defer mon.Task()(&ctx)(&err)
|
||||
|
||||
@ -130,5 +130,19 @@ func TestService(t *testing.T) {
|
||||
require.Error(t, err)
|
||||
require.Equal(t, "service error: project usage error: some buckets still exist", err.Error())
|
||||
})
|
||||
|
||||
t.Run("TestChangeEmail", func(t *testing.T) {
|
||||
const newEmail = "newEmail@example.com"
|
||||
|
||||
err = service.ChangeEmail(authCtx2, newEmail)
|
||||
require.NoError(t, err)
|
||||
|
||||
userWithUpdatedEmail, err := service.GetUserByEmail(authCtx2, newEmail)
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, newEmail, userWithUpdatedEmail.Email)
|
||||
|
||||
err = service.ChangeEmail(authCtx2, newEmail)
|
||||
require.Error(t, err)
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
35
satellite/console/wasm/README.md
Normal file
35
satellite/console/wasm/README.md
Normal file
@ -0,0 +1,35 @@
|
||||
# Using WebAssembly in Storj
|
||||
|
||||
In order to use the uplink library from the browser, we can compile the uplink library to WebAssembly (wasm).
|
||||
|
||||
### Setup
|
||||
|
||||
To generate wasm code that can create access grants in the web browser, run the following from the storj/wasm directory:
|
||||
```
|
||||
$ GOOS=js GOARCH=wasm go build -o access.wasm access.go
|
||||
```
|
||||
|
||||
The `access.wasm` code can then be loaded into the browser in a script tag in an html page. Also needed is a JavaScript support file which ships with golang.
|
||||
|
||||
To copy the JavaScript support file, run:
|
||||
```
|
||||
$ cp "$(go env GOROOT)/misc/wasm/wasm_exec.js" .
|
||||
```
|
||||
Ref: [Golang WebAssembly docs](https://github.com/golang/go/wiki/WebAssembly)
|
||||
|
||||
The HTML file should include the following:
|
||||
```
|
||||
<script type="text/javascript" src="/path/to/wasm_exec.js"></script>
|
||||
<script>
|
||||
const go = new Go();
|
||||
WebAssembly.instantiateStreaming(
|
||||
fetch("/path/to/access.wasm"), go.importObject).then(
|
||||
(result) => {
|
||||
go.run(result.instance);
|
||||
});
|
||||
</script>
|
||||
```
|
||||
|
||||
Additionally, the HTTP `Content-Security-Policy (CSP) script-src` directive will need to be modified to allow wasm code to be executed.
|
||||
|
||||
See: [WebAssembly Content Security Policy docs](https://github.com/WebAssembly/content-security-policy/blob/master/proposals/CSP.md)
|
||||
64
satellite/console/wasm/access.go
Normal file
64
satellite/console/wasm/access.go
Normal file
@ -0,0 +1,64 @@
|
||||
// +build js,wasm
|
||||
// Copyright (C) 2020 Storj Labs, Inc.
|
||||
// See LICENSE for copying information.
|
||||
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"syscall/js"
|
||||
|
||||
"storj.io/common/encryption"
|
||||
"storj.io/common/macaroon"
|
||||
"storj.io/common/storj"
|
||||
"storj.io/uplink/private/access2"
|
||||
)
|
||||
|
||||
func main() {
|
||||
js.Global().Set("generateAccessGrant", generateAccessGrant())
|
||||
<-make(chan bool)
|
||||
}
|
||||
|
||||
func generateAccessGrant() js.Func {
|
||||
return js.FuncOf(func(this js.Value, args []js.Value) interface{} {
|
||||
if len(args) < 4 {
|
||||
return fmt.Sprintf("Error not enough arguments. Need 4, but only %d supplied. The order of arguments are: satellite Node URL, API key, encryption passphrase, and project salt.", len(args))
|
||||
}
|
||||
satelliteNodeURL := args[0].String()
|
||||
apiKey := args[1].String()
|
||||
encryptionPassphrase := args[2].String()
|
||||
projectSalt := args[3].String()
|
||||
|
||||
return genAccessGrant(satelliteNodeURL,
|
||||
apiKey,
|
||||
encryptionPassphrase,
|
||||
projectSalt,
|
||||
)
|
||||
})
|
||||
}
|
||||
|
||||
func genAccessGrant(satelliteNodeURL, apiKey, encryptionPassphrase, projectSalt string) string {
|
||||
parsedAPIKey, err := macaroon.ParseAPIKey(apiKey)
|
||||
if err != nil {
|
||||
return err.Error()
|
||||
}
|
||||
|
||||
const concurrency = 8
|
||||
key, err := encryption.DeriveRootKey([]byte(encryptionPassphrase), []byte(projectSalt), "", concurrency)
|
||||
if err != nil {
|
||||
return err.Error()
|
||||
}
|
||||
|
||||
encAccess := access2.NewEncryptionAccessWithDefaultKey(key)
|
||||
encAccess.SetDefaultPathCipher(storj.EncAESGCM)
|
||||
a := &access2.Access{
|
||||
SatelliteAddress: satelliteNodeURL,
|
||||
APIKey: parsedAPIKey,
|
||||
EncAccess: encAccess,
|
||||
}
|
||||
accessString, err := a.Serialize()
|
||||
if err != nil {
|
||||
return err.Error()
|
||||
}
|
||||
return accessString
|
||||
}
|
||||
@ -32,14 +32,12 @@ func TestChore(t *testing.T) {
|
||||
StorageNodeCount: 8,
|
||||
UplinkCount: 1,
|
||||
Reconfigure: testplanet.Reconfigure{
|
||||
Satellite: func(log *zap.Logger, index int, config *satellite.Config) {
|
||||
config.GracefulExit.MaxInactiveTimeFrame = maximumInactiveTimeFrame
|
||||
|
||||
config.Metainfo.RS.MinThreshold = 4
|
||||
config.Metainfo.RS.RepairThreshold = 6
|
||||
config.Metainfo.RS.SuccessThreshold = 8
|
||||
config.Metainfo.RS.TotalThreshold = 8
|
||||
},
|
||||
Satellite: testplanet.Combine(
|
||||
func(log *zap.Logger, index int, config *satellite.Config) {
|
||||
config.GracefulExit.MaxInactiveTimeFrame = maximumInactiveTimeFrame
|
||||
},
|
||||
testplanet.ReconfigureRS(4, 6, 8, 8),
|
||||
),
|
||||
},
|
||||
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
|
||||
uplinkPeer := planet.Uplinks[0]
|
||||
@ -136,14 +134,12 @@ func TestDurabilityRatio(t *testing.T) {
|
||||
StorageNodeCount: 4,
|
||||
UplinkCount: 1,
|
||||
Reconfigure: testplanet.Reconfigure{
|
||||
Satellite: func(log *zap.Logger, index int, config *satellite.Config) {
|
||||
config.GracefulExit.MaxInactiveTimeFrame = maximumInactiveTimeFrame
|
||||
|
||||
config.Metainfo.RS.MinThreshold = 2
|
||||
config.Metainfo.RS.RepairThreshold = 3
|
||||
config.Metainfo.RS.SuccessThreshold = successThreshold
|
||||
config.Metainfo.RS.TotalThreshold = 4
|
||||
},
|
||||
Satellite: testplanet.Combine(
|
||||
func(log *zap.Logger, index int, config *satellite.Config) {
|
||||
config.GracefulExit.MaxInactiveTimeFrame = maximumInactiveTimeFrame
|
||||
},
|
||||
testplanet.ReconfigureRS(2, 3, successThreshold, 4),
|
||||
),
|
||||
},
|
||||
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
|
||||
uplinkPeer := planet.Uplinks[0]
|
||||
|
||||
@ -255,16 +255,14 @@ func TestRecvTimeout(t *testing.T) {
|
||||
StorageNodeDB: func(index int, db storagenode.DB, log *zap.Logger) (storagenode.DB, error) {
|
||||
return testblobs.NewSlowDB(log.Named("slowdb"), db), nil
|
||||
},
|
||||
Satellite: func(logger *zap.Logger, index int, config *satellite.Config) {
|
||||
// This config value will create a very short timeframe allowed for receiving
|
||||
// data from storage nodes. This will cause context to cancel with timeout.
|
||||
config.GracefulExit.RecvTimeout = 10 * time.Millisecond
|
||||
|
||||
config.Metainfo.RS.MinThreshold = 2
|
||||
config.Metainfo.RS.RepairThreshold = 3
|
||||
config.Metainfo.RS.SuccessThreshold = successThreshold
|
||||
config.Metainfo.RS.TotalThreshold = successThreshold
|
||||
},
|
||||
Satellite: testplanet.Combine(
|
||||
func(log *zap.Logger, index int, config *satellite.Config) {
|
||||
// This config value will create a very short timeframe allowed for receiving
|
||||
// data from storage nodes. This will cause context to cancel with timeout.
|
||||
config.GracefulExit.RecvTimeout = 10 * time.Millisecond
|
||||
},
|
||||
testplanet.ReconfigureRS(2, 3, successThreshold, successThreshold),
|
||||
),
|
||||
StorageNode: func(index int, config *storagenode.Config) {
|
||||
config.GracefulExit = gracefulexit.Config{
|
||||
ChoreInterval: 2 * time.Minute,
|
||||
@ -1240,17 +1238,15 @@ func TestFailureStorageNodeIgnoresTransferMessages(t *testing.T) {
|
||||
StorageNodeCount: 5,
|
||||
UplinkCount: 1,
|
||||
Reconfigure: testplanet.Reconfigure{
|
||||
Satellite: func(logger *zap.Logger, index int, config *satellite.Config) {
|
||||
// We don't care whether a node gracefully exits or not in this test,
|
||||
// so we set the max failures percentage extra high.
|
||||
config.GracefulExit.OverallMaxFailuresPercentage = 101
|
||||
config.GracefulExit.MaxOrderLimitSendCount = maxOrderLimitSendCount
|
||||
|
||||
config.Metainfo.RS.MinThreshold = 2
|
||||
config.Metainfo.RS.RepairThreshold = 3
|
||||
config.Metainfo.RS.SuccessThreshold = 4
|
||||
config.Metainfo.RS.TotalThreshold = 4
|
||||
},
|
||||
Satellite: testplanet.Combine(
|
||||
func(log *zap.Logger, index int, config *satellite.Config) {
|
||||
// We don't care whether a node gracefully exits or not in this test,
|
||||
// so we set the max failures percentage extra high.
|
||||
config.GracefulExit.OverallMaxFailuresPercentage = 101
|
||||
config.GracefulExit.MaxOrderLimitSendCount = maxOrderLimitSendCount
|
||||
},
|
||||
testplanet.ReconfigureRS(2, 3, 4, 4),
|
||||
),
|
||||
},
|
||||
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
|
||||
uplinkPeer := planet.Uplinks[0]
|
||||
@ -1370,15 +1366,13 @@ func TestIneligibleNodeAge(t *testing.T) {
|
||||
StorageNodeCount: 5,
|
||||
UplinkCount: 1,
|
||||
Reconfigure: testplanet.Reconfigure{
|
||||
Satellite: func(logger *zap.Logger, index int, config *satellite.Config) {
|
||||
// Set the required node age to 1 month.
|
||||
config.GracefulExit.NodeMinAgeInMonths = 1
|
||||
|
||||
config.Metainfo.RS.MinThreshold = 2
|
||||
config.Metainfo.RS.RepairThreshold = 3
|
||||
config.Metainfo.RS.SuccessThreshold = 4
|
||||
config.Metainfo.RS.TotalThreshold = 4
|
||||
},
|
||||
Satellite: testplanet.Combine(
|
||||
func(log *zap.Logger, index int, config *satellite.Config) {
|
||||
// Set the required node age to 1 month.
|
||||
config.GracefulExit.NodeMinAgeInMonths = 1
|
||||
},
|
||||
testplanet.ReconfigureRS(2, 3, 4, 4),
|
||||
),
|
||||
},
|
||||
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
|
||||
uplinkPeer := planet.Uplinks[0]
|
||||
|
||||
@ -5,7 +5,10 @@ package metainfo
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"go.uber.org/zap"
|
||||
@ -26,18 +29,73 @@ const (
|
||||
|
||||
// RSConfig is a configuration struct that keeps details about default
|
||||
// redundancy strategy information.
|
||||
//
|
||||
// Can be used as a flag.
|
||||
type RSConfig struct {
|
||||
MaxBufferMem memory.Size `help:"maximum buffer memory to be allocated for read buffers" default:"4.00MiB"`
|
||||
ErasureShareSize memory.Size `help:"the size of each new erasure share in bytes" default:"256B"`
|
||||
MinThreshold int `help:"the minimum pieces required to recover a segment. k." releaseDefault:"29" devDefault:"4"`
|
||||
RepairThreshold int `help:"the minimum safe pieces before a repair is triggered. m." releaseDefault:"35" devDefault:"6"`
|
||||
SuccessThreshold int `help:"the desired total pieces for a segment. o." releaseDefault:"80" devDefault:"8"`
|
||||
TotalThreshold int `help:"the largest amount of pieces to encode to. n." releaseDefault:"110" devDefault:"10"`
|
||||
ErasureShareSize memory.Size
|
||||
Min int
|
||||
Repair int
|
||||
Success int
|
||||
Total int
|
||||
}
|
||||
|
||||
// TODO left for validation until we will remove CreateSegmentOld
|
||||
MinTotalThreshold int `help:"the largest amount of pieces to encode to. n (lower bound for validation)." releaseDefault:"95" devDefault:"10"`
|
||||
MaxTotalThreshold int `help:"the largest amount of pieces to encode to. n (upper bound for validation)." releaseDefault:"130" devDefault:"10"`
|
||||
Validate bool `help:"validate redundancy scheme configuration" default:"true"`
|
||||
// Type implements pflag.Value.
|
||||
func (RSConfig) Type() string { return "metainfo.RSConfig" }
|
||||
|
||||
// String is required for pflag.Value.
|
||||
func (rs *RSConfig) String() string {
|
||||
return fmt.Sprintf("%d/%d/%d/%d-%s",
|
||||
rs.Min,
|
||||
rs.Repair,
|
||||
rs.Success,
|
||||
rs.Total,
|
||||
rs.ErasureShareSize.String())
|
||||
}
|
||||
|
||||
// Set sets the value from a string in the format k/m/o/n-size (min/repair/optimal/total-erasuresharesize).
|
||||
func (rs *RSConfig) Set(s string) error {
|
||||
// Split on dash. Expect two items. First item is RS numbers. Second item is memory.Size.
|
||||
info := strings.Split(s, "-")
|
||||
if len(info) != 2 {
|
||||
return Error.New("Invalid default RS config (expect format k/m/o/n-ShareSize, got %s)", s)
|
||||
}
|
||||
rsNumbersString := info[0]
|
||||
shareSizeString := info[1]
|
||||
|
||||
// Attempt to parse "-size" part of config.
|
||||
shareSizeInt, err := memory.ParseString(shareSizeString)
|
||||
if err != nil {
|
||||
return Error.New("Invalid share size in RS config: '%s', %w", shareSizeString, err)
|
||||
}
|
||||
shareSize := memory.Size(shareSizeInt)
|
||||
|
||||
// Split on forward slash. Expect exactly four positive non-decreasing integers.
|
||||
rsNumbers := strings.Split(rsNumbersString, "/")
|
||||
if len(rsNumbers) != 4 {
|
||||
return Error.New("Invalid default RS numbers (wrong size, expect 4): %s", rsNumbersString)
|
||||
}
|
||||
|
||||
minValue := 1
|
||||
values := []int{}
|
||||
for _, nextValueString := range rsNumbers {
|
||||
nextValue, err := strconv.Atoi(nextValueString)
|
||||
if err != nil {
|
||||
return Error.New("Invalid default RS numbers (should all be valid integers): %s, %w", rsNumbersString, err)
|
||||
}
|
||||
if nextValue < minValue {
|
||||
return Error.New("Invalid default RS numbers (should be non-decreasing): %s", rsNumbersString)
|
||||
}
|
||||
values = append(values, nextValue)
|
||||
minValue = nextValue
|
||||
}
|
||||
|
||||
rs.ErasureShareSize = shareSize
|
||||
rs.Min = values[0]
|
||||
rs.Repair = values[1]
|
||||
rs.Success = values[2]
|
||||
rs.Total = values[3]
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// RateLimiterConfig is a configuration struct for endpoint rate limiting.
|
||||
|
||||
98
satellite/metainfo/config_test.go
Normal file
98
satellite/metainfo/config_test.go
Normal file
@ -0,0 +1,98 @@
|
||||
// Copyright (C) 2020 Storj Labs, Inc.
|
||||
// See LICENSE for copying information.
|
||||
|
||||
package metainfo_test
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/require"
|
||||
|
||||
"storj.io/common/memory"
|
||||
"storj.io/storj/satellite/metainfo"
|
||||
)
|
||||
|
||||
func TestRSConfigValidation(t *testing.T) {
|
||||
tests := []struct {
|
||||
description string
|
||||
configString string
|
||||
expectedConfig metainfo.RSConfig
|
||||
expectError bool
|
||||
}{
|
||||
{
|
||||
description: "valid rs config",
|
||||
configString: "4/8/10/20-256B",
|
||||
expectedConfig: metainfo.RSConfig{
|
||||
ErasureShareSize: 256 * memory.B, Min: 4, Repair: 8, Success: 10, Total: 20,
|
||||
},
|
||||
expectError: false,
|
||||
},
|
||||
{
|
||||
description: "invalid rs config - numbers decrease",
|
||||
configString: "4/8/5/20-256B",
|
||||
expectError: true,
|
||||
},
|
||||
{
|
||||
description: "invalid rs config - starts at 0",
|
||||
configString: "0/2/4/6-256B",
|
||||
expectError: true,
|
||||
},
|
||||
{
|
||||
description: "invalid rs config - strings",
|
||||
configString: "4/a/b/20-256B",
|
||||
expectError: true,
|
||||
},
|
||||
{
|
||||
description: "invalid rs config - floating-point numbers",
|
||||
configString: "4/5.2/7/20-256B",
|
||||
expectError: true,
|
||||
},
|
||||
{
|
||||
description: "invalid rs config - not enough items",
|
||||
configString: "4/5/20-256B",
|
||||
expectError: true,
|
||||
},
|
||||
{
|
||||
description: "invalid rs config - too many items",
|
||||
configString: "4/5/20/30/50-256B",
|
||||
expectError: true,
|
||||
},
|
||||
{
|
||||
description: "invalid rs config - empty numbers",
|
||||
configString: "-256B",
|
||||
expectError: true,
|
||||
},
|
||||
{
|
||||
description: "invalid rs config - empty size",
|
||||
configString: "1/2/3/4-",
|
||||
expectError: true,
|
||||
},
|
||||
{
|
||||
description: "invalid rs config - empty",
|
||||
configString: "",
|
||||
expectError: true,
|
||||
},
|
||||
{
|
||||
description: "invalid valid rs config - invalid share size",
|
||||
configString: "4/8/10/20-256A",
|
||||
expectError: true,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Log(tt.description)
|
||||
|
||||
rsConfig := metainfo.RSConfig{}
|
||||
err := rsConfig.Set(tt.configString)
|
||||
if tt.expectError {
|
||||
require.Error(t, err)
|
||||
} else {
|
||||
require.NoError(t, err)
|
||||
require.EqualValues(t, tt.expectedConfig.ErasureShareSize, rsConfig.ErasureShareSize)
|
||||
require.EqualValues(t, tt.expectedConfig.Min, rsConfig.Min)
|
||||
require.EqualValues(t, tt.expectedConfig.Repair, rsConfig.Repair)
|
||||
require.EqualValues(t, tt.expectedConfig.Success, rsConfig.Success)
|
||||
require.EqualValues(t, tt.expectedConfig.Total, rsConfig.Total)
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -79,6 +79,7 @@ type Endpoint struct {
|
||||
limiterCache *lrucache.ExpiringLRU
|
||||
encInlineSegmentSize int64 // max inline segment size + encryption overhead
|
||||
revocations revocation.DB
|
||||
defaultRS *pb.RedundancyScheme
|
||||
config Config
|
||||
}
|
||||
|
||||
@ -97,6 +98,16 @@ func NewEndpoint(log *zap.Logger, metainfo *Service, deletePieces *piecedeletion
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
defaultRSScheme := &pb.RedundancyScheme{
|
||||
Type: pb.RedundancyScheme_RS,
|
||||
MinReq: int32(config.RS.Min),
|
||||
RepairThreshold: int32(config.RS.Repair),
|
||||
SuccessThreshold: int32(config.RS.Success),
|
||||
Total: int32(config.RS.Total),
|
||||
ErasureShareSize: config.RS.ErasureShareSize.Int32(),
|
||||
}
|
||||
|
||||
return &Endpoint{
|
||||
log: log,
|
||||
metainfo: metainfo,
|
||||
@ -116,6 +127,7 @@ func NewEndpoint(log *zap.Logger, metainfo *Service, deletePieces *piecedeletion
|
||||
}),
|
||||
encInlineSegmentSize: encInlineSegmentSize,
|
||||
revocations: revocations,
|
||||
defaultRS: defaultRSScheme,
|
||||
config: config,
|
||||
}, nil
|
||||
}
|
||||
@ -241,7 +253,7 @@ func (endpoint *Endpoint) GetBucket(ctx context.Context, req *pb.BucketGetReques
|
||||
}
|
||||
|
||||
// override RS to fit satellite settings
|
||||
convBucket, err := convertBucketToProto(bucket, endpoint.redundancyScheme())
|
||||
convBucket, err := convertBucketToProto(bucket, endpoint.defaultRS)
|
||||
if err != nil {
|
||||
return resp, err
|
||||
}
|
||||
@ -316,7 +328,7 @@ func (endpoint *Endpoint) CreateBucket(ctx context.Context, req *pb.BucketCreate
|
||||
}
|
||||
|
||||
// override RS to fit satellite settings
|
||||
convBucket, err := convertBucketToProto(bucket, endpoint.redundancyScheme())
|
||||
convBucket, err := convertBucketToProto(bucket, endpoint.defaultRS)
|
||||
if err != nil {
|
||||
endpoint.log.Error("error while converting bucket to proto", zap.String("bucketName", bucket.Name), zap.Error(err))
|
||||
return nil, rpcstatus.Error(rpcstatus.Internal, "unable to create bucket")
|
||||
@ -375,7 +387,7 @@ func (endpoint *Endpoint) DeleteBucket(ctx context.Context, req *pb.BucketDelete
|
||||
return nil, err
|
||||
}
|
||||
|
||||
convBucket, err = convertBucketToProto(bucket, endpoint.redundancyScheme())
|
||||
convBucket, err = convertBucketToProto(bucket, endpoint.defaultRS)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@ -657,7 +669,7 @@ func (endpoint *Endpoint) BeginObject(ctx context.Context, req *pb.ObjectBeginRe
|
||||
}
|
||||
|
||||
// use only satellite values for Redundancy Scheme
|
||||
pbRS := endpoint.redundancyScheme()
|
||||
pbRS := endpoint.defaultRS
|
||||
|
||||
streamID, err := endpoint.packStreamID(ctx, &internalpb.StreamID{
|
||||
Bucket: req.Bucket,
|
||||
@ -1825,17 +1837,6 @@ func groupPiecesByNodeID(segments []metabase.DeletedSegmentInfo) map[storj.NodeI
|
||||
return piecesToDelete
|
||||
}
|
||||
|
||||
func (endpoint *Endpoint) redundancyScheme() *pb.RedundancyScheme {
|
||||
return &pb.RedundancyScheme{
|
||||
Type: pb.RedundancyScheme_RS,
|
||||
MinReq: int32(endpoint.config.RS.MinThreshold),
|
||||
RepairThreshold: int32(endpoint.config.RS.RepairThreshold),
|
||||
SuccessThreshold: int32(endpoint.config.RS.SuccessThreshold),
|
||||
Total: int32(endpoint.config.RS.TotalThreshold),
|
||||
ErasureShareSize: endpoint.config.RS.ErasureShareSize.Int32(),
|
||||
}
|
||||
}
|
||||
|
||||
// RevokeAPIKey handles requests to revoke an api key.
|
||||
func (endpoint *Endpoint) RevokeAPIKey(ctx context.Context, req *pb.RevokeAPIKeyRequest) (resp *pb.RevokeAPIKeyResponse, err error) {
|
||||
defer mon.Task()(&ctx)(&err)
|
||||
|
||||
@ -215,85 +215,87 @@ func TestInvalidAPIKey(t *testing.T) {
|
||||
require.NoError(t, err)
|
||||
|
||||
for _, invalidAPIKey := range []string{"", "invalid", "testKey"} {
|
||||
client, err := planet.Uplinks[0].DialMetainfo(ctx, planet.Satellites[0], throwawayKey)
|
||||
require.NoError(t, err)
|
||||
defer ctx.Check(client.Close)
|
||||
func() {
|
||||
client, err := planet.Uplinks[0].DialMetainfo(ctx, planet.Satellites[0], throwawayKey)
|
||||
require.NoError(t, err)
|
||||
defer ctx.Check(client.Close)
|
||||
|
||||
client.SetRawAPIKey([]byte(invalidAPIKey))
|
||||
client.SetRawAPIKey([]byte(invalidAPIKey))
|
||||
|
||||
_, err = client.BeginObject(ctx, metainfo.BeginObjectParams{})
|
||||
assertInvalidArgument(t, err, false)
|
||||
_, err = client.BeginObject(ctx, metainfo.BeginObjectParams{})
|
||||
assertInvalidArgument(t, err, false)
|
||||
|
||||
_, err = client.BeginDeleteObject(ctx, metainfo.BeginDeleteObjectParams{})
|
||||
assertInvalidArgument(t, err, false)
|
||||
_, err = client.BeginDeleteObject(ctx, metainfo.BeginDeleteObjectParams{})
|
||||
assertInvalidArgument(t, err, false)
|
||||
|
||||
_, err = client.ListBuckets(ctx, metainfo.ListBucketsParams{})
|
||||
assertInvalidArgument(t, err, false)
|
||||
_, err = client.ListBuckets(ctx, metainfo.ListBucketsParams{})
|
||||
assertInvalidArgument(t, err, false)
|
||||
|
||||
_, _, err = client.ListObjects(ctx, metainfo.ListObjectsParams{})
|
||||
assertInvalidArgument(t, err, false)
|
||||
_, _, err = client.ListObjects(ctx, metainfo.ListObjectsParams{})
|
||||
assertInvalidArgument(t, err, false)
|
||||
|
||||
_, err = client.CreateBucket(ctx, metainfo.CreateBucketParams{})
|
||||
assertInvalidArgument(t, err, false)
|
||||
_, err = client.CreateBucket(ctx, metainfo.CreateBucketParams{})
|
||||
assertInvalidArgument(t, err, false)
|
||||
|
||||
_, err = client.DeleteBucket(ctx, metainfo.DeleteBucketParams{})
|
||||
assertInvalidArgument(t, err, false)
|
||||
_, err = client.DeleteBucket(ctx, metainfo.DeleteBucketParams{})
|
||||
assertInvalidArgument(t, err, false)
|
||||
|
||||
_, err = client.BeginDeleteObject(ctx, metainfo.BeginDeleteObjectParams{})
|
||||
assertInvalidArgument(t, err, false)
|
||||
_, err = client.BeginDeleteObject(ctx, metainfo.BeginDeleteObjectParams{})
|
||||
assertInvalidArgument(t, err, false)
|
||||
|
||||
_, err = client.GetBucket(ctx, metainfo.GetBucketParams{})
|
||||
assertInvalidArgument(t, err, false)
|
||||
_, err = client.GetBucket(ctx, metainfo.GetBucketParams{})
|
||||
assertInvalidArgument(t, err, false)
|
||||
|
||||
_, err = client.GetObject(ctx, metainfo.GetObjectParams{})
|
||||
assertInvalidArgument(t, err, false)
|
||||
_, err = client.GetObject(ctx, metainfo.GetObjectParams{})
|
||||
assertInvalidArgument(t, err, false)
|
||||
|
||||
_, err = client.GetProjectInfo(ctx)
|
||||
assertInvalidArgument(t, err, false)
|
||||
_, err = client.GetProjectInfo(ctx)
|
||||
assertInvalidArgument(t, err, false)
|
||||
|
||||
// these methods needs StreamID to do authentication
|
||||
// these methods needs StreamID to do authentication
|
||||
|
||||
signer := signing.SignerFromFullIdentity(planet.Satellites[0].Identity)
|
||||
satStreamID := &internalpb.StreamID{
|
||||
CreationDate: time.Now(),
|
||||
}
|
||||
signedStreamID, err := satMetainfo.SignStreamID(ctx, signer, satStreamID)
|
||||
require.NoError(t, err)
|
||||
signer := signing.SignerFromFullIdentity(planet.Satellites[0].Identity)
|
||||
satStreamID := &internalpb.StreamID{
|
||||
CreationDate: time.Now(),
|
||||
}
|
||||
signedStreamID, err := satMetainfo.SignStreamID(ctx, signer, satStreamID)
|
||||
require.NoError(t, err)
|
||||
|
||||
encodedStreamID, err := pb.Marshal(signedStreamID)
|
||||
require.NoError(t, err)
|
||||
encodedStreamID, err := pb.Marshal(signedStreamID)
|
||||
require.NoError(t, err)
|
||||
|
||||
streamID, err := storj.StreamIDFromBytes(encodedStreamID)
|
||||
require.NoError(t, err)
|
||||
streamID, err := storj.StreamIDFromBytes(encodedStreamID)
|
||||
require.NoError(t, err)
|
||||
|
||||
err = client.CommitObject(ctx, metainfo.CommitObjectParams{StreamID: streamID})
|
||||
assertInvalidArgument(t, err, false)
|
||||
err = client.CommitObject(ctx, metainfo.CommitObjectParams{StreamID: streamID})
|
||||
assertInvalidArgument(t, err, false)
|
||||
|
||||
_, _, _, err = client.BeginSegment(ctx, metainfo.BeginSegmentParams{StreamID: streamID})
|
||||
assertInvalidArgument(t, err, false)
|
||||
_, _, _, err = client.BeginSegment(ctx, metainfo.BeginSegmentParams{StreamID: streamID})
|
||||
assertInvalidArgument(t, err, false)
|
||||
|
||||
err = client.MakeInlineSegment(ctx, metainfo.MakeInlineSegmentParams{StreamID: streamID})
|
||||
assertInvalidArgument(t, err, false)
|
||||
err = client.MakeInlineSegment(ctx, metainfo.MakeInlineSegmentParams{StreamID: streamID})
|
||||
assertInvalidArgument(t, err, false)
|
||||
|
||||
_, _, err = client.DownloadSegment(ctx, metainfo.DownloadSegmentParams{StreamID: streamID})
|
||||
assertInvalidArgument(t, err, false)
|
||||
_, _, err = client.DownloadSegment(ctx, metainfo.DownloadSegmentParams{StreamID: streamID})
|
||||
assertInvalidArgument(t, err, false)
|
||||
|
||||
// these methods needs SegmentID
|
||||
// these methods needs SegmentID
|
||||
|
||||
signedSegmentID, err := satMetainfo.SignSegmentID(ctx, signer, &internalpb.SegmentID{
|
||||
StreamId: satStreamID,
|
||||
CreationDate: time.Now(),
|
||||
})
|
||||
require.NoError(t, err)
|
||||
signedSegmentID, err := satMetainfo.SignSegmentID(ctx, signer, &internalpb.SegmentID{
|
||||
StreamId: satStreamID,
|
||||
CreationDate: time.Now(),
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
encodedSegmentID, err := pb.Marshal(signedSegmentID)
|
||||
require.NoError(t, err)
|
||||
encodedSegmentID, err := pb.Marshal(signedSegmentID)
|
||||
require.NoError(t, err)
|
||||
|
||||
segmentID, err := storj.SegmentIDFromBytes(encodedSegmentID)
|
||||
require.NoError(t, err)
|
||||
segmentID, err := storj.SegmentIDFromBytes(encodedSegmentID)
|
||||
require.NoError(t, err)
|
||||
|
||||
err = client.CommitSegment(ctx, metainfo.CommitSegmentParams{SegmentID: segmentID})
|
||||
assertInvalidArgument(t, err, false)
|
||||
err = client.CommitSegment(ctx, metainfo.CommitSegmentParams{SegmentID: segmentID})
|
||||
assertInvalidArgument(t, err, false)
|
||||
}()
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
@ -346,14 +346,13 @@ func TestService_DeletePieces_Timeout(t *testing.T) {
|
||||
StorageNodeDB: func(index int, db storagenode.DB, log *zap.Logger) (storagenode.DB, error) {
|
||||
return testblobs.NewSlowDB(log.Named("slowdb"), db), nil
|
||||
},
|
||||
Satellite: func(log *zap.Logger, index int, config *satellite.Config) {
|
||||
config.Metainfo.PieceDeletion.RequestTimeout = 200 * time.Millisecond
|
||||
config.Metainfo.RS.MinThreshold = 2
|
||||
config.Metainfo.RS.RepairThreshold = 2
|
||||
config.Metainfo.RS.SuccessThreshold = 4
|
||||
config.Metainfo.RS.TotalThreshold = 4
|
||||
config.Metainfo.MaxSegmentSize = 15 * memory.KiB
|
||||
},
|
||||
Satellite: testplanet.Combine(
|
||||
func(log *zap.Logger, index int, config *satellite.Config) {
|
||||
config.Metainfo.PieceDeletion.RequestTimeout = 200 * time.Millisecond
|
||||
config.Metainfo.MaxSegmentSize = 15 * memory.KiB
|
||||
},
|
||||
testplanet.ReconfigureRS(2, 2, 4, 4),
|
||||
),
|
||||
},
|
||||
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
|
||||
uplnk := planet.Uplinks[0]
|
||||
|
||||
@ -76,106 +76,108 @@ func TestSettlementWithWindowEndpointManyOrders(t *testing.T) {
|
||||
}
|
||||
|
||||
for _, tt := range testCases {
|
||||
// create serial number to use in test. must be unique for each run.
|
||||
serialNumber1 := testrand.SerialNumber()
|
||||
err = ordersDB.CreateSerialInfo(ctx, serialNumber1, []byte(bucketID), now.AddDate(1, 0, 10))
|
||||
require.NoError(t, err)
|
||||
func() {
|
||||
// create serial number to use in test. must be unique for each run.
|
||||
serialNumber1 := testrand.SerialNumber()
|
||||
err = ordersDB.CreateSerialInfo(ctx, serialNumber1, []byte(bucketID), now.AddDate(1, 0, 10))
|
||||
require.NoError(t, err)
|
||||
|
||||
serialNumber2 := testrand.SerialNumber()
|
||||
err = ordersDB.CreateSerialInfo(ctx, serialNumber2, []byte(bucketID), now.AddDate(1, 0, 10))
|
||||
require.NoError(t, err)
|
||||
serialNumber2 := testrand.SerialNumber()
|
||||
err = ordersDB.CreateSerialInfo(ctx, serialNumber2, []byte(bucketID), now.AddDate(1, 0, 10))
|
||||
require.NoError(t, err)
|
||||
|
||||
piecePublicKey, piecePrivateKey, err := storj.NewPieceKey()
|
||||
require.NoError(t, err)
|
||||
piecePublicKey, piecePrivateKey, err := storj.NewPieceKey()
|
||||
require.NoError(t, err)
|
||||
|
||||
// create signed orderlimit or order to test with
|
||||
limit1 := &pb.OrderLimit{
|
||||
SerialNumber: serialNumber1,
|
||||
SatelliteId: satellite.ID(),
|
||||
UplinkPublicKey: piecePublicKey,
|
||||
StorageNodeId: storagenode.ID(),
|
||||
PieceId: storj.NewPieceID(),
|
||||
Action: pb.PieceAction_PUT,
|
||||
Limit: 1000,
|
||||
PieceExpiration: time.Time{},
|
||||
OrderCreation: tt.orderCreation,
|
||||
OrderExpiration: now.Add(24 * time.Hour),
|
||||
}
|
||||
orderLimit1, err := signing.SignOrderLimit(ctx, signing.SignerFromFullIdentity(satellite.Identity), limit1)
|
||||
require.NoError(t, err)
|
||||
// create signed orderlimit or order to test with
|
||||
limit1 := &pb.OrderLimit{
|
||||
SerialNumber: serialNumber1,
|
||||
SatelliteId: satellite.ID(),
|
||||
UplinkPublicKey: piecePublicKey,
|
||||
StorageNodeId: storagenode.ID(),
|
||||
PieceId: storj.NewPieceID(),
|
||||
Action: pb.PieceAction_PUT,
|
||||
Limit: 1000,
|
||||
PieceExpiration: time.Time{},
|
||||
OrderCreation: tt.orderCreation,
|
||||
OrderExpiration: now.Add(24 * time.Hour),
|
||||
}
|
||||
orderLimit1, err := signing.SignOrderLimit(ctx, signing.SignerFromFullIdentity(satellite.Identity), limit1)
|
||||
require.NoError(t, err)
|
||||
|
||||
order1, err := signing.SignUplinkOrder(ctx, piecePrivateKey, &pb.Order{
|
||||
SerialNumber: serialNumber1,
|
||||
Amount: tt.dataAmount,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
order1, err := signing.SignUplinkOrder(ctx, piecePrivateKey, &pb.Order{
|
||||
SerialNumber: serialNumber1,
|
||||
Amount: tt.dataAmount,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
limit2 := &pb.OrderLimit{
|
||||
SerialNumber: serialNumber2,
|
||||
SatelliteId: satellite.ID(),
|
||||
UplinkPublicKey: piecePublicKey,
|
||||
StorageNodeId: storagenode.ID(),
|
||||
PieceId: storj.NewPieceID(),
|
||||
Action: pb.PieceAction_PUT,
|
||||
Limit: 1000,
|
||||
PieceExpiration: time.Time{},
|
||||
OrderCreation: now,
|
||||
OrderExpiration: now.Add(24 * time.Hour),
|
||||
}
|
||||
orderLimit2, err := signing.SignOrderLimit(ctx, signing.SignerFromFullIdentity(satellite.Identity), limit2)
|
||||
require.NoError(t, err)
|
||||
limit2 := &pb.OrderLimit{
|
||||
SerialNumber: serialNumber2,
|
||||
SatelliteId: satellite.ID(),
|
||||
UplinkPublicKey: piecePublicKey,
|
||||
StorageNodeId: storagenode.ID(),
|
||||
PieceId: storj.NewPieceID(),
|
||||
Action: pb.PieceAction_PUT,
|
||||
Limit: 1000,
|
||||
PieceExpiration: time.Time{},
|
||||
OrderCreation: now,
|
||||
OrderExpiration: now.Add(24 * time.Hour),
|
||||
}
|
||||
orderLimit2, err := signing.SignOrderLimit(ctx, signing.SignerFromFullIdentity(satellite.Identity), limit2)
|
||||
require.NoError(t, err)
|
||||
|
||||
order2, err := signing.SignUplinkOrder(ctx, piecePrivateKey, &pb.Order{
|
||||
SerialNumber: serialNumber2,
|
||||
Amount: tt.dataAmount,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
order2, err := signing.SignUplinkOrder(ctx, piecePrivateKey, &pb.Order{
|
||||
SerialNumber: serialNumber2,
|
||||
Amount: tt.dataAmount,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
// create connection between storagenode and satellite
|
||||
conn, err := storagenode.Dialer.DialNodeURL(ctx, storj.NodeURL{ID: satellite.ID(), Address: satellite.Addr()})
|
||||
require.NoError(t, err)
|
||||
defer ctx.Check(conn.Close)
|
||||
// create connection between storagenode and satellite
|
||||
conn, err := storagenode.Dialer.DialNodeURL(ctx, storj.NodeURL{ID: satellite.ID(), Address: satellite.Addr()})
|
||||
require.NoError(t, err)
|
||||
defer ctx.Check(conn.Close)
|
||||
|
||||
stream, err := pb.NewDRPCOrdersClient(conn).SettlementWithWindow(ctx)
|
||||
require.NoError(t, err)
|
||||
defer ctx.Check(stream.Close)
|
||||
stream, err := pb.NewDRPCOrdersClient(conn).SettlementWithWindow(ctx)
|
||||
require.NoError(t, err)
|
||||
defer ctx.Check(stream.Close)
|
||||
|
||||
// storagenode settles an order and orderlimit
|
||||
err = stream.Send(&pb.SettlementRequest{
|
||||
Limit: orderLimit1,
|
||||
Order: order1,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
err = stream.Send(&pb.SettlementRequest{
|
||||
Limit: orderLimit2,
|
||||
Order: order2,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
resp, err := stream.CloseAndRecv()
|
||||
require.NoError(t, err)
|
||||
// storagenode settles an order and orderlimit
|
||||
err = stream.Send(&pb.SettlementRequest{
|
||||
Limit: orderLimit1,
|
||||
Order: order1,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
err = stream.Send(&pb.SettlementRequest{
|
||||
Limit: orderLimit2,
|
||||
Order: order2,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
resp, err := stream.CloseAndRecv()
|
||||
require.NoError(t, err)
|
||||
|
||||
// the settled amount is only returned during phase3
|
||||
var settled map[int32]int64
|
||||
if satellite.Config.Orders.WindowEndpointRolloutPhase == orders.WindowEndpointRolloutPhase3 {
|
||||
settled = map[int32]int64{int32(pb.PieceAction_PUT): tt.settledAmt}
|
||||
}
|
||||
require.Equal(t, &pb.SettlementWithWindowResponse{
|
||||
Status: pb.SettlementWithWindowResponse_ACCEPTED,
|
||||
ActionSettled: settled,
|
||||
}, resp)
|
||||
// the settled amount is only returned during phase3
|
||||
var settled map[int32]int64
|
||||
if satellite.Config.Orders.WindowEndpointRolloutPhase == orders.WindowEndpointRolloutPhase3 {
|
||||
settled = map[int32]int64{int32(pb.PieceAction_PUT): tt.settledAmt}
|
||||
}
|
||||
require.Equal(t, &pb.SettlementWithWindowResponse{
|
||||
Status: pb.SettlementWithWindowResponse_ACCEPTED,
|
||||
ActionSettled: settled,
|
||||
}, resp)
|
||||
|
||||
// trigger and wait for all of the chores necessary to flush the orders
|
||||
assert.NoError(t, satellite.Accounting.ReportedRollup.RunOnce(ctx, tt.orderCreation))
|
||||
satellite.Orders.Chore.Loop.TriggerWait()
|
||||
// trigger and wait for all of the chores necessary to flush the orders
|
||||
assert.NoError(t, satellite.Accounting.ReportedRollup.RunOnce(ctx, tt.orderCreation))
|
||||
satellite.Orders.Chore.Loop.TriggerWait()
|
||||
|
||||
// assert all the right stuff is in the satellite storagenode and bucket bandwidth tables
|
||||
snbw, err = ordersDB.GetStorageNodeBandwidth(ctx, storagenode.ID(), time.Time{}, tt.orderCreation)
|
||||
require.NoError(t, err)
|
||||
require.EqualValues(t, tt.settledAmt, snbw)
|
||||
// assert all the right stuff is in the satellite storagenode and bucket bandwidth tables
|
||||
snbw, err = ordersDB.GetStorageNodeBandwidth(ctx, storagenode.ID(), time.Time{}, tt.orderCreation)
|
||||
require.NoError(t, err)
|
||||
require.EqualValues(t, tt.settledAmt, snbw)
|
||||
|
||||
newBbw, err := ordersDB.GetBucketBandwidth(ctx, projectID, []byte(bucketname), time.Time{}, tt.orderCreation)
|
||||
require.NoError(t, err)
|
||||
require.EqualValues(t, tt.settledAmt, newBbw)
|
||||
newBbw, err := ordersDB.GetBucketBandwidth(ctx, projectID, []byte(bucketname), time.Time{}, tt.orderCreation)
|
||||
require.NoError(t, err)
|
||||
require.EqualValues(t, tt.settledAmt, newBbw)
|
||||
}()
|
||||
}
|
||||
})
|
||||
}
|
||||
@ -223,72 +225,74 @@ func TestSettlementWithWindowEndpointSingleOrder(t *testing.T) {
|
||||
}
|
||||
|
||||
for _, tt := range testCases {
|
||||
// create signed orderlimit or order to test with
|
||||
limit := &pb.OrderLimit{
|
||||
SerialNumber: serialNumber,
|
||||
SatelliteId: satellite.ID(),
|
||||
UplinkPublicKey: piecePublicKey,
|
||||
StorageNodeId: storagenode.ID(),
|
||||
PieceId: storj.NewPieceID(),
|
||||
Action: pb.PieceAction_PUT,
|
||||
Limit: 1000,
|
||||
PieceExpiration: time.Time{},
|
||||
OrderCreation: now,
|
||||
OrderExpiration: now.Add(24 * time.Hour),
|
||||
}
|
||||
orderLimit, err := signing.SignOrderLimit(ctx, signing.SignerFromFullIdentity(satellite.Identity), limit)
|
||||
require.NoError(t, err)
|
||||
func() {
|
||||
// create signed orderlimit or order to test with
|
||||
limit := &pb.OrderLimit{
|
||||
SerialNumber: serialNumber,
|
||||
SatelliteId: satellite.ID(),
|
||||
UplinkPublicKey: piecePublicKey,
|
||||
StorageNodeId: storagenode.ID(),
|
||||
PieceId: storj.NewPieceID(),
|
||||
Action: pb.PieceAction_PUT,
|
||||
Limit: 1000,
|
||||
PieceExpiration: time.Time{},
|
||||
OrderCreation: now,
|
||||
OrderExpiration: now.Add(24 * time.Hour),
|
||||
}
|
||||
orderLimit, err := signing.SignOrderLimit(ctx, signing.SignerFromFullIdentity(satellite.Identity), limit)
|
||||
require.NoError(t, err)
|
||||
|
||||
order, err := signing.SignUplinkOrder(ctx, piecePrivateKey, &pb.Order{
|
||||
SerialNumber: serialNumber,
|
||||
Amount: tt.dataAmount,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
order, err := signing.SignUplinkOrder(ctx, piecePrivateKey, &pb.Order{
|
||||
SerialNumber: serialNumber,
|
||||
Amount: tt.dataAmount,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
// create connection between storagenode and satellite
|
||||
conn, err := storagenode.Dialer.DialNodeURL(ctx, storj.NodeURL{ID: satellite.ID(), Address: satellite.Addr()})
|
||||
require.NoError(t, err)
|
||||
defer ctx.Check(conn.Close)
|
||||
// create connection between storagenode and satellite
|
||||
conn, err := storagenode.Dialer.DialNodeURL(ctx, storj.NodeURL{ID: satellite.ID(), Address: satellite.Addr()})
|
||||
require.NoError(t, err)
|
||||
defer ctx.Check(conn.Close)
|
||||
|
||||
stream, err := pb.NewDRPCOrdersClient(conn).SettlementWithWindow(ctx)
|
||||
require.NoError(t, err)
|
||||
defer ctx.Check(stream.Close)
|
||||
stream, err := pb.NewDRPCOrdersClient(conn).SettlementWithWindow(ctx)
|
||||
require.NoError(t, err)
|
||||
defer ctx.Check(stream.Close)
|
||||
|
||||
// storagenode settles an order and orderlimit
|
||||
err = stream.Send(&pb.SettlementRequest{
|
||||
Limit: orderLimit,
|
||||
Order: order,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
resp, err := stream.CloseAndRecv()
|
||||
require.NoError(t, err)
|
||||
// storagenode settles an order and orderlimit
|
||||
err = stream.Send(&pb.SettlementRequest{
|
||||
Limit: orderLimit,
|
||||
Order: order,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
resp, err := stream.CloseAndRecv()
|
||||
require.NoError(t, err)
|
||||
|
||||
expected := new(pb.SettlementWithWindowResponse)
|
||||
switch {
|
||||
case satellite.Config.Orders.WindowEndpointRolloutPhase != orders.WindowEndpointRolloutPhase3:
|
||||
expected.Status = pb.SettlementWithWindowResponse_ACCEPTED
|
||||
expected.ActionSettled = nil
|
||||
case tt.expectedStatus == pb.SettlementWithWindowResponse_ACCEPTED:
|
||||
expected.Status = pb.SettlementWithWindowResponse_ACCEPTED
|
||||
expected.ActionSettled = map[int32]int64{int32(pb.PieceAction_PUT): tt.dataAmount}
|
||||
default:
|
||||
expected.Status = pb.SettlementWithWindowResponse_REJECTED
|
||||
expected.ActionSettled = nil
|
||||
}
|
||||
require.Equal(t, expected, resp)
|
||||
expected := new(pb.SettlementWithWindowResponse)
|
||||
switch {
|
||||
case satellite.Config.Orders.WindowEndpointRolloutPhase != orders.WindowEndpointRolloutPhase3:
|
||||
expected.Status = pb.SettlementWithWindowResponse_ACCEPTED
|
||||
expected.ActionSettled = nil
|
||||
case tt.expectedStatus == pb.SettlementWithWindowResponse_ACCEPTED:
|
||||
expected.Status = pb.SettlementWithWindowResponse_ACCEPTED
|
||||
expected.ActionSettled = map[int32]int64{int32(pb.PieceAction_PUT): tt.dataAmount}
|
||||
default:
|
||||
expected.Status = pb.SettlementWithWindowResponse_REJECTED
|
||||
expected.ActionSettled = nil
|
||||
}
|
||||
require.Equal(t, expected, resp)
|
||||
|
||||
// flush all the chores
|
||||
assert.NoError(t, satellite.Accounting.ReportedRollup.RunOnce(ctx, now))
|
||||
satellite.Orders.Chore.Loop.TriggerWait()
|
||||
// flush all the chores
|
||||
assert.NoError(t, satellite.Accounting.ReportedRollup.RunOnce(ctx, now))
|
||||
satellite.Orders.Chore.Loop.TriggerWait()
|
||||
|
||||
// assert all the right stuff is in the satellite storagenode and bucket bandwidth tables
|
||||
snbw, err = ordersDB.GetStorageNodeBandwidth(ctx, storagenode.ID(), time.Time{}, now)
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, dataAmount, snbw)
|
||||
// assert all the right stuff is in the satellite storagenode and bucket bandwidth tables
|
||||
snbw, err = ordersDB.GetStorageNodeBandwidth(ctx, storagenode.ID(), time.Time{}, now)
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, dataAmount, snbw)
|
||||
|
||||
newBbw, err := ordersDB.GetBucketBandwidth(ctx, projectID, []byte(bucketname), time.Time{}, now)
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, dataAmount, newBbw)
|
||||
newBbw, err := ordersDB.GetBucketBandwidth(ctx, projectID, []byte(bucketname), time.Time{}, now)
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, dataAmount, newBbw)
|
||||
}()
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
@ -566,7 +566,11 @@ func (service *Service) CreateGracefulExitPutOrderLimit(ctx context.Context, buc
|
||||
return nil, storj.PiecePrivateKey{}, Error.Wrap(err)
|
||||
}
|
||||
|
||||
nodeURL := storj.NodeURL{ID: nodeID, Address: node.Address.Address}
|
||||
address := node.Address.Address
|
||||
if node.LastIPPort != "" {
|
||||
address = node.LastIPPort
|
||||
}
|
||||
nodeURL := storj.NodeURL{ID: nodeID, Address: address}
|
||||
limit, err = signer.Sign(ctx, nodeURL, pieceNum)
|
||||
if err != nil {
|
||||
return nil, storj.PiecePrivateKey{}, Error.Wrap(err)
|
||||
|
||||
@ -89,35 +89,106 @@ func BenchmarkOverlay(b *testing.B) {
|
||||
}
|
||||
})
|
||||
|
||||
b.Run("UpdateStats", func(b *testing.B) {
|
||||
b.Run("UpdateStatsSuccess", func(b *testing.B) {
|
||||
for i := 0; i < b.N; i++ {
|
||||
id := all[i%len(all)]
|
||||
outcome := overlay.AuditFailure
|
||||
if i&1 == 0 {
|
||||
outcome = overlay.AuditSuccess
|
||||
}
|
||||
_, err := overlaydb.UpdateStats(ctx, &overlay.UpdateRequest{
|
||||
NodeID: id,
|
||||
AuditOutcome: outcome,
|
||||
IsUp: i&2 == 0,
|
||||
AuditOutcome: overlay.AuditSuccess,
|
||||
AuditHistory: testAuditHistoryConfig(),
|
||||
}, time.Now())
|
||||
require.NoError(b, err)
|
||||
}
|
||||
})
|
||||
|
||||
b.Run("BatchUpdateStats", func(b *testing.B) {
|
||||
b.Run("UpdateStatsFailure", func(b *testing.B) {
|
||||
for i := 0; i < b.N; i++ {
|
||||
id := all[i%len(all)]
|
||||
_, err := overlaydb.UpdateStats(ctx, &overlay.UpdateRequest{
|
||||
NodeID: id,
|
||||
AuditOutcome: overlay.AuditFailure,
|
||||
AuditHistory: testAuditHistoryConfig(),
|
||||
}, time.Now())
|
||||
require.NoError(b, err)
|
||||
}
|
||||
})
|
||||
|
||||
b.Run("UpdateStatsUnknown", func(b *testing.B) {
|
||||
for i := 0; i < b.N; i++ {
|
||||
id := all[i%len(all)]
|
||||
_, err := overlaydb.UpdateStats(ctx, &overlay.UpdateRequest{
|
||||
NodeID: id,
|
||||
AuditOutcome: overlay.AuditUnknown,
|
||||
AuditHistory: testAuditHistoryConfig(),
|
||||
}, time.Now())
|
||||
require.NoError(b, err)
|
||||
}
|
||||
})
|
||||
|
||||
b.Run("UpdateStatsOffline", func(b *testing.B) {
|
||||
for i := 0; i < b.N; i++ {
|
||||
id := all[i%len(all)]
|
||||
_, err := overlaydb.UpdateStats(ctx, &overlay.UpdateRequest{
|
||||
NodeID: id,
|
||||
AuditOutcome: overlay.AuditOffline,
|
||||
AuditHistory: testAuditHistoryConfig(),
|
||||
}, time.Now())
|
||||
require.NoError(b, err)
|
||||
}
|
||||
})
|
||||
|
||||
b.Run("BatchUpdateStatsSuccess", func(b *testing.B) {
|
||||
var updateRequests []*overlay.UpdateRequest
|
||||
for i := 0; i < b.N; i++ {
|
||||
id := all[i%len(all)]
|
||||
outcome := overlay.AuditFailure
|
||||
if i&1 == 0 {
|
||||
outcome = overlay.AuditSuccess
|
||||
}
|
||||
updateRequests = append(updateRequests, &overlay.UpdateRequest{
|
||||
NodeID: id,
|
||||
AuditOutcome: outcome,
|
||||
IsUp: i&2 == 0,
|
||||
AuditOutcome: overlay.AuditSuccess,
|
||||
AuditHistory: testAuditHistoryConfig(),
|
||||
})
|
||||
|
||||
}
|
||||
_, err := overlaydb.BatchUpdateStats(ctx, updateRequests, 100, time.Now())
|
||||
require.NoError(b, err)
|
||||
})
|
||||
|
||||
b.Run("BatchUpdateStatsFailure", func(b *testing.B) {
|
||||
var updateRequests []*overlay.UpdateRequest
|
||||
for i := 0; i < b.N; i++ {
|
||||
id := all[i%len(all)]
|
||||
updateRequests = append(updateRequests, &overlay.UpdateRequest{
|
||||
NodeID: id,
|
||||
AuditOutcome: overlay.AuditFailure,
|
||||
AuditHistory: testAuditHistoryConfig(),
|
||||
})
|
||||
|
||||
}
|
||||
_, err := overlaydb.BatchUpdateStats(ctx, updateRequests, 100, time.Now())
|
||||
require.NoError(b, err)
|
||||
})
|
||||
|
||||
b.Run("BatchUpdateStatsUnknown", func(b *testing.B) {
|
||||
var updateRequests []*overlay.UpdateRequest
|
||||
for i := 0; i < b.N; i++ {
|
||||
id := all[i%len(all)]
|
||||
updateRequests = append(updateRequests, &overlay.UpdateRequest{
|
||||
NodeID: id,
|
||||
AuditOutcome: overlay.AuditUnknown,
|
||||
AuditHistory: testAuditHistoryConfig(),
|
||||
})
|
||||
|
||||
}
|
||||
_, err := overlaydb.BatchUpdateStats(ctx, updateRequests, 100, time.Now())
|
||||
require.NoError(b, err)
|
||||
})
|
||||
|
||||
b.Run("BatchUpdateStatsOffline", func(b *testing.B) {
|
||||
var updateRequests []*overlay.UpdateRequest
|
||||
for i := 0; i < b.N; i++ {
|
||||
id := all[i%len(all)]
|
||||
updateRequests = append(updateRequests, &overlay.UpdateRequest{
|
||||
NodeID: id,
|
||||
AuditOutcome: overlay.AuditOffline,
|
||||
AuditHistory: testAuditHistoryConfig(),
|
||||
})
|
||||
|
||||
@ -272,7 +343,6 @@ func BenchmarkNodeSelection(b *testing.B) {
|
||||
if i%2 == 0 { // make half of nodes "new" and half "vetted"
|
||||
_, err = overlaydb.UpdateStats(ctx, &overlay.UpdateRequest{
|
||||
NodeID: nodeID,
|
||||
IsUp: true,
|
||||
AuditOutcome: overlay.AuditSuccess,
|
||||
AuditLambda: 1,
|
||||
AuditWeight: 1,
|
||||
@ -293,7 +363,6 @@ func BenchmarkNodeSelection(b *testing.B) {
|
||||
case 2:
|
||||
err := overlaydb.UpdateCheckIn(ctx, overlay.NodeCheckInInfo{
|
||||
NodeID: nodeID,
|
||||
IsUp: true,
|
||||
Address: &pb.NodeAddress{
|
||||
Address: address,
|
||||
},
|
||||
|
||||
@ -104,7 +104,6 @@ func addNodesToNodesTable(ctx context.Context, t *testing.T, db overlay.DB, coun
|
||||
if i < makeReputable {
|
||||
stats, err := db.UpdateStats(ctx, &overlay.UpdateRequest{
|
||||
NodeID: storj.NodeID{byte(i)},
|
||||
IsUp: true,
|
||||
AuditOutcome: overlay.AuditSuccess,
|
||||
AuditLambda: 1, AuditWeight: 1, AuditDQ: 0.5,
|
||||
AuditHistory: testAuditHistoryConfig(),
|
||||
|
||||
@ -188,7 +188,6 @@ func TestEnsureMinimumRequested(t *testing.T) {
|
||||
reputable[node.ID()] = true
|
||||
_, err := satellite.DB.OverlayCache().UpdateStats(ctx, &overlay.UpdateRequest{
|
||||
NodeID: node.ID(),
|
||||
IsUp: true,
|
||||
AuditOutcome: overlay.AuditSuccess,
|
||||
AuditLambda: 1, AuditWeight: 1, AuditDQ: 0.5,
|
||||
AuditHistory: testAuditHistoryConfig(),
|
||||
@ -232,7 +231,6 @@ func TestEnsureMinimumRequested(t *testing.T) {
|
||||
reputable[node.ID()] = true
|
||||
_, err := satellite.DB.OverlayCache().UpdateStats(ctx, &overlay.UpdateRequest{
|
||||
NodeID: node.ID(),
|
||||
IsUp: true,
|
||||
AuditOutcome: overlay.AuditSuccess,
|
||||
AuditLambda: 1, AuditWeight: 1, AuditDQ: 0.5,
|
||||
AuditHistory: testAuditHistoryConfig(),
|
||||
@ -397,7 +395,6 @@ func TestNodeSelectionGracefulExit(t *testing.T) {
|
||||
for k := 0; k < i; k++ {
|
||||
_, err := satellite.DB.OverlayCache().UpdateStats(ctx, &overlay.UpdateRequest{
|
||||
NodeID: node.ID(),
|
||||
IsUp: true,
|
||||
AuditOutcome: overlay.AuditSuccess,
|
||||
AuditLambda: 1, AuditWeight: 1, AuditDQ: 0.5,
|
||||
AuditHistory: testAuditHistoryConfig(),
|
||||
@ -628,7 +625,6 @@ func TestDistinctIPs(t *testing.T) {
|
||||
for i := 9; i > 7; i-- {
|
||||
_, err := satellite.DB.OverlayCache().UpdateStats(ctx, &overlay.UpdateRequest{
|
||||
NodeID: planet.StorageNodes[i].ID(),
|
||||
IsUp: true,
|
||||
AuditOutcome: overlay.AuditSuccess,
|
||||
AuditLambda: 1,
|
||||
AuditWeight: 1,
|
||||
@ -660,7 +656,6 @@ func TestDistinctIPsWithBatch(t *testing.T) {
|
||||
// These are done individually b/c the previous stat data is important
|
||||
_, err := satellite.Overlay.Service.BatchUpdateStats(ctx, []*overlay.UpdateRequest{{
|
||||
NodeID: planet.StorageNodes[i].ID(),
|
||||
IsUp: true,
|
||||
AuditOutcome: overlay.AuditSuccess,
|
||||
AuditLambda: 1,
|
||||
AuditWeight: 1,
|
||||
|
||||
@ -164,7 +164,6 @@ const (
|
||||
type UpdateRequest struct {
|
||||
NodeID storj.NodeID
|
||||
AuditOutcome AuditType
|
||||
IsUp bool
|
||||
// n.b. these are set values from the satellite.
|
||||
// They are part of the UpdateRequest struct in order to be
|
||||
// more easily accessible in satellite/satellitedb/overlaycache.go.
|
||||
|
||||
@ -131,7 +131,6 @@ func testCache(ctx context.Context, t *testing.T, store overlay.DB) {
|
||||
|
||||
stats, err := service.UpdateStats(ctx, &overlay.UpdateRequest{
|
||||
NodeID: valid1ID,
|
||||
IsUp: true,
|
||||
AuditOutcome: overlay.AuditFailure,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
@ -148,7 +147,6 @@ func testCache(ctx context.Context, t *testing.T, store overlay.DB) {
|
||||
// should not update once already disqualified
|
||||
_, err = service.BatchUpdateStats(ctx, []*overlay.UpdateRequest{{
|
||||
NodeID: valid2ID,
|
||||
IsUp: false,
|
||||
AuditOutcome: overlay.AuditSuccess,
|
||||
}})
|
||||
require.NoError(t, err)
|
||||
@ -197,7 +195,6 @@ func TestRandomizedSelection(t *testing.T) {
|
||||
if i%2 == 0 { // make half of nodes "new" and half "vetted"
|
||||
_, err = cache.UpdateStats(ctx, &overlay.UpdateRequest{
|
||||
NodeID: newID,
|
||||
IsUp: true,
|
||||
AuditOutcome: overlay.AuditSuccess,
|
||||
AuditLambda: 1,
|
||||
AuditWeight: 1,
|
||||
@ -316,7 +313,6 @@ func TestRandomizedSelectionCache(t *testing.T) {
|
||||
if i%2 == 0 { // make half of nodes "new" and half "vetted"
|
||||
_, err = overlaydb.UpdateStats(ctx, &overlay.UpdateRequest{
|
||||
NodeID: newID,
|
||||
IsUp: true,
|
||||
AuditOutcome: overlay.AuditSuccess,
|
||||
AuditLambda: 1,
|
||||
AuditWeight: 1,
|
||||
@ -775,7 +771,6 @@ func TestSuspendedSelection(t *testing.T) {
|
||||
if i%2 == 0 { // make half of nodes "new" and half "vetted"
|
||||
_, err = cache.UpdateStats(ctx, &overlay.UpdateRequest{
|
||||
NodeID: newID,
|
||||
IsUp: true,
|
||||
AuditOutcome: overlay.AuditSuccess,
|
||||
AuditLambda: 1,
|
||||
AuditWeight: 1,
|
||||
@ -835,7 +830,6 @@ func TestConcurrentAudit(t *testing.T) {
|
||||
_, err := planet.Satellites[0].Overlay.Service.UpdateStats(ctx, &overlay.UpdateRequest{
|
||||
NodeID: planet.StorageNodes[0].ID(),
|
||||
AuditOutcome: overlay.AuditSuccess,
|
||||
IsUp: true,
|
||||
})
|
||||
return err
|
||||
})
|
||||
@ -853,7 +847,6 @@ func TestConcurrentAudit(t *testing.T) {
|
||||
{
|
||||
NodeID: planet.StorageNodes[0].ID(),
|
||||
AuditOutcome: overlay.AuditSuccess,
|
||||
IsUp: true,
|
||||
},
|
||||
})
|
||||
return err
|
||||
|
||||
@ -169,7 +169,6 @@ func testDatabase(ctx context.Context, t *testing.T, cache overlay.DB) {
|
||||
updateReq := &overlay.UpdateRequest{
|
||||
NodeID: nodeID,
|
||||
AuditOutcome: overlay.AuditSuccess,
|
||||
IsUp: true,
|
||||
AuditLambda: 0.123, AuditWeight: 0.456,
|
||||
AuditDQ: 0, // don't disqualify for any reason
|
||||
AuditHistory: testAuditHistoryConfig(),
|
||||
@ -186,7 +185,6 @@ func testDatabase(ctx context.Context, t *testing.T, cache overlay.DB) {
|
||||
auditBeta = expectedAuditBeta
|
||||
|
||||
updateReq.AuditOutcome = overlay.AuditFailure
|
||||
updateReq.IsUp = false
|
||||
stats, err = cache.UpdateStats(ctx, updateReq, time.Now())
|
||||
require.NoError(t, err)
|
||||
|
||||
|
||||
@ -66,7 +66,6 @@ func TestAuditSuspendWithUpdateStats(t *testing.T) {
|
||||
_, err = oc.UpdateStats(ctx, &overlay.UpdateRequest{
|
||||
NodeID: nodeID,
|
||||
AuditOutcome: overlay.AuditUnknown,
|
||||
IsUp: true,
|
||||
AuditLambda: 1,
|
||||
AuditWeight: 1,
|
||||
AuditDQ: 0.6,
|
||||
@ -90,7 +89,6 @@ func TestAuditSuspendWithUpdateStats(t *testing.T) {
|
||||
_, err = oc.UpdateStats(ctx, &overlay.UpdateRequest{
|
||||
NodeID: nodeID,
|
||||
AuditOutcome: overlay.AuditSuccess,
|
||||
IsUp: true,
|
||||
AuditLambda: 1,
|
||||
AuditWeight: 1,
|
||||
AuditDQ: 0.6,
|
||||
@ -121,7 +119,6 @@ func TestAuditSuspendFailedAudit(t *testing.T) {
|
||||
_, err = oc.UpdateStats(ctx, &overlay.UpdateRequest{
|
||||
NodeID: nodeID,
|
||||
AuditOutcome: overlay.AuditFailure,
|
||||
IsUp: true,
|
||||
AuditLambda: 1,
|
||||
AuditWeight: 1,
|
||||
AuditDQ: 0.6,
|
||||
@ -277,7 +274,6 @@ func TestAuditSuspendBatchUpdateStats(t *testing.T) {
|
||||
nodeUpdateReq := &overlay.UpdateRequest{
|
||||
NodeID: nodeID,
|
||||
AuditOutcome: overlay.AuditSuccess,
|
||||
IsUp: true,
|
||||
AuditLambda: 1,
|
||||
AuditWeight: 1,
|
||||
AuditDQ: 0.6,
|
||||
@ -343,7 +339,6 @@ func TestOfflineSuspend(t *testing.T) {
|
||||
updateReq := &overlay.UpdateRequest{
|
||||
NodeID: nodeID,
|
||||
AuditOutcome: overlay.AuditOffline,
|
||||
IsUp: false,
|
||||
AuditHistory: overlay.AuditHistoryConfig{
|
||||
WindowSize: time.Hour,
|
||||
TrackingPeriod: 2 * time.Hour,
|
||||
@ -472,12 +467,11 @@ func setOnlineScore(ctx context.Context, reqPtr *overlay.UpdateRequest, desiredS
|
||||
for window := 0; window < windowsPerTrackingPeriod+1; window++ {
|
||||
updateReqs := []*overlay.UpdateRequest{}
|
||||
for i := 0; i < totalAudits; i++ {
|
||||
isUp := true
|
||||
if i >= onlineAudits {
|
||||
isUp = false
|
||||
}
|
||||
updateReq := *reqPtr
|
||||
updateReq.IsUp = isUp
|
||||
updateReq.AuditOutcome = overlay.AuditSuccess
|
||||
if i >= onlineAudits {
|
||||
updateReq.AuditOutcome = overlay.AuditOffline
|
||||
}
|
||||
updateReq.AuditHistory.GracePeriod = gracePeriod
|
||||
|
||||
updateReqs = append(updateReqs, &updateReq)
|
||||
|
||||
@ -391,7 +391,7 @@ func (obs *checkerObserver) InlineSegment(ctx context.Context, segment *metainfo
|
||||
func (checker *Checker) IrreparableProcess(ctx context.Context) (err error) {
|
||||
defer mon.Task()(&ctx)(&err)
|
||||
const limit = 1000
|
||||
var lastSeenSegmentKey metabase.SegmentKey
|
||||
lastSeenSegmentKey := metabase.SegmentKey{}
|
||||
|
||||
for {
|
||||
segments, err := checker.irrdb.GetLimited(ctx, limit, lastSeenSegmentKey)
|
||||
|
||||
@ -13,8 +13,10 @@ import (
|
||||
|
||||
"storj.io/common/pb"
|
||||
"storj.io/common/testcontext"
|
||||
"storj.io/storj/private/testplanet"
|
||||
"storj.io/storj/satellite"
|
||||
"storj.io/storj/satellite/internalpb"
|
||||
"storj.io/storj/satellite/metainfo/metabase"
|
||||
"storj.io/storj/satellite/satellitedb/satellitedbtest"
|
||||
)
|
||||
|
||||
@ -103,3 +105,88 @@ func TestIrreparable(t *testing.T) {
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func TestIrreparableProcess(t *testing.T) {
|
||||
testplanet.Run(t, testplanet.Config{
|
||||
SatelliteCount: 1, StorageNodeCount: 3, UplinkCount: 0,
|
||||
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
|
||||
checker := planet.Satellites[0].Repair.Checker
|
||||
checker.Loop.Stop()
|
||||
checker.IrreparableLoop.Stop()
|
||||
irreparabledb := planet.Satellites[0].DB.Irreparable()
|
||||
queue := planet.Satellites[0].DB.RepairQueue()
|
||||
|
||||
seg := &internalpb.IrreparableSegment{
|
||||
Path: []byte{1},
|
||||
SegmentDetail: &pb.Pointer{
|
||||
Type: pb.Pointer_REMOTE,
|
||||
CreationDate: time.Now(),
|
||||
Remote: &pb.RemoteSegment{
|
||||
Redundancy: &pb.RedundancyScheme{
|
||||
MinReq: 1,
|
||||
RepairThreshold: 2,
|
||||
SuccessThreshold: 3,
|
||||
Total: 4,
|
||||
},
|
||||
RemotePieces: []*pb.RemotePiece{
|
||||
{
|
||||
NodeId: planet.StorageNodes[0].ID(),
|
||||
},
|
||||
{
|
||||
NodeId: planet.StorageNodes[1].ID(),
|
||||
},
|
||||
{
|
||||
NodeId: planet.StorageNodes[2].ID(),
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
LostPieces: int32(4),
|
||||
LastRepairAttempt: time.Now().Unix(),
|
||||
RepairAttemptCount: int64(10),
|
||||
}
|
||||
|
||||
require.NoError(t, irreparabledb.IncrementRepairAttempts(ctx, seg))
|
||||
|
||||
result, err := irreparabledb.Get(ctx, metabase.SegmentKey(seg.GetPath()))
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, result)
|
||||
|
||||
// test healthy segment is removed from irreparable DB
|
||||
require.NoError(t, checker.IrreparableProcess(ctx))
|
||||
|
||||
result, err = irreparabledb.Get(ctx, metabase.SegmentKey(seg.GetPath()))
|
||||
require.Error(t, err)
|
||||
require.Nil(t, result)
|
||||
|
||||
// test unhealthy repairable segment is removed from irreparable DB and inserted into repair queue
|
||||
seg.SegmentDetail.Remote.RemotePieces[0] = &pb.RemotePiece{}
|
||||
seg.SegmentDetail.Remote.RemotePieces[1] = &pb.RemotePiece{}
|
||||
|
||||
require.NoError(t, irreparabledb.IncrementRepairAttempts(ctx, seg))
|
||||
require.NoError(t, checker.IrreparableProcess(ctx))
|
||||
|
||||
result, err = irreparabledb.Get(ctx, metabase.SegmentKey(seg.GetPath()))
|
||||
require.Error(t, err)
|
||||
require.Nil(t, result)
|
||||
|
||||
injured, err := queue.Select(ctx)
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, seg.GetPath(), injured.GetPath())
|
||||
|
||||
n, err := queue.Clean(ctx, time.Now())
|
||||
require.NoError(t, err)
|
||||
require.EqualValues(t, 1, n)
|
||||
|
||||
// test irreparable segment remains in irreparable DB and repair_attempt_count is incremented
|
||||
seg.SegmentDetail.Remote.RemotePieces[2] = &pb.RemotePiece{}
|
||||
|
||||
require.NoError(t, irreparabledb.IncrementRepairAttempts(ctx, seg))
|
||||
require.NoError(t, checker.IrreparableProcess(ctx))
|
||||
|
||||
result, err = irreparabledb.Get(ctx, metabase.SegmentKey(seg.GetPath()))
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, seg.GetPath(), result.Path)
|
||||
require.Equal(t, seg.RepairAttemptCount+1, result.RepairAttemptCount)
|
||||
})
|
||||
}
|
||||
|
||||
@ -53,15 +53,13 @@ func testDataRepair(t *testing.T, inMemoryRepair bool) {
|
||||
StorageNodeCount: 14,
|
||||
UplinkCount: 1,
|
||||
Reconfigure: testplanet.Reconfigure{
|
||||
Satellite: func(log *zap.Logger, index int, config *satellite.Config) {
|
||||
config.Repairer.MaxExcessRateOptimalThreshold = RepairMaxExcessRateOptimalThreshold
|
||||
config.Repairer.InMemoryRepair = inMemoryRepair
|
||||
|
||||
config.Metainfo.RS.MinThreshold = minThreshold
|
||||
config.Metainfo.RS.RepairThreshold = 5
|
||||
config.Metainfo.RS.SuccessThreshold = successThreshold
|
||||
config.Metainfo.RS.TotalThreshold = 9
|
||||
},
|
||||
Satellite: testplanet.Combine(
|
||||
func(log *zap.Logger, index int, config *satellite.Config) {
|
||||
config.Repairer.MaxExcessRateOptimalThreshold = RepairMaxExcessRateOptimalThreshold
|
||||
config.Repairer.InMemoryRepair = inMemoryRepair
|
||||
},
|
||||
testplanet.ReconfigureRS(minThreshold, 5, successThreshold, 9),
|
||||
),
|
||||
},
|
||||
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
|
||||
|
||||
@ -188,15 +186,13 @@ func testCorruptDataRepairFailed(t *testing.T, inMemoryRepair bool) {
|
||||
StorageNodeCount: 14,
|
||||
UplinkCount: 1,
|
||||
Reconfigure: testplanet.Reconfigure{
|
||||
Satellite: func(log *zap.Logger, index int, config *satellite.Config) {
|
||||
config.Repairer.MaxExcessRateOptimalThreshold = RepairMaxExcessRateOptimalThreshold
|
||||
config.Repairer.InMemoryRepair = inMemoryRepair
|
||||
|
||||
config.Metainfo.RS.MinThreshold = 3
|
||||
config.Metainfo.RS.RepairThreshold = 5
|
||||
config.Metainfo.RS.SuccessThreshold = 7
|
||||
config.Metainfo.RS.TotalThreshold = 9
|
||||
},
|
||||
Satellite: testplanet.Combine(
|
||||
func(log *zap.Logger, index int, config *satellite.Config) {
|
||||
config.Repairer.MaxExcessRateOptimalThreshold = RepairMaxExcessRateOptimalThreshold
|
||||
config.Repairer.InMemoryRepair = inMemoryRepair
|
||||
},
|
||||
testplanet.ReconfigureRS(3, 5, 7, 9),
|
||||
),
|
||||
},
|
||||
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
|
||||
uplinkPeer := planet.Uplinks[0]
|
||||
@ -305,15 +301,13 @@ func testCorruptDataRepairSucceed(t *testing.T, inMemoryRepair bool) {
|
||||
StorageNodeCount: 14,
|
||||
UplinkCount: 1,
|
||||
Reconfigure: testplanet.Reconfigure{
|
||||
Satellite: func(log *zap.Logger, index int, config *satellite.Config) {
|
||||
config.Repairer.MaxExcessRateOptimalThreshold = RepairMaxExcessRateOptimalThreshold
|
||||
config.Repairer.InMemoryRepair = inMemoryRepair
|
||||
|
||||
config.Metainfo.RS.MinThreshold = 3
|
||||
config.Metainfo.RS.RepairThreshold = 5
|
||||
config.Metainfo.RS.SuccessThreshold = 7
|
||||
config.Metainfo.RS.TotalThreshold = 9
|
||||
},
|
||||
Satellite: testplanet.Combine(
|
||||
func(log *zap.Logger, index int, config *satellite.Config) {
|
||||
config.Repairer.MaxExcessRateOptimalThreshold = RepairMaxExcessRateOptimalThreshold
|
||||
config.Repairer.InMemoryRepair = inMemoryRepair
|
||||
},
|
||||
testplanet.ReconfigureRS(3, 5, 7, 9),
|
||||
),
|
||||
},
|
||||
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
|
||||
uplinkPeer := planet.Uplinks[0]
|
||||
@ -799,14 +793,12 @@ func testRepairMultipleDisqualifiedAndSuspended(t *testing.T, inMemoryRepair boo
|
||||
StorageNodeCount: 12,
|
||||
UplinkCount: 1,
|
||||
Reconfigure: testplanet.Reconfigure{
|
||||
Satellite: func(log *zap.Logger, index int, config *satellite.Config) {
|
||||
config.Repairer.InMemoryRepair = inMemoryRepair
|
||||
|
||||
config.Metainfo.RS.MinThreshold = 3
|
||||
config.Metainfo.RS.RepairThreshold = 5
|
||||
config.Metainfo.RS.SuccessThreshold = 7
|
||||
config.Metainfo.RS.TotalThreshold = 7
|
||||
},
|
||||
Satellite: testplanet.Combine(
|
||||
func(log *zap.Logger, index int, config *satellite.Config) {
|
||||
config.Repairer.InMemoryRepair = inMemoryRepair
|
||||
},
|
||||
testplanet.ReconfigureRS(3, 5, 7, 7),
|
||||
),
|
||||
},
|
||||
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
|
||||
// first, upload some remote data
|
||||
@ -920,15 +912,13 @@ func testDataRepairOverrideHigherLimit(t *testing.T, inMemoryRepair bool) {
|
||||
StorageNodeCount: 14,
|
||||
UplinkCount: 1,
|
||||
Reconfigure: testplanet.Reconfigure{
|
||||
Satellite: func(log *zap.Logger, index int, config *satellite.Config) {
|
||||
config.Checker.RepairOverride = repairOverride
|
||||
config.Repairer.InMemoryRepair = inMemoryRepair
|
||||
|
||||
config.Metainfo.RS.MinThreshold = 3
|
||||
config.Metainfo.RS.RepairThreshold = 4
|
||||
config.Metainfo.RS.SuccessThreshold = 9
|
||||
config.Metainfo.RS.TotalThreshold = 9
|
||||
},
|
||||
Satellite: testplanet.Combine(
|
||||
func(log *zap.Logger, index int, config *satellite.Config) {
|
||||
config.Repairer.InMemoryRepair = inMemoryRepair
|
||||
config.Checker.RepairOverride = repairOverride
|
||||
},
|
||||
testplanet.ReconfigureRS(3, 4, 9, 9),
|
||||
),
|
||||
},
|
||||
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
|
||||
uplinkPeer := planet.Uplinks[0]
|
||||
@ -1014,15 +1004,13 @@ func testDataRepairOverrideLowerLimit(t *testing.T, inMemoryRepair bool) {
|
||||
StorageNodeCount: 14,
|
||||
UplinkCount: 1,
|
||||
Reconfigure: testplanet.Reconfigure{
|
||||
Satellite: func(log *zap.Logger, index int, config *satellite.Config) {
|
||||
config.Checker.RepairOverride = repairOverride
|
||||
config.Repairer.InMemoryRepair = inMemoryRepair
|
||||
|
||||
config.Metainfo.RS.MinThreshold = 3
|
||||
config.Metainfo.RS.RepairThreshold = 6
|
||||
config.Metainfo.RS.SuccessThreshold = 9
|
||||
config.Metainfo.RS.TotalThreshold = 9
|
||||
},
|
||||
Satellite: testplanet.Combine(
|
||||
func(log *zap.Logger, index int, config *satellite.Config) {
|
||||
config.Repairer.InMemoryRepair = inMemoryRepair
|
||||
config.Checker.RepairOverride = repairOverride
|
||||
},
|
||||
testplanet.ReconfigureRS(3, 6, 9, 9),
|
||||
),
|
||||
},
|
||||
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
|
||||
uplinkPeer := planet.Uplinks[0]
|
||||
@ -1141,15 +1129,13 @@ func testDataRepairUploadLimit(t *testing.T, inMemoryRepair bool) {
|
||||
StorageNodeCount: 13,
|
||||
UplinkCount: 1,
|
||||
Reconfigure: testplanet.Reconfigure{
|
||||
Satellite: func(log *zap.Logger, index int, config *satellite.Config) {
|
||||
config.Repairer.MaxExcessRateOptimalThreshold = RepairMaxExcessRateOptimalThreshold
|
||||
config.Repairer.InMemoryRepair = inMemoryRepair
|
||||
|
||||
config.Metainfo.RS.MinThreshold = 3
|
||||
config.Metainfo.RS.RepairThreshold = repairThreshold
|
||||
config.Metainfo.RS.SuccessThreshold = successThreshold
|
||||
config.Metainfo.RS.TotalThreshold = maxThreshold
|
||||
},
|
||||
Satellite: testplanet.Combine(
|
||||
func(log *zap.Logger, index int, config *satellite.Config) {
|
||||
config.Repairer.MaxExcessRateOptimalThreshold = RepairMaxExcessRateOptimalThreshold
|
||||
config.Repairer.InMemoryRepair = inMemoryRepair
|
||||
},
|
||||
testplanet.ReconfigureRS(3, repairThreshold, successThreshold, maxThreshold),
|
||||
),
|
||||
},
|
||||
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
|
||||
satellite := planet.Satellites[0]
|
||||
@ -1266,14 +1252,12 @@ func testRepairGracefullyExited(t *testing.T, inMemoryRepair bool) {
|
||||
StorageNodeCount: 12,
|
||||
UplinkCount: 1,
|
||||
Reconfigure: testplanet.Reconfigure{
|
||||
Satellite: func(log *zap.Logger, index int, config *satellite.Config) {
|
||||
config.Repairer.InMemoryRepair = inMemoryRepair
|
||||
|
||||
config.Metainfo.RS.MinThreshold = 3
|
||||
config.Metainfo.RS.RepairThreshold = 5
|
||||
config.Metainfo.RS.SuccessThreshold = 7
|
||||
config.Metainfo.RS.TotalThreshold = 7
|
||||
},
|
||||
Satellite: testplanet.Combine(
|
||||
func(log *zap.Logger, index int, config *satellite.Config) {
|
||||
config.Repairer.InMemoryRepair = inMemoryRepair
|
||||
},
|
||||
testplanet.ReconfigureRS(3, 5, 7, 7),
|
||||
),
|
||||
},
|
||||
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
|
||||
// first, upload some remote data
|
||||
|
||||
@ -346,7 +346,6 @@ func (repairer *SegmentRepairer) updateAuditFailStatus(ctx context.Context, fail
|
||||
for i, nodeID := range failedAuditNodeIDs {
|
||||
updateRequests[i] = &overlay.UpdateRequest{
|
||||
NodeID: nodeID,
|
||||
IsUp: true,
|
||||
AuditOutcome: overlay.AuditFailure,
|
||||
}
|
||||
}
|
||||
|
||||
@ -366,7 +366,8 @@ func (cache *overlaycache) BatchUpdateStats(ctx context.Context, updateRequests
|
||||
continue
|
||||
}
|
||||
|
||||
auditHistory, err := cache.updateAuditHistoryWithTx(ctx, tx, updateReq.NodeID, now, updateReq.IsUp, updateReq.AuditHistory)
|
||||
isUp := updateReq.AuditOutcome != overlay.AuditOffline
|
||||
auditHistory, err := cache.updateAuditHistoryWithTx(ctx, tx, updateReq.NodeID, now, isUp, updateReq.AuditHistory)
|
||||
if err != nil {
|
||||
doAppendAll = false
|
||||
return err
|
||||
@ -444,7 +445,8 @@ func (cache *overlaycache) UpdateStats(ctx context.Context, updateReq *overlay.U
|
||||
return nil
|
||||
}
|
||||
|
||||
auditHistory, err := cache.updateAuditHistoryWithTx(ctx, tx, updateReq.NodeID, now, updateReq.IsUp, updateReq.AuditHistory)
|
||||
isUp := updateReq.AuditOutcome != overlay.AuditOffline
|
||||
auditHistory, err := cache.updateAuditHistoryWithTx(ctx, tx, updateReq.NodeID, now, isUp, updateReq.AuditHistory)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@ -1259,7 +1261,8 @@ func (cache *overlaycache) populateUpdateNodeStats(dbNode *dbx.Node, updateReq *
|
||||
mon.FloatVal("audit_online_score").Observe(auditOnlineScore) //mon:locked
|
||||
|
||||
totalUptimeCount := dbNode.TotalUptimeCount
|
||||
if updateReq.IsUp {
|
||||
isUp := updateReq.AuditOutcome != overlay.AuditOffline
|
||||
if isUp {
|
||||
totalUptimeCount++
|
||||
}
|
||||
|
||||
@ -1317,7 +1320,7 @@ func (cache *overlaycache) populateUpdateNodeStats(dbNode *dbx.Node, updateReq *
|
||||
updateFields.UnknownAuditSuspended = timeField{set: true, isNil: true}
|
||||
}
|
||||
|
||||
if updateReq.IsUp {
|
||||
if isUp {
|
||||
updateFields.UptimeSuccessCount = int64Field{set: true, value: dbNode.UptimeSuccessCount + 1}
|
||||
updateFields.LastContactSuccess = timeField{set: true, value: now}
|
||||
} else {
|
||||
|
||||
@ -28,11 +28,11 @@ func TestUpdateStats(t *testing.T) {
|
||||
numAudits := int64(2)
|
||||
numUptimes := int64(3)
|
||||
|
||||
// nodes automatically start with 2 uptimes from testplanet startup
|
||||
// nodeA: 1 audit, 2 uptime -> unvetted
|
||||
updateReq := &overlay.UpdateRequest{
|
||||
NodeID: nodeA.ID(),
|
||||
AuditOutcome: overlay.AuditFailure,
|
||||
IsUp: false,
|
||||
AuditOutcome: overlay.AuditOffline,
|
||||
AuditsRequiredForVetting: numAudits,
|
||||
UptimesRequiredForVetting: numUptimes,
|
||||
AuditHistory: testAuditHistoryConfig(),
|
||||
@ -45,8 +45,7 @@ func TestUpdateStats(t *testing.T) {
|
||||
|
||||
// nodeA: 2 audits, 2 uptimes -> unvetted
|
||||
updateReq.NodeID = nodeA.ID()
|
||||
updateReq.AuditOutcome = overlay.AuditFailure
|
||||
updateReq.IsUp = false
|
||||
updateReq.AuditOutcome = overlay.AuditOffline
|
||||
nodeStats, err = cache.UpdateStats(ctx, updateReq, time.Now())
|
||||
require.NoError(t, err)
|
||||
assert.Nil(t, nodeStats.VettedAt)
|
||||
@ -56,7 +55,6 @@ func TestUpdateStats(t *testing.T) {
|
||||
// nodeA: 3 audits, 3 uptimes -> vetted
|
||||
updateReq.NodeID = nodeA.ID()
|
||||
updateReq.AuditOutcome = overlay.AuditSuccess
|
||||
updateReq.IsUp = true
|
||||
nodeStats, err = cache.UpdateStats(ctx, updateReq, time.Now())
|
||||
require.NoError(t, err)
|
||||
assert.NotNil(t, nodeStats.VettedAt)
|
||||
@ -66,7 +64,6 @@ func TestUpdateStats(t *testing.T) {
|
||||
// nodeB: 1 audit, 3 uptimes -> unvetted
|
||||
updateReq.NodeID = nodeB.ID()
|
||||
updateReq.AuditOutcome = overlay.AuditSuccess
|
||||
updateReq.IsUp = true
|
||||
nodeStats, err = cache.UpdateStats(ctx, updateReq, time.Now())
|
||||
require.NoError(t, err)
|
||||
assert.Nil(t, nodeStats.VettedAt)
|
||||
@ -75,8 +72,7 @@ func TestUpdateStats(t *testing.T) {
|
||||
|
||||
// nodeB: 2 audits, 3 uptimes -> vetted
|
||||
updateReq.NodeID = nodeB.ID()
|
||||
updateReq.AuditOutcome = overlay.AuditFailure
|
||||
updateReq.IsUp = false
|
||||
updateReq.AuditOutcome = overlay.AuditOffline
|
||||
nodeStats, err = cache.UpdateStats(ctx, updateReq, time.Now())
|
||||
require.NoError(t, err)
|
||||
assert.NotNil(t, nodeStats.VettedAt)
|
||||
@ -86,7 +82,6 @@ func TestUpdateStats(t *testing.T) {
|
||||
// Don't overwrite node b's vetted_at timestamp
|
||||
updateReq.NodeID = nodeB.ID()
|
||||
updateReq.AuditOutcome = overlay.AuditSuccess
|
||||
updateReq.IsUp = true
|
||||
nodeStats2, err := cache.UpdateStats(ctx, updateReq, time.Now())
|
||||
require.NoError(t, err)
|
||||
assert.NotNil(t, nodeStats2.VettedAt)
|
||||
@ -110,9 +105,10 @@ func TestBatchUpdateStats(t *testing.T) {
|
||||
numUptimes := int64(3)
|
||||
batchSize := 2
|
||||
|
||||
// nodes automatically start with 2 uptimes from testplanet startup
|
||||
// both nodeA and nodeB unvetted
|
||||
updateReqA := &overlay.UpdateRequest{NodeID: nodeA.ID(), AuditOutcome: overlay.AuditFailure, IsUp: false, AuditsRequiredForVetting: numAudits, UptimesRequiredForVetting: numUptimes, AuditHistory: testAuditHistoryConfig()}
|
||||
updateReqB := &overlay.UpdateRequest{NodeID: nodeB.ID(), AuditOutcome: overlay.AuditSuccess, IsUp: true, AuditsRequiredForVetting: numAudits, UptimesRequiredForVetting: numUptimes, AuditHistory: testAuditHistoryConfig()}
|
||||
updateReqA := &overlay.UpdateRequest{NodeID: nodeA.ID(), AuditOutcome: overlay.AuditOffline, AuditsRequiredForVetting: numAudits, UptimesRequiredForVetting: numUptimes, AuditHistory: testAuditHistoryConfig()}
|
||||
updateReqB := &overlay.UpdateRequest{NodeID: nodeB.ID(), AuditOutcome: overlay.AuditSuccess, AuditsRequiredForVetting: numAudits, UptimesRequiredForVetting: numUptimes, AuditHistory: testAuditHistoryConfig()}
|
||||
updateReqs := []*overlay.UpdateRequest{updateReqA, updateReqB}
|
||||
failed, err := cache.BatchUpdateStats(ctx, updateReqs, batchSize, time.Now())
|
||||
require.NoError(t, err)
|
||||
@ -131,8 +127,8 @@ func TestBatchUpdateStats(t *testing.T) {
|
||||
assert.EqualValues(t, 3, nB.Reputation.UptimeCount)
|
||||
|
||||
// nodeA unvetted, nodeB vetted
|
||||
updateReqA = &overlay.UpdateRequest{NodeID: nodeA.ID(), AuditOutcome: overlay.AuditFailure, IsUp: false, AuditsRequiredForVetting: numAudits, UptimesRequiredForVetting: numUptimes, AuditHistory: testAuditHistoryConfig()}
|
||||
updateReqB = &overlay.UpdateRequest{NodeID: nodeB.ID(), AuditOutcome: overlay.AuditFailure, IsUp: false, AuditsRequiredForVetting: numAudits, UptimesRequiredForVetting: numUptimes, AuditHistory: testAuditHistoryConfig()}
|
||||
updateReqA = &overlay.UpdateRequest{NodeID: nodeA.ID(), AuditOutcome: overlay.AuditOffline, AuditsRequiredForVetting: numAudits, UptimesRequiredForVetting: numUptimes, AuditHistory: testAuditHistoryConfig()}
|
||||
updateReqB = &overlay.UpdateRequest{NodeID: nodeB.ID(), AuditOutcome: overlay.AuditFailure, AuditsRequiredForVetting: numAudits, UptimesRequiredForVetting: numUptimes, AuditHistory: testAuditHistoryConfig()}
|
||||
updateReqs = []*overlay.UpdateRequest{updateReqA, updateReqB}
|
||||
failed, err = cache.BatchUpdateStats(ctx, updateReqs, batchSize, time.Now())
|
||||
require.NoError(t, err)
|
||||
@ -148,11 +144,11 @@ func TestBatchUpdateStats(t *testing.T) {
|
||||
require.NoError(t, err)
|
||||
assert.NotNil(t, nB.Reputation.VettedAt)
|
||||
assert.EqualValues(t, 2, nB.Reputation.AuditCount)
|
||||
assert.EqualValues(t, 3, nB.Reputation.UptimeCount)
|
||||
assert.EqualValues(t, 4, nB.Reputation.UptimeCount)
|
||||
|
||||
// both nodeA and nodeB vetted (don't overwrite timestamp)
|
||||
updateReqA = &overlay.UpdateRequest{NodeID: nodeA.ID(), AuditOutcome: overlay.AuditSuccess, IsUp: true, AuditsRequiredForVetting: numAudits, UptimesRequiredForVetting: numUptimes, AuditHistory: testAuditHistoryConfig()}
|
||||
updateReqB = &overlay.UpdateRequest{NodeID: nodeB.ID(), AuditOutcome: overlay.AuditSuccess, IsUp: true, AuditsRequiredForVetting: numAudits, UptimesRequiredForVetting: numUptimes, AuditHistory: testAuditHistoryConfig()}
|
||||
updateReqA = &overlay.UpdateRequest{NodeID: nodeA.ID(), AuditOutcome: overlay.AuditSuccess, AuditsRequiredForVetting: numAudits, UptimesRequiredForVetting: numUptimes, AuditHistory: testAuditHistoryConfig()}
|
||||
updateReqB = &overlay.UpdateRequest{NodeID: nodeB.ID(), AuditOutcome: overlay.AuditSuccess, AuditsRequiredForVetting: numAudits, UptimesRequiredForVetting: numUptimes, AuditHistory: testAuditHistoryConfig()}
|
||||
updateReqs = []*overlay.UpdateRequest{updateReqA, updateReqB}
|
||||
failed, err = cache.BatchUpdateStats(ctx, updateReqs, batchSize, time.Now())
|
||||
require.NoError(t, err)
|
||||
@ -169,7 +165,7 @@ func TestBatchUpdateStats(t *testing.T) {
|
||||
assert.NotNil(t, nB2.Reputation.VettedAt)
|
||||
assert.Equal(t, nB.Reputation.VettedAt, nB2.Reputation.VettedAt)
|
||||
assert.EqualValues(t, 3, nB2.Reputation.AuditCount)
|
||||
assert.EqualValues(t, 4, nB2.Reputation.UptimeCount)
|
||||
assert.EqualValues(t, 5, nB2.Reputation.UptimeCount)
|
||||
})
|
||||
}
|
||||
|
||||
|
||||
@ -358,72 +358,78 @@ func (db *ProjectAccounting) GetBucketUsageRollups(ctx context.Context, projectI
|
||||
|
||||
var bucketUsageRollups []accounting.BucketUsageRollup
|
||||
for _, bucket := range buckets {
|
||||
bucketRollup := accounting.BucketUsageRollup{
|
||||
ProjectID: projectID,
|
||||
BucketName: []byte(bucket),
|
||||
Since: since,
|
||||
Before: before,
|
||||
}
|
||||
err := func() error {
|
||||
bucketRollup := accounting.BucketUsageRollup{
|
||||
ProjectID: projectID,
|
||||
BucketName: []byte(bucket),
|
||||
Since: since,
|
||||
Before: before,
|
||||
}
|
||||
|
||||
// get bucket_bandwidth_rollups
|
||||
rollupsRows, err := db.db.QueryContext(ctx, roullupsQuery, projectID[:], []byte(bucket), since, before)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer func() { err = errs.Combine(err, rollupsRows.Close()) }()
|
||||
|
||||
// fill egress
|
||||
for rollupsRows.Next() {
|
||||
var action pb.PieceAction
|
||||
var settled, inline int64
|
||||
|
||||
err = rollupsRows.Scan(&settled, &inline, &action)
|
||||
// get bucket_bandwidth_rollups
|
||||
rollupsRows, err := db.db.QueryContext(ctx, roullupsQuery, projectID[:], []byte(bucket), since, before)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
return err
|
||||
}
|
||||
defer func() { err = errs.Combine(err, rollupsRows.Close()) }()
|
||||
|
||||
// fill egress
|
||||
for rollupsRows.Next() {
|
||||
var action pb.PieceAction
|
||||
var settled, inline int64
|
||||
|
||||
err = rollupsRows.Scan(&settled, &inline, &action)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
switch action {
|
||||
case pb.PieceAction_GET:
|
||||
bucketRollup.GetEgress += memory.Size(settled + inline).GB()
|
||||
case pb.PieceAction_GET_AUDIT:
|
||||
bucketRollup.AuditEgress += memory.Size(settled + inline).GB()
|
||||
case pb.PieceAction_GET_REPAIR:
|
||||
bucketRollup.RepairEgress += memory.Size(settled + inline).GB()
|
||||
default:
|
||||
continue
|
||||
}
|
||||
}
|
||||
if err := rollupsRows.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
switch action {
|
||||
case pb.PieceAction_GET:
|
||||
bucketRollup.GetEgress += memory.Size(settled + inline).GB()
|
||||
case pb.PieceAction_GET_AUDIT:
|
||||
bucketRollup.AuditEgress += memory.Size(settled + inline).GB()
|
||||
case pb.PieceAction_GET_REPAIR:
|
||||
bucketRollup.RepairEgress += memory.Size(settled + inline).GB()
|
||||
default:
|
||||
continue
|
||||
bucketStorageTallies, err := storageQuery(ctx,
|
||||
dbx.BucketStorageTally_ProjectId(projectID[:]),
|
||||
dbx.BucketStorageTally_BucketName([]byte(bucket)),
|
||||
dbx.BucketStorageTally_IntervalStart(since),
|
||||
dbx.BucketStorageTally_IntervalStart(before))
|
||||
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if err := rollupsRows.Err(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
bucketStorageTallies, err := storageQuery(ctx,
|
||||
dbx.BucketStorageTally_ProjectId(projectID[:]),
|
||||
dbx.BucketStorageTally_BucketName([]byte(bucket)),
|
||||
dbx.BucketStorageTally_IntervalStart(since),
|
||||
dbx.BucketStorageTally_IntervalStart(before))
|
||||
// fill metadata, objects and stored data
|
||||
// hours calculated from previous tallies,
|
||||
// so we skip the most recent one
|
||||
for i := len(bucketStorageTallies) - 1; i > 0; i-- {
|
||||
current := bucketStorageTallies[i]
|
||||
|
||||
hours := bucketStorageTallies[i-1].IntervalStart.Sub(current.IntervalStart).Hours()
|
||||
|
||||
bucketRollup.RemoteStoredData += memory.Size(current.Remote).GB() * hours
|
||||
bucketRollup.InlineStoredData += memory.Size(current.Inline).GB() * hours
|
||||
bucketRollup.MetadataSize += memory.Size(current.MetadataSize).GB() * hours
|
||||
bucketRollup.RemoteSegments += float64(current.RemoteSegmentsCount) * hours
|
||||
bucketRollup.InlineSegments += float64(current.InlineSegmentsCount) * hours
|
||||
bucketRollup.ObjectCount += float64(current.ObjectCount) * hours
|
||||
}
|
||||
|
||||
bucketUsageRollups = append(bucketUsageRollups, bucketRollup)
|
||||
return nil
|
||||
}()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// fill metadata, objects and stored data
|
||||
// hours calculated from previous tallies,
|
||||
// so we skip the most recent one
|
||||
for i := len(bucketStorageTallies) - 1; i > 0; i-- {
|
||||
current := bucketStorageTallies[i]
|
||||
|
||||
hours := bucketStorageTallies[i-1].IntervalStart.Sub(current.IntervalStart).Hours()
|
||||
|
||||
bucketRollup.RemoteStoredData += memory.Size(current.Remote).GB() * hours
|
||||
bucketRollup.InlineStoredData += memory.Size(current.Inline).GB() * hours
|
||||
bucketRollup.MetadataSize += memory.Size(current.MetadataSize).GB() * hours
|
||||
bucketRollup.RemoteSegments += float64(current.RemoteSegmentsCount) * hours
|
||||
bucketRollup.InlineSegments += float64(current.InlineSegmentsCount) * hours
|
||||
bucketRollup.ObjectCount += float64(current.ObjectCount) * hours
|
||||
}
|
||||
|
||||
bucketUsageRollups = append(bucketUsageRollups, bucketRollup)
|
||||
}
|
||||
|
||||
return bucketUsageRollups, nil
|
||||
|
||||
28
scripts/testdata/satellite-config.yaml.lock
vendored
28
scripts/testdata/satellite-config.yaml.lock
vendored
@ -400,32 +400,8 @@ identity.key-path: /root/.local/share/storj/identity/satellite/identity.key
|
||||
# request rate per project per second.
|
||||
# metainfo.rate-limiter.rate: 1000
|
||||
|
||||
# the size of each new erasure share in bytes
|
||||
# metainfo.rs.erasure-share-size: 256 B
|
||||
|
||||
# maximum buffer memory to be allocated for read buffers
|
||||
# metainfo.rs.max-buffer-mem: 4.0 MiB
|
||||
|
||||
# the largest amount of pieces to encode to. n (upper bound for validation).
|
||||
# metainfo.rs.max-total-threshold: 130
|
||||
|
||||
# the minimum pieces required to recover a segment. k.
|
||||
# metainfo.rs.min-threshold: 29
|
||||
|
||||
# the largest amount of pieces to encode to. n (lower bound for validation).
|
||||
# metainfo.rs.min-total-threshold: 95
|
||||
|
||||
# the minimum safe pieces before a repair is triggered. m.
|
||||
# metainfo.rs.repair-threshold: 35
|
||||
|
||||
# the desired total pieces for a segment. o.
|
||||
# metainfo.rs.success-threshold: 80
|
||||
|
||||
# the largest amount of pieces to encode to. n.
|
||||
# metainfo.rs.total-threshold: 110
|
||||
|
||||
# validate redundancy scheme configuration
|
||||
# metainfo.rs.validate: true
|
||||
# redundancy scheme configuration in the format k/m/o/n-sharesize
|
||||
# metainfo.rs: 29/35/80/110-256 B
|
||||
|
||||
# address(es) to send telemetry to (comma-separated)
|
||||
# metrics.addr: collectora.storj.io:9000
|
||||
|
||||
@ -163,16 +163,18 @@ func testConstraints(t *testing.T, ctx *testcontext.Context, store storage.KeyVa
|
||||
{storage.Value("old-value"), nil},
|
||||
{storage.Value("old-value"), storage.Value("new-value")},
|
||||
} {
|
||||
errTag := fmt.Sprintf("%d. %+v", i, tt)
|
||||
key := storage.Key("test-key")
|
||||
val := storage.Value("test-value")
|
||||
defer func() { _ = store.Delete(ctx, key) }()
|
||||
func() {
|
||||
errTag := fmt.Sprintf("%d. %+v", i, tt)
|
||||
key := storage.Key("test-key")
|
||||
val := storage.Value("test-value")
|
||||
defer func() { _ = store.Delete(ctx, key) }()
|
||||
|
||||
err := store.Put(ctx, key, val)
|
||||
require.NoError(t, err, errTag)
|
||||
err := store.Put(ctx, key, val)
|
||||
require.NoError(t, err, errTag)
|
||||
|
||||
err = store.CompareAndSwap(ctx, key, tt.old, tt.new)
|
||||
assert.True(t, storage.ErrValueChanged.Has(err), "%s: unexpected error: %+v", errTag, err)
|
||||
err = store.CompareAndSwap(ctx, key, tt.old, tt.new)
|
||||
assert.True(t, storage.ErrValueChanged.Has(err), "%s: unexpected error: %+v", errTag, err)
|
||||
}()
|
||||
}
|
||||
})
|
||||
|
||||
|
||||
@ -376,8 +376,15 @@ func TestHeldAmountApi(t *testing.T) {
|
||||
JoinedAt: date,
|
||||
}
|
||||
|
||||
stefanID, err := storj.NodeIDFromString("118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW")
|
||||
require.NoError(t, err)
|
||||
|
||||
held2 := payout.SatelliteHeldHistory{
|
||||
SatelliteID: stefanID,
|
||||
}
|
||||
|
||||
var periods []payout.SatelliteHeldHistory
|
||||
periods = append(periods, held)
|
||||
periods = append(periods, held, held2)
|
||||
|
||||
expected, err := json.Marshal(periods)
|
||||
require.NoError(t, err)
|
||||
|
||||
@ -177,15 +177,14 @@ func TestWorkerFailure_IneligibleNodeAge(t *testing.T) {
|
||||
StorageNodeCount: 5,
|
||||
UplinkCount: 1,
|
||||
Reconfigure: testplanet.Reconfigure{
|
||||
Satellite: func(logger *zap.Logger, index int, config *satellite.Config) {
|
||||
// Set the required node age to 1 month.
|
||||
config.GracefulExit.NodeMinAgeInMonths = 1
|
||||
Satellite: testplanet.Combine(
|
||||
func(log *zap.Logger, index int, config *satellite.Config) {
|
||||
// Set the required node age to 1 month.
|
||||
config.GracefulExit.NodeMinAgeInMonths = 1
|
||||
},
|
||||
testplanet.ReconfigureRS(2, 3, successThreshold, successThreshold),
|
||||
),
|
||||
|
||||
config.Metainfo.RS.MinThreshold = 2
|
||||
config.Metainfo.RS.RepairThreshold = 3
|
||||
config.Metainfo.RS.SuccessThreshold = successThreshold
|
||||
config.Metainfo.RS.TotalThreshold = successThreshold
|
||||
},
|
||||
StorageNode: func(index int, config *storagenode.Config) {
|
||||
config.GracefulExit.NumWorkers = 2
|
||||
config.GracefulExit.NumConcurrentTransfers = 2
|
||||
|
||||
@ -398,7 +398,6 @@ func (service *Service) sendOrdersFromFileStore(ctx context.Context, now time.Ti
|
||||
var group errgroup.Group
|
||||
attemptedSatellites := 0
|
||||
ctx, cancel := context.WithTimeout(ctx, service.config.SenderTimeout)
|
||||
defer cancel()
|
||||
|
||||
for satelliteID, unsentInfo := range ordersBySatellite {
|
||||
satelliteID, unsentInfo := satelliteID, unsentInfo
|
||||
@ -430,6 +429,7 @@ func (service *Service) sendOrdersFromFileStore(ctx context.Context, now time.Ti
|
||||
|
||||
}
|
||||
_ = group.Wait() // doesn't return errors
|
||||
cancel()
|
||||
|
||||
// if all satellites that orders need to be sent to are offline, exit and try again later.
|
||||
if attemptedSatellites == 0 {
|
||||
|
||||
@ -211,7 +211,8 @@ func TestSatellitePayStubPeriodCached(t *testing.T) {
|
||||
heldAmountDB := db.Payout()
|
||||
reputationDB := db.Reputation()
|
||||
satellitesDB := db.Satellites()
|
||||
service := payout.NewService(nil, heldAmountDB, reputationDB, satellitesDB, nil)
|
||||
service, err := payout.NewService(nil, heldAmountDB, reputationDB, satellitesDB, nil)
|
||||
require.NoError(t, err)
|
||||
|
||||
payStub := payout.PayStub{
|
||||
SatelliteID: storj.NodeID{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1},
|
||||
@ -261,7 +262,8 @@ func TestAllPayStubPeriodCached(t *testing.T) {
|
||||
heldAmountDB := db.Payout()
|
||||
reputationDB := db.Reputation()
|
||||
satellitesDB := db.Satellites()
|
||||
service := payout.NewService(nil, heldAmountDB, reputationDB, satellitesDB, nil)
|
||||
service, err := payout.NewService(nil, heldAmountDB, reputationDB, satellitesDB, nil)
|
||||
require.NoError(t, err)
|
||||
|
||||
payStub := payout.PayStub{
|
||||
SatelliteID: storj.NodeID{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1},
|
||||
|
||||
@ -39,6 +39,8 @@ var (
|
||||
type Service struct {
|
||||
log *zap.Logger
|
||||
|
||||
stefanSatellite storj.NodeID
|
||||
|
||||
db DB
|
||||
reputationDB reputation.DB
|
||||
satellitesDB satellites.DB
|
||||
@ -46,14 +48,20 @@ type Service struct {
|
||||
}
|
||||
|
||||
// NewService creates new instance of service.
|
||||
func NewService(log *zap.Logger, db DB, reputationDB reputation.DB, satelliteDB satellites.DB, trust *trust.Pool) *Service {
|
||||
return &Service{
|
||||
log: log,
|
||||
db: db,
|
||||
reputationDB: reputationDB,
|
||||
satellitesDB: satelliteDB,
|
||||
trust: trust,
|
||||
func NewService(log *zap.Logger, db DB, reputationDB reputation.DB, satelliteDB satellites.DB, trust *trust.Pool) (_ *Service, err error) {
|
||||
id, err := storj.NodeIDFromString("118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW")
|
||||
if err != nil {
|
||||
return &Service{}, err
|
||||
}
|
||||
|
||||
return &Service{
|
||||
log: log,
|
||||
stefanSatellite: id,
|
||||
db: db,
|
||||
reputationDB: reputationDB,
|
||||
satellitesDB: satelliteDB,
|
||||
trust: trust,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// SatellitePayStubMonthly retrieves held amount for particular satellite for selected month from storagenode database.
|
||||
@ -161,17 +169,18 @@ func (service *Service) AllPeriods(ctx context.Context) (_ []string, err error)
|
||||
// AllHeldbackHistory retrieves heldback history for all satellites from storagenode database.
|
||||
func (service *Service) AllHeldbackHistory(ctx context.Context) (result []SatelliteHeldHistory, err error) {
|
||||
defer mon.Task()(&ctx)(&err)
|
||||
satellitesIDs := service.trust.GetSatellites(ctx)
|
||||
|
||||
satellites := service.trust.GetSatellites(ctx)
|
||||
for i := 0; i < len(satellites); i++ {
|
||||
satellitesIDs = append(satellitesIDs, service.stefanSatellite)
|
||||
for i := 0; i < len(satellitesIDs); i++ {
|
||||
var history SatelliteHeldHistory
|
||||
|
||||
helds, err := service.db.SatellitesHeldbackHistory(ctx, satellites[i])
|
||||
helds, err := service.db.SatellitesHeldbackHistory(ctx, satellitesIDs[i])
|
||||
if err != nil {
|
||||
return nil, ErrPayoutService.Wrap(err)
|
||||
}
|
||||
|
||||
disposed, err := service.db.SatellitesDisposedHistory(ctx, satellites[i])
|
||||
disposed, err := service.db.SatellitesDisposedHistory(ctx, satellitesIDs[i])
|
||||
if err != nil {
|
||||
return nil, ErrPayoutService.Wrap(err)
|
||||
}
|
||||
@ -192,20 +201,22 @@ func (service *Service) AllHeldbackHistory(ctx context.Context) (result []Satell
|
||||
}
|
||||
|
||||
history.TotalDisposed = disposed
|
||||
history.SatelliteID = satellites[i]
|
||||
url, err := service.trust.GetNodeURL(ctx, satellites[i])
|
||||
history.SatelliteID = satellitesIDs[i]
|
||||
|
||||
if satellitesIDs[i] != service.stefanSatellite {
|
||||
url, err := service.trust.GetNodeURL(ctx, satellitesIDs[i])
|
||||
if err != nil {
|
||||
return nil, ErrPayoutService.Wrap(err)
|
||||
}
|
||||
history.SatelliteName = url.Address
|
||||
}
|
||||
|
||||
stats, err := service.reputationDB.Get(ctx, satellitesIDs[i])
|
||||
if err != nil {
|
||||
return nil, ErrPayoutService.Wrap(err)
|
||||
}
|
||||
|
||||
stats, err := service.reputationDB.Get(ctx, satellites[i])
|
||||
if err != nil {
|
||||
return nil, ErrPayoutService.Wrap(err)
|
||||
}
|
||||
|
||||
history.SatelliteName = url.Address
|
||||
history.JoinedAt = stats.JoinedAt
|
||||
|
||||
result = append(result, history)
|
||||
}
|
||||
|
||||
@ -217,6 +228,8 @@ func (service *Service) AllSatellitesPayoutPeriod(ctx context.Context, period st
|
||||
defer mon.Task()(&ctx)(&err)
|
||||
|
||||
satelliteIDs := service.trust.GetSatellites(ctx)
|
||||
|
||||
satelliteIDs = append(satelliteIDs, service.stefanSatellite)
|
||||
for i := 0; i < len(satelliteIDs); i++ {
|
||||
var payoutForPeriod SatellitePayoutForPeriod
|
||||
paystub, err := service.db.GetPayStub(ctx, satelliteIDs[i], period)
|
||||
@ -249,9 +262,13 @@ func (service *Service) AllSatellitesPayoutPeriod(ctx context.Context, period st
|
||||
return nil, ErrPayoutService.Wrap(err)
|
||||
}
|
||||
|
||||
url, err := service.trust.GetNodeURL(ctx, satelliteIDs[i])
|
||||
if err != nil {
|
||||
return nil, ErrPayoutService.Wrap(err)
|
||||
if satelliteIDs[i] != service.stefanSatellite {
|
||||
url, err := service.trust.GetNodeURL(ctx, satelliteIDs[i])
|
||||
if err != nil {
|
||||
return nil, ErrPayoutService.Wrap(err)
|
||||
}
|
||||
|
||||
payoutForPeriod.SatelliteURL = url.Address
|
||||
}
|
||||
|
||||
if satellite.Status == satellites.ExitSucceeded {
|
||||
@ -281,7 +298,6 @@ func (service *Service) AllSatellitesPayoutPeriod(ctx context.Context, period st
|
||||
payoutForPeriod.Earned = earned
|
||||
payoutForPeriod.SatelliteID = satelliteIDs[i].String()
|
||||
payoutForPeriod.SurgePercent = paystub.SurgePercent
|
||||
payoutForPeriod.SatelliteURL = url.Address
|
||||
payoutForPeriod.Paid = paystub.Paid
|
||||
payoutForPeriod.HeldPercent = heldPercent
|
||||
|
||||
|
||||
@ -555,13 +555,17 @@ func New(log *zap.Logger, full *identity.FullIdentity, db DB, revocationDB exten
|
||||
}
|
||||
|
||||
{ // setup payout service.
|
||||
peer.Payout.Service = payout.NewService(
|
||||
service, err := payout.NewService(
|
||||
peer.Log.Named("payout:service"),
|
||||
peer.DB.Payout(),
|
||||
peer.DB.Reputation(),
|
||||
peer.DB.Satellites(),
|
||||
peer.Storage2.Trust,
|
||||
)
|
||||
if err != nil {
|
||||
return nil, errs.Combine(err, peer.Close())
|
||||
}
|
||||
peer.Payout.Service = service
|
||||
peer.Payout.Endpoint = payout.NewEndpoint(
|
||||
peer.Log.Named("payout:endpoint"),
|
||||
peer.Dialer,
|
||||
|
||||
@ -50,23 +50,25 @@ func TestUploadAndPartialDownload(t *testing.T) {
|
||||
{1513, 1584},
|
||||
{13581, 4783},
|
||||
} {
|
||||
if piecestore.DefaultConfig.InitialStep < tt.size {
|
||||
t.Fatal("test expects initial step to be larger than size to download")
|
||||
}
|
||||
totalDownload += piecestore.DefaultConfig.InitialStep
|
||||
func() {
|
||||
if piecestore.DefaultConfig.InitialStep < tt.size {
|
||||
t.Fatal("test expects initial step to be larger than size to download")
|
||||
}
|
||||
totalDownload += piecestore.DefaultConfig.InitialStep
|
||||
|
||||
download, cleanup, err := planet.Uplinks[0].DownloadStreamRange(ctx, planet.Satellites[0], "testbucket", "test/path", tt.offset, -1)
|
||||
require.NoError(t, err)
|
||||
defer ctx.Check(cleanup)
|
||||
download, cleanup, err := planet.Uplinks[0].DownloadStreamRange(ctx, planet.Satellites[0], "testbucket", "test/path", tt.offset, -1)
|
||||
require.NoError(t, err)
|
||||
defer ctx.Check(cleanup)
|
||||
|
||||
data := make([]byte, tt.size)
|
||||
n, err := io.ReadFull(download, data)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, int(tt.size), n)
|
||||
data := make([]byte, tt.size)
|
||||
n, err := io.ReadFull(download, data)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, int(tt.size), n)
|
||||
|
||||
assert.Equal(t, expectedData[tt.offset:tt.offset+tt.size], data)
|
||||
assert.Equal(t, expectedData[tt.offset:tt.offset+tt.size], data)
|
||||
|
||||
require.NoError(t, download.Close())
|
||||
require.NoError(t, download.Close())
|
||||
}()
|
||||
}
|
||||
|
||||
var totalBandwidthUsage bandwidth.Usage
|
||||
@ -528,8 +530,6 @@ func TestDeletePieces(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestTooManyRequests(t *testing.T) {
|
||||
t.Skip("flaky, because of EOF issues")
|
||||
|
||||
const uplinkCount = 6
|
||||
const maxConcurrent = 3
|
||||
const expectedFailures = uplinkCount - maxConcurrent
|
||||
|
||||
@ -226,45 +226,47 @@ func TestOrderLimitGetValidation(t *testing.T) {
|
||||
err: "expected get or get repair or audit action got PUT",
|
||||
},
|
||||
} {
|
||||
client, err := planet.Uplinks[0].DialPiecestore(ctx, planet.StorageNodes[0])
|
||||
require.NoError(t, err)
|
||||
defer ctx.Check(client.Close)
|
||||
|
||||
signer := signing.SignerFromFullIdentity(planet.Satellites[0].Identity)
|
||||
satellite := planet.Satellites[0].Identity
|
||||
if tt.satellite != nil {
|
||||
signer = signing.SignerFromFullIdentity(tt.satellite)
|
||||
satellite = tt.satellite
|
||||
}
|
||||
|
||||
orderLimit, piecePrivateKey := GenerateOrderLimit(
|
||||
t,
|
||||
satellite.ID,
|
||||
planet.StorageNodes[0].ID(),
|
||||
tt.pieceID,
|
||||
tt.action,
|
||||
tt.serialNumber,
|
||||
tt.pieceExpiration,
|
||||
tt.orderExpiration,
|
||||
tt.limit,
|
||||
)
|
||||
|
||||
orderLimit, err = signing.SignOrderLimit(ctx, signer, orderLimit)
|
||||
require.NoError(t, err)
|
||||
|
||||
downloader, err := client.Download(ctx, orderLimit, piecePrivateKey, 0, tt.limit)
|
||||
require.NoError(t, err)
|
||||
|
||||
buffer, readErr := ioutil.ReadAll(downloader)
|
||||
closeErr := downloader.Close()
|
||||
err = errs.Combine(readErr, closeErr)
|
||||
if tt.err != "" {
|
||||
assert.Equal(t, 0, len(buffer))
|
||||
require.Error(t, err)
|
||||
require.Contains(t, err.Error(), tt.err)
|
||||
} else {
|
||||
func() {
|
||||
client, err := planet.Uplinks[0].DialPiecestore(ctx, planet.StorageNodes[0])
|
||||
require.NoError(t, err)
|
||||
}
|
||||
defer ctx.Check(client.Close)
|
||||
|
||||
signer := signing.SignerFromFullIdentity(planet.Satellites[0].Identity)
|
||||
satellite := planet.Satellites[0].Identity
|
||||
if tt.satellite != nil {
|
||||
signer = signing.SignerFromFullIdentity(tt.satellite)
|
||||
satellite = tt.satellite
|
||||
}
|
||||
|
||||
orderLimit, piecePrivateKey := GenerateOrderLimit(
|
||||
t,
|
||||
satellite.ID,
|
||||
planet.StorageNodes[0].ID(),
|
||||
tt.pieceID,
|
||||
tt.action,
|
||||
tt.serialNumber,
|
||||
tt.pieceExpiration,
|
||||
tt.orderExpiration,
|
||||
tt.limit,
|
||||
)
|
||||
|
||||
orderLimit, err = signing.SignOrderLimit(ctx, signer, orderLimit)
|
||||
require.NoError(t, err)
|
||||
|
||||
downloader, err := client.Download(ctx, orderLimit, piecePrivateKey, 0, tt.limit)
|
||||
require.NoError(t, err)
|
||||
|
||||
buffer, readErr := ioutil.ReadAll(downloader)
|
||||
closeErr := downloader.Close()
|
||||
err = errs.Combine(readErr, closeErr)
|
||||
if tt.err != "" {
|
||||
assert.Equal(t, 0, len(buffer))
|
||||
require.Error(t, err)
|
||||
require.Contains(t, err.Error(), tt.err)
|
||||
} else {
|
||||
require.NoError(t, err)
|
||||
}
|
||||
}()
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
@ -349,103 +349,111 @@ func (db *DB) MigrateToLatest(ctx context.Context) error {
|
||||
// Preflight conducts a pre-flight check to ensure correct schemas and minimal read+write functionality of the database tables.
|
||||
func (db *DB) Preflight(ctx context.Context) (err error) {
|
||||
for dbName, dbContainer := range db.SQLDBs {
|
||||
nextDB := dbContainer.GetDB()
|
||||
// Preflight stage 1: test schema correctness
|
||||
schema, err := sqliteutil.QuerySchema(ctx, nextDB)
|
||||
if err != nil {
|
||||
return ErrPreflight.New("database %q: schema check failed: %v", dbName, err)
|
||||
}
|
||||
// we don't care about changes in versions table
|
||||
schema.DropTable("versions")
|
||||
// if there was a previous pre-flight failure, test_table might still be in the schema
|
||||
schema.DropTable("test_table")
|
||||
|
||||
// If tables and indexes of the schema are empty, set to nil
|
||||
// to help with comparison to the snapshot.
|
||||
if len(schema.Tables) == 0 {
|
||||
schema.Tables = nil
|
||||
}
|
||||
if len(schema.Indexes) == 0 {
|
||||
schema.Indexes = nil
|
||||
}
|
||||
|
||||
// get expected schema
|
||||
expectedSchema := Schema()[dbName]
|
||||
|
||||
// find extra indexes
|
||||
var extraIdxs []*dbschema.Index
|
||||
for _, idx := range schema.Indexes {
|
||||
if _, exists := expectedSchema.FindIndex(idx.Name); exists {
|
||||
continue
|
||||
}
|
||||
|
||||
extraIdxs = append(extraIdxs, idx)
|
||||
}
|
||||
// drop index from schema if it is not unique to not fail preflight
|
||||
for _, idx := range extraIdxs {
|
||||
if !idx.Unique {
|
||||
schema.DropIndex(idx.Name)
|
||||
}
|
||||
}
|
||||
// warn that schema contains unexpected indexes
|
||||
if len(extraIdxs) > 0 {
|
||||
db.log.Warn(fmt.Sprintf("database %q: schema contains unexpected indices %v", dbName, extraIdxs))
|
||||
}
|
||||
|
||||
// expect expected schema to match actual schema
|
||||
if diff := cmp.Diff(expectedSchema, schema); diff != "" {
|
||||
return ErrPreflight.New("database %q: expected schema does not match actual: %s", dbName, diff)
|
||||
}
|
||||
|
||||
// Preflight stage 2: test basic read/write access
|
||||
// for each database, create a new table, insert a row into that table, retrieve and validate that row, and drop the table.
|
||||
|
||||
// drop test table in case the last preflight check failed before table could be dropped
|
||||
_, err = nextDB.ExecContext(ctx, "DROP TABLE IF EXISTS test_table")
|
||||
if err != nil {
|
||||
return ErrPreflight.New("database %q: failed drop if test_table: %w", dbName, err)
|
||||
}
|
||||
_, err = nextDB.ExecContext(ctx, "CREATE TABLE test_table(id int NOT NULL, name varchar(30), PRIMARY KEY (id))")
|
||||
if err != nil {
|
||||
return ErrPreflight.New("database %q: failed create test_table: %w", dbName, err)
|
||||
}
|
||||
|
||||
var expectedID, actualID int
|
||||
var expectedName, actualName string
|
||||
expectedID = 1
|
||||
expectedName = "TEST"
|
||||
_, err = nextDB.ExecContext(ctx, "INSERT INTO test_table VALUES ( ?, ? )", expectedID, expectedName)
|
||||
if err != nil {
|
||||
return ErrPreflight.New("database: %q: failed inserting test value: %w", dbName, err)
|
||||
}
|
||||
|
||||
rows, err := nextDB.QueryContext(ctx, "SELECT id, name FROM test_table")
|
||||
if err != nil {
|
||||
return ErrPreflight.New("database: %q: failed selecting test value: %w", dbName, err)
|
||||
}
|
||||
defer func() { err = errs.Combine(err, rows.Close()) }()
|
||||
if !rows.Next() {
|
||||
return ErrPreflight.New("database %q: no rows in test_table", dbName)
|
||||
}
|
||||
err = rows.Scan(&actualID, &actualName)
|
||||
if err != nil {
|
||||
return ErrPreflight.New("database %q: failed scanning row: %w", dbName, err)
|
||||
}
|
||||
if expectedID != actualID || expectedName != actualName {
|
||||
return ErrPreflight.New("database %q: expected (%d, '%s'), actual (%d, '%s')", dbName, expectedID, expectedName, actualID, actualName)
|
||||
}
|
||||
if rows.Next() {
|
||||
return ErrPreflight.New("database %q: more than one row in test_table", dbName)
|
||||
}
|
||||
|
||||
_, err = nextDB.ExecContext(ctx, "DROP TABLE test_table")
|
||||
if err != nil {
|
||||
return ErrPreflight.New("database %q: failed drop test_table %w", dbName, err)
|
||||
if err := db.preflight(ctx, dbName, dbContainer); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (db *DB) preflight(ctx context.Context, dbName string, dbContainer DBContainer) error {
|
||||
nextDB := dbContainer.GetDB()
|
||||
// Preflight stage 1: test schema correctness
|
||||
schema, err := sqliteutil.QuerySchema(ctx, nextDB)
|
||||
if err != nil {
|
||||
return ErrPreflight.New("database %q: schema check failed: %v", dbName, err)
|
||||
}
|
||||
// we don't care about changes in versions table
|
||||
schema.DropTable("versions")
|
||||
// if there was a previous pre-flight failure, test_table might still be in the schema
|
||||
schema.DropTable("test_table")
|
||||
|
||||
// If tables and indexes of the schema are empty, set to nil
|
||||
// to help with comparison to the snapshot.
|
||||
if len(schema.Tables) == 0 {
|
||||
schema.Tables = nil
|
||||
}
|
||||
if len(schema.Indexes) == 0 {
|
||||
schema.Indexes = nil
|
||||
}
|
||||
|
||||
// get expected schema
|
||||
expectedSchema := Schema()[dbName]
|
||||
|
||||
// find extra indexes
|
||||
var extraIdxs []*dbschema.Index
|
||||
for _, idx := range schema.Indexes {
|
||||
if _, exists := expectedSchema.FindIndex(idx.Name); exists {
|
||||
continue
|
||||
}
|
||||
|
||||
extraIdxs = append(extraIdxs, idx)
|
||||
}
|
||||
// drop index from schema if it is not unique to not fail preflight
|
||||
for _, idx := range extraIdxs {
|
||||
if !idx.Unique {
|
||||
schema.DropIndex(idx.Name)
|
||||
}
|
||||
}
|
||||
// warn that schema contains unexpected indexes
|
||||
if len(extraIdxs) > 0 {
|
||||
db.log.Warn(fmt.Sprintf("database %q: schema contains unexpected indices %v", dbName, extraIdxs))
|
||||
}
|
||||
|
||||
// expect expected schema to match actual schema
|
||||
if diff := cmp.Diff(expectedSchema, schema); diff != "" {
|
||||
return ErrPreflight.New("database %q: expected schema does not match actual: %s", dbName, diff)
|
||||
}
|
||||
|
||||
// Preflight stage 2: test basic read/write access
|
||||
// for each database, create a new table, insert a row into that table, retrieve and validate that row, and drop the table.
|
||||
|
||||
// drop test table in case the last preflight check failed before table could be dropped
|
||||
_, err = nextDB.ExecContext(ctx, "DROP TABLE IF EXISTS test_table")
|
||||
if err != nil {
|
||||
return ErrPreflight.New("database %q: failed drop if test_table: %w", dbName, err)
|
||||
}
|
||||
_, err = nextDB.ExecContext(ctx, "CREATE TABLE test_table(id int NOT NULL, name varchar(30), PRIMARY KEY (id))")
|
||||
if err != nil {
|
||||
return ErrPreflight.New("database %q: failed create test_table: %w", dbName, err)
|
||||
}
|
||||
|
||||
var expectedID, actualID int
|
||||
var expectedName, actualName string
|
||||
expectedID = 1
|
||||
expectedName = "TEST"
|
||||
_, err = nextDB.ExecContext(ctx, "INSERT INTO test_table VALUES ( ?, ? )", expectedID, expectedName)
|
||||
if err != nil {
|
||||
return ErrPreflight.New("database: %q: failed inserting test value: %w", dbName, err)
|
||||
}
|
||||
|
||||
rows, err := nextDB.QueryContext(ctx, "SELECT id, name FROM test_table")
|
||||
if err != nil {
|
||||
return ErrPreflight.New("database: %q: failed selecting test value: %w", dbName, err)
|
||||
}
|
||||
defer func() { err = errs.Combine(err, rows.Close()) }()
|
||||
if !rows.Next() {
|
||||
return ErrPreflight.New("database %q: no rows in test_table", dbName)
|
||||
}
|
||||
err = rows.Scan(&actualID, &actualName)
|
||||
if err != nil {
|
||||
return ErrPreflight.New("database %q: failed scanning row: %w", dbName, err)
|
||||
}
|
||||
if expectedID != actualID || expectedName != actualName {
|
||||
return ErrPreflight.New("database %q: expected (%d, '%s'), actual (%d, '%s')", dbName, expectedID, expectedName, actualID, actualName)
|
||||
}
|
||||
if rows.Next() {
|
||||
return ErrPreflight.New("database %q: more than one row in test_table", dbName)
|
||||
}
|
||||
|
||||
_, err = nextDB.ExecContext(ctx, "DROP TABLE test_table")
|
||||
if err != nil {
|
||||
return ErrPreflight.New("database %q: failed drop test_table %w", dbName, err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// Close closes any resources.
|
||||
func (db *DB) Close() error {
|
||||
return db.closeDatabases()
|
||||
|
||||
@ -67,11 +67,11 @@ docker run -p 8080:8080 storjlabs/satellite-ui:latest
|
||||
- [unit](./unit "unit") folder: contains project unit tests.
|
||||
### Configuration files
|
||||
- **.env**: file for environment level variables.
|
||||
- **.gitignore**: folders, files and extentions which are ignored for git.
|
||||
- **.gitignore**: folders, files and extensions which are ignored for git.
|
||||
- **babel.config.js**: [babel](https://babeljs.io/) configuration for javascript transcompilation.
|
||||
- **index.html**: DOM entry point.
|
||||
- **jestSetup.ts**: [jest](https://jestjs.io/) configuration for unit testing.
|
||||
- **package.json**: file holds various metadata relevant to the project such as version, dependencies, scripts and configurations.
|
||||
- **tsconfig.json**: holds [TypeScript](https://www.typescriptlang.org/) configurations.
|
||||
- **tslint.json**: holds [TypeScript](https://www.typescriptlang.org/) linter configurations.
|
||||
- **vue.config.js**: holds [Vue](https://vuejs.org/) configurations.
|
||||
- **vue.config.js**: holds [Vue](https://vuejs.org/) configurations.
|
||||
|
||||
@ -68,7 +68,6 @@ import { APPSTATE_ACTIONS } from '@/app/store/modules/appState';
|
||||
import { NODE_ACTIONS } from '@/app/store/modules/node';
|
||||
import { NOTIFICATIONS_ACTIONS } from '@/app/store/modules/notifications';
|
||||
import { PAYOUT_ACTIONS } from '@/app/store/modules/payout';
|
||||
import { NotificationsCursor } from '@/app/types/notifications';
|
||||
|
||||
const {
|
||||
GET_NODE_INFO,
|
||||
@ -90,18 +89,23 @@ const {
|
||||
export default class SNOHeader extends Vue {
|
||||
public isNotificationPopupShown: boolean = false;
|
||||
public isOptionsShown: boolean = false;
|
||||
private readonly FIRST_PAGE: number = 1;
|
||||
|
||||
/**
|
||||
* Lifecycle hook before render.
|
||||
* Fetches first page of notifications.
|
||||
*/
|
||||
public beforeMount(): void {
|
||||
public async beforeMount(): Promise<void> {
|
||||
await this.$store.dispatch(APPSTATE_ACTIONS.SET_LOADING, true);
|
||||
|
||||
try {
|
||||
this.$store.dispatch(NODE_ACTIONS.GET_NODE_INFO);
|
||||
this.$store.dispatch(NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS, new NotificationsCursor(1));
|
||||
await this.$store.dispatch(NODE_ACTIONS.GET_NODE_INFO);
|
||||
await this.$store.dispatch(NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS, this.FIRST_PAGE);
|
||||
} catch (error) {
|
||||
console.error(error.message);
|
||||
}
|
||||
|
||||
await this.$store.dispatch(APPSTATE_ACTIONS.SET_LOADING, false);
|
||||
}
|
||||
|
||||
public get nodeId(): string {
|
||||
@ -192,7 +196,7 @@ export default class SNOHeader extends Vue {
|
||||
}
|
||||
|
||||
try {
|
||||
await this.$store.dispatch(NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS, new NotificationsCursor(1));
|
||||
await this.$store.dispatch(NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS, this.FIRST_PAGE);
|
||||
} catch (error) {
|
||||
console.error(error.message);
|
||||
}
|
||||
|
||||
@ -12,10 +12,10 @@
|
||||
<div
|
||||
class="notification-popup-container__content"
|
||||
:class="{'collapsed': isCollapsed}"
|
||||
v-if="latestNotifications.length"
|
||||
v-if="latest.length"
|
||||
>
|
||||
<SNONotification
|
||||
v-for="notification in latestNotifications"
|
||||
v-for="notification in latest"
|
||||
:key="notification.id"
|
||||
is-small="true"
|
||||
:notification="notification"
|
||||
@ -34,6 +34,7 @@ import { Component, Vue } from 'vue-property-decorator';
|
||||
import SNONotification from '@/app/components/notifications/SNONotification.vue';
|
||||
|
||||
import { RouteConfig } from '@/app/router';
|
||||
import { UINotification } from '@/app/types/notifications';
|
||||
|
||||
@Component({
|
||||
components: {
|
||||
@ -49,7 +50,7 @@ export default class NotificationsPopup extends Vue {
|
||||
/**
|
||||
* Represents first page of notifications.
|
||||
*/
|
||||
public get latestNotifications(): Notification[] {
|
||||
public get latest(): UINotification[] {
|
||||
return this.$store.state.notificationsModule.latestNotifications;
|
||||
}
|
||||
|
||||
@ -57,7 +58,7 @@ export default class NotificationsPopup extends Vue {
|
||||
* Indicates if popup is smaller than with scroll.
|
||||
*/
|
||||
public get isCollapsed(): boolean {
|
||||
return this.latestNotifications.length < 4;
|
||||
return this.latest.length < 4;
|
||||
}
|
||||
}
|
||||
</script>
|
||||
|
||||
@ -30,12 +30,12 @@
|
||||
import { Component, Prop, Vue } from 'vue-property-decorator';
|
||||
|
||||
import { NOTIFICATIONS_ACTIONS } from '@/app/store/modules/notifications';
|
||||
import { Notification } from '@/app/types/notifications';
|
||||
import { UINotification } from '@/app/types/notifications';
|
||||
|
||||
@Component
|
||||
export default class SNONotification extends Vue {
|
||||
@Prop({default: () => new Notification()})
|
||||
public readonly notification: Notification;
|
||||
@Prop({default: () => new UINotification()})
|
||||
public readonly notification: UINotification;
|
||||
|
||||
/**
|
||||
* isSmall props indicates if component used in popup.
|
||||
|
||||
@ -89,7 +89,7 @@ export default class HeldProgress extends Vue {
|
||||
),
|
||||
new HeldStep(
|
||||
'+50%',
|
||||
'Month 15',
|
||||
'Month 16',
|
||||
this.monthsOnNetwork > 15,
|
||||
this.monthsOnNetwork < 15,
|
||||
),
|
||||
|
||||
@ -4,17 +4,19 @@
|
||||
import Vue from 'vue';
|
||||
import Vuex from 'vuex';
|
||||
|
||||
import { makeNotificationsModule } from '@/app/store/modules/notifications';
|
||||
import { newNotificationsModule } from '@/app/store/modules/notifications';
|
||||
import { makePayoutModule } from '@/app/store/modules/payout';
|
||||
import { NotificationsHttpApi } from '@/storagenode/api/notifications';
|
||||
import { PayoutHttpApi } from '@/storagenode/api/payout';
|
||||
import { SNOApi } from '@/storagenode/api/storagenode';
|
||||
import { NotificationsService } from '@/storagenode/notifications/service';
|
||||
import { PayoutService } from '@/storagenode/payouts/service';
|
||||
|
||||
import { appStateModule } from './modules/appState';
|
||||
import { makeNodeModule } from './modules/node';
|
||||
|
||||
const notificationsApi = new NotificationsHttpApi();
|
||||
const notificationsService = new NotificationsService(notificationsApi);
|
||||
const payoutApi = new PayoutHttpApi();
|
||||
const payoutService = new PayoutService(payoutApi);
|
||||
const nodeApi = new SNOApi();
|
||||
@ -28,7 +30,7 @@ export const store = new Vuex.Store({
|
||||
modules: {
|
||||
node: makeNodeModule(nodeApi),
|
||||
appStateModule,
|
||||
notificationsModule: makeNotificationsModule(notificationsApi),
|
||||
notificationsModule: newNotificationsModule(notificationsService),
|
||||
payoutModule: makePayoutModule(payoutApi, payoutService),
|
||||
},
|
||||
});
|
||||
|
||||
@ -1,12 +1,8 @@
|
||||
// Copyright (C) 2019 Storj Labs, Inc.
|
||||
// See LICENSE for copying information.
|
||||
|
||||
import {
|
||||
Notification,
|
||||
NotificationsApi,
|
||||
NotificationsCursor,
|
||||
NotificationsState,
|
||||
} from '@/app/types/notifications';
|
||||
import { NotificationsState, UINotification } from '@/app/types/notifications';
|
||||
import { NotificationsService } from '@/storagenode/notifications/service';
|
||||
|
||||
export const NOTIFICATIONS_MUTATIONS = {
|
||||
SET_NOTIFICATIONS: 'SET_NOTIFICATIONS',
|
||||
@ -24,22 +20,22 @@ export const NOTIFICATIONS_ACTIONS = {
|
||||
/**
|
||||
* creates notifications module with all dependencies
|
||||
*
|
||||
* @param api - payments api
|
||||
* @param service - payments service
|
||||
*/
|
||||
export function makeNotificationsModule(api: NotificationsApi) {
|
||||
export function newNotificationsModule(service: NotificationsService) {
|
||||
return {
|
||||
state: new NotificationsState(),
|
||||
mutations: {
|
||||
[NOTIFICATIONS_MUTATIONS.SET_NOTIFICATIONS](state: NotificationsState, notificationsResponse: NotificationsState): void {
|
||||
state.notifications = notificationsResponse.notifications;
|
||||
state.pageCount = notificationsResponse.pageCount;
|
||||
state.unreadCount = notificationsResponse.unreadCount;
|
||||
[NOTIFICATIONS_MUTATIONS.SET_NOTIFICATIONS](state: NotificationsState, notificationsState: NotificationsState): void {
|
||||
state.notifications = notificationsState.notifications;
|
||||
state.pageCount = notificationsState.pageCount;
|
||||
state.unreadCount = notificationsState.unreadCount;
|
||||
},
|
||||
[NOTIFICATIONS_MUTATIONS.SET_LATEST](state: NotificationsState, notificationsResponse: NotificationsState): void {
|
||||
state.latestNotifications = notificationsResponse.notifications;
|
||||
[NOTIFICATIONS_MUTATIONS.SET_LATEST](state: NotificationsState, notificationsState: NotificationsState): void {
|
||||
state.latestNotifications = notificationsState.notifications;
|
||||
},
|
||||
[NOTIFICATIONS_MUTATIONS.MARK_AS_READ](state: NotificationsState, id: string): void {
|
||||
state.notifications = state.notifications.map((notification: Notification) => {
|
||||
state.notifications = state.notifications.map((notification: UINotification) => {
|
||||
if (notification.id === id) {
|
||||
notification.markAsRead();
|
||||
}
|
||||
@ -48,7 +44,7 @@ export function makeNotificationsModule(api: NotificationsApi) {
|
||||
});
|
||||
},
|
||||
[NOTIFICATIONS_MUTATIONS.READ_ALL](state: NotificationsState): void {
|
||||
state.notifications = state.notifications.map((notification: Notification) => {
|
||||
state.notifications = state.notifications.map((notification: UINotification) => {
|
||||
notification.markAsRead();
|
||||
|
||||
return notification;
|
||||
@ -58,24 +54,26 @@ export function makeNotificationsModule(api: NotificationsApi) {
|
||||
},
|
||||
},
|
||||
actions: {
|
||||
[NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS]: async function ({commit}: any, cursor: NotificationsCursor): Promise<NotificationsState> {
|
||||
const notificationsResponse = await api.get(cursor);
|
||||
[NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS]: async function ({commit}: any, pageIndex: number): Promise<void> {
|
||||
const notificationsResponse = await service.notifications(pageIndex);
|
||||
|
||||
commit(NOTIFICATIONS_MUTATIONS.SET_NOTIFICATIONS, notificationsResponse);
|
||||
const notifications = notificationsResponse.page.notifications.map(notification => new UINotification(notification));
|
||||
|
||||
if (cursor.page === 1) {
|
||||
commit(NOTIFICATIONS_MUTATIONS.SET_LATEST, notificationsResponse);
|
||||
const notificationState = new NotificationsState(notifications, notificationsResponse.page.pageCount, notificationsResponse.unreadCount);
|
||||
|
||||
commit(NOTIFICATIONS_MUTATIONS.SET_NOTIFICATIONS, notificationState);
|
||||
|
||||
if (pageIndex === 1) {
|
||||
commit(NOTIFICATIONS_MUTATIONS.SET_LATEST, notificationState);
|
||||
}
|
||||
|
||||
return notificationsResponse;
|
||||
},
|
||||
[NOTIFICATIONS_ACTIONS.MARK_AS_READ]: async function ({commit}: any, id: string): Promise<any> {
|
||||
await api.read(id);
|
||||
[NOTIFICATIONS_ACTIONS.MARK_AS_READ]: async function ({commit}: any, id: string): Promise<void> {
|
||||
await service.readSingeNotification(id);
|
||||
|
||||
commit(NOTIFICATIONS_MUTATIONS.MARK_AS_READ, id);
|
||||
},
|
||||
[NOTIFICATIONS_ACTIONS.READ_ALL]: async function ({commit}: any): Promise<any> {
|
||||
await api.readAll();
|
||||
[NOTIFICATIONS_ACTIONS.READ_ALL]: async function ({commit}: any): Promise<void> {
|
||||
await service.readAllNotifications();
|
||||
|
||||
commit(NOTIFICATIONS_MUTATIONS.READ_ALL);
|
||||
},
|
||||
|
||||
@ -2,23 +2,39 @@
|
||||
// See LICENSE for copying information.
|
||||
|
||||
import { NotificationIcon } from '@/app/utils/notificationIcons';
|
||||
import { Notification, NotificationTypes } from '@/storagenode/notifications/notifications';
|
||||
|
||||
/**
|
||||
* Holds all notifications module state.
|
||||
*/
|
||||
export class NotificationsState {
|
||||
public latestNotifications: UINotification[] = [];
|
||||
|
||||
public constructor(
|
||||
public notifications: UINotification[] = [],
|
||||
public pageCount: number = 0,
|
||||
public unreadCount: number = 0,
|
||||
) { }
|
||||
}
|
||||
|
||||
/**
|
||||
* Describes notification entity.
|
||||
*/
|
||||
export class Notification {
|
||||
export class UINotification {
|
||||
public icon: NotificationIcon;
|
||||
public isRead: boolean;
|
||||
public id: string;
|
||||
public senderId: string;
|
||||
public type: NotificationTypes;
|
||||
public title: string;
|
||||
public message: string;
|
||||
public readAt: Date | null;
|
||||
public createdAt: Date;
|
||||
|
||||
public constructor(
|
||||
public id: string = '',
|
||||
public senderId: string = '',
|
||||
public type: NotificationTypes = NotificationTypes.Custom,
|
||||
public title: string = '',
|
||||
public message: string = '',
|
||||
public isRead: boolean = false,
|
||||
public createdAt: Date = new Date(),
|
||||
) {
|
||||
public constructor(notification: Partial<UINotification> = new Notification()) {
|
||||
Object.assign(this, notification);
|
||||
this.setIcon();
|
||||
this.isRead = !!this.readAt;
|
||||
}
|
||||
|
||||
/**
|
||||
@ -70,60 +86,3 @@ export class Notification {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Describes all current notifications types.
|
||||
*/
|
||||
export enum NotificationTypes {
|
||||
Custom = 0,
|
||||
AuditCheckFailure = 1,
|
||||
UptimeCheckFailure = 2,
|
||||
Disqualification = 3,
|
||||
Suspension = 4,
|
||||
}
|
||||
|
||||
/**
|
||||
* Describes page offset for pagination.
|
||||
*/
|
||||
export class NotificationsCursor {
|
||||
public constructor(
|
||||
public page: number = 0,
|
||||
public limit: number = 7,
|
||||
) { }
|
||||
}
|
||||
|
||||
/**
|
||||
* Holds all notifications module state.
|
||||
*/
|
||||
export class NotificationsState {
|
||||
public latestNotifications: Notification[] = [];
|
||||
|
||||
public constructor(
|
||||
public notifications: Notification[] = [],
|
||||
public pageCount: number = 0,
|
||||
public unreadCount: number = 0,
|
||||
) { }
|
||||
}
|
||||
|
||||
/**
|
||||
* Exposes all notifications-related functionality.
|
||||
*/
|
||||
export interface NotificationsApi {
|
||||
/**
|
||||
* Fetches notifications.
|
||||
* @throws Error
|
||||
*/
|
||||
get(cursor: NotificationsCursor): Promise<NotificationsState>;
|
||||
|
||||
/**
|
||||
* Marks single notification as read.
|
||||
* @throws Error
|
||||
*/
|
||||
read(id: string): Promise<void>;
|
||||
|
||||
/**
|
||||
* Marks all notification as read.
|
||||
* @throws Error
|
||||
*/
|
||||
readAll(): Promise<void>;
|
||||
}
|
||||
|
||||
@ -20,7 +20,6 @@ import { APPSTATE_ACTIONS } from '@/app/store/modules/appState';
|
||||
import { NODE_ACTIONS } from '@/app/store/modules/node';
|
||||
import { NOTIFICATIONS_ACTIONS } from '@/app/store/modules/notifications';
|
||||
import { PAYOUT_ACTIONS } from '@/app/store/modules/payout';
|
||||
import { NotificationsCursor } from '@/app/types/notifications';
|
||||
|
||||
@Component ({
|
||||
components: {
|
||||
@ -43,7 +42,7 @@ export default class Dashboard extends Vue {
|
||||
}
|
||||
|
||||
try {
|
||||
await this.$store.dispatch(NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS, new NotificationsCursor(1));
|
||||
await this.$store.dispatch(NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS, 1);
|
||||
} catch (error) {
|
||||
console.error(error);
|
||||
}
|
||||
|
||||
@ -54,7 +54,7 @@ import VPagination from '@/app/components/VPagination.vue';
|
||||
import BackArrowIcon from '@/../static/images/notifications/backArrow.svg';
|
||||
|
||||
import { NOTIFICATIONS_ACTIONS } from '@/app/store/modules/notifications';
|
||||
import { Notification, NotificationsCursor } from '@/app/types/notifications';
|
||||
import { UINotification } from '@/app/types/notifications';
|
||||
|
||||
@Component ({
|
||||
components: {
|
||||
@ -67,7 +67,7 @@ export default class NotificationsArea extends Vue {
|
||||
/**
|
||||
* Returns notification of current page.
|
||||
*/
|
||||
public get notifications(): Notification[] {
|
||||
public get notifications(): UINotification[] {
|
||||
return this.$store.state.notificationsModule.notifications;
|
||||
}
|
||||
|
||||
@ -92,7 +92,7 @@ export default class NotificationsArea extends Vue {
|
||||
*/
|
||||
public async onPageClick(index: number): Promise<void> {
|
||||
try {
|
||||
await this.$store.dispatch(NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS, new NotificationsCursor(index));
|
||||
await this.$store.dispatch(NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS, index);
|
||||
} catch (error) {
|
||||
console.error(error.message);
|
||||
}
|
||||
|
||||
@ -57,7 +57,6 @@ import { APPSTATE_ACTIONS } from '@/app/store/modules/appState';
|
||||
import { NODE_ACTIONS } from '@/app/store/modules/node';
|
||||
import { NOTIFICATIONS_ACTIONS } from '@/app/store/modules/notifications';
|
||||
import { PAYOUT_ACTIONS } from '@/app/store/modules/payout';
|
||||
import { NotificationsCursor } from '@/app/types/notifications';
|
||||
import { PayoutPeriod, TotalHeldAndPaid } from '@/storagenode/payouts/payouts';
|
||||
|
||||
@Component ({
|
||||
@ -88,7 +87,7 @@ export default class PayoutArea extends Vue {
|
||||
}
|
||||
|
||||
try {
|
||||
await this.$store.dispatch(NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS, new NotificationsCursor(1));
|
||||
await this.$store.dispatch(NOTIFICATIONS_ACTIONS.GET_NOTIFICATIONS, 1);
|
||||
} catch (error) {
|
||||
console.error(error);
|
||||
}
|
||||
|
||||
@ -1,7 +1,12 @@
|
||||
// Copyright (C) 2019 Storj Labs, Inc.
|
||||
// See LICENSE for copying information.
|
||||
|
||||
import { Notification, NotificationsApi, NotificationsCursor, NotificationsState } from '@/app/types/notifications';
|
||||
import {
|
||||
NotificationsApi,
|
||||
NotificationsCursor,
|
||||
NotificationsPage,
|
||||
NotificationsResponse,
|
||||
} from '@/storagenode/notifications/notifications';
|
||||
import { HttpClient } from '@/storagenode/utils/httpClient';
|
||||
|
||||
/**
|
||||
@ -15,10 +20,10 @@ export class NotificationsHttpApi implements NotificationsApi {
|
||||
/**
|
||||
* Fetch notifications.
|
||||
*
|
||||
* @returns notifications state
|
||||
* @returns notifications response.
|
||||
* @throws Error
|
||||
*/
|
||||
public async get(cursor: NotificationsCursor): Promise<NotificationsState> {
|
||||
public async get(cursor: NotificationsCursor): Promise<NotificationsResponse> {
|
||||
const path = `${this.ROOT_PATH}/list?page=${cursor.page}&limit=${cursor.limit}`;
|
||||
const response = await this.client.get(path);
|
||||
|
||||
@ -27,28 +32,12 @@ export class NotificationsHttpApi implements NotificationsApi {
|
||||
}
|
||||
|
||||
const notificationResponse = await response.json();
|
||||
let notifications: Notification[] = [];
|
||||
let pageCount: number = 0;
|
||||
let unreadCount: number = 0;
|
||||
|
||||
if (notificationResponse) {
|
||||
notifications = notificationResponse.page.notifications.map(item =>
|
||||
new Notification(
|
||||
item.id,
|
||||
item.senderId,
|
||||
item.type,
|
||||
item.title,
|
||||
item.message,
|
||||
!!item.readAt,
|
||||
new Date(item.createdAt),
|
||||
),
|
||||
);
|
||||
|
||||
pageCount = notificationResponse.page.pageCount;
|
||||
unreadCount = notificationResponse.unreadCount;
|
||||
}
|
||||
|
||||
return new NotificationsState(notifications, pageCount, unreadCount);
|
||||
return new NotificationsResponse(
|
||||
new NotificationsPage(notificationResponse.page.notifications, notificationResponse.page.pageCount),
|
||||
notificationResponse.unreadCount,
|
||||
notificationResponse.totalCount,
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@ -0,0 +1,88 @@
|
||||
// Copyright (C) 2020 Storj Labs, Inc.
|
||||
// See LICENSE for copying information.
|
||||
|
||||
/**
|
||||
* Exposes all notifications-related functionality.
|
||||
*/
|
||||
export interface NotificationsApi {
|
||||
/**
|
||||
* Fetches notifications.
|
||||
* @throws Error
|
||||
*/
|
||||
get(cursor: NotificationsCursor): Promise<NotificationsResponse>;
|
||||
|
||||
/**
|
||||
* Marks single notification as read.
|
||||
* @throws Error
|
||||
*/
|
||||
read(id: string): Promise<void>;
|
||||
|
||||
/**
|
||||
* Marks all notification as read.
|
||||
* @throws Error
|
||||
*/
|
||||
readAll(): Promise<void>;
|
||||
}
|
||||
|
||||
/**
|
||||
* Describes notification entity.
|
||||
*/
|
||||
export class Notification {
|
||||
public constructor(
|
||||
public id: string = '',
|
||||
public senderId: string = '',
|
||||
public type: NotificationTypes = NotificationTypes.Custom,
|
||||
public title: string = '',
|
||||
public message: string = '',
|
||||
public readAt: Date | null = null,
|
||||
public createdAt: Date = new Date(),
|
||||
) {}
|
||||
}
|
||||
|
||||
/**
|
||||
* Describes all current notifications types.
|
||||
*/
|
||||
export enum NotificationTypes {
|
||||
Custom = 0,
|
||||
AuditCheckFailure = 1,
|
||||
UptimeCheckFailure = 2,
|
||||
Disqualification = 3,
|
||||
Suspension = 4,
|
||||
}
|
||||
|
||||
/**
|
||||
* Describes page offset for pagination.
|
||||
*/
|
||||
export class NotificationsCursor {
|
||||
private DEFAULT_LIMIT: number = 7;
|
||||
|
||||
public constructor(
|
||||
public page: number = 0,
|
||||
public limit: number = 0,
|
||||
) {
|
||||
if (!this.limit) {
|
||||
this.limit = this.DEFAULT_LIMIT;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Describes response object from server.
|
||||
*/
|
||||
export class NotificationsResponse {
|
||||
public constructor(
|
||||
public page: NotificationsPage = new NotificationsPage(),
|
||||
public unreadCount: number = 0,
|
||||
public totalCount: number = 0,
|
||||
) {}
|
||||
}
|
||||
|
||||
/**
|
||||
* Describes page related notification information.
|
||||
*/
|
||||
export class NotificationsPage {
|
||||
public constructor(
|
||||
public notifications: Notification[] = [],
|
||||
public pageCount: number = 0,
|
||||
) {}
|
||||
}
|
||||
43
web/storagenode/src/storagenode/notifications/service.ts
Normal file
43
web/storagenode/src/storagenode/notifications/service.ts
Normal file
@ -0,0 +1,43 @@
|
||||
// Copyright (C) 2020 Storj Labs, Inc.
|
||||
// See LICENSE for copying information.
|
||||
|
||||
import { NotificationsApi, NotificationsCursor, NotificationsResponse } from '@/storagenode/notifications/notifications';
|
||||
|
||||
/**
|
||||
* PayoutService is used to store and handle node paystub information.
|
||||
* PayoutService exposes a business logic related to payouts.
|
||||
*/
|
||||
export class NotificationsService {
|
||||
private readonly api: NotificationsApi;
|
||||
|
||||
public constructor(api: NotificationsApi) {
|
||||
this.api = api;
|
||||
}
|
||||
|
||||
/**
|
||||
* Fetch notifications.
|
||||
*
|
||||
* @returns notifications response.
|
||||
* @throws Error
|
||||
*/
|
||||
public async notifications(index: number, limit?: number): Promise<NotificationsResponse> {
|
||||
const cursor = new NotificationsCursor(index, limit);
|
||||
|
||||
return await this.api.get(cursor);
|
||||
}
|
||||
|
||||
/**
|
||||
* Marks single notification as read on server.
|
||||
* @param id
|
||||
*/
|
||||
public async readSingeNotification(id: string): Promise<void> {
|
||||
await this.api.read(id);
|
||||
}
|
||||
|
||||
/**
|
||||
* Marks all notifications as read on server.
|
||||
*/
|
||||
public async readAllNotifications(): Promise<void> {
|
||||
await this.api.readAll();
|
||||
}
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue
Block a user