2019-01-24 20:15:10 +00:00
|
|
|
// Copyright (C) 2019 Storj Labs, Inc.
|
captplanet (#159)
* captplanet
I kind of went overboard this weekend.
The major goal of this changeset is to provide an environment
for local development where all of the various services can
be easily run together. Developing on Storj v3 should be as
easy as running a setup command and a run command!
To do this, this changeset introduces a new tool called
captplanet, which combines the powers of the Overlay Cache,
the PointerDB, the PieceStore, Kademlia, the Minio Gateway,
etc.
Running 40 farmers and a heavy client inside the same process
forced a rethinking of the "services" that we had. To
avoid confusion by reusing prior terms, this changeset
introduces two new types: Providers and Responsibilities.
I wanted to avoid as many merge conflicts as possible, so
I left the existing Services and code for now, but if people
like this route we can clean up the duplication.
A Responsibility is a collection of gRPC methods and
corresponding state. The following systems are examples of
Responsibilities:
* Kademlia
* OverlayCache
* PointerDB
* StatDB
* PieceStore
* etc.
A Provider is a collection of Responsibilities that
share an Identity, such as:
* The heavy client
* The farmer
* The gateway
An Identity is a public/private key pair, a node id, etc.
Farmers all need different Identities, so captplanet
needs to support running multiple concurrent Providers
with different Identities.
Each Responsibility and Provider should allow for configuration
of multiple copies on its own so creating Responsibilities and
Providers use a new workflow.
To make a Responsibility, one should create a "config"
struct, such as:
```
type Config struct {
RepairThreshold int `help:"If redundancy falls below this number of
pieces, repair is triggered" default:"30"`
SuccessThreshold int `help:"If redundancy is above this number then
no additional uploads are needed" default:"40"`
}
```
To use "config" structs, this changeset introduces another
new library called 'cfgstruct', which allows for the configuration
of arbitrary structs through flagsets, and thus through cobra and
viper.
cfgstruct relies on Go's "struct tags" feature to document
help information and default values. Config structs can be
configured via cfgstruct.Bind for binding the struct to a flagset.
Because this configuration system makes setup and configuration
easier *in general*, additional commands are provided that allow
for easy standup of separate Providers. Please make sure to
check out:
* cmd/captplanet/farmer/main.go (a new farmer binary)
* cmd/captplanet/hc/main.go (a new heavy client binary)
* cmd/captplanet/gw/main.go (a new minio gateway binary)
Usage:
```
$ go install -v storj.io/storj/cmd/captplanet
$ captplanet setup
$ captplanet run
```
Configuration is placed by default in `~/.storj/capt/`
Other changes:
* introduces new config structs for currently existing
Responsibilities that conform to the new Responsibility
interface. Please see the `pkg/*/config.go` files for
examples.
* integrates the PointerDB API key with other global
configuration via flags, instead of through environment
variables through viper like it's been doing. (ultimately
this should also change to use the PointerDB config
struct but this is an okay shortterm solution).
* changes the Overlay cache to use a URL for database
configuration instead of separate redis and bolt config
settings.
* stubs out some peer identity skeleton code (but not the
meat).
* Fixes the SegmentStore to use the overlay client and
pointerdb clients instead of gRPC client code directly
* Leaves a very clear spot where we need to tie the object to
stream to segment store together. There's sort of a "golden
spike" opportunity to connect all the train tracks together
at the bottom of pkg/miniogw/config.go, labeled with a
bunch of TODOs.
Future stuff:
* I now prefer this design over the original
pkg/process.Service thing I had been pushing before (sorry!)
* The experience of trying to have multiple farmers
configurable concurrently led me to prefer config structs
over global flags (I finally came around) or using viper
directly. I think global flags are okay sometimes but in
general going forward we should try and get all relevant
config into config structs.
* If you all like this direction, I think we can go delete my
old Service interfaces and a bunch of flags and clean up a
bunch of stuff.
* If you don't like this direction, it's no sweat at all, and
despite how much code there is here I'm not very tied to any
of this! Considering a lot of this was written between midnight
and 6 am, it might not be any good!
* bind tests
2018-07-24 17:08:28 +01:00
|
|
|
// See LICENSE for copying information.
|
|
|
|
|
|
2019-04-25 09:46:32 +01:00
|
|
|
package metainfo
|
captplanet (#159)
* captplanet
I kind of went overboard this weekend.
The major goal of this changeset is to provide an environment
for local development where all of the various services can
be easily run together. Developing on Storj v3 should be as
easy as running a setup command and a run command!
To do this, this changeset introduces a new tool called
captplanet, which combines the powers of the Overlay Cache,
the PointerDB, the PieceStore, Kademlia, the Minio Gateway,
etc.
Running 40 farmers and a heavy client inside the same process
forced a rethinking of the "services" that we had. To
avoid confusion by reusing prior terms, this changeset
introduces two new types: Providers and Responsibilities.
I wanted to avoid as many merge conflicts as possible, so
I left the existing Services and code for now, but if people
like this route we can clean up the duplication.
A Responsibility is a collection of gRPC methods and
corresponding state. The following systems are examples of
Responsibilities:
* Kademlia
* OverlayCache
* PointerDB
* StatDB
* PieceStore
* etc.
A Provider is a collection of Responsibilities that
share an Identity, such as:
* The heavy client
* The farmer
* The gateway
An Identity is a public/private key pair, a node id, etc.
Farmers all need different Identities, so captplanet
needs to support running multiple concurrent Providers
with different Identities.
Each Responsibility and Provider should allow for configuration
of multiple copies on its own so creating Responsibilities and
Providers use a new workflow.
To make a Responsibility, one should create a "config"
struct, such as:
```
type Config struct {
RepairThreshold int `help:"If redundancy falls below this number of
pieces, repair is triggered" default:"30"`
SuccessThreshold int `help:"If redundancy is above this number then
no additional uploads are needed" default:"40"`
}
```
To use "config" structs, this changeset introduces another
new library called 'cfgstruct', which allows for the configuration
of arbitrary structs through flagsets, and thus through cobra and
viper.
cfgstruct relies on Go's "struct tags" feature to document
help information and default values. Config structs can be
configured via cfgstruct.Bind for binding the struct to a flagset.
Because this configuration system makes setup and configuration
easier *in general*, additional commands are provided that allow
for easy standup of separate Providers. Please make sure to
check out:
* cmd/captplanet/farmer/main.go (a new farmer binary)
* cmd/captplanet/hc/main.go (a new heavy client binary)
* cmd/captplanet/gw/main.go (a new minio gateway binary)
Usage:
```
$ go install -v storj.io/storj/cmd/captplanet
$ captplanet setup
$ captplanet run
```
Configuration is placed by default in `~/.storj/capt/`
Other changes:
* introduces new config structs for currently existing
Responsibilities that conform to the new Responsibility
interface. Please see the `pkg/*/config.go` files for
examples.
* integrates the PointerDB API key with other global
configuration via flags, instead of through environment
variables through viper like it's been doing. (ultimately
this should also change to use the PointerDB config
struct but this is an okay shortterm solution).
* changes the Overlay cache to use a URL for database
configuration instead of separate redis and bolt config
settings.
* stubs out some peer identity skeleton code (but not the
meat).
* Fixes the SegmentStore to use the overlay client and
pointerdb clients instead of gRPC client code directly
* Leaves a very clear spot where we need to tie the object to
stream to segment store together. There's sort of a "golden
spike" opportunity to connect all the train tracks together
at the bottom of pkg/miniogw/config.go, labeled with a
bunch of TODOs.
Future stuff:
* I now prefer this design over the original
pkg/process.Service thing I had been pushing before (sorry!)
* The experience of trying to have multiple farmers
configurable concurrently led me to prefer config structs
over global flags (I finally came around) or using viper
directly. I think global flags are okay sometimes but in
general going forward we should try and get all relevant
config into config structs.
* If you all like this direction, I think we can go delete my
old Service interfaces and a bunch of flags and clean up a
bunch of stuff.
* If you don't like this direction, it's no sweat at all, and
despite how much code there is here I'm not very tied to any
of this! Considering a lot of this was written between midnight
and 6 am, it might not be any good!
* bind tests
2018-07-24 17:08:28 +01:00
|
|
|
|
|
|
|
|
import (
|
2020-04-24 20:15:27 +01:00
|
|
|
"context"
|
2020-11-10 11:56:30 +00:00
|
|
|
"fmt"
|
2020-10-29 16:54:35 +00:00
|
|
|
"io"
|
2020-11-10 11:56:30 +00:00
|
|
|
"strconv"
|
|
|
|
|
"strings"
|
2019-10-01 17:55:02 +01:00
|
|
|
"time"
|
|
|
|
|
|
2019-05-14 16:13:18 +01:00
|
|
|
"go.uber.org/zap"
|
|
|
|
|
|
2019-12-27 11:48:47 +00:00
|
|
|
"storj.io/common/memory"
|
2021-03-08 13:09:32 +00:00
|
|
|
"storj.io/common/storj"
|
2020-11-30 08:39:51 +00:00
|
|
|
"storj.io/common/uuid"
|
2019-11-14 19:46:15 +00:00
|
|
|
"storj.io/storj/private/dbutil"
|
2020-10-29 16:54:35 +00:00
|
|
|
"storj.io/storj/satellite/metainfo/metabase"
|
2020-03-12 07:03:46 +00:00
|
|
|
"storj.io/storj/satellite/metainfo/piecedeletion"
|
2018-10-25 18:11:28 +01:00
|
|
|
"storj.io/storj/storage"
|
2019-12-05 17:22:27 +00:00
|
|
|
"storj.io/storj/storage/cockroachkv"
|
2018-10-25 18:11:28 +01:00
|
|
|
"storj.io/storj/storage/postgreskv"
|
captplanet (#159)
* captplanet
I kind of went overboard this weekend.
The major goal of this changeset is to provide an environment
for local development where all of the various services can
be easily run together. Developing on Storj v3 should be as
easy as running a setup command and a run command!
To do this, this changeset introduces a new tool called
captplanet, which combines the powers of the Overlay Cache,
the PointerDB, the PieceStore, Kademlia, the Minio Gateway,
etc.
Running 40 farmers and a heavy client inside the same process
forced a rethinking of the "services" that we had. To
avoid confusion by reusing prior terms, this changeset
introduces two new types: Providers and Responsibilities.
I wanted to avoid as many merge conflicts as possible, so
I left the existing Services and code for now, but if people
like this route we can clean up the duplication.
A Responsibility is a collection of gRPC methods and
corresponding state. The following systems are examples of
Responsibilities:
* Kademlia
* OverlayCache
* PointerDB
* StatDB
* PieceStore
* etc.
A Provider is a collection of Responsibilities that
share an Identity, such as:
* The heavy client
* The farmer
* The gateway
An Identity is a public/private key pair, a node id, etc.
Farmers all need different Identities, so captplanet
needs to support running multiple concurrent Providers
with different Identities.
Each Responsibility and Provider should allow for configuration
of multiple copies on its own so creating Responsibilities and
Providers use a new workflow.
To make a Responsibility, one should create a "config"
struct, such as:
```
type Config struct {
RepairThreshold int `help:"If redundancy falls below this number of
pieces, repair is triggered" default:"30"`
SuccessThreshold int `help:"If redundancy is above this number then
no additional uploads are needed" default:"40"`
}
```
To use "config" structs, this changeset introduces another
new library called 'cfgstruct', which allows for the configuration
of arbitrary structs through flagsets, and thus through cobra and
viper.
cfgstruct relies on Go's "struct tags" feature to document
help information and default values. Config structs can be
configured via cfgstruct.Bind for binding the struct to a flagset.
Because this configuration system makes setup and configuration
easier *in general*, additional commands are provided that allow
for easy standup of separate Providers. Please make sure to
check out:
* cmd/captplanet/farmer/main.go (a new farmer binary)
* cmd/captplanet/hc/main.go (a new heavy client binary)
* cmd/captplanet/gw/main.go (a new minio gateway binary)
Usage:
```
$ go install -v storj.io/storj/cmd/captplanet
$ captplanet setup
$ captplanet run
```
Configuration is placed by default in `~/.storj/capt/`
Other changes:
* introduces new config structs for currently existing
Responsibilities that conform to the new Responsibility
interface. Please see the `pkg/*/config.go` files for
examples.
* integrates the PointerDB API key with other global
configuration via flags, instead of through environment
variables through viper like it's been doing. (ultimately
this should also change to use the PointerDB config
struct but this is an okay shortterm solution).
* changes the Overlay cache to use a URL for database
configuration instead of separate redis and bolt config
settings.
* stubs out some peer identity skeleton code (but not the
meat).
* Fixes the SegmentStore to use the overlay client and
pointerdb clients instead of gRPC client code directly
* Leaves a very clear spot where we need to tie the object to
stream to segment store together. There's sort of a "golden
spike" opportunity to connect all the train tracks together
at the bottom of pkg/miniogw/config.go, labeled with a
bunch of TODOs.
Future stuff:
* I now prefer this design over the original
pkg/process.Service thing I had been pushing before (sorry!)
* The experience of trying to have multiple farmers
configurable concurrently led me to prefer config structs
over global flags (I finally came around) or using viper
directly. I think global flags are okay sometimes but in
general going forward we should try and get all relevant
config into config structs.
* If you all like this direction, I think we can go delete my
old Service interfaces and a bunch of flags and clean up a
bunch of stuff.
* If you don't like this direction, it's no sweat at all, and
despite how much code there is here I'm not very tied to any
of this! Considering a lot of this was written between midnight
and 6 am, it might not be any good!
* bind tests
2018-07-24 17:08:28 +01:00
|
|
|
)
|
|
|
|
|
|
2018-09-05 17:10:35 +01:00
|
|
|
const (
|
2020-08-11 15:50:01 +01:00
|
|
|
// BoltPointerBucket is the string representing the bucket used for `PointerEntries` in BoltDB.
|
2019-01-23 19:58:44 +00:00
|
|
|
BoltPointerBucket = "pointers"
|
2018-09-05 17:10:35 +01:00
|
|
|
)
|
|
|
|
|
|
2019-06-21 19:15:58 +01:00
|
|
|
// RSConfig is a configuration struct that keeps details about default
|
2020-07-16 15:18:02 +01:00
|
|
|
// redundancy strategy information.
|
2020-11-10 11:56:30 +00:00
|
|
|
//
|
|
|
|
|
// Can be used as a flag.
|
2019-06-21 19:15:58 +01:00
|
|
|
type RSConfig struct {
|
2020-11-10 11:56:30 +00:00
|
|
|
ErasureShareSize memory.Size
|
|
|
|
|
Min int
|
|
|
|
|
Repair int
|
|
|
|
|
Success int
|
|
|
|
|
Total int
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// Type implements pflag.Value.
|
|
|
|
|
func (RSConfig) Type() string { return "metainfo.RSConfig" }
|
|
|
|
|
|
|
|
|
|
// String is required for pflag.Value.
|
|
|
|
|
func (rs *RSConfig) String() string {
|
|
|
|
|
return fmt.Sprintf("%d/%d/%d/%d-%s",
|
|
|
|
|
rs.Min,
|
|
|
|
|
rs.Repair,
|
|
|
|
|
rs.Success,
|
|
|
|
|
rs.Total,
|
|
|
|
|
rs.ErasureShareSize.String())
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// Set sets the value from a string in the format k/m/o/n-size (min/repair/optimal/total-erasuresharesize).
|
|
|
|
|
func (rs *RSConfig) Set(s string) error {
|
|
|
|
|
// Split on dash. Expect two items. First item is RS numbers. Second item is memory.Size.
|
|
|
|
|
info := strings.Split(s, "-")
|
|
|
|
|
if len(info) != 2 {
|
|
|
|
|
return Error.New("Invalid default RS config (expect format k/m/o/n-ShareSize, got %s)", s)
|
|
|
|
|
}
|
|
|
|
|
rsNumbersString := info[0]
|
|
|
|
|
shareSizeString := info[1]
|
|
|
|
|
|
|
|
|
|
// Attempt to parse "-size" part of config.
|
|
|
|
|
shareSizeInt, err := memory.ParseString(shareSizeString)
|
|
|
|
|
if err != nil {
|
|
|
|
|
return Error.New("Invalid share size in RS config: '%s', %w", shareSizeString, err)
|
|
|
|
|
}
|
|
|
|
|
shareSize := memory.Size(shareSizeInt)
|
|
|
|
|
|
|
|
|
|
// Split on forward slash. Expect exactly four positive non-decreasing integers.
|
|
|
|
|
rsNumbers := strings.Split(rsNumbersString, "/")
|
|
|
|
|
if len(rsNumbers) != 4 {
|
|
|
|
|
return Error.New("Invalid default RS numbers (wrong size, expect 4): %s", rsNumbersString)
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
minValue := 1
|
|
|
|
|
values := []int{}
|
|
|
|
|
for _, nextValueString := range rsNumbers {
|
|
|
|
|
nextValue, err := strconv.Atoi(nextValueString)
|
|
|
|
|
if err != nil {
|
|
|
|
|
return Error.New("Invalid default RS numbers (should all be valid integers): %s, %w", rsNumbersString, err)
|
|
|
|
|
}
|
|
|
|
|
if nextValue < minValue {
|
|
|
|
|
return Error.New("Invalid default RS numbers (should be non-decreasing): %s", rsNumbersString)
|
|
|
|
|
}
|
|
|
|
|
values = append(values, nextValue)
|
|
|
|
|
minValue = nextValue
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
rs.ErasureShareSize = shareSize
|
|
|
|
|
rs.Min = values[0]
|
|
|
|
|
rs.Repair = values[1]
|
|
|
|
|
rs.Success = values[2]
|
|
|
|
|
rs.Total = values[3]
|
|
|
|
|
|
|
|
|
|
return nil
|
2019-06-21 19:15:58 +01:00
|
|
|
}
|
|
|
|
|
|
2020-07-16 15:18:02 +01:00
|
|
|
// RateLimiterConfig is a configuration struct for endpoint rate limiting.
|
2020-01-17 15:01:36 +00:00
|
|
|
type RateLimiterConfig struct {
|
2020-01-29 15:22:22 +00:00
|
|
|
Enabled bool `help:"whether rate limiting is enabled." releaseDefault:"true" devDefault:"true"`
|
|
|
|
|
Rate float64 `help:"request rate per project per second." releaseDefault:"1000" devDefault:"100"`
|
|
|
|
|
CacheCapacity int `help:"number of projects to cache." releaseDefault:"10000" devDefault:"10"`
|
|
|
|
|
CacheExpiration time.Duration `help:"how long to cache the projects limiter." releaseDefault:"10m" devDefault:"10s"`
|
2020-01-17 15:01:36 +00:00
|
|
|
}
|
|
|
|
|
|
2020-07-16 15:18:02 +01:00
|
|
|
// ProjectLimitConfig is a configuration struct for default project limits.
|
2020-06-30 22:49:29 +01:00
|
|
|
type ProjectLimitConfig struct {
|
|
|
|
|
MaxBuckets int `help:"max bucket count for a project." default:"100"`
|
2020-12-17 21:27:05 +00:00
|
|
|
DefaultMaxUsage memory.Size `help:"the default storage usage limit" default:"500.00GB"`
|
|
|
|
|
DefaultMaxBandwidth memory.Size `help:"the default bandwidth usage limit" default:"500.00GB"`
|
2020-06-30 22:49:29 +01:00
|
|
|
}
|
|
|
|
|
|
2020-07-16 15:18:02 +01:00
|
|
|
// Config is a configuration struct that is everything you need to start a metainfo.
|
captplanet (#159)
* captplanet
I kind of went overboard this weekend.
The major goal of this changeset is to provide an environment
for local development where all of the various services can
be easily run together. Developing on Storj v3 should be as
easy as running a setup command and a run command!
To do this, this changeset introduces a new tool called
captplanet, which combines the powers of the Overlay Cache,
the PointerDB, the PieceStore, Kademlia, the Minio Gateway,
etc.
Running 40 farmers and a heavy client inside the same process
forced a rethinking of the "services" that we had. To
avoid confusion by reusing prior terms, this changeset
introduces two new types: Providers and Responsibilities.
I wanted to avoid as many merge conflicts as possible, so
I left the existing Services and code for now, but if people
like this route we can clean up the duplication.
A Responsibility is a collection of gRPC methods and
corresponding state. The following systems are examples of
Responsibilities:
* Kademlia
* OverlayCache
* PointerDB
* StatDB
* PieceStore
* etc.
A Provider is a collection of Responsibilities that
share an Identity, such as:
* The heavy client
* The farmer
* The gateway
An Identity is a public/private key pair, a node id, etc.
Farmers all need different Identities, so captplanet
needs to support running multiple concurrent Providers
with different Identities.
Each Responsibility and Provider should allow for configuration
of multiple copies on its own so creating Responsibilities and
Providers use a new workflow.
To make a Responsibility, one should create a "config"
struct, such as:
```
type Config struct {
RepairThreshold int `help:"If redundancy falls below this number of
pieces, repair is triggered" default:"30"`
SuccessThreshold int `help:"If redundancy is above this number then
no additional uploads are needed" default:"40"`
}
```
To use "config" structs, this changeset introduces another
new library called 'cfgstruct', which allows for the configuration
of arbitrary structs through flagsets, and thus through cobra and
viper.
cfgstruct relies on Go's "struct tags" feature to document
help information and default values. Config structs can be
configured via cfgstruct.Bind for binding the struct to a flagset.
Because this configuration system makes setup and configuration
easier *in general*, additional commands are provided that allow
for easy standup of separate Providers. Please make sure to
check out:
* cmd/captplanet/farmer/main.go (a new farmer binary)
* cmd/captplanet/hc/main.go (a new heavy client binary)
* cmd/captplanet/gw/main.go (a new minio gateway binary)
Usage:
```
$ go install -v storj.io/storj/cmd/captplanet
$ captplanet setup
$ captplanet run
```
Configuration is placed by default in `~/.storj/capt/`
Other changes:
* introduces new config structs for currently existing
Responsibilities that conform to the new Responsibility
interface. Please see the `pkg/*/config.go` files for
examples.
* integrates the PointerDB API key with other global
configuration via flags, instead of through environment
variables through viper like it's been doing. (ultimately
this should also change to use the PointerDB config
struct but this is an okay shortterm solution).
* changes the Overlay cache to use a URL for database
configuration instead of separate redis and bolt config
settings.
* stubs out some peer identity skeleton code (but not the
meat).
* Fixes the SegmentStore to use the overlay client and
pointerdb clients instead of gRPC client code directly
* Leaves a very clear spot where we need to tie the object to
stream to segment store together. There's sort of a "golden
spike" opportunity to connect all the train tracks together
at the bottom of pkg/miniogw/config.go, labeled with a
bunch of TODOs.
Future stuff:
* I now prefer this design over the original
pkg/process.Service thing I had been pushing before (sorry!)
* The experience of trying to have multiple farmers
configurable concurrently led me to prefer config structs
over global flags (I finally came around) or using viper
directly. I think global flags are okay sometimes but in
general going forward we should try and get all relevant
config into config structs.
* If you all like this direction, I think we can go delete my
old Service interfaces and a bunch of flags and clean up a
bunch of stuff.
* If you don't like this direction, it's no sweat at all, and
despite how much code there is here I'm not very tied to any
of this! Considering a lot of this was written between midnight
and 6 am, it might not be any good!
* bind tests
2018-07-24 17:08:28 +01:00
|
|
|
type Config struct {
|
2020-11-03 12:58:27 +00:00
|
|
|
DatabaseURL string `help:"the database connection string to use" default:"postgres://"`
|
|
|
|
|
MinRemoteSegmentSize memory.Size `default:"1240" help:"minimum remote segment size"`
|
|
|
|
|
MaxInlineSegmentSize memory.Size `default:"4KiB" help:"maximum inline segment size"`
|
|
|
|
|
MaxSegmentSize memory.Size `default:"64MiB" help:"maximum segment size"`
|
|
|
|
|
MaxMetadataSize memory.Size `default:"2KiB" help:"maximum segment metadata size"`
|
|
|
|
|
MaxCommitInterval time.Duration `default:"48h" help:"maximum time allowed to pass between creating and committing a segment"`
|
|
|
|
|
Overlay bool `default:"true" help:"toggle flag if overlay is enabled"`
|
2020-11-10 12:57:27 +00:00
|
|
|
RS RSConfig `releaseDefault:"29/35/80/110-256B" devDefault:"4/6/8/10-256B" help:"redundancy scheme configuration in the format k/m/o/n-sharesize"`
|
2020-11-03 12:58:27 +00:00
|
|
|
Loop LoopConfig `help:"loop configuration"`
|
|
|
|
|
RateLimiter RateLimiterConfig `help:"rate limiter configuration"`
|
|
|
|
|
ProjectLimits ProjectLimitConfig `help:"project limit configuration"`
|
|
|
|
|
PieceDeletion piecedeletion.Config `help:"piece deletion configuration"`
|
captplanet (#159)
* captplanet
I kind of went overboard this weekend.
The major goal of this changeset is to provide an environment
for local development where all of the various services can
be easily run together. Developing on Storj v3 should be as
easy as running a setup command and a run command!
To do this, this changeset introduces a new tool called
captplanet, which combines the powers of the Overlay Cache,
the PointerDB, the PieceStore, Kademlia, the Minio Gateway,
etc.
Running 40 farmers and a heavy client inside the same process
forced a rethinking of the "services" that we had. To
avoid confusion by reusing prior terms, this changeset
introduces two new types: Providers and Responsibilities.
I wanted to avoid as many merge conflicts as possible, so
I left the existing Services and code for now, but if people
like this route we can clean up the duplication.
A Responsibility is a collection of gRPC methods and
corresponding state. The following systems are examples of
Responsibilities:
* Kademlia
* OverlayCache
* PointerDB
* StatDB
* PieceStore
* etc.
A Provider is a collection of Responsibilities that
share an Identity, such as:
* The heavy client
* The farmer
* The gateway
An Identity is a public/private key pair, a node id, etc.
Farmers all need different Identities, so captplanet
needs to support running multiple concurrent Providers
with different Identities.
Each Responsibility and Provider should allow for configuration
of multiple copies on its own so creating Responsibilities and
Providers use a new workflow.
To make a Responsibility, one should create a "config"
struct, such as:
```
type Config struct {
RepairThreshold int `help:"If redundancy falls below this number of
pieces, repair is triggered" default:"30"`
SuccessThreshold int `help:"If redundancy is above this number then
no additional uploads are needed" default:"40"`
}
```
To use "config" structs, this changeset introduces another
new library called 'cfgstruct', which allows for the configuration
of arbitrary structs through flagsets, and thus through cobra and
viper.
cfgstruct relies on Go's "struct tags" feature to document
help information and default values. Config structs can be
configured via cfgstruct.Bind for binding the struct to a flagset.
Because this configuration system makes setup and configuration
easier *in general*, additional commands are provided that allow
for easy standup of separate Providers. Please make sure to
check out:
* cmd/captplanet/farmer/main.go (a new farmer binary)
* cmd/captplanet/hc/main.go (a new heavy client binary)
* cmd/captplanet/gw/main.go (a new minio gateway binary)
Usage:
```
$ go install -v storj.io/storj/cmd/captplanet
$ captplanet setup
$ captplanet run
```
Configuration is placed by default in `~/.storj/capt/`
Other changes:
* introduces new config structs for currently existing
Responsibilities that conform to the new Responsibility
interface. Please see the `pkg/*/config.go` files for
examples.
* integrates the PointerDB API key with other global
configuration via flags, instead of through environment
variables through viper like it's been doing. (ultimately
this should also change to use the PointerDB config
struct but this is an okay shortterm solution).
* changes the Overlay cache to use a URL for database
configuration instead of separate redis and bolt config
settings.
* stubs out some peer identity skeleton code (but not the
meat).
* Fixes the SegmentStore to use the overlay client and
pointerdb clients instead of gRPC client code directly
* Leaves a very clear spot where we need to tie the object to
stream to segment store together. There's sort of a "golden
spike" opportunity to connect all the train tracks together
at the bottom of pkg/miniogw/config.go, labeled with a
bunch of TODOs.
Future stuff:
* I now prefer this design over the original
pkg/process.Service thing I had been pushing before (sorry!)
* The experience of trying to have multiple farmers
configurable concurrently led me to prefer config structs
over global flags (I finally came around) or using viper
directly. I think global flags are okay sometimes but in
general going forward we should try and get all relevant
config into config structs.
* If you all like this direction, I think we can go delete my
old Service interfaces and a bunch of flags and clean up a
bunch of stuff.
* If you don't like this direction, it's no sweat at all, and
despite how much code there is here I'm not very tied to any
of this! Considering a lot of this was written between midnight
and 6 am, it might not be any good!
* bind tests
2018-07-24 17:08:28 +01:00
|
|
|
}
|
|
|
|
|
|
2019-09-10 14:24:16 +01:00
|
|
|
// PointerDB stores pointers.
|
|
|
|
|
//
|
|
|
|
|
// architecture: Database
|
|
|
|
|
type PointerDB interface {
|
2020-04-24 20:15:27 +01:00
|
|
|
// MigrateToLatest migrates to latest schema version.
|
|
|
|
|
MigrateToLatest(ctx context.Context) error
|
|
|
|
|
|
2019-09-10 14:24:16 +01:00
|
|
|
storage.KeyValueStore
|
|
|
|
|
}
|
|
|
|
|
|
2020-10-29 08:01:53 +00:00
|
|
|
// OpenStore returns database for storing pointer data.
|
2020-12-04 10:24:39 +00:00
|
|
|
func OpenStore(ctx context.Context, logger *zap.Logger, dbURLString string, app string) (db PointerDB, err error) {
|
2019-12-05 17:22:27 +00:00
|
|
|
_, source, implementation, err := dbutil.SplitConnStr(dbURLString)
|
captplanet (#159)
* captplanet
I kind of went overboard this weekend.
The major goal of this changeset is to provide an environment
for local development where all of the various services can
be easily run together. Developing on Storj v3 should be as
easy as running a setup command and a run command!
To do this, this changeset introduces a new tool called
captplanet, which combines the powers of the Overlay Cache,
the PointerDB, the PieceStore, Kademlia, the Minio Gateway,
etc.
Running 40 farmers and a heavy client inside the same process
forced a rethinking of the "services" that we had. To
avoid confusion by reusing prior terms, this changeset
introduces two new types: Providers and Responsibilities.
I wanted to avoid as many merge conflicts as possible, so
I left the existing Services and code for now, but if people
like this route we can clean up the duplication.
A Responsibility is a collection of gRPC methods and
corresponding state. The following systems are examples of
Responsibilities:
* Kademlia
* OverlayCache
* PointerDB
* StatDB
* PieceStore
* etc.
A Provider is a collection of Responsibilities that
share an Identity, such as:
* The heavy client
* The farmer
* The gateway
An Identity is a public/private key pair, a node id, etc.
Farmers all need different Identities, so captplanet
needs to support running multiple concurrent Providers
with different Identities.
Each Responsibility and Provider should allow for configuration
of multiple copies on its own so creating Responsibilities and
Providers use a new workflow.
To make a Responsibility, one should create a "config"
struct, such as:
```
type Config struct {
RepairThreshold int `help:"If redundancy falls below this number of
pieces, repair is triggered" default:"30"`
SuccessThreshold int `help:"If redundancy is above this number then
no additional uploads are needed" default:"40"`
}
```
To use "config" structs, this changeset introduces another
new library called 'cfgstruct', which allows for the configuration
of arbitrary structs through flagsets, and thus through cobra and
viper.
cfgstruct relies on Go's "struct tags" feature to document
help information and default values. Config structs can be
configured via cfgstruct.Bind for binding the struct to a flagset.
Because this configuration system makes setup and configuration
easier *in general*, additional commands are provided that allow
for easy standup of separate Providers. Please make sure to
check out:
* cmd/captplanet/farmer/main.go (a new farmer binary)
* cmd/captplanet/hc/main.go (a new heavy client binary)
* cmd/captplanet/gw/main.go (a new minio gateway binary)
Usage:
```
$ go install -v storj.io/storj/cmd/captplanet
$ captplanet setup
$ captplanet run
```
Configuration is placed by default in `~/.storj/capt/`
Other changes:
* introduces new config structs for currently existing
Responsibilities that conform to the new Responsibility
interface. Please see the `pkg/*/config.go` files for
examples.
* integrates the PointerDB API key with other global
configuration via flags, instead of through environment
variables through viper like it's been doing. (ultimately
this should also change to use the PointerDB config
struct but this is an okay shortterm solution).
* changes the Overlay cache to use a URL for database
configuration instead of separate redis and bolt config
settings.
* stubs out some peer identity skeleton code (but not the
meat).
* Fixes the SegmentStore to use the overlay client and
pointerdb clients instead of gRPC client code directly
* Leaves a very clear spot where we need to tie the object to
stream to segment store together. There's sort of a "golden
spike" opportunity to connect all the train tracks together
at the bottom of pkg/miniogw/config.go, labeled with a
bunch of TODOs.
Future stuff:
* I now prefer this design over the original
pkg/process.Service thing I had been pushing before (sorry!)
* The experience of trying to have multiple farmers
configurable concurrently led me to prefer config structs
over global flags (I finally came around) or using viper
directly. I think global flags are okay sometimes but in
general going forward we should try and get all relevant
config into config structs.
* If you all like this direction, I think we can go delete my
old Service interfaces and a bunch of flags and clean up a
bunch of stuff.
* If you don't like this direction, it's no sweat at all, and
despite how much code there is here I'm not very tied to any
of this! Considering a lot of this was written between midnight
and 6 am, it might not be any good!
* bind tests
2018-07-24 17:08:28 +01:00
|
|
|
if err != nil {
|
2018-10-25 18:11:28 +01:00
|
|
|
return nil, err
|
captplanet (#159)
* captplanet
I kind of went overboard this weekend.
The major goal of this changeset is to provide an environment
for local development where all of the various services can
be easily run together. Developing on Storj v3 should be as
easy as running a setup command and a run command!
To do this, this changeset introduces a new tool called
captplanet, which combines the powers of the Overlay Cache,
the PointerDB, the PieceStore, Kademlia, the Minio Gateway,
etc.
Running 40 farmers and a heavy client inside the same process
forced a rethinking of the "services" that we had. To
avoid confusion by reusing prior terms, this changeset
introduces two new types: Providers and Responsibilities.
I wanted to avoid as many merge conflicts as possible, so
I left the existing Services and code for now, but if people
like this route we can clean up the duplication.
A Responsibility is a collection of gRPC methods and
corresponding state. The following systems are examples of
Responsibilities:
* Kademlia
* OverlayCache
* PointerDB
* StatDB
* PieceStore
* etc.
A Provider is a collection of Responsibilities that
share an Identity, such as:
* The heavy client
* The farmer
* The gateway
An Identity is a public/private key pair, a node id, etc.
Farmers all need different Identities, so captplanet
needs to support running multiple concurrent Providers
with different Identities.
Each Responsibility and Provider should allow for configuration
of multiple copies on its own so creating Responsibilities and
Providers use a new workflow.
To make a Responsibility, one should create a "config"
struct, such as:
```
type Config struct {
RepairThreshold int `help:"If redundancy falls below this number of
pieces, repair is triggered" default:"30"`
SuccessThreshold int `help:"If redundancy is above this number then
no additional uploads are needed" default:"40"`
}
```
To use "config" structs, this changeset introduces another
new library called 'cfgstruct', which allows for the configuration
of arbitrary structs through flagsets, and thus through cobra and
viper.
cfgstruct relies on Go's "struct tags" feature to document
help information and default values. Config structs can be
configured via cfgstruct.Bind for binding the struct to a flagset.
Because this configuration system makes setup and configuration
easier *in general*, additional commands are provided that allow
for easy standup of separate Providers. Please make sure to
check out:
* cmd/captplanet/farmer/main.go (a new farmer binary)
* cmd/captplanet/hc/main.go (a new heavy client binary)
* cmd/captplanet/gw/main.go (a new minio gateway binary)
Usage:
```
$ go install -v storj.io/storj/cmd/captplanet
$ captplanet setup
$ captplanet run
```
Configuration is placed by default in `~/.storj/capt/`
Other changes:
* introduces new config structs for currently existing
Responsibilities that conform to the new Responsibility
interface. Please see the `pkg/*/config.go` files for
examples.
* integrates the PointerDB API key with other global
configuration via flags, instead of through environment
variables through viper like it's been doing. (ultimately
this should also change to use the PointerDB config
struct but this is an okay shortterm solution).
* changes the Overlay cache to use a URL for database
configuration instead of separate redis and bolt config
settings.
* stubs out some peer identity skeleton code (but not the
meat).
* Fixes the SegmentStore to use the overlay client and
pointerdb clients instead of gRPC client code directly
* Leaves a very clear spot where we need to tie the object to
stream to segment store together. There's sort of a "golden
spike" opportunity to connect all the train tracks together
at the bottom of pkg/miniogw/config.go, labeled with a
bunch of TODOs.
Future stuff:
* I now prefer this design over the original
pkg/process.Service thing I had been pushing before (sorry!)
* The experience of trying to have multiple farmers
configurable concurrently led me to prefer config structs
over global flags (I finally came around) or using viper
directly. I think global flags are okay sometimes but in
general going forward we should try and get all relevant
config into config structs.
* If you all like this direction, I think we can go delete my
old Service interfaces and a bunch of flags and clean up a
bunch of stuff.
* If you don't like this direction, it's no sweat at all, and
despite how much code there is here I'm not very tied to any
of this! Considering a lot of this was written between midnight
and 6 am, it might not be any good!
* bind tests
2018-07-24 17:08:28 +01:00
|
|
|
}
|
2019-05-29 14:14:25 +01:00
|
|
|
|
2019-12-05 17:22:27 +00:00
|
|
|
switch implementation {
|
|
|
|
|
case dbutil.Postgres:
|
2020-12-04 10:24:39 +00:00
|
|
|
db, err = postgreskv.Open(ctx, source, app)
|
2019-12-05 17:22:27 +00:00
|
|
|
case dbutil.Cockroach:
|
2020-12-04 10:24:39 +00:00
|
|
|
db, err = cockroachkv.Open(ctx, source, app)
|
2019-05-29 14:14:25 +01:00
|
|
|
default:
|
2019-12-05 17:22:27 +00:00
|
|
|
err = Error.New("unsupported db implementation: %s", dbURLString)
|
captplanet (#159)
* captplanet
I kind of went overboard this weekend.
The major goal of this changeset is to provide an environment
for local development where all of the various services can
be easily run together. Developing on Storj v3 should be as
easy as running a setup command and a run command!
To do this, this changeset introduces a new tool called
captplanet, which combines the powers of the Overlay Cache,
the PointerDB, the PieceStore, Kademlia, the Minio Gateway,
etc.
Running 40 farmers and a heavy client inside the same process
forced a rethinking of the "services" that we had. To
avoid confusion by reusing prior terms, this changeset
introduces two new types: Providers and Responsibilities.
I wanted to avoid as many merge conflicts as possible, so
I left the existing Services and code for now, but if people
like this route we can clean up the duplication.
A Responsibility is a collection of gRPC methods and
corresponding state. The following systems are examples of
Responsibilities:
* Kademlia
* OverlayCache
* PointerDB
* StatDB
* PieceStore
* etc.
A Provider is a collection of Responsibilities that
share an Identity, such as:
* The heavy client
* The farmer
* The gateway
An Identity is a public/private key pair, a node id, etc.
Farmers all need different Identities, so captplanet
needs to support running multiple concurrent Providers
with different Identities.
Each Responsibility and Provider should allow for configuration
of multiple copies on its own so creating Responsibilities and
Providers use a new workflow.
To make a Responsibility, one should create a "config"
struct, such as:
```
type Config struct {
RepairThreshold int `help:"If redundancy falls below this number of
pieces, repair is triggered" default:"30"`
SuccessThreshold int `help:"If redundancy is above this number then
no additional uploads are needed" default:"40"`
}
```
To use "config" structs, this changeset introduces another
new library called 'cfgstruct', which allows for the configuration
of arbitrary structs through flagsets, and thus through cobra and
viper.
cfgstruct relies on Go's "struct tags" feature to document
help information and default values. Config structs can be
configured via cfgstruct.Bind for binding the struct to a flagset.
Because this configuration system makes setup and configuration
easier *in general*, additional commands are provided that allow
for easy standup of separate Providers. Please make sure to
check out:
* cmd/captplanet/farmer/main.go (a new farmer binary)
* cmd/captplanet/hc/main.go (a new heavy client binary)
* cmd/captplanet/gw/main.go (a new minio gateway binary)
Usage:
```
$ go install -v storj.io/storj/cmd/captplanet
$ captplanet setup
$ captplanet run
```
Configuration is placed by default in `~/.storj/capt/`
Other changes:
* introduces new config structs for currently existing
Responsibilities that conform to the new Responsibility
interface. Please see the `pkg/*/config.go` files for
examples.
* integrates the PointerDB API key with other global
configuration via flags, instead of through environment
variables through viper like it's been doing. (ultimately
this should also change to use the PointerDB config
struct but this is an okay shortterm solution).
* changes the Overlay cache to use a URL for database
configuration instead of separate redis and bolt config
settings.
* stubs out some peer identity skeleton code (but not the
meat).
* Fixes the SegmentStore to use the overlay client and
pointerdb clients instead of gRPC client code directly
* Leaves a very clear spot where we need to tie the object to
stream to segment store together. There's sort of a "golden
spike" opportunity to connect all the train tracks together
at the bottom of pkg/miniogw/config.go, labeled with a
bunch of TODOs.
Future stuff:
* I now prefer this design over the original
pkg/process.Service thing I had been pushing before (sorry!)
* The experience of trying to have multiple farmers
configurable concurrently led me to prefer config structs
over global flags (I finally came around) or using viper
directly. I think global flags are okay sometimes but in
general going forward we should try and get all relevant
config into config structs.
* If you all like this direction, I think we can go delete my
old Service interfaces and a bunch of flags and clean up a
bunch of stuff.
* If you don't like this direction, it's no sweat at all, and
despite how much code there is here I'm not very tied to any
of this! Considering a lot of this was written between midnight
and 6 am, it might not be any good!
* bind tests
2018-07-24 17:08:28 +01:00
|
|
|
}
|
2019-05-29 14:14:25 +01:00
|
|
|
|
2019-07-23 15:28:06 +01:00
|
|
|
if err != nil {
|
|
|
|
|
return nil, err
|
|
|
|
|
}
|
|
|
|
|
|
2019-05-14 16:13:18 +01:00
|
|
|
logger.Debug("Connected to:", zap.String("db source", source))
|
2019-07-23 15:28:06 +01:00
|
|
|
return db, nil
|
2018-10-25 18:11:28 +01:00
|
|
|
}
|
2020-10-29 16:54:35 +00:00
|
|
|
|
|
|
|
|
// MetabaseDB stores objects and segments.
|
|
|
|
|
type MetabaseDB interface {
|
|
|
|
|
io.Closer
|
2021-02-17 10:46:44 +00:00
|
|
|
|
2020-10-29 16:54:35 +00:00
|
|
|
// MigrateToLatest migrates to latest schema version.
|
|
|
|
|
MigrateToLatest(ctx context.Context) error
|
2021-02-17 10:46:44 +00:00
|
|
|
// CheckVersion checks the database is the correct version
|
|
|
|
|
CheckVersion(ctx context.Context) error
|
|
|
|
|
|
2021-01-11 10:08:18 +00:00
|
|
|
// DeleteObjectAnyStatusAllVersions deletes all object versions.
|
|
|
|
|
DeleteObjectAnyStatusAllVersions(ctx context.Context, opts metabase.DeleteObjectAnyStatusAllVersions) (result metabase.DeleteObjectResult, err error)
|
2020-11-03 12:58:27 +00:00
|
|
|
// DeleteObjectsAllVersions deletes all versions of multiple objects from the same bucket.
|
|
|
|
|
DeleteObjectsAllVersions(ctx context.Context, opts metabase.DeleteObjectsAllVersions) (result metabase.DeleteObjectResult, err error)
|
2020-12-03 18:04:01 +00:00
|
|
|
// DeletePendingObject deletes a pending object.
|
|
|
|
|
DeletePendingObject(ctx context.Context, opts metabase.DeletePendingObject) (result metabase.DeleteObjectResult, err error)
|
2020-12-09 12:24:37 +00:00
|
|
|
// DeleteBucketObjects deletes all objects in the specified bucket.
|
|
|
|
|
DeleteBucketObjects(ctx context.Context, opts metabase.DeleteBucketObjects) (deletedObjectCount int64, err error)
|
2020-11-30 12:33:06 +00:00
|
|
|
// DeleteExpiredObjects deletes all objects that expired before expiredBefore.
|
2021-03-16 11:54:40 +00:00
|
|
|
DeleteExpiredObjects(ctx context.Context, opts metabase.DeleteExpiredObjects) error
|
2020-12-10 20:49:23 +00:00
|
|
|
// DeleteObjectLatestVersion deletes latest object version.
|
|
|
|
|
DeleteObjectLatestVersion(ctx context.Context, opts metabase.DeleteObjectLatestVersion) (result metabase.DeleteObjectResult, err error)
|
2020-11-06 11:54:52 +00:00
|
|
|
// BeginObjectExactVersion adds a pending object to the database, with specific version.
|
2021-01-12 11:29:13 +00:00
|
|
|
BeginObjectExactVersion(ctx context.Context, opts metabase.BeginObjectExactVersion) (committed metabase.Object, err error)
|
2020-11-06 11:54:52 +00:00
|
|
|
// CommitObject adds a pending object to the database.
|
|
|
|
|
CommitObject(ctx context.Context, opts metabase.CommitObject) (object metabase.Object, err error)
|
|
|
|
|
// BeginSegment verifies whether a new segment upload can be started.
|
|
|
|
|
BeginSegment(ctx context.Context, opts metabase.BeginSegment) (err error)
|
|
|
|
|
// CommitSegment commits segment to the database.
|
|
|
|
|
CommitSegment(ctx context.Context, opts metabase.CommitSegment) (err error)
|
2020-11-11 10:54:10 +00:00
|
|
|
// CommitInlineSegment commits inline segment to the database.
|
|
|
|
|
CommitInlineSegment(ctx context.Context, opts metabase.CommitInlineSegment) (err error)
|
2020-11-06 11:54:52 +00:00
|
|
|
// GetObjectLatestVersion returns object information for latest version.
|
|
|
|
|
GetObjectLatestVersion(ctx context.Context, opts metabase.GetObjectLatestVersion) (_ metabase.Object, err error)
|
2021-01-29 15:28:19 +00:00
|
|
|
// GetSegmentByLocation returns a information about segment on the specified location.
|
|
|
|
|
GetSegmentByLocation(ctx context.Context, opts metabase.GetSegmentByLocation) (segment metabase.Segment, err error)
|
2020-11-06 11:54:52 +00:00
|
|
|
// GetSegmentByPosition returns a information about segment which covers specified offset.
|
|
|
|
|
GetSegmentByPosition(ctx context.Context, opts metabase.GetSegmentByPosition) (segment metabase.Segment, err error)
|
|
|
|
|
// GetLatestObjectLastSegment returns an object last segment information.
|
|
|
|
|
GetLatestObjectLastSegment(ctx context.Context, opts metabase.GetLatestObjectLastSegment) (segment metabase.Segment, err error)
|
2021-03-08 13:09:32 +00:00
|
|
|
// GetStreamPieceCountByNodeID returns piece count by node id.
|
|
|
|
|
GetStreamPieceCountByNodeID(ctx context.Context, opts metabase.GetStreamPieceCountByNodeID) (result map[storj.NodeID]int64, err error)
|
2020-11-10 15:51:37 +00:00
|
|
|
// ListSegments lists specified stream segments.
|
|
|
|
|
ListSegments(ctx context.Context, opts metabase.ListSegments) (result metabase.ListSegmentsResult, err error)
|
2021-03-08 12:10:33 +00:00
|
|
|
// ListStreamPositions lists specified stream segment positions.
|
|
|
|
|
ListStreamPositions(ctx context.Context, opts metabase.ListStreamPositions) (result metabase.ListStreamPositionsResult, err error)
|
2020-12-21 16:43:08 +00:00
|
|
|
// IterateObjectsAllVersions iterates through all versions of all objects.
|
|
|
|
|
IterateObjectsAllVersions(ctx context.Context, opts metabase.IterateObjects, fn func(context.Context, metabase.ObjectsIterator) error) (err error)
|
2020-12-21 15:07:00 +00:00
|
|
|
// IterateObjectsAllVersionsWithStatus iterates through all versions of all objects with specified status.
|
|
|
|
|
IterateObjectsAllVersionsWithStatus(ctx context.Context, opts metabase.IterateObjectsWithStatus, fn func(context.Context, metabase.ObjectsIterator) error) (err error)
|
2021-01-11 12:06:04 +00:00
|
|
|
// IteratePendingObjectsByKey iterates through all StreamID for a given ObjectKey.
|
|
|
|
|
IteratePendingObjectsByKey(ctx context.Context, opts metabase.IteratePendingObjectsByKey, fn func(context.Context, metabase.ObjectsIterator) error) (err error)
|
2021-03-01 14:29:03 +00:00
|
|
|
// IterateLoopObjects iterates through all objects in metabase for metainfo loop purpose.
|
|
|
|
|
IterateLoopObjects(ctx context.Context, opts metabase.IterateLoopObjects, fn func(context.Context, metabase.LoopObjectsIterator) error) (err error)
|
2021-03-01 15:27:04 +00:00
|
|
|
// IterateLoopStreams iterates through all streams passed in as arguments.
|
|
|
|
|
IterateLoopStreams(ctx context.Context, opts metabase.IterateLoopStreams, handleStream func(ctx context.Context, streamID uuid.UUID, next metabase.SegmentIterator) error) (err error)
|
2020-11-17 17:37:58 +00:00
|
|
|
// BucketEmpty returns true if bucket does not contain objects (pending or committed).
|
|
|
|
|
// This method doesn't check bucket existence.
|
|
|
|
|
BucketEmpty(ctx context.Context, opts metabase.BucketEmpty) (empty bool, err error)
|
2020-12-14 12:54:22 +00:00
|
|
|
// UpdateSegmentPieces updates pieces for specified segment. If provided old pieces won't match current database state update will fail.
|
2020-12-14 14:29:48 +00:00
|
|
|
UpdateSegmentPieces(ctx context.Context, opts metabase.UpdateSegmentPieces) (err error)
|
2021-02-09 12:06:18 +00:00
|
|
|
// EnsureNodeAliases ensures that the supplied node ID-s have a alias.
|
|
|
|
|
// It's safe to concurrently try and create node ID-s for the same NodeID.
|
|
|
|
|
EnsureNodeAliases(ctx context.Context, opts metabase.EnsureNodeAliases) (err error)
|
|
|
|
|
// ListNodeAliases lists all node alias mappings.
|
|
|
|
|
ListNodeAliases(ctx context.Context) (_ []metabase.NodeAliasEntry, err error)
|
2020-11-05 13:30:39 +00:00
|
|
|
|
2020-12-03 18:04:01 +00:00
|
|
|
// TestingAllCommittedObjects gets all committed objects from bucket. Use only for testing purposes.
|
2020-11-30 08:39:51 +00:00
|
|
|
TestingAllCommittedObjects(ctx context.Context, projectID uuid.UUID, bucketName string) (objects []metabase.ObjectEntry, err error)
|
2020-12-03 18:04:01 +00:00
|
|
|
// TestingAllPendingObjects gets all pending objects from bucket. Use only for testing purposes.
|
|
|
|
|
TestingAllPendingObjects(ctx context.Context, projectID uuid.UUID, bucketName string) (objects []metabase.ObjectEntry, err error)
|
2020-11-30 08:39:51 +00:00
|
|
|
// TestingAllObjectSegments gets all segments for given object. Use only for testing purposes.
|
|
|
|
|
TestingAllObjectSegments(ctx context.Context, objectLocation metabase.ObjectLocation) (segments []metabase.Segment, err error)
|
2020-12-15 12:58:57 +00:00
|
|
|
// TestingAllObjects gets all objects. Use only for testing purposes.
|
|
|
|
|
TestingAllObjects(ctx context.Context) (segments []metabase.Object, err error)
|
|
|
|
|
// TestingAllSegments gets all segments. Use only for testing purposes.
|
|
|
|
|
TestingAllSegments(ctx context.Context) (segments []metabase.Segment, err error)
|
|
|
|
|
|
2020-11-05 13:30:39 +00:00
|
|
|
// InternalImplementation returns *metabase.DB.
|
|
|
|
|
// TODO: remove.
|
|
|
|
|
InternalImplementation() interface{}
|
2020-10-29 16:54:35 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// OpenMetabase returns database for storing objects and segments.
|
2020-11-05 13:30:39 +00:00
|
|
|
func OpenMetabase(ctx context.Context, log *zap.Logger, dbURLString string) (db MetabaseDB, err error) {
|
2020-10-29 16:54:35 +00:00
|
|
|
_, source, implementation, err := dbutil.SplitConnStr(dbURLString)
|
|
|
|
|
if err != nil {
|
|
|
|
|
return nil, err
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
switch implementation {
|
|
|
|
|
case dbutil.Postgres:
|
2020-11-05 13:30:39 +00:00
|
|
|
db, err = metabase.Open(ctx, log, "pgx", dbURLString)
|
2020-10-29 16:54:35 +00:00
|
|
|
case dbutil.Cockroach:
|
2020-11-05 13:30:39 +00:00
|
|
|
db, err = metabase.Open(ctx, log, "cockroach", dbURLString)
|
2020-10-29 16:54:35 +00:00
|
|
|
default:
|
|
|
|
|
err = Error.New("unsupported db implementation: %s", dbURLString)
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
if err != nil {
|
|
|
|
|
return nil, err
|
|
|
|
|
}
|
|
|
|
|
|
2020-11-05 13:30:39 +00:00
|
|
|
log.Debug("Connected to:", zap.String("db source", source))
|
2020-10-29 16:54:35 +00:00
|
|
|
return db, nil
|
|
|
|
|
}
|
