2019-01-24 20:15:10 +00:00
// Copyright (C) 2019 Storj Labs, Inc.
captplanet (#159)
* captplanet
I kind of went overboard this weekend.
The major goal of this changeset is to provide an environment
for local development where all of the various services can
be easily run together. Developing on Storj v3 should be as
easy as running a setup command and a run command!
To do this, this changeset introduces a new tool called
captplanet, which combines the powers of the Overlay Cache,
the PointerDB, the PieceStore, Kademlia, the Minio Gateway,
etc.
Running 40 farmers and a heavy client inside the same process
forced a rethinking of the "services" that we had. To
avoid confusion by reusing prior terms, this changeset
introduces two new types: Providers and Responsibilities.
I wanted to avoid as many merge conflicts as possible, so
I left the existing Services and code for now, but if people
like this route we can clean up the duplication.
A Responsibility is a collection of gRPC methods and
corresponding state. The following systems are examples of
Responsibilities:
* Kademlia
* OverlayCache
* PointerDB
* StatDB
* PieceStore
* etc.
A Provider is a collection of Responsibilities that
share an Identity, such as:
* The heavy client
* The farmer
* The gateway
An Identity is a public/private key pair, a node id, etc.
Farmers all need different Identities, so captplanet
needs to support running multiple concurrent Providers
with different Identities.
Each Responsibility and Provider should allow for configuration
of multiple copies on its own so creating Responsibilities and
Providers use a new workflow.
To make a Responsibility, one should create a "config"
struct, such as:
```
type Config struct {
RepairThreshold int `help:"If redundancy falls below this number of
pieces, repair is triggered" default:"30"`
SuccessThreshold int `help:"If redundancy is above this number then
no additional uploads are needed" default:"40"`
}
```
To use "config" structs, this changeset introduces another
new library called 'cfgstruct', which allows for the configuration
of arbitrary structs through flagsets, and thus through cobra and
viper.
cfgstruct relies on Go's "struct tags" feature to document
help information and default values. Config structs can be
configured via cfgstruct.Bind for binding the struct to a flagset.
Because this configuration system makes setup and configuration
easier *in general*, additional commands are provided that allow
for easy standup of separate Providers. Please make sure to
check out:
* cmd/captplanet/farmer/main.go (a new farmer binary)
* cmd/captplanet/hc/main.go (a new heavy client binary)
* cmd/captplanet/gw/main.go (a new minio gateway binary)
Usage:
```
$ go install -v storj.io/storj/cmd/captplanet
$ captplanet setup
$ captplanet run
```
Configuration is placed by default in `~/.storj/capt/`
Other changes:
* introduces new config structs for currently existing
Responsibilities that conform to the new Responsibility
interface. Please see the `pkg/*/config.go` files for
examples.
* integrates the PointerDB API key with other global
configuration via flags, instead of through environment
variables through viper like it's been doing. (ultimately
this should also change to use the PointerDB config
struct but this is an okay shortterm solution).
* changes the Overlay cache to use a URL for database
configuration instead of separate redis and bolt config
settings.
* stubs out some peer identity skeleton code (but not the
meat).
* Fixes the SegmentStore to use the overlay client and
pointerdb clients instead of gRPC client code directly
* Leaves a very clear spot where we need to tie the object to
stream to segment store together. There's sort of a "golden
spike" opportunity to connect all the train tracks together
at the bottom of pkg/miniogw/config.go, labeled with a
bunch of TODOs.
Future stuff:
* I now prefer this design over the original
pkg/process.Service thing I had been pushing before (sorry!)
* The experience of trying to have multiple farmers
configurable concurrently led me to prefer config structs
over global flags (I finally came around) or using viper
directly. I think global flags are okay sometimes but in
general going forward we should try and get all relevant
config into config structs.
* If you all like this direction, I think we can go delete my
old Service interfaces and a bunch of flags and clean up a
bunch of stuff.
* If you don't like this direction, it's no sweat at all, and
despite how much code there is here I'm not very tied to any
of this! Considering a lot of this was written between midnight
and 6 am, it might not be any good!
* bind tests
2018-07-24 17:08:28 +01:00
// See LICENSE for copying information.
2019-04-25 09:46:32 +01:00
package metainfo
captplanet (#159)
* captplanet
I kind of went overboard this weekend.
The major goal of this changeset is to provide an environment
for local development where all of the various services can
be easily run together. Developing on Storj v3 should be as
easy as running a setup command and a run command!
To do this, this changeset introduces a new tool called
captplanet, which combines the powers of the Overlay Cache,
the PointerDB, the PieceStore, Kademlia, the Minio Gateway,
etc.
Running 40 farmers and a heavy client inside the same process
forced a rethinking of the "services" that we had. To
avoid confusion by reusing prior terms, this changeset
introduces two new types: Providers and Responsibilities.
I wanted to avoid as many merge conflicts as possible, so
I left the existing Services and code for now, but if people
like this route we can clean up the duplication.
A Responsibility is a collection of gRPC methods and
corresponding state. The following systems are examples of
Responsibilities:
* Kademlia
* OverlayCache
* PointerDB
* StatDB
* PieceStore
* etc.
A Provider is a collection of Responsibilities that
share an Identity, such as:
* The heavy client
* The farmer
* The gateway
An Identity is a public/private key pair, a node id, etc.
Farmers all need different Identities, so captplanet
needs to support running multiple concurrent Providers
with different Identities.
Each Responsibility and Provider should allow for configuration
of multiple copies on its own so creating Responsibilities and
Providers use a new workflow.
To make a Responsibility, one should create a "config"
struct, such as:
```
type Config struct {
RepairThreshold int `help:"If redundancy falls below this number of
pieces, repair is triggered" default:"30"`
SuccessThreshold int `help:"If redundancy is above this number then
no additional uploads are needed" default:"40"`
}
```
To use "config" structs, this changeset introduces another
new library called 'cfgstruct', which allows for the configuration
of arbitrary structs through flagsets, and thus through cobra and
viper.
cfgstruct relies on Go's "struct tags" feature to document
help information and default values. Config structs can be
configured via cfgstruct.Bind for binding the struct to a flagset.
Because this configuration system makes setup and configuration
easier *in general*, additional commands are provided that allow
for easy standup of separate Providers. Please make sure to
check out:
* cmd/captplanet/farmer/main.go (a new farmer binary)
* cmd/captplanet/hc/main.go (a new heavy client binary)
* cmd/captplanet/gw/main.go (a new minio gateway binary)
Usage:
```
$ go install -v storj.io/storj/cmd/captplanet
$ captplanet setup
$ captplanet run
```
Configuration is placed by default in `~/.storj/capt/`
Other changes:
* introduces new config structs for currently existing
Responsibilities that conform to the new Responsibility
interface. Please see the `pkg/*/config.go` files for
examples.
* integrates the PointerDB API key with other global
configuration via flags, instead of through environment
variables through viper like it's been doing. (ultimately
this should also change to use the PointerDB config
struct but this is an okay shortterm solution).
* changes the Overlay cache to use a URL for database
configuration instead of separate redis and bolt config
settings.
* stubs out some peer identity skeleton code (but not the
meat).
* Fixes the SegmentStore to use the overlay client and
pointerdb clients instead of gRPC client code directly
* Leaves a very clear spot where we need to tie the object to
stream to segment store together. There's sort of a "golden
spike" opportunity to connect all the train tracks together
at the bottom of pkg/miniogw/config.go, labeled with a
bunch of TODOs.
Future stuff:
* I now prefer this design over the original
pkg/process.Service thing I had been pushing before (sorry!)
* The experience of trying to have multiple farmers
configurable concurrently led me to prefer config structs
over global flags (I finally came around) or using viper
directly. I think global flags are okay sometimes but in
general going forward we should try and get all relevant
config into config structs.
* If you all like this direction, I think we can go delete my
old Service interfaces and a bunch of flags and clean up a
bunch of stuff.
* If you don't like this direction, it's no sweat at all, and
despite how much code there is here I'm not very tied to any
of this! Considering a lot of this was written between midnight
and 6 am, it might not be any good!
* bind tests
2018-07-24 17:08:28 +01:00
import (
2020-11-10 11:56:30 +00:00
"fmt"
"strconv"
"strings"
2019-10-01 17:55:02 +01:00
"time"
2019-12-27 11:48:47 +00:00
"storj.io/common/memory"
2023-08-04 14:25:36 +01:00
"storj.io/common/uuid"
2022-10-28 15:56:20 +01:00
"storj.io/storj/satellite/metabase"
2022-04-07 15:10:23 +01:00
"storj.io/uplink/private/eestream"
captplanet (#159)
* captplanet
I kind of went overboard this weekend.
The major goal of this changeset is to provide an environment
for local development where all of the various services can
be easily run together. Developing on Storj v3 should be as
easy as running a setup command and a run command!
To do this, this changeset introduces a new tool called
captplanet, which combines the powers of the Overlay Cache,
the PointerDB, the PieceStore, Kademlia, the Minio Gateway,
etc.
Running 40 farmers and a heavy client inside the same process
forced a rethinking of the "services" that we had. To
avoid confusion by reusing prior terms, this changeset
introduces two new types: Providers and Responsibilities.
I wanted to avoid as many merge conflicts as possible, so
I left the existing Services and code for now, but if people
like this route we can clean up the duplication.
A Responsibility is a collection of gRPC methods and
corresponding state. The following systems are examples of
Responsibilities:
* Kademlia
* OverlayCache
* PointerDB
* StatDB
* PieceStore
* etc.
A Provider is a collection of Responsibilities that
share an Identity, such as:
* The heavy client
* The farmer
* The gateway
An Identity is a public/private key pair, a node id, etc.
Farmers all need different Identities, so captplanet
needs to support running multiple concurrent Providers
with different Identities.
Each Responsibility and Provider should allow for configuration
of multiple copies on its own so creating Responsibilities and
Providers use a new workflow.
To make a Responsibility, one should create a "config"
struct, such as:
```
type Config struct {
RepairThreshold int `help:"If redundancy falls below this number of
pieces, repair is triggered" default:"30"`
SuccessThreshold int `help:"If redundancy is above this number then
no additional uploads are needed" default:"40"`
}
```
To use "config" structs, this changeset introduces another
new library called 'cfgstruct', which allows for the configuration
of arbitrary structs through flagsets, and thus through cobra and
viper.
cfgstruct relies on Go's "struct tags" feature to document
help information and default values. Config structs can be
configured via cfgstruct.Bind for binding the struct to a flagset.
Because this configuration system makes setup and configuration
easier *in general*, additional commands are provided that allow
for easy standup of separate Providers. Please make sure to
check out:
* cmd/captplanet/farmer/main.go (a new farmer binary)
* cmd/captplanet/hc/main.go (a new heavy client binary)
* cmd/captplanet/gw/main.go (a new minio gateway binary)
Usage:
```
$ go install -v storj.io/storj/cmd/captplanet
$ captplanet setup
$ captplanet run
```
Configuration is placed by default in `~/.storj/capt/`
Other changes:
* introduces new config structs for currently existing
Responsibilities that conform to the new Responsibility
interface. Please see the `pkg/*/config.go` files for
examples.
* integrates the PointerDB API key with other global
configuration via flags, instead of through environment
variables through viper like it's been doing. (ultimately
this should also change to use the PointerDB config
struct but this is an okay shortterm solution).
* changes the Overlay cache to use a URL for database
configuration instead of separate redis and bolt config
settings.
* stubs out some peer identity skeleton code (but not the
meat).
* Fixes the SegmentStore to use the overlay client and
pointerdb clients instead of gRPC client code directly
* Leaves a very clear spot where we need to tie the object to
stream to segment store together. There's sort of a "golden
spike" opportunity to connect all the train tracks together
at the bottom of pkg/miniogw/config.go, labeled with a
bunch of TODOs.
Future stuff:
* I now prefer this design over the original
pkg/process.Service thing I had been pushing before (sorry!)
* The experience of trying to have multiple farmers
configurable concurrently led me to prefer config structs
over global flags (I finally came around) or using viper
directly. I think global flags are okay sometimes but in
general going forward we should try and get all relevant
config into config structs.
* If you all like this direction, I think we can go delete my
old Service interfaces and a bunch of flags and clean up a
bunch of stuff.
* If you don't like this direction, it's no sweat at all, and
despite how much code there is here I'm not very tied to any
of this! Considering a lot of this was written between midnight
and 6 am, it might not be any good!
* bind tests
2018-07-24 17:08:28 +01:00
)
2018-09-05 17:10:35 +01:00
const (
2020-08-11 15:50:01 +01:00
// BoltPointerBucket is the string representing the bucket used for `PointerEntries` in BoltDB.
2019-01-23 19:58:44 +00:00
BoltPointerBucket = "pointers"
2018-09-05 17:10:35 +01:00
)
2019-06-21 19:15:58 +01:00
// RSConfig is a configuration struct that keeps details about default
2020-07-16 15:18:02 +01:00
// redundancy strategy information.
2020-11-10 11:56:30 +00:00
//
// Can be used as a flag.
2019-06-21 19:15:58 +01:00
type RSConfig struct {
2020-11-10 11:56:30 +00:00
ErasureShareSize memory . Size
Min int
Repair int
Success int
Total int
}
// Type implements pflag.Value.
func ( RSConfig ) Type ( ) string { return "metainfo.RSConfig" }
// String is required for pflag.Value.
func ( rs * RSConfig ) String ( ) string {
return fmt . Sprintf ( "%d/%d/%d/%d-%s" ,
rs . Min ,
rs . Repair ,
rs . Success ,
rs . Total ,
rs . ErasureShareSize . String ( ) )
}
// Set sets the value from a string in the format k/m/o/n-size (min/repair/optimal/total-erasuresharesize).
func ( rs * RSConfig ) Set ( s string ) error {
// Split on dash. Expect two items. First item is RS numbers. Second item is memory.Size.
info := strings . Split ( s , "-" )
if len ( info ) != 2 {
return Error . New ( "Invalid default RS config (expect format k/m/o/n-ShareSize, got %s)" , s )
}
rsNumbersString := info [ 0 ]
shareSizeString := info [ 1 ]
// Attempt to parse "-size" part of config.
shareSizeInt , err := memory . ParseString ( shareSizeString )
if err != nil {
return Error . New ( "Invalid share size in RS config: '%s', %w" , shareSizeString , err )
}
shareSize := memory . Size ( shareSizeInt )
// Split on forward slash. Expect exactly four positive non-decreasing integers.
rsNumbers := strings . Split ( rsNumbersString , "/" )
if len ( rsNumbers ) != 4 {
return Error . New ( "Invalid default RS numbers (wrong size, expect 4): %s" , rsNumbersString )
}
minValue := 1
values := [ ] int { }
for _ , nextValueString := range rsNumbers {
nextValue , err := strconv . Atoi ( nextValueString )
if err != nil {
return Error . New ( "Invalid default RS numbers (should all be valid integers): %s, %w" , rsNumbersString , err )
}
if nextValue < minValue {
return Error . New ( "Invalid default RS numbers (should be non-decreasing): %s" , rsNumbersString )
}
values = append ( values , nextValue )
minValue = nextValue
}
rs . ErasureShareSize = shareSize
rs . Min = values [ 0 ]
rs . Repair = values [ 1 ]
rs . Success = values [ 2 ]
rs . Total = values [ 3 ]
return nil
2019-06-21 19:15:58 +01:00
}
2022-04-07 15:10:23 +01:00
// RedundancyStrategy creates eestream.RedundancyStrategy from config values.
func ( rs * RSConfig ) RedundancyStrategy ( ) ( eestream . RedundancyStrategy , error ) {
2023-10-31 13:05:32 +00:00
fec , err := eestream . NewFEC ( rs . Min , rs . Total )
2022-04-07 15:10:23 +01:00
if err != nil {
return eestream . RedundancyStrategy { } , err
}
erasureScheme := eestream . NewRSScheme ( fec , rs . ErasureShareSize . Int ( ) )
return eestream . NewRedundancyStrategy ( erasureScheme , rs . Repair , rs . Success )
}
2020-07-16 15:18:02 +01:00
// RateLimiterConfig is a configuration struct for endpoint rate limiting.
2020-01-17 15:01:36 +00:00
type RateLimiterConfig struct {
2020-01-29 15:22:22 +00:00
Enabled bool ` help:"whether rate limiting is enabled." releaseDefault:"true" devDefault:"true" `
2022-01-21 22:37:59 +00:00
Rate float64 ` help:"request rate per project per second." releaseDefault:"100" devDefault:"100" testDefault:"1000" `
testplanet/satellite: reduce the number of places default values need to be configured
Satellites set their configuration values to default values using
cfgstruct, however, it turns out our tests don't test these values
at all! Instead, they have a completely separate definition system
that is easy to forget about.
As is to be expected, these values have drifted, and it appears
in a few cases test planet is testing unreasonable values that we
won't see in production, or perhaps worse, features enabled in
production were missed and weren't enabled in testplanet.
This change makes it so all values are configured the same,
systematic way, so it's easy to see when test values are different
than dev values or release values, and it's less hard to forget
to enable features in testplanet.
In terms of reviewing, this change should be actually fairly
easy to review, considering private/testplanet/satellite.go keeps
the current config system and the new one and confirms that they
result in identical configurations, so you can be certain that
nothing was missed and the config is all correct.
You can also check the config lock to see what actual config
values changed.
Change-Id: I6715d0794887f577e21742afcf56fd2b9d12170e
2021-05-31 22:15:00 +01:00
CacheCapacity int ` help:"number of projects to cache." releaseDefault:"10000" devDefault:"10" testDefault:"100" `
2020-01-29 15:22:22 +00:00
CacheExpiration time . Duration ` help:"how long to cache the projects limiter." releaseDefault:"10m" devDefault:"10s" `
2020-01-17 15:01:36 +00:00
}
2023-03-29 16:12:16 +01:00
// UploadLimiterConfig is a configuration struct for endpoint upload limiting.
type UploadLimiterConfig struct {
Enabled bool ` help:"whether rate limiting is enabled." releaseDefault:"true" devDefault:"true" `
SingleObjectLimit time . Duration ` help:"how often we can upload to the single object (the same location) per API instance" default:"1s" devDefault:"1ms" `
CacheCapacity int ` help:"number of object locations to cache." releaseDefault:"10000" devDefault:"10" testDefault:"100" `
}
2020-07-16 15:18:02 +01:00
// ProjectLimitConfig is a configuration struct for default project limits.
2020-06-30 22:49:29 +01:00
type ProjectLimitConfig struct {
2022-07-04 19:44:58 +01:00
MaxBuckets int ` help:"max bucket count for a project." default:"100" testDefault:"10" `
2020-06-30 22:49:29 +01:00
}
2020-07-16 15:18:02 +01:00
// Config is a configuration struct that is everything you need to start a metainfo.
captplanet (#159)
* captplanet
I kind of went overboard this weekend.
The major goal of this changeset is to provide an environment
for local development where all of the various services can
be easily run together. Developing on Storj v3 should be as
easy as running a setup command and a run command!
To do this, this changeset introduces a new tool called
captplanet, which combines the powers of the Overlay Cache,
the PointerDB, the PieceStore, Kademlia, the Minio Gateway,
etc.
Running 40 farmers and a heavy client inside the same process
forced a rethinking of the "services" that we had. To
avoid confusion by reusing prior terms, this changeset
introduces two new types: Providers and Responsibilities.
I wanted to avoid as many merge conflicts as possible, so
I left the existing Services and code for now, but if people
like this route we can clean up the duplication.
A Responsibility is a collection of gRPC methods and
corresponding state. The following systems are examples of
Responsibilities:
* Kademlia
* OverlayCache
* PointerDB
* StatDB
* PieceStore
* etc.
A Provider is a collection of Responsibilities that
share an Identity, such as:
* The heavy client
* The farmer
* The gateway
An Identity is a public/private key pair, a node id, etc.
Farmers all need different Identities, so captplanet
needs to support running multiple concurrent Providers
with different Identities.
Each Responsibility and Provider should allow for configuration
of multiple copies on its own so creating Responsibilities and
Providers use a new workflow.
To make a Responsibility, one should create a "config"
struct, such as:
```
type Config struct {
RepairThreshold int `help:"If redundancy falls below this number of
pieces, repair is triggered" default:"30"`
SuccessThreshold int `help:"If redundancy is above this number then
no additional uploads are needed" default:"40"`
}
```
To use "config" structs, this changeset introduces another
new library called 'cfgstruct', which allows for the configuration
of arbitrary structs through flagsets, and thus through cobra and
viper.
cfgstruct relies on Go's "struct tags" feature to document
help information and default values. Config structs can be
configured via cfgstruct.Bind for binding the struct to a flagset.
Because this configuration system makes setup and configuration
easier *in general*, additional commands are provided that allow
for easy standup of separate Providers. Please make sure to
check out:
* cmd/captplanet/farmer/main.go (a new farmer binary)
* cmd/captplanet/hc/main.go (a new heavy client binary)
* cmd/captplanet/gw/main.go (a new minio gateway binary)
Usage:
```
$ go install -v storj.io/storj/cmd/captplanet
$ captplanet setup
$ captplanet run
```
Configuration is placed by default in `~/.storj/capt/`
Other changes:
* introduces new config structs for currently existing
Responsibilities that conform to the new Responsibility
interface. Please see the `pkg/*/config.go` files for
examples.
* integrates the PointerDB API key with other global
configuration via flags, instead of through environment
variables through viper like it's been doing. (ultimately
this should also change to use the PointerDB config
struct but this is an okay shortterm solution).
* changes the Overlay cache to use a URL for database
configuration instead of separate redis and bolt config
settings.
* stubs out some peer identity skeleton code (but not the
meat).
* Fixes the SegmentStore to use the overlay client and
pointerdb clients instead of gRPC client code directly
* Leaves a very clear spot where we need to tie the object to
stream to segment store together. There's sort of a "golden
spike" opportunity to connect all the train tracks together
at the bottom of pkg/miniogw/config.go, labeled with a
bunch of TODOs.
Future stuff:
* I now prefer this design over the original
pkg/process.Service thing I had been pushing before (sorry!)
* The experience of trying to have multiple farmers
configurable concurrently led me to prefer config structs
over global flags (I finally came around) or using viper
directly. I think global flags are okay sometimes but in
general going forward we should try and get all relevant
config into config structs.
* If you all like this direction, I think we can go delete my
old Service interfaces and a bunch of flags and clean up a
bunch of stuff.
* If you don't like this direction, it's no sweat at all, and
despite how much code there is here I'm not very tied to any
of this! Considering a lot of this was written between midnight
and 6 am, it might not be any good!
* bind tests
2018-07-24 17:08:28 +01:00
type Config struct {
2021-07-29 13:08:19 +01:00
DatabaseURL string ` help:"the database connection string to use" default:"postgres://" `
MinRemoteSegmentSize memory . Size ` default:"1240" testDefault:"0" help:"minimum remote segment size" ` // TODO: fix tests to work with 1024
MaxInlineSegmentSize memory . Size ` default:"4KiB" help:"maximum inline segment size" `
// we have such default value because max value for ObjectKey is 1024(1 Kib) but EncryptedObjectKey
// has encryption overhead 16 bytes. So overall size is 1024 + 16 * 16.
2023-08-21 21:03:16 +01:00
MaxEncryptedObjectKeyLength int ` default:"4000" help:"maximum encrypted object key length" `
2023-06-14 10:02:51 +01:00
MaxSegmentSize memory . Size ` default:"64MiB" help:"maximum segment size" `
MaxMetadataSize memory . Size ` default:"2KiB" help:"maximum segment metadata size" `
MaxCommitInterval time . Duration ` default:"48h" testDefault:"1h" help:"maximum time allowed to pass between creating and committing a segment" `
MinPartSize memory . Size ` default:"5MiB" testDefault:"0" help:"minimum allowed part size (last part has no minimum size limit)" `
MaxNumberOfParts int ` default:"10000" help:"maximum number of parts object can contain" `
Overlay bool ` default:"true" help:"toggle flag if overlay is enabled" `
RS RSConfig ` releaseDefault:"29/35/80/110-256B" devDefault:"4/6/8/10-256B" help:"redundancy scheme configuration in the format k/m/o/n-sharesize" `
RateLimiter RateLimiterConfig ` help:"rate limiter configuration" `
UploadLimiter UploadLimiterConfig ` help:"object upload limiter configuration" `
ProjectLimits ProjectLimitConfig ` help:"project limit configuration" `
2023-06-24 19:31:21 +01:00
2022-02-11 10:59:00 +00:00
// TODO remove this flag when server-side copy implementation will be finished
2023-08-04 10:27:08 +01:00
ServerSideCopy bool ` help:"enable code for server-side copy, deprecated. please leave this to true." default:"true" `
ServerSideCopyDisabled bool ` help:"disable already enabled server-side copy. this is because once server side copy is enabled, delete code should stay changed, even if you want to disable server side copy" default:"false" `
2023-06-24 19:31:21 +01:00
2023-10-30 18:52:10 +00:00
UseBucketLevelObjectVersioning bool ` help:"enable the use of bucket level object versioning" default:"false" `
// flag to simplify testing by enabling bucket level versioning feature only for specific projects
UseBucketLevelObjectVersioningProjects [ ] string ` help:"list of projects which will have UseBucketLevelObjectVersioning feature flag enabled" default:"" hidden:"true" `
2022-09-01 23:13:15 +01:00
// TODO remove when we benchmarking are done and decision is made.
TestListingQuery bool ` default:"false" help:"test the new query for non-recursive listing" `
2023-11-09 11:04:49 +00:00
// flag will be effective only if UseBucketLevelObjectVersioning or UseBucketLevelObjectVersioningProjects are enabled
TestEnableBucketVersioning bool ` default:"false" help:"if enabled all new created buckets will have versioning enabled (use only for testing)" hidden:"true" `
captplanet (#159)
* captplanet
I kind of went overboard this weekend.
The major goal of this changeset is to provide an environment
for local development where all of the various services can
be easily run together. Developing on Storj v3 should be as
easy as running a setup command and a run command!
To do this, this changeset introduces a new tool called
captplanet, which combines the powers of the Overlay Cache,
the PointerDB, the PieceStore, Kademlia, the Minio Gateway,
etc.
Running 40 farmers and a heavy client inside the same process
forced a rethinking of the "services" that we had. To
avoid confusion by reusing prior terms, this changeset
introduces two new types: Providers and Responsibilities.
I wanted to avoid as many merge conflicts as possible, so
I left the existing Services and code for now, but if people
like this route we can clean up the duplication.
A Responsibility is a collection of gRPC methods and
corresponding state. The following systems are examples of
Responsibilities:
* Kademlia
* OverlayCache
* PointerDB
* StatDB
* PieceStore
* etc.
A Provider is a collection of Responsibilities that
share an Identity, such as:
* The heavy client
* The farmer
* The gateway
An Identity is a public/private key pair, a node id, etc.
Farmers all need different Identities, so captplanet
needs to support running multiple concurrent Providers
with different Identities.
Each Responsibility and Provider should allow for configuration
of multiple copies on its own so creating Responsibilities and
Providers use a new workflow.
To make a Responsibility, one should create a "config"
struct, such as:
```
type Config struct {
RepairThreshold int `help:"If redundancy falls below this number of
pieces, repair is triggered" default:"30"`
SuccessThreshold int `help:"If redundancy is above this number then
no additional uploads are needed" default:"40"`
}
```
To use "config" structs, this changeset introduces another
new library called 'cfgstruct', which allows for the configuration
of arbitrary structs through flagsets, and thus through cobra and
viper.
cfgstruct relies on Go's "struct tags" feature to document
help information and default values. Config structs can be
configured via cfgstruct.Bind for binding the struct to a flagset.
Because this configuration system makes setup and configuration
easier *in general*, additional commands are provided that allow
for easy standup of separate Providers. Please make sure to
check out:
* cmd/captplanet/farmer/main.go (a new farmer binary)
* cmd/captplanet/hc/main.go (a new heavy client binary)
* cmd/captplanet/gw/main.go (a new minio gateway binary)
Usage:
```
$ go install -v storj.io/storj/cmd/captplanet
$ captplanet setup
$ captplanet run
```
Configuration is placed by default in `~/.storj/capt/`
Other changes:
* introduces new config structs for currently existing
Responsibilities that conform to the new Responsibility
interface. Please see the `pkg/*/config.go` files for
examples.
* integrates the PointerDB API key with other global
configuration via flags, instead of through environment
variables through viper like it's been doing. (ultimately
this should also change to use the PointerDB config
struct but this is an okay shortterm solution).
* changes the Overlay cache to use a URL for database
configuration instead of separate redis and bolt config
settings.
* stubs out some peer identity skeleton code (but not the
meat).
* Fixes the SegmentStore to use the overlay client and
pointerdb clients instead of gRPC client code directly
* Leaves a very clear spot where we need to tie the object to
stream to segment store together. There's sort of a "golden
spike" opportunity to connect all the train tracks together
at the bottom of pkg/miniogw/config.go, labeled with a
bunch of TODOs.
Future stuff:
* I now prefer this design over the original
pkg/process.Service thing I had been pushing before (sorry!)
* The experience of trying to have multiple farmers
configurable concurrently led me to prefer config structs
over global flags (I finally came around) or using viper
directly. I think global flags are okay sometimes but in
general going forward we should try and get all relevant
config into config structs.
* If you all like this direction, I think we can go delete my
old Service interfaces and a bunch of flags and clean up a
bunch of stuff.
* If you don't like this direction, it's no sweat at all, and
despite how much code there is here I'm not very tied to any
of this! Considering a lot of this was written between midnight
and 6 am, it might not be any good!
* bind tests
2018-07-24 17:08:28 +01:00
}
2022-10-28 15:56:20 +01:00
// Metabase constructs Metabase configuration based on Metainfo configuration with specific application name.
func ( c Config ) Metabase ( applicationName string ) metabase . Config {
return metabase . Config {
ApplicationName : applicationName ,
MinPartSize : c . MinPartSize ,
MaxNumberOfParts : c . MaxNumberOfParts ,
ServerSideCopy : c . ServerSideCopy ,
}
}
2023-08-04 14:25:36 +01:00
// ExtendedConfig extended config keeps additional helper fields and methods around Config.
type ExtendedConfig struct {
Config
2023-10-30 18:52:10 +00:00
useBucketLevelObjectVersioningProjects [ ] uuid . UUID
2023-08-04 14:25:36 +01:00
}
// NewExtendedConfig creates new instance of extended config.
2023-09-11 12:58:34 +01:00
func NewExtendedConfig ( config Config ) ( _ ExtendedConfig , err error ) {
2023-08-04 14:25:36 +01:00
extendedConfig := ExtendedConfig { Config : config }
2023-10-30 18:52:10 +00:00
for _ , projectIDString := range config . UseBucketLevelObjectVersioningProjects {
projectID , err := uuid . FromString ( projectIDString )
if err != nil {
return ExtendedConfig { } , err
}
extendedConfig . useBucketLevelObjectVersioningProjects = append ( extendedConfig . useBucketLevelObjectVersioningProjects , projectID )
}
2023-08-04 14:25:36 +01:00
return extendedConfig , nil
}
2023-10-30 18:52:10 +00:00
// UseBucketLevelObjectVersioningByProject checks if UseBucketLevelObjectVersioning should be enabled for specific project.
func ( ec ExtendedConfig ) UseBucketLevelObjectVersioningByProject ( projectID uuid . UUID ) bool {
// if its globally enabled don't look at projects
if ec . UseBucketLevelObjectVersioning {
return true
}
for _ , p := range ec . useBucketLevelObjectVersioningProjects {
if p == projectID {
return true
}
}
return false
}
