8d8350fea7
* port changes * Task monitor and setup merge from the staging * Restructure + additional interface * Add NewOverlayClient * integrated DHT client interface * added test for interface * PR comments addressed * lint issue * added generated protobuf * adding new interface * added the interface framework * deleted file * fixes compilation errors and integrates new dhtcclient interface * merged netstat latest changes and dht new interface chagnes * fixed the address's port * adding comments * PR comments addressed * netclient interface dial method added * rename and integrated transportclient with minio gateway * rename and code clean up * made changes based on the Dennis's changes on the kad-client * Code review comment changes based on kaloyan review comments * reverted the changes to be similar to master * removed unused file * renamed to transportclient * added the review changes * store the address of the client * updates per the code review comments, changes-> added error retry connection attempt logic, added error conditions including nil parameters * updated the test case to test the bad address passed condition * updated the code per code review comments * Bolt backed overlay cache (#94) * wip * add separate `Process` tests for bolt and redis-backed overlay * more testing * fix gitignore * fix linter error * goimports goimports GOIMPORTS GoImPortS!!!! * fix port madness * forgot to add * add `mux` as handler and shorten context timeouts * gofreakingimports * fix comments * refactor test & add logger/monkit registry * debugging travis * add comment * Set redisAddress to empty string for bolt-test * travis experiment * refactoring tests * Merge remote-tracking branch 'upstream/master' into bolt-backed-overlay-cache * Automatically build, tag and push docker images on merge to master (#103) * port changes * build overlay on successful merge to master * fixes to Makefile * permissions * dep ensure * gopath * let's try vgo * remove dep * maybe alpine is the issue * tagging go version on build * stupid vgo * vgo * adding tags to push * quotes * local linting fixes & stupid travis * prepend storjlabs to docker tag (#108) * port changes * fixing tag name * Use continue instead of return in table tests (#106) I did a dumb mistake for some of the table tests, which made some of the test cases not being executed. * pkg/kademlia tests and restructuring (#97) * port changes * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * files created * Merge remote-tracking branch 'upstream/master' into coyle/kad-tests * wip * Merge remote-tracking branch 'upstream/master' into coyle/kad-tests * wip * remove bkad dependencie from tests * wip * wip * wip * wip * wip * updated coyle/kademlia * wip * cleanup * ports * overlay upgraded * linter fixes * piecestore kademlia newID * add changes from kad demo * PR comments addresses * go func * force travis build * fixed merge conflicts * fixed merge conflicts * Merge branch 'coyle/kad-tests' of https://github.com/coyle/storj into coyle/kad-tests * linter issues * linting issues * fixed merge conflicts * linter is stupid * Coyle/docker fix (#109) * port changes * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * Merge branch 'master' of https://github.com/storj/storj * fixing tag name * no idea * testing * changes * testing on travis * testing * changes to travis build * new approach * Merge branch 'master' into coyle/docker-fix * hardcode version (#111) * hardcode version * adding coveralls / code coverage (#112) * adding coveralls * adding code coverage badge * fixing badges * verbose * swap tests and coverage * extra line * maybe * maybe * moar * gover maybe * testing * cleanup * protos/netstate: remove stuff we're not using (#100) * protos/netstate: remove stuff we're not using * protos/netstate: add metadata field for segmentstore * fix netstate client test * pkg/process: start replacing pkg/process with cobra helpers (#98) * Implement psclient interface (#107) * Implement psclient interface * Add string method to pieceID type * try to fix linter errors * Whoops missed an error * More linter errors * Typo * Lol double typo * Get everything working, begin adding tests for psclient rpc * goimports * Forgot to change the piecestore cli when changed the piecestore code * Fix CLI * remove ID length, added validator to pieceID * Move grpc ranger to client Change client PUT api to take a reader rather than return a writer * GRPCRanger -> PieceRanger; Make PieceRanger a RangeCloser * Forgot to remove offset * Added message upon successful store * Do that thing dennis and kaloyan wanted * goimports * Make closeConn a part of the interface for psclient * Use interface * Removed uneccessary new lines * goimport * Whoops * Actually we don't want to use the interface in Piece Ranger * Renamed piecestore in examples to piecestore-client; moved piecestore-cli to examples * Make comments look nicer * modified transport client based on the the design discussion * modified transport client based on the the design discussion * added the as discussed connection cache interface functionality * added the as discussed connection cache interface functionality * transport client changes * transport client per code review changes * per the code review comments * transport client incorporates review comments * fixes lint warnings * lint warning fixes....client interface has to be Client * initial draft of Objectstore * transport client review changes * client.go changes * transport.go changes * added test case * added test cases * streams iface * comment fix * object store changes * comment fix * initialized the objectstore in gw * Added PutObject with test support for encryption file * added object store test cases * tested & integrated the objectstore with miniogw * handled the ranger and paths * indentation change * Kalyon's code review comments resolution implemented after the 30min code review meeting * Compilation error fix * fixes the tavis build warnings * corrects the ListObject return type to be slice of slice * corrects the ListObject return type to be slice of slice * added the serialization using protobuf * added the unmarshalling of data in getobject() * Jt's Review comments * Kaloyan's review comments, moved the unmarshalling logic and other minor code indentation fixes * more code reivew * more code reivew * Changes the expiration time to zeroTime and added error check in putObject function * Changes the expiration time to zeroTime and added error check in putObject function * minor warning fix- had to add a comment and fix the wording * added a TODO comment per kaloyan * code clean up removed unused variables
190 lines
4.6 KiB
Go
190 lines
4.6 KiB
Go
// Copyright (C) 2018 Storj Labs, Inc.
|
|
// See LICENSE for copying information.
|
|
|
|
package storj
|
|
|
|
import (
|
|
"context"
|
|
"io"
|
|
"time"
|
|
|
|
"github.com/gogo/protobuf/proto"
|
|
"github.com/minio/cli"
|
|
minio "github.com/minio/minio/cmd"
|
|
"github.com/minio/minio/pkg/auth"
|
|
"github.com/minio/minio/pkg/hash"
|
|
"github.com/zeebo/errs"
|
|
monkit "gopkg.in/spacemonkeygo/monkit.v2"
|
|
|
|
"storj.io/storj/pkg/objects"
|
|
"storj.io/storj/pkg/paths"
|
|
mpb "storj.io/storj/protos/objects"
|
|
)
|
|
|
|
var (
|
|
mon = monkit.Package()
|
|
//Error is the errs class of standard End User Client errors
|
|
Error = errs.Class("Storj Gateway error")
|
|
)
|
|
|
|
func init() {
|
|
minio.RegisterGatewayCommand(cli.Command{
|
|
Name: "storj",
|
|
Usage: "Storj",
|
|
Action: storjGatewayMain,
|
|
HideHelpCommand: true,
|
|
})
|
|
}
|
|
|
|
func storjGatewayMain(ctx *cli.Context) {
|
|
s := &Storj{os: mockObjectStore()}
|
|
minio.StartGateway(ctx, s)
|
|
}
|
|
|
|
func mockObjectStore() objects.ObjectStore {
|
|
return &objects.Objects{}
|
|
}
|
|
|
|
// Storj is the implementation of a minio cmd.Gateway
|
|
type Storj struct {
|
|
os objects.ObjectStore
|
|
}
|
|
|
|
// Name implements cmd.Gateway
|
|
func (s *Storj) Name() string {
|
|
return "storj"
|
|
}
|
|
|
|
// NewGatewayLayer implements cmd.Gateway
|
|
func (s *Storj) NewGatewayLayer(creds auth.Credentials) (
|
|
minio.ObjectLayer, error) {
|
|
return &storjObjects{storj: s}, nil
|
|
}
|
|
|
|
// Production implements cmd.Gateway
|
|
func (s *Storj) Production() bool {
|
|
return false
|
|
}
|
|
|
|
type storjObjects struct {
|
|
minio.GatewayUnsupported
|
|
storj *Storj
|
|
}
|
|
|
|
func (s *storjObjects) DeleteBucket(ctx context.Context, bucket string) (err error) {
|
|
defer mon.Task()(&ctx)(&err)
|
|
panic("TODO")
|
|
}
|
|
|
|
func (s *storjObjects) DeleteObject(ctx context.Context, bucket, object string) (err error) {
|
|
defer mon.Task()(&ctx)(&err)
|
|
objpath := paths.New(bucket, object)
|
|
return s.storj.os.DeleteObject(ctx, objpath)
|
|
}
|
|
|
|
func (s *storjObjects) GetBucketInfo(ctx context.Context, bucket string) (
|
|
bucketInfo minio.BucketInfo, err error) {
|
|
defer mon.Task()(&ctx)(&err)
|
|
panic("TODO")
|
|
}
|
|
|
|
func (s *storjObjects) GetObject(ctx context.Context, bucket, object string,
|
|
startOffset int64, length int64, writer io.Writer, etag string) (err error) {
|
|
defer mon.Task()(&ctx)(&err)
|
|
objpath := paths.New(bucket, object)
|
|
rr, _, err := s.storj.os.GetObject(ctx, objpath)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
defer rr.Close()
|
|
r, err := rr.Range(ctx, startOffset, length)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
defer r.Close()
|
|
_, err = io.Copy(writer, r)
|
|
return err
|
|
}
|
|
|
|
func (s *storjObjects) GetObjectInfo(ctx context.Context, bucket,
|
|
object string) (objInfo minio.ObjectInfo, err error) {
|
|
defer mon.Task()(&ctx)(&err)
|
|
objPath := paths.New(bucket, object)
|
|
rr, m, err := s.storj.os.GetObject(ctx, objPath)
|
|
if err != nil {
|
|
return objInfo, err
|
|
}
|
|
defer rr.Close()
|
|
newmetainfo := &mpb.StorjMetaInfo{}
|
|
err = proto.Unmarshal(m.Data, newmetainfo)
|
|
if err != nil {
|
|
return objInfo, err
|
|
}
|
|
return minio.ObjectInfo{
|
|
Name: newmetainfo.GetName(),
|
|
Bucket: newmetainfo.GetBucket(),
|
|
ModTime: m.Modified,
|
|
Size: newmetainfo.GetSize(),
|
|
ETag: newmetainfo.GetETag(),
|
|
}, err
|
|
}
|
|
|
|
func (s *storjObjects) ListBuckets(ctx context.Context) (
|
|
buckets []minio.BucketInfo, err error) {
|
|
defer mon.Task()(&ctx)(&err)
|
|
buckets = nil
|
|
err = nil
|
|
return buckets, err
|
|
}
|
|
|
|
func (s *storjObjects) ListObjects(ctx context.Context, bucket, prefix, marker,
|
|
delimiter string, maxKeys int) (result minio.ListObjectsInfo, err error) {
|
|
defer mon.Task()(&ctx)(&err)
|
|
result = minio.ListObjectsInfo{}
|
|
err = nil
|
|
return result, err
|
|
}
|
|
|
|
func (s *storjObjects) MakeBucketWithLocation(ctx context.Context,
|
|
bucket string, location string) (err error) {
|
|
defer mon.Task()(&ctx)(&err)
|
|
return nil
|
|
}
|
|
|
|
func (s *storjObjects) PutObject(ctx context.Context, bucket, object string,
|
|
data *hash.Reader, metadata map[string]string) (objInfo minio.ObjectInfo,
|
|
err error) {
|
|
defer mon.Task()(&ctx)(&err)
|
|
//metadata serialized
|
|
serMetaInfo := &mpb.StorjMetaInfo{
|
|
ContentType: metadata["content-type"],
|
|
Bucket: bucket,
|
|
Name: object,
|
|
}
|
|
metainfo, err := proto.Marshal(serMetaInfo)
|
|
if err != nil {
|
|
return objInfo, err
|
|
}
|
|
objPath := paths.New(bucket, object)
|
|
// setting zero value means the object never expires
|
|
expTime := time.Time{}
|
|
err = s.storj.os.PutObject(ctx, objPath, data, metainfo, expTime)
|
|
return minio.ObjectInfo{
|
|
Name: object,
|
|
Bucket: bucket,
|
|
// TODO create a followup ticket in JIRA to fix ModTime
|
|
ModTime: time.Now(),
|
|
Size: data.Size(),
|
|
ETag: minio.GenETag(),
|
|
}, err
|
|
}
|
|
|
|
func (s *storjObjects) Shutdown(ctx context.Context) (err error) {
|
|
defer mon.Task()(&ctx)(&err)
|
|
panic("TODO")
|
|
}
|
|
|
|
func (s *storjObjects) StorageInfo(context.Context) minio.StorageInfo {
|
|
return minio.StorageInfo{}
|
|
}
|