storj/pkg/miniogw/gateway-storj.go
aligeti 8d8350fea7 Implement miniogw in terms of ObjectStore (#117)
* port changes

* Task monitor and setup merge from the staging

* Restructure + additional interface

* Add NewOverlayClient

* integrated DHT client interface

* added test for interface

* PR comments addressed

* lint issue

* added generated protobuf

* adding new interface

* added the interface framework

* deleted file

* fixes compilation errors and integrates new dhtcclient interface

* merged netstat latest changes and dht new interface chagnes

* fixed the address's port

* adding comments

* PR comments addressed

* netclient interface dial method added

* rename and integrated transportclient with minio gateway

* rename and code clean up

* made changes based on the Dennis's changes on the kad-client

* Code review comment changes based on kaloyan review comments

* reverted the changes to be similar to master

* removed unused file

* renamed to transportclient

* added the review changes

* store the address of the client

* updates per the code review comments, changes-> added error retry connection attempt logic, added error conditions including nil parameters

* updated the test case to test the bad address passed condition

* updated the code per code review comments

* Bolt backed overlay cache (#94)

* wip

* add separate `Process` tests for bolt and redis-backed overlay

* more testing

* fix gitignore

* fix linter error

* goimports goimports GOIMPORTS GoImPortS!!!!

* fix port madness

* forgot to add

* add `mux` as handler and shorten context timeouts

* gofreakingimports

* fix comments

* refactor test & add logger/monkit registry

* debugging travis

* add comment

* Set redisAddress to empty string for bolt-test

* travis experiment

* refactoring tests

* Merge remote-tracking branch 'upstream/master' into bolt-backed-overlay-cache

* Automatically build, tag and push docker images on merge to master (#103)

* port changes

* build overlay on successful merge to master

* fixes to Makefile

* permissions

* dep ensure

* gopath

* let's try vgo

* remove dep

* maybe alpine is the issue

* tagging go version on build

* stupid vgo

* vgo

* adding tags to push

* quotes

* local linting fixes & stupid travis

* prepend storjlabs to docker tag (#108)

* port changes

* fixing tag name

* Use continue instead of return in table tests (#106)

I did a dumb mistake for some of the table tests, which made some of the
test cases not being executed.

* pkg/kademlia tests and restructuring (#97)

* port changes

* Merge remote-tracking branch 'upstream/master'

* Merge remote-tracking branch 'upstream/master'

* Merge remote-tracking branch 'upstream/master'

* files created

* Merge remote-tracking branch 'upstream/master' into coyle/kad-tests

* wip

* Merge remote-tracking branch 'upstream/master' into coyle/kad-tests

* wip

* remove bkad dependencie from tests

* wip

* wip

* wip

* wip

* wip

* updated coyle/kademlia

* wip

* cleanup

* ports

* overlay upgraded

* linter fixes

* piecestore kademlia newID

* add changes from kad demo

* PR comments addresses

* go func

* force travis build

* fixed merge conflicts

* fixed merge conflicts

* Merge branch 'coyle/kad-tests' of https://github.com/coyle/storj into coyle/kad-tests

* linter issues

* linting issues

* fixed merge conflicts

* linter is stupid

* Coyle/docker fix (#109)

* port changes

* Merge remote-tracking branch 'upstream/master'

* Merge remote-tracking branch 'upstream/master'

* Merge remote-tracking branch 'upstream/master'

* Merge remote-tracking branch 'upstream/master'

* Merge remote-tracking branch 'upstream/master'

* Merge branch 'master' of https://github.com/storj/storj

* fixing tag name

* no idea

* testing

* changes

* testing on travis

* testing

* changes to travis build

* new approach

* Merge branch 'master' into coyle/docker-fix

* hardcode version (#111)

* hardcode version

* adding coveralls / code coverage  (#112)

* adding coveralls

* adding code coverage badge

* fixing badges

* verbose

* swap tests and coverage

* extra line

* maybe

* maybe

* moar

* gover maybe

* testing

* cleanup

* protos/netstate: remove stuff we're not using (#100)

* protos/netstate: remove stuff we're not using

* protos/netstate: add metadata field for segmentstore

* fix netstate client test

* pkg/process: start replacing pkg/process with cobra helpers (#98)

* Implement psclient interface (#107)

* Implement psclient interface

* Add string method to pieceID type

* try to fix linter errors

* Whoops missed an error

* More linter errors

* Typo

* Lol double typo

*  Get everything working, begin adding tests for psclient rpc

* goimports

* Forgot to change the piecestore cli when changed the piecestore code

* Fix CLI

* remove ID length, added validator to pieceID

* Move grpc ranger to client
Change client PUT api to take a reader rather than return a writer

* GRPCRanger -> PieceRanger; Make PieceRanger a RangeCloser

* Forgot to remove offset

* Added message upon successful store

* Do that thing dennis and kaloyan wanted

* goimports

* Make closeConn a part of the interface for psclient

* Use interface

* Removed uneccessary new lines

* goimport

* Whoops

* Actually we don't want to use the interface in Piece Ranger

* Renamed piecestore in examples to piecestore-client; moved piecestore-cli to examples

* Make comments look nicer

* modified transport client based on the the design discussion

* modified transport client based on the the design discussion

* added the as discussed connection cache interface functionality

* added the as discussed connection cache interface functionality

* transport client changes

* transport client per code review changes

* per the code review comments

* transport client incorporates review comments

* fixes lint warnings

* lint warning fixes....client interface has to be Client

* initial draft of Objectstore

* transport client review changes

* client.go changes

* transport.go changes

* added test case

* added test cases

* streams iface

* comment fix

* object store changes

* comment fix

* initialized the objectstore in gw

* Added PutObject with test support for encryption file

* added object store test cases

* tested & integrated the objectstore with miniogw

* handled the ranger and paths

* indentation change

* Kalyon's code review comments resolution implemented after the 30min code review meeting

* Compilation error fix

* fixes the tavis build warnings

* corrects the ListObject  return type to be slice of slice

* corrects the ListObject  return type to be slice of slice

*  added the serialization using protobuf

* added the unmarshalling of data in getobject()

* Jt's Review comments

* Kaloyan's review comments, moved the unmarshalling logic and other minor code indentation fixes

* more code reivew

* more code reivew

* Changes the expiration time to zeroTime and added error check in putObject function

* Changes the expiration time to zeroTime and added error check in putObject function

* minor warning fix- had to add a comment and fix the wording

* added a TODO comment per kaloyan

* code clean up removed unused variables
2018-07-12 11:29:02 -06:00

190 lines
4.6 KiB
Go

// Copyright (C) 2018 Storj Labs, Inc.
// See LICENSE for copying information.
package storj
import (
"context"
"io"
"time"
"github.com/gogo/protobuf/proto"
"github.com/minio/cli"
minio "github.com/minio/minio/cmd"
"github.com/minio/minio/pkg/auth"
"github.com/minio/minio/pkg/hash"
"github.com/zeebo/errs"
monkit "gopkg.in/spacemonkeygo/monkit.v2"
"storj.io/storj/pkg/objects"
"storj.io/storj/pkg/paths"
mpb "storj.io/storj/protos/objects"
)
var (
mon = monkit.Package()
//Error is the errs class of standard End User Client errors
Error = errs.Class("Storj Gateway error")
)
func init() {
minio.RegisterGatewayCommand(cli.Command{
Name: "storj",
Usage: "Storj",
Action: storjGatewayMain,
HideHelpCommand: true,
})
}
func storjGatewayMain(ctx *cli.Context) {
s := &Storj{os: mockObjectStore()}
minio.StartGateway(ctx, s)
}
func mockObjectStore() objects.ObjectStore {
return &objects.Objects{}
}
// Storj is the implementation of a minio cmd.Gateway
type Storj struct {
os objects.ObjectStore
}
// Name implements cmd.Gateway
func (s *Storj) Name() string {
return "storj"
}
// NewGatewayLayer implements cmd.Gateway
func (s *Storj) NewGatewayLayer(creds auth.Credentials) (
minio.ObjectLayer, error) {
return &storjObjects{storj: s}, nil
}
// Production implements cmd.Gateway
func (s *Storj) Production() bool {
return false
}
type storjObjects struct {
minio.GatewayUnsupported
storj *Storj
}
func (s *storjObjects) DeleteBucket(ctx context.Context, bucket string) (err error) {
defer mon.Task()(&ctx)(&err)
panic("TODO")
}
func (s *storjObjects) DeleteObject(ctx context.Context, bucket, object string) (err error) {
defer mon.Task()(&ctx)(&err)
objpath := paths.New(bucket, object)
return s.storj.os.DeleteObject(ctx, objpath)
}
func (s *storjObjects) GetBucketInfo(ctx context.Context, bucket string) (
bucketInfo minio.BucketInfo, err error) {
defer mon.Task()(&ctx)(&err)
panic("TODO")
}
func (s *storjObjects) GetObject(ctx context.Context, bucket, object string,
startOffset int64, length int64, writer io.Writer, etag string) (err error) {
defer mon.Task()(&ctx)(&err)
objpath := paths.New(bucket, object)
rr, _, err := s.storj.os.GetObject(ctx, objpath)
if err != nil {
return err
}
defer rr.Close()
r, err := rr.Range(ctx, startOffset, length)
if err != nil {
return err
}
defer r.Close()
_, err = io.Copy(writer, r)
return err
}
func (s *storjObjects) GetObjectInfo(ctx context.Context, bucket,
object string) (objInfo minio.ObjectInfo, err error) {
defer mon.Task()(&ctx)(&err)
objPath := paths.New(bucket, object)
rr, m, err := s.storj.os.GetObject(ctx, objPath)
if err != nil {
return objInfo, err
}
defer rr.Close()
newmetainfo := &mpb.StorjMetaInfo{}
err = proto.Unmarshal(m.Data, newmetainfo)
if err != nil {
return objInfo, err
}
return minio.ObjectInfo{
Name: newmetainfo.GetName(),
Bucket: newmetainfo.GetBucket(),
ModTime: m.Modified,
Size: newmetainfo.GetSize(),
ETag: newmetainfo.GetETag(),
}, err
}
func (s *storjObjects) ListBuckets(ctx context.Context) (
buckets []minio.BucketInfo, err error) {
defer mon.Task()(&ctx)(&err)
buckets = nil
err = nil
return buckets, err
}
func (s *storjObjects) ListObjects(ctx context.Context, bucket, prefix, marker,
delimiter string, maxKeys int) (result minio.ListObjectsInfo, err error) {
defer mon.Task()(&ctx)(&err)
result = minio.ListObjectsInfo{}
err = nil
return result, err
}
func (s *storjObjects) MakeBucketWithLocation(ctx context.Context,
bucket string, location string) (err error) {
defer mon.Task()(&ctx)(&err)
return nil
}
func (s *storjObjects) PutObject(ctx context.Context, bucket, object string,
data *hash.Reader, metadata map[string]string) (objInfo minio.ObjectInfo,
err error) {
defer mon.Task()(&ctx)(&err)
//metadata serialized
serMetaInfo := &mpb.StorjMetaInfo{
ContentType: metadata["content-type"],
Bucket: bucket,
Name: object,
}
metainfo, err := proto.Marshal(serMetaInfo)
if err != nil {
return objInfo, err
}
objPath := paths.New(bucket, object)
// setting zero value means the object never expires
expTime := time.Time{}
err = s.storj.os.PutObject(ctx, objPath, data, metainfo, expTime)
return minio.ObjectInfo{
Name: object,
Bucket: bucket,
// TODO create a followup ticket in JIRA to fix ModTime
ModTime: time.Now(),
Size: data.Size(),
ETag: minio.GenETag(),
}, err
}
func (s *storjObjects) Shutdown(ctx context.Context) (err error) {
defer mon.Task()(&ctx)(&err)
panic("TODO")
}
func (s *storjObjects) StorageInfo(context.Context) minio.StorageInfo {
return minio.StorageInfo{}
}