Commit Graph

7675 Commits

Author SHA1 Message Date
aligeti
09da23737a
rename farmers to storagenode (#275)
* rename farmers to storagenode

* review changes

* merge conflicts
2018-08-24 21:52:58 -04:00
JT Olio
3154086e58 go1.11: update monotime (#287) 2018-08-24 20:23:09 -04:00
JT Olio
560e9fd9ec
uplink: make ul/uplink consistent (#286)
* uplink: make ul/uplink consistent

* pstore: make test more robust
2018-08-24 14:02:42 -06:00
aligeti
4edba19986 [Ready for review]Rename gw uplink (#273)
Rename gw uplink
2018-08-24 10:22:24 -06:00
Kaloyan Raev
7d1bd090f8
Optimize DecodeReader performance (#256)
* Optimize DecodeReader performance

* A little bit better locking in PieceBuffer.Write

* Fix race issues

* Better fix for race condition in rs_test.go

* Improve PieceBuffer.Read to read the max available in one call

* PieceBuffer.Skip for more efficient discarding of old shares

* Rename bytesRead to nn

* Notify cvNewData only if a complete new share is available

* Small correction in PieceBuffer.Read

* Rename some fields to have longer names
2018-08-24 16:06:27 +03:00
Maximillian von Briesen
553def620a TLS Support (#252)
* begin adding tls

* remove incomplete line in gw/main.go

* identity fixes+:

+ fix `peertls.NewCert` public key issue
+ fix `peertls.verfiyChain` issue
+ fix identity dial option
+ rename `GenerateCA` to `NewCA` and `generateCAWorker` to `newCAWorker` for better consistency/convention

* use pdbclient instead of pointerdb in miniogw

* fix tests

* go fmt

* make review changes

* modify how context.Background() is used

* more context stuff
2018-08-23 22:01:03 -06:00
James Hagans
fdfa6e85c8 Streamstore impl of Stor interface (#191)
* first stab at PUT

* only PUT

* working on PUT

* Put with LimitReader

* start of Get

* reorder of files and proto meta

* working on Meta

* working on Meta

* add aware limit reader

* add size from segment put

* rm if for eof

* update to proto meta

* update gen proto file

* working on get

* working on get

* working on get

* working on list

* working on delete

* working on list

* working on meta method

* fix merge error and working on feedback from PR

* update to proto file

* rm size tuple

* mv eof limit reader to new file

* add toMeta

* rm varible names

* add updates from PR feedback

* updates from PR feedback

* updates from PR feedback

* add toMeta size based on total size

* update toMeta size calculation

* rm passthrough

* add default to config for segment size

* fix get method ranger bug

* add object support for nested stream proto

* rm nested stream meta data

* rm test for another PR
2018-08-23 21:56:38 -06:00
Matt Robinson
3456a127cf
More fixes to the staging deploy process (#278) 2018-08-23 17:17:47 -04:00
Matt Robinson
631331236e
Fix staging deploy of satellite and storage-node (#277) 2018-08-23 17:04:51 -04:00
Matt Robinson
026997e725
All-in-one docker-compose project (#267)
* First pass at an all-in-one compose project

* Sort out dependencies so uplink can build in docker

* Added the final bits to make this work

* Add readme for all-in-one

* Fix sed to work on OS X and Linux

* Make the readme for all-in-one better

* Only bring up the satellite and uplink after fixing the mock overlay

* Use a dummy value for mock overlay for now

* Finishing touches

* Fix url form for aws cli tool

* Move files out of all-in-one directory to help with directory confusion.

* Added a make target to make all-in-one even better
2018-08-23 11:48:03 -04:00
Jennifer Li Johnson
154a1eab3a
implements node pkg query method (#251) 2018-08-23 11:20:11 -04:00
Bryan White
0e2fd97029
Identity fixes+: (#270)
* identity fixes+:

+ fix `peertls.NewCert` public key issue
+ fix `peertls.verfiyChain` issue
+ fix identity dial option
+ rename `GenerateCA` to `NewCA` and `generateCAWorker` to `newCAWorker` for better consistency/convention
2018-08-23 16:08:26 +02:00
Egon Elbre
f80185a807
Preliminary support for windows (#266)
* Don't use url.Parse for bolt paths: filepaths may not be valid URL-s.
* go.mod: update dependencies
* README.md: add Windows instructions
* pkg/overlay: check for the correct path and text in error
* pkg/overlay: fix tests for windows
* pkg/piecestore: make windows tests pass
* pkg/telemetry: skip test, as it doesn't shutdown nicely
* storage/redis: ensure that redis is clean before running tests
2018-08-23 10:05:56 +03:00
Dylan Lott
ce3f9c6699
Fix nil bucket error (#269) 2018-08-22 18:19:46 -06:00
JT Olio
9aee5efd99
pointerdb: separate client and server packages (#261)
* pointerdb: separate client and server packages

the reason for this is so that things the server needs (bolt, auth)
don't get imported if all you need is the client. will result in
smaller binaries and less flag definitions

* review comments
2018-08-22 09:07:00 -06:00
Philip Hutchins
5279283b39 Adding some logging to Jenkinsfile for debug (#265) 2018-08-22 09:18:50 -04:00
JT Olio
ffa6f816d5 gw: support standard configuration via cmdline (#262) 2018-08-22 09:05:16 -04:00
Kaloyan Raev
62521bb3f8 Fixes downloads in Minio Browser (#264) 2018-08-22 13:15:13 +02:00
Egon Elbre
fe3decc42f
all: fix govet warnings (#255)
Fixes go1.11 vet warnings.

Cancel on WithTimeout must always be called to avoid memory leak:

pkg/provider/provider.go:73: the cancel function returned by context.WithTimeout should be called, not discarded, to avoid a context leak

Range over non-copyable things:

pkg/pool/connection_pool_test.go:32: range var v copies lock: struct{pool pool.ConnectionPool; key string; expected pool.TestFoo; expectedError error} contains pool.ConnectionPool contains sync.RWMutex
pkg/pool/connection_pool_test.go:56: range var v copies lock: struct{pool pool.ConnectionPool; key string; value pool.TestFoo; expected pool.TestFoo; expectedError error} contains pool.ConnectionPool contains sync.RWMutex
pkg/pool/connection_pool_test.go:83: range var v copies lock: struct{pool pool.ConnectionPool; key string; value pool.TestFoo; expected interface{}; expectedError error} contains pool.ConnectionPool contains sync.RWMutex

zeebo/errs package always requires formatting directives:

pkg/peertls/peertls.go:50: Class.New call has arguments but no formatting directives
pkg/peertls/utils.go:47: Class.New call has arguments but no formatting directives
pkg/peertls/utils.go:87: Class.New call has arguments but no formatting directives
pkg/overlay/cache.go:94: Class.New call has arguments but no formatting directives
pkg/provider/certificate_authority.go:98: New call has arguments but no formatting directives
pkg/provider/identity.go:96: New call has arguments but no formatting directives
pkg/provider/utils.go:124: New call needs 1 arg but has 2 args
pkg/provider/utils.go:136: New call needs 1 arg but has 2 args
storage/redis/client.go:44: Class.New call has arguments but no formatting directives
storage/redis/client.go:64: Class.New call has arguments but no formatting directives
storage/redis/client.go:75: Class.New call has arguments but no formatting directives
storage/redis/client.go:80: Class.New call has arguments but no formatting directives
storage/redis/client.go:92: Class.New call has arguments but no formatting directives
storage/redis/client.go:96: Class.New call has arguments but no formatting directives
storage/redis/client.go:102: Class.New call has arguments but no formatting directives
storage/redis/client.go:126: Class.New call has arguments but no formatting directives
2018-08-22 09:39:57 +03:00
Matt Robinson
5841065420 Attempt to fix jenkins builds (#260)
* Attempt to clean up docker image tags

* Clean only master

* Add back go version and actually use it
2018-08-21 22:54:56 -04:00
Jennifer Li Johnson
15657071d7
implements connection success and fail on kad routing table (#249)
* implements connection success and fail on kad routing table

* modifications from code review

* todo

* test fixes

* passes in node rather than id

* removes rpath

* test fix
2018-08-21 14:44:42 -04:00
JT Olio
9c252393f2
clabot: add egon (#259) 2018-08-21 12:23:28 -06:00
JT Olio
40f4f7ec31
piecestore: only check ttl db every hour instead of every 300 nanoseconds (#250) 2018-08-20 15:47:24 -04:00
JT Olio
520abc5c73
Update README.md 2018-08-20 13:46:45 -06:00
JT Olio
fa390c9ec9
Mock overlay move (#245)
* move mock overlay from client to server

this doesn't really change much, but it does allow you to
run a standalone gateway against captain planet. it still does
not allow you to run a standalone gateway against a standalone
heavy client

* pointerdb: small error fixes

* some cleanups

* fix tests
2018-08-20 14:24:11 -04:00
Bryan White
f932c3ced8 Overlay cache commands (#236)
* wip overlay cache cmd

* overlay cache add and list commands

* use zap

* fixes

* simplify

* add restrictions

* cleanup

* remove defaults for overlay management commands
2018-08-20 14:21:41 -04:00
Kaloyan Raev
d1c508e24a Avoid logging any minio error in miniogw logging wrapper (#240) 2018-08-20 13:55:39 -04:00
Kaloyan Raev
49577a47f8 Configure erasure-share-size to 1024 bytes (#241) 2018-08-20 13:54:30 -04:00
Kaloyan Raev
130acf57a5
Make PSClient a Closer, so it can be closed with utils.LogClose() (#247) 2018-08-20 18:11:54 +03:00
Jennifer Li Johnson
91bf6e19e8
Routing Table Replacement Cache (#229)
* creating replacement cache

* wip

* wip

* rewrites replacement cache

* replacement cache tests

* update and remove node tests

* check if dropped node got added to replacement cache from add node method

* wip

* wip

* making changes based on pr
2018-08-17 15:11:46 -04:00
Brandon Iglesias
a522cabfbd readme updates (Storj CLI) (#243)
* updates to the readme

Added section for the Storj CLI
Styled AWS S3 CLI section

* Update README.md

* removed file name in upload object commands

* updated title

* added table of contents
2018-08-17 15:05:03 -04:00
Alexander Leitner
b0db33f919
Bandwidth accounting (#134)
* captplanet standalone farmer setup

* Bandwidth Allocation

* utils.Close method changed to utils.LogClose

* Get build temporarily working

* Get/Put for PSClient should take payer bandwidth allocations rather than the NewPSClient function

* Update example client to reflect changes in client API

* Update ecclient to use latest PSClient, Make NewPSClient return error also

* Updated pieceranger tests to check for errors; sign method should take byte array

* Handle defers in store.go better

* Fix defer functions in psdb.go

* fun times

* Protobuf bandwidthallocation data is now a byte array

* Remove psservice package and merge it into pstore server

* Write wrapper for database calls

* Change all expiration names in protobuf to be more informative; add defer in retrieve; remove old comment

* Make PSDB tests implementation independent rather than method independent

* get rid of payer, renter in ecclient

* add context monitoring in store and retrieve
2018-08-17 13:40:15 -04:00
Kaloyan Raev
ae430b8c10 Use utils.LogClose for deferred closing in miniogw (#239) 2018-08-17 13:17:58 -04:00
Cameron
906643fce2
Storj CLT (#201)
* WIP first commit

* change args check

* rename to cli, rename folder to storj, const defaultConfDir, use process.Exec()

* typos

* WIP create helper for getting minio objectlayer, changed setup command to try to match changes in setting up identity

* add rb command

* changes for setting up identity

* remove concurrency field from setupCfg

* rename variable returned by url.Parse to u across all commands for consistency

* add marker functionality to ls command

* add download functionality to cp command

* fix typo in rb init, use short variable names
2018-08-16 14:34:55 -04:00
Philip Hutchins
bfac1297f2
Updating entrypoint and dockerfile (#237) 2018-08-16 14:03:15 -04:00
Natalie Villasana
ff65663867
adds Bucket Store (#213)
* adds foundation for bucketStore

* adds prefixedObjStore to buckets package, adjusts gateway-storj accordingly

* fixes multi value assignment problems in gateway-storj

* fixes more multi value assignment errors in gateway-storj

* starts changing miniogw tests to accommodate buckets

* creates bucket store mock

* wip - fixing test cases in object tests

* adds get, put, and list object tests, comments out two test cases

* adds happy scenario tests for bucket methods

* fixes bug in list, removes redundant parts from gateway tests

* fixes nit

* Clean up tests from #188

* Fix bug with timestamp conversion in segment store

* fixes segments.Meta test

* Fix regression in listing objects in a bucket

* adds check to see if bucket is empty before deleting

* updates DeleteBucket test to account for empty/full bucket

* adds TODOs for DeleteBucket and MakeBucket for some cases, adjusts tests, filters out minio errors in logging.go

* adds checks for if buckets already exist or not in DeleteBucket and MakeBucket functions; adjusts tests

* adds BucketNotFound error check in bucket store, removes todo

* adds make_bucket to Travis test, updates boltdb client constructor to always create a bucket (table)
2018-08-16 10:32:28 -04:00
Brandon Iglesias
0cdc331585
styling changes to the code of conduct (#234)
adding bullet points in "Our Standards" section
2018-08-16 10:07:30 -04:00
Brandon Iglesias
c17fc062ff Adding the Storj Code of Conduct (#233)
* Adding the Storj Code of Conduct to Github

* fixing styling

* Update CODE_OF_CONDUCT.md

* Update CODE_OF_CONDUCT.md
2018-08-15 14:22:58 -07:00
Matt Robinson
1687ce25c7 Fix the sha3 error with the build (#230)
* Fix the build
2018-08-15 14:49:30 -04:00
Maximillian von Briesen
f80ddf9358
Test captplanet with travis (#226)
* test captplanet in travis

* add comparison between uploaded and downloaded file

* test inline and remote files

* move captplanet testing to bash script
2018-08-14 12:58:16 -04:00
nfarah86
d396eb6be9
Bug/download (#216)
* fixed downloading empty file

* fixed formatting

* fixed error handling

* fixed cache_test.go to accomodate for error

* fixed error verbose messaging for wrong downlaods

* clean up
2018-08-14 08:22:29 -07:00
Natalie Villasana
6ab323bfec
adds stripe_size field to pointerdb proto file (#218)
* adds erasure_share_size to RS in pointerdb proto

* adds makeErasureScheme helper method to segments store

* cleans up makeErasureScheme function
2018-08-14 11:15:22 -04:00
Philip Hutchins
0440d5fbe6
Adding auto deploy to staging in kube (#219) 2018-08-14 10:00:39 -05:00
JT Olio
c7adc1dfb8
shorter ids (#227)
* shorter ids

* fix test
2018-08-13 17:05:31 -06:00
JT Olio
1412d5a110
fix gw and hc (#225)
* fix gw and hc

* resolve paths

* i keep forgetting gw is different
2018-08-13 12:29:13 -06:00
JT Olio
df726b3ee8
fix services after identity merge (#224)
unfortunately, we can't enable server-side tls yet because
client-side tls hasn't been implemented
2018-08-13 09:39:42 -06:00
JT Olio
1f8db2a4c7
captplanet standalone farmer setup (#202) 2018-08-13 09:33:21 -06:00
JT Olio
ab029301bd
Consolidate command configuration and setup (#221) 2018-08-13 09:07:05 -06:00
Bryan White
5d20cf8829
Node Identity (#193)
* peertls: don't log errors for double close

understood that this part of the code is undergoing heavy change
right now, but just want to make sure this fix gets incorporated
somewhere

* git cleanup: node-id stuff

* cleanup

* rename identity_util.go

* wip `CertificateAuthority` refactor

* refactoring

* gitignore update

* wip

* Merge remote-tracking branch 'storj/doubleclose' into node-id3

* storj/doubleclose:
  peertls: don't log errors for double close

* add peertls tests & gomports

* wip:

+ refactor
+ style changes
+ cleanup
+ [wip] add version to CA and identity configs
+ [wip] heavy client setup

* refactor

* wip:

+ refactor
+ style changes
+ add `CAConfig.Load`
+ add `CAConfig.Save`

* wip:

+ add `LoadOrCreate` and `Create` to CA and Identity configs
+ add overwrite to CA and identity configs
+ heavy client setup
+ refactor
+ style changes
+ cleanup

* wip

* fixing things

* fixing things

* wip hc setup

* hc setup:

+ refactor
+ bugfixing

* improvements based on reveiw feedback

* goimports

* improvements:

+ responding to review feedback
+ refactor

* feedback-based improvements

* feedback-based improvements

* feedback-based improvements

* feedback-based improvements

* feedback-based improvements

* feedback-based improvements

* cleanup

* refactoring CA and Identity structs

* Merge branch 'master' into node-id3

* move version field to setup config structs for CA and identity

* fix typo

* responding to revieiw feedback

* responding to revieiw feedback

* responding to revieiw feedback

* responding to revieiw feedback

* responding to revieiw feedback

* responding to revieiw feedback

* Merge branch 'master' into node-id3

* fix gateway setup finally

* go imports

* fix `FullCertificateAuthority.GenerateIdentity`

* cleanup overlay tests

* bugfixing

* update ca/identity setup

* go imports

* fix peertls test copy/paste fail

* responding to review feedback

* setup tweaking

* update farmer setup
2018-08-13 10:39:45 +02:00
Jennifer Li Johnson
198f7fd506
uncomments oc=globalmockoverlay (#222) 2018-08-10 19:19:19 -04:00