Commit Graph

95 Commits

Author SHA1 Message Date
Maximillian von Briesen
821d0b6f1d
Stream encryption (#302)
* begin adding encryption for remote pieces

* begin adding decryption

* add encryption key as arg to Put and Get

* move encryption/decryption to object store

* Add encryption key to object store constructor

* Add the erasure scheme to object store constructor

* Ensure decrypter is initialized with the stripe size used by encrypter

* Revert "Ensure decrypter is initialized with the stripe size used by encrypter"

This reverts commit 07272333f461606edfb43ad106cc152f37a3bd46.

* Revert "Add the erasure scheme to object store constructor"

This reverts commit ea5e793b536159d993b96e3db69a37c1656a193c.

* move encryption to stream store

* move decryption stuff to stream store

* revert changes in object store

* add encryptedBlockSize and close rangers on error during Get

* calculate padding sizes correctly

* encryptedBlockSize -> encryptionBlockSize

* pass encryption key and block size into stream store

* remove encryption key and block size from object store constructor

* move encrypter/decrypter initialization

* remove unnecessary cast

* Fix padding issue

* Fix linter

* add todos

* use random encryption key for data encryption. Store an encrypted copy of this key in segment metadata

* use different encryption key for each segment

* encrypt data in one step if it is small enough

* refactor and move encryption stuff

* fix errors related to nil slices passed to copy

* fix encrypter vs. decrypter bug

* put encryption stuff in eestream

* get captplanet test to pass

* fix linting errors

* add types for encryption keys/nonces and clean up

* fix tests

* more review changes

* add Cipher type for encryption stuff

* fix rs_test

* Simplify type casting of key and nonce

* Init starting nonce to the segment index

* don't copy derived key

* remove default encryption key; force user to explicitly set it

* move getSegmentPath to streams package

* dont require user to specify encryption key for captplanet

* rename GenericKey and GenericNonce to Key and Nonce

* review changes

* fix linting error

* Download uses the encryption type from metadata

* Store enc block size in metadata and use it for download
2018-09-26 09:32:23 -04:00
Brandon Iglesias
f1a82e588c updating the reed solomon numbers (#350)
* Update config.go

* updating captplanet

* Fix captplanet setup command for >= 100 nodes
2018-09-25 18:38:37 +03:00
Egon Elbre
8d73b20a63
Basic Multipart support (#351) 2018-09-21 22:44:45 +03:00
Matt Robinson
d2dcd8a136 Revert break for local files (#366) 2018-09-20 14:44:28 -04:00
Alexander Leitner
5a4f5e55ff
Add progress bar (#358)
* Add progress bar

* Turn progress bar on and off

* fix dependencies
2018-09-20 14:41:42 -04:00
Matt Robinson
e1f44ec782 Make the next shirt, better (#362) 2018-09-18 09:26:59 -06:00
Egon Elbre
b6b6111173
Flatten proto definitions into a single package (#360)
* protos: move streams to pb
* protos: move overlay to pb
* protos: move pointerdb to pb
* protos: move piecestore to pb
* fix statdb import naming
2018-09-18 07:39:06 +03:00
Kaloyan Raev
ee5f32b0f7
ConcatRanger closes segment readers as soon as data is read (#349)
* ConcatRanger closes a segment readers as soon as data is read

* Fix linter errors

* Fix data race

* Nicer call to CombineErrors
2018-09-14 10:10:43 -04:00
Stefan Benten
05ce4e5269
Fix path handling (#354) 2018-09-14 15:14:04 +02:00
aligeti
3571886fe7 check uplink setup (#339)
* check uplink setup

* code review comments changes
2018-09-13 13:34:01 -04:00
Egon Elbre
a7d34bea4b
Add missing copyright headers (#352) 2018-09-13 16:30:28 +03:00
Matt Robinson
0b947355da A bunch of small fixes to make OPsy things better (#346) 2018-09-12 11:02:53 -04:00
Michal Niewrzal
e90c2140b8 Show file size for uplink ls command (#344)
* Show file size for uplink ls command

* align size to right

* Fix stream store list method to handle meta.Size flag
2018-09-12 09:03:31 -04:00
Egon Elbre
10d9420f24
command-line benchmark and visualization (#342) 2018-09-11 18:35:46 +03:00
Egon Elbre
e7e2d4d7c9
use golangci-lint and add missing error checks (#340) 2018-09-11 16:13:25 +03:00
Cameron
555e7f1897 CLI commands with bucket store API (#304)
* add mb command

* forgot colon

* add command descriptions

* use utils.ParseURL in commands

* return error message instead of minio.BucketAlreadyExists in mb

* ls command with bucket store functionality

* rb command with bucket store functionality

* rm command with bucket store functionality

* newline

* use print rather than errs for messages, add no buckets messsage

* cp command with bucket store functionality

* remove deprecated getStorjObjects function

* defer utils.LogClose(f) on instead of defer f.Close()

* Check for no buckets after for loop

* add checks for unspecified bucket in bucket store methods

* fix incorrect return types

* add no path error messages in object store methods

* split copy into helpers

* srcObj scheme check in download

* print buckets instead of appending to slice

* check if destObj.Host != srcObj.Host

* better method of handling destination name if not specified

* uplink rename

* final cleanups

* trailing slash fixes

* linting

* more linting

* helpful error messages

* Adjust startAfter after merging #328

* Improve output messages

* Improved error check for empty bucket and path

* No page limit on client side. Rely on server side limit.

* Better time formatting

* Fix paths in recursive list results
2018-09-09 11:31:26 -06:00
JT Olio
e0226790e6
miniogw: simplify output (#330) 2018-09-08 12:53:52 -06:00
Matt Robinson
75c64035a2 Normalize the Docker Bits (#325)
* Normalize the dockerfiles and entrypoints

* fix formattting and docker-compose file

* add missing env var to satellite dockerfile

* Remove build args no longer in use

* More changes to make it all work
2018-09-07 15:46:08 -04:00
JT Olio
8f3398b349
merge uplink and storj (#323)
also remove testing package from import dependency chain
(testing defines some flags)
2018-09-07 09:01:04 -06:00
nfarah86
f58bb3e41d removed bin (#319) 2018-09-06 16:31:50 -04:00
nfarah86
766a82c51f
Create secret key (#309)
* initial aws creds for uplink

* uplink  random key works

* fixed pr per  suggestion
2018-09-06 11:04:41 -07:00
aligeti
75110cf2da
check for unsupported flags while running uplink (#316)
* uplink handle invalid or more than needed arguments

* updated the error reporting

* rebase from master

* check for unsupported flags
2018-09-06 12:43:08 -04:00
Matt Robinson
2c315db84b Fix the binaries and Docker Images (#306)
* Finally fix the binary builds for a couple os/arch combinations

* Make the file executable before adding it to the zip

* This cache hack wasn't suppose to make it into the PR

* Guard against calling the binary target by itself

* Order matters on OS X

* Vendored my image

* No 386 builds until storage is fixed.
2018-09-05 16:40:47 -04:00
aligeti
2686476364
uplink handle invalid or more than needed arguments (#305)
* uplink handle invalid or more than needed arguments

* updated the error reporting
2018-09-05 12:46:48 -04:00
Matt Robinson
0a78e741ec
Build and upload binaries (#296)
* First pass at building binaries

* Finished building images

* Reworked some bits to save binaries in a better place.

* First pass at uploading binaries

* Builds for freebsd fail currently

* Add ignore file for docker builds to not invalidate the cache as often

* Docker image for 1.11-alpine exists now

* Name windows binaries correctly

* The makefile is used in tests.

* Take a swing at updating the tests for 1.11

* Switch to something with glibc so the race detector works.

* Remove unused .PHONY targets
2018-08-31 11:21:44 -04:00
aligeti
743ec65204
hc to satellite rename (#282)
* hc to satellite rename

* rebase with master and fix spelling

* missed out rename

* rebase merge fix
2018-08-29 14:32:41 -04:00
Cameron
ea9f5b4e63
CopyObject (#242)
* add CopyObject method

* use utils.LogClose

* extract common code from GetObject to getObject helper

* remove rr.Range from getObject helper, create helper putObject

* return rr, err in getObject helper

* extract code from PutObject into putObject helper

* remove commented out text

* remove other commented out code

* WIP trying to get storj cp command to work with copyObject

* fix typo in rb and now it works

* use rr.Size() instead of srcInfo.Size

* Revert "WIP trying to get storj cp command to work with copyObject"

This reverts commit e256b9f9a0fda728d41eb5b9d7a98b5446825842.

* add CopyObject test

* rebase and fix merge conflicts

* check error in gateway-storj test

* fix typo
2018-08-29 11:00:05 -04:00
Egon Elbre
23231a29b1
Reduce linter concurrency (#297)
Limit concurrency to reduce memory usage.
2018-08-28 18:00:18 +03:00
Bryan White
746b63f685
CA and identity commands (#235)
* wip ca/ident cmds

* minor improvements and commenting

* combine id and ca commands and add $CONFDIR

* add `NewIdenity` test

* refactor `NewCA` benchmarks

* linter fixes
2018-08-28 00:23:48 +02:00
JT Olio
5f54f2ca8b
captplanet: customizable mock overlay host (#284) 2018-08-27 12:17:19 -06:00
Egon Elbre
0f5a2f4ef5 Enable more linters (#272)
* enable more linters

* Run gofmt -s

* run goimports

* run unconvert

* fix naked return

* fix misspellings

* fix ineffectual assigments

* fix missing declaration

* don't use deprecated grpc.Errof

* check errors in tests

* run gofmt -w -r "assert.Nil(err) -> assert.NoError(err)"

* fix directory permissions

* don't use nil Context

* simplify boolean expressions

* use bytes.Equal instead of bytes.Compare

* merge variable declarations, remove redundant returns

* fix some golint errors

* run goimports

* handle more errors

* delete empty TestMain

* delete empty TestMain

* ignore examples for now

* fix lint errors

* remove unused values

* more fixes

* run gofmt -w -s .

* add more comments

* fix naming

* more lint fixes

* try switching travis to go1.11

* fix unnecessary conversions

* fix deprecated methods

* use go1.10 and disable gofmt/goimports for now

* switch to 1.10

* don't re-enable gofmt and goimports

* switch covermode to atomic because of -race

* gofmt
2018-08-27 11:28:16 -06:00
aligeti
09da23737a
rename farmers to storagenode (#275)
* rename farmers to storagenode

* review changes

* merge conflicts
2018-08-24 21:52:58 -04:00
JT Olio
560e9fd9ec
uplink: make ul/uplink consistent (#286)
* uplink: make ul/uplink consistent

* pstore: make test more robust
2018-08-24 14:02:42 -06:00
aligeti
4edba19986 [Ready for review]Rename gw uplink (#273)
Rename gw uplink
2018-08-24 10:22:24 -06:00
Matt Robinson
026997e725
All-in-one docker-compose project (#267)
* First pass at an all-in-one compose project

* Sort out dependencies so uplink can build in docker

* Added the final bits to make this work

* Add readme for all-in-one

* Fix sed to work on OS X and Linux

* Make the readme for all-in-one better

* Only bring up the satellite and uplink after fixing the mock overlay

* Use a dummy value for mock overlay for now

* Finishing touches

* Fix url form for aws cli tool

* Move files out of all-in-one directory to help with directory confusion.

* Added a make target to make all-in-one even better
2018-08-23 11:48:03 -04:00
JT Olio
ffa6f816d5 gw: support standard configuration via cmdline (#262) 2018-08-22 09:05:16 -04:00
Matt Robinson
5841065420 Attempt to fix jenkins builds (#260)
* Attempt to clean up docker image tags

* Clean only master

* Add back go version and actually use it
2018-08-21 22:54:56 -04:00
JT Olio
fa390c9ec9
Mock overlay move (#245)
* move mock overlay from client to server

this doesn't really change much, but it does allow you to
run a standalone gateway against captain planet. it still does
not allow you to run a standalone gateway against a standalone
heavy client

* pointerdb: small error fixes

* some cleanups

* fix tests
2018-08-20 14:24:11 -04:00
Bryan White
f932c3ced8 Overlay cache commands (#236)
* wip overlay cache cmd

* overlay cache add and list commands

* use zap

* fixes

* simplify

* add restrictions

* cleanup

* remove defaults for overlay management commands
2018-08-20 14:21:41 -04:00
Alexander Leitner
b0db33f919
Bandwidth accounting (#134)
* captplanet standalone farmer setup

* Bandwidth Allocation

* utils.Close method changed to utils.LogClose

* Get build temporarily working

* Get/Put for PSClient should take payer bandwidth allocations rather than the NewPSClient function

* Update example client to reflect changes in client API

* Update ecclient to use latest PSClient, Make NewPSClient return error also

* Updated pieceranger tests to check for errors; sign method should take byte array

* Handle defers in store.go better

* Fix defer functions in psdb.go

* fun times

* Protobuf bandwidthallocation data is now a byte array

* Remove psservice package and merge it into pstore server

* Write wrapper for database calls

* Change all expiration names in protobuf to be more informative; add defer in retrieve; remove old comment

* Make PSDB tests implementation independent rather than method independent

* get rid of payer, renter in ecclient

* add context monitoring in store and retrieve
2018-08-17 13:40:15 -04:00
Cameron
906643fce2
Storj CLT (#201)
* WIP first commit

* change args check

* rename to cli, rename folder to storj, const defaultConfDir, use process.Exec()

* typos

* WIP create helper for getting minio objectlayer, changed setup command to try to match changes in setting up identity

* add rb command

* changes for setting up identity

* remove concurrency field from setupCfg

* rename variable returned by url.Parse to u across all commands for consistency

* add marker functionality to ls command

* add download functionality to cp command

* fix typo in rb init, use short variable names
2018-08-16 14:34:55 -04:00
Philip Hutchins
bfac1297f2
Updating entrypoint and dockerfile (#237) 2018-08-16 14:03:15 -04:00
Matt Robinson
1687ce25c7 Fix the sha3 error with the build (#230)
* Fix the build
2018-08-15 14:49:30 -04:00
JT Olio
1412d5a110
fix gw and hc (#225)
* fix gw and hc

* resolve paths

* i keep forgetting gw is different
2018-08-13 12:29:13 -06:00
JT Olio
df726b3ee8
fix services after identity merge (#224)
unfortunately, we can't enable server-side tls yet because
client-side tls hasn't been implemented
2018-08-13 09:39:42 -06:00
JT Olio
1f8db2a4c7
captplanet standalone farmer setup (#202) 2018-08-13 09:33:21 -06:00
JT Olio
ab029301bd
Consolidate command configuration and setup (#221) 2018-08-13 09:07:05 -06:00
Bryan White
5d20cf8829
Node Identity (#193)
* peertls: don't log errors for double close

understood that this part of the code is undergoing heavy change
right now, but just want to make sure this fix gets incorporated
somewhere

* git cleanup: node-id stuff

* cleanup

* rename identity_util.go

* wip `CertificateAuthority` refactor

* refactoring

* gitignore update

* wip

* Merge remote-tracking branch 'storj/doubleclose' into node-id3

* storj/doubleclose:
  peertls: don't log errors for double close

* add peertls tests & gomports

* wip:

+ refactor
+ style changes
+ cleanup
+ [wip] add version to CA and identity configs
+ [wip] heavy client setup

* refactor

* wip:

+ refactor
+ style changes
+ add `CAConfig.Load`
+ add `CAConfig.Save`

* wip:

+ add `LoadOrCreate` and `Create` to CA and Identity configs
+ add overwrite to CA and identity configs
+ heavy client setup
+ refactor
+ style changes
+ cleanup

* wip

* fixing things

* fixing things

* wip hc setup

* hc setup:

+ refactor
+ bugfixing

* improvements based on reveiw feedback

* goimports

* improvements:

+ responding to review feedback
+ refactor

* feedback-based improvements

* feedback-based improvements

* feedback-based improvements

* feedback-based improvements

* feedback-based improvements

* feedback-based improvements

* cleanup

* refactoring CA and Identity structs

* Merge branch 'master' into node-id3

* move version field to setup config structs for CA and identity

* fix typo

* responding to revieiw feedback

* responding to revieiw feedback

* responding to revieiw feedback

* responding to revieiw feedback

* responding to revieiw feedback

* responding to revieiw feedback

* Merge branch 'master' into node-id3

* fix gateway setup finally

* go imports

* fix `FullCertificateAuthority.GenerateIdentity`

* cleanup overlay tests

* bugfixing

* update ca/identity setup

* go imports

* fix peertls test copy/paste fail

* responding to review feedback

* setup tweaking

* update farmer setup
2018-08-13 10:39:45 +02:00
Philip Hutchins
1f7d77ae69
Fixing Docker entrypoint for storage node (#212) 2018-08-09 12:22:39 -04:00
Philip Hutchins
80c0a0ed2b
Fixing docker entrypoint for satellite (#208) 2018-08-08 19:34:25 -04:00