Commit Graph

509 Commits

Author SHA1 Message Date
Michal Niewrzal
dfc52e4394 [WIP] Piece Rangers should connect lazily to storage nodes (second approach) (#321)
* Piece Rangers should connect lazily to storage nodes

* code reorganization

* fixed unit tests

* remove if
2018-09-08 16:52:19 +03:00
Matt Robinson
75c64035a2 Normalize the Docker Bits (#325)
* Normalize the dockerfiles and entrypoints

* fix formattting and docker-compose file

* add missing env var to satellite dockerfile

* Remove build args no longer in use

* More changes to make it all work
2018-09-07 15:46:08 -04:00
JT Olio
8f3398b349
merge uplink and storj (#323)
also remove testing package from import dependency chain
(testing defines some flags)
2018-09-07 09:01:04 -06:00
JT Olio
1ea3f9f03b eestream: better error reporting when downloads fail (#295)
* eestream: better error reporting when downloads fail

* Fix failing tests

* Make the check in hasEnoughShares a little bit cleaner
2018-09-07 10:37:13 -04:00
Egon Elbre
0a2839b2d7
Add IsPrefix support to piecestore (#313) 2018-09-07 17:20:15 +03:00
Egon Elbre
706a6ac26e
Use better test ports (#311)
Other interface or ports may be blocked or require permissions when running tests.
2018-09-07 12:42:04 +03:00
Egon Elbre
79354bf0ab
Enforce lookup limit for storage (#312) 2018-09-07 12:00:00 +03:00
Egon Elbre
ff57fa5c00
use better redis server setup with fallback (#310) 2018-09-07 10:33:33 +03:00
JT Olio
8da996d61a infectious: pull in optimizations (#320) 2018-09-07 10:32:54 +03:00
Egon Elbre
8a4151f397
Fix Redis Iterators (#318)
Redis SCAN may return same item multiple times, add a basic check to avoid duplicate entries.
2018-09-06 23:50:22 +03:00
nfarah86
f58bb3e41d removed bin (#319) 2018-09-06 16:31:50 -04:00
nfarah86
766a82c51f
Create secret key (#309)
* initial aws creds for uplink

* uplink  random key works

* fixed pr per  suggestion
2018-09-06 11:04:41 -07:00
aligeti
75110cf2da
check for unsupported flags while running uplink (#316)
* uplink handle invalid or more than needed arguments

* updated the error reporting

* rebase from master

* check for unsupported flags
2018-09-06 12:43:08 -04:00
Brandon Iglesias
f168e19fb4 adding mniewrzal to the CLA bot (#315)
* adding mniewrzal to the CLA bot

* adding stefanbenten
2018-09-06 17:23:57 +03:00
Matt Robinson
2c315db84b Fix the binaries and Docker Images (#306)
* Finally fix the binary builds for a couple os/arch combinations

* Make the file executable before adding it to the zip

* This cache hack wasn't suppose to make it into the PR

* Guard against calling the binary target by itself

* Order matters on OS X

* Vendored my image

* No 386 builds until storage is fixed.
2018-09-05 16:40:47 -04:00
JT Olio
a4d1070d68
storage/ec: fix some Get socket leaks (#308)
* storage/ec: fix some Get socket leaks

* fix linters
2018-09-05 10:08:39 -07:00
aligeti
2686476364
uplink handle invalid or more than needed arguments (#305)
* uplink handle invalid or more than needed arguments

* updated the error reporting
2018-09-05 12:46:48 -04:00
Egon Elbre
83df0ee1b0
Implement ListV2 with storage rework (#303)
1. Added KeyValueStore.Iterate for implementing the different List, ListV2 etc. implementations. This allows for more efficient use of memory depending on the situation.
2. Implemented an inmemory teststore for running tests. This should allow to replace MockKeyValueStore in most places.
3. Rewrote tests
4. Pulled out logger from bolt implementation so it can be used for all other storage implementations.
5. Fixed multiple things in bolt and redis implementations.
2018-09-05 19:10:35 +03:00
Cameron
6251b8e090 extract bucketstore code from NewGateway into helper (#300)
* extract bucketstore code from NewGateway into helper

* fix lines breaks
2018-09-05 19:03:20 +03:00
Egon Elbre
7ecca9988a
add reed-solomon encoding benchmark (#307) 2018-09-05 17:33:20 +03:00
Matt Robinson
0a78e741ec
Build and upload binaries (#296)
* First pass at building binaries

* Finished building images

* Reworked some bits to save binaries in a better place.

* First pass at uploading binaries

* Builds for freebsd fail currently

* Add ignore file for docker builds to not invalidate the cache as often

* Docker image for 1.11-alpine exists now

* Name windows binaries correctly

* The makefile is used in tests.

* Take a swing at updating the tests for 1.11

* Switch to something with glibc so the race detector works.

* Remove unused .PHONY targets
2018-08-31 11:21:44 -04:00
aligeti
743ec65204
hc to satellite rename (#282)
* hc to satellite rename

* rebase with master and fix spelling

* missed out rename

* rebase merge fix
2018-08-29 14:32:41 -04:00
JT Olio
2bba44fa1c
Update README.md (#299)
* Update README.md

* cla mention

* fix a-names

* fix a-names again
2018-08-29 09:28:29 -06:00
Cameron
ea9f5b4e63
CopyObject (#242)
* add CopyObject method

* use utils.LogClose

* extract common code from GetObject to getObject helper

* remove rr.Range from getObject helper, create helper putObject

* return rr, err in getObject helper

* extract code from PutObject into putObject helper

* remove commented out text

* remove other commented out code

* WIP trying to get storj cp command to work with copyObject

* fix typo in rb and now it works

* use rr.Size() instead of srcInfo.Size

* Revert "WIP trying to get storj cp command to work with copyObject"

This reverts commit e256b9f9a0fda728d41eb5b9d7a98b5446825842.

* add CopyObject test

* rebase and fix merge conflicts

* check error in gateway-storj test

* fix typo
2018-08-29 11:00:05 -04:00
Egon Elbre
23231a29b1
Reduce linter concurrency (#297)
Limit concurrency to reduce memory usage.
2018-08-28 18:00:18 +03:00
Bryan White
746b63f685
CA and identity commands (#235)
* wip ca/ident cmds

* minor improvements and commenting

* combine id and ca commands and add $CONFDIR

* add `NewIdenity` test

* refactor `NewCA` benchmarks

* linter fixes
2018-08-28 00:23:48 +02:00
JT Olio
58c810519f
cross-compilation: fix darwin (#293)
* cross-compilation: fix darwin

* minio without fsnotify

* update Gopkg.lock

* make release target
2018-08-27 15:06:55 -06:00
Alexander Leitner
899e1e68f1
Add functions for signing and verifying during bandwidth exchange (#246)
* Added initial functions for signing and verifying

* whoops

* Get client up to speed

* Added initial functions for signing and verifying

* whoops

* Get client up to speed

* wip

* wip

* actual signatures in tests

(cherry picked from commit 1464853b737f1d712d64fbf90147f535525c8fd9)

* bugfixing

* Generate private key in example

* Generate signatures for pieceranger tests

* Update examples to use TLS

* Use private key from identity inside of example

* Use crypto.PrivateKey interface

* Change err name in defers

* Pass tests

* Pass identity Key to PSClient

* Get tests passing on travis

* Resolve linter complaints
2018-08-27 14:35:27 -04:00
JT Olio
75de358740
grpc-debug tool (#283)
* grpc-debug tool

* review comments
2018-08-27 12:32:03 -06:00
JT Olio
5f54f2ca8b
captplanet: customizable mock overlay host (#284) 2018-08-27 12:17:19 -06:00
Egon Elbre
0f5a2f4ef5 Enable more linters (#272)
* enable more linters

* Run gofmt -s

* run goimports

* run unconvert

* fix naked return

* fix misspellings

* fix ineffectual assigments

* fix missing declaration

* don't use deprecated grpc.Errof

* check errors in tests

* run gofmt -w -r "assert.Nil(err) -> assert.NoError(err)"

* fix directory permissions

* don't use nil Context

* simplify boolean expressions

* use bytes.Equal instead of bytes.Compare

* merge variable declarations, remove redundant returns

* fix some golint errors

* run goimports

* handle more errors

* delete empty TestMain

* delete empty TestMain

* ignore examples for now

* fix lint errors

* remove unused values

* more fixes

* run gofmt -w -s .

* add more comments

* fix naming

* more lint fixes

* try switching travis to go1.11

* fix unnecessary conversions

* fix deprecated methods

* use go1.10 and disable gofmt/goimports for now

* switch to 1.10

* don't re-enable gofmt and goimports

* switch covermode to atomic because of -race

* gofmt
2018-08-27 11:28:16 -06:00
Dylan Lott
7e52c81ebc Segment validation (#280)
* WIP adding inline segment size validation

* add better error handling

* updates error handling and removes unnecessary logs

* update minimum inline size check

* update check if condition and inline min config type
2018-08-27 10:04:46 -04:00
JT Olio
2e5b54a435
go 1.11! woohoo! simplify readme (#288)
* go 1.11! woohoo! simplify readme

* gopath comment

* more clarification
2018-08-26 10:38:38 -06:00
nfarah86
5f46a0696f
writing definition for LISTv2 (#289)
* writing definition for LISTv2

* fixed method to call storage.xx

* regenerate kvstore file

* fixed minor error in storage

* added panic

* added panic
2018-08-25 20:00:49 -07:00
aligeti
09da23737a
rename farmers to storagenode (#275)
* rename farmers to storagenode

* review changes

* merge conflicts
2018-08-24 21:52:58 -04:00
JT Olio
3154086e58 go1.11: update monotime (#287) 2018-08-24 20:23:09 -04:00
JT Olio
560e9fd9ec
uplink: make ul/uplink consistent (#286)
* uplink: make ul/uplink consistent

* pstore: make test more robust
2018-08-24 14:02:42 -06:00
aligeti
4edba19986 [Ready for review]Rename gw uplink (#273)
Rename gw uplink
2018-08-24 10:22:24 -06:00
Kaloyan Raev
7d1bd090f8
Optimize DecodeReader performance (#256)
* Optimize DecodeReader performance

* A little bit better locking in PieceBuffer.Write

* Fix race issues

* Better fix for race condition in rs_test.go

* Improve PieceBuffer.Read to read the max available in one call

* PieceBuffer.Skip for more efficient discarding of old shares

* Rename bytesRead to nn

* Notify cvNewData only if a complete new share is available

* Small correction in PieceBuffer.Read

* Rename some fields to have longer names
2018-08-24 16:06:27 +03:00
Maximillian von Briesen
553def620a TLS Support (#252)
* begin adding tls

* remove incomplete line in gw/main.go

* identity fixes+:

+ fix `peertls.NewCert` public key issue
+ fix `peertls.verfiyChain` issue
+ fix identity dial option
+ rename `GenerateCA` to `NewCA` and `generateCAWorker` to `newCAWorker` for better consistency/convention

* use pdbclient instead of pointerdb in miniogw

* fix tests

* go fmt

* make review changes

* modify how context.Background() is used

* more context stuff
2018-08-23 22:01:03 -06:00
James Hagans
fdfa6e85c8 Streamstore impl of Stor interface (#191)
* first stab at PUT

* only PUT

* working on PUT

* Put with LimitReader

* start of Get

* reorder of files and proto meta

* working on Meta

* working on Meta

* add aware limit reader

* add size from segment put

* rm if for eof

* update to proto meta

* update gen proto file

* working on get

* working on get

* working on get

* working on list

* working on delete

* working on list

* working on meta method

* fix merge error and working on feedback from PR

* update to proto file

* rm size tuple

* mv eof limit reader to new file

* add toMeta

* rm varible names

* add updates from PR feedback

* updates from PR feedback

* updates from PR feedback

* add toMeta size based on total size

* update toMeta size calculation

* rm passthrough

* add default to config for segment size

* fix get method ranger bug

* add object support for nested stream proto

* rm nested stream meta data

* rm test for another PR
2018-08-23 21:56:38 -06:00
Matt Robinson
3456a127cf
More fixes to the staging deploy process (#278) 2018-08-23 17:17:47 -04:00
Matt Robinson
631331236e
Fix staging deploy of satellite and storage-node (#277) 2018-08-23 17:04:51 -04:00
Matt Robinson
026997e725
All-in-one docker-compose project (#267)
* First pass at an all-in-one compose project

* Sort out dependencies so uplink can build in docker

* Added the final bits to make this work

* Add readme for all-in-one

* Fix sed to work on OS X and Linux

* Make the readme for all-in-one better

* Only bring up the satellite and uplink after fixing the mock overlay

* Use a dummy value for mock overlay for now

* Finishing touches

* Fix url form for aws cli tool

* Move files out of all-in-one directory to help with directory confusion.

* Added a make target to make all-in-one even better
2018-08-23 11:48:03 -04:00
Jennifer Li Johnson
154a1eab3a
implements node pkg query method (#251) 2018-08-23 11:20:11 -04:00
Bryan White
0e2fd97029
Identity fixes+: (#270)
* identity fixes+:

+ fix `peertls.NewCert` public key issue
+ fix `peertls.verfiyChain` issue
+ fix identity dial option
+ rename `GenerateCA` to `NewCA` and `generateCAWorker` to `newCAWorker` for better consistency/convention
2018-08-23 16:08:26 +02:00
Egon Elbre
f80185a807
Preliminary support for windows (#266)
* Don't use url.Parse for bolt paths: filepaths may not be valid URL-s.
* go.mod: update dependencies
* README.md: add Windows instructions
* pkg/overlay: check for the correct path and text in error
* pkg/overlay: fix tests for windows
* pkg/piecestore: make windows tests pass
* pkg/telemetry: skip test, as it doesn't shutdown nicely
* storage/redis: ensure that redis is clean before running tests
2018-08-23 10:05:56 +03:00
Dylan Lott
ce3f9c6699
Fix nil bucket error (#269) 2018-08-22 18:19:46 -06:00
JT Olio
9aee5efd99
pointerdb: separate client and server packages (#261)
* pointerdb: separate client and server packages

the reason for this is so that things the server needs (bolt, auth)
don't get imported if all you need is the client. will result in
smaller binaries and less flag definitions

* review comments
2018-08-22 09:07:00 -06:00
Philip Hutchins
5279283b39 Adding some logging to Jenkinsfile for debug (#265) 2018-08-22 09:18:50 -04:00