Commit Graph

823 Commits

Author SHA1 Message Date
Michal Niewrzal
5cf89858c2
Add protolock to build (#1321) 2019-02-18 08:43:46 +01:00
Dylan Lott
6d7fe7c7d8
Add uplink tests (#1310)
* initial wireup of test script

* making some progress

* working on getting gateway config to work

* get script working for testing uplink

* removes unnecessary stuff

* more removal of unnecessary change

* remove unnecessary pointerdb set in setup

* fix dest dir in test-aws script
2019-02-13 14:44:36 -07:00
Egon Elbre
60b0275a44
Enable ipv6 tests (#1204) 2019-02-06 14:47:00 +02:00
Egon Elbre
0b6e8f0764 restore previous folder layout in travis (#1240) 2019-02-05 13:09:29 -06:00
Michal Niewrzal
53c11dfc5d
Stop storagenode on ctrl+c (#1220)
* Stop storagenode on ctrl+c

* Cancel grpc server

* handle error

* handle error

* use errgroup

* fix check-travis-tidy

* remove pipefail
2019-02-04 15:50:55 +01:00
Egon Elbre
7163dcc403
Fix typos in check-travis-tidy.sh (#1223) 2019-02-04 15:40:36 +02:00
Egon Elbre
5f59c46ed9
Only run gospace tidy when go.mod or go.sum change (#1214) 2019-02-01 20:11:57 +02:00
Jennifer Li Johnson
856b98997c
updates copyright 2018 to 2019 (#1133) 2019-01-24 15:15:10 -05:00
Bryan White
8edfd43abb
certificates config and --certs-dir fixes (#1093) 2019-01-22 13:35:48 +01:00
Egon Elbre
c89609abbe
Remove captplanet (#1070) 2019-01-17 04:35:22 -05:00
Egon Elbre
9dbb6a101d
Rename storj-sdk to storj-sim (#1078) 2019-01-16 18:09:57 -05:00
Egon Elbre
b86c4dc5eb
add make install-sdk target (#1069) 2019-01-16 13:50:16 -05:00
JT Olio
728908fac4
lockedgen: support nesting for Console (#1060)
Change-Id: Ie340d3e9beca55171c473d8fc990af4305f1025b
2019-01-15 13:29:52 -05:00
Maximillian von Briesen
43ed9dea86
Remove overwrite flags (#1040)
remove overwrite flags from uplink, satellite, storagenode, captplanet, bootstrap

https://storjlabs.atlassian.net/browse/V3-1013
2019-01-14 10:57:58 -05:00
Egon Elbre
10005d869c
Fail integration tests on panic (#1032) 2019-01-11 18:18:16 +02:00
Egon Elbre
12eec57abf
Add arguments to lockedgen for using it in other places (#1030) 2019-01-11 18:07:26 +02:00
Bryan White
9e1ec633d8
Improve cert signing integration test+: (#997) 2019-01-11 15:59:35 +01:00
Egon Elbre
7721c594a2
Fail integration tests on race (#990) 2019-01-08 18:59:09 +02:00
Egon Elbre
387fb14e23
storj-sdk (#969) 2019-01-08 17:24:15 +02:00
Egon Elbre
db5a990719
lint imports grouping (#993) 2019-01-08 16:05:14 +02:00
Bryan White
6372190873
Integration test, travis quick-fix (#974) 2019-01-05 22:24:31 +01:00
Bryan White
b6611e2800
automate certificate signing in storage node setup (#954) 2019-01-04 18:23:23 +01:00
Egon Elbre
5984bc9549
captplanet: use -race flag in script (#970) 2019-01-04 18:38:57 +02:00
Matt Robinson
c0e6b62708
Test all-in-one (#900)
* Add test for aio

* Don't trust the user to have images built for a version

* Make travis run the aio test

* Add missing values to docker-compose, sort some things, consider the gateway image

* today's changes

* config changed, again

* more fixes

* Expose satellite port on localhost:7778

* Add retries and a timeout around the big-testfile test in AIO

* Another config value changed

* Make this error message a little more useful

* Fix nil condition
2019-01-03 14:54:27 -05:00
Bryan White
d8db7c3049
RevocationDB fixes (#866) 2018-12-14 21:45:53 +01:00
Dylan Lott
9bdee7b106
Discovery service (#805)
* WIP possible discovery service impl

* Adds discovery service to CaptPlanet

* Updates the config and server for discovery service

* updates testplanet to use discovery package

* update satellite imports

* Removes unnecessary cache test

* linter fixes

* adds discovery startup to captplanet

* invoke refresh

* updates to discovery refresh cycle

* Make implementation more consistent with previous implementation

* add wait before trying to upload

* sleep a bit more

* remove kademlia bootstrap

* updates

* remove comments
2018-12-14 12:23:54 -07:00
Bryan White
2016ce9fd6
Certificate revocation (#836)
* wip certificate revocation

* refactor tests

* wip testing

* testing

* review fixes

* integration fix attempt #1

* review fixes

* integration fix attempt #2

* linter fixes

* add copywrite

* integration fix attemp #3

* more testing

* more tests

* go mod tidy

* review fixes

* linter fixes
2018-12-13 21:01:43 +01:00
Egon Elbre
ae21a6692b
add wait before trying to upload (#843) 2018-12-12 18:34:20 +02:00
Dylan Lott
ef5adfb65e
fix test-captplanet script for mac (#818) 2018-12-11 11:32:51 -07:00
Egon Elbre
4224be5bb1
install specific gogo (#757) 2018-12-05 11:06:57 +02:00
Matt Robinson
4237afb1c1
Attempt at fixing the all-in-one environment (#736) 2018-12-04 16:10:23 -05:00
Michal Niewrzal
1206ef9126
Use OS related default setup directory (#741)
* Use OS related default setup directory

* added missing comment

* adjust test-captplanet.sh

* ApplicationDir moved to fpath
2018-12-03 16:51:56 +01:00
Egon Elbre
3e461df470
Linting protobuf files (#676) 2018-11-30 17:02:01 +02:00
Egon Elbre
1e4556f88a
Fix import groupings (#739) 2018-11-30 15:40:13 +02:00
Bryan White
2a0c4e60d2
preparing for use of customtype gogo extension with NodeID type (#693)
* preparing for use of `customtype` gogo extension with `NodeID` type

* review changes

* preparing for use of `customtype` gogo extension with `NodeID` type

* review changes

* wip

* tests passing

* wip fixing tests

* more wip test fixing

* remove NodeIDList from proto files

* linter fixes

* linter fixes

* linter/review fixes

* more freaking linter fixes

* omg just kill me - linterrrrrrrr

* travis linter, i will muder you and your family in your sleep

* goimports everything - burn in hell travis

* goimports update

* go mod tidy
2018-11-29 19:39:27 +01:00
Egon Elbre
3a87de391f
Add copyright checks for vue, js and ts (#712) 2018-11-27 14:16:52 +02:00
Dylan Lott
07b4f3d7e3
updates the refresh interval in captplanet test script (#695)
* updates the refresh interval in captplanet test script

* remove sleep 25 line because unnecessary
2018-11-20 16:08:52 -07:00
Egon Elbre
5bcb6fa4fe
Block ports in range 0 to 10000 when running tests (#666) 2018-11-19 17:40:44 +02:00
Egon Elbre
2e0fa0c7e3
Fail tests when they leave garbage behind (#667) 2018-11-16 20:26:13 +02:00
Egon Elbre
42ff835e6e
Fail linter when copyright is missing. (#664) 2018-11-15 14:47:52 +02:00
Egon Elbre
6d401a4351
Check for go.mod validity (#605) 2018-11-09 15:32:35 +02:00
Bryan White
7257079c67
Waitlist script (#573)
* bash refactoring

* add storj.sh and user.sh

* reformat/refactor

* formatting

* fix typo
2018-11-06 18:43:20 +01:00
paul cannon
7ba2fba03b fix test-captplanet.sh script on MacOS/BSD (#575)
only GNU sed understands -i with no argument.
2018-11-06 14:34:22 +02:00
littleskunk
8c740f8390 reproduce repair error (#566) 2018-11-01 12:46:14 -04:00
Bryan White
8b9711cb5e
better waitlist-gating (#557)
* better waitlist-gating

(cherry picked from commit 490fe02b7c3558da18678dfb651c92ec9c4a75b5)

* fix broken test

* linter fixes

* linter fixes

* make extension verification optional

* add certifcate gating script for captplanet

* fixing tests

* linter fixes

* linter fixes?

* moar linter fixes

* Revert "moar linter fixes"

This reverts commit 8139ccbd73cbbead987b7667567844f50f7df2c8.

* just kill me

* refactor

* refactor tests

* liniter...

* cleanup
2018-11-01 16:48:43 +01:00
Egon Elbre
2bb2c50d79
Travis Windows (#519)
* Make Travis run Windows tests
* Simplify git checkout
2018-10-24 14:51:49 +03:00
Dylan Lott
28ee32f322
Updates captainplanet test scripts to use /tmp/capt for test files (#517)
* creates tmp dir and cleans it up after script runs

* updates captplanet test script to use tmpdir

* updates tmpdir to use windows-compatible mktemp
2018-10-19 11:14:29 -06:00
Egon Elbre
51ae3d6e72
Travis with go1.11 and modules (#479)
* Travis uses Go 1.11
* Use go modules instead of storj-vendor
* Automatic caching of downloaded dependencies
* Ensures that modules incompatible linters run with modules
2018-10-16 20:02:00 +03:00
littleskunk
f900167974 reproduce captplanet kademlia problems (#465) 2018-10-12 13:58:32 +03:00
Matt Robinson
a1f93285e2 figure out which deployments need to be patched during a deployment instead of guessing (#438) 2018-10-08 12:52:27 -04:00
Dennis Coyle
dee2c137c8
Remove BKAD dependency from pkg/kademlia (#294)
* slowly but surely

* hardcode ID for tests so we can get predictable results

* skipping bad test

* removing tests of bkad

* wip

* new algorithm for worker

* clean up

* remove skipped test

* changes

* uncomment

* fixed conflicts

* maybe done ?

* cleanup

* boot bkad

* wip

* cleanup

* undo change

* fixes

* wip

* wip

* moving nodeID around

* wip

* wip

* fixes

* fixes after merge

* added TODO

* fixed tests post identity

* linter fixes

* wip

* PR review comments

* wip

* fixing tests

* fix tests

* force db directory

* bad test

* fixes race condition

* small cleanups

* adding db folder

* testing

* wip

* cleanup

* cleanup

* linters

* export Restrict

* add timeout

* testing

* linters

* forgot one

* moar fixes from master merge

* PR comments

* moar PR comments

* removed stun flag

* remove duplicate declaration

* remove old tests

* remove timeout

* fix tests

* missed one

* changed StringToID >> IDFromString

* PR comments

* stupid linter

* moevd overlay mock

* fixed merge conflicts

* fixes

* linter
2018-10-08 11:09:37 -04:00
Maximillian von Briesen
821d0b6f1d
Stream encryption (#302)
* begin adding encryption for remote pieces

* begin adding decryption

* add encryption key as arg to Put and Get

* move encryption/decryption to object store

* Add encryption key to object store constructor

* Add the erasure scheme to object store constructor

* Ensure decrypter is initialized with the stripe size used by encrypter

* Revert "Ensure decrypter is initialized with the stripe size used by encrypter"

This reverts commit 07272333f461606edfb43ad106cc152f37a3bd46.

* Revert "Add the erasure scheme to object store constructor"

This reverts commit ea5e793b536159d993b96e3db69a37c1656a193c.

* move encryption to stream store

* move decryption stuff to stream store

* revert changes in object store

* add encryptedBlockSize and close rangers on error during Get

* calculate padding sizes correctly

* encryptedBlockSize -> encryptionBlockSize

* pass encryption key and block size into stream store

* remove encryption key and block size from object store constructor

* move encrypter/decrypter initialization

* remove unnecessary cast

* Fix padding issue

* Fix linter

* add todos

* use random encryption key for data encryption. Store an encrypted copy of this key in segment metadata

* use different encryption key for each segment

* encrypt data in one step if it is small enough

* refactor and move encryption stuff

* fix errors related to nil slices passed to copy

* fix encrypter vs. decrypter bug

* put encryption stuff in eestream

* get captplanet test to pass

* fix linting errors

* add types for encryption keys/nonces and clean up

* fix tests

* more review changes

* add Cipher type for encryption stuff

* fix rs_test

* Simplify type casting of key and nonce

* Init starting nonce to the segment index

* don't copy derived key

* remove default encryption key; force user to explicitly set it

* move getSegmentPath to streams package

* dont require user to specify encryption key for captplanet

* rename GenericKey and GenericNonce to Key and Nonce

* review changes

* fix linting error

* Download uses the encryption type from metadata

* Store enc block size in metadata and use it for download
2018-09-26 09:32:23 -04:00
Egon Elbre
c65b114941
Add captplanet tests for IPv6 (#371) 2018-09-25 21:23:21 +03:00
Matt Robinson
5c226238f3
Update kubernetes info for staging cluster (#382) 2018-09-25 10:55:31 -04:00
Egon Elbre
8d73b20a63
Basic Multipart support (#351) 2018-09-21 22:44:45 +03:00
Egon Elbre
ebb839da7d
Add ListV2 method (#365) 2018-09-18 18:26:53 +03:00
Egon Elbre
50cd8211c2
Use specific aws-cli version (#364) 2018-09-18 17:30:09 +03:00
Matt Robinson
0b947355da A bunch of small fixes to make OPsy things better (#346) 2018-09-12 11:02:53 -04:00
Jennifer Li Johnson
d0f87f0de1
Overlay Bulk Lookup (#279)
* implements bulk node lookup from overlay cache
2018-09-11 00:52:14 -04:00
Matt Robinson
75c64035a2 Normalize the Docker Bits (#325)
* Normalize the dockerfiles and entrypoints

* fix formattting and docker-compose file

* add missing env var to satellite dockerfile

* Remove build args no longer in use

* More changes to make it all work
2018-09-07 15:46:08 -04:00
Matt Robinson
2c315db84b Fix the binaries and Docker Images (#306)
* Finally fix the binary builds for a couple os/arch combinations

* Make the file executable before adding it to the zip

* This cache hack wasn't suppose to make it into the PR

* Guard against calling the binary target by itself

* Order matters on OS X

* Vendored my image

* No 386 builds until storage is fixed.
2018-09-05 16:40:47 -04:00
Matt Robinson
026997e725
All-in-one docker-compose project (#267)
* First pass at an all-in-one compose project

* Sort out dependencies so uplink can build in docker

* Added the final bits to make this work

* Add readme for all-in-one

* Fix sed to work on OS X and Linux

* Make the readme for all-in-one better

* Only bring up the satellite and uplink after fixing the mock overlay

* Use a dummy value for mock overlay for now

* Finishing touches

* Fix url form for aws cli tool

* Move files out of all-in-one directory to help with directory confusion.

* Added a make target to make all-in-one even better
2018-08-23 11:48:03 -04:00
Natalie Villasana
ff65663867
adds Bucket Store (#213)
* adds foundation for bucketStore

* adds prefixedObjStore to buckets package, adjusts gateway-storj accordingly

* fixes multi value assignment problems in gateway-storj

* fixes more multi value assignment errors in gateway-storj

* starts changing miniogw tests to accommodate buckets

* creates bucket store mock

* wip - fixing test cases in object tests

* adds get, put, and list object tests, comments out two test cases

* adds happy scenario tests for bucket methods

* fixes bug in list, removes redundant parts from gateway tests

* fixes nit

* Clean up tests from #188

* Fix bug with timestamp conversion in segment store

* fixes segments.Meta test

* Fix regression in listing objects in a bucket

* adds check to see if bucket is empty before deleting

* updates DeleteBucket test to account for empty/full bucket

* adds TODOs for DeleteBucket and MakeBucket for some cases, adjusts tests, filters out minio errors in logging.go

* adds checks for if buckets already exist or not in DeleteBucket and MakeBucket functions; adjusts tests

* adds BucketNotFound error check in bucket store, removes todo

* adds make_bucket to Travis test, updates boltdb client constructor to always create a bucket (table)
2018-08-16 10:32:28 -04:00
Maximillian von Briesen
f80ddf9358
Test captplanet with travis (#226)
* test captplanet in travis

* add comparison between uploaded and downloaded file

* test inline and remote files

* move captplanet testing to bash script
2018-08-14 12:58:16 -04:00
Philip Hutchins
0440d5fbe6
Adding auto deploy to staging in kube (#219) 2018-08-14 10:00:39 -05:00
JT Olio
8e7f4f6ebe protos: update protobufs with go generate (#169)
the import for timestamp and duration should use
the path provided by a standard protocol buffer library
installation
2018-07-26 13:20:02 -07:00
Dennis Coyle
2181932e31 Automatically build, tag and push docker images on merge to master (#103)
* port changes

* build overlay on successful merge to master

* fixes to Makefile

* permissions

* dep ensure

* gopath

* let's try vgo

* remove dep

* maybe alpine is the issue

* tagging go version on build

* stupid vgo

* vgo

* adding tags to push

* quotes

* local linting fixes & stupid travis
2018-06-21 15:36:39 -04:00
JT Olds
dd2fbb5069 don't require shallow clones of old commits (#38)
* don't require shallow clones of old commits

doesn't look like github allows it. hints here:
https://stackoverflow.com/questions/31278902/how-to-shallow-clone-a-specific-commit-with-depth-1

* don't try to update submodules
2018-05-09 16:21:53 +03:00
JT Olds
fc7cfde9f7 pkg/miniogw: a minio-based s3 gateway for storj. just stubs right now (#34) 2018-05-08 19:49:54 -04:00
JT Olds
2ef8e581b2 fix travis, and get a clean lint (#32)
* fix travis

* lint fixes

* travis: enable tests

* fix some tests, disable a broken eestream test for now
2018-05-08 18:02:01 -04:00
Dennis Coyle
fe60ba02a3 lint step for copyright headers (#16)
* overlay proto

* grpc server and client

* fix import to storj

* tests

* change imports

* imports cleanup/comments

* PR comments addressed from @jtolds

* lint step for copyright headers

* spelling

* PR comments addressed from @jtolds
2018-04-23 10:05:52 -06:00
Dennis Coyle
1975b98eba grpc server and client 2018-04-16 10:09:01 -06:00
Dennis Coyle
b236d7dffb keep 2018-04-06 12:36:55 -04:00