Commit Graph

279 Commits

Author SHA1 Message Date
JT Olio
ae855fd6eb
Forward-port release-alpha8 build script issues (#1726)
* releases: update alpha dockerfiles to embed release tags

Change-Id: I2978d3ed77d1b768be24ec9194bc404cd92e4700

* release script use plain shell

Change-Id: Ie4ca6633014e3bf00ac0c82e28905174a6ade95b

* removed /scripts from .dockerignore

Change-Id: Ib555236d62fb3dc354689284ac4a032d5b4b3aa8

* switch back to bash

Change-Id: Ie0958cde71cd396f44ac28c177f48df0bea7ba22

* fix go1.12 modules

Change-Id: If4c8364e7271f09febd1a6c8bfc731d723d344d9

* make sure the work tree is clean

Change-Id: Ie6e52af28e2bc52880d92a3b574d1f859fc8e88e
2019-04-09 23:01:10 -06:00
Stefan Benten
2cf86703a3
Add Versioning Server (#1576)
* Initial Webserver Draft for Version Controlling

* Rename type to avoid confusion

* Move Function Calls into Version Package

* Fix Linting and Language Typos

* Fix Linting and Spelling Mistakes

* Include Copyright

* Include Copyright

* Adjust Version-Control Server to return list of Versions

* Linting

* Improve Request Handling and Readability

* Add Configuration File Option
Add Systemd Service file

* Add Logging to File

* Smaller Changes

* Add Semantic Versioning and refuses outdated Software from Startup (#1612)

* implements internal Semantic Version library

* adds version logging + reporting to process

* Advance SemVer struct for easier handling

* Add Accepted Version Store

* Fix Function

* Restructure

* Type Conversion

* Handle Version String properly

* Add Note about array index

* Set temporary Default Version

* Add Copyright

* Adding Version to Dashboard

* Adding Version Info Log

* Renaming and adding CheckerProcess

* Iteration Sync

* Iteration V2

* linting

* made LogAndReportVersion a go routine

* Refactor to Go Routine

* Add Context to Go Routine and allow Operation if Lookup to Control Server fails

* Handle Unmarshal properly

* Linting

* Relocate Version Checks

* Relocating Version Check and specified default Version for now

* Linting Error Prevention

* Refuse Startup on outdated Version

* Add Startup Check Function

* Straighten Logging

* Dont force Shutdown if --dev flag is set

* Create full Service/Peer Structure for ControlServer

* Linting

* Straighting Naming

* Finish VersionControl Service Layout

* Improve Error Handling

* Change Listening Address

* Move Checker Function

* Remove VersionControl Peer

* Linting

* Linting

* Create VersionClient Service

* Renaming

* Add Version Client to Peer Definitions

* Linting and Renaming

* Linting

* Remove Transport Checks for now

* Move to Client Side Flag

* Remove check

* Linting

* Transport Client Version Intro

* Adding Version Client to Transport Client

* Add missing parameter

* Adding Version Check, to set Allowed = true

* Set Default to true, testing

* Restructuring Code

* Uplink Changes

* Add more proper Defaults

* Renaming of Version struct

* Dont pass Service use Pointer

* Set Defaults for Versioning Checks

* Put HTTP Server in go routine

* Add Versioncontrol to Storj-Sim

* Testplanet Fixes

* Linting

* Add Error Handling and new Server Struct

* Move Lock slightly

* Reduce Race Potentials

* Remove unnecessary files

* Linting

* Add Proper Transport Handling

* small fixes

* add fence for allowed check

* Add Startup Version Check and Service Naming

* make errormessage private

* Add Comments about VersionedClient

* Linting

* Remove Checks that refuse outgoing connections

* Remove release cmd

* Add Release Script

* Linting

* Update to use correct Values

* Move vars private and set minimum default versions for testing builds

* Remove VersionedClient

* Better Error Handling and naked return removal

* Straighten the Regex and string conversion

* Change Check to allows testplanet and storj-sim to run without the
need to pass an LDFlag

* Cosmetic Change to Dashboard

* Cleanup Returns and remove commented code

* Remove Version Check if no build options are passed in

* Pass in Config Values instead of Pointers

* Handle missed Error

* Update Endpoint URL

* Change Type of Release Flag

* Add additional Logging

* Remove Versions Logging of other Services

* minor fixes

Change-Id: I5cc04a410ea6b2008d14dffd63eb5f36dd348a8b
2019-04-03 21:13:39 +02:00
Egon Elbre
694b6dc1da
make tests run faster (#1553) 2019-03-22 15:14:17 +02:00
paul cannon
1d78ddc3df
Test that network stalls don't cause indefinite hangs in uplink (#1530)
* test-network-stalls tests... network stalls!

in particular, right now, it just tests whether an uplink correctly
times out after some amount of time when one of the nodes it's talking
to suddenly goes offline.

This tool is meant to be run under `storj-sim network test`.

Also included here:

* fix storj-sim-related test scripts on Mac

the default storj config dir on Mac has a space in it
('~/Library/Application Support/Storj'), which breaks everywhere it
shows up in an unquoted variable in a sh/bash script. easy enough to fix
as a one-off, but quoting bash vars avoids a dozen other potential
problems too.

change a few things using `head -c` to use `dd`. `head -c` works,
but is not as widely understood (as evidenced by each of these scripts
getting through code review, one at a time, with the comments not
matching the numbers actually used).

* storj-sim reports PIDs of worker processes to test

so that the tests can cause unfortunate "accidents" to befall the worker
processes in the course of the test, and find out whether everything
reacts correctly.
2019-03-20 08:58:07 -06:00
Egon Elbre
05d148aeb5
Storage node and upload/download protocol refactor (#1422)
refactor storage node server
refactor upload and download protocol
2019-03-18 12:55:06 +02:00
Jess G
3c9d83dbfe
convert psserver dashboard into an inspector (#1407)
* Convert psserver dashboard into an inspector

* remove dashboard stream, update ps.pb.mock

* fixes for lint errs
2019-03-05 15:48:37 -05:00
Michal Niewrzal
5cf89858c2
Add protolock to build (#1321) 2019-02-18 08:43:46 +01:00
Dylan Lott
6d7fe7c7d8
Add uplink tests (#1310)
* initial wireup of test script

* making some progress

* working on getting gateway config to work

* get script working for testing uplink

* removes unnecessary stuff

* more removal of unnecessary change

* remove unnecessary pointerdb set in setup

* fix dest dir in test-aws script
2019-02-13 14:44:36 -07:00
Egon Elbre
60b0275a44
Enable ipv6 tests (#1204) 2019-02-06 14:47:00 +02:00
Egon Elbre
0b6e8f0764 restore previous folder layout in travis (#1240) 2019-02-05 13:09:29 -06:00
Michal Niewrzal
53c11dfc5d
Stop storagenode on ctrl+c (#1220)
* Stop storagenode on ctrl+c

* Cancel grpc server

* handle error

* handle error

* use errgroup

* fix check-travis-tidy

* remove pipefail
2019-02-04 15:50:55 +01:00
Egon Elbre
7163dcc403
Fix typos in check-travis-tidy.sh (#1223) 2019-02-04 15:40:36 +02:00
Egon Elbre
5f59c46ed9
Only run gospace tidy when go.mod or go.sum change (#1214) 2019-02-01 20:11:57 +02:00
Jennifer Li Johnson
856b98997c
updates copyright 2018 to 2019 (#1133) 2019-01-24 15:15:10 -05:00
Bryan White
8edfd43abb
certificates config and --certs-dir fixes (#1093) 2019-01-22 13:35:48 +01:00
Egon Elbre
c89609abbe
Remove captplanet (#1070) 2019-01-17 04:35:22 -05:00
Egon Elbre
9dbb6a101d
Rename storj-sdk to storj-sim (#1078) 2019-01-16 18:09:57 -05:00
Egon Elbre
b86c4dc5eb
add make install-sdk target (#1069) 2019-01-16 13:50:16 -05:00
JT Olio
728908fac4
lockedgen: support nesting for Console (#1060)
Change-Id: Ie340d3e9beca55171c473d8fc990af4305f1025b
2019-01-15 13:29:52 -05:00
Maximillian von Briesen
43ed9dea86
Remove overwrite flags (#1040)
remove overwrite flags from uplink, satellite, storagenode, captplanet, bootstrap

https://storjlabs.atlassian.net/browse/V3-1013
2019-01-14 10:57:58 -05:00
Egon Elbre
10005d869c
Fail integration tests on panic (#1032) 2019-01-11 18:18:16 +02:00
Egon Elbre
12eec57abf
Add arguments to lockedgen for using it in other places (#1030) 2019-01-11 18:07:26 +02:00
Bryan White
9e1ec633d8
Improve cert signing integration test+: (#997) 2019-01-11 15:59:35 +01:00
Egon Elbre
7721c594a2
Fail integration tests on race (#990) 2019-01-08 18:59:09 +02:00
Egon Elbre
387fb14e23
storj-sdk (#969) 2019-01-08 17:24:15 +02:00
Egon Elbre
db5a990719
lint imports grouping (#993) 2019-01-08 16:05:14 +02:00
Bryan White
6372190873
Integration test, travis quick-fix (#974) 2019-01-05 22:24:31 +01:00
Bryan White
b6611e2800
automate certificate signing in storage node setup (#954) 2019-01-04 18:23:23 +01:00
Egon Elbre
5984bc9549
captplanet: use -race flag in script (#970) 2019-01-04 18:38:57 +02:00
Matt Robinson
c0e6b62708
Test all-in-one (#900)
* Add test for aio

* Don't trust the user to have images built for a version

* Make travis run the aio test

* Add missing values to docker-compose, sort some things, consider the gateway image

* today's changes

* config changed, again

* more fixes

* Expose satellite port on localhost:7778

* Add retries and a timeout around the big-testfile test in AIO

* Another config value changed

* Make this error message a little more useful

* Fix nil condition
2019-01-03 14:54:27 -05:00
Bryan White
d8db7c3049
RevocationDB fixes (#866) 2018-12-14 21:45:53 +01:00
Dylan Lott
9bdee7b106
Discovery service (#805)
* WIP possible discovery service impl

* Adds discovery service to CaptPlanet

* Updates the config and server for discovery service

* updates testplanet to use discovery package

* update satellite imports

* Removes unnecessary cache test

* linter fixes

* adds discovery startup to captplanet

* invoke refresh

* updates to discovery refresh cycle

* Make implementation more consistent with previous implementation

* add wait before trying to upload

* sleep a bit more

* remove kademlia bootstrap

* updates

* remove comments
2018-12-14 12:23:54 -07:00
Bryan White
2016ce9fd6
Certificate revocation (#836)
* wip certificate revocation

* refactor tests

* wip testing

* testing

* review fixes

* integration fix attempt #1

* review fixes

* integration fix attempt #2

* linter fixes

* add copywrite

* integration fix attemp #3

* more testing

* more tests

* go mod tidy

* review fixes

* linter fixes
2018-12-13 21:01:43 +01:00
Egon Elbre
ae21a6692b
add wait before trying to upload (#843) 2018-12-12 18:34:20 +02:00
Dylan Lott
ef5adfb65e
fix test-captplanet script for mac (#818) 2018-12-11 11:32:51 -07:00
Egon Elbre
4224be5bb1
install specific gogo (#757) 2018-12-05 11:06:57 +02:00
Matt Robinson
4237afb1c1
Attempt at fixing the all-in-one environment (#736) 2018-12-04 16:10:23 -05:00
Michal Niewrzal
1206ef9126
Use OS related default setup directory (#741)
* Use OS related default setup directory

* added missing comment

* adjust test-captplanet.sh

* ApplicationDir moved to fpath
2018-12-03 16:51:56 +01:00
Egon Elbre
3e461df470
Linting protobuf files (#676) 2018-11-30 17:02:01 +02:00
Egon Elbre
1e4556f88a
Fix import groupings (#739) 2018-11-30 15:40:13 +02:00
Bryan White
2a0c4e60d2
preparing for use of customtype gogo extension with NodeID type (#693)
* preparing for use of `customtype` gogo extension with `NodeID` type

* review changes

* preparing for use of `customtype` gogo extension with `NodeID` type

* review changes

* wip

* tests passing

* wip fixing tests

* more wip test fixing

* remove NodeIDList from proto files

* linter fixes

* linter fixes

* linter/review fixes

* more freaking linter fixes

* omg just kill me - linterrrrrrrr

* travis linter, i will muder you and your family in your sleep

* goimports everything - burn in hell travis

* goimports update

* go mod tidy
2018-11-29 19:39:27 +01:00
Egon Elbre
3a87de391f
Add copyright checks for vue, js and ts (#712) 2018-11-27 14:16:52 +02:00
Dylan Lott
07b4f3d7e3
updates the refresh interval in captplanet test script (#695)
* updates the refresh interval in captplanet test script

* remove sleep 25 line because unnecessary
2018-11-20 16:08:52 -07:00
Egon Elbre
5bcb6fa4fe
Block ports in range 0 to 10000 when running tests (#666) 2018-11-19 17:40:44 +02:00
Egon Elbre
2e0fa0c7e3
Fail tests when they leave garbage behind (#667) 2018-11-16 20:26:13 +02:00
Egon Elbre
42ff835e6e
Fail linter when copyright is missing. (#664) 2018-11-15 14:47:52 +02:00
Egon Elbre
6d401a4351
Check for go.mod validity (#605) 2018-11-09 15:32:35 +02:00
Bryan White
7257079c67
Waitlist script (#573)
* bash refactoring

* add storj.sh and user.sh

* reformat/refactor

* formatting

* fix typo
2018-11-06 18:43:20 +01:00
paul cannon
7ba2fba03b fix test-captplanet.sh script on MacOS/BSD (#575)
only GNU sed understands -i with no argument.
2018-11-06 14:34:22 +02:00
littleskunk
8c740f8390 reproduce repair error (#566) 2018-11-01 12:46:14 -04:00
Bryan White
8b9711cb5e
better waitlist-gating (#557)
* better waitlist-gating

(cherry picked from commit 490fe02b7c3558da18678dfb651c92ec9c4a75b5)

* fix broken test

* linter fixes

* linter fixes

* make extension verification optional

* add certifcate gating script for captplanet

* fixing tests

* linter fixes

* linter fixes?

* moar linter fixes

* Revert "moar linter fixes"

This reverts commit 8139ccbd73cbbead987b7667567844f50f7df2c8.

* just kill me

* refactor

* refactor tests

* liniter...

* cleanup
2018-11-01 16:48:43 +01:00
Egon Elbre
2bb2c50d79
Travis Windows (#519)
* Make Travis run Windows tests
* Simplify git checkout
2018-10-24 14:51:49 +03:00
Dylan Lott
28ee32f322
Updates captainplanet test scripts to use /tmp/capt for test files (#517)
* creates tmp dir and cleans it up after script runs

* updates captplanet test script to use tmpdir

* updates tmpdir to use windows-compatible mktemp
2018-10-19 11:14:29 -06:00
Egon Elbre
51ae3d6e72
Travis with go1.11 and modules (#479)
* Travis uses Go 1.11
* Use go modules instead of storj-vendor
* Automatic caching of downloaded dependencies
* Ensures that modules incompatible linters run with modules
2018-10-16 20:02:00 +03:00
littleskunk
f900167974 reproduce captplanet kademlia problems (#465) 2018-10-12 13:58:32 +03:00
Matt Robinson
a1f93285e2 figure out which deployments need to be patched during a deployment instead of guessing (#438) 2018-10-08 12:52:27 -04:00
Dennis Coyle
dee2c137c8
Remove BKAD dependency from pkg/kademlia (#294)
* slowly but surely

* hardcode ID for tests so we can get predictable results

* skipping bad test

* removing tests of bkad

* wip

* new algorithm for worker

* clean up

* remove skipped test

* changes

* uncomment

* fixed conflicts

* maybe done ?

* cleanup

* boot bkad

* wip

* cleanup

* undo change

* fixes

* wip

* wip

* moving nodeID around

* wip

* wip

* fixes

* fixes after merge

* added TODO

* fixed tests post identity

* linter fixes

* wip

* PR review comments

* wip

* fixing tests

* fix tests

* force db directory

* bad test

* fixes race condition

* small cleanups

* adding db folder

* testing

* wip

* cleanup

* cleanup

* linters

* export Restrict

* add timeout

* testing

* linters

* forgot one

* moar fixes from master merge

* PR comments

* moar PR comments

* removed stun flag

* remove duplicate declaration

* remove old tests

* remove timeout

* fix tests

* missed one

* changed StringToID >> IDFromString

* PR comments

* stupid linter

* moevd overlay mock

* fixed merge conflicts

* fixes

* linter
2018-10-08 11:09:37 -04:00
Maximillian von Briesen
821d0b6f1d
Stream encryption (#302)
* begin adding encryption for remote pieces

* begin adding decryption

* add encryption key as arg to Put and Get

* move encryption/decryption to object store

* Add encryption key to object store constructor

* Add the erasure scheme to object store constructor

* Ensure decrypter is initialized with the stripe size used by encrypter

* Revert "Ensure decrypter is initialized with the stripe size used by encrypter"

This reverts commit 07272333f461606edfb43ad106cc152f37a3bd46.

* Revert "Add the erasure scheme to object store constructor"

This reverts commit ea5e793b536159d993b96e3db69a37c1656a193c.

* move encryption to stream store

* move decryption stuff to stream store

* revert changes in object store

* add encryptedBlockSize and close rangers on error during Get

* calculate padding sizes correctly

* encryptedBlockSize -> encryptionBlockSize

* pass encryption key and block size into stream store

* remove encryption key and block size from object store constructor

* move encrypter/decrypter initialization

* remove unnecessary cast

* Fix padding issue

* Fix linter

* add todos

* use random encryption key for data encryption. Store an encrypted copy of this key in segment metadata

* use different encryption key for each segment

* encrypt data in one step if it is small enough

* refactor and move encryption stuff

* fix errors related to nil slices passed to copy

* fix encrypter vs. decrypter bug

* put encryption stuff in eestream

* get captplanet test to pass

* fix linting errors

* add types for encryption keys/nonces and clean up

* fix tests

* more review changes

* add Cipher type for encryption stuff

* fix rs_test

* Simplify type casting of key and nonce

* Init starting nonce to the segment index

* don't copy derived key

* remove default encryption key; force user to explicitly set it

* move getSegmentPath to streams package

* dont require user to specify encryption key for captplanet

* rename GenericKey and GenericNonce to Key and Nonce

* review changes

* fix linting error

* Download uses the encryption type from metadata

* Store enc block size in metadata and use it for download
2018-09-26 09:32:23 -04:00
Egon Elbre
c65b114941
Add captplanet tests for IPv6 (#371) 2018-09-25 21:23:21 +03:00
Matt Robinson
5c226238f3
Update kubernetes info for staging cluster (#382) 2018-09-25 10:55:31 -04:00
Egon Elbre
8d73b20a63
Basic Multipart support (#351) 2018-09-21 22:44:45 +03:00
Egon Elbre
ebb839da7d
Add ListV2 method (#365) 2018-09-18 18:26:53 +03:00
Egon Elbre
50cd8211c2
Use specific aws-cli version (#364) 2018-09-18 17:30:09 +03:00
Matt Robinson
0b947355da A bunch of small fixes to make OPsy things better (#346) 2018-09-12 11:02:53 -04:00
Jennifer Li Johnson
d0f87f0de1
Overlay Bulk Lookup (#279)
* implements bulk node lookup from overlay cache
2018-09-11 00:52:14 -04:00
Matt Robinson
75c64035a2 Normalize the Docker Bits (#325)
* Normalize the dockerfiles and entrypoints

* fix formattting and docker-compose file

* add missing env var to satellite dockerfile

* Remove build args no longer in use

* More changes to make it all work
2018-09-07 15:46:08 -04:00
Matt Robinson
2c315db84b Fix the binaries and Docker Images (#306)
* Finally fix the binary builds for a couple os/arch combinations

* Make the file executable before adding it to the zip

* This cache hack wasn't suppose to make it into the PR

* Guard against calling the binary target by itself

* Order matters on OS X

* Vendored my image

* No 386 builds until storage is fixed.
2018-09-05 16:40:47 -04:00
Matt Robinson
026997e725
All-in-one docker-compose project (#267)
* First pass at an all-in-one compose project

* Sort out dependencies so uplink can build in docker

* Added the final bits to make this work

* Add readme for all-in-one

* Fix sed to work on OS X and Linux

* Make the readme for all-in-one better

* Only bring up the satellite and uplink after fixing the mock overlay

* Use a dummy value for mock overlay for now

* Finishing touches

* Fix url form for aws cli tool

* Move files out of all-in-one directory to help with directory confusion.

* Added a make target to make all-in-one even better
2018-08-23 11:48:03 -04:00
Natalie Villasana
ff65663867
adds Bucket Store (#213)
* adds foundation for bucketStore

* adds prefixedObjStore to buckets package, adjusts gateway-storj accordingly

* fixes multi value assignment problems in gateway-storj

* fixes more multi value assignment errors in gateway-storj

* starts changing miniogw tests to accommodate buckets

* creates bucket store mock

* wip - fixing test cases in object tests

* adds get, put, and list object tests, comments out two test cases

* adds happy scenario tests for bucket methods

* fixes bug in list, removes redundant parts from gateway tests

* fixes nit

* Clean up tests from #188

* Fix bug with timestamp conversion in segment store

* fixes segments.Meta test

* Fix regression in listing objects in a bucket

* adds check to see if bucket is empty before deleting

* updates DeleteBucket test to account for empty/full bucket

* adds TODOs for DeleteBucket and MakeBucket for some cases, adjusts tests, filters out minio errors in logging.go

* adds checks for if buckets already exist or not in DeleteBucket and MakeBucket functions; adjusts tests

* adds BucketNotFound error check in bucket store, removes todo

* adds make_bucket to Travis test, updates boltdb client constructor to always create a bucket (table)
2018-08-16 10:32:28 -04:00
Maximillian von Briesen
f80ddf9358
Test captplanet with travis (#226)
* test captplanet in travis

* add comparison between uploaded and downloaded file

* test inline and remote files

* move captplanet testing to bash script
2018-08-14 12:58:16 -04:00
Philip Hutchins
0440d5fbe6
Adding auto deploy to staging in kube (#219) 2018-08-14 10:00:39 -05:00
JT Olio
8e7f4f6ebe protos: update protobufs with go generate (#169)
the import for timestamp and duration should use
the path provided by a standard protocol buffer library
installation
2018-07-26 13:20:02 -07:00
Dennis Coyle
2181932e31 Automatically build, tag and push docker images on merge to master (#103)
* port changes

* build overlay on successful merge to master

* fixes to Makefile

* permissions

* dep ensure

* gopath

* let's try vgo

* remove dep

* maybe alpine is the issue

* tagging go version on build

* stupid vgo

* vgo

* adding tags to push

* quotes

* local linting fixes & stupid travis
2018-06-21 15:36:39 -04:00
JT Olds
dd2fbb5069 don't require shallow clones of old commits (#38)
* don't require shallow clones of old commits

doesn't look like github allows it. hints here:
https://stackoverflow.com/questions/31278902/how-to-shallow-clone-a-specific-commit-with-depth-1

* don't try to update submodules
2018-05-09 16:21:53 +03:00
JT Olds
fc7cfde9f7 pkg/miniogw: a minio-based s3 gateway for storj. just stubs right now (#34) 2018-05-08 19:49:54 -04:00
JT Olds
2ef8e581b2 fix travis, and get a clean lint (#32)
* fix travis

* lint fixes

* travis: enable tests

* fix some tests, disable a broken eestream test for now
2018-05-08 18:02:01 -04:00
Dennis Coyle
fe60ba02a3 lint step for copyright headers (#16)
* overlay proto

* grpc server and client

* fix import to storj

* tests

* change imports

* imports cleanup/comments

* PR comments addressed from @jtolds

* lint step for copyright headers

* spelling

* PR comments addressed from @jtolds
2018-04-23 10:05:52 -06:00
Dennis Coyle
1975b98eba grpc server and client 2018-04-16 10:09:01 -06:00
Dennis Coyle
b236d7dffb keep 2018-04-06 12:36:55 -04:00