Commit Graph

99 Commits

Author SHA1 Message Date
Kaloyan Raev
30dfc2b20c
Remove PointerDB client (#1520) 2019-03-22 11:01:49 +02:00
Bill Thorp
66718cc5e6
Development defaults for configuration (#1430)
added --dev command line option, cfgstruct.DevFlag(), and cfgstruct.SetupFlag()
2019-03-12 08:51:06 -04:00
Bryan White
675e0ef683 [V3-1320] fix empty node ID verification non-error (#1395)
* small identity refactor:

+ Optimize? iterative cert chain methods to use array instead of slice
+ Add `ToChain` helper for converting 1d to 2d cert chain
  TODO: replace literal declarations with this
+ rename `ChainRaw/RestChainRaw` to `RawChain/RawRestChain`
  (adjective noun, instead of nound adjective)

* add regression tests for V3-1320

* fix V3-1320

* separate `DialUnverifiedIDOption` from `DialOption`

* separate `PingNode` and `DialNode` from `PingAddress` and `DialAddress`

* update node ID while bootstrapping

* goimports & fix comment

* add test case
2019-03-04 15:03:33 -05:00
Bryan White
fde0020c68
dial node/address with bad server cert (#1342) 2019-02-26 19:35:16 +01:00
JT Olio
2a59679766 pkg/transport: require tls configuration for dialing (#1286)
* separate TLS options from server options (because we need them for dialing too)
* stop creating transports in multiple places
* ensure that we actually check revocation, whitelists, certificate signing, etc, for all connections.
2019-02-11 13:17:32 +02:00
Kaloyan Raev
dd76829d10
Improve logic for cutting the long tail during upload (#909) 2019-02-05 12:54:25 +02:00
Egon Elbre
4117600f32
Cleanup repository a bit (#1203) 2019-02-01 20:21:00 +02:00
Egon Elbre
d5346982c2
Delete provider package (#1177) 2019-01-30 22:47:21 +02:00
Jennifer Li Johnson
856b98997c
updates copyright 2018 to 2019 (#1133) 2019-01-24 15:15:10 -05:00
Alexander Leitner
aabcceef52 Don't create identity when running dash (#1098)
* Don't create identity when running dash

* only use configs for connecting with dash, delete unused client example

* use server address
2019-01-18 18:08:45 +01:00
Dylan Lott
82b9b54695
storagenode dashboard is working and passing tests locally (#1072)
* storagenode dashboard is working and passing tests locally

* linter fixes

* linter fixes

* moar linter fixes god

* review fixes and updates

* linter fixes
2019-01-16 16:09:13 -05:00
Alexander Leitner
bfde515391
Clean up Storage node setup (#1013)
* Edit config on Setup

* Default to 1TiB storage space and 500GiB bandwidth

* Use human readable formats

* Use memory

* units of 1024 are measured with KiB/MiB etc

* pkg/cfgstruct: allow values to be configured with human readable sizes

Change-Id: Ic4e9ae461516d1d26fb81f6e44c5ac5cfccf777f

* Modify tests

* Removed comments

* More merge conflict stuff resolved

* Fix lint

* test fixin

Change-Id: I3a008206bf03a4446da19f642a2f9c1f9acaae36

* Remove commented code but secretly leave it in the histroy forever

* Move flag definition to struct
2019-01-14 16:19:15 -05:00
Egon Elbre
8893884044
convert piecestorage into a struct (#1024) 2019-01-11 13:26:39 +02:00
Philip Hutchins
06bdcb1915 Added systemd script to service folder at the top level (#973)
* Added systemd script to service folder at the top level

* Added help details to the SystemD script

* Moving service file to examples folder
2019-01-09 12:48:58 +01:00
Jennifer Li Johnson
a2fa5c4c5a Proper NodeType Handling (#873)
* adds enums to nodetype

* updating nodetype todos

* ran pb updates

* reorder nodetypes

* adding checks

* wip

* wip

* wip

* bug in test-captplanet

* wip

* add values to storagenode, satellite, captplanet binaries

* Cleanup

* more cleanup

* wip

* lint

* lint

* wip

* fixes bug

* regenerate protos

Change-Id: Id270212e8c7479e52641058042cf23b5317ab773

* limit node type changes to kademlia

Change-Id: I9c1a6cc4a79e05086627f0fdeb5028c62ce754f4

* dpanic

Change-Id: Id952a2ad13c807ebaea0ec0a875405e267d81c3e

* review comments

Change-Id: I7f9b77ef22779dd012fd490375b136014f51f834
2019-01-02 11:47:34 -07:00
Egon Elbre
cc8cce58dd
statreceiver nitpicking (#926) 2019-01-01 11:41:27 +02:00
JT Olio
362f447d9f
cmd/statreceiver: lua-scriptable stat receiver (#636)
* cmd/statreceiver: lua-scriptable stat receiver

Change-Id: I3ce0fe3f1ef4b1f4f27eed90bac0e91cfecf22d7

* some updates

Change-Id: I7c3485adcda1278fce01ae077b4761b3ddb9fb7a

* more comments

Change-Id: I0bb22993cd934c3d40fc1da80d07e49e686b80dd

* linter fixes

Change-Id: Ied014304ecb9aadcf00a6b66ad28f856a428d150

* catch errors

Change-Id: I6e1920f1fd941e66199b30bc427285c19769fc70

* review feedback

Change-Id: I9d4051851eab18970c5f5ddcf4ff265508e541d3

* errorgroup improvements

Change-Id: I4699dda3022f0485fbb50c9dafe692d3921734ff

* too tricky

the previous thing was better for memory with lots of errors at a time
but https://play.golang.org/p/RweTMRjoSCt is too much of a foot gun

Change-Id: I23f0b3d77dd4288fcc20b3756a7110359576bf44
2018-12-11 11:24:31 -07:00
Maximillian von Briesen
2ab15196d9
Get rid of statdb grpc stuff (#761)
* remove sdbclient

* do not register statdb grpc server/remove apikey from statdb

* start audit service after statdb in captplanet
2018-12-04 13:47:58 -05:00
Egon Elbre
1e4556f88a
Fix import groupings (#739) 2018-11-30 15:40:13 +02:00
Bryan White
2a0c4e60d2
preparing for use of customtype gogo extension with NodeID type (#693)
* preparing for use of `customtype` gogo extension with `NodeID` type

* review changes

* preparing for use of `customtype` gogo extension with `NodeID` type

* review changes

* wip

* tests passing

* wip fixing tests

* more wip test fixing

* remove NodeIDList from proto files

* linter fixes

* linter fixes

* linter/review fixes

* more freaking linter fixes

* omg just kill me - linterrrrrrrr

* travis linter, i will muder you and your family in your sleep

* goimports everything - burn in hell travis

* goimports update

* go mod tidy
2018-11-29 19:39:27 +01:00
Egon Elbre
0ae05cf834
Ensure that we only choose storage nodes (#732) 2018-11-29 16:57:00 +02:00
Michal Niewrzal
8dd669b37f
Bandwidth allocations need method designation (GET/PUT) (#708)
* Bandwidth allocations need method designation (GET/PUT)

* add method comment

* goimports
2018-11-26 19:21:44 +01:00
Maximillian von Briesen
5014a785a0
Node selection 0/4: Update statdb auth/functionality (#698)
* remove api key from statdb server reqs; add statdb UpdateUptime and UpdateAuditSuccess to server

* update api key authentication in statdb server

* add todos for future statdb updates

* add UpdateUptime and UpdateAuditSuccess to statdb server

* fix apikey stuff in config.go and statdb_test.go

* fix tests

* update sdbclient.NewClient call in audit package

* fix UpdateUptime and UpdateAuditSuccess in sdbclient

* set api key from statdb/config.go

* change package for statdb tests

* linter fixes

* remove todo comments

* fix sdbclient err checking

* move validate auth functionality to auth package

* update description for statdb api key

* remove import
2018-11-26 12:08:29 -05:00
Bryan White
54ccb460a9
rename size fields in protobuf messages for gogo compatibility (#690)
* rename `size` fields in protobuf messages for gogo compatibility

* linter fixes
2018-11-20 18:09:35 +01:00
Maximillian von Briesen
2c36dcc2c8
Update statdb (#650)
* add sdbclient.UpdateUptime; update args for sdbclient.CreateEntryIfNotExists

* add auditcount to node stats; restructure statdb.CreateEntryIfNotExists

* add noop mock sdbclient

* add the ability to create a node in statdb without "default" stats

* update statdb.CreateEntryIfNotExists

* take fewer args for sdbclient.CreateWithStats/FindValidNodes

* add sdbclient.UpdateAuditSuccess

* update sdbclient.Update so that all fields are updated when called (reduce args)

* update error checking in statdb.Create
2018-11-14 19:03:19 -05:00
Dennis Coyle
e4baec8fce
Coyle/tls verify (#612)
* add nodeID to dial opts

* fix tests to use real identity
2018-11-09 17:08:33 -05:00
Alexander Leitner
320c93368a
Add creation time to Payer Bandwidth Allocation (#581)
* Add creation time to Payer Bandwidth Allocation

* goimports
2018-11-08 11:04:52 -05:00
Bryan White
ee62e2a9d8
Use transport client and cleanup all the clients (#574)
* wip

* linter fixes

* linter fixes

* test fixes

* linter fixes

* fix merge + restructure piecestore packages

* review feedback

* linter fixes

* linter fixes

* remove unnecessary aliases to piecestore

* more merge fixing
2018-11-06 18:49:17 +01:00
Michal Niewrzal
deb015970d
Use Nodes array from pdb.Get (#578)
* Use Nodes array from pdb.Get

* fix problems with captplanet tests

* better comments
2018-11-06 18:03:11 +01:00
aligeti
a760e48d4d
Satellite bw usage v3 121 (#547)
* protobuf for sending bandwidth agreements to satellite from storage nodes

* Setup process for sending agreements

* Add payer_id to db with bandwidth agreements for better sorting

* Linter errors

* Read agreements from PSDB

* Try writing message to server

* Cleanup

* Basic functionality

* Better error handelling

* Fix test

* setup config and server structure for receiving bandwidth agreements

* Resolve linter issues

* Optional commit for if we want to handle deletes all at once

* add identity to Server, add logic for receiving bandwidth messsages

* Bandwidth agreement DBX creation and integration with bw agreement endpoint

Co-authored-by: Kishore <kishore@storj.io>
Co-authored-by: Cam <cameron@storj.io>

* protobuf for sending bandwidth agreements to satellite from storage nodes

* Setup process for sending agreements

* Add payer_id to db with bandwidth agreements for better sorting

* Linter errors

* Read agreements from PSDB

* Try writing message to server

* Cleanup

* Basic functionality

* Better error handelling

* Fix test

* setup config and server structure for receiving bandwidth agreements

* Resolve linter issues

* Optional commit for if we want to handle deletes all at once

* add identity to Server, add logic for receiving bandwidth messsages

* Bandwidth agreement DBX creation and integration with bw agreement endpoint

Co-authored-by: Kishore <kishore@storj.io>
Co-authored-by: Cam <cameron@storj.io>

* added postgres create/read/delete test function

Co-authored-by: kishore <kishore@storj.io
Co-authored-by: cam <cameron@storj.io>

* edit comment

* removed sqlite3 driver from dbx

* remove generated sqlite code, add dbx read limitoffset

* remove getServerAndDB function, rename getDBPath to getPSQLInfo

* WIP writing server endpoint test

* code review changes
2018-11-05 10:23:54 -05:00
Bryan White
8b9711cb5e
better waitlist-gating (#557)
* better waitlist-gating

(cherry picked from commit 490fe02b7c3558da18678dfb651c92ec9c4a75b5)

* fix broken test

* linter fixes

* linter fixes

* make extension verification optional

* add certifcate gating script for captplanet

* fixing tests

* linter fixes

* linter fixes?

* moar linter fixes

* Revert "moar linter fixes"

This reverts commit 8139ccbd73cbbead987b7667567844f50f7df2c8.

* just kill me

* refactor

* refactor tests

* liniter...

* cleanup
2018-11-01 16:48:43 +01:00
Alexander Leitner
db80358f32
bandwidth agreements protobuf (#505)
* protobuf for sending bandwidth agreements to satellite from storage nodes

* Setup process for sending agreements

* Add payer_id to db with bandwidth agreements for better sorting

* Renamed payer to satellite in psdb
2018-10-31 14:47:25 -04:00
Alexander Leitner
7e4149138a
Piecestore proto change (#556)
* add action and storage node id to piecestore.proto. renamed payer and renter to satellite_id and uplink_id
2018-10-30 15:03:41 -04:00
Kaloyan Raev
99640225fd
Refactor Path type (#522)
The old paths.Path type is now replaced with the new storj.Path.

storj.Path is simply an alias to the built-in string type. As such it can be used just as any string, which simplifies a lot working with paths. No more conversions paths.New and path.String().

As an alias storj.Path does not define any methods. However, any functions applying to strings (like those from the strings package) gracefully apply to storj.Path too. In addition we have a few more functions defined:

    storj.SplitPath
    storj.JoinPaths
    encryption.EncryptPath
    encryption.DecryptPath
    encryption.DerivePathKey
    encryption.DeriveContentKey

All code in master is migrated to the new storj.Path type.

The Path example is also updated and is good for reference: /pkg/encryption/examples_test.go

This PR also resolve a nonce misuse issue in path encryption: https://storjlabs.atlassian.net/browse/V3-545
2018-10-25 23:28:16 +03:00
Kaloyan Raev
ac4805a841
Adapt storj and encryption packages (#513) 2018-10-19 16:38:13 +03:00
Kaloyan Raev
b34adc310b
Extract encryption package from eestream (#497) 2018-10-18 14:10:29 +03:00
Michal Niewrzal
f9248c21d4
Satellite verification on storage node (#469)
* Satellite verification on storage node

* fix formatting

* fix formatting

* rename SignatureAuth to SignedMessage

* fixes after review

* fix linter errors

* improve errors handling

* remove SignedMessageProvider

* fix liter errors

* params changed to authorization, signed message in audit, minor fixes

* fix formatting
2018-10-17 13:40:11 +02:00
Natalie Villasana
1687111cca
adds Reporter to the audit package to record failed audits (#467) 2018-10-16 13:40:34 -04:00
Egon Elbre
2dbada73ab
Separate storj.io packages in imports (#481)
Format with goimports -w -local storj.io
Add flag to golang lint
Remove commented out imports as they mess with goimports
2018-10-16 14:43:44 +03:00
aligeti
2b2e0996b0 updated based on review comments (#456) 2018-10-15 15:59:05 -04:00
Michal Niewrzal
ad327bedb1
Use context to propagate API Key (#383)
* Satellite signs proofs

* wip

* remove direct apikey usage from pdbclient

* adjusting unit tests

* fix linter errors

* unit tests

* linter errors

* remove usless interface

* remove unused code

* improve unit tests

* signature generation

* code review changes

* code review comments

* back to satellite-id signature generation

* remove go-grpc-middlewar dependency

* small step back

* linter fixes

* fix tests

* packages reorganization

* Move TestAPIKeyInjector to grpcauth package
2018-10-09 16:39:14 +02:00
Maximillian von Briesen
81b106fb72 Add StatDB to Satellite (#396) 2018-10-08 18:15:54 -04:00
Egon Elbre
d10554eb8c
remove udp forwarder (#404) 2018-10-04 15:08:24 +03:00
Egon Elbre
c3a1d71616
examples: turn paths binary to Go documentation example (#405)
* examples: turn paths binary to Go documentation example
* remove bip39 dependency
2018-10-03 15:15:54 +03:00
Egon Elbre
ec0850fa9f
examples: piecestore-cli use cobra directly (#409) 2018-10-02 17:48:09 +03:00
Egon Elbre
dac03acbc4
examples: piecestore-client, use cobra directly (#410) 2018-10-02 17:47:39 +03:00
Egon Elbre
67784e0667
Lint examples folder (#391) 2018-09-28 21:08:44 +03:00
Kaloyan Raev
f94c584768
Rename: DecodedBlockSize>StripeSize, EncodedBlockSize>ErasureShareSize (#392) 2018-09-27 14:52:18 +03:00
Maximillian von Briesen
821d0b6f1d
Stream encryption (#302)
* begin adding encryption for remote pieces

* begin adding decryption

* add encryption key as arg to Put and Get

* move encryption/decryption to object store

* Add encryption key to object store constructor

* Add the erasure scheme to object store constructor

* Ensure decrypter is initialized with the stripe size used by encrypter

* Revert "Ensure decrypter is initialized with the stripe size used by encrypter"

This reverts commit 07272333f461606edfb43ad106cc152f37a3bd46.

* Revert "Add the erasure scheme to object store constructor"

This reverts commit ea5e793b536159d993b96e3db69a37c1656a193c.

* move encryption to stream store

* move decryption stuff to stream store

* revert changes in object store

* add encryptedBlockSize and close rangers on error during Get

* calculate padding sizes correctly

* encryptedBlockSize -> encryptionBlockSize

* pass encryption key and block size into stream store

* remove encryption key and block size from object store constructor

* move encrypter/decrypter initialization

* remove unnecessary cast

* Fix padding issue

* Fix linter

* add todos

* use random encryption key for data encryption. Store an encrypted copy of this key in segment metadata

* use different encryption key for each segment

* encrypt data in one step if it is small enough

* refactor and move encryption stuff

* fix errors related to nil slices passed to copy

* fix encrypter vs. decrypter bug

* put encryption stuff in eestream

* get captplanet test to pass

* fix linting errors

* add types for encryption keys/nonces and clean up

* fix tests

* more review changes

* add Cipher type for encryption stuff

* fix rs_test

* Simplify type casting of key and nonce

* Init starting nonce to the segment index

* don't copy derived key

* remove default encryption key; force user to explicitly set it

* move getSegmentPath to streams package

* dont require user to specify encryption key for captplanet

* rename GenericKey and GenericNonce to Key and Nonce

* review changes

* fix linting error

* Download uses the encryption type from metadata

* Store enc block size in metadata and use it for download
2018-09-26 09:32:23 -04:00
aligeti
97cbacb299
storage node quick check and startup validation (#368)
* storage node quick check and startup validation

*  rearranged the startup validation and quick check logic

* travis lint warning fixes

* travis lint warning fixes

* travis lint warning fixes

* code changes per review comments

* code clean dev debug info

* travis lint wranings

* code changes per code review comments

* code changes per code review comments

* code update per review

* sqlite SUM is having issue when getting the SUM of an empty column; filepath was checking a directory that doesn't exist when starting server; Example updated to print allocated and used space

* storage node quick check and startup validation

*  rearranged the startup validation and quick check logic

* travis lint warning fixes

* travis lint warning fixes

* travis lint warning fixes

* code changes per review comments

* code clean dev debug info

* travis lint wranings

* code changes per code review comments

* code changes per code review comments

* code update per review

* no file or directory error

* Updated mock PSClient
2018-09-25 12:49:55 -04:00