Commit Graph

143 Commits

Author SHA1 Message Date
Egon Elbre
05d148aeb5
Storage node and upload/download protocol refactor (#1422)
refactor storage node server
refactor upload and download protocol
2019-03-18 12:55:06 +02:00
littleskunk
afad4f7010 Improve bwa tests (#1408) 2019-03-08 13:23:43 +01:00
littleskunk
11048bed2c psdb: index on serial number for bandwidth_agreements 2019-03-08 11:53:04 +02:00
Jess G
193a70f0a6
add private listener to grpc server (#1398)
* add private listener to grpc server

* add changes per init CR

* fix server.close

* add insecure grpc connection, update logs msg

* fix tests, move insecure client

* add private ports to storj-sim, add insecure client to other inspectors

* add ports to test so there arent conflicts

* fix lint err

* fix node started log msg, close public listener

* remove commented out line
2019-03-07 13:19:37 -05:00
Egon Elbre
ead4a0ae95
Add new piece id definition (#1411) 2019-03-06 11:17:35 +02:00
aligeti
15034526cd Enhance the Storage Node to keep all BWA's for 90 days (#1374)
* Enhance the Storage Node to keep all BWA's for 90 days

* warn -> error

* rename enum
2019-03-05 17:20:46 -05:00
Jess G
3c9d83dbfe
convert psserver dashboard into an inspector (#1407)
* Convert psserver dashboard into an inspector

* remove dashboard stream, update ps.pb.mock

* fixes for lint errs
2019-03-05 15:48:37 -05:00
Bryan White
675e0ef683 [V3-1320] fix empty node ID verification non-error (#1395)
* small identity refactor:

+ Optimize? iterative cert chain methods to use array instead of slice
+ Add `ToChain` helper for converting 1d to 2d cert chain
  TODO: replace literal declarations with this
+ rename `ChainRaw/RestChainRaw` to `RawChain/RawRestChain`
  (adjective noun, instead of nound adjective)

* add regression tests for V3-1320

* fix V3-1320

* separate `DialUnverifiedIDOption` from `DialOption`

* separate `PingNode` and `DialNode` from `PingAddress` and `DialAddress`

* update node ID while bootstrapping

* goimports & fix comment

* add test case
2019-03-04 15:03:33 -05:00
Michal Niewrzal
cce84c9914
Cleanup errors handling around piece hash verification (#1382) 2019-03-04 15:29:35 +01:00
Stefan Benten
5f57d2c906
Reduce Log Messages on Production Settings (#1396)
* Reduce Tally Log Messages on Production Settings

* Reduce Logging in Piecestore Client as well
2019-03-04 11:12:43 +01:00
Egon Elbre
3f3209c8d5
fixes to piecestore and psdb (#1380)
* replace direct reference with an interface in various places
* hide piecePath
* ensure psserver tests don't use path
* ensure psserver tests don't use sql queries directly
2019-03-01 07:46:16 +02:00
Cameron
018cdd4037
uplink ttl (#1365)
* add expires flag to uplink cp

* fix ttlDBs delete expired query
2019-02-28 12:51:24 -05:00
aligeti
993af7ed37 fixes the bwa deletes to be done by serialnum, rather by signature (#1364) 2019-02-27 18:29:23 +02:00
Egon Elbre
b7f8e309d9 add piecestore.Storage interface (#1369) 2019-02-27 10:56:16 +01:00
Michal Niewrzal
81408a3c9e
Use SignedHash on client/uplink side (#1354)
* psclient receives storage node hash and compare it to own hash for verification
* uplink sends delete request when hashes don't match
* valid hashes are propagated up to segments.Store for future sending to satellite
2019-02-25 16:57:54 +01:00
Michal Niewrzal
6186b3f90a
Storage node hash calculation on upload (#1347)
Storage node is calculating hash of uploaded data and send it back to uplink with signature
2019-02-23 11:46:07 +01:00
Bill Thorp
373b301736
BWA aliases (#1333)
aliased RBAs and PBAs
2019-02-22 16:17:35 -05:00
aligeti
4ccb80257d
Refactor how bandwidth allocations are stored on storage nodes (#1280)
* integrated with db migration framework

* updating the migrationdb test

* updating the migrationdb sql v0 and v1 files

* lint warnings

*  code review changes

* fix migration scripts, upated test data

* fixed the local/UTC timing issues

* code review updates

* updated code review comments
2019-02-22 10:51:39 -05:00
Natalie Villasana
c3d3f41d30 removes some SignedMessage use (#1258)
Removes most instances of pb.SignedMessage (there's more to take out but they shouldn't hurt anyone as is).

There used to be places in psserver where a PieceID was hmac'd with the SatelliteID, which was gotten from a SignedMessage. This PR makes it so some functions access the SatelliteID from the Payer Bandwidth Allocation instead.

This requires passing a SatelliteID into psserver functions where they weren't before, so the following proto messages have been changed:

 * PieceId - satellite_id field added
   This is so the psserver.Piece function has access to the SatelliteID when it needs to get the namespaced pieceID.
   This proto message should probably be renamed to PieceRequest, or a new PieceRequest message should be created so this isn't misnamed.

 * PieceDelete - satellite_id field added
   This is so the psserver.Delete function has access to the SatelliteID when receiving a request to Delete.
2019-02-19 23:36:08 -06:00
Michal Niewrzal
8d685217e4
Storagenode migrations (#1299)
* creates initial migration for psdb
* add test mechanism to validate migration to every version
* fix few small issues in versions.go and context.go
2019-02-19 10:39:04 +01:00
Bill Thorp
c598ed034d kad.FindNear fix (#1320)
We realized that the Kademlia FindNear() function was

1. not using XOR distance (AKA _totally broken_)
2. largely a duplicate of the RoutingTable FindNear() function

Changes in this PR:

1. upgraded RoutingTable FindNear() to use iterator and restrictions
2. removed unneeded RoutingTable interface
3. made Kademlia wrap methods that were previously accessed via RoutingTable
4. fixed the tests
2019-02-15 22:23:35 -05:00
Stefan Benten
661df2a625 Reduce Log Messages on Level Info (#1313) 2019-02-14 13:50:35 +01:00
JT Olio
2a59679766 pkg/transport: require tls configuration for dialing (#1286)
* separate TLS options from server options (because we need them for dialing too)
* stop creating transports in multiple places
* ensure that we actually check revocation, whitelists, certificate signing, etc, for all connections.
2019-02-11 13:17:32 +02:00
Egon Elbre
995927dcd2
Disable psserver TestRetrieve (#1273) 2019-02-08 11:54:50 +02:00
paul cannon
c35b93766d
Unite all cryptographic signing and verifying (#1244)
this change removes the cryptopasta dependency.

a couple possible sources of problem with this change:

 * the encoding used for ECDSA signatures on SignedMessage has changed.
   the encoding employed by cryptopasta was workable, but not the same
   as the encoding used for such signatures in the rest of the world
   (most particularly, on ECDSA signatures in X.509 certificates). I
   think we'll be best served by using one ECDSA signature encoding from
   here on, but if we need to use the old encoding for backwards
   compatibility with existing nodes, that can be arranged.

 * since there's already a breaking change in SignedMessage, I changed
   it to send and receive public keys in raw PKIX format, instead of
   PEM. PEM just adds unhelpful overhead for this case.
2019-02-07 14:39:20 -06:00
Egon Elbre
fdbe2db273
Remove node package and simplify DHT interface (#1233) 2019-02-06 14:37:17 +02:00
Egon Elbre
218e95728c
psserver: swallow io.EOF when getting full content in FileRanger (#1238) 2019-02-06 07:41:41 +02:00
Bill Thorp
690e8b2061
storage node cert cache (#1226)
* draft

* still errors

* double close fix

* added tests

* weird, goimports must not be working

* renames

* missed one

* forgot to save:
2019-02-05 12:57:56 -05:00
Egon Elbre
07412698a9
Use lower bcrypt cost for console.Service when running tests and disable flaky tests (#1236) 2019-02-05 19:31:53 +02:00
Kaloyan Raev
dd76829d10
Improve logic for cutting the long tail during upload (#909) 2019-02-05 12:54:25 +02:00
Egon Elbre
87d6410b50 Revert "Remove node package and simplify DHT interface."
This reverts commit 03ec1ff92d.
2019-02-05 10:38:48 +02:00
Egon Elbre
03ec1ff92d Remove node package and simplify DHT interface. 2019-02-05 10:37:24 +02:00
Bill Thorp
0f662b8e38 sql based tally (#1166)
* WIP

* wacky changes

* more

* it builds.... IT BUILDS!!!

* fixed SQL, broke out saving tallies for testing

* shorter lines

* fixed SQL, moved tally_test to testplanet

* lint

* WIP logic error preventing PUT and GETs to same serialnum

* fixed BWA test

* fixed temporary brain failure

* eliminated magic numbers

* cleaned up satellite uplink stats

* use errs.Combine instead

* thrashing

* fixed tally erroneous error msg

* fixed tally test

* lint

* SQL syntax attempt to fix

* spelling error

* made bwa db test resist old postgres data

* postgres pk error msg hunting

* postgres pk error msg hunting

* postgres

* err might be nil?

* fixed error logging bug

* hopefully solved postgres issue

* using rebind

* moved tests to _test package

* fixing test dirs

* finally made sense of Egons package name feedback

* UTC, array fixes
2019-02-01 13:50:12 -05:00
Kaloyan Raev
5c2b67db63 Fix EOF error on audits (storage node) (#1208)
* Fix EOF error on audits

* Specify the correct number of bytes to CopyN instead of checking for EOF

* Fix imports order
2019-02-01 10:57:11 -07:00
Egon Elbre
d5346982c2
Delete provider package (#1177) 2019-01-30 22:47:21 +02:00
Alexander Leitner
7c741c16d1
Readable sizes in dashboard (#1180)
* Readable sizes

* Use tabwriter

* Added more addresses to dashboard stats

* Use kad to look up addresses

* Move dashboard command to dashboard.go
2019-01-30 14:49:25 -05:00
Egon Elbre
1df81b1460
Separate garbage collect logic from psdb (#1167) 2019-01-29 17:41:01 +02:00
Dylan Lott
bb2588e9ae
Refactors kademlia inspector to use FindNear (#1157)
* WIP

* update identity paths to get dump-nodes working

* Remove commented out code

* remove comment

* updates from code reviews

* updates linter
2019-01-28 15:53:37 -07:00
Bill Thorp
60946c2024
make bandwidth agreements sensible: without []byte's (#1152)
removed []byte's from bandwidth agreement protocol buffers
2019-01-28 14:45:25 -05:00
Egon Elbre
cecd4b0816
Remove server aliases (#1154) 2019-01-28 17:04:53 +02:00
Egon Elbre
49dacb662c
Improve bwagreement tests (#1149) 2019-01-26 01:06:38 +02:00
Bill Thorp
8ba4b61e59
Make Bandwidth Agreements Secure / Trustable (#1117)
* Added cert chains and nodeid verification to bandwidth agreement
2019-01-25 13:05:21 -05:00
Jennifer Li Johnson
856b98997c
updates copyright 2018 to 2019 (#1133) 2019-01-24 15:15:10 -05:00
Alexander Leitner
c4afec9328
quick log cleanup for storage node (#1121)
* Clean up logs
2019-01-24 11:09:37 -05:00
Egon Elbre
5de7f8af7f
Satellite Peer (#1119) 2019-01-23 21:58:44 +02:00
Cameron
95d2d54fc3
storagenode approvedSatelliteIDs (#1116)
* add config fields for satellite restriction on psserver

* add whitelistedSatIDs to psserver Server struct

* check pbwa satellite ID against whitelist

* add whitelist to psserver tests

* reword help message, make approved() a method on server
2019-01-23 12:56:12 -05:00
Egon Elbre
b6c61cdd55
Use storagenode.Peer for storagenode (#1107) 2019-01-23 12:39:03 +02:00
Egon Elbre
99d3b7a3c8
Fix import grouping (#1111) 2019-01-22 17:48:23 +02:00
Alexander Leitner
aabcceef52 Don't create identity when running dash (#1098)
* Don't create identity when running dash

* only use configs for connecting with dash, delete unused client example

* use server address
2019-01-18 18:08:45 +01:00
Dylan Lott
7bc1944a08 Dashboard updates (#1087) 2019-01-18 05:41:25 -05:00