JakeHillion/storj
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
1,636 Commits 5 Branches 0 Tags 94 MiB
268dc6b7e4
Commit Graph

960 Commits

Author SHA1 Message Date
Egon Elbre
de15a4fdcf
remove utils.CombineErrors and utils.ErrorGroup (#1603) 2019-03-29 14:30:23 +02:00
Kaloyan Raev
ddce5a4bf8
Fix incorrect usage of NodeID and ErrNodeID in PieceID impl (#1601) 2019-03-29 13:13:03 +02:00
Kaloyan Raev
f9ba935286
Merge overlay_cache_nodes into nodes table (#1581) 2019-03-29 10:53:43 +02:00
Egon Elbre
be06fdfd6c Create orders.Service (#1593) 2019-03-28 22:09:23 +02:00
Bill Thorp
40e7fae564 not overflow int32 (#1594) 2019-03-28 13:08:32 +02:00
Stefan Benten
7336e87e1c
Fixing Check for a writable folder of the identity tool (#1587) 
* Fixing Check for a writable folder

* Removing fmt statements

* Check the folder not the file
 2019-03-27 23:20:10 +01:00
Cameron
cac55a29e4
Add used egress/ingress to storage node dashboard (#1565) 
* add egress and ingress to StatSummaryResponse

* print egress and ingress to storagenode dashboard
 2019-03-27 15:44:18 -04:00
Michal Niewrzal
bfdfebbde2
Satellite orders receiving (#1564) 
This change adds satellite endpoint for receiving OrderLimits sent by storage node.
Change includes:
* wire up orders sender in storage node (also in testplanet)
* saving serial number for OrderLimit in serial_numbers table
* satellite endpoint for receiving, verifying and storing OrderLimit and Order serial number
* initial implementation for Orders DB
* basic test for sending orders to satellite
 2019-03-27 11:24:35 +01:00
Natalie Villasana
5b48a48a79
adds test for correct download with non-critical amount of nodes offline (#1574) 2019-03-26 14:09:44 -04:00
JT Olio
d7ac48ead7
pkg/process: indicate if an error is causing program death (#1573) 
Change-Id: I4f97cbd1004bcbda8b25a238a6c30ca88a20ac43
 2019-03-26 07:47:55 -06:00
Kaloyan Raev
d1639c4157 Merge statdb pkg into overlay pkg (#1570) 2019-03-25 18:25:09 -04:00
Bryan White
ef99c1657f
TLS extension handling overhaul (#1458) 2019-03-25 22:52:12 +01:00
Bill Thorp
dbd97aaaf9
SNs should NOT be audited for expired segments (#1559) 
delete expired pointers during audit
 2019-03-23 06:52:51 -04:00
Egon Elbre
94e79eda6d
remove overlay endpoint (#1521) 2019-03-23 10:06:11 +02:00
Dylan Lott
c6d5ff5dc2
remove unused bootstrap function from discovery/service.go (#1558) 
* remove unused bootstrap function from discovery/service.go

* Linter fix missing semicolon
 2019-03-22 15:00:38 -06:00
Natalie Villasana
0fa1d536e7
removes pingbackTimeout (#1556) 2019-03-22 16:06:57 -04:00
Natalie Villasana
ea4a61f0c0
adds slow kad dialer tests, adds timeout interceptor to transport (#1545) 2019-03-22 13:09:37 -04:00
Jess G
9236ac4bdf
change default to true to enforce sa whitelist (#1549) 
* change default to true to enable whitelist

* add devDefault
 2019-03-22 07:21:16 -07:00
Egon Elbre
1d96d25f3f
kademlia ping tracking (#1538) 2019-03-22 15:27:59 +02:00
Egon Elbre
694b6dc1da
make tests run faster (#1553) 2019-03-22 15:14:17 +02:00
Kaloyan Raev
30dfc2b20c
Remove PointerDB client (#1520) 2019-03-22 11:01:49 +02:00
Maximillian von Briesen
db64d6590b Add repairer tests (#1494) 2019-03-21 16:26:56 +02:00
Kaloyan Raev
2c410f51bb
Fix CalcEncryptedSize and CalcPieceSize (#1525) 2019-03-21 15:47:48 +02:00
Egon Elbre
2c5c2c29da
storage node order sending (#1535) 2019-03-21 15:24:26 +02:00
Kaloyan Raev
b66b49d824 Print Satellite ID in logs when SettleAgreement fails (#1542) 2019-03-21 12:59:22 +02:00
Michal Niewrzal
d7feafe56b Move psserver tests (#1522) 2019-03-20 23:12:00 +02:00
Natalie Villasana
e390605b81 rearranges code in verifier.go (#1529) 2019-03-20 11:54:37 +01:00
Natalie Villasana
b4b99fa979 adds context timeout to transport.DialAddress (#1528) 2019-03-20 12:11:07 +02:00
Bryan White
724aaab78d fix kademlia bootstrap and getting peer identity from context (#1434) 2019-03-20 10:30:42 +02:00
Natalie Villasana
61ee04d363
adds pingbackTimeout to kademlia endpoint (#1518) 2019-03-19 14:30:27 -04:00
Natalie Villasana
a09f568fdb
audit service won't hang on bad storage node conns (#1503) 2019-03-19 13:37:26 -04:00
Kaloyan Raev
d057efb05e
Add Repair method to ECClient (#1509) 2019-03-19 15:14:59 +02:00
Egon Elbre
26497df0e2
Faster agreement sender (#1507) 2019-03-19 14:56:59 +02:00
Egon Elbre
636d73934e Fix storage node path handling (#1519) 2019-03-19 11:10:23 +02:00
Michal Niewrzal
56251570ef Remove overlay client (#1510) 2019-03-18 21:46:10 +02:00
Egon Elbre
117edec54c
Add serial number type (#1508) 2019-03-18 15:08:24 +02:00
Egon Elbre
05d148aeb5
Storage node and upload/download protocol refactor (#1422) 
refactor storage node server
refactor upload and download protocol
 2019-03-18 12:55:06 +02:00
Cameron
c7ffbe1c28
Add ability to view irreparable segments on satellite (#1448) 
* define irreparable inspector protobuf

* add IrreparableDB method GetLimited

* fill out irreparable inspector API

* add IrreparableInspector server to satellite, fix small error

* refactor IrreparableDB to use pb.IrreparableSegment instead of irreparable.RemoteSegmentInfo
 2019-03-15 16:21:52 -04:00
Bill Thorp
665fd33e3c
Repair queue isolation level fix (#1466) 
Implemented custom SQLite and Postgres Repairqueue Dequeue handlers
 2019-03-14 17:12:47 -04:00
Bill Thorp
52e829c6de
framework for deleting expired Orders from Satellite (#1436) 
framework for deleting old bandwidth agreements
 2019-03-12 16:57:21 -04:00
Dylan Lott
59f1e267c9
Removes concept of email from kademlia metadata (#1435) 
* Removes concept of Email from Kademlia

* Removes kad email

* adds emails back to operator config for satellite

* replace operator configs in testplanet
 2019-03-12 14:05:18 -06:00
Egon Elbre
0560f076a5
fix piece id derivation (#1462) 2019-03-12 19:35:42 +02:00
Alexander Leitner
af889f1554
Move iswriteable to fpath (#1464) 2019-03-12 12:13:40 -04:00
Alexander Leitner
bb77d9b4a6
Warn about permissions when creating identity (#1384) 
* Warn about permissions when creating identity

* Function to determine if directory is writeable

* Check if writable before authorizing

* Remove redeclatarion

* remove windows specific utils

* Nat nits

* Actually test if directory is writeable with file creation
 2019-03-12 10:42:38 -04:00
JT Olio
56ace481c1 judicious import path checking (#1425) 
see also
3ee9780df1
and
https://golang.org/cmd/go/#hdr-Remote_import_paths
and
https://golang.org/cmd/go/#hdr-Import_path_checking

Change-Id: I2ac906b1a1befde21edbcfc9b0a1b6a4863520da
 2019-03-12 14:58:40 +01:00
Bill Thorp
1f4e1a3ab0
framework to delete raw accounting tallies (#1439) 
framework to delete raw accounting tallies, renamed Query
 2019-03-12 09:08:23 -04:00
Bill Thorp
66718cc5e6
Development defaults for configuration (#1430) 
added --dev command line option, cfgstruct.DevFlag(), and cfgstruct.SetupFlag()
 2019-03-12 08:51:06 -04:00
littleskunk
afad4f7010 Improve bwa tests (#1408) 2019-03-08 13:23:43 +01:00
littleskunk
11048bed2c psdb: index on serial number for bandwidth_agreements 2019-03-08 11:53:04 +02:00
Jess G
193a70f0a6
add private listener to grpc server (#1398) 
* add private listener to grpc server

* add changes per init CR

* fix server.close

* add insecure grpc connection, update logs msg

* fix tests, move insecure client

* add private ports to storj-sim, add insecure client to other inspectors

* add ports to test so there arent conflicts

* fix lint err

* fix node started log msg, close public listener

* remove commented out line
 2019-03-07 13:19:37 -05:00
Yaroslav Vorobiov
9cf56c03fa
Bucket usage info rollup table creation and implementation (#1340) 2019-03-06 17:54:48 +02:00
Bryan White
c607abf27c [V3-1147] Ensure certificate validation happens properly (#1403) 
* add regression test & update transport tests

* separate client and server verificiation functions

* goimports
 2019-03-06 09:42:34 -05:00
Egon Elbre
ead4a0ae95
Add new piece id definition (#1411) 2019-03-06 11:17:35 +02:00
Natalie Villasana
036b2f3b49
allocates 1kb for audit bandwidth agreements instead of 256kb (#1404) 2019-03-05 18:58:25 -05:00
aligeti
15034526cd Enhance the Storage Node to keep all BWA's for 90 days (#1374) 
* Enhance the Storage Node to keep all BWA's for 90 days

* warn -> error

* rename enum
 2019-03-05 17:20:46 -05:00
Jess G
3c9d83dbfe
convert psserver dashboard into an inspector (#1407) 
* Convert psserver dashboard into an inspector

* remove dashboard stream, update ps.pb.mock

* fixes for lint errs
 2019-03-05 15:48:37 -05:00
Bryan White
5742131941
tlsconfig refactor / cleanup (#1399) 2019-03-04 21:40:18 +01:00
Bryan White
675e0ef683 [V3-1320] fix empty node ID verification non-error (#1395) 
* small identity refactor:

+ Optimize? iterative cert chain methods to use array instead of slice
+ Add `ToChain` helper for converting 1d to 2d cert chain
  TODO: replace literal declarations with this
+ rename `ChainRaw/RestChainRaw` to `RawChain/RawRestChain`
  (adjective noun, instead of nound adjective)

* add regression tests for V3-1320

* fix V3-1320

* separate `DialUnverifiedIDOption` from `DialOption`

* separate `PingNode` and `DialNode` from `PingAddress` and `DialAddress`

* update node ID while bootstrapping

* goimports & fix comment

* add test case
 2019-03-04 15:03:33 -05:00
aligeti
588e2a51d2 Handles zero bucket count and bucket with no files (#1391) 
* handles zero bucket count and bucket with no files
 2019-03-04 12:47:08 -05:00
Michal Niewrzal
cce84c9914
Cleanup errors handling around piece hash verification (#1382) 2019-03-04 15:29:35 +01:00
Kaloyan Raev
5fa7a4a7c6 Ensure ECClient upload timer is stopped when no more status is expected (#1397) 
This change ensures that the upload timer of ECClient is always stopped after no more status is expected from uploaded pieces. It also ensures that the "Timer expired" message will be logged only if the context is not already cancelled.

This is to avoid confusing logs where a "Timer expired" message is logged significantly later and mixes with similar messages logged from the upload of the next file segments.
 2019-03-04 14:48:13 +01:00
Stefan Benten
5f57d2c906
Reduce Log Messages on Production Settings (#1396) 
* Reduce Tally Log Messages on Production Settings

* Reduce Logging in Piecestore Client as well
 2019-03-04 11:12:43 +01:00
Kaloyan Raev
54f68347ad
Use DialNode for requesting node info (#1394) 2019-03-02 09:34:08 +02:00
Dylan Lott
9abf191b01
Change payments to use statDB for wallet addresses rather than Overlay cache (#1368) 
* Wiring up DumpNodes response for Inspector

* Finalize everything and test that it works

* Get Count and DumpNodes working for Overlay Cache

* WIP updating payment rollup to check statDB instead of overlay

* FIrst pass at updating statDB to take wallet and email

* Passing tests

* use pb.NodeOperator instead of Meta struct

* remove TODO

* revert go.mod

* Get SQL migration working correctly

* Changes Meta to Operator in NodeStats struct

* Adds update operator logic for statDB

* Fix db migrate tests - added v5 snapshot

* User friendly msg for missing snapshot version

* Passing tests

* Change node update to happen in discovery instead of in overlay

* Fix logic and update function calls

* Update comment on UpdateOperator interface method

* Update name of parameter

* Change type of argument to UpdateOperator

* Updates statDB tests
 2019-03-01 10:46:34 -07:00
Cameron
3fd55c7034
remove unused date field in csv row (#1388) 2019-03-01 08:23:27 -05:00
Egon Elbre
3f3209c8d5
fixes to piecestore and psdb (#1380) 
* replace direct reference with an interface in various places
* hide piecePath
* ensure psserver tests don't use path
* ensure psserver tests don't use sql queries directly
 2019-03-01 07:46:16 +02:00
Dylan Lott
bacd683879
Adds DumpNodes method to Kademlia Inspector (#1328) 
* Wiring up DumpNodes response for Inspector

* Finalize everything and test that it works

* remove changes to overlay inspector

* Replace list with iterate method in DumpNodes

* generate proto

* Merge in nodeinfo changes

* Update iterate function in DumpNodes

* Fix import order
 2019-02-28 12:55:27 -07:00
Cameron
018cdd4037
uplink ttl (#1365) 
* add expires flag to uplink cp

* fix ttlDBs delete expired query
 2019-02-28 12:51:24 -05:00
Jennifer Li Johnson
6ecda190cb
Jj/payments cleanup (#1379) 
* Removes date from payments csv and re-query of wallet addr

* removes dbx get wallet addr

* switch order in csv
 2019-02-28 12:13:59 -05:00
Michal Niewrzal
4122c98cb7
Validate piece hash on satellite (#1359) 
The satellite receives pieces signed hashes in Pointer. If signed hash cannot be validated then piece is removed from Pointer and not saved in DB.
 2019-02-28 15:14:54 +01:00
aligeti
993af7ed37 fixes the bwa deletes to be done by serialnum, rather by signature (#1364) 2019-02-27 18:29:23 +02:00
Egon Elbre
b7f8e309d9 add piecestore.Storage interface (#1369) 2019-02-27 10:56:16 +01:00
Bryan White
fde0020c68
dial node/address with bad server cert (#1342) 2019-02-26 19:35:16 +01:00
Jennifer Li Johnson
cefdff535a
Replace list calls with iterates in kademlia routing table (#1318) 
* notes

* adds bool for option to skip self

* wip getKBucketID is still breaking TestAddNode

* fix (#1322)

* remove panic

* remove iteratebuckets

* implements getKBucketRange

* getNodeIDsWithinKBucket

* removes determineFurthestIDWithinK

* wip

* nodeIsWithinNearestK

* removes bucketIDToKey

* removes extra xor in test

* wip

* rebased

* uses create rt w opts

* lint

* fix lint
 2019-02-26 11:07:53 -05:00
aligeti
c6ad7644d2
Total file count through Monkit (#1351) 
* segment, file, byte stats, total and per-bucket; checker: report segment health stats; reports the total num of lost files

* code review updates
 2019-02-26 10:17:51 -05:00
Bryan White
03747bad51
some certificates cmd cleanup: (#1361) 2019-02-26 09:55:52 +01:00
Kaloyan Raev
1ec17653d4 Endpoint for local node info (#1355) 
Adds a new `Info` method to the Kademlia endpoint that returns the following local node info:
* ID
* Type
* Metadata (email and wallet)
* Restrictions (free storage and bandwidth)

The new endpoint is exposed as `inspector kad node-info` command too.
 2019-02-25 19:41:51 +01:00
Michal Niewrzal
81408a3c9e
Use SignedHash on client/uplink side (#1354) 
* psclient receives storage node hash and compare it to own hash for verification
* uplink sends delete request when hashes don't match
* valid hashes are propagated up to segments.Store for future sending to satellite
 2019-02-25 16:57:54 +01:00
Bryan White
b652b6a522 use peer ca whitelist in testplanet (#1337) 2019-02-25 09:38:03 +02:00
Michal Niewrzal
6186b3f90a
Storage node hash calculation on upload (#1347) 
Storage node is calculating hash of uploaded data and send it back to uplink with signature
 2019-02-23 11:46:07 +01:00
Bill Thorp
373b301736
BWA aliases (#1333) 
aliased RBAs and PBAs
 2019-02-22 16:17:35 -05:00
Jennifer Li Johnson
b5447c6608
Routing table tests (#1270) 
This PR includes a new package called testrouting, which implements a very algorithmically slow but hopefully easy-to-keep-operationally-correct in-memory routing table. The routing table also supports writing out its current structure as a DOT graph for visualization. testrouting is primarily meant to help in coming up with generic routing table integration tests.

This PR also includes a new routing table integration test suite that runs against all current routing table implementations. Our existing routing table passes a lot of the tests, but not all of them, still debugging why. I have confirmed the tests should pass with the visualization graphs though.
 2019-02-22 13:39:29 -05:00
aligeti
4ccb80257d
Refactor how bandwidth allocations are stored on storage nodes (#1280) 
* integrated with db migration framework

* updating the migrationdb test

* updating the migrationdb sql v0 and v1 files

* lint warnings

*  code review changes

* fix migration scripts, upated test data

* fixed the local/UTC timing issues

* code review updates

* updated code review comments
 2019-02-22 10:51:39 -05:00
Natalie Villasana
c3d3f41d30 removes some SignedMessage use (#1258) 
Removes most instances of pb.SignedMessage (there's more to take out but they shouldn't hurt anyone as is).

There used to be places in psserver where a PieceID was hmac'd with the SatelliteID, which was gotten from a SignedMessage. This PR makes it so some functions access the SatelliteID from the Payer Bandwidth Allocation instead.

This requires passing a SatelliteID into psserver functions where they weren't before, so the following proto messages have been changed:

 * PieceId - satellite_id field added
   This is so the psserver.Piece function has access to the SatelliteID when it needs to get the namespaced pieceID.
   This proto message should probably be renamed to PieceRequest, or a new PieceRequest message should be created so this isn't misnamed.

 * PieceDelete - satellite_id field added
   This is so the psserver.Delete function has access to the SatelliteID when receiving a request to Delete.
 2019-02-19 23:36:08 -06:00
Dennis Coyle
36e4d8c1c2
restrict actions on PayerBandwidthAllocation (#1326) 
* restrict actions on PayerBandwidthAllocation

* defaults to error if action unknown
 2019-02-19 16:00:58 -05:00
Jennifer Li Johnson
a5aa5f84a1 handle server termination in kademlia tests (#1323) 2019-02-19 20:41:32 +02:00
Michal Niewrzal
8d685217e4
Storagenode migrations (#1299) 
* creates initial migration for psdb
* add test mechanism to validate migration to every version
* fix few small issues in versions.go and context.go
 2019-02-19 10:39:04 +01:00
Bill Thorp
c598ed034d kad.FindNear fix (#1320) 
We realized that the Kademlia FindNear() function was

1. not using XOR distance (AKA _totally broken_)
2. largely a duplicate of the RoutingTable FindNear() function

Changes in this PR:

1. upgraded RoutingTable FindNear() to use iterator and restrictions
2. removed unneeded RoutingTable interface
3. made Kademlia wrap methods that were previously accessed via RoutingTable
4. fixed the tests
 2019-02-15 22:23:35 -05:00
Egon Elbre
12261c0624
Satellite migrations (#1301) 2019-02-14 23:55:21 +02:00
Bryan White
d93c57cffa add certificates claim delete & refactor (#1317) 
* add \`certificates claim delete\` & refactor

* quickfix

* goimports
 2019-02-14 20:17:26 +01:00
Bill Thorp
dfd6589fbe
fix kad refresh bad bucket range (#1315) 
fix kad refresh bad bucket range
 2019-02-14 10:47:03 -05:00
Michal Niewrzal
58792533b9
Fix invalid config key msg for dashbord (#1314) 2019-02-14 14:06:15 +01:00
Stefan Benten
661df2a625 Reduce Log Messages on Level Info (#1313) 2019-02-14 13:50:35 +01:00
Bill Thorp
9b580c5fb6 Repair checker is checking the same 1000 elements all the time (#1297) 
* removed limit on repair, now using cycle

* added BatchIteratorOptions

* consolidated boltdb common.go

* PR feedback cleanup
 2019-02-14 13:33:41 +01:00
Egon Elbre
1a5a9903a2
Move SplitConnstr to a nicer place (#1308) 2019-02-13 23:54:59 +02:00
Bill Thorp
b53f9896d3
Removed ReverseList from KeyValueStore interfaces (#1306) 
Removed ReverseList from KeyValueStore interfaces
 2019-02-13 12:27:03 -05:00
Bill Thorp
2c5716e874
pb.Equal fix (#1304) 
* pb.Equal fix

* file capitalization

* oh snake case
 2019-02-13 09:08:57 -05:00
Michal Niewrzal
b2f9453184
Disable Checker subsystem in tests (#1279) 
* Disable Checker subsystem in tests

* rename field

* remove sleeps and errgroup.Group
 2019-02-11 22:06:39 +01:00
Maximillian von Briesen
f94a2e1161
Node selection method name change (#1296) 
* SelectNodes -> SelectStorageNodes
 2019-02-11 14:24:51 -05:00
Maximillian von Briesen
c9d00d8241
test random selection for new nodes (#1294) 2019-02-11 12:10:32 -05:00
Maximillian von Briesen
c6c23a319b
move node type definition to db level for node selection (#1295) 2019-02-11 11:35:28 -05:00
Egon Elbre
18bd2d7b1a
Fix randomized selection test limit (#1290) 2019-02-11 14:04:00 +02:00
JT Olio
2a59679766 pkg/transport: require tls configuration for dialing (#1286) 
* separate TLS options from server options (because we need them for dialing too)
* stop creating transports in multiple places
* ensure that we actually check revocation, whitelists, certificate signing, etc, for all connections.
 2019-02-11 13:17:32 +02:00
JT Olio
ea28a9a581 pkg/transport: fail on bad certificates (#1285) 2019-02-11 13:02:49 +02:00
Maximillian von Briesen
690d5447a1
Add test for randomized node selection (#1271) 2019-02-09 16:17:49 -05:00
Dylan Lott
612919d95b
Adds tests to discovery/graveyard (#1272) 
* adds tests for cache graveyard

* updates tests for cache graveyard to use sync2
 2019-02-08 13:56:24 -07:00
Egon Elbre
17a7f9d002
Enable planet Merging test (#1281) 2019-02-08 22:35:59 +02:00
Egon Elbre
e37e0c1b5f
Fix server config usage (#1282) 2019-02-08 20:57:17 +02:00
Kaloyan Raev
4e738bbd44
Fix randomness in stopping nodes in TestGetObjectStream (#1277) 2019-02-08 15:20:12 +02:00
Michal Niewrzal
c7ad427d95
Move metainfo config (#1234) 
* Move metainfo config

* move config to uplink

* reorganize code

* use satellite apikey

* comment update
 2019-02-08 13:57:35 +01:00
Egon Elbre
995927dcd2
Disable psserver TestRetrieve (#1273) 2019-02-08 11:54:50 +02:00
Egon Elbre
9c1e299f3c
Ensure everyone sees everyone else (#1275) 2019-02-08 11:25:13 +02:00
Egon Elbre
5a13bdec18
close revocation database in test (#1274) 2019-02-08 11:13:32 +02:00
paul cannon
c35b93766d
Unite all cryptographic signing and verifying (#1244) 
this change removes the cryptopasta dependency.

a couple possible sources of problem with this change:

 * the encoding used for ECDSA signatures on SignedMessage has changed.
   the encoding employed by cryptopasta was workable, but not the same
   as the encoding used for such signatures in the rest of the world
   (most particularly, on ECDSA signatures in X.509 certificates). I
   think we'll be best served by using one ECDSA signature encoding from
   here on, but if we need to use the old encoding for backwards
   compatibility with existing nodes, that can be arranged.

 * since there's already a breaking change in SignedMessage, I changed
   it to send and receive public keys in raw PKIX format, instead of
   PEM. PEM just adds unhelpful overhead for this case.
 2019-02-07 14:39:20 -06:00
Bill Thorp
0b35762105
Convert Payments to use SQL, for SUM() and Wallet (#1266) 
* payments query no longer DBX, using SQL

* sum in SQL

* removed old function

* fixed rollup test

* wrap errors

* removed DBX code
 2019-02-07 15:26:55 -05:00
aligeti
b736ae4823
Store the uplinks public key on the satellite so that it can verify bandwidth requests in the future (#1042) 
* integrated with bwagreement & psserver

* integrated with pointerdb

* code review updates

* refactor after code review

* uplinkdb rename to certdb

* Code review changes
 2019-02-07 14:22:49 -05:00
paul cannon
ef61c170b1
Consolidate key/cert/signature encoding and decoding (#1243) 2019-02-07 12:40:28 -06:00
paul cannon
32882daf38
SHA256Hash() and TBSBytes() needn't return error (#1242) 2019-02-07 11:08:52 -06:00
paul cannon
0032147665 regroup things related to public-key cryptography (#1241) 2019-02-07 10:04:29 +01:00
Cameron
bfe8060f4d
Rollup tests (#1224) 
* createdAt field no longer autoinsert

* rollup tests
 2019-02-06 23:16:24 -05:00
Bryan White
3b54cf0e15
identity improvements: (#1215) 2019-02-06 17:40:55 +01:00
Egon Elbre
30f89b0362
Ensure overlay tests run against postgres (#1232) 2019-02-06 15:32:42 +02:00
Egon Elbre
bb11d83ed0
Proper planet shutdown (#1249) 2019-02-06 15:19:14 +02:00
Michal Niewrzal
aa3836c06a
Add checker irreparable test (#1246) 
* Add checker irreparable test

* use RepairUnixSec

* wait for second check

* use storagenode.ID()

* Update comments
 2019-02-06 14:03:38 +01:00
Egon Elbre
fdbe2db273
Remove node package and simplify DHT interface (#1233) 2019-02-06 14:37:17 +02:00
Egon Elbre
2d6b5faf6d
Fix discovery busy-spin (#1248) 2019-02-06 14:25:29 +02:00
Bryan White
00554f0876
add certificates claims+ (#1216) 2019-02-06 12:00:26 +01:00
Bryan White
7b7e6c43f8
better batch-generation (#1219) 2019-02-06 09:04:12 +01:00
Egon Elbre
218e95728c
psserver: swallow io.EOF when getting full content in FileRanger (#1238) 2019-02-06 07:41:41 +02:00
paul cannon
331f0cc845 don't put nil nodes in GetResponse (#1218) 2019-02-05 23:12:27 +02:00
Bill Thorp
690e8b2061
storage node cert cache (#1226) 
* draft

* still errors

* double close fix

* added tests

* weird, goimports must not be working

* renames

* missed one

* forgot to save:
 2019-02-05 12:57:56 -05:00
Egon Elbre
07412698a9
Use lower bcrypt cost for console.Service when running tests and disable flaky tests (#1236) 2019-02-05 19:31:53 +02:00
Yaroslav Vorobiov
2ff0d9d435
Add bucket to project relationship on satellite (#1143) 
* integrate console api keys with buckets in pointerdb

* fix test

* fix tests kvmetainfo

* linter fix

* disable account activation

* fix test

* review fixes

* fix comments

* little refactoring

* remove debug println

* fix typo

* disable activation in a propper way

* fix test

* fix imports

* fix uplink count in testplanet

* move key creation to planet.newUplink
 2019-02-05 19:22:17 +02:00
Egon Elbre
b16f27c54b
Ensure discovery tests run against postgres (#1230) 2019-02-05 19:03:16 +02:00
Kaloyan Raev
3c73d3a33c Fixes intermitent failures in storage/ec TestPut (#1239) 2019-02-05 18:49:52 +02:00
Egon Elbre
39c1e5ccec
Ensure checker tests run against postgres (#1229) 2019-02-05 18:00:52 +02:00
Egon Elbre
94ee200e35
Ensure cursor tests run against postgres (#1228) 2019-02-05 17:45:09 +02:00
Egon Elbre
6564357e13
Ensure that tally tests run against postgres (#1227) 2019-02-05 16:23:45 +02:00
Kaloyan Raev
dd76829d10
Improve logic for cutting the long tail during upload (#909) 2019-02-05 12:54:25 +02:00
Egon Elbre
87d6410b50 Revert "Remove node package and simplify DHT interface." 
This reverts commit 03ec1ff92d.
 2019-02-05 10:38:48 +02:00
Egon Elbre
03ec1ff92d Remove node package and simplify DHT interface. 2019-02-05 10:37:24 +02:00
Dylan Lott
21a58d0ece
node graveyard resurrection (#1186) 
* deletes nodes from cache if unresponsive

* update error messages

* WIP restructuring discovery tickers

* passes tests

* Removes log messages

* Passes up error in graveyard ticker
 2019-02-04 14:52:06 -07:00
Egon Elbre
c37e0c1b6d
testplanet.Run (#1222) 2019-02-04 22:37:46 +02:00
Michal Niewrzal
2f8b3da1af
Upload/download for testplanet Uplink (#1183) 
* Upload/download for testplanet Uplink

* check error in tests

* cleanup

* refactor node -> uplink

* add missing test file

* rest of refactoring

* workaround to resolve cycles in tests

* rename method

* add missing comments

* review comments

* use KiB
 2019-02-04 17:56:10 +01:00
Michal Niewrzal
53c11dfc5d
Stop storagenode on ctrl+c (#1220) 
* Stop storagenode on ctrl+c

* Cancel grpc server

* handle error

* handle error

* use errgroup

* fix check-travis-tidy

* remove pipefail
 2019-02-04 15:50:55 +01:00
paul cannon
0238a7785b
make a map object before inserting into it (#1217) 
I think this is causing panics.
 2019-02-03 18:42:20 -06:00
Bill Thorp
0f662b8e38 sql based tally (#1166) 
* WIP

* wacky changes

* more

* it builds.... IT BUILDS!!!

* fixed SQL, broke out saving tallies for testing

* shorter lines

* fixed SQL, moved tally_test to testplanet

* lint

* WIP logic error preventing PUT and GETs to same serialnum

* fixed BWA test

* fixed temporary brain failure

* eliminated magic numbers

* cleaned up satellite uplink stats

* use errs.Combine instead

* thrashing

* fixed tally erroneous error msg

* fixed tally test

* lint

* SQL syntax attempt to fix

* spelling error

* made bwa db test resist old postgres data

* postgres pk error msg hunting

* postgres pk error msg hunting

* postgres

* err might be nil?

* fixed error logging bug

* hopefully solved postgres issue

* using rebind

* moved tests to _test package

* fixing test dirs

* finally made sense of Egons package name feedback

* UTC, array fixes
 2019-02-01 13:50:12 -05:00
Egon Elbre
4117600f32
Cleanup repository a bit (#1203) 2019-02-01 20:21:00 +02:00
Kaloyan Raev
5c2b67db63 Fix EOF error on audits (storage node) (#1208) 
* Fix EOF error on audits

* Specify the correct number of bytes to CopyN instead of checking for EOF

* Fix imports order
 2019-02-01 10:57:11 -07:00
Kaloyan Raev
a90aa18582 Fix panic in PointerDB.Get (#1209) 
* Fix panic in PointerDB.Get

* Allow nil nodes, check if v is nil instead

* More checks like this

* Remove unnecessary check

* More checks
 2019-02-01 10:55:47 -07:00
JT Olio
41d19f3c3d
miniogw: fix buffer size alignment (#1213) 
Change-Id: I47e0e67e385d236ac0338fe36f11cc7c00e23dba
 2019-02-01 10:42:30 -07:00
Dylan Lott
c77a647174
Updates transport dialopts to use WithBlock() (#1201) 
* Updates transport opts to use WithBlock()

* Removes unnecessary withBlock's and adds WithBlock to DialAddress
 2019-02-01 10:40:35 -07:00
Egon Elbre
5a63c00442
Fix issues with blocking during startup (#1212) 2019-02-01 19:28:40 +02:00
Bryan White
3e3088e37c improve identity create ux (#1211) 2019-02-01 11:58:33 -05:00
Egon Elbre
7a85c5c1c6
Fix audit node ids (#1207) 2019-02-01 16:48:57 +02:00
Egon Elbre
00570df1f9
Add top flag to tparse (#1205) 2019-02-01 16:35:13 +02:00
Egon Elbre
b91d77436f
Test merging planets (#1181) 2019-02-01 15:32:28 +02:00
Egon Elbre
0c366c1ed2 Clearer code for node selection (#1173) 2019-01-31 13:49:00 -05:00
Kaloyan Raev
8c357ac571
Correctly update uptime when ping fails (#1190) 2019-01-31 16:51:38 +02:00
Dylan Lott
6025f9f19e
deletes nodes from cache if unresponsive (#1185) 
* deletes nodes from cache if unresponsive

* update error messages
 2019-01-30 14:15:24 -07:00
Egon Elbre
97714d150f
Remove unused kademlia.Disconnect method (#1178) 2019-01-30 22:57:13 +02:00
Egon Elbre
d5346982c2
Delete provider package (#1177) 2019-01-30 22:47:21 +02:00
Egon Elbre
19de56bf83
Fix race in discovery test (#1184) 2019-01-30 22:29:33 +02:00
Alexander Leitner
7c741c16d1
Readable sizes in dashboard (#1180) 
* Readable sizes

* Use tabwriter

* Added more addresses to dashboard stats

* Use kad to look up addresses

* Move dashboard command to dashboard.go
 2019-01-30 14:49:25 -05:00
Michal Niewrzal
14434594c4
Re-enable repair checker tests (#1172) 
* Re-enable repair checker tests

* simpler NodeID construction

* adjust benchmark
 2019-01-30 19:34:44 +01:00
Dylan Lott
5e1ab841cf Cache refresher (#1171) 
* got tests passed

* wire up paginate function for cache node retrieval

* Add tests for paginate but they're failing

* fix the test arguments

* Updates paginate function to return more variable

* Updates

* Some test and logic tweaks

* improves config handling in discovery

* adds refresh offset to discovery struct
 2019-01-30 11:29:18 -05:00
Egon Elbre
6132ce86b7
Remove utils.LogClose (#1169) 2019-01-29 22:42:27 +02:00
Natalie Villasana
8d7944bcf8
new node selection in overlay (#1136) 2019-01-29 14:42:43 -05:00
Egon Elbre
1df81b1460
Separate garbage collect logic from psdb (#1167) 2019-01-29 17:41:01 +02:00
Egon Elbre
54eb1b099c
don't display message unless logger has been specified (#1165) 2019-01-29 14:12:24 +02:00
Egon Elbre
e1a8bbdcb6
Kademlia flags cleanup (#1137) 2019-01-29 08:51:07 +02:00
Dylan Lott
bb2588e9ae
Refactors kademlia inspector to use FindNear (#1157) 
* WIP

* update identity paths to get dump-nodes working

* Remove commented out code

* remove comment

* updates from code reviews

* updates linter
 2019-01-28 15:53:37 -07:00
Bill Thorp
60946c2024
make bandwidth agreements sensible: without []byte's (#1152) 
removed []byte's from bandwidth agreement protocol buffers
 2019-01-28 14:45:25 -05:00
Dylan Lott
df903ea215
Decreases refresh time per bucket in Kademlia (#1150) 2019-01-28 12:36:20 -07:00
Egon Elbre
d50c07e56c
Implement WorkGroup (#1151) 2019-01-28 21:04:42 +02:00
Egon Elbre
cecd4b0816
Remove server aliases (#1154) 2019-01-28 17:04:53 +02:00
Bryan White
2b20acbec9
identity cleanup (#1145) 2019-01-26 15:59:53 +01:00
Egon Elbre
49dacb662c
Improve bwagreement tests (#1149) 2019-01-26 01:06:38 +02:00
Egon Elbre
f7e18882c4
Kademlia closing race (#1139) 2019-01-26 00:33:20 +02:00
Bill Thorp
8ba4b61e59
Make Bandwidth Agreements Secure / Trustable (#1117) 
* Added cert chains and nodeid verification to bandwidth agreement
 2019-01-25 13:05:21 -05:00
Bryan White
7bed8050aa
Improve identity cli ux: (#1142) 2019-01-25 17:55:45 +01:00
Jennifer Li Johnson
5397efe5c1
removes standalone payments service (#1134) 
* remove standalone payments service - will integrate within satellite directly

* regenerate proto files
 2019-01-25 11:23:25 -05:00
Egon Elbre
85b43926b4
Separate identity from server config (#1138) 2019-01-25 16:54:54 +02:00
Egon Elbre
a1027c7836
Make inmemory a separate command (#1127) 2019-01-24 23:04:29 +02:00
Egon Elbre
78b0414e6c
internal/sync2: fix file handle leak (#1124) 2019-01-24 22:48:54 +02:00
Cameron
f5b1e77cf2 rollup timestamp fix (#1132) 
* create timestamp if isNew

* pass isNil into SaveRollup
 2019-01-24 15:41:22 -05:00
Jennifer Li Johnson
856b98997c
updates copyright 2018 to 2019 (#1133) 2019-01-24 15:15:10 -05:00
Jennifer Li Johnson
030d294399
adds logs in rollup and tally run (#1131) 2019-01-24 15:05:53 -05:00
Jennifer Li Johnson
70940b8710
Making sure payments works (#1122) 
* byte hours

* updates comment

* removes payment test data methods

* wip

* adds tally and rollup to peer

* remove sim test for payment (for now)

* lint error
 2019-01-24 13:44:14 -05:00
Alexander Leitner
c4afec9328
quick log cleanup for storage node (#1121) 
* Clean up logs
 2019-01-24 11:09:37 -05:00
Egon Elbre
28427e7fad
Change identity command (#1128) 2019-01-24 17:41:16 +02:00
Egon Elbre
5de7f8af7f
Satellite Peer (#1119) 2019-01-23 21:58:44 +02:00
Cameron
95d2d54fc3
storagenode approvedSatelliteIDs (#1116) 
* add config fields for satellite restriction on psserver

* add whitelistedSatIDs to psserver Server struct

* check pbwa satellite ID against whitelist

* add whitelist to psserver tests

* reword help message, make approved() a method on server
 2019-01-23 12:56:12 -05:00
Michal Niewrzal
3fdb47e31c Try accessing bootstrap node for verification (#1089) 2019-01-23 17:48:46 +02:00
Egon Elbre
8677c68b68
Print successfully connected message every 100 times (#1118) 2019-01-23 14:07:22 +02:00
Egon Elbre
05b96f95a3
Change default difficulty to 30 (#1113) 2019-01-23 13:36:19 +02:00
Egon Elbre
b6c61cdd55
Use storagenode.Peer for storagenode (#1107) 2019-01-23 12:39:03 +02:00
Egon Elbre
99d3b7a3c8
Fix import grouping (#1111) 2019-01-22 17:48:23 +02:00
Egon Elbre
ad49f4b994
Enable linters (#1105) 2019-01-22 17:09:02 +02:00
Bryan White
322a2813f7
oops (#1110) 2019-01-22 15:34:40 +01:00