JakeHillion/storj
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
1,974 Commits 5 Branches 0 Tags 94 MiB
16156e3b3d
Commit Graph

1127 Commits

Author SHA1 Message Date
Egon Elbre
636d73934e Fix storage node path handling (#1519) 2019-03-19 11:10:23 +02:00
Michal Niewrzal
56251570ef Remove overlay client (#1510) 2019-03-18 21:46:10 +02:00
Egon Elbre
117edec54c
Add serial number type (#1508) 2019-03-18 15:08:24 +02:00
Egon Elbre
05d148aeb5
Storage node and upload/download protocol refactor (#1422) 
refactor storage node server
refactor upload and download protocol
 2019-03-18 12:55:06 +02:00
Cameron
c7ffbe1c28
Add ability to view irreparable segments on satellite (#1448) 
* define irreparable inspector protobuf

* add IrreparableDB method GetLimited

* fill out irreparable inspector API

* add IrreparableInspector server to satellite, fix small error

* refactor IrreparableDB to use pb.IrreparableSegment instead of irreparable.RemoteSegmentInfo
 2019-03-15 16:21:52 -04:00
Bill Thorp
665fd33e3c
Repair queue isolation level fix (#1466) 
Implemented custom SQLite and Postgres Repairqueue Dequeue handlers
 2019-03-14 17:12:47 -04:00
Bill Thorp
52e829c6de
framework for deleting expired Orders from Satellite (#1436) 
framework for deleting old bandwidth agreements
 2019-03-12 16:57:21 -04:00
Dylan Lott
59f1e267c9
Removes concept of email from kademlia metadata (#1435) 
* Removes concept of Email from Kademlia

* Removes kad email

* adds emails back to operator config for satellite

* replace operator configs in testplanet
 2019-03-12 14:05:18 -06:00
Egon Elbre
0560f076a5
fix piece id derivation (#1462) 2019-03-12 19:35:42 +02:00
Alexander Leitner
af889f1554
Move iswriteable to fpath (#1464) 2019-03-12 12:13:40 -04:00
Alexander Leitner
bb77d9b4a6
Warn about permissions when creating identity (#1384) 
* Warn about permissions when creating identity

* Function to determine if directory is writeable

* Check if writable before authorizing

* Remove redeclatarion

* remove windows specific utils

* Nat nits

* Actually test if directory is writeable with file creation
 2019-03-12 10:42:38 -04:00
JT Olio
56ace481c1 judicious import path checking (#1425) 
see also
3ee9780df1
and
https://golang.org/cmd/go/#hdr-Remote_import_paths
and
https://golang.org/cmd/go/#hdr-Import_path_checking

Change-Id: I2ac906b1a1befde21edbcfc9b0a1b6a4863520da
 2019-03-12 14:58:40 +01:00
Bill Thorp
1f4e1a3ab0
framework to delete raw accounting tallies (#1439) 
framework to delete raw accounting tallies, renamed Query
 2019-03-12 09:08:23 -04:00
Bill Thorp
66718cc5e6
Development defaults for configuration (#1430) 
added --dev command line option, cfgstruct.DevFlag(), and cfgstruct.SetupFlag()
 2019-03-12 08:51:06 -04:00
littleskunk
afad4f7010 Improve bwa tests (#1408) 2019-03-08 13:23:43 +01:00
littleskunk
11048bed2c psdb: index on serial number for bandwidth_agreements 2019-03-08 11:53:04 +02:00
Jess G
193a70f0a6
add private listener to grpc server (#1398) 
* add private listener to grpc server

* add changes per init CR

* fix server.close

* add insecure grpc connection, update logs msg

* fix tests, move insecure client

* add private ports to storj-sim, add insecure client to other inspectors

* add ports to test so there arent conflicts

* fix lint err

* fix node started log msg, close public listener

* remove commented out line
 2019-03-07 13:19:37 -05:00
Yaroslav Vorobiov
9cf56c03fa
Bucket usage info rollup table creation and implementation (#1340) 2019-03-06 17:54:48 +02:00
Bryan White
c607abf27c [V3-1147] Ensure certificate validation happens properly (#1403) 
* add regression test & update transport tests

* separate client and server verificiation functions

* goimports
 2019-03-06 09:42:34 -05:00
Egon Elbre
ead4a0ae95
Add new piece id definition (#1411) 2019-03-06 11:17:35 +02:00
Natalie Villasana
036b2f3b49
allocates 1kb for audit bandwidth agreements instead of 256kb (#1404) 2019-03-05 18:58:25 -05:00
aligeti
15034526cd Enhance the Storage Node to keep all BWA's for 90 days (#1374) 
* Enhance the Storage Node to keep all BWA's for 90 days

* warn -> error

* rename enum
 2019-03-05 17:20:46 -05:00
Jess G
3c9d83dbfe
convert psserver dashboard into an inspector (#1407) 
* Convert psserver dashboard into an inspector

* remove dashboard stream, update ps.pb.mock

* fixes for lint errs
 2019-03-05 15:48:37 -05:00
Bryan White
5742131941
tlsconfig refactor / cleanup (#1399) 2019-03-04 21:40:18 +01:00
Bryan White
675e0ef683 [V3-1320] fix empty node ID verification non-error (#1395) 
* small identity refactor:

+ Optimize? iterative cert chain methods to use array instead of slice
+ Add `ToChain` helper for converting 1d to 2d cert chain
  TODO: replace literal declarations with this
+ rename `ChainRaw/RestChainRaw` to `RawChain/RawRestChain`
  (adjective noun, instead of nound adjective)

* add regression tests for V3-1320

* fix V3-1320

* separate `DialUnverifiedIDOption` from `DialOption`

* separate `PingNode` and `DialNode` from `PingAddress` and `DialAddress`

* update node ID while bootstrapping

* goimports & fix comment

* add test case
 2019-03-04 15:03:33 -05:00
aligeti
588e2a51d2 Handles zero bucket count and bucket with no files (#1391) 
* handles zero bucket count and bucket with no files
 2019-03-04 12:47:08 -05:00
Michal Niewrzal
cce84c9914
Cleanup errors handling around piece hash verification (#1382) 2019-03-04 15:29:35 +01:00
Kaloyan Raev
5fa7a4a7c6 Ensure ECClient upload timer is stopped when no more status is expected (#1397) 
This change ensures that the upload timer of ECClient is always stopped after no more status is expected from uploaded pieces. It also ensures that the "Timer expired" message will be logged only if the context is not already cancelled.

This is to avoid confusing logs where a "Timer expired" message is logged significantly later and mixes with similar messages logged from the upload of the next file segments.
 2019-03-04 14:48:13 +01:00
Stefan Benten
5f57d2c906
Reduce Log Messages on Production Settings (#1396) 
* Reduce Tally Log Messages on Production Settings

* Reduce Logging in Piecestore Client as well
 2019-03-04 11:12:43 +01:00
Kaloyan Raev
54f68347ad
Use DialNode for requesting node info (#1394) 2019-03-02 09:34:08 +02:00
Dylan Lott
9abf191b01
Change payments to use statDB for wallet addresses rather than Overlay cache (#1368) 
* Wiring up DumpNodes response for Inspector

* Finalize everything and test that it works

* Get Count and DumpNodes working for Overlay Cache

* WIP updating payment rollup to check statDB instead of overlay

* FIrst pass at updating statDB to take wallet and email

* Passing tests

* use pb.NodeOperator instead of Meta struct

* remove TODO

* revert go.mod

* Get SQL migration working correctly

* Changes Meta to Operator in NodeStats struct

* Adds update operator logic for statDB

* Fix db migrate tests - added v5 snapshot

* User friendly msg for missing snapshot version

* Passing tests

* Change node update to happen in discovery instead of in overlay

* Fix logic and update function calls

* Update comment on UpdateOperator interface method

* Update name of parameter

* Change type of argument to UpdateOperator

* Updates statDB tests
 2019-03-01 10:46:34 -07:00
Cameron
3fd55c7034
remove unused date field in csv row (#1388) 2019-03-01 08:23:27 -05:00
Egon Elbre
3f3209c8d5
fixes to piecestore and psdb (#1380) 
* replace direct reference with an interface in various places
* hide piecePath
* ensure psserver tests don't use path
* ensure psserver tests don't use sql queries directly
 2019-03-01 07:46:16 +02:00
Dylan Lott
bacd683879
Adds DumpNodes method to Kademlia Inspector (#1328) 
* Wiring up DumpNodes response for Inspector

* Finalize everything and test that it works

* remove changes to overlay inspector

* Replace list with iterate method in DumpNodes

* generate proto

* Merge in nodeinfo changes

* Update iterate function in DumpNodes

* Fix import order
 2019-02-28 12:55:27 -07:00
Cameron
018cdd4037
uplink ttl (#1365) 
* add expires flag to uplink cp

* fix ttlDBs delete expired query
 2019-02-28 12:51:24 -05:00
Jennifer Li Johnson
6ecda190cb
Jj/payments cleanup (#1379) 
* Removes date from payments csv and re-query of wallet addr

* removes dbx get wallet addr

* switch order in csv
 2019-02-28 12:13:59 -05:00
Michal Niewrzal
4122c98cb7
Validate piece hash on satellite (#1359) 
The satellite receives pieces signed hashes in Pointer. If signed hash cannot be validated then piece is removed from Pointer and not saved in DB.
 2019-02-28 15:14:54 +01:00
aligeti
993af7ed37 fixes the bwa deletes to be done by serialnum, rather by signature (#1364) 2019-02-27 18:29:23 +02:00
Egon Elbre
b7f8e309d9 add piecestore.Storage interface (#1369) 2019-02-27 10:56:16 +01:00
Bryan White
fde0020c68
dial node/address with bad server cert (#1342) 2019-02-26 19:35:16 +01:00
Jennifer Li Johnson
cefdff535a
Replace list calls with iterates in kademlia routing table (#1318) 
* notes

* adds bool for option to skip self

* wip getKBucketID is still breaking TestAddNode

* fix (#1322)

* remove panic

* remove iteratebuckets

* implements getKBucketRange

* getNodeIDsWithinKBucket

* removes determineFurthestIDWithinK

* wip

* nodeIsWithinNearestK

* removes bucketIDToKey

* removes extra xor in test

* wip

* rebased

* uses create rt w opts

* lint

* fix lint
 2019-02-26 11:07:53 -05:00
aligeti
c6ad7644d2
Total file count through Monkit (#1351) 
* segment, file, byte stats, total and per-bucket; checker: report segment health stats; reports the total num of lost files

* code review updates
 2019-02-26 10:17:51 -05:00
Bryan White
03747bad51
some certificates cmd cleanup: (#1361) 2019-02-26 09:55:52 +01:00
Kaloyan Raev
1ec17653d4 Endpoint for local node info (#1355) 
Adds a new `Info` method to the Kademlia endpoint that returns the following local node info:
* ID
* Type
* Metadata (email and wallet)
* Restrictions (free storage and bandwidth)

The new endpoint is exposed as `inspector kad node-info` command too.
 2019-02-25 19:41:51 +01:00
Michal Niewrzal
81408a3c9e
Use SignedHash on client/uplink side (#1354) 
* psclient receives storage node hash and compare it to own hash for verification
* uplink sends delete request when hashes don't match
* valid hashes are propagated up to segments.Store for future sending to satellite
 2019-02-25 16:57:54 +01:00
Bryan White
b652b6a522 use peer ca whitelist in testplanet (#1337) 2019-02-25 09:38:03 +02:00
Michal Niewrzal
6186b3f90a
Storage node hash calculation on upload (#1347) 
Storage node is calculating hash of uploaded data and send it back to uplink with signature
 2019-02-23 11:46:07 +01:00
Bill Thorp
373b301736
BWA aliases (#1333) 
aliased RBAs and PBAs
 2019-02-22 16:17:35 -05:00
Jennifer Li Johnson
b5447c6608
Routing table tests (#1270) 
This PR includes a new package called testrouting, which implements a very algorithmically slow but hopefully easy-to-keep-operationally-correct in-memory routing table. The routing table also supports writing out its current structure as a DOT graph for visualization. testrouting is primarily meant to help in coming up with generic routing table integration tests.

This PR also includes a new routing table integration test suite that runs against all current routing table implementations. Our existing routing table passes a lot of the tests, but not all of them, still debugging why. I have confirmed the tests should pass with the visualization graphs though.
 2019-02-22 13:39:29 -05:00
aligeti
4ccb80257d
Refactor how bandwidth allocations are stored on storage nodes (#1280) 
* integrated with db migration framework

* updating the migrationdb test

* updating the migrationdb sql v0 and v1 files

* lint warnings

*  code review changes

* fix migration scripts, upated test data

* fixed the local/UTC timing issues

* code review updates

* updated code review comments
 2019-02-22 10:51:39 -05:00
Natalie Villasana
c3d3f41d30 removes some SignedMessage use (#1258) 
Removes most instances of pb.SignedMessage (there's more to take out but they shouldn't hurt anyone as is).

There used to be places in psserver where a PieceID was hmac'd with the SatelliteID, which was gotten from a SignedMessage. This PR makes it so some functions access the SatelliteID from the Payer Bandwidth Allocation instead.

This requires passing a SatelliteID into psserver functions where they weren't before, so the following proto messages have been changed:

 * PieceId - satellite_id field added
   This is so the psserver.Piece function has access to the SatelliteID when it needs to get the namespaced pieceID.
   This proto message should probably be renamed to PieceRequest, or a new PieceRequest message should be created so this isn't misnamed.

 * PieceDelete - satellite_id field added
   This is so the psserver.Delete function has access to the SatelliteID when receiving a request to Delete.
 2019-02-19 23:36:08 -06:00
Dennis Coyle
36e4d8c1c2
restrict actions on PayerBandwidthAllocation (#1326) 
* restrict actions on PayerBandwidthAllocation

* defaults to error if action unknown
 2019-02-19 16:00:58 -05:00
Jennifer Li Johnson
a5aa5f84a1 handle server termination in kademlia tests (#1323) 2019-02-19 20:41:32 +02:00
Michal Niewrzal
8d685217e4
Storagenode migrations (#1299) 
* creates initial migration for psdb
* add test mechanism to validate migration to every version
* fix few small issues in versions.go and context.go
 2019-02-19 10:39:04 +01:00
Bill Thorp
c598ed034d kad.FindNear fix (#1320) 
We realized that the Kademlia FindNear() function was

1. not using XOR distance (AKA _totally broken_)
2. largely a duplicate of the RoutingTable FindNear() function

Changes in this PR:

1. upgraded RoutingTable FindNear() to use iterator and restrictions
2. removed unneeded RoutingTable interface
3. made Kademlia wrap methods that were previously accessed via RoutingTable
4. fixed the tests
 2019-02-15 22:23:35 -05:00
Egon Elbre
12261c0624
Satellite migrations (#1301) 2019-02-14 23:55:21 +02:00
Bryan White
d93c57cffa add certificates claim delete & refactor (#1317) 
* add \`certificates claim delete\` & refactor

* quickfix

* goimports
 2019-02-14 20:17:26 +01:00
Bill Thorp
dfd6589fbe
fix kad refresh bad bucket range (#1315) 
fix kad refresh bad bucket range
 2019-02-14 10:47:03 -05:00
Michal Niewrzal
58792533b9
Fix invalid config key msg for dashbord (#1314) 2019-02-14 14:06:15 +01:00
Stefan Benten
661df2a625 Reduce Log Messages on Level Info (#1313) 2019-02-14 13:50:35 +01:00
Bill Thorp
9b580c5fb6 Repair checker is checking the same 1000 elements all the time (#1297) 
* removed limit on repair, now using cycle

* added BatchIteratorOptions

* consolidated boltdb common.go

* PR feedback cleanup
 2019-02-14 13:33:41 +01:00
Egon Elbre
1a5a9903a2
Move SplitConnstr to a nicer place (#1308) 2019-02-13 23:54:59 +02:00
Bill Thorp
b53f9896d3
Removed ReverseList from KeyValueStore interfaces (#1306) 
Removed ReverseList from KeyValueStore interfaces
 2019-02-13 12:27:03 -05:00
Bill Thorp
2c5716e874
pb.Equal fix (#1304) 
* pb.Equal fix

* file capitalization

* oh snake case
 2019-02-13 09:08:57 -05:00
Michal Niewrzal
b2f9453184
Disable Checker subsystem in tests (#1279) 
* Disable Checker subsystem in tests

* rename field

* remove sleeps and errgroup.Group
 2019-02-11 22:06:39 +01:00
Maximillian von Briesen
f94a2e1161
Node selection method name change (#1296) 
* SelectNodes -> SelectStorageNodes
 2019-02-11 14:24:51 -05:00
Maximillian von Briesen
c9d00d8241
test random selection for new nodes (#1294) 2019-02-11 12:10:32 -05:00
Maximillian von Briesen
c6c23a319b
move node type definition to db level for node selection (#1295) 2019-02-11 11:35:28 -05:00
Egon Elbre
18bd2d7b1a
Fix randomized selection test limit (#1290) 2019-02-11 14:04:00 +02:00
JT Olio
2a59679766 pkg/transport: require tls configuration for dialing (#1286) 
* separate TLS options from server options (because we need them for dialing too)
* stop creating transports in multiple places
* ensure that we actually check revocation, whitelists, certificate signing, etc, for all connections.
 2019-02-11 13:17:32 +02:00
JT Olio
ea28a9a581 pkg/transport: fail on bad certificates (#1285) 2019-02-11 13:02:49 +02:00
Maximillian von Briesen
690d5447a1
Add test for randomized node selection (#1271) 2019-02-09 16:17:49 -05:00
Dylan Lott
612919d95b
Adds tests to discovery/graveyard (#1272) 
* adds tests for cache graveyard

* updates tests for cache graveyard to use sync2
 2019-02-08 13:56:24 -07:00
Egon Elbre
17a7f9d002
Enable planet Merging test (#1281) 2019-02-08 22:35:59 +02:00
Egon Elbre
e37e0c1b5f
Fix server config usage (#1282) 2019-02-08 20:57:17 +02:00
Kaloyan Raev
4e738bbd44
Fix randomness in stopping nodes in TestGetObjectStream (#1277) 2019-02-08 15:20:12 +02:00
Michal Niewrzal
c7ad427d95
Move metainfo config (#1234) 
* Move metainfo config

* move config to uplink

* reorganize code

* use satellite apikey

* comment update
 2019-02-08 13:57:35 +01:00
Egon Elbre
995927dcd2
Disable psserver TestRetrieve (#1273) 2019-02-08 11:54:50 +02:00
Egon Elbre
9c1e299f3c
Ensure everyone sees everyone else (#1275) 2019-02-08 11:25:13 +02:00
Egon Elbre
5a13bdec18
close revocation database in test (#1274) 2019-02-08 11:13:32 +02:00
paul cannon
c35b93766d
Unite all cryptographic signing and verifying (#1244) 
this change removes the cryptopasta dependency.

a couple possible sources of problem with this change:

 * the encoding used for ECDSA signatures on SignedMessage has changed.
   the encoding employed by cryptopasta was workable, but not the same
   as the encoding used for such signatures in the rest of the world
   (most particularly, on ECDSA signatures in X.509 certificates). I
   think we'll be best served by using one ECDSA signature encoding from
   here on, but if we need to use the old encoding for backwards
   compatibility with existing nodes, that can be arranged.

 * since there's already a breaking change in SignedMessage, I changed
   it to send and receive public keys in raw PKIX format, instead of
   PEM. PEM just adds unhelpful overhead for this case.
 2019-02-07 14:39:20 -06:00
Bill Thorp
0b35762105
Convert Payments to use SQL, for SUM() and Wallet (#1266) 
* payments query no longer DBX, using SQL

* sum in SQL

* removed old function

* fixed rollup test

* wrap errors

* removed DBX code
 2019-02-07 15:26:55 -05:00
aligeti
b736ae4823
Store the uplinks public key on the satellite so that it can verify bandwidth requests in the future (#1042) 
* integrated with bwagreement & psserver

* integrated with pointerdb

* code review updates

* refactor after code review

* uplinkdb rename to certdb

* Code review changes
 2019-02-07 14:22:49 -05:00
paul cannon
ef61c170b1
Consolidate key/cert/signature encoding and decoding (#1243) 2019-02-07 12:40:28 -06:00
paul cannon
32882daf38
SHA256Hash() and TBSBytes() needn't return error (#1242) 2019-02-07 11:08:52 -06:00
paul cannon
0032147665 regroup things related to public-key cryptography (#1241) 2019-02-07 10:04:29 +01:00
Cameron
bfe8060f4d
Rollup tests (#1224) 
* createdAt field no longer autoinsert

* rollup tests
 2019-02-06 23:16:24 -05:00
Bryan White
3b54cf0e15
identity improvements: (#1215) 2019-02-06 17:40:55 +01:00
Egon Elbre
30f89b0362
Ensure overlay tests run against postgres (#1232) 2019-02-06 15:32:42 +02:00
Egon Elbre
bb11d83ed0
Proper planet shutdown (#1249) 2019-02-06 15:19:14 +02:00
Michal Niewrzal
aa3836c06a
Add checker irreparable test (#1246) 
* Add checker irreparable test

* use RepairUnixSec

* wait for second check

* use storagenode.ID()

* Update comments
 2019-02-06 14:03:38 +01:00
Egon Elbre
fdbe2db273
Remove node package and simplify DHT interface (#1233) 2019-02-06 14:37:17 +02:00
Egon Elbre
2d6b5faf6d
Fix discovery busy-spin (#1248) 2019-02-06 14:25:29 +02:00
Bryan White
00554f0876
add certificates claims+ (#1216) 2019-02-06 12:00:26 +01:00
Bryan White
7b7e6c43f8
better batch-generation (#1219) 2019-02-06 09:04:12 +01:00
Egon Elbre
218e95728c
psserver: swallow io.EOF when getting full content in FileRanger (#1238) 2019-02-06 07:41:41 +02:00
paul cannon
331f0cc845 don't put nil nodes in GetResponse (#1218) 2019-02-05 23:12:27 +02:00
Bill Thorp
690e8b2061
storage node cert cache (#1226) 
* draft

* still errors

* double close fix

* added tests

* weird, goimports must not be working

* renames

* missed one

* forgot to save:
 2019-02-05 12:57:56 -05:00
Egon Elbre
07412698a9
Use lower bcrypt cost for console.Service when running tests and disable flaky tests (#1236) 2019-02-05 19:31:53 +02:00
Yaroslav Vorobiov
2ff0d9d435
Add bucket to project relationship on satellite (#1143) 
* integrate console api keys with buckets in pointerdb

* fix test

* fix tests kvmetainfo

* linter fix

* disable account activation

* fix test

* review fixes

* fix comments

* little refactoring

* remove debug println

* fix typo

* disable activation in a propper way

* fix test

* fix imports

* fix uplink count in testplanet

* move key creation to planet.newUplink
 2019-02-05 19:22:17 +02:00
Egon Elbre
b16f27c54b
Ensure discovery tests run against postgres (#1230) 2019-02-05 19:03:16 +02:00
Kaloyan Raev
3c73d3a33c Fixes intermitent failures in storage/ec TestPut (#1239) 2019-02-05 18:49:52 +02:00
Egon Elbre
39c1e5ccec
Ensure checker tests run against postgres (#1229) 2019-02-05 18:00:52 +02:00
Egon Elbre
94ee200e35
Ensure cursor tests run against postgres (#1228) 2019-02-05 17:45:09 +02:00
Egon Elbre
6564357e13
Ensure that tally tests run against postgres (#1227) 2019-02-05 16:23:45 +02:00
Kaloyan Raev
dd76829d10
Improve logic for cutting the long tail during upload (#909) 2019-02-05 12:54:25 +02:00
Egon Elbre
87d6410b50 Revert "Remove node package and simplify DHT interface." 
This reverts commit 03ec1ff92d.
 2019-02-05 10:38:48 +02:00
Egon Elbre
03ec1ff92d Remove node package and simplify DHT interface. 2019-02-05 10:37:24 +02:00
Dylan Lott
21a58d0ece
node graveyard resurrection (#1186) 
* deletes nodes from cache if unresponsive

* update error messages

* WIP restructuring discovery tickers

* passes tests

* Removes log messages

* Passes up error in graveyard ticker
 2019-02-04 14:52:06 -07:00
Egon Elbre
c37e0c1b6d
testplanet.Run (#1222) 2019-02-04 22:37:46 +02:00
Michal Niewrzal
2f8b3da1af
Upload/download for testplanet Uplink (#1183) 
* Upload/download for testplanet Uplink

* check error in tests

* cleanup

* refactor node -> uplink

* add missing test file

* rest of refactoring

* workaround to resolve cycles in tests

* rename method

* add missing comments

* review comments

* use KiB
 2019-02-04 17:56:10 +01:00
Michal Niewrzal
53c11dfc5d
Stop storagenode on ctrl+c (#1220) 
* Stop storagenode on ctrl+c

* Cancel grpc server

* handle error

* handle error

* use errgroup

* fix check-travis-tidy

* remove pipefail
 2019-02-04 15:50:55 +01:00
paul cannon
0238a7785b
make a map object before inserting into it (#1217) 
I think this is causing panics.
 2019-02-03 18:42:20 -06:00
Bill Thorp
0f662b8e38 sql based tally (#1166) 
* WIP

* wacky changes

* more

* it builds.... IT BUILDS!!!

* fixed SQL, broke out saving tallies for testing

* shorter lines

* fixed SQL, moved tally_test to testplanet

* lint

* WIP logic error preventing PUT and GETs to same serialnum

* fixed BWA test

* fixed temporary brain failure

* eliminated magic numbers

* cleaned up satellite uplink stats

* use errs.Combine instead

* thrashing

* fixed tally erroneous error msg

* fixed tally test

* lint

* SQL syntax attempt to fix

* spelling error

* made bwa db test resist old postgres data

* postgres pk error msg hunting

* postgres pk error msg hunting

* postgres

* err might be nil?

* fixed error logging bug

* hopefully solved postgres issue

* using rebind

* moved tests to _test package

* fixing test dirs

* finally made sense of Egons package name feedback

* UTC, array fixes
 2019-02-01 13:50:12 -05:00
Egon Elbre
4117600f32
Cleanup repository a bit (#1203) 2019-02-01 20:21:00 +02:00
Kaloyan Raev
5c2b67db63 Fix EOF error on audits (storage node) (#1208) 
* Fix EOF error on audits

* Specify the correct number of bytes to CopyN instead of checking for EOF

* Fix imports order
 2019-02-01 10:57:11 -07:00
Kaloyan Raev
a90aa18582 Fix panic in PointerDB.Get (#1209) 
* Fix panic in PointerDB.Get

* Allow nil nodes, check if v is nil instead

* More checks like this

* Remove unnecessary check

* More checks
 2019-02-01 10:55:47 -07:00
JT Olio
41d19f3c3d
miniogw: fix buffer size alignment (#1213) 
Change-Id: I47e0e67e385d236ac0338fe36f11cc7c00e23dba
 2019-02-01 10:42:30 -07:00
Dylan Lott
c77a647174
Updates transport dialopts to use WithBlock() (#1201) 
* Updates transport opts to use WithBlock()

* Removes unnecessary withBlock's and adds WithBlock to DialAddress
 2019-02-01 10:40:35 -07:00
Egon Elbre
5a63c00442
Fix issues with blocking during startup (#1212) 2019-02-01 19:28:40 +02:00
Bryan White
3e3088e37c improve identity create ux (#1211) 2019-02-01 11:58:33 -05:00
Egon Elbre
7a85c5c1c6
Fix audit node ids (#1207) 2019-02-01 16:48:57 +02:00
Egon Elbre
00570df1f9
Add top flag to tparse (#1205) 2019-02-01 16:35:13 +02:00
Egon Elbre
b91d77436f
Test merging planets (#1181) 2019-02-01 15:32:28 +02:00
Egon Elbre
0c366c1ed2 Clearer code for node selection (#1173) 2019-01-31 13:49:00 -05:00
Kaloyan Raev
8c357ac571
Correctly update uptime when ping fails (#1190) 2019-01-31 16:51:38 +02:00
Dylan Lott
6025f9f19e
deletes nodes from cache if unresponsive (#1185) 
* deletes nodes from cache if unresponsive

* update error messages
 2019-01-30 14:15:24 -07:00
Egon Elbre
97714d150f
Remove unused kademlia.Disconnect method (#1178) 2019-01-30 22:57:13 +02:00
Egon Elbre
d5346982c2
Delete provider package (#1177) 2019-01-30 22:47:21 +02:00
Egon Elbre
19de56bf83
Fix race in discovery test (#1184) 2019-01-30 22:29:33 +02:00
Alexander Leitner
7c741c16d1
Readable sizes in dashboard (#1180) 
* Readable sizes

* Use tabwriter

* Added more addresses to dashboard stats

* Use kad to look up addresses

* Move dashboard command to dashboard.go
 2019-01-30 14:49:25 -05:00
Michal Niewrzal
14434594c4
Re-enable repair checker tests (#1172) 
* Re-enable repair checker tests

* simpler NodeID construction

* adjust benchmark
 2019-01-30 19:34:44 +01:00
Dylan Lott
5e1ab841cf Cache refresher (#1171) 
* got tests passed

* wire up paginate function for cache node retrieval

* Add tests for paginate but they're failing

* fix the test arguments

* Updates paginate function to return more variable

* Updates

* Some test and logic tweaks

* improves config handling in discovery

* adds refresh offset to discovery struct
 2019-01-30 11:29:18 -05:00
Egon Elbre
6132ce86b7
Remove utils.LogClose (#1169) 2019-01-29 22:42:27 +02:00
Natalie Villasana
8d7944bcf8
new node selection in overlay (#1136) 2019-01-29 14:42:43 -05:00
Egon Elbre
1df81b1460
Separate garbage collect logic from psdb (#1167) 2019-01-29 17:41:01 +02:00
Egon Elbre
54eb1b099c
don't display message unless logger has been specified (#1165) 2019-01-29 14:12:24 +02:00
Egon Elbre
e1a8bbdcb6
Kademlia flags cleanup (#1137) 2019-01-29 08:51:07 +02:00
Dylan Lott
bb2588e9ae
Refactors kademlia inspector to use FindNear (#1157) 
* WIP

* update identity paths to get dump-nodes working

* Remove commented out code

* remove comment

* updates from code reviews

* updates linter
 2019-01-28 15:53:37 -07:00
Bill Thorp
60946c2024
make bandwidth agreements sensible: without []byte's (#1152) 
removed []byte's from bandwidth agreement protocol buffers
 2019-01-28 14:45:25 -05:00
Dylan Lott
df903ea215
Decreases refresh time per bucket in Kademlia (#1150) 2019-01-28 12:36:20 -07:00
Egon Elbre
d50c07e56c
Implement WorkGroup (#1151) 2019-01-28 21:04:42 +02:00
Egon Elbre
cecd4b0816
Remove server aliases (#1154) 2019-01-28 17:04:53 +02:00
Bryan White
2b20acbec9
identity cleanup (#1145) 2019-01-26 15:59:53 +01:00
Egon Elbre
49dacb662c
Improve bwagreement tests (#1149) 2019-01-26 01:06:38 +02:00
Egon Elbre
f7e18882c4
Kademlia closing race (#1139) 2019-01-26 00:33:20 +02:00
Bill Thorp
8ba4b61e59
Make Bandwidth Agreements Secure / Trustable (#1117) 
* Added cert chains and nodeid verification to bandwidth agreement
 2019-01-25 13:05:21 -05:00
Bryan White
7bed8050aa
Improve identity cli ux: (#1142) 2019-01-25 17:55:45 +01:00
Jennifer Li Johnson
5397efe5c1
removes standalone payments service (#1134) 
* remove standalone payments service - will integrate within satellite directly

* regenerate proto files
 2019-01-25 11:23:25 -05:00
Egon Elbre
85b43926b4
Separate identity from server config (#1138) 2019-01-25 16:54:54 +02:00
Egon Elbre
a1027c7836
Make inmemory a separate command (#1127) 2019-01-24 23:04:29 +02:00
Egon Elbre
78b0414e6c
internal/sync2: fix file handle leak (#1124) 2019-01-24 22:48:54 +02:00
Cameron
f5b1e77cf2 rollup timestamp fix (#1132) 
* create timestamp if isNew

* pass isNil into SaveRollup
 2019-01-24 15:41:22 -05:00
Jennifer Li Johnson
856b98997c
updates copyright 2018 to 2019 (#1133) 2019-01-24 15:15:10 -05:00
Jennifer Li Johnson
030d294399
adds logs in rollup and tally run (#1131) 2019-01-24 15:05:53 -05:00
Jennifer Li Johnson
70940b8710
Making sure payments works (#1122) 
* byte hours

* updates comment

* removes payment test data methods

* wip

* adds tally and rollup to peer

* remove sim test for payment (for now)

* lint error
 2019-01-24 13:44:14 -05:00
Alexander Leitner
c4afec9328
quick log cleanup for storage node (#1121) 
* Clean up logs
 2019-01-24 11:09:37 -05:00
Egon Elbre
28427e7fad
Change identity command (#1128) 2019-01-24 17:41:16 +02:00
Egon Elbre
5de7f8af7f
Satellite Peer (#1119) 2019-01-23 21:58:44 +02:00
Cameron
95d2d54fc3
storagenode approvedSatelliteIDs (#1116) 
* add config fields for satellite restriction on psserver

* add whitelistedSatIDs to psserver Server struct

* check pbwa satellite ID against whitelist

* add whitelist to psserver tests

* reword help message, make approved() a method on server
 2019-01-23 12:56:12 -05:00
Michal Niewrzal
3fdb47e31c Try accessing bootstrap node for verification (#1089) 2019-01-23 17:48:46 +02:00
Egon Elbre
8677c68b68
Print successfully connected message every 100 times (#1118) 2019-01-23 14:07:22 +02:00
Egon Elbre
05b96f95a3
Change default difficulty to 30 (#1113) 2019-01-23 13:36:19 +02:00
Egon Elbre
b6c61cdd55
Use storagenode.Peer for storagenode (#1107) 2019-01-23 12:39:03 +02:00
Egon Elbre
99d3b7a3c8
Fix import grouping (#1111) 2019-01-22 17:48:23 +02:00
Egon Elbre
ad49f4b994
Enable linters (#1105) 2019-01-22 17:09:02 +02:00
Bryan White
322a2813f7
oops (#1110) 2019-01-22 15:34:40 +01:00
Egon Elbre
6c48d69c38
Disable mounting for bsd-like systems (#1109) 
* Disable mounting for bsd-like
* Remove debug code
 2019-01-22 15:28:57 +02:00
Bryan White
8edfd43abb
certificates config and --certs-dir fixes (#1093) 2019-01-22 13:35:48 +01:00
Michal Niewrzal
eb2d858ffc Use pointerdb.Service instead of pointerdb.Server (#1099) 2019-01-19 13:58:53 -05:00
Alexander Leitner
aabcceef52 Don't create identity when running dash (#1098) 
* Don't create identity when running dash

* only use configs for connecting with dash, delete unused client example

* use server address
 2019-01-18 18:08:45 +01:00
Cameron
ecd704692e
wip testing and timestamp fix (#1100) 
* wip testing and timestamp fix

* fixed import cycle
 2019-01-18 11:53:23 -05:00
Michal Niewrzal
cc1bdef8b7
Move pointerdb logic to a service (#1037) 
* test

* Move pointerdb logic to service

* tset

* reorder constructor params

* restore field
 2019-01-18 10:10:21 -05:00
JT Olio
2d53a33bc5 pkg/node: fix connection logging (#1096) 
* pkg/node: fix connection logging

Change-Id: Iac552ac5eb126642fc1d01bae5f44720fafad536

* debug logging

Change-Id: Ibb95bc7baf1d4d0f07640ae807b2eb2f943aaab0
 2019-01-18 10:03:19 -05:00
Egon Elbre
2d00f882a3
Kademlia Dialer (#1092) 2019-01-18 10:00:56 -05:00
Egon Elbre
78dc02b758 Satellite Peer (#1034) 
* add satellite peer

* Add overlay

* reorganize kademlia

* add RunRefresh

* add refresh to storagenode.Peer

* add discovery

* add agreements and metainfo

* rename

* add datarepair checker

* add repair

* add todo notes for audit

* add testing interface

* add into testplanet

* fixes

* fix compilation errors

* fix compilation errors

* make testplanet run

* remove audit refrences

* ensure that audit tests run

* dev

* checker tests compilable

* fix discovery

* fix compilation

* fix

* fix

* dev

* fix

* disable auth

* fixes

* revert go.mod/sum

* fix linter errors

* fix

* fix copyright

* Add address param for SN dashboard (#1076)

* Rename storj-sdk to storj-sim (#1078)

* Storagenode logs and config improvements  (#1075)

* Add more info to SN logs

* remove config-dir from user config

* add output where config was stored

* add message for successful connection

* fix linter

* remove storage.path from user config

* resolve config path

* move success  message to info

* log improvements

* Remove captplanet (#1070)

* pkg/server: include production cert (#1082)

Change-Id: Ie8e6fe78550be83c3bd797db7a1e58d37c684792

* Generate Payments Report (#1079)

* memory.Size: autoformat sizes based on value entropy (#1081)

* Jj/bytes (#1085)

* run tally and rollup

* sets dev default tally and rollup intervals

* nonessential storj-sim edits (#1086)

* Closing context doesn't stop storage node (#1084)

* Print when cancelled

* Close properly

* Don't log nil

* Don't print error when closing dashboard

* Fix panic in inspector if ping fails (#1088)

* Consolidate identity management to identity cli commands (#1083)

* Consolidate identity management:

Move identity cretaion/signing out of storagenode setup command.

* fixes

* linters

* Consolidate identity management:

Move identity cretaion/signing out of storagenode setup command.

* fixes

* sava backups before saving signed certs

* add "-prebuilt-test-cmds" test flag

* linters

* prepare cli tests for travis

* linter fixes

* more fixes

* linter gods

* sp/sdk/sim

* remove ca.difficulty

* remove unused difficulty

* return setup to its rightful place

* wip travis

* Revert "wip travis"

This reverts commit 56834849dcf066d3cc0a4f139033fc3f6d7188ca.

* typo in travis.yaml

* remove tests

* remove more

* make it only create one identity at a time for consistency

* add config-dir for consitency

* add identity creation to storj-sim

* add flags

* simplify

* fix nolint and compile

* prevent overwrite and pass difficulty, concurrency, and parent creds

* goimports
 2019-01-18 08:54:08 -05:00
Dylan Lott
7bc1944a08 Dashboard updates (#1087) 2019-01-18 05:41:25 -05:00
Egon Elbre
bbf81f2479 Consolidate identity management to identity cli commands (#1083) 
* Consolidate identity management:

Move identity cretaion/signing out of storagenode setup command.

* fixes

* linters

* Consolidate identity management:

Move identity cretaion/signing out of storagenode setup command.

* fixes

* sava backups before saving signed certs

* add "-prebuilt-test-cmds" test flag

* linters

* prepare cli tests for travis

* linter fixes

* more fixes

* linter gods

* sp/sdk/sim

* remove ca.difficulty

* remove unused difficulty

* return setup to its rightful place

* wip travis

* Revert "wip travis"

This reverts commit 56834849dcf066d3cc0a4f139033fc3f6d7188ca.

* typo in travis.yaml

* remove tests

* remove more

* make it only create one identity at a time for consistency

* add config-dir for consitency

* add identity creation to storj-sim

* add flags

* simplify

* fix nolint and compile

* prevent overwrite and pass difficulty, concurrency, and parent creds

* goimports
 2019-01-18 11:36:58 +01:00
Alexander Leitner
944daeab02
Closing context doesn't stop storage node (#1084) 
* Print when cancelled

* Close properly

* Don't log nil

* Don't print error when closing dashboard
 2019-01-17 16:08:05 -05:00
Jennifer Li Johnson
c4b876ddc5
Jj/bytes (#1085) 
* run tally and rollup

* sets dev default tally and rollup intervals
 2019-01-17 14:39:32 -05:00
JT Olio
a255e385df memory.Size: autoformat sizes based on value entropy (#1081) 2019-01-17 20:47:57 +02:00
Jennifer Li Johnson
e6fbf63620
Generate Payments Report (#1079) 2019-01-17 13:34:13 -05:00
JT Olio
b2e69d847f pkg/server: include production cert (#1082) 
Change-Id: Ie8e6fe78550be83c3bd797db7a1e58d37c684792
 2019-01-17 12:36:45 -05:00
Michal Niewrzal
f9abb4584c
Storagenode logs and config improvements (#1075) 
* Add more info to SN logs

* remove config-dir from user config

* add output where config was stored

* add message for successful connection

* fix linter

* remove storage.path from user config

* resolve config path

* move success  message to info

* log improvements
 2019-01-16 21:41:43 -05:00
Dylan Lott
82b9b54695
storagenode dashboard is working and passing tests locally (#1072) 
* storagenode dashboard is working and passing tests locally

* linter fixes

* linter fixes

* moar linter fixes god

* review fixes and updates

* linter fixes
 2019-01-16 16:09:13 -05:00
paul cannon
a2333c4809
trivial: log bootstrap node address used to DEBUG (#1063) 2019-01-16 15:43:00 -05:00
Bill Thorp
342dc857f5 rollup query (#1056) 
* implemention notes

* more notes

* starting rollup query

* not working yet

* fixed build

* fixed cfg bug

* change context cancelled errs to debugs

* using byte hours for at rest tally

* revert changes to go.mod

* comment fixes

* prevent double recording tallies in rollup

* linting

* stop leaking dbx

* nodeid changes

* fix build
 2019-01-16 14:30:33 -05:00
paul cannon
b3713b7d92
trivial: log "connection to node failed" at DEBUG level (#1064) 
instead of ERROR, because errors contacting other nodes in the network
are entirely normal and expected, and they don't need urgent attention.
 2019-01-16 13:01:59 -05:00
Michal Niewrzal
01e4811a6f
Fix ignored overridden flag in user config (#1059) 
* Fix ignored overridden flag in user config

* rename variable
 2019-01-15 16:03:24 -05:00
Egon Elbre
f8906ce000
Change overlay cache database interface (#1047) 2019-01-15 11:08:45 -05:00
Michal Niewrzal
ab0c18de3a
Fix SN overrided params (#1049) 
* Fix SN overrided params

* remove one more override

* fix missing var

* overriding default for storage node

* restore removed code

* fix docker
 2019-01-15 10:43:10 -05:00
paul cannon
248ee6438f
use Node ID for metrics ID (#1052) 
This change removes automatic metrics reporting for everything going
through process.Exec(), and re-adds metrics reporting for those commands
which are expected to be long-lived. Other commands (which may have been
intermittently sending metrics before this, if they ran unusually long)
will no longer send any metrics.

For commands where it makes sense, a node ID is used as the metrics ID.
 2019-01-15 10:02:54 -05:00
Kaloyan Raev
a63abf8fab
Save only user-specific flags to storage node's config.yaml (#1051) 2019-01-15 15:55:33 +02:00
Yehor Butko
e38cf8f50d
Renaming and moving pkg/satellite to satellite/console (#1054) 
* [WIP] V3-853 Merge the satellite DB into the master database

* Removing consoleDB from satelliteDB

* Fixing tests for satellite/console

* fixing linter

* sorting imports in satellite/console

* fixing console config

* fixing linter
 2019-01-15 15:03:24 +02:00
littleskunk
2aebbeed34
add more bwagreement unit tests (#1046) 
* add more bwagreement unit tests
 2019-01-15 05:44:56 +01:00
Alexander Leitner
bfde515391
Clean up Storage node setup (#1013) 
* Edit config on Setup

* Default to 1TiB storage space and 500GiB bandwidth

* Use human readable formats

* Use memory

* units of 1024 are measured with KiB/MiB etc

* pkg/cfgstruct: allow values to be configured with human readable sizes

Change-Id: Ic4e9ae461516d1d26fb81f6e44c5ac5cfccf777f

* Modify tests

* Removed comments

* More merge conflict stuff resolved

* Fix lint

* test fixin

Change-Id: I3a008206bf03a4446da19f642a2f9c1f9acaae36

* Remove commented code but secretly leave it in the histroy forever

* Move flag definition to struct
 2019-01-14 16:19:15 -05:00
JT Olio
87925789de pkg/inspector: split functionality to be per-service (#1029) 
now kad inspector features exist on every server that has
kademlia running. likewise, overlay and statdb.

this means kad inspection features are now available on
storage nodes

Change-Id: I343c873552341de13302bfb7a5d79cccc84fc6b8
 2019-01-14 11:47:22 -07:00
Bill Thorp
8b82114b7f
agreement sender bug fix (#1044) 2019-01-11 19:31:34 -05:00
Bill Thorp
273c3924ca
continue agreement sender loop on error (#1020) 
* continue agreement sender loop on error
* added REJECTED grpc status
 2019-01-11 13:15:49 -05:00
Bryan White
9e1ec633d8
Improve cert signing integration test+: (#997) 2019-01-11 15:59:35 +01:00
Michal Niewrzal
e13f7c7a7d
Fix unstarted pointerdb responsibility (#1033) 2019-01-11 13:45:11 +01:00
Egon Elbre
8893884044
convert piecestorage into a struct (#1024) 2019-01-11 13:26:39 +02:00
Yehor Butko
6e3cbb9712
V3-978 Create missing unit tests for satellitedb (#1025) 
* V3-978 Create missing unit tests for satellitedb

* fixing linter

* removing redundant checks from tests
 2019-01-11 12:08:51 +02:00
aligeti
b0c59a2afc Empty satellite signature crashing (#1007) 2019-01-10 20:31:47 -07:00
Maximillian von Briesen
058ef585aa
do not try to dial nil node from audit verifier (#1027) 
* do not try to dial nil node from audit verifier

* add todo
 2019-01-10 15:13:40 -05:00
Cameron
ef50bbf8b6
satellites reject expired BWAs (#1015) 
* add 45 day expiration to PBAs

* add expiration field to relevant areas, DeleteExpired placeholder

* reject expired BWAs

* test for expired BWAs

* add BwExpiration config value
 2019-01-10 13:30:55 -05:00
Maximillian von Briesen
0642592420 Replace pdb client with server in audit service (#1016) 
* add logger to audit service

* use pointerdb instead of pdbclient in audit

* linter fixes
 2019-01-10 11:35:18 -05:00
Bogdan Artemenko
625ae46ae5
Satellite web Endless scroll and 'Sort by' for Team Members page. (#999) 
* Implemented endless scroll and 'sort by' for Team Members page.
* Implemented actions name constants usage instead raw strings.
 2019-01-10 16:44:15 +02:00
Egon Elbre
eb69ecadec
Storage Node Peer (#1005) 2019-01-10 15:13:27 +02:00
Michal Niewrzal
e5920ae0a4
Storage node config params descriptions cleanup (#1004) 
* Storage node params descriptions cleanup

* reorg agreements sender param

* review changes
 2019-01-10 13:50:50 +01:00
Michal Niewrzal
cfbaeaf8f2
Some pointerdb methods doesn't validate api key (#1019) 2019-01-10 13:07:08 +01:00
Bill Thorp
fb4a11ebb0
Differentiate GET and PUT operations in the tally service (#965) 
* draft of new bandwidth agreement types
* updated storagenode report for new types
* use correct pba types
 2019-01-10 06:41:57 -05:00
JT Olio
f8ee5f88e5 cmd/bootstrap: a kademlia-only server (#1001) 2019-01-09 17:59:51 +02:00
Egon Elbre
756f49fea1
remove discovery endpoint (#1006) 2019-01-09 17:18:42 +02:00
aligeti
457f909a74 Satellite accepts manipulated paychecks V3-1023 (#972) 
* fix - Satellite crashing on receiving a manipulated bandwidthagreement

* provider.PeerIdentityFromContext called twice. Remove one

* add storage node ID to serial number

* remove serialNum query and transaction

* add uuid to GeneratePayerBandwidthAllocation for testing

* enable expected failure on duplicate serialnum cases

* Revert "enable expected failure on duplicate serialnum cases"

This reverts commit 5948f43ed1741c280f0bb34a86c1c490365417bc.

* enable expected failure on duplicate serialnum cases
 2019-01-09 10:02:03 -05:00
Bill Thorp
ea47d27c1b
Get satellites to send BW agreements to via KAD (#985) 
* moved from hardcoded overlay IP to kad
 2019-01-09 09:41:50 -05:00
Egon Elbre
2ee4eed465
Add missing satellite arguments (#998) 2019-01-09 01:41:01 +02:00
Dennis Coyle
add442aaa6
change default BootstrapAddr (#994) 2019-01-08 15:50:28 -05:00
Egon Elbre
9e55a7209d
Fix data-race in UpdateSelf (#991) 2019-01-08 18:01:22 +02:00
Yaroslav Vorobiov
dfeaf406f0
Satellite api tests (#967) 2019-01-08 16:54:35 +02:00
Egon Elbre
db5a990719
lint imports grouping (#993) 2019-01-08 16:05:14 +02:00
Yaroslav Vorobiov
425ac45d89
normalize login email (#992) 
transform email to lower case before user creating, updating, authorization
 2019-01-08 15:54:12 +02:00
Egon Elbre
320a0708f4
Allow empty key and better error handling (#986) 2019-01-08 10:22:54 +02:00
Alexander Leitner
c9d900fe4d Process for updating storage node stats in kademlia (#904) 
* Process for updating node stats in kademlia

* Mutex lock

* Rename and set up the refresher

* Wrap errors

* Lock should be around the if

* Address comments; Build updateSelf funciton in Kademlia Routing Table

* Added test

* Address comments
 2019-01-07 22:05:11 -05:00
JT Olio
1faeeb49d5 prepare key generation for launch (#979) 
* pkg/identity: use sha256 instead of sha3 for pow

Change-Id: I9b7a4f2c3e624a6e248a233e3653eaccaf23c6f3

* pkg/identity: restructure key generation a bit

Change-Id: I0061a5cc62f04b0c86ffbf046519d5c0a154e896

* cmd/identity: indefinite key generation command

you can start this command and leave it running and it will fill up your
hard drive with node certificate authority private keys ordered by
difficulty.

Change-Id: I61c7a3438b9ff6656e74b8d74fef61e557e4d95a

* pkg/storj: more node id difficulty testing

Change-Id: Ie56b1859aa14ec6ef5973caf42aacb4c494b87c7

* review comments

Change-Id: Iff019aa8121a7804f10c248bf2e578189e5b829d
 2019-01-07 13:02:22 -05:00
littleskunk
9a536f2cc4
Test bwagreement serial number (#978) 
* bwagreement serial number tests
 2019-01-07 16:19:05 +01:00
Michal Niewrzal
4e857ea133
Add setup tag for config structs (#968) 
* Fix ignored setup arguments

* fix linter errors

* small params cleanup

* fix integration tests

* cleanup in configs

* Add `setup` tag for config structs

* fix broken if statement

* cleanup captplanet config

* remove reduntant return

* add missing Signer config

* review comments

* local variable renamed

* remove unused var
 2019-01-07 12:06:10 +01:00
Michal Niewrzal
bacc1b13b4
Verifier should use payer bandwidth alloc from satellite (#960) 
* Verifier should use payer bandwidth alloc from satellite

* unit test added

* fix typo

* review comments applied

* fix renamed field
 2019-01-06 19:51:01 +01:00
Jennifer Li Johnson
4e3e1be1a0
returns nil if no nodeData present in CalculateAtRestRaw (#976) 2019-01-04 14:54:54 -05:00
Bryan White
b6611e2800
automate certificate signing in storage node setup (#954) 2019-01-04 18:23:23 +01:00
aligeti
ce1db97ce6
fix manipulated paychecks V3-1023 (#961) 
Satellite crashing on receiving a manipulated bandwidthagreement V3-1022
 2019-01-04 11:26:26 -05:00
Yehor Butko
262753bde6
V3-1029 Update list Project Logic (#966) 2019-01-04 13:32:21 +02:00
Matt Robinson
c0e6b62708
Test all-in-one (#900) 
* Add test for aio

* Don't trust the user to have images built for a version

* Make travis run the aio test

* Add missing values to docker-compose, sort some things, consider the gateway image

* today's changes

* config changed, again

* more fixes

* Expose satellite port on localhost:7778

* Add retries and a timeout around the big-testfile test in AIO

* Another config value changed

* Make this error message a little more useful

* Fix nil condition
 2019-01-03 14:54:27 -05:00
Cameron
9fa874459a
remove starting node log in piecestore server (#964) 2019-01-03 13:57:12 -05:00
Egon Elbre
011b889ba7
pkg/process: properly quote arguments (#963) 2019-01-03 20:00:59 +02:00
Egon Elbre
26c2564bd8
pkg/kademlia: simplify code (#958) 2019-01-02 20:57:11 +02:00
Jennifer Li Johnson
a2fa5c4c5a Proper NodeType Handling (#873) 
* adds enums to nodetype

* updating nodetype todos

* ran pb updates

* reorder nodetypes

* adding checks

* wip

* wip

* wip

* bug in test-captplanet

* wip

* add values to storagenode, satellite, captplanet binaries

* Cleanup

* more cleanup

* wip

* lint

* lint

* wip

* fixes bug

* regenerate protos

Change-Id: Id270212e8c7479e52641058042cf23b5317ab773

* limit node type changes to kademlia

Change-Id: I9c1a6cc4a79e05086627f0fdeb5028c62ce754f4

* dpanic

Change-Id: Id952a2ad13c807ebaea0ec0a875405e267d81c3e

* review comments

Change-Id: I7f9b77ef22779dd012fd490375b136014f51f834
 2019-01-02 11:47:34 -07:00
Egon Elbre
89db0fe9f5
storj-sdk base code (#799) 2019-01-02 20:07:49 +02:00
Egon Elbre
0ca03b41e2
Use generated locking database implementation (#947) 2019-01-02 19:53:27 +02:00
Bryan White
249244536a
CSR Service (part 2): cert signing rpc (#950) 
* CSR Service:

+ implement certificate sign rpc method
+ implement certificate signer client/server
+ refactor `AuthorizationDB#Create`
+ refactor `NewTestIdentity`
+ add `AuthorizationDB#Claim`
+ add `Token#Equal`
+ fix `Authorizations#Marshal` when marshaling identities and certificates
+ tweak `Authorization#String` format
+ cert debugging improvements (jsondiff)
+ receive context arg in `NewTestIdentity`
+ misc. fixes
 2019-01-02 12:39:17 -05:00
Michal Niewrzal
b712fbcbb0
Fix 'empty queue' error when satellite starts (#939) 2019-01-02 17:00:32 +01:00
Bill Thorp
e4b3dab446
Add config editing via the CLI for storage nodes (#919) 
* added editor and updated storagenode cmd
* switched to custom yaml gen
 2019-01-02 10:12:07 -05:00
Matt Robinson
cbc5fdf99a
Fix CA Timeout and show progress to the user (#915) 
* Fix idle after CA timeout and show progress to the user so they don't think we've hung

* fix timeout again since it was lost in a rebase
 2019-01-02 09:41:01 -05:00
JT Olio
3fbd8c7d15 kademlia: remove a redundant argument (#951) 2019-01-02 12:57:06 +02:00
JT Olio
d4d0b0d13b inspector: use existing identity (#952) 2019-01-02 12:45:47 +02:00
JT Olio
4a0749e44a Remove farmer references (#949) 2019-01-02 12:31:49 +02:00
JT Olio
2c916a04c3 pkg/provider: split into pkg/server, pkg/identity (#953) 2019-01-02 12:23:25 +02:00
Egon Elbre
cc8cce58dd
statreceiver nitpicking (#926) 2019-01-01 11:41:27 +02:00
Bryan White
4efb5c0a75
Authorization improvements (#925) 2018-12-31 10:45:43 -05:00
Yehor Butko
e82edc68c8
V3-976 Create db query for filtering team mebers (#940) 
* V3-976 Create db query for filtering team mebers

* fixing linter

* fixing linter

* sql injection fixed

* getOrder renamed, tests added
 2018-12-28 14:07:35 +02:00
Yaroslav Vorobiov
d65cefcac7
Satellite api keys api (#936) 2018-12-27 17:30:15 +02:00
Egon Elbre
4346cd060f
Implement mutex around satellitedb (#932) 2018-12-27 11:56:25 +02:00
Michal Niewrzal
a9572b7169
Fix ignored config logger settings (#938) 2018-12-27 10:38:11 +01:00
Nikolay Yurchenko
312fd5a5fd
[V3-963] Satellite team members add dialog (#935) 
* Implemented team member add dialog without api.

* logic changed according to api

* added initial app reducer and team members deletion fixed
 2018-12-26 17:05:33 +02:00
Yaroslav Vorobiov
3b9f5fa633
satellite api keys repo and service method added (#931) 2018-12-26 16:00:53 +02:00
Yehor Butko
5ef427265e
web/satellite notifications added, users and projects api updated (#934) 
* web/satellite notifications added, users and projects api updated

* fix users api
 2018-12-24 14:52:52 +02:00
Dylan Lott
7569b7d71b
Node lifecycle hooks (#920) 
* wires up first draft of lifecycle methods

* creates interface on transport

* node lifecycle hooks works

* linter fixes

* adds error log at connection success

* chnages Observer interface to use context

* Makes Discovery take its own logger

* WIP

* linter fixes

* Test fixes

* adds in ConnFailure code for cache
 2018-12-21 21:51:42 -07:00
Dylan Lott
2f6835a3be
Adds a logger to RoutingTable (#922) 
* adds delete functions and tests for cache

* adds logger to routing table struct and passes it through to logger

* add loggers
 2018-12-21 17:48:53 -07:00
Dylan Lott
05c1644fec
adds delete functions and tests for cache (#921) 2018-12-21 11:35:21 -07:00
Yaroslav Vorobiov
16db288c85
Satellite add/delete project members by email api (#928) 
* Satellite add/delete project members by email api
 2018-12-21 17:41:53 +02:00
Cameron
f70b826fd4
repair queue masterDB support (#865) 
* add injuredsegment model to satellitedb.dbx

* add context to queue.RepairQueue interface

* use queue.RepairQueue interface, use masterdb
 2018-12-21 10:11:19 -05:00
Bill Thorp
3fa094f3da
Add the ability for Kademlia to do a refresh for each bucket (#908) 
* added random id in range, added refresh
 2018-12-20 16:45:06 -05:00
Dennis Coyle
18c4972205
add monkit to Satellite DB Service (#903) 
* add monkit to Satelite DB Service

* goimports
 2018-12-20 15:10:27 -05:00
Bryan White
398379b149
CSR service (part 1): authorizations (#906) 2018-12-20 19:29:05 +01:00
Yehor Butko
ac860c1984
Removing company and project ownership (#917) 2018-12-20 18:18:08 +02:00
Yaroslav Vorobiov
1c119c6215
(#916) delete and add projectmember api now accepts list of users 
* delete and add projectmember api now accepts list of users

* review changes

* simplification

* use errs.Combine
 2018-12-20 17:36:32 +02:00
Egon Elbre
d9b9ae6ffa
Cleanup overlay methods and names. (#914) 2018-12-20 15:57:54 +02:00
Maximillian von Briesen
f18c38628e
Refactor statdb (#887) 
Update statdb args/return values to minimize structs
    Simplify statdb.Update() to update all stats instead of an arbitrary subset determined by flags
    Remove CreateIfNotExists logic from statdb.Update()
    Simplify audit code structure
 2018-12-19 13:44:03 -05:00
Yaroslav Vorobiov
76018231c6
satellite project members pagination api (#907) 
* satellite project members pagination api

* rename GetByProjectIDPaged to GetByProjectID

* add comment to max limit
 2018-12-19 15:03:12 +02:00
Dennis Coyle
1b7ab9600d
Do not error if Discovery can not find Node (#902) 2018-12-18 15:01:15 -05:00
aligeti
3c7f62be73
KBucket clean up (#898) 2018-12-18 13:04:46 -05:00
Yaroslav Vorobiov
efdfff9704
updated project and user deletion (#884) 
* update project and user deletion

* linter fixes
 2018-12-18 19:43:02 +02:00
Jennifer Li Johnson
01df85704c
Updates accounting tables and allows tally to update at raw table for at rest data (#874) 2018-12-18 12:18:42 -05:00
Egon Elbre
d9a13667fc
Use fixed logger for Kademlia and make not being able to connect in discovery debug statement. (#899) 2018-12-18 17:13:32 +02:00
Bryan White
4eb55017c8
Cert revocation CLI (#848) 
* wip

* allow identity and CA configs to save cert/key separately

* fixes

* linter and default path fixes

* review fixes

* fixes:

+ review fixes
+ bug fixes
+ add extensions command

* linter fixes

* fix ca revoke description

* review fixes
 2018-12-18 12:55:55 +01:00
Natalie Villasana
17c60c1f06
moves node selection config setup from uplink to satellite (#891) 2018-12-17 16:05:05 -05:00
Michal Niewrzal
a819d819e3
Overlay Cache master db merge (#877) 
* Overlay Cache master db merge

* test update

* review comments

* Fix transaction usage

* removed unused variable

* added better limit handling

* better error handling
 2018-12-17 21:14:16 +01:00
Egon Elbre
6b5945139f
Overlay error handling (#880) 2018-12-17 20:47:26 +02:00
Egon Elbre
5c26aa3e4a
Use storj.NodeID directly in statdb tests (#881) 2018-12-17 20:47:04 +02:00
Natalie Villasana
d97e27ed80
removes a few todos (#867) 2018-12-17 13:11:29 -05:00
Michal Niewrzal
1d2523cf60
Payments API protobuf definition (#876) 2018-12-17 18:36:50 +01:00
Dylan Lott
706f20611d
adds discovery function to overlay for random lookups (#860) 
* adds discovery function to overlay for random lookups

* uses different variable name

* linter fixes
 2018-12-17 09:31:14 -07:00
Cameron
0d17c21a1b
Add logger to piecestore server/node started log (#856) 
* add logger to psserver struct

* node started log

* rename initialize to NewEndpoint

* return err from WriteBandwidthAllocToDB
 2018-12-17 10:23:02 -05:00
Egon Elbre
d8df4b5f6b
Simplify provider tests (#862) 2018-12-17 17:09:52 +02:00
Yehor Butko
90f79e585c
V3-975 Create db pagination query for team members (#871) 
* V3-975 Create db pagination query for team members

* fix go mod tidy

* Methods renamed. Picked go.sum from storj/master.

* fixing linter
 2018-12-17 16:28:58 +02:00
Michal Niewrzal
f0662675bc
Add '--dir' param for all CLI parts (replace --base-path) (#833) 
* Add '--dir' param for all CLI parts (replace --base-path)

* FindDirParam method moved

* fix compilation error

* make param global

* remove unused fields

* rename param

* remove config flag

* goimports
 2018-12-14 22:14:59 +01:00
Bryan White
d8db7c3049
RevocationDB fixes (#866) 2018-12-14 21:45:53 +01:00
aligeti
5e1b02ca8b
Statdb master db v3 848 (#830) 
* intial changes to migrate statdb to masterdb framework

* statdb refactor compiles

* added TestCreateDoesNotExist testcase

* Initial port of statdb to masterdb framework working

* refactored statdb proto def to pkg/statdb

* removed statdb/proto folder

* moved pb.Node to storj.NodeID

* CreateEntryIfNotExistsRequest moved pd.Node to storj.NodeID

* moved the fields from pb.Node to statdb.UpdateRequest

ported TestUpdateExists, TestUpdateUptimeExists, TestUpdateAuditSuccessExists TestUpdateBatchExists
 2018-12-14 15:17:30 -05:00
Dylan Lott
9bdee7b106
Discovery service (#805) 
* WIP possible discovery service impl

* Adds discovery service to CaptPlanet

* Updates the config and server for discovery service

* updates testplanet to use discovery package

* update satellite imports

* Removes unnecessary cache test

* linter fixes

* adds discovery startup to captplanet

* invoke refresh

* updates to discovery refresh cycle

* Make implementation more consistent with previous implementation

* add wait before trying to upload

* sleep a bit more

* remove kademlia bootstrap

* updates

* remove comments
 2018-12-14 12:23:54 -07:00
Yaroslav Vorobiov
60fb655db2
satellite user delete updated to require user password (#852) 2018-12-14 18:14:17 +02:00
Michal Niewrzal
725ed44ce0
Add gen.go for protobuf (#863) 2018-12-14 16:55:55 +01:00
Michal Niewrzal
db5a3e0d0a
Parameters cleanup (#864) 
* Parameters cleanup

* goimports
 2018-12-14 16:55:45 +01:00
Bill Thorp
f9845e7e92
changes to masterdb to support accountingDB (#846) 
* added rollup to captplanet, moved accountingDB to masterdb
 2018-12-14 09:27:21 -05:00
Egon Elbre
7363e6cfd8
Better transport tests (#849) 2018-12-14 12:44:48 +02:00
Bill Thorp
c17a6ba584
fixed temp dir issues w/ rev db (#859) 2018-12-13 21:24:35 -05:00
Bryan White
2016ce9fd6
Certificate revocation (#836) 
* wip certificate revocation

* refactor tests

* wip testing

* testing

* review fixes

* integration fix attempt #1

* review fixes

* integration fix attempt #2

* linter fixes

* add copywrite

* integration fix attemp #3

* more testing

* more tests

* go mod tidy

* review fixes

* linter fixes
 2018-12-13 21:01:43 +01:00
Egon Elbre
08394389d3 Fix ConfNested on Windows (#851) 2018-12-13 07:34:38 -07:00
Egon Elbre
969fcabf72
Remove utils.LogClose from identity saving (#837) 2018-12-13 15:50:05 +02:00
Kaloyan Raev
2a8fef8062
Data repairer should use the redundancy strategy from segment's pointer (#838) 2018-12-13 09:12:36 +02:00
Jennifer Li Johnson
6642f97142
Jj/accounting updates (#820) 
* tallies up data stored on each node in pointerdb

* adds comments for data type enums

* changes Open to BeginTx because Go convention

* removes online status check from identify active nodes

* changes identifyactivenodes to calculatestaticdata

* updates accounting dbx names
 2018-12-12 16:24:08 -05:00
Maximillian von Briesen
a6c7306350
Cut off piecestore Puts if they exceed alloced bandwidth/space (#819) 
* add bandwidth/storage limits to StreamWriter

* add StreamWriter tests for bandwidth/storage limits
 2018-12-12 14:14:51 -05:00
Egon Elbre
7a80e7bf2f
Use testplanet in node tests (#841) 2018-12-12 17:40:33 +02:00
Egon Elbre
c56307f5b9
datarepair: remove unneeded debug message (#839) 
* datarepair: remove unneeded debug message
* Make overlay-cache and routing table logs more specific
 2018-12-12 17:39:16 +02:00
Egon Elbre
38f72df8b5
Faster tests (#834) 2018-12-12 16:05:47 +02:00
Bill Thorp
ba5f71810e
added tally to capt run.go (#751) 
* added tally to capt run.go, ParseURL -> SplitDBURL
 2018-12-12 08:15:34 -05:00
JT Olio
4a4f6ad53e
cfgstruct.Bind: support nested config paths (#785) 
this will allow some config cleanups in a future pr

Change-Id: Ie873bcee567a72956d9337dfc13ab6ba46c9d1a0
 2018-12-11 11:41:19 -07:00
JT Olio
ceb590fa67
capt: reduce nodes to 10 (#793) 
Change-Id: Ief380fe29e3043657705cd7505c266fd774181a4
 2018-12-11 11:40:54 -07:00
JT Olio
362f447d9f
cmd/statreceiver: lua-scriptable stat receiver (#636) 
* cmd/statreceiver: lua-scriptable stat receiver

Change-Id: I3ce0fe3f1ef4b1f4f27eed90bac0e91cfecf22d7

* some updates

Change-Id: I7c3485adcda1278fce01ae077b4761b3ddb9fb7a

* more comments

Change-Id: I0bb22993cd934c3d40fc1da80d07e49e686b80dd

* linter fixes

Change-Id: Ied014304ecb9aadcf00a6b66ad28f856a428d150

* catch errors

Change-Id: I6e1920f1fd941e66199b30bc427285c19769fc70

* review feedback

Change-Id: I9d4051851eab18970c5f5ddcf4ff265508e541d3

* errorgroup improvements

Change-Id: I4699dda3022f0485fbb50c9dafe692d3921734ff

* too tricky

the previous thing was better for memory with lots of errors at a time
but https://play.golang.org/p/RweTMRjoSCt is too much of a foot gun

Change-Id: I23f0b3d77dd4288fcc20b3756a7110359576bf44
 2018-12-11 11:24:31 -07:00
Natalie Villasana
9e1ec97b31
adds node selection config (#782) 2018-12-11 12:30:14 -05:00
Kaloyan Raev
9449010c15
Enable overlay by default (#835) 2018-12-11 18:52:45 +02:00
Kaloyan Raev
252da15f0d
Randomize node selection during GETs (#827) 2018-12-11 18:05:14 +02:00
Yaroslav Vorobiov
175e25f93a
Satellite graphql: take authorized user id if no id was provided in iput (#828) 
satellite graphql: take authorized user id if no id was provided by client
 2018-12-11 17:54:45 +02:00
Egon Elbre
492639aef7
Skip TestUploadDownload (#826) 2018-12-11 16:52:03 +02:00
Egon Elbre
8e14d2cb94
Reduce ranger test output (#825) 2018-12-11 11:38:56 +02:00
Michal Niewrzal
7e2f66ee24
Small fixes around master db (#822) 2018-12-11 10:30:09 +01:00
Egon Elbre
26dbf788ad
Optimize compilation by removing logging tests (#816) 2018-12-11 10:46:52 +02:00
aligeti
d27bcddaab
Merge the irreparable DB into the master database (#777) 
* initial refactoring to merge irreparabledb to master db

* refactor

* code review comment updates
 2018-12-10 14:08:45 -05:00
Yehor Butko
a8a082f4c4
V3-808 Implement graphql queries for team members page (#815) 2018-12-10 19:32:15 +02:00
Yaroslav Vorobiov
63998b4a62
updated user update api, separated company and user creation (#767) 2018-12-10 17:57:06 +02:00
Kaloyan Raev
3175eb74a6
Improve piece retrieving log with offset and length (#813) 2018-12-10 17:22:49 +02:00
Michal Niewrzal
a15099d5f5
Simplify testing with all supported DBs (#809) 
* Simplify testing with all supported DBs

* move method to separate package

* method renamed to Run

* fixed comment
 2018-12-10 15:50:12 +01:00
Kaloyan Raev
3cd0b4adb2
Verify bucket exists before starting multipart upload (#812) 2018-12-10 16:41:43 +02:00
Kaloyan Raev
4cc49a8fa3
Better error handling of stream.Upload.Close (#811) 2018-12-10 16:35:22 +02:00
Yehor Butko
b1605a7331
Updating users repository (#808) 2018-12-10 15:47:48 +02:00
Yehor Butko
678eb81d54
V3-898 Create ability for querying db in transaction scope (#781) 
* V3-898 Create ability for querying db in transaction scope

* handling errors

* fixing linter

* Transaction updated

* fixing empty lines linter

* tx signatures changed

* fixing comments
 2018-12-10 14:29:01 +02:00
Yaroslav Vorobiov
8ff6cea5a9
[WIP] get project members api (#801) 2018-12-10 13:38:42 +02:00
Egon Elbre
c7cf97cc23
Fix sqlite registration (#806) 2018-12-10 10:32:49 +02:00
Cameron
f46fcd0303
upload/download tests with test-planet (#718) 
* add upload/download test

* use minio gateway for upload download

* move test into miniogw package

* logging edits, remove logging_test.go
 2018-12-07 14:26:39 -05:00
Kaloyan Raev
c025c13706
Fix deadlock on testplanet shutdown (#798) 2018-12-07 20:31:59 +02:00
Kaloyan Raev
72a20af569
Migrate Minio Gateway to the new Metainfo API (#775) 2018-12-07 20:31:29 +02:00
Egon Elbre
554daf17e6
Remove mock kademlia (#802) 2018-12-07 19:36:20 +02:00
JT Olio
890c87914f
pstore: fix blob permissions (#788) 
Change-Id: I004747657221c6ac53894fc5c5917e3f92436990
 2018-12-07 07:47:45 -07:00
JT Olio
1c96db01ba
better database error handling (#784) 
* better database error handling

Change-Id: I28dbd69cf6c2fa268e02405521ff6e6c1a68a702

* missing comments added

* missing comment added
 2018-12-07 07:46:42 -07:00
Brandon Iglesias
023eb95a9b
cleaning up todos in Kademlia pkg (#763) 
* updating comments in KAD code

* deleting todo comment because they are not needed anymore
 2018-12-07 08:57:42 -05:00
Bryan White
228aa34ff6
TLS extension processing (#771) 2018-12-07 14:44:25 +01:00
Michal Niewrzal
4608976526
Bwagreement tests cleanup (#795) 2018-12-07 13:11:35 +01:00
Michal Niewrzal
d62896bbaf
Use testcontext with tally tests (#796) 2018-12-07 12:55:25 +01:00
Michal Niewrzal
034a34069a
Merge bwagreement db into satellite master db (#770) 
* Merge bwagreement db into satellite master db

* adjust to recent tally changes

* linter problems

* linter problems

* returning db structs in more optimal way

* added pointer for assignment

* error message changed

* better param message
 2018-12-07 10:59:31 +01:00
Jennifer Li Johnson
dac79fd762
removes check for mock overlay in data repair checker config (#794) 2018-12-07 00:42:16 -05:00
JT Olio
b6c936340b
clean up uplink config section (#786) 2018-12-06 17:18:11 -07:00
Alexander Leitner
04ff6b8e11
Use audit results in checker (#754) 
* Use statdb to get audit results

* Find invalid nodes
 2018-12-06 13:51:23 -05:00
Kaloyan Raev
b043fd76dd
Fix race in connection pool (#778) 2018-12-06 18:05:57 +02:00
Yehor Butko
0c95fba4c8
Satellite Projects db entity and api updated (#772) 
* Satellite Projects db entity and api updated

* get users projects query updated with dbx
 2018-12-06 17:19:47 +02:00
Yaroslav Vorobiov
d6f6ab56a1
satellite projectmembers add and delete api (#776) 2018-12-06 16:40:32 +02:00
Egon Elbre
fc8e9c6572
Move irreparabledb under datarepair (#764) 2018-12-06 14:32:21 +02:00
Maximillian von Briesen
828a8b6907
Change statdb.FindValidNodes to FindInvalidNodes (#774) 
* Remove statdb.FindValidNodes
* Add statdb.FindInvalidNodes for getting nodes that fall below a reputation threshold
 2018-12-05 15:23:32 -05:00
Dennis Coyle
fffb093056
SelfAddr -> ExternalAddress (#768) 2018-12-05 11:36:03 -05:00
Egon Elbre
c4033b15af
Simplify peer discovery implementation (#765) 2018-12-05 16:32:37 +02:00
Bill Thorp
b2a8a10b16
redo tally query (#742) 
* redo tally query

* broken, sharing w/ Jen

* tally tests added, failiing

* fix txn issue in tally

* fix lint
 2018-12-05 09:03:23 -05:00
Dennis Coyle
e138056943
listen to configured address (#755) 2018-12-05 05:37:03 -05:00
Dylan Lott
69281552b3
Pretty prints node output from kad lookup tool (#762) 
* Pretty prints node output from kad lookup tool

* Linter fix
 2018-12-04 17:23:39 -07:00
Dylan Lott
c6d790d58e
Improvements to Kad Lookups (#731) 
* adds channel for getting node out of lookup

* WIP adding the channels to lookups

* WIP adding channel to node lookups

* Wires up FindNodes method with channels

* WIP adds a test suite for lookup - tests are still failing

* WIP wires up use of testplanet for kademlia lookup tests

* WIP merging in node id changes

* Merges in pkg/storj node type changes

* Tests passing

* Lookup node working via Inspector now

* updates

* WIP working on getting tests passing

* WIP getting tests passing

* FindNode works

* Linter fix

* Adds copyrights to lookup_test

* removes a fmt.Printf I missed

* Removes commented out lines
 2018-12-04 14:39:28 -07:00
Matt Robinson
4237afb1c1
Attempt at fixing the all-in-one environment (#736) 2018-12-04 16:10:23 -05:00
Maximillian von Briesen
6c655d117b
Filter nodes by reputation and IP address (node selection) (#720) 
*   Pulls statdb stats into overlay cache whenever cache.Put() is called
* Updates overlay.FindStorageNodes()/overlayClient.Choose() to filter based on node stats
* Updates overlay.FindStorageNodes()/overlayClient.Choose() to exclude duplicate IP addresses
 2018-12-04 15:18:26 -05:00
Maximillian von Briesen
2ab15196d9
Get rid of statdb grpc stuff (#761) 
* remove sdbclient

* do not register statdb grpc server/remove apikey from statdb

* start audit service after statdb in captplanet
 2018-12-04 13:47:58 -05:00
Kaloyan Raev
b7c45c7c16
Migrate Uplink CLI to Metainfo API (#745) 2018-12-04 18:46:02 +02:00
aligeti
9f6e5d6731
Keep track of 'irreparable' segments in a database (#686) 
* initial irreparable development

* added the Open, Insert, Update, Get, Delete, Close  support

* added the Unit test cases

* code review changes
 2018-12-04 11:26:30 -05:00
Egon Elbre
ae790dfd9f
Fix queue implementation (#758) 2018-12-04 17:46:53 +02:00
Egon Elbre
efe475b448
Ensure that restrictions are consistent. (#759) 2018-12-04 16:02:03 +02:00
Bryan White
24f6da2dd5
Kad inspector fixes (#746) 
* pretty print kad inspector

* fix format string

* cleanup

* cleanup

* usage, args, print improvements

* kad inspector tweaks

* pretty print kad inspector

* cleanup

* cleanup

* kad inspector tweaks

* linter fixes
 2018-12-03 21:01:54 +01:00
Maximillian von Briesen
c6dd70541e
move sdb apikey to config struct (#753) 2018-12-03 13:40:03 -05:00
Kaloyan Raev
0018ebf63e
Object.Bucket as storj.Bucket instead of string (#747) 2018-12-03 16:38:03 +02:00
Bryan White
120d875e06
consolidate NodeRep and NodeStats (#744) 
* consolidate `NodeRep` and `NodeStats`

* regenerate

* review fixes

* go travis
 2018-12-03 14:23:12 +01:00
Bryan White
fe8c1823b7 Cleanup debug code 2018-12-03 12:45:27 +02:00
Bryan White
71ef93feed
pretty print kad inspector (#743) 
* pretty print kad inspector

* fix format string

* cleanup

* cleanup

* usage, args, print improvements
 2018-11-30 18:36:05 +01:00
Egon Elbre
3e461df470
Linting protobuf files (#676) 2018-11-30 17:02:01 +02:00
Kaloyan Raev
66476da1ae
metainfo: basic upload (#726) 2018-11-30 15:50:52 +02:00
Egon Elbre
1e4556f88a
Fix import groupings (#739) 2018-11-30 15:40:13 +02:00
Bryan White
66b3a20595
cleanup and bugfix (#737) 2018-11-30 14:04:59 +01:00
Egon Elbre
9ac3517432
Reduce noise in logging (#733) 2018-11-29 22:59:26 +02:00
Bryan White
2a0c4e60d2
preparing for use of customtype gogo extension with NodeID type (#693) 
* preparing for use of `customtype` gogo extension with `NodeID` type

* review changes

* preparing for use of `customtype` gogo extension with `NodeID` type

* review changes

* wip

* tests passing

* wip fixing tests

* more wip test fixing

* remove NodeIDList from proto files

* linter fixes

* linter fixes

* linter/review fixes

* more freaking linter fixes

* omg just kill me - linterrrrrrrr

* travis linter, i will muder you and your family in your sleep

* goimports everything - burn in hell travis

* goimports update

* go mod tidy
 2018-11-29 19:39:27 +01:00
Yaroslav Vorobiov
a21855ac37
satellite user validation (#729) 2018-11-29 18:23:44 +02:00
Egon Elbre
0ae05cf834
Ensure that we only choose storage nodes (#732) 2018-11-29 16:57:00 +02:00
Yehor Butko
7b628e8af6
V3-763 Create logic for project details (#728) 
* V3-763 Create logic for project details

* updating markup

* icon fixed
 2018-11-28 18:20:23 +02:00
Yaroslav Vorobiov
c27431d64f
company update api (#725) 2018-11-28 14:30:38 +02:00
Yaroslav Vorobiov
24159216bc
update user api (#715) 2018-11-28 12:31:15 +02:00
Kaloyan Raev
f46d43b0d0
Fix panic in Audit service (#713) 2018-11-28 09:33:17 +02:00
Dylan Lott
a245aa7827
Inspector - Kad PING (#722) 
* initial commit of inspector gadget wireup

* change name of comman dline tool, setup grpc server

* Get inspector cli working with grpc client

* Wired up CountNodes command

* WIP getting buckets response working

* Added GetBucket command

* WIP working on get buckets command

* WIP working on bucket list

* Still WIP

* WIP getting bucket counts to work

* Some clean up of unnecessary changes

* List Buckets and Get Bucket are working

* Removing logs, getting ready for review

* initial commit of inspector gadget wireup

* change name of comman dline tool, setup grpc server

* Get inspector cli working with grpc client

* Wired up CountNodes command

* WIP getting buckets response working

* Added GetBucket command

* WIP working on get buckets command

* WIP working on bucket list

* Still WIP

* WIP getting bucket counts to work

* Some clean up of unnecessary changes

* List Buckets and Get Bucket are working

* Removing logs, getting ready for review

* Fix error return

* Trying to get tests passing

* Adds method on dht mock for tests

* Add dbx files back

* Fix package import error in dbx file

* Adds copyrights to pass linter

* tidy go mod

* Updates from code review

* Updates inspector to take flag arguments for address

* Format list-buckets output more prettier

* Wiring up PING in kad inspector tools
 2018-11-27 15:50:52 -07:00
Dylan Lott
fd42795456
updates dbx/gen.go to name generated package correctly (#723) 2018-11-27 13:52:19 -07:00
Maximillian von Briesen
7cf16503b8
Add statdb to overlay cache (node selection) (#711) 
* make overlay node rep consistent with statdb node rep

* add statdb to testplanet

* add statdb to overlay cache

* tests
 2018-11-27 12:46:12 -05:00
Michal Niewrzal
376bd74bed
Stops logging repairer errors for empty queue (#717) 2018-11-27 16:57:51 +01:00
Yaroslav Vorobiov
a0d84ccd67
Satellite user delete (#707) 
* auth from context

* user delete api
 2018-11-27 16:20:58 +02:00
Yehor Butko
784f3a887d
V3-760 Create logic for New project popup (#714) 
* V3-760 Create logic for New project popup

* removing path to static files from gateway

* added missing copyrights
 2018-11-27 13:14:10 +00:00
Kaloyan Raev
ea9bf38d85
eestream avoids waiting for pending readers if lots of errors (#709) 2018-11-27 10:17:57 +02:00
Bill Thorp
b8b0d75cc6
made accounting use migrate (#688) 
* made accounting use migrate
 2018-11-26 16:49:55 -05:00
Michal Niewrzal
8dd669b37f
Bandwidth allocations need method designation (GET/PUT) (#708) 
* Bandwidth allocations need method designation (GET/PUT)

* add method comment

* goimports
 2018-11-26 19:21:44 +01:00
Maximillian von Briesen
5014a785a0
Node selection 0/4: Update statdb auth/functionality (#698) 
* remove api key from statdb server reqs; add statdb UpdateUptime and UpdateAuditSuccess to server

* update api key authentication in statdb server

* add todos for future statdb updates

* add UpdateUptime and UpdateAuditSuccess to statdb server

* fix apikey stuff in config.go and statdb_test.go

* fix tests

* update sdbclient.NewClient call in audit package

* fix UpdateUptime and UpdateAuditSuccess in sdbclient

* set api key from statdb/config.go

* change package for statdb tests

* linter fixes

* remove todo comments

* fix sdbclient err checking

* move validate auth functionality to auth package

* update description for statdb api key

* remove import
 2018-11-26 12:08:29 -05:00
Yehor Butko
1f1f9f5875
Satellitedb company-user relation changed to 1-to-1 (#706) 
* Satellitedb company-user relation changed to 1-to-1

* fixing comments
 2018-11-26 14:57:55 +00:00
Yehor Butko
a998665491
Satellite projects api (#703) 
* V3-645 Create Project API

* fixing linter

* update project refactored
 2018-11-26 10:47:23 +00:00
Kaloyan Raev
67db26b36e
Metainfo stream download (#697) 2018-11-26 09:39:05 +02:00
Maximillian von Briesen
b884ee4626
Update protos to fit style guide (easy review) (#702) 
* fix overlay.proto and piecestore.proto to meet style guide

* update code to be consistent with regenerated protos
 2018-11-23 21:46:53 -05:00
Yaroslav Vorobiov
bdebe36ae0
updated login api (#699) 2018-11-22 12:38:58 +02:00
Maximillian von Briesen
06cf1c71ed
Statdb inspection tool (#626) 
* begin adding stadb inspection tool

* add command for creating statdb nodes from csv

* add getCSVStats and create helper func for getting sdb client

* fix createcsvstats, add createstats

* fix linter errors
 2018-11-21 17:01:09 -05:00
Dylan Lott
0d05cb26bf
Kademlia Inspector CLI (#657) 
* initial commit of inspector gadget wireup

* change name of comman dline tool, setup grpc server

* Get inspector cli working with grpc client

* Wired up CountNodes command

* WIP getting buckets response working

* Added GetBucket command

* WIP working on get buckets command

* WIP working on bucket list

* Still WIP

* WIP getting bucket counts to work

* Some clean up of unnecessary changes

* List Buckets and Get Bucket are working

* Removing logs, getting ready for review

* initial commit of inspector gadget wireup

* change name of comman dline tool, setup grpc server

* Get inspector cli working with grpc client

* Wired up CountNodes command

* WIP getting buckets response working

* Added GetBucket command

* WIP working on get buckets command

* WIP working on bucket list

* Still WIP

* WIP getting bucket counts to work

* Some clean up of unnecessary changes

* List Buckets and Get Bucket are working

* Removing logs, getting ready for review

* Fix error return

* Trying to get tests passing

* Adds method on dht mock for tests

* Add dbx files back

* Fix package import error in dbx file

* Adds copyrights to pass linter

* tidy go mod

* Updates from code review

* Updates inspector to take flag arguments for address

* Format list-buckets output more prettier
 2018-11-21 10:31:27 -07:00
Yaroslav Vorobiov
c829835dc1
updated createuser api (#687) 2018-11-21 17:51:43 +02:00
Michal Niewrzal
f72832ee69
Add metadata to pb.Node (email and wallet) (#671) 
* Add metadata to pb.Node (email and wallet)

* goimports

* reorganize config

* fix typo

* remove 'foo' id

* add basic test
 2018-11-21 16:07:18 +01:00
Kaloyan Raev
aec2c41911
Metainfo remote segment test (#696) 2018-11-21 16:35:53 +02:00
Kaloyan Raev
8c56198090
Use designated error classes for common bucket and object errors (#691) 2018-11-21 13:17:28 +02:00
Bryan White
dabd87cb36
swap golang/protobuf out for gogo/protobuf (#692) 
* swap golang/protobuf out for gogo/protobuf

* revert go.mod and go.sum changes

* linter fixes
 2018-11-20 19:29:07 +01:00
Bryan White
54ccb460a9
rename size fields in protobuf messages for gogo compatibility (#690) 
* rename `size` fields in protobuf messages for gogo compatibility

* linter fixes
 2018-11-20 18:09:35 +01:00