Commit Graph

508 Commits

Author SHA1 Message Date
Kaloyan Raev
3175eb74a6
Improve piece retrieving log with offset and length (#813) 2018-12-10 17:22:49 +02:00
Michal Niewrzal
a15099d5f5
Simplify testing with all supported DBs (#809)
* Simplify testing with all supported DBs

* move method to separate package

* method renamed to Run

* fixed comment
2018-12-10 15:50:12 +01:00
Kaloyan Raev
3cd0b4adb2
Verify bucket exists before starting multipart upload (#812) 2018-12-10 16:41:43 +02:00
Kaloyan Raev
4cc49a8fa3
Better error handling of stream.Upload.Close (#811) 2018-12-10 16:35:22 +02:00
Yehor Butko
b1605a7331
Updating users repository (#808) 2018-12-10 15:47:48 +02:00
Yehor Butko
678eb81d54
V3-898 Create ability for querying db in transaction scope (#781)
* V3-898 Create ability for querying db in transaction scope

* handling errors

* fixing linter

* Transaction updated

* fixing empty lines linter

* tx signatures changed

* fixing comments
2018-12-10 14:29:01 +02:00
Yaroslav Vorobiov
8ff6cea5a9
[WIP] get project members api (#801) 2018-12-10 13:38:42 +02:00
Egon Elbre
c7cf97cc23
Fix sqlite registration (#806) 2018-12-10 10:32:49 +02:00
Cameron
f46fcd0303
upload/download tests with test-planet (#718)
* add upload/download test

* use minio gateway for upload download

* move test into miniogw package

* logging edits, remove logging_test.go
2018-12-07 14:26:39 -05:00
Kaloyan Raev
c025c13706
Fix deadlock on testplanet shutdown (#798) 2018-12-07 20:31:59 +02:00
Kaloyan Raev
72a20af569
Migrate Minio Gateway to the new Metainfo API (#775) 2018-12-07 20:31:29 +02:00
Egon Elbre
554daf17e6
Remove mock kademlia (#802) 2018-12-07 19:36:20 +02:00
JT Olio
890c87914f
pstore: fix blob permissions (#788)
Change-Id: I004747657221c6ac53894fc5c5917e3f92436990
2018-12-07 07:47:45 -07:00
JT Olio
1c96db01ba
better database error handling (#784)
* better database error handling

Change-Id: I28dbd69cf6c2fa268e02405521ff6e6c1a68a702

* missing comments added

* missing comment added
2018-12-07 07:46:42 -07:00
Brandon Iglesias
023eb95a9b
cleaning up todos in Kademlia pkg (#763)
* updating comments in KAD code

* deleting todo comment because they are not needed anymore
2018-12-07 08:57:42 -05:00
Bryan White
228aa34ff6
TLS extension processing (#771) 2018-12-07 14:44:25 +01:00
Michal Niewrzal
4608976526
Bwagreement tests cleanup (#795) 2018-12-07 13:11:35 +01:00
Michal Niewrzal
d62896bbaf
Use testcontext with tally tests (#796) 2018-12-07 12:55:25 +01:00
Michal Niewrzal
034a34069a
Merge bwagreement db into satellite master db (#770)
* Merge bwagreement db into satellite master db

* adjust to recent tally changes

* linter problems

* linter problems

* returning db structs in more optimal way

* added pointer for assignment

* error message changed

* better param message
2018-12-07 10:59:31 +01:00
Jennifer Li Johnson
dac79fd762
removes check for mock overlay in data repair checker config (#794) 2018-12-07 00:42:16 -05:00
JT Olio
b6c936340b
clean up uplink config section (#786) 2018-12-06 17:18:11 -07:00
Alexander Leitner
04ff6b8e11
Use audit results in checker (#754)
* Use statdb to get audit results

* Find invalid nodes
2018-12-06 13:51:23 -05:00
Kaloyan Raev
b043fd76dd
Fix race in connection pool (#778) 2018-12-06 18:05:57 +02:00
Yehor Butko
0c95fba4c8
Satellite Projects db entity and api updated (#772)
* Satellite Projects db entity and api updated

* get users projects query updated with dbx
2018-12-06 17:19:47 +02:00
Yaroslav Vorobiov
d6f6ab56a1
satellite projectmembers add and delete api (#776) 2018-12-06 16:40:32 +02:00
Egon Elbre
fc8e9c6572
Move irreparabledb under datarepair (#764) 2018-12-06 14:32:21 +02:00
Maximillian von Briesen
828a8b6907
Change statdb.FindValidNodes to FindInvalidNodes (#774)
* Remove statdb.FindValidNodes
* Add statdb.FindInvalidNodes for getting nodes that fall below a reputation threshold
2018-12-05 15:23:32 -05:00
Dennis Coyle
fffb093056
SelfAddr -> ExternalAddress (#768) 2018-12-05 11:36:03 -05:00
Egon Elbre
c4033b15af
Simplify peer discovery implementation (#765) 2018-12-05 16:32:37 +02:00
Bill Thorp
b2a8a10b16
redo tally query (#742)
* redo tally query

* broken, sharing w/ Jen

* tally tests added, failiing

* fix txn issue in tally

* fix lint
2018-12-05 09:03:23 -05:00
Dennis Coyle
e138056943
listen to configured address (#755) 2018-12-05 05:37:03 -05:00
Dylan Lott
69281552b3
Pretty prints node output from kad lookup tool (#762)
* Pretty prints node output from kad lookup tool

* Linter fix
2018-12-04 17:23:39 -07:00
Dylan Lott
c6d790d58e
Improvements to Kad Lookups (#731)
* adds channel for getting node out of lookup

* WIP adding the channels to lookups

* WIP adding channel to node lookups

* Wires up FindNodes method with channels

* WIP adds a test suite for lookup - tests are still failing

* WIP wires up use of testplanet for kademlia lookup tests

* WIP merging in node id changes

* Merges in pkg/storj node type changes

* Tests passing

* Lookup node working via Inspector now

* updates

* WIP working on getting tests passing

* WIP getting tests passing

* FindNode works

* Linter fix

* Adds copyrights to lookup_test

* removes a fmt.Printf I missed

* Removes commented out lines
2018-12-04 14:39:28 -07:00
Matt Robinson
4237afb1c1
Attempt at fixing the all-in-one environment (#736) 2018-12-04 16:10:23 -05:00
Maximillian von Briesen
6c655d117b
Filter nodes by reputation and IP address (node selection) (#720)
*   Pulls statdb stats into overlay cache whenever cache.Put() is called
* Updates overlay.FindStorageNodes()/overlayClient.Choose() to filter based on node stats
* Updates overlay.FindStorageNodes()/overlayClient.Choose() to exclude duplicate IP addresses
2018-12-04 15:18:26 -05:00
Maximillian von Briesen
2ab15196d9
Get rid of statdb grpc stuff (#761)
* remove sdbclient

* do not register statdb grpc server/remove apikey from statdb

* start audit service after statdb in captplanet
2018-12-04 13:47:58 -05:00
Kaloyan Raev
b7c45c7c16
Migrate Uplink CLI to Metainfo API (#745) 2018-12-04 18:46:02 +02:00
aligeti
9f6e5d6731
Keep track of 'irreparable' segments in a database (#686)
* initial irreparable development

* added the Open, Insert, Update, Get, Delete, Close  support

* added the Unit test cases

* code review changes
2018-12-04 11:26:30 -05:00
Egon Elbre
ae790dfd9f
Fix queue implementation (#758) 2018-12-04 17:46:53 +02:00
Egon Elbre
efe475b448
Ensure that restrictions are consistent. (#759) 2018-12-04 16:02:03 +02:00
Bryan White
24f6da2dd5
Kad inspector fixes (#746)
* pretty print kad inspector

* fix format string

* cleanup

* cleanup

* usage, args, print improvements

* kad inspector tweaks

* pretty print kad inspector

* cleanup

* cleanup

* kad inspector tweaks

* linter fixes
2018-12-03 21:01:54 +01:00
Maximillian von Briesen
c6dd70541e
move sdb apikey to config struct (#753) 2018-12-03 13:40:03 -05:00
Kaloyan Raev
0018ebf63e
Object.Bucket as storj.Bucket instead of string (#747) 2018-12-03 16:38:03 +02:00
Bryan White
120d875e06
consolidate NodeRep and NodeStats (#744)
* consolidate `NodeRep` and `NodeStats`

* regenerate

* review fixes

* go travis
2018-12-03 14:23:12 +01:00
Bryan White
fe8c1823b7 Cleanup debug code 2018-12-03 12:45:27 +02:00
Bryan White
71ef93feed
pretty print kad inspector (#743)
* pretty print kad inspector

* fix format string

* cleanup

* cleanup

* usage, args, print improvements
2018-11-30 18:36:05 +01:00
Egon Elbre
3e461df470
Linting protobuf files (#676) 2018-11-30 17:02:01 +02:00
Kaloyan Raev
66476da1ae
metainfo: basic upload (#726) 2018-11-30 15:50:52 +02:00
Egon Elbre
1e4556f88a
Fix import groupings (#739) 2018-11-30 15:40:13 +02:00
Bryan White
66b3a20595
cleanup and bugfix (#737) 2018-11-30 14:04:59 +01:00
Egon Elbre
9ac3517432
Reduce noise in logging (#733) 2018-11-29 22:59:26 +02:00
Bryan White
2a0c4e60d2
preparing for use of customtype gogo extension with NodeID type (#693)
* preparing for use of `customtype` gogo extension with `NodeID` type

* review changes

* preparing for use of `customtype` gogo extension with `NodeID` type

* review changes

* wip

* tests passing

* wip fixing tests

* more wip test fixing

* remove NodeIDList from proto files

* linter fixes

* linter fixes

* linter/review fixes

* more freaking linter fixes

* omg just kill me - linterrrrrrrr

* travis linter, i will muder you and your family in your sleep

* goimports everything - burn in hell travis

* goimports update

* go mod tidy
2018-11-29 19:39:27 +01:00
Yaroslav Vorobiov
a21855ac37
satellite user validation (#729) 2018-11-29 18:23:44 +02:00
Egon Elbre
0ae05cf834
Ensure that we only choose storage nodes (#732) 2018-11-29 16:57:00 +02:00
Yehor Butko
7b628e8af6
V3-763 Create logic for project details (#728)
* V3-763 Create logic for project details

* updating markup

* icon fixed
2018-11-28 18:20:23 +02:00
Yaroslav Vorobiov
c27431d64f
company update api (#725) 2018-11-28 14:30:38 +02:00
Yaroslav Vorobiov
24159216bc
update user api (#715) 2018-11-28 12:31:15 +02:00
Kaloyan Raev
f46d43b0d0
Fix panic in Audit service (#713) 2018-11-28 09:33:17 +02:00
Dylan Lott
a245aa7827
Inspector - Kad PING (#722)
* initial commit of inspector gadget wireup

* change name of comman dline tool, setup grpc server

* Get inspector cli working with grpc client

* Wired up CountNodes command

* WIP getting buckets response working

* Added GetBucket command

* WIP working on get buckets command

* WIP working on bucket list

* Still WIP

* WIP getting bucket counts to work

* Some clean up of unnecessary changes

* List Buckets and Get Bucket are working

* Removing logs, getting ready for review

* initial commit of inspector gadget wireup

* change name of comman dline tool, setup grpc server

* Get inspector cli working with grpc client

* Wired up CountNodes command

* WIP getting buckets response working

* Added GetBucket command

* WIP working on get buckets command

* WIP working on bucket list

* Still WIP

* WIP getting bucket counts to work

* Some clean up of unnecessary changes

* List Buckets and Get Bucket are working

* Removing logs, getting ready for review

* Fix error return

* Trying to get tests passing

* Adds method on dht mock for tests

* Add dbx files back

* Fix package import error in dbx file

* Adds copyrights to pass linter

* tidy go mod

* Updates from code review

* Updates inspector to take flag arguments for address

* Format list-buckets output more prettier

* Wiring up PING in kad inspector tools
2018-11-27 15:50:52 -07:00
Dylan Lott
fd42795456
updates dbx/gen.go to name generated package correctly (#723) 2018-11-27 13:52:19 -07:00
Maximillian von Briesen
7cf16503b8
Add statdb to overlay cache (node selection) (#711)
* make overlay node rep consistent with statdb node rep

* add statdb to testplanet

* add statdb to overlay cache

* tests
2018-11-27 12:46:12 -05:00
Michal Niewrzal
376bd74bed
Stops logging repairer errors for empty queue (#717) 2018-11-27 16:57:51 +01:00
Yaroslav Vorobiov
a0d84ccd67
Satellite user delete (#707)
* auth from context

* user delete api
2018-11-27 16:20:58 +02:00
Yehor Butko
784f3a887d
V3-760 Create logic for New project popup (#714)
* V3-760 Create logic for New project popup

* removing path to static files from gateway

* added missing copyrights
2018-11-27 13:14:10 +00:00
Kaloyan Raev
ea9bf38d85
eestream avoids waiting for pending readers if lots of errors (#709) 2018-11-27 10:17:57 +02:00
Bill Thorp
b8b0d75cc6
made accounting use migrate (#688)
* made accounting use migrate
2018-11-26 16:49:55 -05:00
Michal Niewrzal
8dd669b37f
Bandwidth allocations need method designation (GET/PUT) (#708)
* Bandwidth allocations need method designation (GET/PUT)

* add method comment

* goimports
2018-11-26 19:21:44 +01:00
Maximillian von Briesen
5014a785a0
Node selection 0/4: Update statdb auth/functionality (#698)
* remove api key from statdb server reqs; add statdb UpdateUptime and UpdateAuditSuccess to server

* update api key authentication in statdb server

* add todos for future statdb updates

* add UpdateUptime and UpdateAuditSuccess to statdb server

* fix apikey stuff in config.go and statdb_test.go

* fix tests

* update sdbclient.NewClient call in audit package

* fix UpdateUptime and UpdateAuditSuccess in sdbclient

* set api key from statdb/config.go

* change package for statdb tests

* linter fixes

* remove todo comments

* fix sdbclient err checking

* move validate auth functionality to auth package

* update description for statdb api key

* remove import
2018-11-26 12:08:29 -05:00
Yehor Butko
1f1f9f5875
Satellitedb company-user relation changed to 1-to-1 (#706)
* Satellitedb company-user relation changed to 1-to-1

* fixing comments
2018-11-26 14:57:55 +00:00
Yehor Butko
a998665491
Satellite projects api (#703)
* V3-645 Create Project API

* fixing linter

* update project refactored
2018-11-26 10:47:23 +00:00
Kaloyan Raev
67db26b36e
Metainfo stream download (#697) 2018-11-26 09:39:05 +02:00
Maximillian von Briesen
b884ee4626
Update protos to fit style guide (easy review) (#702)
* fix overlay.proto and piecestore.proto to meet style guide

* update code to be consistent with regenerated protos
2018-11-23 21:46:53 -05:00
Yaroslav Vorobiov
bdebe36ae0
updated login api (#699) 2018-11-22 12:38:58 +02:00
Maximillian von Briesen
06cf1c71ed
Statdb inspection tool (#626)
* begin adding stadb inspection tool

* add command for creating statdb nodes from csv

* add getCSVStats and create helper func for getting sdb client

* fix createcsvstats, add createstats

* fix linter errors
2018-11-21 17:01:09 -05:00
Dylan Lott
0d05cb26bf
Kademlia Inspector CLI (#657)
* initial commit of inspector gadget wireup

* change name of comman dline tool, setup grpc server

* Get inspector cli working with grpc client

* Wired up CountNodes command

* WIP getting buckets response working

* Added GetBucket command

* WIP working on get buckets command

* WIP working on bucket list

* Still WIP

* WIP getting bucket counts to work

* Some clean up of unnecessary changes

* List Buckets and Get Bucket are working

* Removing logs, getting ready for review

* initial commit of inspector gadget wireup

* change name of comman dline tool, setup grpc server

* Get inspector cli working with grpc client

* Wired up CountNodes command

* WIP getting buckets response working

* Added GetBucket command

* WIP working on get buckets command

* WIP working on bucket list

* Still WIP

* WIP getting bucket counts to work

* Some clean up of unnecessary changes

* List Buckets and Get Bucket are working

* Removing logs, getting ready for review

* Fix error return

* Trying to get tests passing

* Adds method on dht mock for tests

* Add dbx files back

* Fix package import error in dbx file

* Adds copyrights to pass linter

* tidy go mod

* Updates from code review

* Updates inspector to take flag arguments for address

* Format list-buckets output more prettier
2018-11-21 10:31:27 -07:00
Yaroslav Vorobiov
c829835dc1
updated createuser api (#687) 2018-11-21 17:51:43 +02:00
Michal Niewrzal
f72832ee69
Add metadata to pb.Node (email and wallet) (#671)
* Add metadata to pb.Node (email and wallet)

* goimports

* reorganize config

* fix typo

* remove 'foo' id

* add basic test
2018-11-21 16:07:18 +01:00
Kaloyan Raev
aec2c41911
Metainfo remote segment test (#696) 2018-11-21 16:35:53 +02:00
Kaloyan Raev
8c56198090
Use designated error classes for common bucket and object errors (#691) 2018-11-21 13:17:28 +02:00
Bryan White
dabd87cb36
swap golang/protobuf out for gogo/protobuf (#692)
* swap golang/protobuf out for gogo/protobuf

* revert go.mod and go.sum changes

* linter fixes
2018-11-20 19:29:07 +01:00
Bryan White
54ccb460a9
rename size fields in protobuf messages for gogo compatibility (#690)
* rename `size` fields in protobuf messages for gogo compatibility

* linter fixes
2018-11-20 18:09:35 +01:00
Dennis Coyle
e7e0d1daaa
Coyle/disable mock (#627)
* disabled mock overlay from captplanet
2018-11-20 11:54:52 -05:00
Jennifer Li Johnson
93c5f385a8
Enable checker in captplanet and staging (#643)
* enable checker

* add option to use mock overlay in checker

* adds logs to checker

* appease linter
2018-11-20 10:54:22 -05:00
Yehor Butko
4734a7447b
V3-805 Extend satellitedb with Project members entity (#683)
* V3-805 Extend satellitedb with Project members entity

* fixing linter

* small refactoring

* tests refactoring
2018-11-20 14:50:47 +00:00
Egon Elbre
76af9f5171
Add overlay server to testplanet (#677) 2018-11-19 22:39:25 +02:00
Egon Elbre
832317b0ee
Remove unused workers (#640) 2018-11-19 17:07:24 +02:00
Egon Elbre
d07433c150
Use testplanet to test Refresh (#678) 2018-11-19 16:40:01 +02:00
Kaloyan Raev
bb1cf151e3
Metainfo handle inline segments in ReadOnlyStream (#674) 2018-11-19 13:08:28 +02:00
Egon Elbre
6fe16e48ba
Simplify and fix tests (#660) 2018-11-16 18:31:14 +02:00
Yaroslav Vorobiov
3e99861e04
operationName and variables fields added to graphql json parsing (#668) 2018-11-16 16:34:35 +02:00
Kaloyan Raev
f730ce451a
Telemetry in metainfo implementation (#670) 2018-11-16 15:59:27 +02:00
aligeti
7958994ae2
Diagnostic tool to inspect repair queue (#656)
* initial repair queue diag tool development

* fixes linter warnings

* code review updates
2018-11-16 08:31:33 -05:00
Alexander Leitner
5e9377524f
Refactor basystem (#641)
* Signature verification

* Clean up agreement sender to have less errors

* overlay address in captnplanet

* Refactor bandwidth.proto to not use streams

* Make sure the send worked

* Handle connection to satellite

* Save renter public key inside of renter bandwidth allocations

* Default diag to sqlite. Make configurable

* Separate bw server and dbm; regenerate dbx files

* Make sure test uses protobufs

* Demonstrate creating bandwidth allocations
2018-11-15 14:06:09 -05:00
Egon Elbre
c4b90f84dd
Use better defaults and naming for postgres database (#659) 2018-11-15 20:36:57 +02:00
Kaloyan Raev
0357e61bbd
metainfo objects tests (#662) 2018-11-15 17:31:33 +02:00
Yaroslav Vorobiov
92b1bceca7
added copyright to files in satelliteauth pkg (#663) 2018-11-15 14:25:04 +02:00
Yaroslav Vorobiov
3f2ce11315
added copyright to every file in satellite pkg (#661) 2018-11-15 14:00:08 +02:00
Egon Elbre
21a102a05f
testplanet: add logger (#649) 2018-11-15 10:57:47 +02:00
Maximillian von Briesen
2c36dcc2c8
Update statdb (#650)
* add sdbclient.UpdateUptime; update args for sdbclient.CreateEntryIfNotExists

* add auditcount to node stats; restructure statdb.CreateEntryIfNotExists

* add noop mock sdbclient

* add the ability to create a node in statdb without "default" stats

* update statdb.CreateEntryIfNotExists

* take fewer args for sdbclient.CreateWithStats/FindValidNodes

* add sdbclient.UpdateAuditSuccess

* update sdbclient.Update so that all fields are updated when called (reduce args)

* update error checking in statdb.Create
2018-11-14 19:03:19 -05:00
Bill Thorp
d3c347a0ac
Verify overlay cache is accurately and fully tested (#540)
* added intro node id, moved port range to 1024, listen in go routines
2018-11-14 16:30:07 -05:00