Commit Graph

697 Commits

Author SHA1 Message Date
aligeti
ce1db97ce6
fix manipulated paychecks V3-1023 (#961)
Satellite crashing on receiving a manipulated bandwidthagreement V3-1022
2019-01-04 11:26:26 -05:00
Yehor Butko
262753bde6
V3-1029 Update list Project Logic (#966) 2019-01-04 13:32:21 +02:00
Matt Robinson
c0e6b62708
Test all-in-one (#900)
* Add test for aio

* Don't trust the user to have images built for a version

* Make travis run the aio test

* Add missing values to docker-compose, sort some things, consider the gateway image

* today's changes

* config changed, again

* more fixes

* Expose satellite port on localhost:7778

* Add retries and a timeout around the big-testfile test in AIO

* Another config value changed

* Make this error message a little more useful

* Fix nil condition
2019-01-03 14:54:27 -05:00
Cameron
9fa874459a
remove starting node log in piecestore server (#964) 2019-01-03 13:57:12 -05:00
Egon Elbre
011b889ba7
pkg/process: properly quote arguments (#963) 2019-01-03 20:00:59 +02:00
Egon Elbre
26c2564bd8
pkg/kademlia: simplify code (#958) 2019-01-02 20:57:11 +02:00
Jennifer Li Johnson
a2fa5c4c5a Proper NodeType Handling (#873)
* adds enums to nodetype

* updating nodetype todos

* ran pb updates

* reorder nodetypes

* adding checks

* wip

* wip

* wip

* bug in test-captplanet

* wip

* add values to storagenode, satellite, captplanet binaries

* Cleanup

* more cleanup

* wip

* lint

* lint

* wip

* fixes bug

* regenerate protos

Change-Id: Id270212e8c7479e52641058042cf23b5317ab773

* limit node type changes to kademlia

Change-Id: I9c1a6cc4a79e05086627f0fdeb5028c62ce754f4

* dpanic

Change-Id: Id952a2ad13c807ebaea0ec0a875405e267d81c3e

* review comments

Change-Id: I7f9b77ef22779dd012fd490375b136014f51f834
2019-01-02 11:47:34 -07:00
Egon Elbre
89db0fe9f5
storj-sdk base code (#799) 2019-01-02 20:07:49 +02:00
Egon Elbre
0ca03b41e2
Use generated locking database implementation (#947) 2019-01-02 19:53:27 +02:00
Bryan White
249244536a
CSR Service (part 2): cert signing rpc (#950)
* CSR Service:

+ implement certificate sign rpc method
+ implement certificate signer client/server
+ refactor `AuthorizationDB#Create`
+ refactor `NewTestIdentity`
+ add `AuthorizationDB#Claim`
+ add `Token#Equal`
+ fix `Authorizations#Marshal` when marshaling identities and certificates
+ tweak `Authorization#String` format
+ cert debugging improvements (jsondiff)
+ receive context arg in `NewTestIdentity`
+ misc. fixes
2019-01-02 12:39:17 -05:00
Michal Niewrzal
b712fbcbb0
Fix 'empty queue' error when satellite starts (#939) 2019-01-02 17:00:32 +01:00
Bill Thorp
e4b3dab446
Add config editing via the CLI for storage nodes (#919)
* added editor and updated storagenode cmd
* switched to custom yaml gen
2019-01-02 10:12:07 -05:00
Matt Robinson
cbc5fdf99a
Fix CA Timeout and show progress to the user (#915)
* Fix idle after CA timeout and show progress to the user so they don't think we've hung

* fix timeout again since it was lost in a rebase
2019-01-02 09:41:01 -05:00
JT Olio
3fbd8c7d15 kademlia: remove a redundant argument (#951) 2019-01-02 12:57:06 +02:00
JT Olio
d4d0b0d13b inspector: use existing identity (#952) 2019-01-02 12:45:47 +02:00
JT Olio
4a0749e44a Remove farmer references (#949) 2019-01-02 12:31:49 +02:00
JT Olio
2c916a04c3 pkg/provider: split into pkg/server, pkg/identity (#953) 2019-01-02 12:23:25 +02:00
Egon Elbre
cc8cce58dd
statreceiver nitpicking (#926) 2019-01-01 11:41:27 +02:00
Bryan White
4efb5c0a75
Authorization improvements (#925) 2018-12-31 10:45:43 -05:00
Yehor Butko
e82edc68c8
V3-976 Create db query for filtering team mebers (#940)
* V3-976 Create db query for filtering team mebers

* fixing linter

* fixing linter

* sql injection fixed

* getOrder renamed, tests added
2018-12-28 14:07:35 +02:00
Yaroslav Vorobiov
d65cefcac7
Satellite api keys api (#936) 2018-12-27 17:30:15 +02:00
Egon Elbre
4346cd060f
Implement mutex around satellitedb (#932) 2018-12-27 11:56:25 +02:00
Michal Niewrzal
a9572b7169
Fix ignored config logger settings (#938) 2018-12-27 10:38:11 +01:00
Nikolay Yurchenko
312fd5a5fd
[V3-963] Satellite team members add dialog (#935)
* Implemented team member add dialog without api.

* logic changed according to api

* added initial app reducer and team members deletion fixed
2018-12-26 17:05:33 +02:00
Yaroslav Vorobiov
3b9f5fa633
satellite api keys repo and service method added (#931) 2018-12-26 16:00:53 +02:00
Yehor Butko
5ef427265e
web/satellite notifications added, users and projects api updated (#934)
* web/satellite notifications added, users and projects api updated

* fix users api
2018-12-24 14:52:52 +02:00
Dylan Lott
7569b7d71b
Node lifecycle hooks (#920)
* wires up first draft of lifecycle methods

* creates interface on transport

* node lifecycle hooks works

* linter fixes

* adds error log at connection success

* chnages Observer interface to use context

* Makes Discovery take its own logger

* WIP

* linter fixes

* Test fixes

* adds in ConnFailure code for cache
2018-12-21 21:51:42 -07:00
Dylan Lott
2f6835a3be
Adds a logger to RoutingTable (#922)
* adds delete functions and tests for cache

* adds logger to routing table struct and passes it through to logger

* add loggers
2018-12-21 17:48:53 -07:00
Dylan Lott
05c1644fec
adds delete functions and tests for cache (#921) 2018-12-21 11:35:21 -07:00
Yaroslav Vorobiov
16db288c85
Satellite add/delete project members by email api (#928)
* Satellite add/delete project members by email api
2018-12-21 17:41:53 +02:00
Cameron
f70b826fd4
repair queue masterDB support (#865)
* add injuredsegment model to satellitedb.dbx

* add context to queue.RepairQueue interface

* use queue.RepairQueue interface, use masterdb
2018-12-21 10:11:19 -05:00
Bill Thorp
3fa094f3da
Add the ability for Kademlia to do a refresh for each bucket (#908)
* added random id in range, added refresh
2018-12-20 16:45:06 -05:00
Dennis Coyle
18c4972205
add monkit to Satellite DB Service (#903)
* add monkit to Satelite DB Service

* goimports
2018-12-20 15:10:27 -05:00
Bryan White
398379b149
CSR service (part 1): authorizations (#906) 2018-12-20 19:29:05 +01:00
Yehor Butko
ac860c1984
Removing company and project ownership (#917) 2018-12-20 18:18:08 +02:00
Yaroslav Vorobiov
1c119c6215
(#916) delete and add projectmember api now accepts list of users
* delete and add projectmember api now accepts list of users

* review changes

* simplification

* use errs.Combine
2018-12-20 17:36:32 +02:00
Egon Elbre
d9b9ae6ffa
Cleanup overlay methods and names. (#914) 2018-12-20 15:57:54 +02:00
Maximillian von Briesen
f18c38628e
Refactor statdb (#887)
Update statdb args/return values to minimize structs
    Simplify statdb.Update() to update all stats instead of an arbitrary subset determined by flags
    Remove CreateIfNotExists logic from statdb.Update()
    Simplify audit code structure
2018-12-19 13:44:03 -05:00
Yaroslav Vorobiov
76018231c6
satellite project members pagination api (#907)
* satellite project members pagination api

* rename GetByProjectIDPaged to GetByProjectID

* add comment to max limit
2018-12-19 15:03:12 +02:00
Dennis Coyle
1b7ab9600d
Do not error if Discovery can not find Node (#902) 2018-12-18 15:01:15 -05:00
aligeti
3c7f62be73
KBucket clean up (#898) 2018-12-18 13:04:46 -05:00
Yaroslav Vorobiov
efdfff9704
updated project and user deletion (#884)
* update project and user deletion

* linter fixes
2018-12-18 19:43:02 +02:00
Jennifer Li Johnson
01df85704c
Updates accounting tables and allows tally to update at raw table for at rest data (#874) 2018-12-18 12:18:42 -05:00
Egon Elbre
d9a13667fc
Use fixed logger for Kademlia and make not being able to connect in discovery debug statement. (#899) 2018-12-18 17:13:32 +02:00
Bryan White
4eb55017c8
Cert revocation CLI (#848)
* wip

* allow identity and CA configs to save cert/key separately

* fixes

* linter and default path fixes

* review fixes

* fixes:

+ review fixes
+ bug fixes
+ add extensions command

* linter fixes

* fix ca revoke description

* review fixes
2018-12-18 12:55:55 +01:00
Natalie Villasana
17c60c1f06
moves node selection config setup from uplink to satellite (#891) 2018-12-17 16:05:05 -05:00
Michal Niewrzal
a819d819e3
Overlay Cache master db merge (#877)
* Overlay Cache master db merge

* test update

* review comments

* Fix transaction usage

* removed unused variable

* added better limit handling

* better error handling
2018-12-17 21:14:16 +01:00
Egon Elbre
6b5945139f
Overlay error handling (#880) 2018-12-17 20:47:26 +02:00
Egon Elbre
5c26aa3e4a
Use storj.NodeID directly in statdb tests (#881) 2018-12-17 20:47:04 +02:00
Natalie Villasana
d97e27ed80
removes a few todos (#867) 2018-12-17 13:11:29 -05:00
Michal Niewrzal
1d2523cf60
Payments API protobuf definition (#876) 2018-12-17 18:36:50 +01:00
Dylan Lott
706f20611d
adds discovery function to overlay for random lookups (#860)
* adds discovery function to overlay for random lookups

* uses different variable name

* linter fixes
2018-12-17 09:31:14 -07:00
Cameron
0d17c21a1b
Add logger to piecestore server/node started log (#856)
* add logger to psserver struct

* node started log

* rename initialize to NewEndpoint

* return err from WriteBandwidthAllocToDB
2018-12-17 10:23:02 -05:00
Egon Elbre
d8df4b5f6b
Simplify provider tests (#862) 2018-12-17 17:09:52 +02:00
Yehor Butko
90f79e585c
V3-975 Create db pagination query for team members (#871)
* V3-975 Create db pagination query for team members

* fix go mod tidy

* Methods renamed. Picked go.sum from storj/master.

* fixing linter
2018-12-17 16:28:58 +02:00
Michal Niewrzal
f0662675bc
Add '--dir' param for all CLI parts (replace --base-path) (#833)
* Add '--dir' param for all CLI parts (replace --base-path)

* FindDirParam method moved

* fix compilation error

* make param global

* remove unused fields

* rename param

* remove config flag

* goimports
2018-12-14 22:14:59 +01:00
Bryan White
d8db7c3049
RevocationDB fixes (#866) 2018-12-14 21:45:53 +01:00
aligeti
5e1b02ca8b
Statdb master db v3 848 (#830)
* intial changes to migrate statdb to masterdb framework

* statdb refactor compiles

* added TestCreateDoesNotExist testcase

* Initial port of statdb to masterdb framework working

* refactored statdb proto def to pkg/statdb

* removed statdb/proto folder

* moved pb.Node to storj.NodeID

* CreateEntryIfNotExistsRequest moved pd.Node to storj.NodeID

* moved the fields from pb.Node to statdb.UpdateRequest

ported TestUpdateExists, TestUpdateUptimeExists, TestUpdateAuditSuccessExists TestUpdateBatchExists
2018-12-14 15:17:30 -05:00
Dylan Lott
9bdee7b106
Discovery service (#805)
* WIP possible discovery service impl

* Adds discovery service to CaptPlanet

* Updates the config and server for discovery service

* updates testplanet to use discovery package

* update satellite imports

* Removes unnecessary cache test

* linter fixes

* adds discovery startup to captplanet

* invoke refresh

* updates to discovery refresh cycle

* Make implementation more consistent with previous implementation

* add wait before trying to upload

* sleep a bit more

* remove kademlia bootstrap

* updates

* remove comments
2018-12-14 12:23:54 -07:00
Yaroslav Vorobiov
60fb655db2
satellite user delete updated to require user password (#852) 2018-12-14 18:14:17 +02:00
Michal Niewrzal
725ed44ce0
Add gen.go for protobuf (#863) 2018-12-14 16:55:55 +01:00
Michal Niewrzal
db5a3e0d0a
Parameters cleanup (#864)
* Parameters cleanup

* goimports
2018-12-14 16:55:45 +01:00
Bill Thorp
f9845e7e92
changes to masterdb to support accountingDB (#846)
* added rollup to captplanet, moved accountingDB to masterdb
2018-12-14 09:27:21 -05:00
Egon Elbre
7363e6cfd8
Better transport tests (#849) 2018-12-14 12:44:48 +02:00
Bill Thorp
c17a6ba584
fixed temp dir issues w/ rev db (#859) 2018-12-13 21:24:35 -05:00
Bryan White
2016ce9fd6
Certificate revocation (#836)
* wip certificate revocation

* refactor tests

* wip testing

* testing

* review fixes

* integration fix attempt #1

* review fixes

* integration fix attempt #2

* linter fixes

* add copywrite

* integration fix attemp #3

* more testing

* more tests

* go mod tidy

* review fixes

* linter fixes
2018-12-13 21:01:43 +01:00
Egon Elbre
08394389d3 Fix ConfNested on Windows (#851) 2018-12-13 07:34:38 -07:00
Egon Elbre
969fcabf72
Remove utils.LogClose from identity saving (#837) 2018-12-13 15:50:05 +02:00
Kaloyan Raev
2a8fef8062
Data repairer should use the redundancy strategy from segment's pointer (#838) 2018-12-13 09:12:36 +02:00
Jennifer Li Johnson
6642f97142
Jj/accounting updates (#820)
* tallies up data stored on each node in pointerdb

* adds comments for data type enums

* changes Open to BeginTx because Go convention

* removes online status check from identify active nodes

* changes identifyactivenodes to calculatestaticdata

* updates accounting dbx names
2018-12-12 16:24:08 -05:00
Maximillian von Briesen
a6c7306350
Cut off piecestore Puts if they exceed alloced bandwidth/space (#819)
* add bandwidth/storage limits to StreamWriter

* add StreamWriter tests for bandwidth/storage limits
2018-12-12 14:14:51 -05:00
Egon Elbre
7a80e7bf2f
Use testplanet in node tests (#841) 2018-12-12 17:40:33 +02:00
Egon Elbre
c56307f5b9
datarepair: remove unneeded debug message (#839)
* datarepair: remove unneeded debug message
* Make overlay-cache and routing table logs more specific
2018-12-12 17:39:16 +02:00
Egon Elbre
38f72df8b5
Faster tests (#834) 2018-12-12 16:05:47 +02:00
Bill Thorp
ba5f71810e
added tally to capt run.go (#751)
* added tally to capt run.go, ParseURL -> SplitDBURL
2018-12-12 08:15:34 -05:00
JT Olio
4a4f6ad53e
cfgstruct.Bind: support nested config paths (#785)
this will allow some config cleanups in a future pr

Change-Id: Ie873bcee567a72956d9337dfc13ab6ba46c9d1a0
2018-12-11 11:41:19 -07:00
JT Olio
ceb590fa67
capt: reduce nodes to 10 (#793)
Change-Id: Ief380fe29e3043657705cd7505c266fd774181a4
2018-12-11 11:40:54 -07:00
JT Olio
362f447d9f
cmd/statreceiver: lua-scriptable stat receiver (#636)
* cmd/statreceiver: lua-scriptable stat receiver

Change-Id: I3ce0fe3f1ef4b1f4f27eed90bac0e91cfecf22d7

* some updates

Change-Id: I7c3485adcda1278fce01ae077b4761b3ddb9fb7a

* more comments

Change-Id: I0bb22993cd934c3d40fc1da80d07e49e686b80dd

* linter fixes

Change-Id: Ied014304ecb9aadcf00a6b66ad28f856a428d150

* catch errors

Change-Id: I6e1920f1fd941e66199b30bc427285c19769fc70

* review feedback

Change-Id: I9d4051851eab18970c5f5ddcf4ff265508e541d3

* errorgroup improvements

Change-Id: I4699dda3022f0485fbb50c9dafe692d3921734ff

* too tricky

the previous thing was better for memory with lots of errors at a time
but https://play.golang.org/p/RweTMRjoSCt is too much of a foot gun

Change-Id: I23f0b3d77dd4288fcc20b3756a7110359576bf44
2018-12-11 11:24:31 -07:00
Natalie Villasana
9e1ec97b31
adds node selection config (#782) 2018-12-11 12:30:14 -05:00
Kaloyan Raev
9449010c15
Enable overlay by default (#835) 2018-12-11 18:52:45 +02:00
Kaloyan Raev
252da15f0d
Randomize node selection during GETs (#827) 2018-12-11 18:05:14 +02:00
Yaroslav Vorobiov
175e25f93a
Satellite graphql: take authorized user id if no id was provided in iput (#828)
satellite graphql: take authorized user id if no id was provided by client
2018-12-11 17:54:45 +02:00
Egon Elbre
492639aef7
Skip TestUploadDownload (#826) 2018-12-11 16:52:03 +02:00
Egon Elbre
8e14d2cb94
Reduce ranger test output (#825) 2018-12-11 11:38:56 +02:00
Michal Niewrzal
7e2f66ee24
Small fixes around master db (#822) 2018-12-11 10:30:09 +01:00
Egon Elbre
26dbf788ad
Optimize compilation by removing logging tests (#816) 2018-12-11 10:46:52 +02:00
aligeti
d27bcddaab
Merge the irreparable DB into the master database (#777)
* initial refactoring to merge irreparabledb to master db

* refactor

* code review comment updates
2018-12-10 14:08:45 -05:00
Yehor Butko
a8a082f4c4
V3-808 Implement graphql queries for team members page (#815) 2018-12-10 19:32:15 +02:00
Yaroslav Vorobiov
63998b4a62
updated user update api, separated company and user creation (#767) 2018-12-10 17:57:06 +02:00
Kaloyan Raev
3175eb74a6
Improve piece retrieving log with offset and length (#813) 2018-12-10 17:22:49 +02:00
Michal Niewrzal
a15099d5f5
Simplify testing with all supported DBs (#809)
* Simplify testing with all supported DBs

* move method to separate package

* method renamed to Run

* fixed comment
2018-12-10 15:50:12 +01:00
Kaloyan Raev
3cd0b4adb2
Verify bucket exists before starting multipart upload (#812) 2018-12-10 16:41:43 +02:00
Kaloyan Raev
4cc49a8fa3
Better error handling of stream.Upload.Close (#811) 2018-12-10 16:35:22 +02:00
Yehor Butko
b1605a7331
Updating users repository (#808) 2018-12-10 15:47:48 +02:00
Yehor Butko
678eb81d54
V3-898 Create ability for querying db in transaction scope (#781)
* V3-898 Create ability for querying db in transaction scope

* handling errors

* fixing linter

* Transaction updated

* fixing empty lines linter

* tx signatures changed

* fixing comments
2018-12-10 14:29:01 +02:00
Yaroslav Vorobiov
8ff6cea5a9
[WIP] get project members api (#801) 2018-12-10 13:38:42 +02:00
Egon Elbre
c7cf97cc23
Fix sqlite registration (#806) 2018-12-10 10:32:49 +02:00
Cameron
f46fcd0303
upload/download tests with test-planet (#718)
* add upload/download test

* use minio gateway for upload download

* move test into miniogw package

* logging edits, remove logging_test.go
2018-12-07 14:26:39 -05:00
Kaloyan Raev
c025c13706
Fix deadlock on testplanet shutdown (#798) 2018-12-07 20:31:59 +02:00
Kaloyan Raev
72a20af569
Migrate Minio Gateway to the new Metainfo API (#775) 2018-12-07 20:31:29 +02:00
Egon Elbre
554daf17e6
Remove mock kademlia (#802) 2018-12-07 19:36:20 +02:00
JT Olio
890c87914f
pstore: fix blob permissions (#788)
Change-Id: I004747657221c6ac53894fc5c5917e3f92436990
2018-12-07 07:47:45 -07:00
JT Olio
1c96db01ba
better database error handling (#784)
* better database error handling

Change-Id: I28dbd69cf6c2fa268e02405521ff6e6c1a68a702

* missing comments added

* missing comment added
2018-12-07 07:46:42 -07:00
Brandon Iglesias
023eb95a9b
cleaning up todos in Kademlia pkg (#763)
* updating comments in KAD code

* deleting todo comment because they are not needed anymore
2018-12-07 08:57:42 -05:00
Bryan White
228aa34ff6
TLS extension processing (#771) 2018-12-07 14:44:25 +01:00
Michal Niewrzal
4608976526
Bwagreement tests cleanup (#795) 2018-12-07 13:11:35 +01:00
Michal Niewrzal
d62896bbaf
Use testcontext with tally tests (#796) 2018-12-07 12:55:25 +01:00
Michal Niewrzal
034a34069a
Merge bwagreement db into satellite master db (#770)
* Merge bwagreement db into satellite master db

* adjust to recent tally changes

* linter problems

* linter problems

* returning db structs in more optimal way

* added pointer for assignment

* error message changed

* better param message
2018-12-07 10:59:31 +01:00
Jennifer Li Johnson
dac79fd762
removes check for mock overlay in data repair checker config (#794) 2018-12-07 00:42:16 -05:00
JT Olio
b6c936340b
clean up uplink config section (#786) 2018-12-06 17:18:11 -07:00
Alexander Leitner
04ff6b8e11
Use audit results in checker (#754)
* Use statdb to get audit results

* Find invalid nodes
2018-12-06 13:51:23 -05:00
Kaloyan Raev
b043fd76dd
Fix race in connection pool (#778) 2018-12-06 18:05:57 +02:00
Yehor Butko
0c95fba4c8
Satellite Projects db entity and api updated (#772)
* Satellite Projects db entity and api updated

* get users projects query updated with dbx
2018-12-06 17:19:47 +02:00
Yaroslav Vorobiov
d6f6ab56a1
satellite projectmembers add and delete api (#776) 2018-12-06 16:40:32 +02:00
Egon Elbre
fc8e9c6572
Move irreparabledb under datarepair (#764) 2018-12-06 14:32:21 +02:00
Maximillian von Briesen
828a8b6907
Change statdb.FindValidNodes to FindInvalidNodes (#774)
* Remove statdb.FindValidNodes
* Add statdb.FindInvalidNodes for getting nodes that fall below a reputation threshold
2018-12-05 15:23:32 -05:00
Dennis Coyle
fffb093056
SelfAddr -> ExternalAddress (#768) 2018-12-05 11:36:03 -05:00
Egon Elbre
c4033b15af
Simplify peer discovery implementation (#765) 2018-12-05 16:32:37 +02:00
Bill Thorp
b2a8a10b16
redo tally query (#742)
* redo tally query

* broken, sharing w/ Jen

* tally tests added, failiing

* fix txn issue in tally

* fix lint
2018-12-05 09:03:23 -05:00
Dennis Coyle
e138056943
listen to configured address (#755) 2018-12-05 05:37:03 -05:00
Dylan Lott
69281552b3
Pretty prints node output from kad lookup tool (#762)
* Pretty prints node output from kad lookup tool

* Linter fix
2018-12-04 17:23:39 -07:00
Dylan Lott
c6d790d58e
Improvements to Kad Lookups (#731)
* adds channel for getting node out of lookup

* WIP adding the channels to lookups

* WIP adding channel to node lookups

* Wires up FindNodes method with channels

* WIP adds a test suite for lookup - tests are still failing

* WIP wires up use of testplanet for kademlia lookup tests

* WIP merging in node id changes

* Merges in pkg/storj node type changes

* Tests passing

* Lookup node working via Inspector now

* updates

* WIP working on getting tests passing

* WIP getting tests passing

* FindNode works

* Linter fix

* Adds copyrights to lookup_test

* removes a fmt.Printf I missed

* Removes commented out lines
2018-12-04 14:39:28 -07:00
Matt Robinson
4237afb1c1
Attempt at fixing the all-in-one environment (#736) 2018-12-04 16:10:23 -05:00
Maximillian von Briesen
6c655d117b
Filter nodes by reputation and IP address (node selection) (#720)
*   Pulls statdb stats into overlay cache whenever cache.Put() is called
* Updates overlay.FindStorageNodes()/overlayClient.Choose() to filter based on node stats
* Updates overlay.FindStorageNodes()/overlayClient.Choose() to exclude duplicate IP addresses
2018-12-04 15:18:26 -05:00
Maximillian von Briesen
2ab15196d9
Get rid of statdb grpc stuff (#761)
* remove sdbclient

* do not register statdb grpc server/remove apikey from statdb

* start audit service after statdb in captplanet
2018-12-04 13:47:58 -05:00
Kaloyan Raev
b7c45c7c16
Migrate Uplink CLI to Metainfo API (#745) 2018-12-04 18:46:02 +02:00
aligeti
9f6e5d6731
Keep track of 'irreparable' segments in a database (#686)
* initial irreparable development

* added the Open, Insert, Update, Get, Delete, Close  support

* added the Unit test cases

* code review changes
2018-12-04 11:26:30 -05:00
Egon Elbre
ae790dfd9f
Fix queue implementation (#758) 2018-12-04 17:46:53 +02:00
Egon Elbre
efe475b448
Ensure that restrictions are consistent. (#759) 2018-12-04 16:02:03 +02:00
Bryan White
24f6da2dd5
Kad inspector fixes (#746)
* pretty print kad inspector

* fix format string

* cleanup

* cleanup

* usage, args, print improvements

* kad inspector tweaks

* pretty print kad inspector

* cleanup

* cleanup

* kad inspector tweaks

* linter fixes
2018-12-03 21:01:54 +01:00
Maximillian von Briesen
c6dd70541e
move sdb apikey to config struct (#753) 2018-12-03 13:40:03 -05:00
Kaloyan Raev
0018ebf63e
Object.Bucket as storj.Bucket instead of string (#747) 2018-12-03 16:38:03 +02:00
Bryan White
120d875e06
consolidate NodeRep and NodeStats (#744)
* consolidate `NodeRep` and `NodeStats`

* regenerate

* review fixes

* go travis
2018-12-03 14:23:12 +01:00
Bryan White
fe8c1823b7 Cleanup debug code 2018-12-03 12:45:27 +02:00
Bryan White
71ef93feed
pretty print kad inspector (#743)
* pretty print kad inspector

* fix format string

* cleanup

* cleanup

* usage, args, print improvements
2018-11-30 18:36:05 +01:00
Egon Elbre
3e461df470
Linting protobuf files (#676) 2018-11-30 17:02:01 +02:00
Kaloyan Raev
66476da1ae
metainfo: basic upload (#726) 2018-11-30 15:50:52 +02:00
Egon Elbre
1e4556f88a
Fix import groupings (#739) 2018-11-30 15:40:13 +02:00
Bryan White
66b3a20595
cleanup and bugfix (#737) 2018-11-30 14:04:59 +01:00
Egon Elbre
9ac3517432
Reduce noise in logging (#733) 2018-11-29 22:59:26 +02:00
Bryan White
2a0c4e60d2
preparing for use of customtype gogo extension with NodeID type (#693)
* preparing for use of `customtype` gogo extension with `NodeID` type

* review changes

* preparing for use of `customtype` gogo extension with `NodeID` type

* review changes

* wip

* tests passing

* wip fixing tests

* more wip test fixing

* remove NodeIDList from proto files

* linter fixes

* linter fixes

* linter/review fixes

* more freaking linter fixes

* omg just kill me - linterrrrrrrr

* travis linter, i will muder you and your family in your sleep

* goimports everything - burn in hell travis

* goimports update

* go mod tidy
2018-11-29 19:39:27 +01:00
Yaroslav Vorobiov
a21855ac37
satellite user validation (#729) 2018-11-29 18:23:44 +02:00
Egon Elbre
0ae05cf834
Ensure that we only choose storage nodes (#732) 2018-11-29 16:57:00 +02:00
Yehor Butko
7b628e8af6
V3-763 Create logic for project details (#728)
* V3-763 Create logic for project details

* updating markup

* icon fixed
2018-11-28 18:20:23 +02:00
Yaroslav Vorobiov
c27431d64f
company update api (#725) 2018-11-28 14:30:38 +02:00
Yaroslav Vorobiov
24159216bc
update user api (#715) 2018-11-28 12:31:15 +02:00
Kaloyan Raev
f46d43b0d0
Fix panic in Audit service (#713) 2018-11-28 09:33:17 +02:00
Dylan Lott
a245aa7827
Inspector - Kad PING (#722)
* initial commit of inspector gadget wireup

* change name of comman dline tool, setup grpc server

* Get inspector cli working with grpc client

* Wired up CountNodes command

* WIP getting buckets response working

* Added GetBucket command

* WIP working on get buckets command

* WIP working on bucket list

* Still WIP

* WIP getting bucket counts to work

* Some clean up of unnecessary changes

* List Buckets and Get Bucket are working

* Removing logs, getting ready for review

* initial commit of inspector gadget wireup

* change name of comman dline tool, setup grpc server

* Get inspector cli working with grpc client

* Wired up CountNodes command

* WIP getting buckets response working

* Added GetBucket command

* WIP working on get buckets command

* WIP working on bucket list

* Still WIP

* WIP getting bucket counts to work

* Some clean up of unnecessary changes

* List Buckets and Get Bucket are working

* Removing logs, getting ready for review

* Fix error return

* Trying to get tests passing

* Adds method on dht mock for tests

* Add dbx files back

* Fix package import error in dbx file

* Adds copyrights to pass linter

* tidy go mod

* Updates from code review

* Updates inspector to take flag arguments for address

* Format list-buckets output more prettier

* Wiring up PING in kad inspector tools
2018-11-27 15:50:52 -07:00
Dylan Lott
fd42795456
updates dbx/gen.go to name generated package correctly (#723) 2018-11-27 13:52:19 -07:00
Maximillian von Briesen
7cf16503b8
Add statdb to overlay cache (node selection) (#711)
* make overlay node rep consistent with statdb node rep

* add statdb to testplanet

* add statdb to overlay cache

* tests
2018-11-27 12:46:12 -05:00
Michal Niewrzal
376bd74bed
Stops logging repairer errors for empty queue (#717) 2018-11-27 16:57:51 +01:00
Yaroslav Vorobiov
a0d84ccd67
Satellite user delete (#707)
* auth from context

* user delete api
2018-11-27 16:20:58 +02:00
Yehor Butko
784f3a887d
V3-760 Create logic for New project popup (#714)
* V3-760 Create logic for New project popup

* removing path to static files from gateway

* added missing copyrights
2018-11-27 13:14:10 +00:00
Kaloyan Raev
ea9bf38d85
eestream avoids waiting for pending readers if lots of errors (#709) 2018-11-27 10:17:57 +02:00
Bill Thorp
b8b0d75cc6
made accounting use migrate (#688)
* made accounting use migrate
2018-11-26 16:49:55 -05:00
Michal Niewrzal
8dd669b37f
Bandwidth allocations need method designation (GET/PUT) (#708)
* Bandwidth allocations need method designation (GET/PUT)

* add method comment

* goimports
2018-11-26 19:21:44 +01:00
Maximillian von Briesen
5014a785a0
Node selection 0/4: Update statdb auth/functionality (#698)
* remove api key from statdb server reqs; add statdb UpdateUptime and UpdateAuditSuccess to server

* update api key authentication in statdb server

* add todos for future statdb updates

* add UpdateUptime and UpdateAuditSuccess to statdb server

* fix apikey stuff in config.go and statdb_test.go

* fix tests

* update sdbclient.NewClient call in audit package

* fix UpdateUptime and UpdateAuditSuccess in sdbclient

* set api key from statdb/config.go

* change package for statdb tests

* linter fixes

* remove todo comments

* fix sdbclient err checking

* move validate auth functionality to auth package

* update description for statdb api key

* remove import
2018-11-26 12:08:29 -05:00
Yehor Butko
1f1f9f5875
Satellitedb company-user relation changed to 1-to-1 (#706)
* Satellitedb company-user relation changed to 1-to-1

* fixing comments
2018-11-26 14:57:55 +00:00
Yehor Butko
a998665491
Satellite projects api (#703)
* V3-645 Create Project API

* fixing linter

* update project refactored
2018-11-26 10:47:23 +00:00
Kaloyan Raev
67db26b36e
Metainfo stream download (#697) 2018-11-26 09:39:05 +02:00
Maximillian von Briesen
b884ee4626
Update protos to fit style guide (easy review) (#702)
* fix overlay.proto and piecestore.proto to meet style guide

* update code to be consistent with regenerated protos
2018-11-23 21:46:53 -05:00
Yaroslav Vorobiov
bdebe36ae0
updated login api (#699) 2018-11-22 12:38:58 +02:00
Maximillian von Briesen
06cf1c71ed
Statdb inspection tool (#626)
* begin adding stadb inspection tool

* add command for creating statdb nodes from csv

* add getCSVStats and create helper func for getting sdb client

* fix createcsvstats, add createstats

* fix linter errors
2018-11-21 17:01:09 -05:00
Dylan Lott
0d05cb26bf
Kademlia Inspector CLI (#657)
* initial commit of inspector gadget wireup

* change name of comman dline tool, setup grpc server

* Get inspector cli working with grpc client

* Wired up CountNodes command

* WIP getting buckets response working

* Added GetBucket command

* WIP working on get buckets command

* WIP working on bucket list

* Still WIP

* WIP getting bucket counts to work

* Some clean up of unnecessary changes

* List Buckets and Get Bucket are working

* Removing logs, getting ready for review

* initial commit of inspector gadget wireup

* change name of comman dline tool, setup grpc server

* Get inspector cli working with grpc client

* Wired up CountNodes command

* WIP getting buckets response working

* Added GetBucket command

* WIP working on get buckets command

* WIP working on bucket list

* Still WIP

* WIP getting bucket counts to work

* Some clean up of unnecessary changes

* List Buckets and Get Bucket are working

* Removing logs, getting ready for review

* Fix error return

* Trying to get tests passing

* Adds method on dht mock for tests

* Add dbx files back

* Fix package import error in dbx file

* Adds copyrights to pass linter

* tidy go mod

* Updates from code review

* Updates inspector to take flag arguments for address

* Format list-buckets output more prettier
2018-11-21 10:31:27 -07:00
Yaroslav Vorobiov
c829835dc1
updated createuser api (#687) 2018-11-21 17:51:43 +02:00
Michal Niewrzal
f72832ee69
Add metadata to pb.Node (email and wallet) (#671)
* Add metadata to pb.Node (email and wallet)

* goimports

* reorganize config

* fix typo

* remove 'foo' id

* add basic test
2018-11-21 16:07:18 +01:00
Kaloyan Raev
aec2c41911
Metainfo remote segment test (#696) 2018-11-21 16:35:53 +02:00
Kaloyan Raev
8c56198090
Use designated error classes for common bucket and object errors (#691) 2018-11-21 13:17:28 +02:00
Bryan White
dabd87cb36
swap golang/protobuf out for gogo/protobuf (#692)
* swap golang/protobuf out for gogo/protobuf

* revert go.mod and go.sum changes

* linter fixes
2018-11-20 19:29:07 +01:00
Bryan White
54ccb460a9
rename size fields in protobuf messages for gogo compatibility (#690)
* rename `size` fields in protobuf messages for gogo compatibility

* linter fixes
2018-11-20 18:09:35 +01:00
Dennis Coyle
e7e0d1daaa
Coyle/disable mock (#627)
* disabled mock overlay from captplanet
2018-11-20 11:54:52 -05:00
Jennifer Li Johnson
93c5f385a8
Enable checker in captplanet and staging (#643)
* enable checker

* add option to use mock overlay in checker

* adds logs to checker

* appease linter
2018-11-20 10:54:22 -05:00
Yehor Butko
4734a7447b
V3-805 Extend satellitedb with Project members entity (#683)
* V3-805 Extend satellitedb with Project members entity

* fixing linter

* small refactoring

* tests refactoring
2018-11-20 14:50:47 +00:00
Egon Elbre
76af9f5171
Add overlay server to testplanet (#677) 2018-11-19 22:39:25 +02:00
Egon Elbre
832317b0ee
Remove unused workers (#640) 2018-11-19 17:07:24 +02:00
Egon Elbre
d07433c150
Use testplanet to test Refresh (#678) 2018-11-19 16:40:01 +02:00
Kaloyan Raev
bb1cf151e3
Metainfo handle inline segments in ReadOnlyStream (#674) 2018-11-19 13:08:28 +02:00
Egon Elbre
6fe16e48ba
Simplify and fix tests (#660) 2018-11-16 18:31:14 +02:00
Yaroslav Vorobiov
3e99861e04
operationName and variables fields added to graphql json parsing (#668) 2018-11-16 16:34:35 +02:00
Kaloyan Raev
f730ce451a
Telemetry in metainfo implementation (#670) 2018-11-16 15:59:27 +02:00
aligeti
7958994ae2
Diagnostic tool to inspect repair queue (#656)
* initial repair queue diag tool development

* fixes linter warnings

* code review updates
2018-11-16 08:31:33 -05:00
Alexander Leitner
5e9377524f
Refactor basystem (#641)
* Signature verification

* Clean up agreement sender to have less errors

* overlay address in captnplanet

* Refactor bandwidth.proto to not use streams

* Make sure the send worked

* Handle connection to satellite

* Save renter public key inside of renter bandwidth allocations

* Default diag to sqlite. Make configurable

* Separate bw server and dbm; regenerate dbx files

* Make sure test uses protobufs

* Demonstrate creating bandwidth allocations
2018-11-15 14:06:09 -05:00
Egon Elbre
c4b90f84dd
Use better defaults and naming for postgres database (#659) 2018-11-15 20:36:57 +02:00
Kaloyan Raev
0357e61bbd
metainfo objects tests (#662) 2018-11-15 17:31:33 +02:00
Yaroslav Vorobiov
92b1bceca7
added copyright to files in satelliteauth pkg (#663) 2018-11-15 14:25:04 +02:00
Yaroslav Vorobiov
3f2ce11315
added copyright to every file in satellite pkg (#661) 2018-11-15 14:00:08 +02:00
Egon Elbre
21a102a05f
testplanet: add logger (#649) 2018-11-15 10:57:47 +02:00
Maximillian von Briesen
2c36dcc2c8
Update statdb (#650)
* add sdbclient.UpdateUptime; update args for sdbclient.CreateEntryIfNotExists

* add auditcount to node stats; restructure statdb.CreateEntryIfNotExists

* add noop mock sdbclient

* add the ability to create a node in statdb without "default" stats

* update statdb.CreateEntryIfNotExists

* take fewer args for sdbclient.CreateWithStats/FindValidNodes

* add sdbclient.UpdateAuditSuccess

* update sdbclient.Update so that all fields are updated when called (reduce args)

* update error checking in statdb.Create
2018-11-14 19:03:19 -05:00
Bill Thorp
d3c347a0ac
Verify overlay cache is accurately and fully tested (#540)
* added intro node id, moved port range to 1024, listen in go routines
2018-11-14 16:30:07 -05:00
Yaroslav Vorobiov
f28af4cdfb
added creation date to graphql user and removed redundant check from graphql api handler (#654) 2018-11-14 17:13:13 +02:00
Yehor Butko
8990fea63c
Satellitedb refactoring (#647) 2018-11-14 12:45:49 +00:00
Yaroslav Vorobiov
c442205b3a
account api: infrastructure, login, register, getUser (#611) 2018-11-14 12:50:15 +02:00
Egon Elbre
17519a7532 metainfo: Fix listing paths (#645) 2018-11-14 12:40:53 +02:00
Kaloyan Raev
5cdb03515f
Configure path encryption in metainfo API (#644) 2018-11-14 11:26:18 +02:00
Jennifer Li Johnson
377832c705
Create Tally and Rollup pkgs in accounting (#642)
* creates separate tally and rollup packages and writes skeleton for rollup

* TODO add rollupDB and rawDB to rollup struct

* TODO add rawDB to tally struct
2018-11-13 20:22:18 -05:00
Alexander Leitner
50d657af11
Cleanup agreement sender (#630)
* Signature verification

* code review comments & lint warnings

* Clean up agreement sender to have less errors

* overlay address in captnplanet
2018-11-13 10:37:49 -05:00
Kaloyan Raev
514ca50611
Configurable encryption type for paths (#637) 2018-11-13 14:21:52 +02:00
Yehor Butko
75213fa633
V3-663 Extend database with Project entity (#622)
* V3-663 Extend database with Project entity

* fixing linter
2018-11-13 10:27:42 +02:00
aligeti
6576d31b29 Signature verification (#604)
* Signature verification

* code review comments & lint warnings
2018-11-12 16:59:30 -05:00
Bill Thorp
ad39201ba9
Issues in overlay rebuild allowed to continue (#625)
* Issues in overlay rebuild allowed to continue
2018-11-12 14:53:20 -05:00