JakeHillion/storj
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
1,781 Commits 5 Branches 0 Tags 94 MiB
ddcf4fc2a3
Commit Graph

110 Commits

Author SHA1 Message Date
Egon Elbre
05d148aeb5
Storage node and upload/download protocol refactor (#1422) 
refactor storage node server
refactor upload and download protocol
 2019-03-18 12:55:06 +02:00
Natalie Villasana
036b2f3b49
allocates 1kb for audit bandwidth agreements instead of 256kb (#1404) 2019-03-05 18:58:25 -05:00
Bill Thorp
373b301736
BWA aliases (#1333) 
aliased RBAs and PBAs
 2019-02-22 16:17:35 -05:00
Natalie Villasana
c3d3f41d30 removes some SignedMessage use (#1258) 
Removes most instances of pb.SignedMessage (there's more to take out but they shouldn't hurt anyone as is).

There used to be places in psserver where a PieceID was hmac'd with the SatelliteID, which was gotten from a SignedMessage. This PR makes it so some functions access the SatelliteID from the Payer Bandwidth Allocation instead.

This requires passing a SatelliteID into psserver functions where they weren't before, so the following proto messages have been changed:

 * PieceId - satellite_id field added
   This is so the psserver.Piece function has access to the SatelliteID when it needs to get the namespaced pieceID.
   This proto message should probably be renamed to PieceRequest, or a new PieceRequest message should be created so this isn't misnamed.

 * PieceDelete - satellite_id field added
   This is so the psserver.Delete function has access to the SatelliteID when receiving a request to Delete.
 2019-02-19 23:36:08 -06:00
Egon Elbre
9c1e299f3c
Ensure everyone sees everyone else (#1275) 2019-02-08 11:25:13 +02:00
Egon Elbre
94ee200e35
Ensure cursor tests run against postgres (#1228) 2019-02-05 17:45:09 +02:00
Egon Elbre
5a63c00442
Fix issues with blocking during startup (#1212) 2019-02-01 19:28:40 +02:00
Egon Elbre
7a85c5c1c6
Fix audit node ids (#1207) 2019-02-01 16:48:57 +02:00
Egon Elbre
d5346982c2
Delete provider package (#1177) 2019-01-30 22:47:21 +02:00
Egon Elbre
6132ce86b7
Remove utils.LogClose (#1169) 2019-01-29 22:42:27 +02:00
Bill Thorp
60946c2024
make bandwidth agreements sensible: without []byte's (#1152) 
removed []byte's from bandwidth agreement protocol buffers
 2019-01-28 14:45:25 -05:00
Jennifer Li Johnson
856b98997c
updates copyright 2018 to 2019 (#1133) 2019-01-24 15:15:10 -05:00
Egon Elbre
5de7f8af7f
Satellite Peer (#1119) 2019-01-23 21:58:44 +02:00
Egon Elbre
ad49f4b994
Enable linters (#1105) 2019-01-22 17:09:02 +02:00
Michal Niewrzal
eb2d858ffc Use pointerdb.Service instead of pointerdb.Server (#1099) 2019-01-19 13:58:53 -05:00
Egon Elbre
78dc02b758 Satellite Peer (#1034) 
* add satellite peer

* Add overlay

* reorganize kademlia

* add RunRefresh

* add refresh to storagenode.Peer

* add discovery

* add agreements and metainfo

* rename

* add datarepair checker

* add repair

* add todo notes for audit

* add testing interface

* add into testplanet

* fixes

* fix compilation errors

* fix compilation errors

* make testplanet run

* remove audit refrences

* ensure that audit tests run

* dev

* checker tests compilable

* fix discovery

* fix compilation

* fix

* fix

* dev

* fix

* disable auth

* fixes

* revert go.mod/sum

* fix linter errors

* fix

* fix copyright

* Add address param for SN dashboard (#1076)

* Rename storj-sdk to storj-sim (#1078)

* Storagenode logs and config improvements  (#1075)

* Add more info to SN logs

* remove config-dir from user config

* add output where config was stored

* add message for successful connection

* fix linter

* remove storage.path from user config

* resolve config path

* move success  message to info

* log improvements

* Remove captplanet (#1070)

* pkg/server: include production cert (#1082)

Change-Id: Ie8e6fe78550be83c3bd797db7a1e58d37c684792

* Generate Payments Report (#1079)

* memory.Size: autoformat sizes based on value entropy (#1081)

* Jj/bytes (#1085)

* run tally and rollup

* sets dev default tally and rollup intervals

* nonessential storj-sim edits (#1086)

* Closing context doesn't stop storage node (#1084)

* Print when cancelled

* Close properly

* Don't log nil

* Don't print error when closing dashboard

* Fix panic in inspector if ping fails (#1088)

* Consolidate identity management to identity cli commands (#1083)

* Consolidate identity management:

Move identity cretaion/signing out of storagenode setup command.

* fixes

* linters

* Consolidate identity management:

Move identity cretaion/signing out of storagenode setup command.

* fixes

* sava backups before saving signed certs

* add "-prebuilt-test-cmds" test flag

* linters

* prepare cli tests for travis

* linter fixes

* more fixes

* linter gods

* sp/sdk/sim

* remove ca.difficulty

* remove unused difficulty

* return setup to its rightful place

* wip travis

* Revert "wip travis"

This reverts commit 56834849dcf066d3cc0a4f139033fc3f6d7188ca.

* typo in travis.yaml

* remove tests

* remove more

* make it only create one identity at a time for consistency

* add config-dir for consitency

* add identity creation to storj-sim

* add flags

* simplify

* fix nolint and compile

* prevent overwrite and pass difficulty, concurrency, and parent creds

* goimports
 2019-01-18 08:54:08 -05:00
Egon Elbre
f8906ce000
Change overlay cache database interface (#1047) 2019-01-15 11:08:45 -05:00
Michal Niewrzal
e13f7c7a7d
Fix unstarted pointerdb responsibility (#1033) 2019-01-11 13:45:11 +01:00
Maximillian von Briesen
058ef585aa
do not try to dial nil node from audit verifier (#1027) 
* do not try to dial nil node from audit verifier

* add todo
 2019-01-10 15:13:40 -05:00
Maximillian von Briesen
0642592420 Replace pdb client with server in audit service (#1016) 
* add logger to audit service

* use pointerdb instead of pdbclient in audit

* linter fixes
 2019-01-10 11:35:18 -05:00
Bill Thorp
fb4a11ebb0
Differentiate GET and PUT operations in the tally service (#965) 
* draft of new bandwidth agreement types
* updated storagenode report for new types
* use correct pba types
 2019-01-10 06:41:57 -05:00
Michal Niewrzal
bacc1b13b4
Verifier should use payer bandwidth alloc from satellite (#960) 
* Verifier should use payer bandwidth alloc from satellite

* unit test added

* fix typo

* review comments applied

* fix renamed field
 2019-01-06 19:51:01 +01:00
Jennifer Li Johnson
a2fa5c4c5a Proper NodeType Handling (#873) 
* adds enums to nodetype

* updating nodetype todos

* ran pb updates

* reorder nodetypes

* adding checks

* wip

* wip

* wip

* bug in test-captplanet

* wip

* add values to storagenode, satellite, captplanet binaries

* Cleanup

* more cleanup

* wip

* lint

* lint

* wip

* fixes bug

* regenerate protos

Change-Id: Id270212e8c7479e52641058042cf23b5317ab773

* limit node type changes to kademlia

Change-Id: I9c1a6cc4a79e05086627f0fdeb5028c62ce754f4

* dpanic

Change-Id: Id952a2ad13c807ebaea0ec0a875405e267d81c3e

* review comments

Change-Id: I7f9b77ef22779dd012fd490375b136014f51f834
 2019-01-02 11:47:34 -07:00
Egon Elbre
89db0fe9f5
storj-sdk base code (#799) 2019-01-02 20:07:49 +02:00
Bryan White
249244536a
CSR Service (part 2): cert signing rpc (#950) 
* CSR Service:

+ implement certificate sign rpc method
+ implement certificate signer client/server
+ refactor `AuthorizationDB#Create`
+ refactor `NewTestIdentity`
+ add `AuthorizationDB#Claim`
+ add `Token#Equal`
+ fix `Authorizations#Marshal` when marshaling identities and certificates
+ tweak `Authorization#String` format
+ cert debugging improvements (jsondiff)
+ receive context arg in `NewTestIdentity`
+ misc. fixes
 2019-01-02 12:39:17 -05:00
Egon Elbre
d9b9ae6ffa
Cleanup overlay methods and names. (#914) 2018-12-20 15:57:54 +02:00
Maximillian von Briesen
f18c38628e
Refactor statdb (#887) 
Update statdb args/return values to minimize structs
    Simplify statdb.Update() to update all stats instead of an arbitrary subset determined by flags
    Remove CreateIfNotExists logic from statdb.Update()
    Simplify audit code structure
 2018-12-19 13:44:03 -05:00
aligeti
5e1b02ca8b
Statdb master db v3 848 (#830) 
* intial changes to migrate statdb to masterdb framework

* statdb refactor compiles

* added TestCreateDoesNotExist testcase

* Initial port of statdb to masterdb framework working

* refactored statdb proto def to pkg/statdb

* removed statdb/proto folder

* moved pb.Node to storj.NodeID

* CreateEntryIfNotExistsRequest moved pd.Node to storj.NodeID

* moved the fields from pb.Node to statdb.UpdateRequest

ported TestUpdateExists, TestUpdateUptimeExists, TestUpdateAuditSuccessExists TestUpdateBatchExists
 2018-12-14 15:17:30 -05:00
Maximillian von Briesen
2ab15196d9
Get rid of statdb grpc stuff (#761) 
* remove sdbclient

* do not register statdb grpc server/remove apikey from statdb

* start audit service after statdb in captplanet
 2018-12-04 13:47:58 -05:00
Egon Elbre
1e4556f88a
Fix import groupings (#739) 2018-11-30 15:40:13 +02:00
Bryan White
2a0c4e60d2
preparing for use of customtype gogo extension with NodeID type (#693) 
* preparing for use of `customtype` gogo extension with `NodeID` type

* review changes

* preparing for use of `customtype` gogo extension with `NodeID` type

* review changes

* wip

* tests passing

* wip fixing tests

* more wip test fixing

* remove NodeIDList from proto files

* linter fixes

* linter fixes

* linter/review fixes

* more freaking linter fixes

* omg just kill me - linterrrrrrrr

* travis linter, i will muder you and your family in your sleep

* goimports everything - burn in hell travis

* goimports update

* go mod tidy
 2018-11-29 19:39:27 +01:00
Egon Elbre
0ae05cf834
Ensure that we only choose storage nodes (#732) 2018-11-29 16:57:00 +02:00
Kaloyan Raev
f46d43b0d0
Fix panic in Audit service (#713) 2018-11-28 09:33:17 +02:00
Maximillian von Briesen
7cf16503b8
Add statdb to overlay cache (node selection) (#711) 
* make overlay node rep consistent with statdb node rep

* add statdb to testplanet

* add statdb to overlay cache

* tests
 2018-11-27 12:46:12 -05:00
Michal Niewrzal
8dd669b37f
Bandwidth allocations need method designation (GET/PUT) (#708) 
* Bandwidth allocations need method designation (GET/PUT)

* add method comment

* goimports
 2018-11-26 19:21:44 +01:00
Maximillian von Briesen
5014a785a0
Node selection 0/4: Update statdb auth/functionality (#698) 
* remove api key from statdb server reqs; add statdb UpdateUptime and UpdateAuditSuccess to server

* update api key authentication in statdb server

* add todos for future statdb updates

* add UpdateUptime and UpdateAuditSuccess to statdb server

* fix apikey stuff in config.go and statdb_test.go

* fix tests

* update sdbclient.NewClient call in audit package

* fix UpdateUptime and UpdateAuditSuccess in sdbclient

* set api key from statdb/config.go

* change package for statdb tests

* linter fixes

* remove todo comments

* fix sdbclient err checking

* move validate auth functionality to auth package

* update description for statdb api key

* remove import
 2018-11-26 12:08:29 -05:00
Bryan White
54ccb460a9
rename size fields in protobuf messages for gogo compatibility (#690) 
* rename `size` fields in protobuf messages for gogo compatibility

* linter fixes
 2018-11-20 18:09:35 +01:00
Egon Elbre
6fe16e48ba
Simplify and fix tests (#660) 2018-11-16 18:31:14 +02:00
Natalie Villasana
f11f4653e4
removes audit erroring when no stripe is given (#603) 2018-11-08 22:36:44 +00:00
Alexander Leitner
320c93368a
Add creation time to Payer Bandwidth Allocation (#581) 
* Add creation time to Payer Bandwidth Allocation

* goimports
 2018-11-08 11:04:52 -05:00
Maximillian von Briesen
66ee332446
Fix audit stuff breaking test-captplanet (#594) 
* fix audit stripe selector to work if last segment is smaller than stripe size

* fix audit bug related to indexing an incomplete list of nodes returned by overlay
 2018-11-07 16:23:05 -05:00
Egon Elbre
d5b3d5f395
Don't panic in case of a 0-size pointer (#590) 2018-11-07 18:51:36 +02:00
Natalie Villasana
3b572264ca
adds audit job (#521) 2018-11-07 01:16:43 +00:00
Bryan White
ee62e2a9d8
Use transport client and cleanup all the clients (#574) 
* wip

* linter fixes

* linter fixes

* test fixes

* linter fixes

* fix merge + restructure piecestore packages

* review feedback

* linter fixes

* linter fixes

* remove unnecessary aliases to piecestore

* more merge fixing
 2018-11-06 18:49:17 +01:00
Michal Niewrzal
deb015970d
Use Nodes array from pdb.Get (#578) 
* Use Nodes array from pdb.Get

* fix problems with captplanet tests

* better comments
 2018-11-06 18:03:11 +01:00
Michal Niewrzal
1129ead6ef
Fix 'message not signed by the satellite' problem (#560) 
* Fix 'message not signed by the satellite' problem

* fixed build

* fix errors
 2018-11-05 16:12:19 +01:00
Bryan White
8b9711cb5e
better waitlist-gating (#557) 
* better waitlist-gating

(cherry picked from commit 490fe02b7c3558da18678dfb651c92ec9c4a75b5)

* fix broken test

* linter fixes

* linter fixes

* make extension verification optional

* add certifcate gating script for captplanet

* fixing tests

* linter fixes

* linter fixes?

* moar linter fixes

* Revert "moar linter fixes"

This reverts commit 8139ccbd73cbbead987b7667567844f50f7df2c8.

* just kill me

* refactor

* refactor tests

* liniter...

* cleanup
 2018-11-01 16:48:43 +01:00
Egon Elbre
2a8b681c4d
Run repairer and checker early (#565) 
* Run repairers, checker, auditors first time they run to detect potential setup problems.
* Fix error handling in audit.Service
 2018-11-01 16:03:45 +02:00
Alexander Leitner
7e4149138a
Piecestore proto change (#556) 
* add action and storage node id to piecestore.proto. renamed payer and renter to satellite_id and uplink_id
 2018-10-30 15:03:41 -04:00
Michal Niewrzal
0d03f2fbb5
Send bandwidth alloc from satellite to storage node (#538) 
* Send bandwidth alloc from satellite to storage node

* Remove unecessary nil checks

* set Renter field

* fix tests

* goimports

* Update README.md

* Update README.md

* Update README.md

* Update README.md (#550)

* Skip flaky TestPing (#552)
 2018-10-30 17:24:46 +01:00
Kaloyan Raev
99640225fd
Refactor Path type (#522) 
The old paths.Path type is now replaced with the new storj.Path.

storj.Path is simply an alias to the built-in string type. As such it can be used just as any string, which simplifies a lot working with paths. No more conversions paths.New and path.String().

As an alias storj.Path does not define any methods. However, any functions applying to strings (like those from the strings package) gracefully apply to storj.Path too. In addition we have a few more functions defined:

    storj.SplitPath
    storj.JoinPaths
    encryption.EncryptPath
    encryption.DecryptPath
    encryption.DerivePathKey
    encryption.DeriveContentKey

All code in master is migrated to the new storj.Path type.

The Path example is also updated and is good for reference: /pkg/encryption/examples_test.go

This PR also resolve a nonce misuse issue in path encryption: https://storjlabs.atlassian.net/browse/V3-545
 2018-10-25 23:28:16 +03:00
Michal Niewrzal
f9248c21d4
Satellite verification on storage node (#469) 
* Satellite verification on storage node

* fix formatting

* fix formatting

* rename SignatureAuth to SignedMessage

* fixes after review

* fix linter errors

* improve errors handling

* remove SignedMessageProvider

* fix liter errors

* params changed to authorization, signed message in audit, minor fixes

* fix formatting
 2018-10-17 13:40:11 +02:00
James Hagans
115d257bb1
Audit cron (#466) 2018-10-16 16:02:18 -04:00
Natalie Villasana
1687111cca
adds Reporter to the audit package to record failed audits (#467) 2018-10-16 13:40:34 -04:00
Michal Niewrzal
351dc523d6
Satellite signature generation (#453) 
* Satellite signature generation

* Add unit test

* remove unused var

* remove base64 encoding from signature generation
 2018-10-11 16:35:55 +02:00
Natalie Villasana
404fc0329f
remove todo from audit pkg (#457) 2018-10-10 15:07:14 -04:00
Natalie Villasana
14b67af327
reorganizes audit package into Service, Cursor, and Verifier (#450) 2018-10-10 14:25:46 -04:00
Egon Elbre
b546ed9510
Remove unneeded constants (#452) 2018-10-10 11:33:17 +03:00
Natalie Villasana
b04ea4639f moves all audit files to pkg/audit (#449) 2018-10-09 18:05:42 -04:00
Natalie Villasana
d36eb9c742
audits stripes (#380) 2018-10-09 17:10:37 -04:00