Commit Graph

61 Commits

Author SHA1 Message Date
Egon Elbre
2680bae88c private/testplanet: remove dependency to uplink
Remove direct dependency on uplink.RSConfig, this simplifies
moving the config file without introducing weird dependencies.

Change-Id: I7fd2a145401e0205d7047631df9d2810241efeec
2020-01-02 09:40:46 +00:00
Egon Elbre
6615ecc9b6 common: separate repository
Change-Id: Ibb89c42060450e3839481a7e495bbe3ad940610a
2019-12-27 14:11:15 +02:00
Yingrong Zhao
7af42e3c10
satellite/metainfo, satellite/repair, uplink/eestream: add metric for download failed due to not enough pieces available (#3665) 2019-12-04 16:24:36 -05:00
Egon Elbre
ee6c1cac8a
private: rename internal to private (#3573) 2019-11-14 21:46:15 +02:00
Ethan Adams
3e0d12354a
storagenode/gracefulexit: Implement storage node graceful exit worker - part 1 (#3322) 2019-10-22 16:42:21 -04:00
Egon Elbre
3c438f31bd
satellite/satellitedb: remove sqlite support (#3296) 2019-10-19 00:27:57 +03:00
Ethan Adams
a1275746b4
satellite/gracefulexit: Implement the 'process' endpoint on the satellite (#3223) 2019-10-11 17:18:05 -04:00
Jeff Wendling
098cbc9c67 all: use pkg/rpc instead of pkg/transport
all of the packages and tests work with both grpc and
drpc. we'll probably need to do some jenkins pipelines
to run the tests with drpc as well.

most of the changes are really due to a bit of cleanup
of the pkg/transport.Client api into an rpc.Dialer in
the spirit of a net.Dialer. now that we don't need
observers, we can pass around stateless configuration
to everything rather than stateful things that issue
observations. it also adds a DialAddressID for the
case where we don't have a pb.Node, but we do have an
address and want to assert some ID. this happened
pretty frequently, and now there's no more weird
contortions creating custom tls options, etc.

a lot of the other changes are being consistent/using
the abstractions in the rpc package to do rpc style
things like finding peer information, or checking
status codes.

Change-Id: Ief62875e21d80a21b3c56a5a37f45887679f9412
2019-09-25 15:37:06 -06:00
Jeff Wendling
0dcbd3dc08 bootstrap/satellite/certificate/storagenode: register drpc services
Change-Id: Id29f14b76a8c9cb2be31001b9a7a4356a4bda183
2019-09-12 15:09:46 -06:00
Natalie Villasana
aa3567187e
satellite/audit: worker now verifies and reverifies (#2965) 2019-09-11 18:37:01 -04:00
Egon Elbre
a801fab66a
all: add archview annotations (#2964) 2019-09-10 16:24:16 +03:00
ethanadams
4ede12a2ab
satellite/orders: Fix for V3-2529: Release v0.19.0 storage nodes can't submit orders, duplicate key value violates unique constraint (#2900)
* V3-2529: Add DB savepoint to fix issue with postgres. Add test force a rejected order

Co-Authored-By: Ivan Fraixedes <ivan@fraixed.es>

* Update satellite/satellitedb/orders.go
2019-08-29 11:14:10 -04:00
Cameron
599324c364
satellite/dbcleanup: delete expired serials from satellite (#2867)
Creates a new chore, dbcleanup, which can be used for routine deletion of items from the satellite database and adds functionality for deletion of expired serial numbers
2019-08-27 13:12:38 -04:00
Cameron
3d9441999a
storagenode/orders: add archive cleanup to orders service (#2821)
This PR introduces functionality for routine deletion of archived orders.

The user may specify an interval at which to run archive cleanup and a TTL for archived items. During each cleanup, all items that have reached the TTL are deleted

This archive cleanup job is combined with the order sender into a new combined orders service
2019-08-22 10:33:14 -04:00
Egon Elbre
2d69d47655
all: fix Error.New formatting (#2840) 2019-08-21 19:30:29 +03:00
ethanadams
1a69ec8318
satellite/orders: document protocol and fix typos (#2813)
* Addressing comments from PR 2762
* Rebuild of orders.pb.go after comments added to proto file
* run update-satellite-config-lock for spelling fix.
2019-08-19 09:36:11 -04:00
Ivan Fraixedes
e47b8ed131
storagenode: No FATAL error when unsent orders aren't found (#2801)
* pkg/process: Fatal show complete error information
  Change the general process execution function to not using the sugared
  logger for outputting the full error information.
  Delete some unreachable code because Zap logger Fatal method calls exit
  1 internally.
* storagenode/storagenodedb: Add info to error
  Add more information to an error returned due to some data
  inconsistency.
* storagenode/orders: Don't use sugared logger
  Don't use sugar logger and provide better contextualized error messages
  in settle method.
* storagenode/orders: Add some log fields to error msgs
  Add some relevant log fields to some logged errors of the sender settle
  method.
* satellite/orders: Remove always nil error from debug
  Remove an error which as logged in debug level which was always nil and
  makes the logic that used this variable clear.
* storagenode/orders: Don't return error Archiving unsent
  Don't stop the process which archive unsent orders if some of them
  aren't found the DB because it cause the Storage Node to stop with a
  fatal error.
2019-08-16 16:53:22 +02:00
ethanadams
8df683a265
Update satellite settlement endpoint to batch order processing into transactions. (#2762)
Update satellite settlement endpoint to batch order processing into transactions
2019-08-15 15:05:43 -04:00
Egon Elbre
48211daa9d
uplink/piecestore: handle Download errors better (#2771) 2019-08-14 12:02:58 +03:00
aligeti
32f95a14fd
satellite/certdb: remove certdb that was used to store uplink certificates (#2760)
* satellitedb/certDB: refactors of the node certificate storage DB table

The existing implementation doesnt allow to store the complete certificate chain of uplinkIDs or storagenodeIDs, so the current table is dropped and new table will be added which addresses the storage and retrieval of certificates

pkg/identity: fixes spelling mistakes that I missed on PR#2754

Fixes V3-1992/V3-2388
2019-08-12 10:41:34 -04:00
Egon Elbre
c8edeb0257
satellite/overlay: rename overlay.Cache to overlay.Service (#2717) 2019-08-06 19:35:59 +03:00
Egon Elbre
5d0816430f
rename all the things (#2531)
* rename pkg/linksharing to linksharing
* rename pkg/httpserver to linksharing/httpserver
* rename pkg/eestream to uplink/eestream
* rename pkg/stream to uplink/stream
* rename pkg/metainfo/kvmetainfo to uplink/metainfo/kvmetainfo
* rename pkg/auth/signing to pkg/signing
* rename pkg/storage to uplink/storage
* rename pkg/accounting to satellite/accounting
* rename pkg/audit to satellite/audit
* rename pkg/certdb to satellite/certdb
* rename pkg/discovery to satellite/discovery
* rename pkg/overlay to satellite/overlay
* rename pkg/datarepair to satellite/repair
2019-07-28 08:55:36 +03:00
Ivan Fraixedes
f420b29d35
[V3-1927] Repairer uploads to max threshold instead of success… (#2423)
* pkg/datarepair: Add test to check num upload pieces
  Add a new test for ensuring the number of pieces that the repair process
  upload when a segment is injured.
* satellite/orders: Don't create "put order limits" over total
  Repair must not create "put order limits" more than the total count.
* pkg/datarepair: Update upload repair pieces test
  Update the test which checks the number of pieces which are uploaded
  during a repair for using the same excess over the success threshold
  value than the implementation.
* satellites/orders: Limit repair put order for not being total
  Limit the number of put orders to be used by repair for only uploading
  pieces to a % excess over the successful threshold.
* pkg/datarepair: Change DataRepair test to pass again
  Make some changes in the DataRepair test to make pass again after the
  repair upload repaired pieces only until a % excess over success
  threshold.
  Also update the steps description of the DataRepair test after it has been
  changed, to match on what's now, besides to leave it more generic for
  avoiding having to update it on minimal future refactorings.
* satellite: Make repair excess optimal threshold configurable
  Add a new configuration parameter to the satellite for being able to
  configure the percentage excess over the optimal threshold, used for
  determining how many pieces should be repaired/uploaded, rather than
  having the value hard coded.
* repairer: Add configurable param to segments/repairer
  Add a new parameters to the segment/repairer to calculate the maximum
  number of excess nodes, based on the optimal threshold, that repaired
  pieces can be uploaded.
  This new parameter has been added for not returning more nodes than the
  number of upload orders for data repair satellite service calculate for
  repairing pieces.
* pkg/storage/ec: Update log message in clien.Repair
* satellite: Update configuration lock file
2019-07-12 00:44:47 +02:00
Egon Elbre
d52f764e54
protocol: implement new piece signing and verification (#2525) 2019-07-11 16:51:40 -04:00
Bill Thorp
0e463dccfd
7 day validity window for order limits (#2520)
* 7 day limit
2019-07-10 17:17:00 -04:00
Alexander Leitner
1c5db71faf
Change protobuf expirations to use time.Time (#2509)
* Change protobuf expirations to use time.Time instead of timestamp.Timestamp
2019-07-09 17:54:00 -04:00
Michal Niewrzal
2ee5bada2c
Add pieceNum to PieceID derivation function (#2193) 2019-07-03 18:53:15 +02:00
Alexander Leitner
6d55bbdb57
OrderLimit creation date time limit (#2412)
* Limit by order creation
2019-07-02 12:06:12 -04:00
Egon Elbre
385c046723
pkg/pb: rename Order2 to Order, OrderLimit2 to OrderLimit (#2406) 2019-07-01 18:54:11 +03:00
Egon Elbre
b6ad3e9c9f
internal/testrand: new package for random data (#2282) 2019-06-26 13:38:51 +03:00
Egon Elbre
caa2fcf62b satellite/orders: don't panic (#2331) 2019-06-26 09:26:33 +02:00
Jess G
e5c48fab74
fix ordersDB methods to take correct args (#2314)
* fix orderdDB methods to take correct args

* update tally to save projectID in correct format

* update var names in splitBucket test

* changes per CR comments
2019-06-25 08:58:42 -07:00
Kaloyan Raev
7bf5a09638 Use ErrNodeOffline when returning error for offline node (#2312) 2019-06-25 00:15:38 +02:00
Fadila
8226024ca8
Do not use disqualified nodes when asking for get order limits (#2303)
Add checks about disqualification when creating order limits
2019-06-24 16:46:10 +02:00
littleskunk
a5baebfa65 reject invalid orders (#2262) 2019-06-21 13:44:14 +02:00
JT Olio
568b000e9b satellite: make order expiration configurable (#2251) 2019-06-21 13:38:40 +03:00
JT Olio
76b54458e9 satellite: send external address in order limits (#2278) 2019-06-21 12:19:52 +03:00
Natalie Villasana
9386187fe6
add disqualification and new reputation system into overlay cache (#2227) 2019-06-20 09:56:04 -04:00
Kaloyan Raev
8e29ef8a6b Use zap.Stringer instead of zap.String (#2223) 2019-06-18 01:37:43 +02:00
JT Olio
14486b1885 satellite/orders: TODOs about transactions (#2134) 2019-06-12 17:00:29 +02:00
Egon Elbre
61c8c3fb49
satellite/orders: batch update storage node allocations (#2146) 2019-06-10 17:58:28 +03:00
JT Olio
f360097b24 satellite/orders: monitor order settlement more (#2128) 2019-06-05 17:43:41 +02:00
JT Olio
ccb158c99b
pkg/auth: add monkit task to missing places (#2123)
What: add monkit.Task to a bunch of functions that are missing it

Why: this will significantly help our instrumentation, data collection, and tracing about what's going on in the network
2019-06-05 07:47:01 -06:00
JT Olio
29d16b4d68 satellite: add monkit task to missing places (#2108) 2019-06-04 13:55:37 +02:00
Natalie Villasana
aa6ff17b70 add Reverify to auditing (#2041)
Co-authored-by: Maximillian von Briesen <mobyvb@gmail.com>
Co-authored-by: Kaloyan Raev <kaloyan@storj.io>
2019-05-27 14:13:47 +03:00
littleskunk
73a039afa7
named log (#2034) 2019-05-24 00:47:05 +02:00
Bill Thorp
cd4a3e06d8
wired up IsHealthy to config (#1820)
* wired up IsHealthy to config
2019-04-23 18:45:50 -04:00
Egon Elbre
bdd0d778eb
interface tests belong to the interface, not the implementation (#1794) 2019-04-23 11:47:16 +03:00
Kaloyan Raev
8fc5fe1d6f
Refactor pb.Node protobuf (#1785) 2019-04-22 12:07:50 +03:00
Cameron
32192aca10
convert times to UTC before entering as utimestamp (#1708) 2019-04-09 15:12:58 -04:00