Commit Graph

310 Commits

Author SHA1 Message Date
Egon Elbre
afe05edff2 {storagenode,satellite}/gracefulexit: ensure workers finish their work
Fixes a data race caused by not waiting for workers to finish
before shutting down. Currently this ended up failing logging
because it was closed when test tried to write to it.

Change-Id: I074045cd83bbf49e658f51353aa7901e9a5d074b
2019-12-17 17:21:52 +02:00
Egon Elbre
7a36507a0a private/testcontext: ensure we call cleanup everywhere
Change-Id: Icb921144b651611d78f3736629430d05c3b8a7d3
2019-12-17 14:16:09 +00:00
littleskunk
08947e177d storagenode/garbagecollection: enable in production
Change-Id: I627b7a37ca4a85eb19936ca2c7ca907d7cc63f5b
2019-12-16 22:44:04 +00:00
Vitalii Shpital
53d9bc4530
storagenode/notifications: db created (#3707) 2019-12-16 19:59:01 +02:00
littleskunk
c2ea75208f
storagenode/orderdb: fix db lock
Change-Id: Id1add0ba7ae1b20bd98099bd4d3aff0fcfdd90c9
2019-12-15 23:41:22 +01:00
Andrew Harding
cb89496569 storagenode/trust: wire up list into pool
- also updated ping chore to pick up trust changes
- fixed small typo in blueprint
- fixed flags for storj-sim
- wired up changes to testplanet

Change-Id: I02982f3a63a1b4150b82a009ee126b25ed51917d
2019-12-13 20:32:50 +00:00
Andrew Harding
2867b6a466 storagenode/trust: list implementation
Change-Id: Ia886e84990efaf2c783f199741552a7a8ff41d4e
2019-12-12 17:15:47 +00:00
Jeff Wendling
fb8e78132d storagenodedb: reenable utccheck in tests
Change-Id: If7d64dd4ae58e4b656ff9122ae3195b2a5173cb3
2019-12-10 23:17:14 +00:00
Andrew Harding
5ed9373dba storagenode/trust: source entry cache
Implements a cache that can persist trust entries returned by sources

Change-Id: I72579e42e9f72d34a54b7510c9b665844f187314
2019-12-10 21:45:01 +00:00
Andrew Harding
715d97e3d8 storagenode/trust: rule and excluders
Change-Id: I84ed542e1ef3cfaa5cc3d3f631cdc295393bf978
2019-12-10 21:08:12 +00:00
Cameron Ayer
6fae361c31 replace planet.Start in tests with planet.Run
planet.Start starts a testplanet system, whereas planet.Run starts a testplanet
and runs a test against it with each DB backend (cockroach compat).

Change-Id: I39c9da26d9619ee69a2b718d24ab00271f9e9bc2
2019-12-10 16:55:54 +00:00
Andrew Harding
eb52ac623b storagenode/trust: source implementations
Change-Id: Ie36e79cc15257db88051f63e5b9463fd9d7b4736
2019-12-09 20:00:02 +00:00
Andrew Harding
7d0aadfeca storagenode/trust: satellite URL implementation
Satellite URL is a stricter form of the STORJ Node URL. It requires both
the ID and port specifier.

Change-Id: I7fd302064f864c1de8240a7915bf5263b898dfd1
2019-12-09 17:05:57 +00:00
littleskunk
9d1faeee58 storagenode/garbagecollection: increase MaxTimeSkew to be higher than satellite MaxCommitInterval
Change-Id: I86f8d0b44bea3aa005ff26d52588611c59df5e9a
2019-12-09 16:03:55 +00:00
Ethan Adams
9420fa9fc5 satellite/gracefulexit: Add graceful exit completed/failed receipt verification to satellite CLI (#3679) 2019-12-03 17:09:39 -05:00
Ivan Fraixedes
42c61138e8
storage: Improve doc comments delete methods (#3591)
Improve the documentation of several methods involved in the delete
operation to make clear their behavior without having to inspect their
logic.
2019-12-02 12:18:20 +01:00
Ivan Fraixedes
bf97ef06fc
storagenode: Add new endpoint to receive satellite requests for… (#3590)
* pkg/pg: Add new service function storage node

  Add a new service function to the storage node piece store for deleting
  pieces when satellites request them.

* storagenode/piecestore: Add endpoint to delete piece

  Add a new endpoint to receive from trusted satellites to delete a piece.

* private/testplanet: Fix storagenode mock

  Add to the storagenode mock the new endpoint method.

* proto.lock: Update it with the last protbuff changes

* storagenode/piecestore: Reuse test piece upload

  Extract the repeated logic from several tests functions for uploading a
  test piece to a test helper function.

* uplink/piecestore: Implement client side method

  Implement the client side method of the new piecestore RPC function.

* storagenode/piecestore: Add test DeletePiece endpoint

  Implement a test for the DeletePiece new endpoint method.
2019-11-26 18:47:19 +01:00
Yingrong Zhao
66f1a1680f
add completion receipt to exit-status cli command on storage node (#3650) 2019-11-26 12:32:26 -05:00
Isaac Hess
56f8fd2dd7
storagenode/pieces: Add EmptyTrash functionality (#3640)
* storagenode/pieces: Add EmptyTrash functionality

* storagenode/pieces: Fix err

* storagenode/pieces: Fix lint
2019-11-26 09:25:21 -07:00
Vitalii Shpital
038ac58600
web/storagenode: minimal allowed version view implemented (#3583) 2019-11-26 18:08:24 +02:00
littleskunk
8842b0c252 storagenode/gracefulexit: improve logging (#3633) 2019-11-21 21:10:02 -05:00
Rafael Antonio Ribeiro Gomes
2739771761
storagenode: add bandwidth metrics (#3623)
* storagenode: add bandwidth metrics

* remove unecessary metric
2019-11-21 16:51:40 -03:00
Isaac Hess
6aeddf2f53
storagenode/pieces: Add Trash and RestoreTrash to piecestore (#3575)
* storagenode/pieces: Add Trash and RestoreTrash to piecestore

* Add index for expiration trash
2019-11-20 09:28:49 -07:00
Kaloyan Raev
6d728d6ea0
storagenode/collect: delete piece 24 hours after expiration (#3613) 2019-11-20 17:02:57 +02:00
Vitalii Shpital
61c8bcc9a6
web/storagenode: egress chart implemented (#3574) 2019-11-20 16:37:57 +02:00
Rafael Antonio Ribeiro Gomes
da39c71d35
storagenode: add new metric satellite.request (#3610)
* storagenode: add new metric satellite.request

* storagenode: metrics fixed

* switch from Counter to Meter
2019-11-19 18:11:31 -03:00
Ivan Fraixedes
8e1e4cc342
piecestore: Fix invalid comment and typos (#3604) 2019-11-19 16:30:48 +01:00
Nikolai Siedov
24318d74b3
storagenode/console: show satellite url in satellite selection (#3602) 2019-11-19 14:16:56 +02:00
Nikolai Siedov
0d35505fe1
SNOboard/console: router changed for gorillaMux, caching added (#3577) 2019-11-15 14:36:43 +02:00
Egon Elbre
ee6c1cac8a
private: rename internal to private (#3573) 2019-11-14 21:46:15 +02:00
Egon Elbre
1a54007f1c
storagenode/storagenodedb: dont log opening of each database (#3571) 2019-11-14 17:08:16 +02:00
Egon Elbre
1e64006e32 lint: add staticcheck as a separate step (#3569) 2019-11-14 10:31:30 +02:00
paul cannon
bd89f51c66
Keep v0pieceinfo database isolated (#3364)
* put TestCreateV0 back in StoreForTest
* avoid direct handles to V0 pieceinfo db
* type mismatch fix
* use storage.Blobs interface in store_test.go

..instead of filestore.Store. this will allow filestore.Store to become
unexported.

* unexport filestore.Store

rename it to blobStore. things should use the storage.Blobs interface
instead. changes in this commit are purely mechanical (made through the
"refactor" tool in Gocode followed by search/replace on the word "Store"
within the storage/filestore/ directory).

* kill filestore.StoreForTest

now that filestore.blobStore is unexported, there isn't a need for a
specialized wrapper type. this (not coincidentally) also makes it
possible for the WriterForFormatVersion() method on
storagenode/pieces.StoreForTest to work, without requiring everything to
wrap the store.blobs attribute in a filestore.StoreForTest, which was
impractical.
2019-11-13 13:15:31 -06:00
Yingrong Zhao
db8294cfba
storagenode/gracefulexit: get hash and limit using original piece ID (#3557) 2019-11-13 12:45:55 -05:00
Jeff Wendling
ebcd37c572 storagenode/contact: fix connection leak with contact checkin
Change-Id: If86002557144d5d8dbff939d2b6a2dfec6537577
2019-11-06 18:00:09 +00:00
littleskunk
7eb6724c92
logging: unify logging around satellite ID, node ID and piece ID (#3491)
* logging: unify logging around satellite ID, node ID and piece ID

* unify segment index
2019-11-05 22:04:07 +01:00
Maximillian von Briesen
257d3946d5
storagenode/gracefulexit: allow storagenodes to concurrently transfer pieces for graceful exit (#3478) 2019-11-05 10:33:44 -05:00
Jennifer Li Johnson
11f0ea3258
5s (#3477) 2019-11-04 16:20:31 -05:00
Jennifer Li Johnson
aa7d15a365
storagenode/contact: exponential backoff retries for pinging Satellites (#3372) 2019-11-04 16:03:21 -05:00
Jess G
5abb91afcf
satellite: change the Peer name to Core (#3472)
* change satellite.Peer name to Core

* change to Core in testplanet

* missed a few places

* keep shared stuff in peer.go to stay consistent with storj/docs
2019-11-04 11:01:02 -08:00
Isaac Hess
4d26d0a6a6 storagenode/pieces: Add migration from v0 piece to v1 piece (#3401) 2019-11-04 17:59:45 +01:00
Egon Elbre
87687938d1 storagenode/contact: fix panic in ping satellites (#3447) 2019-11-01 16:20:53 +01:00
Ethan Adams
43103ae13f
lower storage node counts in tests (#3427) 2019-10-31 10:57:54 -04:00
Jess G
4d85b11574
satellite/contact: improve errors in contact endpoints (#3356)
* improve errors in satellite contact endpoints

* add changes per CR comments

* update pingback method so it still updates node table

* fix err and returns

* fix zap logging to be better
2019-10-30 11:57:21 -07:00
Natalie Villasana
4878135068
satellite/gracefulexit, storagenode/gracefulexit: add timeouts (#3407) 2019-10-30 13:40:57 -04:00
Natalie Villasana
5453886231 satellite/repair, uplink/ecclient: remove unused expiration arg from ec.Repair and ec.putPiece (#3416) 2019-10-30 11:35:00 -04:00
Yingrong Zhao
3ee0b89f8f
storagenode/gracefulexit: delete pieces when receive Delete or Completed message from satellite (#3406) 2019-10-30 10:46:56 -04:00
Egon Elbre
65a8e0bcbc
{satellite,storagenode}/gracefulexit: clearer log messages (#3413) 2019-10-30 10:21:27 +02:00
Isaac Hess
1defd4dbfe
storagenode/piecestore: Respect config.MaxConcurrentRequests for drpc (#3402) 2019-10-28 13:12:49 -06:00
Ethan Adams
5b0398a718 storagenode/gracefulexit: Exclude finished exits from chore/worker processing. Fix update status bug (#3399) 2019-10-28 13:59:45 -04:00