Commit Graph

318 Commits

Author SHA1 Message Date
Bryan White
6b40866ed8
Re-implement libstorj API (V2) using libuplink (V3) (#2573) 2019-07-30 13:40:05 +02:00
ethanadams
8f8b13abb9
Re-enable SN bandwidth rollups. Fix SN bandwidth rollup unique constraint issue. Re-organize service code (#2617)
* re-organizing into bandwidth service. re-enable rollup loop
* Prevent uniqueness failure in bandwidth rollup
* Add test to make sure the rollup select date range works correctly
* add bandwidth config for rollup interval
2019-07-29 10:07:52 -04:00
Egon Elbre
5d0816430f
rename all the things (#2531)
* rename pkg/linksharing to linksharing
* rename pkg/httpserver to linksharing/httpserver
* rename pkg/eestream to uplink/eestream
* rename pkg/stream to uplink/stream
* rename pkg/metainfo/kvmetainfo to uplink/metainfo/kvmetainfo
* rename pkg/auth/signing to pkg/signing
* rename pkg/storage to uplink/storage
* rename pkg/accounting to satellite/accounting
* rename pkg/audit to satellite/audit
* rename pkg/certdb to satellite/certdb
* rename pkg/discovery to satellite/discovery
* rename pkg/overlay to satellite/overlay
* rename pkg/datarepair to satellite/repair
2019-07-28 08:55:36 +03:00
Maximillian von Briesen
906c77b55a
Add RetainStatus to storagenode config (#2633)
--storage2.retain-status = "disabled" (default), "debug", or "enabled"
2019-07-26 16:49:08 -04:00
Egon Elbre
0cdeae1922 add missing error handling (#2630) 2019-07-25 17:01:44 +02:00
Natalie Villasana
f11413bc8e Implement garbage collection on satellite (#2577)
* Added a gc package at satellite/gc, which contains the gc.Service, which runs garbage collection integrated with the metainfoloop, and the gc PieceTracker, which implements the metainfo loop Observer interface and stores all of the filters (about which pieces are good) for each node.
* Added a gc config located at satellite/gc/service.go (loop disabled by default in release)
* Creates bloom filters with pieces to be retained inside the metainfo loop
* Sends RetainRequests (or filters with good piece ids) to all storage nodes.
2019-07-24 13:26:43 -04:00
Michal Niewrzal
5710dc3a32
Metainfo RPC segment methods (part 2) (#2616) 2019-07-24 13:33:23 +02:00
Jess G
353b089927
update testplanet with libuplink (#2618)
* update testplanet uplink upload with libuplink

* add libuplink to testplanet download

* update createbucket and delete obj with libuplink

* update downloadStream, fix tests

* fix test

* updates for CR comments
2019-07-23 07:58:45 -07:00
Jennifer Li Johnson
53d96be44a
Stylistic Go Cleanup (#2524) 2019-07-22 15:10:04 -04:00
Maximillian von Briesen
6c1c3fb4a7
Add metainfo loop service (#2563)
Add a metainfo loop service on the satellite that can be subscribed to by various services that need to make use of metainfo information
2019-07-22 09:34:12 -04:00
Egon Elbre
13dd501042
storagenode/storagenodedb: move tests near the interface rather than the implementation (#2596) 2019-07-19 20:40:27 +03:00
Egon Elbre
f6f65a80d7
storagenode/trust: implement fetching peer identity without kademlia and endpoint (#2584) 2019-07-17 21:14:44 +03:00
Yehor Butko
5f194b4533
SNO Dashboard API (#2427)
* SNO Dashboard API
2019-07-17 14:42:00 +03:00
Michal Niewrzal
260d9c49a8
Metainfo RPC objects methods (#2534) 2019-07-16 12:39:23 +02:00
Jeff Wendling
3a34a0df7b repair: fix data race in reliability cache (#2561) 2019-07-15 22:58:39 +03:00
Jeff Wendling
a2418b22af storagenodedb: optimize index usage and queries (#2545)
- Drops some unused indexes
- Applies a computed index to timestamp columns
- Applies a partial index for expired pieces
- Uses BETWEEN to avoid some datetime calls
- Filters expired piece search by those that aren't NULL
2019-07-12 15:29:09 -04:00
Alexander Leitner
64b2769de3
discovery: parallelize refresh (#2535)
* parallelize discovery refresh

* add paginateQualifiedtest, address pr comments

* Remove duplicate uptime update

* Lower concurrency in Testplanet for discovery
2019-07-12 10:35:48 -04:00
Ivan Fraixedes
f420b29d35
[V3-1927] Repairer uploads to max threshold instead of success… (#2423)
* pkg/datarepair: Add test to check num upload pieces
  Add a new test for ensuring the number of pieces that the repair process
  upload when a segment is injured.
* satellite/orders: Don't create "put order limits" over total
  Repair must not create "put order limits" more than the total count.
* pkg/datarepair: Update upload repair pieces test
  Update the test which checks the number of pieces which are uploaded
  during a repair for using the same excess over the success threshold
  value than the implementation.
* satellites/orders: Limit repair put order for not being total
  Limit the number of put orders to be used by repair for only uploading
  pieces to a % excess over the successful threshold.
* pkg/datarepair: Change DataRepair test to pass again
  Make some changes in the DataRepair test to make pass again after the
  repair upload repaired pieces only until a % excess over success
  threshold.
  Also update the steps description of the DataRepair test after it has been
  changed, to match on what's now, besides to leave it more generic for
  avoiding having to update it on minimal future refactorings.
* satellite: Make repair excess optimal threshold configurable
  Add a new configuration parameter to the satellite for being able to
  configure the percentage excess over the optimal threshold, used for
  determining how many pieces should be repaired/uploaded, rather than
  having the value hard coded.
* repairer: Add configurable param to segments/repairer
  Add a new parameters to the segment/repairer to calculate the maximum
  number of excess nodes, based on the optimal threshold, that repaired
  pieces can be uploaded.
  This new parameter has been added for not returning more nodes than the
  number of upload orders for data repair satellite service calculate for
  repairing pieces.
* pkg/storage/ec: Update log message in clien.Repair
* satellite: Update configuration lock file
2019-07-12 00:44:47 +02:00
Egon Elbre
d52f764e54
protocol: implement new piece signing and verification (#2525) 2019-07-11 16:51:40 -04:00
Maximillian von Briesen
8b507f3d73 Address concerns with storagenode Retain endpoint (#2527) 2019-07-11 16:04:21 -04:00
Bill Thorp
0e463dccfd
7 day validity window for order limits (#2520)
* 7 day limit
2019-07-10 17:17:00 -04:00
JT Olio
a79c7d77f3 overlay cache: slight modification of node-is-online rules (#2490) 2019-07-09 22:36:09 -04:00
Jeff Wendling
d616be8ae0 storagenode: use minimum time in the order for expiration (#2504) 2019-07-09 17:16:30 -04:00
Stefan Benten
16156e3b3d
Ensure we force a segment size and account storage before committing them (#2473) 2019-07-08 18:24:38 -04:00
Egon Elbre
674742d1a7
satellite/datarepair: use reliability cache (#1976) 2019-07-09 01:04:35 +03:00
Alexander Leitner
19ab9852f2
Update node.proto to use time.Time instead of timestamp (#2482) 2019-07-08 14:24:42 -04:00
Kaloyan Raev
f9ed0dc1a8
Improve stability of TestDownloadSharesDownloadTimeout (#2210) 2019-07-05 19:04:15 +03:00
aligeti
ae8b9698f9
Rename/remove EncryptionScheme -> EncryptionParame… (#2363)
* rename/remove EncryptionScheme -> EncryptionParameters
2019-07-03 14:07:44 -04:00
Cameron
d499d162f4
implement storj.NodeURL in trusted satellites (#2388)
* implement storj.NodeURL in trusted satellites
2019-07-03 13:29:18 -04:00
Kaloyan Raev
ca0058c9f1
Set MinDownloadTimeout to 5s in testplanet (#2447) 2019-07-03 17:49:08 +03:00
Egon Elbre
38f3d860a4
storagenode: decline uploads when there are too many live requests (#2397) 2019-07-03 16:47:55 +03:00
Stefan Benten
cded0b412d Properly format Log Message (#2438) 2019-07-03 11:16:26 +02:00
Alexander Leitner
6d55bbdb57
OrderLimit creation date time limit (#2412)
* Limit by order creation
2019-07-02 12:06:12 -04:00
Stefan Benten
3583c65f5b
Move from allowed range to minimum for Version Control (#2421) 2019-07-02 17:28:06 +02:00
Maximillian von Briesen
52e5a4eee3 pass logger into repairer and ecclient (#2365) 2019-07-02 13:08:02 +03:00
Michal Niewrzal
3337087a94
Add storj.StreamID (#2407) 2019-07-02 11:16:06 +02:00
Egon Elbre
e8605d312e satellite/rewards: use USD type (#2384)
* fix String converison

* add method

* rename to USD

* fix types

* fix parsing of forms

* fix tests

* fix header

* use larger type

* use int64

* rename func

* move currency to separate package

* convert types, renames

* fix usercredits

* remove unnecessary conversion

* fix comment and named params
2019-07-01 15:16:49 -04:00
Yaroslav Vorobiov
314837dd6f Fix smtp sender return error on close (#2162) 2019-07-01 19:43:10 +02:00
Natalie Villasana
3f643551e7 remove flakiness in TestDataRepair and TestSegmentStoreRepair (#2335)
* stop audit loop in repair tests to prevent possible timeout
2019-07-01 11:15:45 -04:00
Egon Elbre
2b68a72428
internal/testplanet: ensure that metainfo connections get closed (#2381) 2019-07-01 17:35:10 +03:00
Jeff Wendling
efcdaa43a3
lib/uplink: encryption context (#2349)
* lib/uplink: encryption context

Change-Id: I5c23dca3286a46b713b30c4997e9ae6e630b2280

* lib/uplink: bucket operation examples

Change-Id: Ia0f6e69f365dcff0cf11c731f51b30842bce053b

* lib/uplink: encryption key sharing test cases

Change-Id: I3a172d565f33f4e591402cdcb9460664a7cc7fbe

* fix encrypted path prefix restriction issue

Change-Id: I8f3921f9d52aaf4b84039de608b8cbbc88769554

* implement panics in libuplink encryption code

todo on cipher suite selection as well as an api concern

Change-Id: Ifa39eb3cc4b3443f7d96f9304df9b2ac4ec4085d

* implement GetProjectInfo api call to get salt

Change-Id: Ic5f6b3be9ea35df48c1aa214ab5d355fb328e2cf

* some fixes and accessors for encryption store

Change-Id: I3bb61f6712a037900e2a96e72ad4029ec1d3f718

* general fixes to builds/tests/etc

Change-Id: I9930fa96acb3b221d9a001f8e274af5729cc8a47

* java bindings changes

Change-Id: Ia2bd4c9c69739c8d3154d79616cff1f36fb403b6

* get libuplink examples passing

Change-Id: I828f09a144160e0a5dd932324f78491ae2ec8a07

* fix proto.lock file

Change-Id: I2fbbf4d0976a7d0473c2645e6dcb21aaa3be7651

* fix proto.lock again

Change-Id: I92702cf49e1a340eef6379c2be4f7c4a268112a9

* fix golint issues

Change-Id: I631ff9f43307a58e3b25a58cbb4a4cc2495f5eb6

* more linting fixes

Change-Id: I51f8f30b367b5bca14c94b15417b9a4c9e7aa0ce

* bug fixed by structs bump

Change-Id: Ibb03c691fce7606c35c08721b3ef0781ab48a38a

* retrigger

Change-Id: Ieee0470b6a2d07168a1578552e8e7f271ae93a13

* retrigger

Change-Id: I753d63853171e6a436c104ce176048892eb974c5

* semantic merge conflict

Change-Id: I9419448496de90340569047a6a16a1b858a7978a

* update total to match prod defaults

Change-Id: I693d55c1ebb28b5803ee1d26e9e198decf82308b

* retrigger

Change-Id: I28b74d5d6202f61aa3866fe407d423f6a0a14b9e

* retrigger

Change-Id: I6fd054885c715f602e2cef623fd464c42e88742c

* retrigger

Change-Id: I6a01bae88c72406d4ed5a8f13bf8a2b3c650bd2d
2019-06-27 17:36:51 +00:00
Egon Elbre
615bfca135 Fix TestGetSignee flakiness (#2350)
* add IsCanceled

* fixes to error handling

* fix imports

* retrigger jenkins
2019-06-26 09:30:37 -06:00
Egon Elbre
7b66e0cd7c Use dial to clarify that it's internally closing the connection. (#2347) 2019-06-26 15:14:48 +03:00
Egon Elbre
b6ad3e9c9f
internal/testrand: new package for random data (#2282) 2019-06-26 13:38:51 +03:00
Kaloyan Raev
ae36979804 Take advantage of Is and IsFunc from zeebo/errs (#2310) 2019-06-26 09:38:07 +02:00
Egon Elbre
c7679b9b30
Fix some leaks and add notes about close handling (#2334) 2019-06-25 23:00:51 +03:00
Egon Elbre
c28f800098
Skip TestDataRepair and TestUplinksParallel, because they are flaky (#2337) 2019-06-25 19:30:39 +03:00
Egon Elbre
414648d660
Fix some metainfo.Client leaks (#2327) 2019-06-25 18:36:23 +03:00
Egon Elbre
6502143e79
fix import ordering (#2322) 2019-06-25 12:46:29 +03:00
Yingrong Zhao
bbedff12a6 satellite: rearrange marketing package (#2268)
* move offer out of marketing package and remove marketing package

* fix imports

* fix rename errors

* remove offer service

* change package name from offers to rewards

* fix linting

* remove unused code and use appropriate comment
2019-06-24 16:51:54 -04:00