Commit Graph

402 Commits

Author SHA1 Message Date
Marc Schubert
64199d37bd
Improve Node Selection Query (#2648)
* Update overlaycache.go

Removes one select statement and columns gets filtered in first query. 

Needs to be tested agains real database that this query is working and faster!

* Correct linting

reorder scans that this fit to new sql result order
2019-07-29 14:32:43 +02:00
Egon Elbre
e75813d094 satellite/repair: move segment repairer to satellite and simplify (#2651) 2019-07-29 13:24:56 +02:00
Egon Elbre
dd7c8610bb
satellite/repair: move test files (#2649) 2019-07-28 12:15:34 +03:00
Egon Elbre
5d0816430f
rename all the things (#2531)
* rename pkg/linksharing to linksharing
* rename pkg/httpserver to linksharing/httpserver
* rename pkg/eestream to uplink/eestream
* rename pkg/stream to uplink/stream
* rename pkg/metainfo/kvmetainfo to uplink/metainfo/kvmetainfo
* rename pkg/auth/signing to pkg/signing
* rename pkg/storage to uplink/storage
* rename pkg/accounting to satellite/accounting
* rename pkg/audit to satellite/audit
* rename pkg/certdb to satellite/certdb
* rename pkg/discovery to satellite/discovery
* rename pkg/overlay to satellite/overlay
* rename pkg/datarepair to satellite/repair
2019-07-28 08:55:36 +03:00
Yingrong Zhao
3b6c69c594
fix mutation (#2641) 2019-07-26 10:58:17 -07:00
Faris Huskovic
2144181c99
satellite/marketingweb: Add Partner Offer Type (#2540) 2019-07-25 18:06:23 -04:00
Stefan Benten
e260f9f1cc
Remove unnecessary UNION and Selects (#2624) 2019-07-25 15:17:12 -04:00
Kaloyan Raev
734c793deb
Add UpdatePieces method to Metainfo service (#2572) 2019-07-25 19:59:46 +03:00
Egon Elbre
0cdeae1922 add missing error handling (#2630) 2019-07-25 17:01:44 +02:00
Natalie Villasana
f11413bc8e Implement garbage collection on satellite (#2577)
* Added a gc package at satellite/gc, which contains the gc.Service, which runs garbage collection integrated with the metainfoloop, and the gc PieceTracker, which implements the metainfo loop Observer interface and stores all of the filters (about which pieces are good) for each node.
* Added a gc config located at satellite/gc/service.go (loop disabled by default in release)
* Creates bloom filters with pieces to be retained inside the metainfo loop
* Sends RetainRequests (or filters with good piece ids) to all storage nodes.
2019-07-24 13:26:43 -04:00
Michal Niewrzal
5710dc3a32
Metainfo RPC segment methods (part 2) (#2616) 2019-07-24 13:33:23 +02:00
Ivan Fraixedes
b24e60a33f
satellite/satellitedb: Use var block for single variable declar… (#2622)
* satellite/satellitedb: User var block for Error
  To follow with the code style of the majority of the sources of the
  current code base the Error variable should be in a block.
  Replacing a single var expression to a block one makes the godoc more
  consistent across the repository.

* satellite/satellitedb: Remove empty spaces end of line
2019-07-23 19:58:43 +02:00
Yingrong Zhao
aaf3283b4e
satellite/console: referral link for individual users (#2614)
* init

* change give credit to be invitee credits for free credits

* remove unused variables

* fix tests
2019-07-23 12:08:07 -04:00
Jess G
353b089927
update testplanet with libuplink (#2618)
* update testplanet uplink upload with libuplink

* add libuplink to testplanet download

* update createbucket and delete obj with libuplink

* update downloadStream, fix tests

* fix test

* updates for CR comments
2019-07-23 07:58:45 -07:00
Ivan Fraixedes
3c8f1370d2
[v3 2137] - Add more info to find out repair failures (#2623)
* pkg/datarepair/repairer: Track always time for repair
  Make a minor change in the worker function of the repairer, that when
  successful, always track the metric time for repair independently if the
  time since checker queue metric can be tracked.

* storage/postgreskv: Wrap error in Get func
  Wrap the returned error of the Get function as it is done when the
  query doesn't return any row.

* satellite/metainfo: Move debug msg to the right place
  NewStore function was writing a debug log message when the DB was
  connected, however it was always writing it out despite if an error
  happened when getting the connection.

* pkg/datarepair/repairer: Wrap error before logging it
  Wrap the error returned by process which is executed by the Run method
  of the repairer service to add context to the error log message.

* pkg/datarepair/repairer: Make errors more specific in worker
  Make the error messages of the "worker" method of the Service more
  specific and the logged message for such errors.

* pkg/storage/repair: Improve error reporting Repair
  In order of improving the error reporting by the
  pkg/storage/repair.Repair method, several errors of this method and
  functions/methods which this one relies one have been updated to be
  wrapper into their corresponding classes.

* pkg/storage/segments: Track path param of Repair method
  Track in monkit the path parameter passed to the Repair method.

* satellite/satellitedb: Wrap Error returned by Delete
  Wrap the error returned by repairQueue.Delete method to enhance the
  error with a class and stack and the
  pkg/storage/segments.Repairer.Repair method get a more contextualized
  error from it.
2019-07-23 16:28:06 +02:00
Michal Niewrzal
cba008d7df
Add GetObject method to Metainfo (#2611) 2019-07-23 13:09:12 +02:00
Jennifer Li Johnson
53d96be44a
Stylistic Go Cleanup (#2524) 2019-07-22 15:10:04 -04:00
Michal Niewrzal
6f2b85603d
Metainfo RPC segment methods (part 1) (#2567) 2019-07-22 16:45:18 +02:00
Maximillian von Briesen
537d1cf09f
Add test to ensure correct order of repair queue selection (#2551) 2019-07-22 10:10:42 -04:00
Maximillian von Briesen
6c1c3fb4a7
Add metainfo loop service (#2563)
Add a metainfo loop service on the satellite that can be subscribed to by various services that need to make use of metainfo information
2019-07-22 09:34:12 -04:00
Yingrong Zhao
6778caf846
satellite/console: add referral link logic (#2576)
* setup referral route

* referredBy

* add user id

* modify user query

* separate optional field from userInfo

* get current reward on init of satellite gui

* remove unsed code

* fix format

* only apply 0 credit on registration

* only pass required information for rewards

* fix time parsing

* fix test and linter

* rename method

* add todo

* remove user referral logic

* add null check and fix format

* get current offer

* remove partnerID on CreateUser struct

* fix storj-sim user creation

* only redeem credit when there's an offer

* fix default offer configuration

* fix migration

* Add helper function for get correct credit duration

* add comment

* only store userid into user_credit table

* add check for partner id to set correct offer type

* change free credit to use invitee credits

* remove unecessary code
2019-07-19 14:22:10 -04:00
aligeti
29b576961f
value attribution merge fix and more test cases (#2588)
* value attribution merge fix and more test cases
2019-07-19 11:17:34 -04:00
Bill Thorp
a7cc940776
Nodes should not be able to fail the same audit multiple times (#2404)
update pointer on audit failure
2019-07-18 14:08:15 -04:00
Yingrong Zhao
3e9f90b7ee
satellite/satellitedb: fix nil value from offers table (#2587)
* fix nil value

* Oops
2019-07-18 13:26:04 -04:00
Jess G
3af9250659
update irreparableDB.GetLimited query to use where instead of offset (#2585)
* update query to use where instead of offset, update tests

* update cmd/inspector irreparable

* add comment for offset
2019-07-18 09:21:21 -07:00
ethanadams
af7ffb8072 v3-2156: Add partner_id on user creation (#2571)
* Add partnerID on user creation

* added support for partner ID on create user in consoleql User

* add partner ID to api key if the user creating it has a partner ID associated with it

* updates for consoleal user and userinfo
2019-07-17 16:53:14 -04:00
Michal Niewrzal
0b827f1375
Speedup metainfo tests (#2556) 2019-07-16 21:16:41 +02:00
Kaloyan Raev
d453cd148e Consider encryption overhead when validating max segment size (#2569) 2019-07-16 15:30:10 +02:00
Michal Niewrzal
260d9c49a8
Metainfo RPC objects methods (#2534) 2019-07-16 12:39:23 +02:00
Yingrong Zhao
c12a4aed3b satellite/console: add redeem reward method (#2484)
* add RedeemRewards method

* remove redeem from reward.db

* add redeemable cap check in redeem

* rename offerCap to redeemableCap

* remove redeem test

* update error message

* fix build

* Trigger Jenkins

* use correct credit setting for redeem

* fix comment

* change create qury to get redeemable_cap from offers table

* change referredBy to a pointer in user credit struct
2019-07-15 19:30:00 -04:00
Jess G
aa99482fa7
Jg/add tests (#2547)
* add a test to confirm only objects in pointerDB

* fix lint

* fix link
2019-07-15 09:30:26 -07:00
aligeti
3a01106ab4
quick fix (#2548) 2019-07-12 14:12:02 -04:00
aligeti
daa3b32ee2
Add Attribution Columns to appropriate tables for OSPP referral link (#2516)
* adds "partner_id" column to user, project, api_key & bucket_metainfo tables
2019-07-12 13:59:19 -04:00
Maximillian von Briesen
b590e53d64
Order by attempted time in injured segments select (#2533) 2019-07-12 13:35:20 -04:00
Alexander Leitner
64b2769de3
discovery: parallelize refresh (#2535)
* parallelize discovery refresh

* add paginateQualifiedtest, address pr comments

* Remove duplicate uptime update

* Lower concurrency in Testplanet for discovery
2019-07-12 10:35:48 -04:00
Yingrong Zhao
d887ffec62
satellite/satellitedb: add default offer for offers table (#2522)
* add default offer for offers table

* fix migration test

* Trigger Jenkins

* set the default value to be correct type

* skip soon will deleted test

* fix test data

* add orderby for ListAll

* change durations, redeemable cap to be a nullable field

* remove unecessary code
2019-07-12 10:19:38 -04:00
Faris Huskovic
003828c348
web/marketing/pages: partner UI table (#2523)
* add UI table for partner offers

* remove award credit from partner table, add referral link column
2019-07-12 09:52:00 -04:00
Yingrong Zhao
6e57b102c5
satellite/satellitedb: remove num_redeemed in offers table (#2510)
* remove num_redeemed

* remove the field frol Offer struct

* remove the field in the UI templates

* fix merge conflict

* fix migration

* fix merge conflict

* fix typo
2019-07-12 09:19:18 -04:00
Jess G
f11bf46a11
Jg/1967 mv bucket metadata uplink (#2505)
* add bucketstore, add init uplink bucket

* update uplink to use bucket rpc

* fix tests

* wrap metainfo client errors

* add allowedBucket struct, fix tests

* update comment

* add paging

* updates per CR

* add test for pagination

* fix lint

* fix uplink test so its easier tyo understand

* fix gateway pagination bug

* changes per cr

* fix bug w/allowedBuckets, add test to catch
2019-07-12 08:57:02 -04:00
Ivan Fraixedes
f420b29d35
[V3-1927] Repairer uploads to max threshold instead of success… (#2423)
* pkg/datarepair: Add test to check num upload pieces
  Add a new test for ensuring the number of pieces that the repair process
  upload when a segment is injured.
* satellite/orders: Don't create "put order limits" over total
  Repair must not create "put order limits" more than the total count.
* pkg/datarepair: Update upload repair pieces test
  Update the test which checks the number of pieces which are uploaded
  during a repair for using the same excess over the success threshold
  value than the implementation.
* satellites/orders: Limit repair put order for not being total
  Limit the number of put orders to be used by repair for only uploading
  pieces to a % excess over the successful threshold.
* pkg/datarepair: Change DataRepair test to pass again
  Make some changes in the DataRepair test to make pass again after the
  repair upload repaired pieces only until a % excess over success
  threshold.
  Also update the steps description of the DataRepair test after it has been
  changed, to match on what's now, besides to leave it more generic for
  avoiding having to update it on minimal future refactorings.
* satellite: Make repair excess optimal threshold configurable
  Add a new configuration parameter to the satellite for being able to
  configure the percentage excess over the optimal threshold, used for
  determining how many pieces should be repaired/uploaded, rather than
  having the value hard coded.
* repairer: Add configurable param to segments/repairer
  Add a new parameters to the segment/repairer to calculate the maximum
  number of excess nodes, based on the optimal threshold, that repaired
  pieces can be uploaded.
  This new parameter has been added for not returning more nodes than the
  number of upload orders for data repair satellite service calculate for
  repairing pieces.
* pkg/storage/ec: Update log message in clien.Repair
* satellite: Update configuration lock file
2019-07-12 00:44:47 +02:00
Egon Elbre
d52f764e54
protocol: implement new piece signing and verification (#2525) 2019-07-11 16:51:40 -04:00
Michal Niewrzal
268c629ba8
Replace base64 encoding for path segments (#2345) 2019-07-11 13:26:07 -04:00
Bill Thorp
0e463dccfd
7 day validity window for order limits (#2520)
* 7 day limit
2019-07-10 17:17:00 -04:00
Bogdan Artemenko
32e0227c45
Project Payment methods (#2037) 2019-07-10 23:29:26 +03:00
Faris Huskovic
0d294103e9
satellite/rewards: nicer offers handling (#2390)
* organize offers

* revert changes to go.mod and go.sum

* change OfferStatus enums back to original

* revert modified auto-gen files

* don't render empty row if offers is empty

* change return val of ListAll to Offers

* fix build

* add method to check for empty offer when rendering template

* fix typo

* fix lint and typos

* lean out IsEmpty

* dont use named return vals

* better clarify offer statuses

* change back order of setting offer.Status

* lint

* satellite/marketingweb: allow disabling rewards (#2392)

* implement handler for stop offer endpoint

* use proper text and fix data-target for free-credit stop modal
2019-07-10 13:12:40 -04:00
JT Olio
a79c7d77f3 overlay cache: slight modification of node-is-online rules (#2490) 2019-07-09 22:36:09 -04:00
Alexander Leitner
1c5db71faf
Change protobuf expirations to use time.Time (#2509)
* Change protobuf expirations to use time.Time instead of timestamp.Timestamp
2019-07-09 17:54:00 -04:00
Egon Elbre
910e4744ad
satellite/metainfo: optimize pointerdb size (#2506) 2019-07-09 16:36:18 -04:00
Egon Elbre
203f36a41f
rename order.go to order.pb.go (#2496) 2019-07-09 13:24:41 -04:00
Jess G
f9696d6c5e
satellite/metainfo: add buckets RPC and database (#2460)
* add db interface and methods, add sa metainfo endpoints and svc

* add bucket metainfo svc funcs

* add sadb bucekts

* bucket list gets all buckets

* filter buckets list on macaroon restrictions

* update pb cipher suite to be enum

* add conversion funcs

* updates per comments

* bucket settings should say default

* add direction to list buckets, add tests

* fix test bucket names

* lint err

* only support forward direction

* add comments

* minor refactoring

* make sure list up to limit

* update test

* update protolock file

* fix lint

* change per PR
2019-07-08 15:32:18 -07:00