Commit Graph

625 Commits

Author SHA1 Message Date
Egon Elbre
e8605d312e satellite/rewards: use USD type (#2384)
* fix String converison

* add method

* rename to USD

* fix types

* fix parsing of forms

* fix tests

* fix header

* use larger type

* use int64

* rename func

* move currency to separate package

* convert types, renames

* fix usercredits

* remove unnecessary conversion

* fix comment and named params
2019-07-01 15:16:49 -04:00
Egon Elbre
385c046723
pkg/pb: rename Order2 to Order, OrderLimit2 to OrderLimit (#2406) 2019-07-01 18:54:11 +03:00
Egon Elbre
2b68a72428
internal/testplanet: ensure that metainfo connections get closed (#2381) 2019-07-01 17:35:10 +03:00
Michal Niewrzal
ff92a5bcf6 Improve RS validation message (#2377) 2019-06-29 17:05:46 +02:00
Faris Huskovic
ebd976ec28
satellite/marketing: Create New Offer (#2186)
* update UI to reflect final mockups

* implement create handler and render offers table data to UI

* fix line-height unit and remove important from selectors

* update file names and ids for clarity

* shorten 'label' in ids

* localize global vars, fix endpoint names, remove unnecessary receiver, fix comments

* fix unnecessary initialization of pointer

* correct file-naming conventions

* register timeConverter in an init func for safety and remove unnecessary important from css

* consolidate create endpoints and add comments

* register timeConverter in init func

* add copyright to files

* introduce require pkg

* add proper http server unit test

* update linting and create offers concurrently in unit test

* fix getOffers comment

* add copy-right to unit-test

* fix data-races

* fix linting

* remove converter in NewServer

* requested changes in progress

* add require for checking status code

* renamed template file

* fix 400 handler

* fix missing copyright and remove extra line

* fix build

* run goroutine for testing parallel

* evaluate reqType with switch stmt and promp for credit amount in cents

* fix lint issue

* add default case

* remove unnecessary var

* fix range scope error

* remove empty lines and use long form for struct field

* fix merge conflicts

* fix template reference

* fix modal id

* not delete package

* add currency formatting and requested changes

* markup formatting

* lean out currency logic and move wait outside loop

* pass ToDollars func to home template

* fix lint
2019-06-28 10:34:10 -04:00
Egon Elbre
827fb92b47
satellite/console: nicer error handling in tests (#2378) 2019-06-28 16:04:50 +03:00
Egon Elbre
e83ebd7cde
jenkins: avoid using goimports and distribute load better (#2359) 2019-06-27 21:52:50 +03:00
Jeff Wendling
efcdaa43a3
lib/uplink: encryption context (#2349)
* lib/uplink: encryption context

Change-Id: I5c23dca3286a46b713b30c4997e9ae6e630b2280

* lib/uplink: bucket operation examples

Change-Id: Ia0f6e69f365dcff0cf11c731f51b30842bce053b

* lib/uplink: encryption key sharing test cases

Change-Id: I3a172d565f33f4e591402cdcb9460664a7cc7fbe

* fix encrypted path prefix restriction issue

Change-Id: I8f3921f9d52aaf4b84039de608b8cbbc88769554

* implement panics in libuplink encryption code

todo on cipher suite selection as well as an api concern

Change-Id: Ifa39eb3cc4b3443f7d96f9304df9b2ac4ec4085d

* implement GetProjectInfo api call to get salt

Change-Id: Ic5f6b3be9ea35df48c1aa214ab5d355fb328e2cf

* some fixes and accessors for encryption store

Change-Id: I3bb61f6712a037900e2a96e72ad4029ec1d3f718

* general fixes to builds/tests/etc

Change-Id: I9930fa96acb3b221d9a001f8e274af5729cc8a47

* java bindings changes

Change-Id: Ia2bd4c9c69739c8d3154d79616cff1f36fb403b6

* get libuplink examples passing

Change-Id: I828f09a144160e0a5dd932324f78491ae2ec8a07

* fix proto.lock file

Change-Id: I2fbbf4d0976a7d0473c2645e6dcb21aaa3be7651

* fix proto.lock again

Change-Id: I92702cf49e1a340eef6379c2be4f7c4a268112a9

* fix golint issues

Change-Id: I631ff9f43307a58e3b25a58cbb4a4cc2495f5eb6

* more linting fixes

Change-Id: I51f8f30b367b5bca14c94b15417b9a4c9e7aa0ce

* bug fixed by structs bump

Change-Id: Ibb03c691fce7606c35c08721b3ef0781ab48a38a

* retrigger

Change-Id: Ieee0470b6a2d07168a1578552e8e7f271ae93a13

* retrigger

Change-Id: I753d63853171e6a436c104ce176048892eb974c5

* semantic merge conflict

Change-Id: I9419448496de90340569047a6a16a1b858a7978a

* update total to match prod defaults

Change-Id: I693d55c1ebb28b5803ee1d26e9e198decf82308b

* retrigger

Change-Id: I28b74d5d6202f61aa3866fe407d423f6a0a14b9e

* retrigger

Change-Id: I6fd054885c715f602e2cef623fd464c42e88742c

* retrigger

Change-Id: I6a01bae88c72406d4ed5a8f13bf8a2b3c650bd2d
2019-06-27 17:36:51 +00:00
aligeti
7facda4422
value attribution integration with libuplink (#2297)
* integrates the value attribution with libuplink
2019-06-26 12:22:01 -04:00
Yehor Butko
8bf7c5c671
SNO Dashboard http status codes updated (#2333) 2019-06-26 16:36:47 +03:00
Egon Elbre
b6ad3e9c9f
internal/testrand: new package for random data (#2282) 2019-06-26 13:38:51 +03:00
Michal Niewrzal
6b23380d6b
Make RS validation always enabled (#2336) 2019-06-26 12:27:36 +02:00
Egon Elbre
caa2fcf62b satellite/orders: don't panic (#2331) 2019-06-26 09:26:33 +02:00
ethanadams
0e528bc56e
Add attribution report to the satellite CLI (#2288)
* added satalite partner value attribution report. WIP

* WIP

* basic attribution report test completed. still a WIP

* cleanup

* fixed projectID conversion

* report display cleanup

* cleanup .added more test data

* added partnerID to query results

* fixed lint issues

* fix import order

* suggestions from PR review

* updated doc to reflect implementation

* clarification comments in the report SQL

* Changed based on PR suggestion

* More changes based on PR suggestions

* Changes based on PR suggestions

* reordered tests to make consistant with previous 2

* small comments cleanup

* More PR suggestions

* fixed lint issue and removed printf

* fixed var name

* Updates based on PR suggestions

* fixed message

* fixed test

* changes required after merge from master
2019-06-25 16:58:38 -04:00
Egon Elbre
c7679b9b30
Fix some leaks and add notes about close handling (#2334) 2019-06-25 23:00:51 +03:00
ethanadams
605e3fb730
v3-2023: Add migration for project_id change in V3-2010 (#2332)
* v3-2023: add project_id migration for bucket_storage_tallies and bucket_bandwidth_rollups

* added test data for migration 37

* corrected data format

* test sql update

* migrate script updates

* adding previous data
2019-06-25 15:21:34 -04:00
Jess G
e5c48fab74
fix ordersDB methods to take correct args (#2314)
* fix orderdDB methods to take correct args

* update tally to save projectID in correct format

* update var names in splitBucket test

* changes per CR comments
2019-06-25 08:58:42 -07:00
Egon Elbre
414648d660
Fix some metainfo.Client leaks (#2327) 2019-06-25 18:36:23 +03:00
Yaroslav Vorobiov
eb5dc4ba2b
Add node stats GRPC endpoint to the satellite (#2281) 2019-06-25 13:40:32 +03:00
Jeff Wendling
c35c8e4c24 allow reading bucket metadata with restricted keys (#2321)
Change-Id: I47d3a2f5f02744ae6c51d54963cdf2dff24134e2
2019-06-24 17:36:09 -06:00
Kaloyan Raev
7bf5a09638 Use ErrNodeOffline when returning error for offline node (#2312) 2019-06-25 00:15:38 +02:00
Yingrong Zhao
bbedff12a6 satellite: rearrange marketing package (#2268)
* move offer out of marketing package and remove marketing package

* fix imports

* fix rename errors

* remove offer service

* change package name from offers to rewards

* fix linting

* remove unused code and use appropriate comment
2019-06-24 16:51:54 -04:00
JT Olio
e17908a266
metainfo: add project info rpc for getting a project salt using an api key (#2311)
The API exposed in #2302 needs to be able to get a project-level salt (https://github.com/storj/storj/pull/2302/files#diff-72b734f4a2e408a544fef071d2565fc8R200), see https://github.com/storj/storj/blob/master/docs/design/password-key-derivation.md also

I realized this morning that if we need it later this week, we need to get it into today's release.

Please describe the tests: tests will come as part of #2302

Please describe the performance impact: none
2019-06-24 11:15:45 -06:00
Stefan Benten
01beaa289a
Mask IP Addresses to subnets (#2305) 2019-06-24 17:33:18 +02:00
Fadila
8226024ca8
Do not use disqualified nodes when asking for get order limits (#2303)
Add checks about disqualification when creating order limits
2019-06-24 16:46:10 +02:00
Michal Niewrzal
fdeb834801
Bucket name validation (#2244) 2019-06-24 11:52:25 +02:00
paul cannon
8948459166
do ip filtering in a more correct way? (#2301)
This doesn't solve much of the performance difficulty but ought to be
lots more correct in terms of proper node selection semantics.
2019-06-23 16:16:45 -05:00
Bill Thorp
d1e9829e84 preserve reputation migration (#2295)
Adds a migration step to pull in old reputation success / total counts into modern alpha / beta scores

If audit success count is less than 50, audit alpha will be set to 50
If uptime success count is less than 100, uptime alpha will be set to 100
This helps us deal with cases where nodes have not been audited or checked for uptime yet, in which case alpha/beta values of 0/0 would cause a node to be considered disqualified.

A node with audit alpha/beta of 50/0 will be disqualified on the 19th check
A node with uptime alpha/beta of 100/0 will be disqualified on the 44th check

This does not affect brand new nodes (nodes that were not in the database before this change). The alpha/beta values for those nodes will be set to 1/0 as before
2019-06-22 21:18:54 -04:00
Cameron
1283036e37
add storage node voucher request service (#2158)
* add voucher service on storage node

* config field tag syntax, go routines for requests

* hook up voucher service in storagenode/peer.go

* add voucher config to testplanet

* add voucher config to testplanet

* add voucher response status INVALID, ACCEPTED, REJECTED

* add a test for vouchers service

* handle no row from GetValid, test it

* add trust pool to voucher service

* use trusted list to get satellites

* verify vouchers upon receipt

* test VerifyVoucher
2019-06-21 18:48:52 -04:00
aligeti
d8e62bc06e
support value attribution endpoint (#2231)
Support Value attribution Endpoint
2019-06-21 15:14:34 -04:00
aligeti
043d603cbe
satellite rs config check with validation check set to false default (#2229)
* satellite rs config check with validation check
2019-06-21 14:15:58 -04:00
Bill Thorp
8f47fca5d3
Remove audit / uptime ratio fields (#2247)
* removed ratios
2019-06-21 13:14:53 -04:00
ethanadams
4f2e893e68
Fix the way project_id is stored in bucket_storage_tallies and bucket_bandwidth_rollups (#2283)
* fixing issues where projectID is stored as the byte representation of a UUID string, instead of bytes of the UUID

* added test for spitBucketID
2019-06-21 11:38:37 -04:00
Maximillian von Briesen
d103dd2c4e
Include information about disqualified nodes in the storage node payment report (#2239) 2019-06-21 10:21:15 -04:00
littleskunk
a5baebfa65 reject invalid orders (#2262) 2019-06-21 13:44:14 +02:00
JT Olio
568b000e9b satellite: make order expiration configurable (#2251) 2019-06-21 13:38:40 +03:00
JT Olio
76b54458e9 satellite: send external address in order limits (#2278) 2019-06-21 12:19:52 +03:00
Natalie Villasana
9386187fe6
add disqualification and new reputation system into overlay cache (#2227) 2019-06-20 09:56:04 -04:00
Yingrong Zhao
1c2067a090
Create graphql query for getting user credit usage (#2255)
* set up creditUsage query

* rename function name to follow convention

* remove unused code

* add coipyright

* fix linting
2019-06-20 09:09:50 -04:00
Yingrong Zhao
d583ab707b add user credit usage method into console service (#2240) 2019-06-19 22:49:04 +02:00
Kaloyan Raev
964c87c476 Fix checks around repair threshold (#2246) 2019-06-19 22:13:11 +02:00
aligeti
7b450927ff
code updates with comments (#2245) 2019-06-19 08:50:50 -04:00
Egon Elbre
8f4a6afc8a Fixes for value attribution (#2238)
* Move value attribution to satellite/

* remove unnecessary conversion
2019-06-19 08:02:37 -04:00
Maximillian von Briesen
ad8cad4909
Expand the inspector tool to provide node id's for each segment, rather than just numeric totals (#2205) 2019-06-18 18:22:14 -04:00
Bill Thorp
119a8fd3cc removed fields (#2234)
* removed fields

* sql tweaks
2019-06-18 13:40:28 -04:00
Yaroslav Vorobiov
1334933cdc Console: Remove payments checks during project creation and member deletion (#2236)
* remove payments check during project creation and member deletion

* remove project payment info creation
2019-06-18 13:09:34 -04:00
Yingrong Zhao
954ca3c6ee
add db implementation for user_credits table (#2169)
* add dbx queries

* add migration file

* start service

* Add TotalReferredCountByUserId and availableCreditsByUserID

* implement UserCredits interface and UserCredit struct type

* add UserCredits into consoledb

* add setupData helper function

* add test for update

* update lock file

* fix lint error

* add invalidUserCredits tests

* rename method

* adds comments

* add checks for erros in setupData

* change update method to only execute one query per request

* rename vairable

* should return a signal from Update method if the charge is not fully complete

* changes for readability

* prevent sql injection

* rename

* improve readability
2019-06-18 11:55:47 -04:00
Bill Thorp
81134e97cc
nodes db - adds alpha and beta reputation fields (#2215)
* node dbx and migrate adding alpha and beta reputation
2019-06-18 09:45:02 -04:00
Yaroslav Vorobiov
503b951bcd
modify build scripts for satellite ui (#2228) 2019-06-18 16:36:54 +03:00
aligeti
09940d4e0b
value attribution DB interface Insert & Get method support (#2200)
* value attribution DB interface methods support
2019-06-18 09:06:33 -04:00
Maximillian von Briesen
8398fae9b5
Add node churn and containment/reverify monkit stats (#2217)
* add counters for nodes that have/have not been seen in the past 24 hours/week

* add additional uptime counters

* add monkit stats for containment mode
2019-06-18 08:54:52 -04:00
Ivan Fraixedes
35c8648330
[v3-1914] Storage node disqualification: Change type from bool to timestamp (#2212)
* satellite/satellitedb: Alter nodes disqualification column
  Change the type of the 'disqualification' column of the nodes table from
  boolean to timestamp.
* overlay/cache: Change Disqualified field type
  Change the Disqualified field type the NodeDossier struct type from bool
  to time.Time to match with the disqualified type used by the DB layer.
* satellite/satellitedb: Update queries uses disqualified
  Update the queries which uses the disqualified column due to the column
  type has been changed from boolean to nullable timestamp.
* docs/design: Update disqualification due impl changes
  Update the disqualification design document to contain the architectural
  change required to be able to restore unfair disqualified nodes in case
  of an unexpected cause (bug, mistake, hard network disconnection, etc.).
2019-06-18 11:14:31 +02:00
Faris Huskovic
7795ba8169
update UI to reflect final mockups (#2175)
* update UI to reflect final mockups

* fix line-height unit and remove important from selectors

* update file names and ids for clarity

* shorten 'label' in ids

* correct file-naming conventions

* add copyright to files

* check if break in imports is causing lint error

* resolve lint

* tidy go mod

* fix shorthands
2019-06-17 21:57:04 -04:00
Egon Elbre
f4f776d092 Use mail.test as domain in emails (#2224) 2019-06-18 02:28:40 +02:00
Kaloyan Raev
8e29ef8a6b Use zap.Stringer instead of zap.String (#2223) 2019-06-18 01:37:43 +02:00
JT Olio
e58a06bd0c config: update release values to match prod (#2192) 2019-06-15 18:19:19 +02:00
Faris Huskovic
c0fe061a67 add user credits table (#2085)
* add user credits table

* change primary key, change type for credit_type, and change relation kind of foreign keys from cascade to restrict

* modify table and query methods

* modify schema

* add dbx queries

* add migration file

* add orderby to read available credit entries
2019-06-13 09:52:33 -04:00
ethanadams
380684e766 Removing bucket_id and adding project_id and bucket_name as primary key (#2182)
* removing bucket_id and adding project_id and bucket_name as primary key

* cleanup
2019-06-13 08:28:55 -04:00
aligeti
8e8a5eb89c
initial value attribution check development (#2091)
* initial value attribution check development

* code reveiew updates
2019-06-12 21:35:37 -04:00
Yingrong Zhao
474d9e7492
only parse templates for the first requests (#2180)
* only parse templates for the first requests

* fix race condition isse

* parse all templates before start the new server

* add comment for parseTemplates and remove unused code

* remove unused type
2019-06-12 16:27:07 -04:00
Yingrong Zhao
a4d6e8c8ca split update methods into two for offer data (#2133)
* change Offers interface to separate Update method into two.

* Implement Finish and Redeem method to avoid concurrent updates

* Implement FinishOffer and RedeemOffer service methods

* add tests

* fix linting issue

* add tests for checking Finish and Redeem's results to work as expected

* fix linting error
2019-06-12 11:53:19 -04:00
JT Olio
14486b1885 satellite/orders: TODOs about transactions (#2134) 2019-06-12 17:00:29 +02:00
Egon Elbre
fdddaa2a47 Fix marketingweb code (#2177)
* set to only listen on 127.0.0.1, move static files to same location, better template handling

* handle error

* fix path in storj-sim

* revert template handling changes

* code shouldn't panic on invalid tempalte

* do not rewrite once writing has started

* write correct error code

* use filepath for path handling

* revert change

* fix

* fix mod tidy

* use correct error code for not found, avoid infinite loop on failure
2019-06-12 09:42:39 -04:00
Yingrong Zhao
af66d9c6e4
Open a new port on satellite for admin GUI (#1901)
* Set up new port 8090 for in offers

Clean up commented code

Rename offers to offersweb

Remove unused code

Add todos for adding front-end templates

Add middleware for only allow local access

Add comment

Fix linting error

Remove commented code

Update storj-sim

Check request IP against Host IP

Use net pakcage to retrieve IP address

Rename service to marketing

* Add wrapper for all errors

* fix conflicts

* update the config file

* fix linting error

* remove unused packages

* remove global runtime var and add flag to storj-sim for mar static dir

* remove debugging lines

* add new config for test data and check if static dir flag is set before passing to mux

* change 'console' to 'marketing' for test data config

* fix linting errors

* update config flag

* Trigger Jenkins

* Trigger CLA
2019-06-11 11:00:59 -04:00
Kaloyan Raev
252c8ac189
Add email to self node info (#2171) 2019-06-11 16:30:28 +03:00
Egon Elbre
61c8c3fb49
satellite/orders: batch update storage node allocations (#2146) 2019-06-10 17:58:28 +03:00
Ivan Fraixedes
cac4637670
satellite: Fix Peer.Run method doc comment (#2147) 2019-06-07 13:40:20 +02:00
Yaroslav Vorobiov
51db703b2b
Console add payments to service (#2100) 2019-06-06 19:07:14 +03:00
Egon Elbre
91a0ece959
Remove bwagreement leftovers (#2140) 2019-06-06 18:16:14 +03:00
Egon Elbre
28a1201590 pkg/bwagreements: remove service (#2138) 2019-06-06 15:57:58 +02:00
Natalie Villasana
25d7dda135 add disqualified check to node selection queries (#2102)
add disqualified check to queries, skip TestStatDB
2019-06-05 20:21:32 -04:00
Michal Niewrzal
b5ac4f3eac
Better metainfo Create/Commit request validation (#2088) 2019-06-05 18:41:02 +02:00
Dennis Coyle
b9d586901e
Value Attribution DBX Model (#2116)
* adds model to satellite dbx

* cleans up model spacing

* generated golang from dbx

* added migration steps

* Added testdata

* changed node_id -> bucket_id

* adds -- NEW DATA -- to testdata

* more testdata changes

* adds -- NEW DATA -- line

* dbx makes the table plural

* missed a singular value_attribution

* restart jenkins

* Update satellitedb.dbx

* adjust to PR comments

* autogenerated dbx models

* restart jenkins
2019-06-05 12:06:14 -04:00
JT Olio
f360097b24 satellite/orders: monitor order settlement more (#2128) 2019-06-05 17:43:41 +02:00
JT Olio
d7f3a5f811
internal,lib,uplink: add monkit task to missing places (#2118)
* internal,lib,uplink: add monkit task to missing places

Change-Id: I490053eee4ed517502f9fe00c6394f0095bd13d0

* Include Monkit

* Add missing context

* Another missing ctx

* More ctx missing

* Linting

* go imports

Change-Id: Ibf0ed072eba339f027727ed8039f7bce1f223fa7

* fix semantic merge conflict

Change-Id: I67fb1f4e7b6cd5e89d69987ed7b3966b7d30ee37
2019-06-05 09:03:11 -06:00
JT Olio
f1641af802 storage: add monkit task to missing places (#2122)
* storage: add monkit task to missing places

Change-Id: I9e17a6b14f7c25bbf698eeecf32785e9add3f26e

* fix tests

Change-Id: Id078276fa3de61a28eb3d01d4e751732ecbb173f

* import order

Change-Id: I814e33755b9f10b5219af37cd828cd75eb3da1a4

* remove part of other commit

Change-Id: Idaa4c95cd65e97567fb466de49718db8203cfbe1
2019-06-05 16:23:10 +02:00
JT Olio
ccb158c99b
pkg/auth: add monkit task to missing places (#2123)
What: add monkit.Task to a bunch of functions that are missing it

Why: this will significantly help our instrumentation, data collection, and tracing about what's going on in the network
2019-06-05 07:47:01 -06:00
JT Olio
d02427e41a db: set max open conns, conn max lifetime, add db stat monitoring (#2117) 2019-06-04 23:30:21 +02:00
Yingrong Zhao
09b0c2a630
create db implementation for offer table (#2031)
* init marketing service

Fix linting error

Create offerdb implementation

Create offers service

Add update method

Create offer table and migration

Fix linting error

fix conflicts

Insert new data

Change duration to have clear indication to be based on days

add error wrapper

Change from using uuid to int for id field

* Create Marketing service

* make error virable name more readable

* add condition in update service method to check offer status

* generate lock file

Change get to listAllOffers

* Add method for getting current offer

wip

* add check for expires_at in update method

* Fix conflicts

* add copyright header

* Fix linting error

* only allow update to active offers

* add isDefault argument to GetCurrent

* Update lock file

* add migration file

* finish migrate for adding credit_in_cents for both award and invitee

* save 100 years as expiration date for default offers

* create crud test for offers

* add GetCurrent test

* modify doc

* Fix GetCurrent to work with default offer

* fix linting issue

* add more tests and address feedbacks

* fix migration file

* add type column back to match with mockup design

* add type column back to match with mockup design

* move doc changes to new pr

* add comments

* change GetCurrent to GetCurrentByType

* fix typo
2019-06-04 15:17:01 -04:00
paul cannon
d15eaed588 add capability of logging all GRPC calls/payloads (#2067) 2019-06-04 14:55:24 +02:00
JT Olio
29d16b4d68 satellite: add monkit task to missing places (#2108) 2019-06-04 13:55:37 +02:00
Cameron
e077b0d380
rename VetNode to IsVetted (#2097)
* rename VetNode to IsVetted
2019-06-03 10:53:30 -04:00
Egon Elbre
b8e0ac6377
satellitedb/overlaycache: avoid tx leak in case of error (#2095) 2019-06-03 17:37:43 +03:00
Yaroslav Vorobiov
6809129e6f
Console add stripe service (#2080) 2019-06-03 16:46:57 +03:00
Yaroslav Vorobiov
79a0085103
Fix storage and object_count calculations on console (#2081) 2019-06-03 15:54:06 +03:00
Natalie Villasana
6db9388082 add disqualified column to nodes table (#2086)
* add disqualified column to nodes table, update migrate script and testdata

* fix crazy formatting of postgres.v25.sql
2019-05-30 17:38:23 -04:00
Cameron
590b1a5a1d
Satellite voucher service (#2043)
* set up voucher service skeleton, basic test

* add VetNode db method

* basic test for VetNode

* encode and sign voucher functions

* fill out and sign vouchers

* test pass/fail voucher request

* match EncodeVoucher to other Encode functions
2019-05-30 15:52:33 -04:00
aligeti
934ebf9cbf
Added the irreparable repair functionality (#1955)
* Added the irreparable repair functionality
2019-05-30 11:18:20 -04:00
ethanadams
16e3b77cf5
Enable Scopelint Linter (#2049)
* added scopelint and correcte issues found

* corrected scopelint issue

* made updates based on Ivan's suggestions

Most were around naming conventions
Some were false positives, but I kept them since the test.Run could eventually be changed to run in parallel, which could cause a bug
Others were false positives.  Added // nolint: scopelint
2019-05-29 09:30:16 -04:00
ethanadams
268dc6b7e4
Enable gocritic linter (#2051)
* first round cleanup based on go-critic

* more issues resolved for ifelsechain and unlambda checks

* updated from master and gocritic found a new ifElseChain issue

* disable appendAssign. i reports false positives

* re-enabled go-critic appendAssign and disabled lint check at code line level

* fixed go-critic lint error

* fixed // nolint add gocritic specifically
2019-05-29 09:14:25 -04:00
Michal Niewrzal
03bae48ad6
Remove unused parameter (#2069) 2019-05-29 11:15:15 +02:00
Michal Niewrzal
f731267e8c Per-project usage limiting (#2036)
What: Changes to support custom usage limit for the project. With this implementation by default project usage limit is taken from configuration flag. If project DB field usage_limit will be set to value larger than 0 it will become custom usage limit and we will be used to verify is limit was exceeded.

Whats changed:

usage_limit (bigint) field added to projects table (with migration)
things related to project usage moved from metainfo endpoint to project usage type
accounting.ProjectAccounting extended with GetProjectUsageLimits() method
Why: We need to have different usage limits per project. https://storjlabs.atlassian.net/browse/V3-1814
2019-05-28 09:36:52 -06:00
Stefan Benten
8912d7149c
Fixes Auth Issue (#2064) 2019-05-28 16:32:51 +02:00
Natalie Villasana
aa6ff17b70 add Reverify to auditing (#2041)
Co-authored-by: Maximillian von Briesen <mobyvb@gmail.com>
Co-authored-by: Kaloyan Raev <kaloyan@storj.io>
2019-05-27 14:13:47 +03:00
Fadila
8f078d2841 Delete pending audit entries from containment DB when files get deleted (#2050) 2019-05-24 15:56:08 -04:00
Jeff Wendling
1bd52b9f90 server side macaroons (#1945)
What: Adds macaroon support to the server side

Why: So that api keys are now macaroons
2019-05-24 10:51:27 -06:00
littleskunk
73a039afa7
named log (#2034) 2019-05-24 00:47:05 +02:00
Kaloyan Raev
de8070730a Save hash of expected share from missing audited node (#2025) 2019-05-23 16:07:19 -04:00
Jeff Wendling
b0ca12e31e
log errors only when an error happens (#2029) 2019-05-23 17:20:01 +00:00
Natalie Villasana
8b31c4b91f
remove unnecessary containmentDB wrapper (#2027) 2019-05-23 10:37:23 -04:00
Yingrong Zhao
f9045f8385 Create offer table in satellitedb for free credit program (#1975) 2019-05-22 23:41:55 +02:00
Cameron
4058c29ca4
filter duplicate node IPs (#1890)
* add last_ip field to dbx model node, generate dbx

* add last_ip to node proto, generate pb

* migrate

* resolve address in transport.DialNode, update lastIp in cache.UpdateAddress

* use net.SplitHostPort to isolate host address from port

* define DistinctIPs flag

* add test for GetIP

* select last_ip when querying for nodes

* if distinctIPs flag == true, query for nodes with distinct IPs

* some basic tests

* change last_ip to field 14 in proto

* remove comments

* check err

* change distinctIPs to distinctIP

* exclude IPs from newNodes in query for reputable nodes

* add index on last_ip

* only add to excludedIPs if flag is true

* test half new nodes returns distinct IPs

* fix alignment

* add test

* rework ip filter query, add retry logic, add switch for database driver

* add retry to SelectNewNodes

* change discovery intervals so IPs don't get overwritten

* remove TestGetIP

* edit updating node stats in test

* split exclude into nodeIDs and IPs

* separate non-distinct IP query into other function

* trigger checks

* remove else block
2019-05-22 16:06:27 -04:00
Natalie Villasana
acfb59ae48
adds db layer functions for containment (#1972) 2019-05-22 10:50:22 -04:00
Bill Thorp
a6c4019288
using DB time only (#2018)
* using DB time only, using UTC
2019-05-21 12:50:55 -04:00
Egon Elbre
9c23c2d427 db: set max idle connections higher to avoid redialing all the time (#1991) 2019-05-21 17:30:06 +03:00
Egon Elbre
42562429f5 Optimize KnownUnreliableOrOffline SQL query (#1968) 2019-05-19 17:10:46 +02:00
Bill Thorp
91721f63ba
Bt/repair no nodes (#1974)
* handle cases where repair is equal to total
2019-05-17 15:02:40 -04:00
Egon Elbre
859e04f8f3 satellite/console: use require.NoError in tests (#1986) 2019-05-17 09:31:14 -04:00
Egon Elbre
748b2b05a4 satellite/mailservice: wait for goroutines to finish (#1985)
* satellite/mailservice: wait for goroutines to finish

* fix tests
2019-05-17 14:29:35 +02:00
Maximillian von Briesen
cc020dfdea
Create containment mode database table and migrate scripts (#1970) 2019-05-16 10:11:15 -04:00
Yaroslav Vorobiov
2d2301d5ff
Console buckets page (#1847) 2019-05-16 13:43:46 +03:00
Bogdan Artemenko
38acc4dba6
Resend account activation email (#1961) 2019-05-15 11:28:36 +03:00
Jess G
8518618b7a
add postgres support to storj-sim (#1908)
* add flags to sotrj-sim for SA dbs

* add schema to postgres

* add createschema with parse to sa

* add metainfo db postgres support

* add kv default as bolt

* add debug log to see db source

* add env var for postgres to test-sim.sh

* fix lint errs

* dynamically add postgres to args

* add postgres to integration tests

* add sqlite and postgres integration jenkins

* fix db name

* merge integration tests into one step

* test integration tests w/psql

* try using different schema

* debug failure

* use correct host for running storj-sim

* rm sqlite integration

* add back integration
2019-05-14 08:13:18 -07:00
littleskunk
13c812dbbd fix node selection (#1958) 2019-05-13 19:55:51 -04:00
Bogdan Artemenko
afddaba551
Removed redundant logs from satellite. (#1954) 2019-05-13 19:21:20 +03:00
Bogdan Artemenko
f46487b015
ResetPassword Table and all CRUD methods. (#1916) 2019-05-13 18:53:52 +03:00
Jennifer Li Johnson
5395ff5fe6
Refactor accountingdb interface (#1897)
* splits accounting db into storagenodeaccounting and projectaccounting interfaces and renames methods to match
2019-05-10 15:05:42 -04:00
paul cannon
02be91b029
real-time tracking of space used per project (#1910)
Ran into difficulties trying to find the ideal solution for sharing
these counts between multiple satellite servers, so for now this is a
dumb solution storing recent space-usage changes in a big dumb in-memory
map with a big dumb lock around it. The interface used, though, should
allow us to swap out the implementation without much difficulty
elsewhere once we know what we want it to be.
2019-05-09 20:39:21 -05:00
Natalie Villasana
b48f584cea
repair checker resumes iterating where left off (#1879) 2019-05-08 13:59:50 -04:00
Bill Thorp
ea978dd674
hopefully sensible satellite defaults (#1888)
* hopefully sensible satellite defaults
2019-05-07 10:44:47 -04:00
Bill Thorp
d903865346
audit and uptime ratios default to 1 2019-05-07 10:23:08 -04:00
Egon Elbre
ecde1bd251 jenkins: use -race for check-imports and fix dbx check (#1873)
What: Use -race for check-imports, this means it will use the cached build files.

Why:
2019-05-01 09:44:12 -06:00
Bill Thorp
6ece4f11ad
moved invalid/offline back into SQL (#1838)
* moved invalid/offline back into SQL, removed GetAll()
2019-05-01 09:45:52 -04:00
Bill Thorp
f6b0a6b269
made inspector segment health care about healthy nodes (#1858) 2019-04-29 09:22:51 -04:00
Bill Thorp
2367918331 removed unused .List() method (#1853)
* removed unused .List()

* removed unused test
2019-04-26 17:41:13 +02:00
Egon Elbre
db939d37ec
cover all the things (#1818) 2019-04-26 16:39:11 +03:00
Bill Thorp
a11dc76169
wired up online config (#1827)
* wired up online config
2019-04-26 08:15:06 -04:00
Nikolay Yurchenko
42116594d1
[V3-1614] Account settings rework (#1831) 2019-04-26 01:06:19 +10:00
Michal Niewrzal
fe3dfc1587
Move pointerdb.Service to satellite (#1826) 2019-04-25 10:46:32 +02:00
Bill Thorp
cd4a3e06d8
wired up IsHealthy to config (#1820)
* wired up IsHealthy to config
2019-04-23 18:45:50 -04:00
Fadila
8ddf481b33 Checker: invalid and offline nodes search update (#1812)
* simplified invalid and offline login into getMissingPieces
2019-04-23 16:54:39 -04:00
Bill Thorp
2029c558af
updated audit configs (#1814)
updated audit configs
2019-04-23 16:47:11 -04:00
Egon Elbre
fe2bf7d4a6 fix saving uplink public key on satellite (#1797)
* fix save into cert records

* fix transaction logic

* test shouldn't be flaky anymore

* add missing cause method

* restart jenkins
2019-04-23 15:48:57 -04:00
Cameron
9f6b010748 rollup interval fixes (#1739)
* set all intervals to UTC in rollupStats map, only delete latest day after both rollups

* clean up usage of interval, use intervalEndTime rather than createdAt

* change some variable names, add comments

* add flag for tally deletion

* adds deletetallies flag to testplanet

* space

* Removes println:

* adds test for deletes false
2019-04-23 15:21:30 -04:00
Yehor Butko
4185f2d8ac
satellite bugfix (#1816)
* satellite bugfix
2019-04-23 17:46:54 +03:00
Yaroslav Vorobiov
75870005b2
api keys change encoding to base32 (#1813) 2019-04-23 16:00:31 +03:00
Yaroslav Vorobiov
f7409ab52b
Change time params to UNIX timestamp in console usage-report query (#1757) 2019-04-23 15:56:15 +03:00
Egon Elbre
f7ed63a119
handle database error checks properly (#1796) 2019-04-23 14:13:57 +03:00
Egon Elbre
bdd0d778eb
interface tests belong to the interface, not the implementation (#1794) 2019-04-23 11:47:16 +03:00
Natalie Villasana
33ed8ca4c9
skips TestUploadDownloadOneUplinksInParallel and TestParallel (#1806) 2019-04-22 16:04:17 -04:00
Bill Thorp
e8e6eda0f5 non-unique path for injured segments fix (#1803) 2019-04-22 14:55:48 -04:00
Natalie Villasana
37d95f164c
deletes duplicates from existing injuredsegments table (#1781) 2019-04-22 12:41:53 -04:00
Natalie Villasana
8d1f614662 removes unused queue code, moves queue_test.go to repairqueue_test.go in satellitedb dir (#1783) 2019-04-22 13:35:52 +03:00
Kaloyan Raev
8fc5fe1d6f
Refactor pb.Node protobuf (#1785) 2019-04-22 12:07:50 +03:00
Egon Elbre
bbeb0eac4e
disable pdb logging, it's usually not that useful (#1790) 2019-04-19 16:30:55 +03:00
Bogdan Artemenko
6e88f410e7
Implemented send email for support from static email. (#1780)
* Implemented send email for support from static email.

* Removed redundant 'LetUsKnow' reference.
2019-04-19 13:51:11 +03:00
Egon Elbre
5b3c146d8a
Check context cancellation more nicely (#1752) 2019-04-17 13:09:44 +03:00
Bill Thorp
17a227e6e9
refactor injuredsegments db so that we can't have duplicates (#1717)
made repairqueue not use a true queue, forbid duplicates
2019-04-16 14:14:09 -04:00
Stefan Benten
0cefb12a28
Set Default Node Type to invalid, to avoid cache issues (#1746) 2019-04-12 15:30:19 +02:00
Egon Elbre
0eee46524d update linter to v1.16 (#1741)
* update to v1.16

* use full version number

* fix linter issues

* fix comment

* nicer comment

* restart travis

* restart travis
2019-04-11 13:32:40 -04:00
Bogdan Artemenko
6a50b187eb
Implemented password reset on satellite console web. (#1665) 2019-04-10 22:16:10 +03:00
Stefan Benten
bae4c820ee
Add Version Information into KAD Network and SatelliteDB & Change Selection Process (#1648)
* Initial Webserver Draft for Version Controlling

* Rename type to avoid confusion

* Move Function Calls into Version Package

* Fix Linting and Language Typos

* Fix Linting and Spelling Mistakes

* Include Copyright

* Include Copyright

* Adjust Version-Control Server to return list of Versions

* Linting

* Improve Request Handling and Readability

* Add Configuration File Option
Add Systemd Service file

* Add Logging to File

* Smaller Changes

* Add Semantic Versioning and refuses outdated Software from Startup (#1612)

* implements internal Semantic Version library

* adds version logging + reporting to process

* Advance SemVer struct for easier handling

* Add Accepted Version Store

* Fix Function

* Restructure

* Type Conversion

* Handle Version String properly

* Add Note about array index

* Set temporary Default Version

* Add Copyright

* Adding Version to Dashboard

* Adding Version Info Log

* Renaming and adding CheckerProcess

* Iteration Sync

* Iteration V2

* linting

* made LogAndReportVersion a go routine

* Refactor to Go Routine

* Add Context to Go Routine and allow Operation if Lookup to Control Server fails

* Handle Unmarshal properly

* Linting

* Relocate Version Checks

* Relocating Version Check and specified default Version for now

* Linting Error Prevention

* Refuse Startup on outdated Version

* Add Startup Check Function

* Straighten Logging

* Dont force Shutdown if --dev flag is set

* Create full Service/Peer Structure for ControlServer

* Linting

* Straighting Naming

* Finish VersionControl Service Layout

* Improve Error Handling

* Change Listening Address

* Move Checker Function

* Remove VersionControl Peer

* Linting

* Linting

* Create VersionClient Service

* Renaming

* Add Version Client to Peer Definitions

* Linting and Renaming

* Linting

* Remove Transport Checks for now

* Move to Client Side Flag

* Remove check

* Linting

* Transport Client Version Intro

* Adding Version Client to Transport Client

* Add missing parameter

* Adding Version Check, to set Allowed = true

* Set Default to true, testing

* Restructuring Code

* Uplink Changes

* Add more proper Defaults

* Renaming of Version struct

* Dont pass Service use Pointer

* Set Defaults for Versioning Checks

* Put HTTP Server in go routine

* Add Versioncontrol to Storj-Sim

* Testplanet Fixes

* Linting

* Add Error Handling and new Server Struct

* Move Lock slightly

* Reduce Race Potentials

* Remove unnecessary files

* Linting

* Add Proper Transport Handling

* small fixes

* add fence for allowed check

* Add Startup Version Check and Service Naming

* make errormessage private

* Add Comments about VersionedClient

* Linting

* Remove Checks that refuse outgoing connections

* Remove release cmd

* Add Release Script

* Linting

* Update to use correct Values

* Change Timestamp handling

* Adding Protobuf changes back in

* Adding SatelliteDB Changes and adding Storj Node Version to PB

* Add Migration Table

* Add Default Stats for Creation

* Move to BigInt

* Proper SQL Migration

* Ensure minimum Version is passed to the node selection

* Linting...

* Remove VersionedClient and adjust smaller changes from prior merge

* Linting

* Fix PB Message Handling and Query for Node Selection

* some future-proofing type changes

Change-Id: I3cb5018dcccdbc9739fe004d859065992720caaf

* fix a compiler error

Change-Id: If66bb92d8b98e31cd618ecec9c6448ab9b037fa5

* Comment on Constant for Overlay

* Remove NOT NULL and add epoch call as function

* add versions to bootstrap and satellites

Change-Id: I436944589ea5f21600cdd997742a84fe0b16e47b

* Change Update Migration

* Fix DB Migration

* Increase Timeout temporarily, to see whats going on

* Remove unnecessary const and vars
Cleanup Function calls from deprecated NodeVersion struct

* Updated Protopuf, removed depcreated Code from Inspector

* Implement NodeVersion into InfoResponse

* Regenerated locked.go

* Linting

* Fix Tests

* Remove unnecessary constant

* Update Function and Flag Description

* Remove Empty Stat Creation

* return properly with error

* Remove unnecessary struct

* simplify migration step

* Update Inspector to return Version Info

* Update local Endpoint Version Handling

* Reset Travis Timeout

* Add Default for CommitHash

* single quotes
2019-04-10 08:04:24 +02:00
Yehor Butko
ba891538af
V3-1399 Error messages (#1715)
* V3-1399 Error messages
2019-04-10 03:15:12 +03:00
Yaroslav Vorobiov
71843632a0
Console add bucket usage report (#1706) 2019-04-09 19:14:19 -04:00
Cameron
32192aca10
convert times to UTC before entering as utimestamp (#1708) 2019-04-09 15:12:58 -04:00
Jess G
55bfe507e5
add tests for tally and accounting db (#1675)
* add tally calcAtRestData test

* add account db tests

* move tests from sadb to accounting pkg

* add func to calcEncSize

* remove unneeded pad changes

* update comment

* add changes to comments
2019-04-09 09:48:35 -04:00
Michal Niewrzal
a684b7b953
Test number of pieces lower than repair threshold (#1683) 2019-04-09 15:31:19 +02:00
Yaroslav Vorobiov
2218793c55 Added error logging to account activation handler (#1647) 2019-04-09 08:20:29 -04:00
Natalie Villasana
2533aa1ab1
logs dbx constraint violation error (#1712) 2019-04-08 18:03:48 -04:00
Jess G
5dfe28a8c3
use allocated instead of settled (#1700)
* use allocated instead of settled

* add expansion factor

* changes per CR
2019-04-08 14:35:54 -07:00
Bill Thorp
061deb6add
only showing online node count (#1702) 2019-04-08 16:55:04 -04:00
Natalie Villasana
0d006661f4
returns an error if tx.Get_Node_By_Id returns nil node (#1698) 2019-04-08 13:52:53 -04:00
Maximillian von Briesen
bb3b4e4816 Data repair integration test (#1582) 2019-04-08 13:33:47 -04:00
Yehor Butko
f4430f9063
Satellite frontned empty states (#1682)
* V3-1421 Empty State Pages in no Project Case
2019-04-05 18:08:14 +03:00
Kaloyan Raev
92c1121072
Fix "Number of valid pieces is lower then success threshold" (#1679) 2019-04-05 13:19:20 +03:00
Michal Niewrzal
5ea797889b
Store inline bandiwdth (#1619) 2019-04-05 09:42:56 +02:00
Alexander Leitner
06e2e7a1e0
Filehealth cli (#1649)
* print CSV formatted data for checking object and segment health
2019-04-04 13:11:19 -04:00
Kaloyan Raev
bfdee78f05
Introduce NodeDossier type and cleanup overlay.DB interface (#1626)
Co-authored-by: Natalie Villasana <navillasa@gmail.com>
Co-authored-by: Bill Thorp <bill3000@hotmail.com>
2019-04-04 19:34:36 +03:00
Jess G
1638d96b16
update ProjectStorageTotals func to get all records for project (#1657)
* update ProjectStorageTotals func to get all records for project

* fix var name

* update test to catch bug

* fix spelling

* modify query so we only have to make 1
2019-04-04 09:01:04 -07:00
JT Olio
09be9964eb internal/version: do version checks much earlier in the process initialization, take 2 (#1666)
* internal/version: do version checks much earlier in the process initialization, take 2

Change-Id: Ida8c7e3757e0deea0ec7aea867d3d27ce97dc134

* linter and test failures

Change-Id: I45b02a16ec1c0f0981227dc842e68dbdf67fdbf4
2019-04-04 17:40:07 +02:00
Jennifer Li Johnson
8549421385
Remove bw from tally service + query bandwidth in rollup service (#1618) 2019-04-04 11:20:59 -04:00
Yaroslav Vorobiov
b38b87cb14
Console usage rollup api (#1664) 2019-04-04 17:56:20 +03:00
Michal Niewrzal
6d86610bcc
Fix UseSerialNumber panic (#1663)
This change fixes UseSerialNumber and case when given serial number doesn't exists in DB.
2019-04-04 16:42:01 +02:00
Stefan Benten
2cf86703a3
Add Versioning Server (#1576)
* Initial Webserver Draft for Version Controlling

* Rename type to avoid confusion

* Move Function Calls into Version Package

* Fix Linting and Language Typos

* Fix Linting and Spelling Mistakes

* Include Copyright

* Include Copyright

* Adjust Version-Control Server to return list of Versions

* Linting

* Improve Request Handling and Readability

* Add Configuration File Option
Add Systemd Service file

* Add Logging to File

* Smaller Changes

* Add Semantic Versioning and refuses outdated Software from Startup (#1612)

* implements internal Semantic Version library

* adds version logging + reporting to process

* Advance SemVer struct for easier handling

* Add Accepted Version Store

* Fix Function

* Restructure

* Type Conversion

* Handle Version String properly

* Add Note about array index

* Set temporary Default Version

* Add Copyright

* Adding Version to Dashboard

* Adding Version Info Log

* Renaming and adding CheckerProcess

* Iteration Sync

* Iteration V2

* linting

* made LogAndReportVersion a go routine

* Refactor to Go Routine

* Add Context to Go Routine and allow Operation if Lookup to Control Server fails

* Handle Unmarshal properly

* Linting

* Relocate Version Checks

* Relocating Version Check and specified default Version for now

* Linting Error Prevention

* Refuse Startup on outdated Version

* Add Startup Check Function

* Straighten Logging

* Dont force Shutdown if --dev flag is set

* Create full Service/Peer Structure for ControlServer

* Linting

* Straighting Naming

* Finish VersionControl Service Layout

* Improve Error Handling

* Change Listening Address

* Move Checker Function

* Remove VersionControl Peer

* Linting

* Linting

* Create VersionClient Service

* Renaming

* Add Version Client to Peer Definitions

* Linting and Renaming

* Linting

* Remove Transport Checks for now

* Move to Client Side Flag

* Remove check

* Linting

* Transport Client Version Intro

* Adding Version Client to Transport Client

* Add missing parameter

* Adding Version Check, to set Allowed = true

* Set Default to true, testing

* Restructuring Code

* Uplink Changes

* Add more proper Defaults

* Renaming of Version struct

* Dont pass Service use Pointer

* Set Defaults for Versioning Checks

* Put HTTP Server in go routine

* Add Versioncontrol to Storj-Sim

* Testplanet Fixes

* Linting

* Add Error Handling and new Server Struct

* Move Lock slightly

* Reduce Race Potentials

* Remove unnecessary files

* Linting

* Add Proper Transport Handling

* small fixes

* add fence for allowed check

* Add Startup Version Check and Service Naming

* make errormessage private

* Add Comments about VersionedClient

* Linting

* Remove Checks that refuse outgoing connections

* Remove release cmd

* Add Release Script

* Linting

* Update to use correct Values

* Move vars private and set minimum default versions for testing builds

* Remove VersionedClient

* Better Error Handling and naked return removal

* Straighten the Regex and string conversion

* Change Check to allows testplanet and storj-sim to run without the
need to pass an LDFlag

* Cosmetic Change to Dashboard

* Cleanup Returns and remove commented code

* Remove Version Check if no build options are passed in

* Pass in Config Values instead of Pointers

* Handle missed Error

* Update Endpoint URL

* Change Type of Release Flag

* Add additional Logging

* Remove Versions Logging of other Services

* minor fixes

Change-Id: I5cc04a410ea6b2008d14dffd63eb5f36dd348a8b
2019-04-03 21:13:39 +02:00
Maximillian von Briesen
6028d8c3de
Fix repair order limit creation (#1650)
* fix repair order limit creation (use piece size instead of share size)
* fix counter in CreateGetRepairOrderLimits
2019-04-03 09:17:32 -04:00
Jess G
d51bdf14df
project usage limiting (#1561)
* reorg uplink cmd files for consistency

* init implementation of usage limiting

* Revert "reorg uplink cmd files for consistency"

This reverts commit 91ced7639bf36fc8af1db237b01e233ca92f1890.

* add changes per CR comments

* fix custom query to use rebind

* updates per convo about what to limit on

* changes per comments

* fix syntax and comments

* add integration test, add db methods for test

* update migration, add rebind to query

* update testdata for psql

* remove unneeded drop index statement

* fix migrations, fix calculate usage limit

* fix comment

* add audit test back

* change methods to use bucketName/projectID, fix tests

* add changes per CR comments

* add test for uplink upload and err ssg

* changes per CR comments

* check get/put limit separately
2019-04-02 11:21:18 -07:00
Simon Guindon
d3885b7b78
Change crypto/rand to use math/rand in pgutil (#1589)
* Added retry logic and tests for handling if crypto/rand fails.

* Added retry logic and tests for handling if crypto/rand fails.

* Fixing linting error.

* Removing use of `crypto/rand` for use of `math/rand`.

* Changing code comment about why we ignore this error.
2019-04-02 12:52:25 -04:00
Alexander Leitner
59b3eb190e
API endpoint checking object and segment health from the satellite (#1546)
* Generate protobuf for checking object and segment health

* Initial cmd client for health check

* Register the health inspector on the satellite peer
2019-04-02 10:55:58 -04:00
Michal Niewrzal
f80750693c Store bandwidth from orders on satellite (#1586) 2019-04-01 16:14:58 -04:00
Kaloyan Raev
034c283eda Set last_contact_success when a node is initially added to overlay cache (#1632) 2019-04-01 12:42:06 -06:00
Cameron
6d43832c4f
record bucket data into bucket_storage_tally table (#1595)
* add MetadataSize to stats

* add logic for accumulating bucket stats in calculateAtRestData

* rename stats to BucketTally, move to accounting package

* define method on accountingDB for inserting bucketTallies

* insert bucketTallies into bucket_storage_tally table
2019-04-01 09:42:17 -04:00
Egon Elbre
872bd5d7c1
Remove pointerdb.Server (#1609) 2019-03-30 13:21:49 +02:00
paul cannon
3b12b5e85c
non-racy CreateEntryIfNotExists() (#1611)
This changes semantics slightly! with this change,
CreateEntryIfNotExists() will do a cache Update with every node passed
in, whether it exists or not. Update() already does a race-free upsert
operation, so that change removes the problematic race in
CreateEntryIfNotExists(). As far as I can tell, this semantic change
doesn't break any expectations of callers, and shouldn't affect
performance in a significant way, as we already have an awful lot of
round-trips to the db either way. But if I've misunderstood the
intention of the method, someone ought to catch it during review.
2019-03-29 15:07:07 -06:00
Yehor Butko
66a7d6bf3a
Satellite password check simplification (#1610) 2019-03-29 19:42:20 +02:00
Egon Elbre
9dbc7c719e
fix error handling in GetByProjectID (#1604) 2019-03-29 14:55:05 +02:00
Egon Elbre
de15a4fdcf
remove utils.CombineErrors and utils.ErrorGroup (#1603) 2019-03-29 14:30:23 +02:00
Yaroslav Vorobiov
0ce6d4ab81
Project api auth update (#1572) 2019-03-29 14:13:37 +02:00
Kaloyan Raev
9fb99c8484
Ensure serial number is not saved if error during order limit creation (#1602) 2019-03-29 11:53:53 +02:00
Kaloyan Raev
f9ba935286
Merge overlay_cache_nodes into nodes table (#1581) 2019-03-29 10:53:43 +02:00
Egon Elbre
be06fdfd6c Create orders.Service (#1593) 2019-03-28 22:09:23 +02:00
Jennifer Li Johnson
b3a8beb1bc
Updates accounting tables models and migration (#1584)
* go gen

* undo changes to bucket usage

* update locked

* spacing

* moves changes to migration v11

* minor changes to fix lint and test err

* change sql to fix errs
2019-03-27 22:46:49 -04:00
Yehor Butko
86bf3dee9f
V3-1307 combine first and last name to full name (#1569)
* V3-1307 combine first and last name to full name
2019-03-27 14:33:32 +02:00
Michal Niewrzal
bfdfebbde2
Satellite orders receiving (#1564)
This change adds satellite endpoint for receiving OrderLimits sent by storage node.
Change includes:
* wire up orders sender in storage node (also in testplanet)
* saving serial number for OrderLimit in serial_numbers table
* satellite endpoint for receiving, verifying and storing OrderLimit and Order serial number
* initial implementation for Orders DB
* basic test for sending orders to satellite
2019-03-27 11:24:35 +01:00
Natalie Villasana
5b48a48a79
adds test for correct download with non-critical amount of nodes offline (#1574) 2019-03-26 14:09:44 -04:00
Yaroslav Vorobiov
b2b86bee56
[V3-1366] Email project name bug (#1571) 2019-03-26 17:56:16 +02:00
Stefan Benten
3db903fe40
Move from Unique to Index on SatelliteDB (#1563)
* Move from Unique to Index

* Remove Index

* Make some more Indexes Unique and adjust migration

* Fix Migration Statements

* Fix Typo

* Fix Migration of older Table

* Exchange DROP statement

* Remove "if not exists"

* Revert Change in old Migration
2019-03-26 11:34:30 +01:00
Kaloyan Raev
d1639c4157 Merge statdb pkg into overlay pkg (#1570) 2019-03-25 18:25:09 -04:00
Yaroslav Vorobiov
cce6fa8fd3
added LOGIN auth scheme to satellite mail service (#1552) 2019-03-24 02:08:41 +02:00
Egon Elbre
94e79eda6d
remove overlay endpoint (#1521) 2019-03-23 10:06:11 +02:00
Natalie Villasana
0fa1d536e7
removes pingbackTimeout (#1556) 2019-03-22 16:06:57 -04:00
Egon Elbre
297b393dfd database for tracking information without storing orders / agreements (#1551)
* initial schema declaration

* fix typo

* split database

* fix syntax errors and generate code

* add queries for serial number manipulation

* update comment

* add migration script

* fix comments

* add new data

* fix bucket id
2019-03-22 11:54:22 -07:00
Egon Elbre
694b6dc1da
make tests run faster (#1553) 2019-03-22 15:14:17 +02:00
Natalie Villasana
61ee04d363
adds pingbackTimeout to kademlia endpoint (#1518) 2019-03-19 14:30:27 -04:00
Yehor Butko
c6f8d82978
V3-1319 Add an authorization token field for account creation backend (#1492)
* V3-1319 Add an authorization token field for account creation on the satellite UI backend
2019-03-19 19:55:43 +02:00
Natalie Villasana
a09f568fdb
audit service won't hang on bad storage node conns (#1503) 2019-03-19 13:37:26 -04:00
Michal Niewrzal
b05cf05649
Restrict slash in bucket name (#1524) 2019-03-19 15:37:28 +01:00
paul cannon
cd91a22e0f better way to check if err is ErrorCode_NoRows (#1453)
* better way to check if err is ErrorCode_NoRows
2019-03-18 20:15:27 -04:00
Egon Elbre
a24c74c502 fix message formatting (#1512) 2019-03-18 18:02:37 +01:00
Egon Elbre
05d148aeb5
Storage node and upload/download protocol refactor (#1422)
refactor storage node server
refactor upload and download protocol
2019-03-18 12:55:06 +02:00
Cameron
c7ffbe1c28
Add ability to view irreparable segments on satellite (#1448)
* define irreparable inspector protobuf

* add IrreparableDB method GetLimited

* fill out irreparable inspector API

* add IrreparableInspector server to satellite, fix small error

* refactor IrreparableDB to use pb.IrreparableSegment instead of irreparable.RemoteSegmentInfo
2019-03-15 16:21:52 -04:00
Bill Thorp
665fd33e3c
Repair queue isolation level fix (#1466)
Implemented custom SQLite and Postgres Repairqueue Dequeue handlers
2019-03-14 17:12:47 -04:00
Bill Thorp
52e829c6de
framework for deleting expired Orders from Satellite (#1436)
framework for deleting old bandwidth agreements
2019-03-12 16:57:21 -04:00
Dylan Lott
59f1e267c9
Removes concept of email from kademlia metadata (#1435)
* Removes concept of Email from Kademlia

* Removes kad email

* adds emails back to operator config for satellite

* replace operator configs in testplanet
2019-03-12 14:05:18 -06:00
Stefan Benten
12c5f42c30
Add Application Name to PostgreSQL Connection String (#1451)
* Add Application Name to PostgreSQL Connection String,
if not existing already

* Relocate Check

* Handle Application Name only for Postgres

* URL Encode

* Relocate Function into pgutil

* Fix Error, when ApplicationName is set

* Rename parameter

* Rename Check Value

* Straightline Comment

* Remove fmt Dependency

* Fix Linting Recommendation
2019-03-12 14:29:13 +01:00
Bill Thorp
1f4e1a3ab0
framework to delete raw accounting tallies (#1439)
framework to delete raw accounting tallies, renamed Query
2019-03-12 09:08:23 -04:00
littleskunk
b26e660ec3 add index created_at on bwagreements (#1438) 2019-03-11 08:09:50 +02:00
Yehor Butko
3ab696164f
Satellite console new activation flow (#1421)
Satellite console new activation flow
2019-03-08 16:01:11 +02:00
Jess G
193a70f0a6
add private listener to grpc server (#1398)
* add private listener to grpc server

* add changes per init CR

* fix server.close

* add insecure grpc connection, update logs msg

* fix tests, move insecure client

* add private ports to storj-sim, add insecure client to other inspectors

* add ports to test so there arent conflicts

* fix lint err

* fix node started log msg, close public listener

* remove commented out line
2019-03-07 13:19:37 -05:00
Yaroslav Vorobiov
9cf56c03fa
Bucket usage info rollup table creation and implementation (#1340) 2019-03-06 17:54:48 +02:00
Yehor Butko
af86d8e368
V3-1095 Update service to send invitations in project (#1412)
* V3-1095 Update service to send invitations in project
2019-03-06 17:42:19 +02:00
Yehor Butko
3e2c101bd8
V3-1152 Node bootstrap web backend (#1327)
* V3-1152 Node bootstrap
2019-03-05 12:38:21 +02:00
Yaroslav Vorobiov
a30ba4eca8
Add mail service to the satellite (#1302) 2019-03-02 17:22:20 +02:00
Dylan Lott
9abf191b01
Change payments to use statDB for wallet addresses rather than Overlay cache (#1368)
* Wiring up DumpNodes response for Inspector

* Finalize everything and test that it works

* Get Count and DumpNodes working for Overlay Cache

* WIP updating payment rollup to check statDB instead of overlay

* FIrst pass at updating statDB to take wallet and email

* Passing tests

* use pb.NodeOperator instead of Meta struct

* remove TODO

* revert go.mod

* Get SQL migration working correctly

* Changes Meta to Operator in NodeStats struct

* Adds update operator logic for statDB

* Fix db migrate tests - added v5 snapshot

* User friendly msg for missing snapshot version

* Passing tests

* Change node update to happen in discovery instead of in overlay

* Fix logic and update function calls

* Update comment on UpdateOperator interface method

* Update name of parameter

* Change type of argument to UpdateOperator

* Updates statDB tests
2019-03-01 10:46:34 -07:00
JT Olio
2efaa27318 accounting: support null wallet addresses (#1387) 2019-03-01 07:34:46 +02:00
Philip Hutchins
1f452e67dc Adding dockerfile for running the web UI for Satellite (#1366)
* Adding dockerfile for running the web UI for Satellite

* Updating to work with Makefile and from root directory of repo

* Updating satellite ui build process to run in a more production like mode by generating the assets the pulling those into the satellite container

* Updates to allow external traffic to UI, updates to storagenode for identity creation, and logging for bug tracking

* Adding auto cert generation for storagenode

* removing satellite-ui-image from main images flow in Makefile and adding latest tag to docker build for it

* Adding solid defaults, tuning dockerfiles, and moving to standard logging methods

* Updating logging to be more standard

* Updating to logger.Debug

* Removing unused library and unused identity creation code

Change-Id: I956453037e303693ea37f94318180af0ab7984d5
2019-02-28 13:18:53 -07:00
Jennifer Li Johnson
6ecda190cb
Jj/payments cleanup (#1379)
* Removes date from payments csv and re-query of wallet addr

* removes dbx get wallet addr

* switch order in csv
2019-02-28 12:13:59 -05:00
Bill Thorp
373b301736
BWA aliases (#1333)
aliased RBAs and PBAs
2019-02-22 16:17:35 -05:00
Egon Elbre
12261c0624
Satellite migrations (#1301) 2019-02-14 23:55:21 +02:00
Egon Elbre
1a5a9903a2
Move SplitConnstr to a nicer place (#1308) 2019-02-13 23:54:59 +02:00
Egon Elbre
497fb756fe
create dbutil package for migration testing (#1305) 2019-02-13 18:06:34 +02:00
Bogdan Artemenko
ad95d881c7
Implemented Multiple API key deletion. Added Notification for API key deletion result. (#1298)
* Implemented Notification for API key deletion result.

* Moved several API keys deletion to Service side.
2019-02-13 13:34:40 +02:00
Egon Elbre
3beaeebdb6
Update dbx (#1300) 2019-02-12 21:46:03 +02:00
Michal Niewrzal
b2f9453184
Disable Checker subsystem in tests (#1279)
* Disable Checker subsystem in tests

* rename field

* remove sleeps and errgroup.Group
2019-02-11 22:06:39 +01:00
Maximillian von Briesen
f94a2e1161
Node selection method name change (#1296)
* SelectNodes -> SelectStorageNodes
2019-02-11 14:24:51 -05:00
Maximillian von Briesen
c6c23a319b
move node type definition to db level for node selection (#1295) 2019-02-11 11:35:28 -05:00
JT Olio
2a59679766 pkg/transport: require tls configuration for dialing (#1286)
* separate TLS options from server options (because we need them for dialing too)
* stop creating transports in multiple places
* ensure that we actually check revocation, whitelists, certificate signing, etc, for all connections.
2019-02-11 13:17:32 +02:00
JT Olio
ea28a9a581 pkg/transport: fail on bad certificates (#1285) 2019-02-11 13:02:49 +02:00
Yehor Butko
8156d911fa
Updating account activation flow (#1251)
* Updating account activation flow

* Updated integration tests, createUserMutation updated

* removing redundant index

* removed redundant testing code

* review comments fixed
2019-02-11 12:33:56 +02:00
Egon Elbre
17a7f9d002
Enable planet Merging test (#1281) 2019-02-08 22:35:59 +02:00
Egon Elbre
e37e0c1b5f
Fix server config usage (#1282) 2019-02-08 20:57:17 +02:00
Egon Elbre
9c1e299f3c
Ensure everyone sees everyone else (#1275) 2019-02-08 11:25:13 +02:00
paul cannon
c35b93766d
Unite all cryptographic signing and verifying (#1244)
this change removes the cryptopasta dependency.

a couple possible sources of problem with this change:

 * the encoding used for ECDSA signatures on SignedMessage has changed.
   the encoding employed by cryptopasta was workable, but not the same
   as the encoding used for such signatures in the rest of the world
   (most particularly, on ECDSA signatures in X.509 certificates). I
   think we'll be best served by using one ECDSA signature encoding from
   here on, but if we need to use the old encoding for backwards
   compatibility with existing nodes, that can be arranged.

 * since there's already a breaking change in SignedMessage, I changed
   it to send and receive public keys in raw PKIX format, instead of
   PEM. PEM just adds unhelpful overhead for this case.
2019-02-07 14:39:20 -06:00
Bill Thorp
0b35762105
Convert Payments to use SQL, for SUM() and Wallet (#1266)
* payments query no longer DBX, using SQL

* sum in SQL

* removed old function

* fixed rollup test

* wrap errors

* removed DBX code
2019-02-07 15:26:55 -05:00
aligeti
b736ae4823
Store the uplinks public key on the satellite so that it can verify bandwidth requests in the future (#1042)
* integrated with bwagreement & psserver

* integrated with pointerdb

* code review updates

* refactor after code review

* uplinkdb rename to certdb

* Code review changes
2019-02-07 14:22:49 -05:00
Cameron
bfe8060f4d
Rollup tests (#1224)
* createdAt field no longer autoinsert

* rollup tests
2019-02-06 23:16:24 -05:00
Bill Thorp
dc7deab695
go test fail due to consoleql UTC issues (#1198)
* is this right?

* used time.Equal for conceptual clarity
2019-02-06 09:48:30 -05:00
Egon Elbre
bb11d83ed0
Proper planet shutdown (#1249) 2019-02-06 15:19:14 +02:00
Michal Niewrzal
aa3836c06a
Add checker irreparable test (#1246)
* Add checker irreparable test

* use RepairUnixSec

* wait for second check

* use storagenode.ID()

* Update comments
2019-02-06 14:03:38 +01:00
Egon Elbre
fdbe2db273
Remove node package and simplify DHT interface (#1233) 2019-02-06 14:37:17 +02:00
Egon Elbre
ab83f0f077
Fix parallel execution sharing (#1247) 2019-02-06 11:16:05 +02:00
Egon Elbre
b5d86aa0b4
Make testplanet.Run and satellitedbtest.Run parallel by default (#1231) 2019-02-05 21:44:00 +02:00
Egon Elbre
07412698a9
Use lower bcrypt cost for console.Service when running tests and disable flaky tests (#1236) 2019-02-05 19:31:53 +02:00
Yaroslav Vorobiov
2ff0d9d435
Add bucket to project relationship on satellite (#1143)
* integrate console api keys with buckets in pointerdb

* fix test

* fix tests kvmetainfo

* linter fix

* disable account activation

* fix test

* review fixes

* fix comments

* little refactoring

* remove debug println

* fix typo

* disable activation in a propper way

* fix test

* fix imports

* fix uplink count in testplanet

* move key creation to planet.newUplink
2019-02-05 19:22:17 +02:00
paul cannon
1573cbf4c4 pass UTC times to db so that sqlite3 understands (#1237)
* pass UTC times to db so that sqlite3 understands

this fixes the pkg/accounting/tally tests, at least for me.

(credit to Bill Thorp)

* also fix this one
2019-02-05 11:33:14 -05:00
Egon Elbre
87d6410b50 Revert "Remove node package and simplify DHT interface."
This reverts commit 03ec1ff92d.
2019-02-05 10:38:48 +02:00
Egon Elbre
03ec1ff92d Remove node package and simplify DHT interface. 2019-02-05 10:37:24 +02:00
Egon Elbre
c37e0c1b6d
testplanet.Run (#1222) 2019-02-04 22:37:46 +02:00
Michal Niewrzal
383cb05e8e
Divide uplink and gateway params set (#1026)
* Divide uplink and gateway params set

* attempt to fix docker

* attempt to fix all in one

* test

* more reorganization

* fix compilation error

* fix imports order

* fix dependency

* rename structs

* keep minio params for now

* review comments

* remove manual flag check
2019-02-04 16:50:06 +01:00
Bill Thorp
0f662b8e38 sql based tally (#1166)
* WIP

* wacky changes

* more

* it builds.... IT BUILDS!!!

* fixed SQL, broke out saving tallies for testing

* shorter lines

* fixed SQL, moved tally_test to testplanet

* lint

* WIP logic error preventing PUT and GETs to same serialnum

* fixed BWA test

* fixed temporary brain failure

* eliminated magic numbers

* cleaned up satellite uplink stats

* use errs.Combine instead

* thrashing

* fixed tally erroneous error msg

* fixed tally test

* lint

* SQL syntax attempt to fix

* spelling error

* made bwa db test resist old postgres data

* postgres pk error msg hunting

* postgres pk error msg hunting

* postgres

* err might be nil?

* fixed error logging bug

* hopefully solved postgres issue

* using rebind

* moved tests to _test package

* fixing test dirs

* finally made sense of Egons package name feedback

* UTC, array fixes
2019-02-01 13:50:12 -05:00
Egon Elbre
5a63c00442
Fix issues with blocking during startup (#1212) 2019-02-01 19:28:40 +02:00
Yaroslav Vorobiov
ba07f337c0
Satellite api keys frontend (#1039)
* api keys frontend

* fix linter

* fixes, copy to clipboard, disable account activation

* activation is removed in https://github.com/storj/storj/pull/1143
2019-02-01 18:19:30 +02:00
Egon Elbre
5a0306ce0c
Fix typo - second edition (#1206) 2019-02-01 16:09:34 +02:00
Egon Elbre
b91d77436f
Test merging planets (#1181) 2019-02-01 15:32:28 +02:00
Egon Elbre
73ccd8411a
Fix select nodes equation (#1200) 2019-01-31 22:45:37 +02:00
Maximillian von Briesen
c46864449d
Order nodes by random (#1197) 2019-01-31 14:33:07 -05:00
Egon Elbre
ba5097c13d Postgres schemas for testing (#1194)
* add schema creation code
2019-01-31 14:17:12 -05:00
Egon Elbre
0c366c1ed2 Clearer code for node selection (#1173) 2019-01-31 13:49:00 -05:00
Bogdan Artemenko
cec2e2d0d5
[V3-1071] Enhanced project members search to full name. (#1192) 2019-01-31 16:11:53 +02:00
Egon Elbre
5e27d6ec59
Console postgres tests (#1188) 2019-01-31 15:01:13 +02:00
Dylan Lott
5e1ab841cf Cache refresher (#1171)
* got tests passed

* wire up paginate function for cache node retrieval

* Add tests for paginate but they're failing

* fix the test arguments

* Updates paginate function to return more variable

* Updates

* Some test and logic tweaks

* improves config handling in discovery

* adds refresh offset to discovery struct
2019-01-30 11:29:18 -05:00
Yehor Butko
19bc01c19a
V3-1091 Extend Users table with IsActive functionality (#1170)
* V3-1091 Extend Users table with IsActive functionality

* fixed review comments
2019-01-30 17:04:40 +02:00
Natalie Villasana
8d7944bcf8
new node selection in overlay (#1136) 2019-01-29 14:42:43 -05:00
Egon Elbre
e1a8bbdcb6
Kademlia flags cleanup (#1137) 2019-01-29 08:51:07 +02:00
Bill Thorp
e94f6408ff
bw agreement db refactor (#1158) 2019-01-28 16:16:21 -05:00
Bill Thorp
60946c2024
make bandwidth agreements sensible: without []byte's (#1152)
removed []byte's from bandwidth agreement protocol buffers
2019-01-28 14:45:25 -05:00
Egon Elbre
d50c07e56c
Implement WorkGroup (#1151) 2019-01-28 21:04:42 +02:00
Bogdan Artemenko
b75f06e229
Removed TermsAccepted in project from satellite console and web. (#1155)
* Removed TermsAccepted from satellite console and web.
2019-01-28 20:20:33 +02:00
Egon Elbre
cecd4b0816
Remove server aliases (#1154) 2019-01-28 17:04:53 +02:00
Egon Elbre
f7e18882c4
Kademlia closing race (#1139) 2019-01-26 00:33:20 +02:00
Bill Thorp
8ba4b61e59
Make Bandwidth Agreements Secure / Trustable (#1117)
* Added cert chains and nodeid verification to bandwidth agreement
2019-01-25 13:05:21 -05:00
Jennifer Li Johnson
5397efe5c1
removes standalone payments service (#1134)
* remove standalone payments service - will integrate within satellite directly

* regenerate proto files
2019-01-25 11:23:25 -05:00
Cameron
f5b1e77cf2 rollup timestamp fix (#1132)
* create timestamp if isNew

* pass isNil into SaveRollup
2019-01-24 15:41:22 -05:00
Egon Elbre
187e9b2138
Code consistency between peers (#1126) 2019-01-24 22:28:06 +02:00
Jennifer Li Johnson
856b98997c
updates copyright 2018 to 2019 (#1133) 2019-01-24 15:15:10 -05:00
Jennifer Li Johnson
b424345edf
Jj/audit (#1130) 2019-01-24 14:52:16 -05:00
Jennifer Li Johnson
70940b8710
Making sure payments works (#1122)
* byte hours

* updates comment

* removes payment test data methods

* wip

* adds tally and rollup to peer

* remove sim test for payment (for now)

* lint error
2019-01-24 13:44:14 -05:00
Yaroslav Vorobiov
f437effaef
Add console to satellite peer (#1120) 2019-01-24 18:26:36 +02:00
Egon Elbre
5de7f8af7f
Satellite Peer (#1119) 2019-01-23 21:58:44 +02:00
Egon Elbre
99d3b7a3c8
Fix import grouping (#1111) 2019-01-22 17:48:23 +02:00
Michal Niewrzal
eb2d858ffc Use pointerdb.Service instead of pointerdb.Server (#1099) 2019-01-19 13:58:53 -05:00
Cameron
ecd704692e
wip testing and timestamp fix (#1100)
* wip testing and timestamp fix

* fixed import cycle
2019-01-18 11:53:23 -05:00
Michal Niewrzal
cc1bdef8b7
Move pointerdb logic to a service (#1037)
* test

* Move pointerdb logic to service

* tset

* reorder constructor params

* restore field
2019-01-18 10:10:21 -05:00
Egon Elbre
78dc02b758 Satellite Peer (#1034)
* add satellite peer

* Add overlay

* reorganize kademlia

* add RunRefresh

* add refresh to storagenode.Peer

* add discovery

* add agreements and metainfo

* rename

* add datarepair checker

* add repair

* add todo notes for audit

* add testing interface

* add into testplanet

* fixes

* fix compilation errors

* fix compilation errors

* make testplanet run

* remove audit refrences

* ensure that audit tests run

* dev

* checker tests compilable

* fix discovery

* fix compilation

* fix

* fix

* dev

* fix

* disable auth

* fixes

* revert go.mod/sum

* fix linter errors

* fix

* fix copyright

* Add address param for SN dashboard (#1076)

* Rename storj-sdk to storj-sim (#1078)

* Storagenode logs and config improvements  (#1075)

* Add more info to SN logs

* remove config-dir from user config

* add output where config was stored

* add message for successful connection

* fix linter

* remove storage.path from user config

* resolve config path

* move success  message to info

* log improvements

* Remove captplanet (#1070)

* pkg/server: include production cert (#1082)

Change-Id: Ie8e6fe78550be83c3bd797db7a1e58d37c684792

* Generate Payments Report (#1079)

* memory.Size: autoformat sizes based on value entropy (#1081)

* Jj/bytes (#1085)

* run tally and rollup

* sets dev default tally and rollup intervals

* nonessential storj-sim edits (#1086)

* Closing context doesn't stop storage node (#1084)

* Print when cancelled

* Close properly

* Don't log nil

* Don't print error when closing dashboard

* Fix panic in inspector if ping fails (#1088)

* Consolidate identity management to identity cli commands (#1083)

* Consolidate identity management:

Move identity cretaion/signing out of storagenode setup command.

* fixes

* linters

* Consolidate identity management:

Move identity cretaion/signing out of storagenode setup command.

* fixes

* sava backups before saving signed certs

* add "-prebuilt-test-cmds" test flag

* linters

* prepare cli tests for travis

* linter fixes

* more fixes

* linter gods

* sp/sdk/sim

* remove ca.difficulty

* remove unused difficulty

* return setup to its rightful place

* wip travis

* Revert "wip travis"

This reverts commit 56834849dcf066d3cc0a4f139033fc3f6d7188ca.

* typo in travis.yaml

* remove tests

* remove more

* make it only create one identity at a time for consistency

* add config-dir for consitency

* add identity creation to storj-sim

* add flags

* simplify

* fix nolint and compile

* prevent overwrite and pass difficulty, concurrency, and parent creds

* goimports
2019-01-18 08:54:08 -05:00
Jennifer Li Johnson
e6fbf63620
Generate Payments Report (#1079) 2019-01-17 13:34:13 -05:00
Yaroslav Vorobiov
d832789481
ConsoleDB merge and buckets table creation (#1066)
* init

* db merge

* bucket table renamed to bucket info

* remove id and fix comments

* fix imports
2019-01-16 15:23:28 -05:00
Bill Thorp
342dc857f5 rollup query (#1056)
* implemention notes

* more notes

* starting rollup query

* not working yet

* fixed build

* fixed cfg bug

* change context cancelled errs to debugs

* using byte hours for at rest tally

* revert changes to go.mod

* comment fixes

* prevent double recording tallies in rollup

* linting

* stop leaking dbx

* nodeid changes

* fix build
2019-01-16 14:30:33 -05:00
Egon Elbre
f8906ce000
Change overlay cache database interface (#1047) 2019-01-15 11:08:45 -05:00
Yehor Butko
e38cf8f50d
Renaming and moving pkg/satellite to satellite/console (#1054)
* [WIP] V3-853 Merge the satellite DB into the master database

* Removing consoleDB from satelliteDB

* Fixing tests for satellite/console

* fixing linter

* sorting imports in satellite/console

* fixing console config

* fixing linter
2019-01-15 15:03:24 +02:00
Egon Elbre
12eec57abf
Add arguments to lockedgen for using it in other places (#1030) 2019-01-11 18:07:26 +02:00
Cameron
ef50bbf8b6
satellites reject expired BWAs (#1015)
* add 45 day expiration to PBAs

* add expiration field to relevant areas, DeleteExpired placeholder

* reject expired BWAs

* test for expired BWAs

* add BwExpiration config value
2019-01-10 13:30:55 -05:00
Bill Thorp
fb4a11ebb0
Differentiate GET and PUT operations in the tally service (#965)
* draft of new bandwidth agreement types
* updated storagenode report for new types
* use correct pba types
2019-01-10 06:41:57 -05:00
aligeti
457f909a74 Satellite accepts manipulated paychecks V3-1023 (#972)
* fix - Satellite crashing on receiving a manipulated bandwidthagreement

* provider.PeerIdentityFromContext called twice. Remove one

* add storage node ID to serial number

* remove serialNum query and transaction

* add uuid to GeneratePayerBandwidthAllocation for testing

* enable expected failure on duplicate serialnum cases

* Revert "enable expected failure on duplicate serialnum cases"

This reverts commit 5948f43ed1741c280f0bb34a86c1c490365417bc.

* enable expected failure on duplicate serialnum cases
2019-01-09 10:02:03 -05:00
Egon Elbre
0ca03b41e2
Use generated locking database implementation (#947) 2019-01-02 19:53:27 +02:00
Michal Niewrzal
b712fbcbb0
Fix 'empty queue' error when satellite starts (#939) 2019-01-02 17:00:32 +01:00
JT Olio
4a0749e44a Remove farmer references (#949) 2019-01-02 12:31:49 +02:00
Michal Niewrzal
8c00b2109e
Format satellite.dbx (#930)
* Format satellite.dbx

* remove empty lines
2018-12-27 22:21:50 +01:00
Egon Elbre
4346cd060f
Implement mutex around satellitedb (#932) 2018-12-27 11:56:25 +02:00
Cameron
f70b826fd4
repair queue masterDB support (#865)
* add injuredsegment model to satellitedb.dbx

* add context to queue.RepairQueue interface

* use queue.RepairQueue interface, use masterdb
2018-12-21 10:11:19 -05:00
Maximillian von Briesen
1aa8bb03ef
Statdb transactions (#911)
add sql transactions to statdb
    update statdb errors to use satellitedb error class
2018-12-19 17:34:20 -05:00
Maximillian von Briesen
f18c38628e
Refactor statdb (#887)
Update statdb args/return values to minimize structs
    Simplify statdb.Update() to update all stats instead of an arbitrary subset determined by flags
    Remove CreateIfNotExists logic from statdb.Update()
    Simplify audit code structure
2018-12-19 13:44:03 -05:00
Michal Niewrzal
bbe1dbb92c
Fix overlay commands after master db merge (#895)
* Fix overlay commands after master db merge

* remove unused var

* remove unused comment
2018-12-18 21:41:31 +01:00
Jennifer Li Johnson
01df85704c
Updates accounting tables and allows tally to update at raw table for at rest data (#874) 2018-12-18 12:18:42 -05:00
Michal Niewrzal
a819d819e3
Overlay Cache master db merge (#877)
* Overlay Cache master db merge

* test update

* review comments

* Fix transaction usage

* removed unused variable

* added better limit handling

* better error handling
2018-12-17 21:14:16 +01:00
Michal Niewrzal
de4ab85ebe
Fix irreparable db transaction usage (#888) 2018-12-17 19:54:46 +01:00
aligeti
5e1b02ca8b
Statdb master db v3 848 (#830)
* intial changes to migrate statdb to masterdb framework

* statdb refactor compiles

* added TestCreateDoesNotExist testcase

* Initial port of statdb to masterdb framework working

* refactored statdb proto def to pkg/statdb

* removed statdb/proto folder

* moved pb.Node to storj.NodeID

* CreateEntryIfNotExistsRequest moved pd.Node to storj.NodeID

* moved the fields from pb.Node to statdb.UpdateRequest

ported TestUpdateExists, TestUpdateUptimeExists, TestUpdateAuditSuccessExists TestUpdateBatchExists
2018-12-14 15:17:30 -05:00
Bill Thorp
f9845e7e92
changes to masterdb to support accountingDB (#846)
* added rollup to captplanet, moved accountingDB to masterdb
2018-12-14 09:27:21 -05:00
Bill Thorp
ba5f71810e
added tally to capt run.go (#751)
* added tally to capt run.go, ParseURL -> SplitDBURL
2018-12-12 08:15:34 -05:00
Michal Niewrzal
7e2f66ee24
Small fixes around master db (#822) 2018-12-11 10:30:09 +01:00
aligeti
d27bcddaab
Merge the irreparable DB into the master database (#777)
* initial refactoring to merge irreparabledb to master db

* refactor

* code review comment updates
2018-12-10 14:08:45 -05:00
Kevin Leffew
4a128e65d9 Removed unnecessary comment (#804) 2018-12-10 13:54:52 -05:00
Michal Niewrzal
a15099d5f5
Simplify testing with all supported DBs (#809)
* Simplify testing with all supported DBs

* move method to separate package

* method renamed to Run

* fixed comment
2018-12-10 15:50:12 +01:00
Egon Elbre
c7cf97cc23
Fix sqlite registration (#806) 2018-12-10 10:32:49 +02:00
JT Olio
1c96db01ba
better database error handling (#784)
* better database error handling

Change-Id: I28dbd69cf6c2fa268e02405521ff6e6c1a68a702

* missing comments added

* missing comment added
2018-12-07 07:46:42 -07:00
Michal Niewrzal
034a34069a
Merge bwagreement db into satellite master db (#770)
* Merge bwagreement db into satellite master db

* adjust to recent tally changes

* linter problems

* linter problems

* returning db structs in more optimal way

* added pointer for assignment

* error message changed

* better param message
2018-12-07 10:59:31 +01:00
Michal Niewrzal
44a6cfa1c1
Initial satellite master database setup (#735)
* Initial satellite master database setup

* fixed unsed ctx and /nolint added for temp change

* tests for two db drivers + connection param renamed
2018-12-05 10:35:50 +01:00