Commit Graph

101 Commits

Author SHA1 Message Date
Cameron
963e1b9710 Rename voucher service to endpoint (#2467) 2019-07-08 14:48:47 +03:00
Cameron
d499d162f4
implement storj.NodeURL in trusted satellites (#2388)
* implement storj.NodeURL in trusted satellites
2019-07-03 13:29:18 -04:00
Yingrong Zhao
f56cb1c612 satellite/console: add get current reward offer to console (#2341)
* expose reward offer to satellite gui

* named return
2019-07-02 10:36:54 -04:00
Yaroslav Vorobiov
f4dfb8fb95
Add daily node space usage to satellite nodestats endpoint (#2361) 2019-07-02 13:42:09 +03:00
Maximillian von Briesen
52e5a4eee3 pass logger into repairer and ecclient (#2365) 2019-07-02 13:08:02 +03:00
Jennifer Li Johnson
699ccea19f
Creates Routing Table Antechamber (#2318) 2019-07-01 17:20:19 -04:00
Faris Huskovic
ebd976ec28
satellite/marketing: Create New Offer (#2186)
* update UI to reflect final mockups

* implement create handler and render offers table data to UI

* fix line-height unit and remove important from selectors

* update file names and ids for clarity

* shorten 'label' in ids

* localize global vars, fix endpoint names, remove unnecessary receiver, fix comments

* fix unnecessary initialization of pointer

* correct file-naming conventions

* register timeConverter in an init func for safety and remove unnecessary important from css

* consolidate create endpoints and add comments

* register timeConverter in init func

* add copyright to files

* introduce require pkg

* add proper http server unit test

* update linting and create offers concurrently in unit test

* fix getOffers comment

* add copy-right to unit-test

* fix data-races

* fix linting

* remove converter in NewServer

* requested changes in progress

* add require for checking status code

* renamed template file

* fix 400 handler

* fix missing copyright and remove extra line

* fix build

* run goroutine for testing parallel

* evaluate reqType with switch stmt and promp for credit amount in cents

* fix lint issue

* add default case

* remove unnecessary var

* fix range scope error

* remove empty lines and use long form for struct field

* fix merge conflicts

* fix template reference

* fix modal id

* not delete package

* add currency formatting and requested changes

* markup formatting

* lean out currency logic and move wait outside loop

* pass ToDollars func to home template

* fix lint
2019-06-28 10:34:10 -04:00
Yaroslav Vorobiov
eb5dc4ba2b
Add node stats GRPC endpoint to the satellite (#2281) 2019-06-25 13:40:32 +03:00
Yingrong Zhao
bbedff12a6 satellite: rearrange marketing package (#2268)
* move offer out of marketing package and remove marketing package

* fix imports

* fix rename errors

* remove offer service

* change package name from offers to rewards

* fix linting

* remove unused code and use appropriate comment
2019-06-24 16:51:54 -04:00
Cameron
1283036e37
add storage node voucher request service (#2158)
* add voucher service on storage node

* config field tag syntax, go routines for requests

* hook up voucher service in storagenode/peer.go

* add voucher config to testplanet

* add voucher config to testplanet

* add voucher response status INVALID, ACCEPTED, REJECTED

* add a test for vouchers service

* handle no row from GetValid, test it

* add trust pool to voucher service

* use trusted list to get satellites

* verify vouchers upon receipt

* test VerifyVoucher
2019-06-21 18:48:52 -04:00
aligeti
d8e62bc06e
support value attribution endpoint (#2231)
Support Value attribution Endpoint
2019-06-21 15:14:34 -04:00
aligeti
043d603cbe
satellite rs config check with validation check set to false default (#2229)
* satellite rs config check with validation check
2019-06-21 14:15:58 -04:00
JT Olio
568b000e9b satellite: make order expiration configurable (#2251) 2019-06-21 13:38:40 +03:00
JT Olio
76b54458e9 satellite: send external address in order limits (#2278) 2019-06-21 12:19:52 +03:00
Egon Elbre
8f4a6afc8a Fixes for value attribution (#2238)
* Move value attribution to satellite/

* remove unnecessary conversion
2019-06-19 08:02:37 -04:00
aligeti
09940d4e0b
value attribution DB interface Insert & Get method support (#2200)
* value attribution DB interface methods support
2019-06-18 09:06:33 -04:00
Yingrong Zhao
474d9e7492
only parse templates for the first requests (#2180)
* only parse templates for the first requests

* fix race condition isse

* parse all templates before start the new server

* add comment for parseTemplates and remove unused code

* remove unused type
2019-06-12 16:27:07 -04:00
Yingrong Zhao
af66d9c6e4
Open a new port on satellite for admin GUI (#1901)
* Set up new port 8090 for in offers

Clean up commented code

Rename offers to offersweb

Remove unused code

Add todos for adding front-end templates

Add middleware for only allow local access

Add comment

Fix linting error

Remove commented code

Update storj-sim

Check request IP against Host IP

Use net pakcage to retrieve IP address

Rename service to marketing

* Add wrapper for all errors

* fix conflicts

* update the config file

* fix linting error

* remove unused packages

* remove global runtime var and add flag to storj-sim for mar static dir

* remove debugging lines

* add new config for test data and check if static dir flag is set before passing to mux

* change 'console' to 'marketing' for test data config

* fix linting errors

* update config flag

* Trigger Jenkins

* Trigger CLA
2019-06-11 11:00:59 -04:00
Kaloyan Raev
252c8ac189
Add email to self node info (#2171) 2019-06-11 16:30:28 +03:00
Ivan Fraixedes
cac4637670
satellite: Fix Peer.Run method doc comment (#2147) 2019-06-07 13:40:20 +02:00
Egon Elbre
28a1201590 pkg/bwagreements: remove service (#2138) 2019-06-06 15:57:58 +02:00
JT Olio
d7f3a5f811
internal,lib,uplink: add monkit task to missing places (#2118)
* internal,lib,uplink: add monkit task to missing places

Change-Id: I490053eee4ed517502f9fe00c6394f0095bd13d0

* Include Monkit

* Add missing context

* Another missing ctx

* More ctx missing

* Linting

* go imports

Change-Id: Ibf0ed072eba339f027727ed8039f7bce1f223fa7

* fix semantic merge conflict

Change-Id: I67fb1f4e7b6cd5e89d69987ed7b3966b7d30ee37
2019-06-05 09:03:11 -06:00
Yingrong Zhao
09b0c2a630
create db implementation for offer table (#2031)
* init marketing service

Fix linting error

Create offerdb implementation

Create offers service

Add update method

Create offer table and migration

Fix linting error

fix conflicts

Insert new data

Change duration to have clear indication to be based on days

add error wrapper

Change from using uuid to int for id field

* Create Marketing service

* make error virable name more readable

* add condition in update service method to check offer status

* generate lock file

Change get to listAllOffers

* Add method for getting current offer

wip

* add check for expires_at in update method

* Fix conflicts

* add copyright header

* Fix linting error

* only allow update to active offers

* add isDefault argument to GetCurrent

* Update lock file

* add migration file

* finish migrate for adding credit_in_cents for both award and invitee

* save 100 years as expiration date for default offers

* create crud test for offers

* add GetCurrent test

* modify doc

* Fix GetCurrent to work with default offer

* fix linting issue

* add more tests and address feedbacks

* fix migration file

* add type column back to match with mockup design

* add type column back to match with mockup design

* move doc changes to new pr

* add comments

* change GetCurrent to GetCurrentByType

* fix typo
2019-06-04 15:17:01 -04:00
paul cannon
d15eaed588 add capability of logging all GRPC calls/payloads (#2067) 2019-06-04 14:55:24 +02:00
JT Olio
29d16b4d68 satellite: add monkit task to missing places (#2108) 2019-06-04 13:55:37 +02:00
Yaroslav Vorobiov
6809129e6f
Console add stripe service (#2080) 2019-06-03 16:46:57 +03:00
Cameron
590b1a5a1d
Satellite voucher service (#2043)
* set up voucher service skeleton, basic test

* add VetNode db method

* basic test for VetNode

* encode and sign voucher functions

* fill out and sign vouchers

* test pass/fail voucher request

* match EncodeVoucher to other Encode functions
2019-05-30 15:52:33 -04:00
aligeti
934ebf9cbf
Added the irreparable repair functionality (#1955)
* Added the irreparable repair functionality
2019-05-30 11:18:20 -04:00
ethanadams
268dc6b7e4
Enable gocritic linter (#2051)
* first round cleanup based on go-critic

* more issues resolved for ifelsechain and unlambda checks

* updated from master and gocritic found a new ifElseChain issue

* disable appendAssign. i reports false positives

* re-enabled go-critic appendAssign and disabled lint check at code line level

* fixed go-critic lint error

* fixed // nolint add gocritic specifically
2019-05-29 09:14:25 -04:00
Michal Niewrzal
03bae48ad6
Remove unused parameter (#2069) 2019-05-29 11:15:15 +02:00
Michal Niewrzal
f731267e8c Per-project usage limiting (#2036)
What: Changes to support custom usage limit for the project. With this implementation by default project usage limit is taken from configuration flag. If project DB field usage_limit will be set to value larger than 0 it will become custom usage limit and we will be used to verify is limit was exceeded.

Whats changed:

usage_limit (bigint) field added to projects table (with migration)
things related to project usage moved from metainfo endpoint to project usage type
accounting.ProjectAccounting extended with GetProjectUsageLimits() method
Why: We need to have different usage limits per project. https://storjlabs.atlassian.net/browse/V3-1814
2019-05-28 09:36:52 -06:00
Stefan Benten
8912d7149c
Fixes Auth Issue (#2064) 2019-05-28 16:32:51 +02:00
Fadila
8f078d2841 Delete pending audit entries from containment DB when files get deleted (#2050) 2019-05-24 15:56:08 -04:00
Kaloyan Raev
de8070730a Save hash of expected share from missing audited node (#2025) 2019-05-23 16:07:19 -04:00
Natalie Villasana
8b31c4b91f
remove unnecessary containmentDB wrapper (#2027) 2019-05-23 10:37:23 -04:00
Natalie Villasana
acfb59ae48
adds db layer functions for containment (#1972) 2019-05-22 10:50:22 -04:00
Egon Elbre
748b2b05a4 satellite/mailservice: wait for goroutines to finish (#1985)
* satellite/mailservice: wait for goroutines to finish

* fix tests
2019-05-17 14:29:35 +02:00
Jess G
8518618b7a
add postgres support to storj-sim (#1908)
* add flags to sotrj-sim for SA dbs

* add schema to postgres

* add createschema with parse to sa

* add metainfo db postgres support

* add kv default as bolt

* add debug log to see db source

* add env var for postgres to test-sim.sh

* fix lint errs

* dynamically add postgres to args

* add postgres to integration tests

* add sqlite and postgres integration jenkins

* fix db name

* merge integration tests into one step

* test integration tests w/psql

* try using different schema

* debug failure

* use correct host for running storj-sim

* rm sqlite integration

* add back integration
2019-05-14 08:13:18 -07:00
Jennifer Li Johnson
5395ff5fe6
Refactor accountingdb interface (#1897)
* splits accounting db into storagenodeaccounting and projectaccounting interfaces and renames methods to match
2019-05-10 15:05:42 -04:00
paul cannon
02be91b029
real-time tracking of space used per project (#1910)
Ran into difficulties trying to find the ideal solution for sharing
these counts between multiple satellite servers, so for now this is a
dumb solution storing recent space-usage changes in a big dumb in-memory
map with a big dumb lock around it. The interface used, though, should
allow us to swap out the implementation without much difficulty
elsewhere once we know what we want it to be.
2019-05-09 20:39:21 -05:00
Michal Niewrzal
fe3dfc1587
Move pointerdb.Service to satellite (#1826) 2019-04-25 10:46:32 +02:00
Cameron
9f6b010748 rollup interval fixes (#1739)
* set all intervals to UTC in rollupStats map, only delete latest day after both rollups

* clean up usage of interval, use intervalEndTime rather than createdAt

* change some variable names, add comments

* add flag for tally deletion

* adds deletetallies flag to testplanet

* space

* Removes println:

* adds test for deletes false
2019-04-23 15:21:30 -04:00
Kaloyan Raev
8fc5fe1d6f
Refactor pb.Node protobuf (#1785) 2019-04-22 12:07:50 +03:00
Egon Elbre
bbeb0eac4e
disable pdb logging, it's usually not that useful (#1790) 2019-04-19 16:30:55 +03:00
Egon Elbre
5b3c146d8a
Check context cancellation more nicely (#1752) 2019-04-17 13:09:44 +03:00
Stefan Benten
bae4c820ee
Add Version Information into KAD Network and SatelliteDB & Change Selection Process (#1648)
* Initial Webserver Draft for Version Controlling

* Rename type to avoid confusion

* Move Function Calls into Version Package

* Fix Linting and Language Typos

* Fix Linting and Spelling Mistakes

* Include Copyright

* Include Copyright

* Adjust Version-Control Server to return list of Versions

* Linting

* Improve Request Handling and Readability

* Add Configuration File Option
Add Systemd Service file

* Add Logging to File

* Smaller Changes

* Add Semantic Versioning and refuses outdated Software from Startup (#1612)

* implements internal Semantic Version library

* adds version logging + reporting to process

* Advance SemVer struct for easier handling

* Add Accepted Version Store

* Fix Function

* Restructure

* Type Conversion

* Handle Version String properly

* Add Note about array index

* Set temporary Default Version

* Add Copyright

* Adding Version to Dashboard

* Adding Version Info Log

* Renaming and adding CheckerProcess

* Iteration Sync

* Iteration V2

* linting

* made LogAndReportVersion a go routine

* Refactor to Go Routine

* Add Context to Go Routine and allow Operation if Lookup to Control Server fails

* Handle Unmarshal properly

* Linting

* Relocate Version Checks

* Relocating Version Check and specified default Version for now

* Linting Error Prevention

* Refuse Startup on outdated Version

* Add Startup Check Function

* Straighten Logging

* Dont force Shutdown if --dev flag is set

* Create full Service/Peer Structure for ControlServer

* Linting

* Straighting Naming

* Finish VersionControl Service Layout

* Improve Error Handling

* Change Listening Address

* Move Checker Function

* Remove VersionControl Peer

* Linting

* Linting

* Create VersionClient Service

* Renaming

* Add Version Client to Peer Definitions

* Linting and Renaming

* Linting

* Remove Transport Checks for now

* Move to Client Side Flag

* Remove check

* Linting

* Transport Client Version Intro

* Adding Version Client to Transport Client

* Add missing parameter

* Adding Version Check, to set Allowed = true

* Set Default to true, testing

* Restructuring Code

* Uplink Changes

* Add more proper Defaults

* Renaming of Version struct

* Dont pass Service use Pointer

* Set Defaults for Versioning Checks

* Put HTTP Server in go routine

* Add Versioncontrol to Storj-Sim

* Testplanet Fixes

* Linting

* Add Error Handling and new Server Struct

* Move Lock slightly

* Reduce Race Potentials

* Remove unnecessary files

* Linting

* Add Proper Transport Handling

* small fixes

* add fence for allowed check

* Add Startup Version Check and Service Naming

* make errormessage private

* Add Comments about VersionedClient

* Linting

* Remove Checks that refuse outgoing connections

* Remove release cmd

* Add Release Script

* Linting

* Update to use correct Values

* Change Timestamp handling

* Adding Protobuf changes back in

* Adding SatelliteDB Changes and adding Storj Node Version to PB

* Add Migration Table

* Add Default Stats for Creation

* Move to BigInt

* Proper SQL Migration

* Ensure minimum Version is passed to the node selection

* Linting...

* Remove VersionedClient and adjust smaller changes from prior merge

* Linting

* Fix PB Message Handling and Query for Node Selection

* some future-proofing type changes

Change-Id: I3cb5018dcccdbc9739fe004d859065992720caaf

* fix a compiler error

Change-Id: If66bb92d8b98e31cd618ecec9c6448ab9b037fa5

* Comment on Constant for Overlay

* Remove NOT NULL and add epoch call as function

* add versions to bootstrap and satellites

Change-Id: I436944589ea5f21600cdd997742a84fe0b16e47b

* Change Update Migration

* Fix DB Migration

* Increase Timeout temporarily, to see whats going on

* Remove unnecessary const and vars
Cleanup Function calls from deprecated NodeVersion struct

* Updated Protopuf, removed depcreated Code from Inspector

* Implement NodeVersion into InfoResponse

* Regenerated locked.go

* Linting

* Fix Tests

* Remove unnecessary constant

* Update Function and Flag Description

* Remove Empty Stat Creation

* return properly with error

* Remove unnecessary struct

* simplify migration step

* Update Inspector to return Version Info

* Update local Endpoint Version Handling

* Reset Travis Timeout

* Add Default for CommitHash

* single quotes
2019-04-10 08:04:24 +02:00
JT Olio
09be9964eb internal/version: do version checks much earlier in the process initialization, take 2 (#1666)
* internal/version: do version checks much earlier in the process initialization, take 2

Change-Id: Ida8c7e3757e0deea0ec7aea867d3d27ce97dc134

* linter and test failures

Change-Id: I45b02a16ec1c0f0981227dc842e68dbdf67fdbf4
2019-04-04 17:40:07 +02:00
Jennifer Li Johnson
8549421385
Remove bw from tally service + query bandwidth in rollup service (#1618) 2019-04-04 11:20:59 -04:00
Stefan Benten
2cf86703a3
Add Versioning Server (#1576)
* Initial Webserver Draft for Version Controlling

* Rename type to avoid confusion

* Move Function Calls into Version Package

* Fix Linting and Language Typos

* Fix Linting and Spelling Mistakes

* Include Copyright

* Include Copyright

* Adjust Version-Control Server to return list of Versions

* Linting

* Improve Request Handling and Readability

* Add Configuration File Option
Add Systemd Service file

* Add Logging to File

* Smaller Changes

* Add Semantic Versioning and refuses outdated Software from Startup (#1612)

* implements internal Semantic Version library

* adds version logging + reporting to process

* Advance SemVer struct for easier handling

* Add Accepted Version Store

* Fix Function

* Restructure

* Type Conversion

* Handle Version String properly

* Add Note about array index

* Set temporary Default Version

* Add Copyright

* Adding Version to Dashboard

* Adding Version Info Log

* Renaming and adding CheckerProcess

* Iteration Sync

* Iteration V2

* linting

* made LogAndReportVersion a go routine

* Refactor to Go Routine

* Add Context to Go Routine and allow Operation if Lookup to Control Server fails

* Handle Unmarshal properly

* Linting

* Relocate Version Checks

* Relocating Version Check and specified default Version for now

* Linting Error Prevention

* Refuse Startup on outdated Version

* Add Startup Check Function

* Straighten Logging

* Dont force Shutdown if --dev flag is set

* Create full Service/Peer Structure for ControlServer

* Linting

* Straighting Naming

* Finish VersionControl Service Layout

* Improve Error Handling

* Change Listening Address

* Move Checker Function

* Remove VersionControl Peer

* Linting

* Linting

* Create VersionClient Service

* Renaming

* Add Version Client to Peer Definitions

* Linting and Renaming

* Linting

* Remove Transport Checks for now

* Move to Client Side Flag

* Remove check

* Linting

* Transport Client Version Intro

* Adding Version Client to Transport Client

* Add missing parameter

* Adding Version Check, to set Allowed = true

* Set Default to true, testing

* Restructuring Code

* Uplink Changes

* Add more proper Defaults

* Renaming of Version struct

* Dont pass Service use Pointer

* Set Defaults for Versioning Checks

* Put HTTP Server in go routine

* Add Versioncontrol to Storj-Sim

* Testplanet Fixes

* Linting

* Add Error Handling and new Server Struct

* Move Lock slightly

* Reduce Race Potentials

* Remove unnecessary files

* Linting

* Add Proper Transport Handling

* small fixes

* add fence for allowed check

* Add Startup Version Check and Service Naming

* make errormessage private

* Add Comments about VersionedClient

* Linting

* Remove Checks that refuse outgoing connections

* Remove release cmd

* Add Release Script

* Linting

* Update to use correct Values

* Move vars private and set minimum default versions for testing builds

* Remove VersionedClient

* Better Error Handling and naked return removal

* Straighten the Regex and string conversion

* Change Check to allows testplanet and storj-sim to run without the
need to pass an LDFlag

* Cosmetic Change to Dashboard

* Cleanup Returns and remove commented code

* Remove Version Check if no build options are passed in

* Pass in Config Values instead of Pointers

* Handle missed Error

* Update Endpoint URL

* Change Type of Release Flag

* Add additional Logging

* Remove Versions Logging of other Services

* minor fixes

Change-Id: I5cc04a410ea6b2008d14dffd63eb5f36dd348a8b
2019-04-03 21:13:39 +02:00
Jess G
d51bdf14df
project usage limiting (#1561)
* reorg uplink cmd files for consistency

* init implementation of usage limiting

* Revert "reorg uplink cmd files for consistency"

This reverts commit 91ced7639bf36fc8af1db237b01e233ca92f1890.

* add changes per CR comments

* fix custom query to use rebind

* updates per convo about what to limit on

* changes per comments

* fix syntax and comments

* add integration test, add db methods for test

* update migration, add rebind to query

* update testdata for psql

* remove unneeded drop index statement

* fix migrations, fix calculate usage limit

* fix comment

* add audit test back

* change methods to use bucketName/projectID, fix tests

* add changes per CR comments

* add test for uplink upload and err ssg

* changes per CR comments

* check get/put limit separately
2019-04-02 11:21:18 -07:00