Commit Graph

115 Commits

Author SHA1 Message Date
Kaloyan Raev
aac1e3c45f satellite/payments: move inspector commands to satellite cli
This allows to seeing logs in the output of the invoice commands.

Existing ensure-stripe-customer commands is moved from the 'reports' to
the new 'billing' root command.

Change-Id: I752c7ab6ca59bfac8e0f174a45d2ab45fc18e467
2020-05-18 16:12:45 +00:00
Michal Niewrzal
2eb2c25e51 satellite/payments/stripecoinpayments: add StripeClient interface to be
able to cover more testing scenarios

Currently, its hard to implement test suite for payments because
mockpayments is on to high level and we cannot emulate many things e.g.
adding credit card. This change is first to be able to add mock for
Stripe client and do more granular tests.

Change-Id: Ied85d4bd0642debdffe1161657c1e475202e9d23
2020-05-15 10:52:44 +02:00
Ethan
159df8b2e4 Add logging listener for retrieving and setting log levels
See https://storjlabs.atlassian.net/browse/SM-752

These changes allow us to change the log level at runtime through a handler off of the debug endpoint.

Examples of changing the log level on storj-sim

To get the current level for the satellite api process:
curl -XGET 'http://127.0.0.1:10009/logging' --header 'Content-Type: text/plain'

To change the log level:
curl -XPUT 'http://127.0.0.1:10009/logging' --header 'Content-Type: text/plain' --data-raw '{"level":"error"}'

Change-Id: I05d164b290929fa06b6d78c01075ee41f8238044
2020-05-12 16:38:06 -04:00
Stefan Benten
e23bd806b4
satellite/accounting: separate usage and bandwidth limit (#3878) 2020-05-12 15:01:15 +02:00
Egon Elbre
7d29f2e0d3 all: remove drpc wrappers
Change-Id: I45016f7d2a771dc00776196c1f531f3343e93b40
2020-05-11 08:20:34 +03:00
Egon Elbre
e6d5ce6b77 all: remove grpc
It seems everyone has migrated to drpc.

Change-Id: Ica6b2d0bdef68c6603083f2963458843eca71e9e
2020-05-10 06:36:09 +00:00
Jess G
75b9a5971e
satellite: update log levels (#3851)
* satellite: update log levels

Change-Id: I86bc32e042d742af6dbc469a294291a2e667e81f

* log version on start up for every service

Change-Id: Ic128bb9c5ac52d4dc6d6c4cb3059fbad73f5d3de

* Use monkit for tracking failed ip resolutions

Change-Id: Ia5aa71d315515e0c5f62c98d9d115ef984cd50c2

* fix compile errors

Change-Id: Ia33c8b6e34e780bd1115120dc347a439d99e83bf

* add request limit value to storage node rpc err

Change-Id: I1ad6706a60237928e29da300d96a1bafa94156e5

* we cant track storage node ids in monkit metrics so lets use logging to track that for expired orders

Change-Id: I1cc1d240b29019ae2f8c774792765df3cbeac887

* fix build errs

Change-Id: I6d0ffe058e9a38b7ed031c85a29440f3d68e8d47
2020-04-15 12:32:22 -07:00
Egon Elbre
d3ce845f82 satellite: revert log lines used to figure out node id
Currently storj-sim relies on the log lines to be exactly the same,
when they change it cannot find the necessary information from log.

Change-Id: Ia039915ef3375a7cf60f107b2c05c958de15b6d5
2020-04-15 17:07:56 +03:00
Kaloyan Raev
a2ce836761 remove sugar logging
Change-Id: I6b6ca9704837cb3f5f5449ba7f55661487814d9f
2020-04-15 12:37:47 +00:00
Michal Niewrzal
f36e8548f1 satellite/metainfo: adjust max inline segment size validation to
potential encryption overhead.

This is the same approach we have for validating remote segment size.

https://storjlabs.atlassian.net/browse/USR-619

Change-Id: I2597ee734313a3068fd986001680bbedbf1bed2a
2020-04-09 12:34:10 +00:00
Qweder93
3a9422cc9a satellite/nodestats: add pricing model to endpoint
Change-Id: Iddace8e437216a343458f440b543cee61164f233
2020-04-08 14:29:51 +00:00
Cameron Ayer
42be4bdc0f satellite/contact: add timeout to PingBack method
Change-Id: I2ec2f82e2e10d8be16f82e9de13ce42358e47c98
2020-04-04 18:26:30 +00:00
Michal Niewrzal
c178a08cb8 satellite/metainfo: add max segment size and max inline size to
BeginObject response

We want to control inline segment size and segment size on satellite
side. We need to return such information to uplink like with redundancy
scheme.

Change-Id: If04b0a45a2757a01c0cc046432c115f475e9323c
2020-04-02 12:41:28 +00:00
Egon Elbre
c715c75fea pkg/server: add counters for grpc calls
This will help to determine how many grpc calls are made to the
satellite.

Also remove the grpc funcs that have been added to upstream.

Change-Id: I91878f4fd10f9bfe601c94222c102eaaf4d35963
2020-03-25 21:38:13 +02:00
Yingrong Zhao
a731472496 bump storj.io/common to latest and storj.io/drpc to v0.0.11
Change-Id: I7a6e823b441eeff4621dfdf2d6577be76c9761c8
2020-03-24 15:17:10 -04:00
Michal Niewrzal
fdf40a7526 storj: remove storj/private/version package which was moved to
`storj/private` repo

Change-Id: I81c3f5b9d5e4fe7bca760999eb045ee9734e5e2e
2020-03-24 14:31:33 +00:00
Michal Niewrzal
f0aeda3091 storj: remove from storj/pkg packages moved to storj/private repo
* debug
* traces
* cfgstruct
* process

Package `storj/private/version` will be removed as a separate change.

Change-Id: Iadc40faa782e6225513b28218952f02d9c240a9f
2020-03-24 09:56:29 +01:00
Egon Elbre
09e0f3de63 satellite/metainfo/piecedeletion: add Service
Change-Id: Id7e32ed569701fa0be66f9527c43a67052994570
2020-03-18 14:50:08 +00:00
Stefan Benten
52590197c2
satellite/payments: More Cleanup and Satellite command to ensure we have stripe customers (#3805) 2020-03-16 20:34:15 +01:00
Qweder93
9f84261c36 storagenode/cache heldamount added
Change-Id: I7fc807789de63e8a9b8ca2018fd73bdb9e01ad0d
2020-03-16 00:28:35 +02:00
Jess G
39cb821196
satellite/overlay: rm combinedcache, fix IP naming to be network (#3798)
* rn combinedcache, rm dns node lookup

Change-Id: I239f07211764b097d851230d8c81900a47756e9e

* excludeIPs -> excludedNetworks

Change-Id: Ifa6f44ab17457cdd5aff4cd5694296867c18b179

* use lowercase var name

Change-Id: I825aad2b718c71f455e747be18f8cabd02aabe55

* update Getnetwork name

Change-Id: I002a1b7bc6b4ef40159c0cd2b0ef209f80a9c503

* fix comments

Change-Id: Ibddf5b9ffa9d685af6c392d893db063ef18e45fa

* update comments with ipv6

Change-Id: I31758b7d4979e7c27d014668f4fb532ad838cda2

Co-authored-by: Stefan Benten <mail@stefan-benten.de>
2020-03-12 11:37:57 -07:00
JT Olio
051569c69f
satellite: enable open registration (and add flag that disables it) SM-441
Change-Id: I47bfedb312089f6d2bfbab013bd74ad4b8aa5f5e
2020-03-11 03:53:34 +01:00
Qweder93
484ec7463a storagenode: notifications on outdated software version
Change-Id: If19b075c78a7b2c441e11b783c3c09fed55060c7
2020-03-02 16:48:02 +00:00
Egon Elbre
64330c55b3 all: use pbgrpc
common/pb moved grpc to a separate package common/pb/pbgrpc.
This updates this repository to use it.

Change-Id: I2de2a190688871cf9cb61f7ea511f8a01e264e4e
2020-02-26 21:27:47 +02:00
Yingrong Zhao
77f67a8086 satellite/metainfo: add timeout for delete request
Change-Id: I9cad6d7ea185fc2c0ed4e58b42e4e3a78178a79f
2020-02-20 09:10:16 +00:00
Yingrong Zhao
e6da8d0249 satellite/metainfo: use global limiter for DeletePieces Service
we want to return back to the user as quick as possible but also keep
deleting remaining pieces on the storagenodes

Change-Id: I04e9e7a80b17a8c474c841cceae02bb21d2e796f
2020-02-19 12:17:36 +00:00
Qweder93
dc075eaa96 satellite/payments : deposit bonuses (credits) added
Change-Id: Ib151bbb9b02d655fa619c53bfbc04ed6f3bb39e0
2020-02-11 11:11:42 +00:00
Jeff Wendling
7999d24f81 all: use monkit v3
this commit updates our monkit dependency to the v3 version where
it outputs in an influx style. this makes discovery much easier
as many tools are built to look at it this way.

graphite and rothko will suffer some due to no longer being a tree
based on dots. hopefully time will exist to update rothko to
index based on the new metric format.

it adds an influx output for the statreceiver so that we can
write to influxdb v1 or v2 directly.

Change-Id: Iae9f9494a6d29cfbd1f932a5e71a891b490415ff
2020-02-05 23:53:17 +00:00
Egon Elbre
8dea4f52db satellite: add control panel
Change-Id: Id48246e9bcd4c6ec643277fe740937b2e42ad85b
2020-01-30 08:06:43 -05:00
Egon Elbre
4e2bf81719 pkg/debug: add better title
Change-Id: Icc6114f4e7523cfe6c7984ef1f6eec664ae4ee65
2020-01-30 07:49:40 -05:00
Egon Elbre
d10d6fd153 storagenode,satellite: ignore error on listening debug port
Change-Id: Id3a6d153535776ce41f8edf2bd6f6dad5e2a60bf
2020-01-29 18:06:02 -05:00
Egon Elbre
f237d70098 storagenode,satellite: use pkg/debug
Use debug.Server in storage node and satellite for customizing debug server.

Change-Id: I7979412376d028cadf29656d838ab94f18e2aa99
2020-01-29 16:30:31 -05:00
Yaroslav Vorobiov
083b396c16 satellite/payments: allow floating point numbers for pricing
Change-Id: I78b60134cf043746efef5371b761939a10f75aaf
2020-01-28 22:52:13 -05:00
Egon Elbre
e319660f7a private/lifecycle: implement Group
lifecycle.Group implements controlling multiple items such
that their startup and close works.

Change-Id: Idb4f4a6c3a1f07cdcf44d3147a6c959686df0007
2020-01-29 00:37:33 +00:00
Isaac Hess
40a890639d satellite/orders: Flush all pending bandwidth rollup writes on shutdown
Currently we risk losing pending bandwidth rollup writes even on a clean
shutdown. This change ensures that all pending writes are actually
written to the db when shutting down the satellite.

Change-Id: Ideab62fa9808937d3dce9585c52405d8c8a0e703
2020-01-23 08:12:41 -07:00
Ethan
21a5d70a83 satellite/metainfo: Rate limiting - API requests
Limits how many times metainfo APIs can be called per second by project ID. If limit is exceeded, the API will return Unauthorized/Too Many requests.

Limit per second and the size of the limiter cache per project are configurable, as well as whether the limiter is enabled.

Tests added/updated for the new rate_limit field in projects table.
Tests added for exceeding limits and disableing limiter.

Change-Id: Ic8ad102de3b690a475809d4f684156d5715f20fa
2020-01-21 14:25:04 +00:00
stefanbenten
f4097d518c satellite: reduce logging of node status
Change-Id: I6618cf4bf31b856acd7a28b54011a943c03ab22a
2020-01-18 17:47:59 +00:00
Ivan Fraixedes
6dc948da46
satellite/metainfo: Create service for deleting pieces
Create a service for deleting pieces of storage nodes.

Currently the DeletePieces method returns after a success threshold,
completion or a timeout.

The end goal is to return when reaching the success threshold and
leaving the remaining goroutines running after DeletePieces method
returns and add a life cycle to the service that it waits for them when
it closes.

This is the first commit for ticket:
https://storjlabs.atlassian.net/browse/V3-3476

Change-Id: If740bbf57c741f880449980b8176b036dd956c7b
2020-01-15 15:24:20 +01:00
Isaac Hess
4950d7106a satellite/orders: Add write cache for bw rollups
Change-Id: I8ba454cb2ab4742cafd6ed09120e4240874831fc
2020-01-13 22:40:51 +00:00
Egon Elbre
f41d440944 all: reduce number of log messages
Remove starting up messages from peers. We expect all of them to start,
if they don't, then they should return an error why they don't start.
The only informative message is when a service is disabled.

When doing initial database setup then each migration step isn't
informative, hence print only a single line with the final version.

Also use shorter log scopes.

Change-Id: Ic8b61411df2eeae2a36d600a0c2fbc97a84a5b93
2020-01-06 19:03:46 +00:00
Egon Elbre
6615ecc9b6 common: separate repository
Change-Id: Ibb89c42060450e3839481a7e495bbe3ad940610a
2019-12-27 14:11:15 +02:00
Ivan Fraixedes
366f4b9493
satellite: Create method for deleting pieces of SNs
Create a method for deleting pieces of the storage nodes and add a test
for it.

Change-Id: I5fffc8c43d900d317961098b1d135ce3223b73ea
2019-12-18 10:38:16 +01:00
Yaroslav Vorobiov
77839dd41b
satellite/console: project usage limits api (#3702) 2019-12-12 14:58:15 +02:00
Egon Elbre
36fead0093 satellite/metainfo: add UserAgent support to endpoints (#3548) 2019-11-26 03:12:37 -08:00
Yingrong Zhao
79a4fff6c7
satellite/referrals: set up referrals service and http endpoints (#3566) 2019-11-25 16:36:36 -05:00
Nikolai Siedov
3fe518d547
satellite: added ability to inject stripe public key post build (#3560) 2019-11-18 13:38:43 +02:00
Yaroslav Vorobiov
53c6741ba6
satellite/payments: add API for retrieving conversion ratio, convert tokens to USD before applying to balance (#3530) 2019-11-15 16:59:39 +02:00
Yehor Butko
a8e4e9cb03
satellite/payments: project usage charges (#3512) 2019-11-15 16:27:44 +02:00
Egon Elbre
ee6c1cac8a
private: rename internal to private (#3573) 2019-11-14 21:46:15 +02:00
Yaroslav Vorobiov
0b32690d0a satellite/peer: add payments config (#3488)
* satellite/peer: add payments config

* remove stripe-key from console config

* update config lock

* fix imports

* fix config-lock
2019-11-05 21:26:19 +01:00
Yaroslav Vorobiov
35edc2bcc3 satellite/payments: invoice creation (#3468) 2019-11-05 15:16:02 +02:00
Egon Elbre
9c59efd33d
satellite/rewards: ensure that partner information is asked from a service (#3275) 2019-11-05 14:58:09 +02:00
Yehor Butko
5cb46d2ce3
satellite/payments: mock payment service created, api calls from frontend returned (#3448) 2019-11-04 12:54:25 +02:00
Maximillian von Briesen
590312970d satellite/gracefulexit: add flag for enabling/disabling graceful exit on the satellite (#3437) 2019-11-01 16:21:24 +02:00
Yaroslav Vorobiov
30a3205745
satellite/payments: update account balance (#3379) 2019-10-29 18:04:34 +02:00
Ethan Adams
e54d290d2e satellite/gracefulexit: Add signatures for success/failed exit finished messages. (#3368)
* add signatures, fix process loop bug, move delete to on success

* added tests for signatures

* PR comment updates

* fixed setting reason by default.

* updates for PR comments

* added signed failure when verificationi fails

* moved to sign_test

* fix panic

* removed testplanet from test
2019-10-25 16:36:26 -04:00
Natalie Villasana
696c567e89
satellite/gracefulexit: add piece hash validation for successful transfer (#3313) 2019-10-24 15:38:40 -04:00
Yaroslav Vorobiov
1a304f5ef9
satellite/payments: add payments loop, update pending transactions (#3318) 2019-10-23 15:04:54 +03:00
Bryan White
f468816f13
{internal/version,versioncontrol,cmd/storagenode-updater}: add rollout to storagenode updater (#3276) 2019-10-21 12:50:59 +02:00
Bryan White
243ba1cb17
{versioncontrol,internal/version,cmd/*}: refactor version control (#3253) 2019-10-20 09:56:23 +02:00
Yehor Butko
26cc625dc6
satellite/console: payments api (#3297) 2019-10-17 17:42:18 +03:00
Cameron
76ad83f12c
satellite/accounting: add redis support to live accounting (#3213)
* set up redis support in live accounting

* move live.Service interface into accounting package and rename to Cache, pass into satellite

* refactor Cache to store one int64 total, add IncrBy method to redis client implementation

* add monkit tracing to live accounting
2019-10-16 12:50:29 -04:00
Jess G
87a426f228 internal/testplanet: add satellite.API to testplanet (#3237) 2019-10-14 16:01:53 -04:00
Jennifer Li Johnson
b185dbbee2
satellite/discovery: remove discovery related code (#3175) 2019-10-14 10:57:01 -04:00
Jess G
6e6d0ad9b8 split satellite: create separate API process (#3152) 2019-10-02 19:02:47 -04:00