Commit Graph

4694 Commits

Author SHA1 Message Date
Moby von Briesen
708cb48aa6 storagenode/orders: implement orders filestore on storagenode
* Add all new orders to the orders filestore instead of the database.
* Submit orders from the filestore to the new satellite SettleWindow
endpoint.

The orders filestore will eventually replace the orders DB completely.
For now, we will still be checking the orders DB and submitting those
orders if they exist. In a later release, we will completely remove the
orders DB, but we need both the DB and filestore for the transitionary
period.

Change-Id: Iac8780fd5ab770296181bbd313e1d335f072d4dc
2020-08-19 15:00:35 +00:00
Ethan
5445d595c0 storagenode/gracefulexit: Wait for the worker delete and transfer goroutines to finish before completing the exit
A failed test showed the same piece being deleted twice. This happens if the graceful exit completes before a previous piece deletion finishes. This change adds a "wait" on the limiter before executing the delete all step when GE is done.

Change-Id: I1c8c49d1e501c2728c80d4224a4854e742be27da
2020-08-19 14:20:26 +00:00
Ivan Fraixedes
7f8df74070
private/testplanet: Use config with name set when empty
In testplanet Run function we create a new configuration variable on
each t.Run for setting the value to the config name field when it's
empty, however the new copy of the configuration was not used.

Change-Id: I9da34e743f9648850c96556eab0349e742db3aac
2020-08-19 13:12:10 +02:00
Egon Elbre
be3fd0147e storagenode/storagenodedb: database name in all preflight errors
Shorten the error strings and include database name in all potential
preflight errors.

Change-Id: Ic92ca1ec6e14ffbddb0a0cf89e357eec9532d27e
2020-08-18 16:31:19 +03:00
Egon Elbre
b4c8e219c7 satellite/orders: calculate order expiration inside signer
Change-Id: I07f79eeb1ab41b061a1f3146f684bd21291cffb0
2020-08-18 13:21:16 +03:00
Egon Elbre
189ab07846 satellite/orders: use Signer in CreateGetOrderLimits
Change-Id: Icb7ed4f1af1dabbbb68cb6f6e1f86d93a9b5faa3
2020-08-18 13:20:00 +03:00
Egon Elbre
cd5e99ea6b satellite/orders: Signer for simplifying signing logic
Create a separate struct for signing order limits.

Change-Id: I8f8f5245040efa8c03138512be9248d4834f3f36
2020-08-18 13:19:16 +03:00
Qweder93
01bb2bd17d satellite/audit: verifier checks if node made sucess GE before auditing
Change-Id: Ia6cde4e9fcf11020a5301d38065f7159f276eb80
2020-08-17 23:37:57 +03:00
VitaliiShpital
3383acc3c8 web/satellite: low balance banner removed for no paywall users
WHAT:
yellow low balance banner removed from billing page for no paywall users

WHY:
better user experience for no paywall users

Change-Id: I1c254d7696e166073c66a6fbb82c134903cbc920
2020-08-14 16:20:47 +00:00
Yingrong Zhao
0518b16370 satellite/piecedeletion: move node info retrieval into the service
This change will require less work for the user of peiecedeletion
service by moving overlay database call into the package.

Change-Id: I14a150ab71fe885780e7a7a74db006a779507ae5
2020-08-13 16:07:54 +00:00
stefanbenten
c7b86a3481 satellite/admin: add check project usage endpoint and fix some leftover http.Error handling
Change-Id: I1ae3e7cb723a553f9c5a3a752beab0a27b0293bc
2020-08-13 13:48:34 +00:00
NickolaiYurchenko
4cdba365ef web/storagenode: payout history table
Change-Id: I448ea8424baf31400d9868ef9ca2b8002caa7bbd
2020-08-13 12:05:56 +00:00
Matt Robinson
c548475662
build: Go 1.14.7 (#3926)
Co-authored-by: Egon Elbre <egonelbre@gmail.com>
2020-08-12 12:42:08 +02:00
Yingrong Zhao
b71da59f8a satellite/metainfo: fix client context cancelation
We should still ignore client cancelation.
Also remove logger dependency on `Report`.

Change-Id: I59987c6b1aaa93202b07e5ef6e207d5191542b52
2020-08-11 20:58:23 +00:00
Isaac Hess
a14887e20b satellite/metainfo: Add GetObjectIPs method
This adds the unimplemented GetObjectIPs method to metainfo endpoint so
we can import new common protobuf definitions.

Change-Id: I154f26baccb6bb3c66de3eb25611930545c9754b
2020-08-11 11:39:15 -06:00
Egon Elbre
94a09ce20b all: add missing dots
Change-Id: I93b86c9fb3398c5d3c9121b8859dad1c615fa23a
2020-08-11 17:50:01 +03:00
Ivan Fraixedes
0525858b72 docs/blueprints: Fix typos horizontal scale GC
Fix a few typos in the horizontal scale garbages collection blueprint
document.

Change-Id: I25a0e59ae526e6f270ea4e8b5eac36d779d597dc
2020-08-11 13:53:52 +00:00
VitaliiShpital
c921710247 web/satellite: confirm saving API key modal implemented
WHAT:
modal with saving API key confirmation implemented for onboarding tour

WHY:
ensure user saved API key

Change-Id: I9bc000cf5a19a9d6fc9eb3a4bfb9156d6b4bc78f
2020-08-11 13:19:01 +00:00
Caleb Case
00f9882ad5 bump to uplink v1.2.0
Also fixup config and related tests for storj.io/common
bee93690ab909580993f402b594ad5a14760c39f

Change-Id: I6897d925cd28b139c1fc2b43e63b08c0cc3fb688
2020-08-10 17:11:09 +00:00
Ethan
ab1d0f097d satellite/storageusage: Group accounting rollups at_rest_total by day
When investigating a gap in storage usage data in the SN dashboard, I noticed that there were 2 entries in the accounting_rollups table on the date of the gap.
This change accounts for multiple entries in the accounting_rollups table for a given day.

Change-Id: Ibf2b5d0455117cb0417163e8fcfb7e509d594171
2020-08-10 15:03:15 +00:00
Kaloyan Raev
7552ff26ec satellite/db: drop project_invoice_stamps table
It's an obsolete table from earlier state of Stripe invoices
implementation. No code is currently using it. It is confirmed that this
table is currently empty across all satellites.

Change-Id: I12d2756578faf8418ea8f3b09088e885694b8925
2020-08-10 13:22:10 +00:00
Qweder93
4ee1b2d45a storagenode/console: added list of all audits per satellite to sno dashboard/satellites
Change-Id: I52e58748d6467f372d9a308347fc77e400d137e2
2020-08-10 12:55:07 +00:00
Qweder93
373934efb2 storagenode/heldamount: payout history: removed extra doubling with surge percent, added held percent
Change-Id: Idd3927c3130bff771e5437b9b18b4a4907f787e4
2020-08-10 15:29:34 +03:00
NickolaiYurchenko
14bb00f713 web/storagenode: all stats held history
Change-Id: Ief40a97a7e120dccefc04c3c2eb78b8492dff17d
2020-08-07 17:34:21 +03:00
NickolaiYurchenko
6ec7bc8b5d web/storagenode: last month estimated payout
Change-Id: I8b8c96dd3b4a8112abb2dbae5f09c97941012e8c
2020-08-07 13:46:05 +00:00
NickolaiYurchenko
32e1f16b48 web/storagenode: current month held amount for all satellites
Change-Id: I716a2695e5da19bfbec55d03e96bde0d2faa5ced
2020-08-07 13:45:52 +00:00
Michal Niewrzal
6e1ff78cdd satellite/metainfo: extend TestAttributionReport
Small extension to test case where another partner is upload/downloading
to/from the same bucket as partner which creates this bucket.

Change-Id: Ib674fe5f95f868b71341e30aba5e2440847738f4
2020-08-07 11:17:52 +00:00
Yingrong Zhao
2b2cb468bf satellite/metainfo: implement DeleteObjectPieces with objectdeletion package
Use new objectdeletion package for deleting pointers.

In the best case scenario, it will make on database call to fetch
information about the number of segments. And another request to delete
and fetch information about other segments.

This PR also changes our object deletion API to return no error when an
object is not found but instead consider such operation as success. This
behavior is asligned with S3 API and makes the code less complex.

Change-Id: I280c56e8b5d815a8c4dafe8227689467e899775a
2020-08-06 16:10:13 -04:00
Michal Niewrzal
ceb2eee494 satellite/metainfo: make overwrites impossible without delete permission
https://storjlabs.atlassian.net/browse/USR-1007

Change-Id: Ic7fa600a00a53b80a37acee4ac994be6ea4cb279
2020-08-06 18:01:44 +00:00
Qweder93
6e90ca8b0e cmd/storagenode: CLI dashboard last contact renamed to status
Change-Id: I29bcd62d0d2970d6294dc642b442b60a3331bbd6
2020-08-06 13:54:07 +00:00
Michal Niewrzal
88dcc93f3c satellite/metainfo: use user PartnerID for bucket attribution
Change-Id: I20f1bd432333f9b37ca8fb457c349eff94ffb392
2020-08-06 13:14:07 +00:00
crawter
68a2726e2a web/storagenode: telemetry removed
Change-Id: I26e9d96178539d76a0aecaf5e62d76a56b6559a3
2020-08-06 12:36:59 +03:00
Malcolm Bouzi
2ef9f7f644
change doc link (#3924)
Co-authored-by: VitaliiShpital <takur12345@gmail.com>
2020-08-06 10:51:07 +02:00
stefanbenten
d654ab5fa0 satellite/admin: change returned data to be json encoded
Change-Id: I239779fe54328fe854fcbdd71f7906875cd76635
2020-08-05 15:33:39 +02:00
Qweder93
f804f03b1f storagenode/heldamount: payuout history updated
Change-Id: I6dc91e9eed51f9b81af3e47a45168c43d254356a
2020-08-05 09:58:34 +00:00
Moby von Briesen
e02adfe5e9 satellite/overlay/config.go: Add AuditHistoryConfig to overlay
Adds AuditHistory{WindowSize, TrackingPeriod, GracePeriod,
OfflineThreshold}. These values will be used to track offline audits over
time, and to suspend/disqualify nodes for being offline for too long.

Change-Id: I05f7dbc3c034bdc53c4fbd7719c71a44f37ec6a5
2020-08-04 18:18:56 +00:00
Moby von Briesen
5dfe27f175 satellite/{repair,overlay}: Use overlay NodeSelectionCache for repair uploads
This change removes the overlay function FindStorageNodesForRepair,
which skips using the node selection cache and hits the database
directly. Otherwise, it is functionally identical to
FindStorageNodesForUpload, which checks the node selection cache first.

When selecting nodes for PUT_REPAIRs, we now call
FindStorageNodesForUpload instead of FindStorageNodesForRepair to reduce
database load.

Change-Id: If34e109695b2ed2b8fb6759115bf769a3459684e
2020-08-04 12:50:12 -04:00
Jessica Grebenschikov
f27285197d docs/blueprint: scale garbage collection
Change-Id: Iacd6f7f0a24ce91fa3da21a04707f3a6d27766f6
2020-08-03 23:52:40 +00:00
Jeff Wendling
85a74b47e7 satellite/orders: 3-phase rollout
This adds a config flag orders.window-endpoint-rollout-phase
that can take on the values phase1, phase2 or phase3.

In phase1, the current orders endpoint continues to work as
usual, and the windowed orders endpoint uses the same backend
as the current one (but also does a bit extra).

In phase2, the current orders endpoint is disabled and the
windowed orders endpoint continues to use the same backend.

In phase3, the current orders endpoint is still disabled and
the windowed orders endpoint uses the new backend that requires
much less database traffic and state.

The intention is to deploy in phase1, roll out code to nodes
to have them use the windowed endpoint, switch to phase2, wait
a couple days for all existing orders to expire, then switch
to phase3.

Additionally, it fixes a bug where a node could submit a bunch
of orders and rack up charges for a bucket.

Change-Id: Ifdc10e09ae1645159cbec7ace687dcb2d594c76d
2020-08-03 17:01:42 +00:00
Rafael Gomes
935f44ddb7 satellite/metainfo: Add Delete Service config
Change-Id: I0a6e3ce1adfe1488eb23da9dda92877af1834599
2020-08-03 14:28:02 +00:00
Rafael Gomes
50ac53f764 satellite/metainfo/objectdeletion: Implement batching
Implement an upper bound for how many objects the service
will handle at a time.

Change-Id: Id90a5c6197ed510a569f4f15134d06e1a9b315a1
2020-08-03 14:09:56 +00:00
Qweder93
53a5d18e1a storagenode: fixed logging about piece being moved to trash, and added logging when piece was actually deleted
Change-Id: I46f6a141b27033c2087b5c4681506d80b90f4a18
2020-08-02 20:00:05 +03:00
Michal Niewrzal
20184d3604 satellite/metainfo: move TestAttributionReport to attribution tests
Additionally test was simplified by adding ability to set user agent for
testplanet uplink.

Change-Id: I82942c2280562b5118a42aa8e1e0f53092f8dbe1
2020-07-30 19:18:15 +00:00
Michal Niewrzal
4561d9bdb0 satellite/console/consoleweb: add support for partner parameter for
signup page

With this change partner id can be associated with user during creating
account by adding `?partner=partner_name` parameter to signup page url
e.g. https://tardigrade.io/signup?partner=mongodb

https://storjlabs.atlassian.net/browse/USR-999

Change-Id: I12a5ebec92a6f5135909447172ef24da57fb1c68
2020-07-30 11:29:31 +00:00
Kaloyan Raev
edfd3d7661 satellite/payments: delete credits and credits_spendings db tables
Jira: https://storjlabs.atlassian.net/browse/USR-822

This the last step of dropping these 2 db tables. It also deletes all
code associate with them.

Change-Id: I8be840dc2a7be255cf6308c9434b729fe4d9391e
2020-07-30 12:19:57 +03:00
NickolaiYurchenko
52c651286a web/storagenode: node id copy
Change-Id: I07f19c2ff3d1771761b16f6adfea741b2eac01fc
2020-07-29 20:47:11 +00:00
NickolaiYurchenko
89dd0b475b web/storagenode: added ability to show and copy satellite id
Change-Id: I0f7ec1f0b92ab9044cfcbe8b7550db42c25c08a1
2020-07-29 20:46:50 +00:00
Caleb Case
7735b797c0
Bump dependencies for various fixes.
Change-Id: I79e4a985aef38f48ef49683eb19ce9dc0b3a2f85
2020-07-29 11:27:35 -04:00
Jeff Wendling
bdb3a10d39 mod: bump drpc
Change-Id: I584a2f55a391ce26e9fb1675e9f8a678febe649f
2020-07-29 14:24:07 +00:00
Michal Niewrzal
14f49558ac satellite/admin: fix test failing at the end of month
Change-Id: I6db6080614eaa0d775c730a7eec571b02da03e7f
2020-07-29 14:17:11 +02:00