Commit Graph

5638 Commits

Author SHA1 Message Date
Yaroslav Vorobiov
a5fd903177 storagenode/reputation: add vetted at timestamp
Change-Id: I02d59414b6b172cf7f7bfc92df222cf4a5574e0e
2021-07-07 18:11:54 +03:00
Michał Niewrzał
2e9d3a737c satellite/metabase: fix listing prefixes with cursor set
We were not skipping the initial prefix from the cursor.

Change-Id: I2bb472e960b92cae77fd1226de0b26fac79c429b
2021-07-07 14:30:46 +00:00
Qweder93
8f4505f532 multinode: handling offline nodes added
nodes/service and payouts/service returns node with status offline
or 0 in case if node is offline.

Change-Id: I74c03dcba9ddc9c05885ab329f80f3b14fe8c0fa
2021-07-07 13:25:32 +00:00
Kaloyan Raev
73b9223758 satellite/metainfo: implement UpdateObjectMetadata
Change-Id: I7d791bb84b73299d395b587074e721c14e4be31c
2021-07-07 12:57:32 +00:00
Yaroslav Vorobiov
818f6c6ea6 multinode/console: add summary to storage usage API
Change-Id: Ia8a1e598d667f25461f73f1626da22113cb7caeb
2021-07-07 15:00:05 +03:00
Igor
b0ef9b9b67 Create MAINTAINERS.md 2021-07-07 13:27:07 +02:00
Jeff Wendling
ef7b89cc03 cmd/uplinkng: remove global flags
this changes globalFlags to be a ulext.External
interface value that is passed to each command.

rather than have the ulext.External have a Setup
call in the way that the projectProvider used to
we make all of the state arguments to the functions
and have the commands call setup themselves.

the reason it is in its own package is so that
cmd/uplinkng can import cmd/uplinkng/ultest
but cmd/uplinkng/ultest needs to refer to whatever
the interface type is to call the function that
creates the commands.

there's also quite a bit of shuffling around of
code and names. sorry if that makes it tricky
to review. there should be no logic changes, though.

a side benefit is there's no longer a need to do
a type assertion in ultest to make it set the
fake filesystem to use. that can be passed in
directly now. additionally, this makes the
access commands much easier to test.

Change-Id: I29cf6a2144248a58b7a605a7ae0a5ada5cfd57b6
2021-07-06 17:26:51 -04:00
Jeff Wendling
f474bb6179 cmd/uplinkng: update for breaking clingy changes
clingy changed some and this is just fixes for that

Change-Id: I729aed6329fe0988fcb9b4407f16966a753b3204
2021-07-06 17:26:51 -04:00
Vitalii Shpital
bbd3efaeed web/satellite: Paid Tier add payment modal implemented
Added new PaidTier-related modal where user can add CC or STORJ Tokens.
Becomes visible on CTA click on Paid Tier banner at the top.

Change-Id: I51015e95d396e21d5c1a1728b8f753798626c09e
2021-07-06 16:16:18 +00:00
NickolaiYurchenko
4d418c13c3 web/multinode: routes restriction in no node added
checks if there is a try to navigate to some of internal route and no nodes added so far.
redirects to add first node screen if so.

Change-Id: Ibde47c5b81ae33462afe00f2fd7a54e8b295933a
2021-07-06 15:52:46 +00:00
crawter
b8e6e5c6c8 web/multinode/nodes: status calculation placed to backend
Change-Id: Id3ee876155d64485e115458a079617ac2db5310a
2021-07-06 15:06:56 +00:00
Moby von Briesen
e36001b7cf satellite/console: Implement paid tier
When a user adds a credit card, switch them to the paid tier and update
their projects with new bandwidth/storage limits. New projects for the
paid tier user will also have the updated limits.

The new limits are:
* storage per project - 50 GB free/25 TB paid
* bandwidth per project - 50 GB free/100 TB paid

Change-Id: I7d6467d077e8bb2bbe4bcf88ab8d75490f83165e
2021-07-06 14:30:12 +00:00
crawter
3b0b0ba3c4 multinode/nodes: node status extended with NotReachable
Change-Id: Ibd104945a79e07497f4963442c12a555ea02fbaf
2021-07-06 17:04:19 +03:00
Malcolm Bouzi
ecf3de3c3b web/satellite: Add new intro section to register page
- With the removal of the sign up page in Webflow, we need to add section with some product intro

Change-Id: Ifa84c670a6cd93ed79e1791e07d40647c43017cd
2021-07-06 13:17:14 +00:00
nadimhq
60717e356d
UItests: added an onboarding wizard test
These tests are used to check that newly created accounts are able to use the built-in wizard to introduce new users on how to get started using STORJ DCS during the onboarding phase
2021-07-06 07:36:42 -04:00
Qweder93
6c02258925 storagenode/peer: register multinode Payouts endpoint
Change-Id: Iaaba0157e605bb51a73244517986d6b49a59d8d5
2021-07-06 10:09:24 +00:00
Egon Elbre
74af44c12e satellite/metabase: make IteratePendingObjectsByKey recursive
Change-Id: I593164302559bec69393947e19820f8eef338830
2021-07-06 07:27:22 +00:00
Kaloyan Raev
a6086d421f satellite/metabase: implement SetObjectMetadataLatestVersion
It replaces UpdateMetadataObject that hasn't been used anywhere yet.

Change-Id: I1b2131acd75924c487a1c3ca099674251c5c9b56
2021-07-06 01:06:16 +00:00
Yaroslav Vorobiov
68627e7d80 multinode/console: add reputation satellite api
Change-Id: I7cef6c1c271607f7485f604d5b61587558a31878
2021-07-05 15:32:22 +00:00
Egon Elbre
bf5194d134 satellite/satellitedb: optimize ProjectAccounting.ArchiveRollupsBefore
The previous query was making a full table scan. This modifies code to
do the queries separately on each action. It will probably be slower on
a small table, however there should be a several magnitude boost on
large tables.

Change-Id: Ib8885024d8a5a0102bbab4ce09bd6af9047930c9
2021-07-02 17:06:58 +00:00
Egon Elbre
c248651f3f satellite/metabase/{meta,segment}loop: fix test flakiness
When the delta is very small from the bounds then the ratio calculation
doesn't work that well. Let's allow 100 from the bounds, since that
would be expected in any case.

We won't add a configuration for it, since it's not that useful.

Change-Id: I049066a42470b825f430b7f32ebe92d544c6cc8b
2021-07-02 19:16:11 +03:00
Michał Niewrzał
7cbff95090 cmd: delete metabase-expireat-migration
This migration was applied to all satellites and can be removed.

Change-Id: I6c63eeb60da17f6bb8b2d4dd3cd5e6e3d9612638
2021-07-01 18:11:48 +02:00
Vitalii Shpital
bab43af6ce web/satellite: added Upgrade to Paid Tier banner
Added new info banner to show user their used and total storage values with a button to upgrade to Paid Tier with auto limit increase

Change-Id: I827818dcb5179358df246218a47feb61bc1a1bac
2021-07-01 15:05:03 +00:00
NickolaiYurchenko
1230cde317 web/multinode: disk space usage and stat charts
Change-Id: Ib2b1d6f2b78d57bd43ae1152921d98a1d6c31243
2021-07-01 14:21:08 +00:00
Michał Niewrzał
b900f6b4f9 satellite/repair/checker: move checker to segment loop
Change-Id: I04b25e4fa14c822c9524586c25bde89db2a6cad9
2021-07-01 13:51:56 +00:00
NickolaiYurchenko
8686267e06 web/multinode: storage api, service and store module created
Change-Id: Ieb3dbcd9c967388315f5203598ff56a848477476
2021-07-01 12:32:09 +00:00
NickolaiYurchenko
55754df110 web/multinode: bandwidth related unit tests
Change-Id: If8da681e15db9e6f9f333bacdeab7670c48f2dfb
2021-07-01 12:32:04 +00:00
NickolaiYurchenko
cece8e4110 web/multinode: bandwidth charts added
bandwidth/egress/ingress chart and api/vuex connection

Change-Id: I16ba2bb82854a1d198384b3b8e6ffc4e58d8bb91
2021-07-01 12:31:58 +00:00
NickolaiYurchenko
cc5de4288b web/multinode: bandwidth api and store module created
Change-Id: I5a280f836d73ca9c34bab7dd5371bf1dce5a3974
2021-07-01 12:31:53 +00:00
Michał Niewrzał
0ca7583282 satellite/accounting: add total for bytes and segments to tallies
We want to calculate bucket tally only from iterating objects.
Object currently has an info about totals for bytes and segments.
We need to adjust tallies to keep those totals. Older entries will
be untouched and code will use totals only if available. Change
is adding columns for totals to bucket_storage_tally table and
is adding general handling for them.

Next step is to start using total columns instead of inline/remote.
This will be done with next change.

Change-Id: I37fed1b327789efcf1d0570318aee3045db17fad
2021-07-01 08:52:32 +00:00
TungHoang
e1379bea0f
storagenode/piecestore: allow rejecting slow clients
Estimate speed of the uploads by calculating the time a client has been in a non-congested state. Storage node is uncongested when it has used up over 80% of its maximum concurrent requests capacity.

Currently it's disabled by default.
2021-07-01 09:59:04 +03:00
Michał Niewrzał
4335b21332 satellite/metabase: add TotalEncryptedSize to LoopObjectEntry
We gonna need TotalEncryptedSize for buckets tally migrated to objects
iterator from metaloop.

Change-Id: I64774fdbda47abdaf5fbd370bd015b14bac9afdf
2021-06-30 16:43:31 +00:00
Michał Niewrzał
d53aacc058 satellite/repair: migrate to new repair_queue table
We want to use StreamID/Position to identify injured
segment. As it is hard to alter existing injuredsegments
table we are adding a new table that will replace existing
one. Old table will be dropped later.

Change-Id: I0d3b06522645013178b6678c19378ebafe485c49
2021-06-30 17:12:24 +02:00
Moby von Briesen
0ec3867ec0 satellitedb: add paid_tier column to users table
So that we can easily see whether a user is in the paid tier without
querying for payment methods.

Change-Id: I122566ddd0953203f852741fa12c71795bc1ec5c
2021-06-30 15:00:38 +00:00
Michał Niewrzał
aa7fd8c9cd satellite/satellitedb: fix ProjectAccounting project bandwidth calculation
Period end was calculated
incorrectly as it was still in current month but
should be the first day of next month.

Change-Id: I37451d29a9b901b69e6c3c401b333c58b3376d61
2021-06-30 14:54:12 +02:00
JT Olio
1c3688f1d6 storage/filestore: include satellite id in open_file_in_trash metrics
Change-Id: Ib7aa15f84c3acd9eedc4778bd2cde3072d0bf64b
2021-06-29 23:25:25 +00:00
JT Olio
cb18dc77fc satellite/stripecoinpayments: version is the wrong name
this service exists to do currency conversions, which is
the best I can assume that this was meant to be named.

Change-Id: Ia2416f5475749e8bfe8d05bf491649576f6d77bf
2021-06-29 15:31:11 +00:00
Moby von Briesen
d999a963ca satellite/console/.../consoleapi: Standardize serveJSONError
This change removes all the separate implementations for
`apiservice.serveJSONError()` and defines one for every service to use
in `consoleapi/common.go`.

Change-Id: Iabf184e5cba69a98eb25936ce11ebd07f02c8ff3
2021-06-29 09:40:07 -04:00
Michał Niewrzał
371517d93b satellite/accounting: fix as of system interval value for some tests
Live accounting has now as of system interval config but it was set incorrectly for some tests.

Change-Id: I6397d858c50e129bb8f9cd1a972047582ee9d59d
2021-06-29 10:16:48 +00:00
Moby von Briesen
4e95d27033 web,satellite: Remove paywall-related functionality
Because of our free/paid tier plan, we do not need a paywall anymore. We
have not used it in a while, but still have leftover code laying around.

Change-Id: Iaea8c39faf042a2f7a6b837727bb135c8bdf2907
2021-06-29 02:47:48 +02:00
Michał Niewrzał
4267a958d4 satellite/satellitedb: use AS OF SYSTEM TIME for GetProjectBandwidth query
Adding AS OF SYSTEM TIME to query that is calculating project bandiwdth.
As an addition method for setting interval is added as test doesn't
work well with default interval.

Change-Id: Id1e15be4f6afff13b9dc2b7f595e2edb6de28db9
2021-06-28 15:19:47 +00:00
Michał Niewrzał
70e6cdfd06 satellite/audit: move to segmentloop
Change-Id: I10e63a1e4b6b62f5cd3098f5922ad3de1ec5af51
2021-06-28 11:32:00 +00:00
Michał Niewrzał
8ce619706b satellite/audit: migrate to new segment_pending_audit table
Currently, pending audit is finding segment by segment location
(path) because we want to move audit to segmentloop and we will
have only StreamID and Position we need to add columns for those
fields. Altering existing table can cause issues while
migration and deployment. Cleaner choise is to make new table.
This change contains migration with new segment_pending_audit
table that will replace pending_audits table and adjustments
to use new table in the code.

Table pending_audits will be dropped with next release.

Change-Id: Id507e29c152da594bac1fd812c78d7ecf45ec51f
2021-06-28 13:19:49 +02:00
Fadila Khadar
a89c0763a0 satellite/gracefulexit: create table graceful_exit_segment_transfer_queue
table graceful_exit_segment_transfer_queue will be used to replace graceful_exit_transfer_queue. Currently, it uses the path of a segment to keep track of pieces to be transferred. As we want to use the segment metainfo loop, we will need to record stream_id and position of the segment instead of relying on object path.

This change also add a uses_segment_transfer_queue column to the graceful_exit_progress table to be able to know if a transfer has been initiated while using the old table.

Change-Id: Iafb1e8e65ba124e20de4a9ff76da181c3222de7e
2021-06-28 12:55:57 +02:00
Vitalii Shpital
d30fd77652 satellite/console: new endpoint to get total usage and limits for all the projects user owns
Added new endpoint and service method to return total usage and limits for all the projects that user owns.
It is needed for new paid tier UI

Change-Id: Ic5b67ca7b275ec4930d976a007168235c0500b70
2021-06-25 21:22:59 +00:00
Egon Elbre
4031336cbd private/lifecycle: monitor unexpected shutdowns
Change-Id: I9af3a95a1b60c1572cd57bb5e9539d60aa0337bb
2021-06-25 19:26:23 +03:00
Egon Elbre
3c8e7e679a mod: bump storj.io/private and monkit
This gives us nicer output for time.Time formatting when looking at
ongoing database queries.

Change-Id: I00c1ae4d162f537bd69fc2c8132c6507c5ee98cb
2021-06-25 16:57:00 +03:00
Michał Niewrzał
1f5fbbd24a satellite/metabase: while deleting bucket objects delete pieces in batches
It's possible that single object will have a lot of segments and
at the moment we are trying to collect all pieces at once and
send to storage nodes information about deletion. Such
approach my lead to using extensive amount of memory. This
change is handling this problem by calling DeletePieces
function multiple times with only part of pieces to delete for
a single call.

Change-Id: Ie1e66cd9d86d130eb89a61cf6e23f38b8cb8859e
2021-06-25 13:27:00 +00:00
Stefan Benten
a59232bb1d satellite: return directly to avoid shadowed err variable
Change-Id: I40046b37ba47f5038e8e0dd303b3ecb279441259
2021-06-24 20:52:13 +02:00
Cameron Ayer
8c124c6fa4 satellite/{reputation,overlay,satellitedb}: create reputation service, DB, add overlay method UpdateReputation
Define service and DB interface for storing node reputation data
and updating the overlay cache.
Add overlay service and DB method UpdateReputation.
See https://github.com/storj/storj/pull/4144

Change-Id: Iedd8bd3274457d26c595919303d55327c1464b8c
2021-06-24 16:19:15 +00:00