Commit Graph

5876 Commits

Author SHA1 Message Date
Michał Niewrzał
237360e765 metabase-orphaned-segments: fix args binding for delete command
As an additon I clarify log message a little bit.

Change-Id: Id2487cc218e3dd62ad5a05b56e8b1d5d070676a7
2021-07-22 11:00:37 +02:00
Fadila Khadar
6d60d412f0 satellite/gracefulexit: use segment loop
Join the segment loop instead of the metainfo loop, to iterate only over segments.

Change-Id: I06259d363b98d4e191f2bf2d82c9b47255ee484a
2021-07-21 15:12:25 +00:00
Fadila Khadar
c4202b9451 satellite/gracefulexit: use graceful_exit_segment_transfer_queue
For being able to use the segment metainfo loop, graceful exit transfers have to include the segment stream_id/position instead of the path. For this, we created a new table graceful_exit_segment_transfer_queue that will replace the graceful_exit_transfer_queue. The table has been created in a previous migration and made accessible through graceful exit db in another one.
This changes makes graceful exit enqueue transfer items for new exiting nodes in the new table.

Change-Id: I7bd00de13e749be521d63ef3b80c168df66b9433
2021-07-21 14:02:20 +00:00
Cameron Ayer
449c873681 satellite/repair/repairer: attempt repair GETs using nodes' last IP and port first
Sometimes we see timeouts from DNS lookups when trying to do
repair GETs. Solution: try using node's last IP and port first.
If we can't connect, retry with DNS lookup.

Change-Id: I59e223aebb436118779fb18378f6e09d072f12be
2021-07-21 13:13:06 +00:00
Fadila Khadar
b0d98b1c1a satellite/gracefulexit: allow use of graceful_exit_segment_transfer_queue
For being able to use the segment metainfo loop, graceful exit transfers have to include the segment stream_id/position instead of the path. For this, we created a new table graceful_exit_segment_transfer_queue that will replace the graceful_exit_transfer_queue. The table has been created in a previous migration.
This change gives access to this table.
Graceful Exit doesn't use the table yet, this will be done in a next change.

Change-Id: I6c09cff4cc45f0529813a8898ddb2d14aadb2cb8
2021-07-21 12:34:44 +00:00
Qweder93
73cdefbc41 munltinode/console: node online statuses extended with StatusUnauthorized and StatusStorageNodeInternal
Change-Id: I12c90169f8959e5eafe2a64bf8b412f7afc48c1c
2021-07-21 15:00:41 +03:00
Michał Niewrzał
f73fcee892 cmd: add tool for orphaned segments
This tool has two commands to execute. One is to 'report'
orphaned segments. Second to 'delete' orphaned
segments.

To find such segments tool is first finding all unique
segment stream ids. As a next step its removing from this
list stream ids of existing objects. What is left is a list of
orphaned segments.

Change-Id: I4a0ae3ad0b10a8d16572bfd22ac92cfa15ca19b3
2021-07-21 10:10:58 +00:00
Cameron Ayer
373ba8fd27 satellite/repair/repairer: metrics for repair bytes uploaded and downloaded
Change-Id: Icb0850692ecc155f6c8169edf1b045b2b546ff48
2021-07-21 09:23:19 +00:00
Cameron Ayer
adc0fbddfa satellite/audit: don't fail nodes for audit if not enough pieces downloaded
In most situations where we would not get enough shares to complete
an audit, something has probably gone wrong like a forgotten delete,
and nodes should not be failed. We have an alert when this occurs.
Check the logs to see what happened. If we decide the nodes should
get audit failures, we can do it manually.

Change-Id: Ib6e408082048d31197c37ebfd7f9031135fc938f
2021-07-20 20:28:18 +00:00
Michał Niewrzał
27a714e8b0 satellite/accounting/tally: use objects iterator instead metaloop
Bucket tally calculation will be removed from metaloop and will
use metabase objects iterator directly.

At the moment only bucket tally needs objects so it make no sense
to implement separate objects loop.

Change-Id: Iee60059fc8b9a1bf64d01cafe9659b69b0e27eb1
2021-07-20 15:52:18 +00:00
Vitalii Shpital
4c7384ca2f web/satellite: removed redundant files splitting
Since we removed login/registration explicit templates and styles from branding repo we don't need file splitting in main repo anymore.
So moved html and css code into components

Change-Id: Ib297e2576f16a7fddd17cc62f6c655adf96a59b6
2021-07-20 12:44:02 +03:00
Vitalii Shpital
2489b78d22 web/satellite: move user MFA secret generation to server-side
It was decided to move user MFA secret generation to server-side and be fetched by http request

Change-Id: I5e31d35fb78d1d9f3280518fe3eb543bb8ed4377
2021-07-19 13:48:09 +00:00
Egon Elbre
5a56021bca satellite/metabase: add intLimitRange
We have quite a bit code duplication in ensuring valid int ranges.

Change-Id: Ib31db8c4b2be6a677625cab5bcdddefb60401fe3
2021-07-19 07:51:16 +00:00
Moby von Briesen
bc2f81c2fa web/satellite: Add beta checkbox to registration page
Only if is-beta-satellite is set to true.
Also cleans up some styles.

Change-Id: I2273936ae2b41a3409fb985b13cc987224a3ba85
2021-07-16 15:55:11 +00:00
Moby von Briesen
7624bdd090 web/satellite: Return server-side error message during registration
It turns out, there are multiple different 400 errors that might be
returned from the server during registration. Rather than display
hardcoded text on the client ("Validation of reCAPTCHA was
unsuccessful"), this change simply displays the error sent by the
server.

As mentioned in the comment, we should eventually do this for all
errors, but that will be a more in-depth change. The purpose of this
commit is to unblock a point release.

Change-Id: Ideca107cc4039a0dabfa0fb02c943da920f7ff4f
2021-07-16 09:43:11 -04:00
Jeremy Wharton
1e0a1b15d4 web/satellite: Repair reCAPTCHA resetting upon registration error
Previously, the reCAPTCHA would only reset if a Bad Request
error was received. Resets should initiate regardless of error
type to prevent the same reCAPTCHA response from being submitted
more then once.

Additionally, registration errors could trigger a reset for a
nonexistent reCAPTCHA element.

Change-Id: Ib405707f1803cf41e5de192f31d75153136e6c67
2021-07-15 15:42:52 -05:00
Qweder93
4a98dd40e2 storagenode/payouts: historical payouts use satellitesDB instead of trustPool
Change-Id: I39f4215f4ebf91bd1b38fbcb5c58e6ba53ceff1b
2021-07-15 16:19:18 +03:00
Bill Thorp
18c3252025 cmd/uplink: recommend notAfter expiration caveat be set
The auth service has no way to remove access grant registrations that
lack expiration dates.  We want to encourage people to set them, so as
to slow the rate at which the auth service DB fills up.

Change-Id: I1ccf629cd995dc184d2d90333166eab34d34ae07
2021-07-14 17:05:28 +00:00
Vitalii Shpital
22e88c8f0a web/satellite: disable MFA functionality
Added disable user MFA functionality to account settings.
Disable MFA popup where user will have to enter MFA passcode first.
2 buttons (enable, disable)) are visible for now until backend is ready.

Change-Id: Iff8b497a370fc1c6c08c5ccaf01ce1f2dc317126
2021-07-14 17:38:57 +03:00
Vitalii Shpital
e463eb17ac web/satellite: added enabling user MFA functionality to account settings
Added feature flagged functionality for enabling user MFA.
Added new Popup where user will scan qr code and confirm enabling
by entering passcode from MFA app. Also recovery codes will be visible afterwords

Change-Id: Ie8d1bc83c941a08fd8701442601a2d20126c8892
2021-07-14 15:13:59 +03:00
Jeremy Wharton
0d8010e353 satellite/satellitedb: Implement multi-factor authentication db
Columns for MFA status, secret key, and JSON-encoded array of
recovery codes are added to the users table.

Change-Id: Ifed7e50ec9767c1670d9682df1575678984daa60
2021-07-13 18:18:36 -05:00
Moby von Briesen
4c912c7479 web/satellite: Update paid tier banner copy
Also, show paid tier banner if a user has not added a credit card,
rather than if they are flagged as "paid_tier" in the database. This
addresses issues where we might display the banner for users who have
already added a credit card before the paid tier commit: e36001b7cf

Change-Id: I0352d48ae9f97ceab81ae065ccb97be3e5516857
2021-07-13 13:34:47 +00:00
Yaroslav Vorobiov
6db6b76b27 multinode/console: embed web assets
Embed web static files in multinode binary to be able to
release multinode as single binary.
Add make commands to build multinode binary with embeded web
assets.

Change-Id: I348aff7a7d847fae5c021cbf59abc7f892c0df80
2021-07-12 18:51:24 +03:00
Moby von Briesen
30cd7d3da3 satellite/console: Update CSP for satellite UI
* Add recaptcha to Content Security Policy
* Remove segment and customer.io from Content Security Policy

Change-Id: Ie6d767d8c023f7e69f475c37f1535e4db90953fc
2021-07-12 10:32:15 -04:00
Moby von Briesen
5870502589 cmd/satellite: Add billing command for converting customers to paid tier
We have implemented the paid tier, but it currently only handles new
users entering paid tier. It does not convert users who have already
added a credit card previously. We still want to convert these users'
project limits. This billing command can be run once to convert all old
customers with a credti card. Afterwards, we should be able to safely
remove it.

Change-Id: Ia496580b8e72ef436375b74f590fe57cca704fa8
2021-07-12 13:37:37 +00:00
Vitalii Shpital
8855c0dff7 web/satellite: added MFA feature flag, updated client-side api and Vuex store module
Added feature flag for MFA
Added new client-side api call to enable MFA returning secret
Updated users Vuex module to include new API call

Change-Id: Ia9e10f68c4a7da39b4f7c1073e657c2de98fb0db
2021-07-12 10:12:59 +00:00
Brandon Iglesias
ee107fe8cd
satellite/rewards: adding partners to list (#4159) 2021-07-09 22:47:25 +02:00
Michał Niewrzał
76c8d5d2cb satellite/metrics: move metrics to segment loop
Change-Id: Iccc0e4b6d531590c9cc57c7c74a6ce4c46b7d6a2
2021-07-08 20:29:10 +00:00
NickolaiYurchenko
8c052048b6 web/multinode: redirect fix before route enters dashboard
makink node/infos request if trying to reach protected routes

Change-Id: I34af0437437f8c670ac2a6eb61902fbc9114e540
2021-07-08 19:08:24 +03:00
Qweder93
4d0fe39235 storagenode/satellites: address added, caching satellite's addresses from trust
Change-Id: Ica3eea5b8d81b176c6a4385fea803730b08ece16
2021-07-08 15:38:23 +00:00
Kaloyan Raev
a767aed591
satellite/{metabase,metainfo}: require StreamID for UpdateObjectMetadata
This avoids corrupting objects if reuploads and metadata updates happen
concurrently.
2021-07-08 17:50:37 +03:00
Michał Niewrzał
cbbbfca439 satellite/accounting: move storage node tally to separate service
Current tally is calculating storage both for buckets and
storage nodes. This change is moving nodes storage
calculation to separate service that will be using
segment loop.

Change-Id: I9e68bfa0bc751c82ff738c71ca58d311f257bd8d
2021-07-08 14:02:33 +00:00
NickolaiYurchenko
f8eebbc115 web/multinode: npm packages version audit & update
Change-Id: Ib0c2b5c50bdd6d1284ba732ead2b20a5b187f53e
2021-07-08 07:58:22 +00:00
Jeremy Wharton
a5f6bb9cc0 satellite/console: Add reCAPTCHA verification step to registration
The user must complete a reCAPTCHA in order to register.
ReCAPTCHA verification failure results in rejection of the
registration attempt.

Change-Id: I34ba7db414d756fd1aaebdc3d19cccbfc7fc1ea3
2021-07-07 21:34:07 +00:00
Jeremy Wharton
ec9ad5bd7d web/satellite: Don't require passphrase entry after generation
Remove the Enter Encryption Passphrase screen that
immediately follows the Generate Passphrase screen upon entering
the Object Browser for the first time.

Change-Id: I04fb00325e7798096dc209473f5553d6e5df0dd3
2021-07-07 21:11:17 +00:00
crawter
0495703848 web/multinode: reputation domain and api created
Change-Id: I9bf92015f32cdec1ef15f8d4a02b94987de12462
2021-07-07 16:07:47 +00:00
crawter
f79d697717 multinode/nodes: checking node availability before adding it
Change-Id: I7792ed359713f12fee9fb4ac0ea46703dfc8406e
2021-07-07 15:45:09 +00:00
Yaroslav Vorobiov
cbb4cd3fc3 multinode/reputation: add vetted at timestamp
Change-Id: Id35cb6cfdabf4bf2762e4a162cf3157afb0ff170
2021-07-07 18:11:54 +03:00
Yaroslav Vorobiov
a5fd903177 storagenode/reputation: add vetted at timestamp
Change-Id: I02d59414b6b172cf7f7bfc92df222cf4a5574e0e
2021-07-07 18:11:54 +03:00
Michał Niewrzał
2e9d3a737c satellite/metabase: fix listing prefixes with cursor set
We were not skipping the initial prefix from the cursor.

Change-Id: I2bb472e960b92cae77fd1226de0b26fac79c429b
2021-07-07 14:30:46 +00:00
Qweder93
8f4505f532 multinode: handling offline nodes added
nodes/service and payouts/service returns node with status offline
or 0 in case if node is offline.

Change-Id: I74c03dcba9ddc9c05885ab329f80f3b14fe8c0fa
2021-07-07 13:25:32 +00:00
Kaloyan Raev
73b9223758 satellite/metainfo: implement UpdateObjectMetadata
Change-Id: I7d791bb84b73299d395b587074e721c14e4be31c
2021-07-07 12:57:32 +00:00
Yaroslav Vorobiov
818f6c6ea6 multinode/console: add summary to storage usage API
Change-Id: Ia8a1e598d667f25461f73f1626da22113cb7caeb
2021-07-07 15:00:05 +03:00
Igor
b0ef9b9b67 Create MAINTAINERS.md 2021-07-07 13:27:07 +02:00
Jeff Wendling
ef7b89cc03 cmd/uplinkng: remove global flags
this changes globalFlags to be a ulext.External
interface value that is passed to each command.

rather than have the ulext.External have a Setup
call in the way that the projectProvider used to
we make all of the state arguments to the functions
and have the commands call setup themselves.

the reason it is in its own package is so that
cmd/uplinkng can import cmd/uplinkng/ultest
but cmd/uplinkng/ultest needs to refer to whatever
the interface type is to call the function that
creates the commands.

there's also quite a bit of shuffling around of
code and names. sorry if that makes it tricky
to review. there should be no logic changes, though.

a side benefit is there's no longer a need to do
a type assertion in ultest to make it set the
fake filesystem to use. that can be passed in
directly now. additionally, this makes the
access commands much easier to test.

Change-Id: I29cf6a2144248a58b7a605a7ae0a5ada5cfd57b6
2021-07-06 17:26:51 -04:00
Jeff Wendling
f474bb6179 cmd/uplinkng: update for breaking clingy changes
clingy changed some and this is just fixes for that

Change-Id: I729aed6329fe0988fcb9b4407f16966a753b3204
2021-07-06 17:26:51 -04:00
Vitalii Shpital
bbd3efaeed web/satellite: Paid Tier add payment modal implemented
Added new PaidTier-related modal where user can add CC or STORJ Tokens.
Becomes visible on CTA click on Paid Tier banner at the top.

Change-Id: I51015e95d396e21d5c1a1728b8f753798626c09e
2021-07-06 16:16:18 +00:00
NickolaiYurchenko
4d418c13c3 web/multinode: routes restriction in no node added
checks if there is a try to navigate to some of internal route and no nodes added so far.
redirects to add first node screen if so.

Change-Id: Ibde47c5b81ae33462afe00f2fd7a54e8b295933a
2021-07-06 15:52:46 +00:00
crawter
b8e6e5c6c8 web/multinode/nodes: status calculation placed to backend
Change-Id: Id3ee876155d64485e115458a079617ac2db5310a
2021-07-06 15:06:56 +00:00
Moby von Briesen
e36001b7cf satellite/console: Implement paid tier
When a user adds a credit card, switch them to the paid tier and update
their projects with new bandwidth/storage limits. New projects for the
paid tier user will also have the updated limits.

The new limits are:
* storage per project - 50 GB free/25 TB paid
* bandwidth per project - 50 GB free/100 TB paid

Change-Id: I7d6467d077e8bb2bbe4bcf88ab8d75490f83165e
2021-07-06 14:30:12 +00:00