Commit Graph

5711 Commits

Author SHA1 Message Date
Michał Niewrzał
be92ff1751 satellite/metabase: fix finishing bucket objects deletion
In case when number of deleted segments in a single
batch was multiplication of opts.DeletePiecesBatchSize it
was possible to finish deletion of objects to early.

Change-Id: I9181d4a3c64053d9df46a11a6e0cd22153c73ee9
2021-08-02 09:00:02 +00:00
Michał Niewrzał
804d221d42 satellite/satellitedb: remove tables unused after metaloop refactoring
Change-Id: I6e6c2ec32e43eb0c24c4ea0522443487634a5164
2021-08-02 08:01:26 +00:00
Michał Niewrzał
011b944382 satellite/metrics: fix metrics for total inline/remote bytes and segments
Change-Id: I567ce127590a4712cab296d28a19838e3a632021
2021-07-30 18:19:51 +02:00
paul cannon
fda316b461 Makefile: update channel name of #team-data
Avoids confusion when metrics need to be changed or updated.

Change-Id: I1d0bf939bd30cd66128eb04ccee5c4133447bfe9
2021-07-30 09:58:57 -05:00
Artur M. Wolff
10d7a63ab0 Makefile: include arm64 (instead of arm) executables in arm64v8 images
Change-Id: Ic9cd5cd7b052efcb75499c338dd60480f960c4fa
2021-07-30 14:01:48 +02:00
Vitalii Shpital
c6c2a1daf9 web/satellite: added MFA login step
Added Enter MFA passcode step to login after first attempt.
Connected client-side and server-side.

Change-Id: I4482afde50172bbf2c598aa01bbd220763b4f6d1
2021-07-30 08:50:19 +00:00
Egon Elbre
2118c89e9a web/multinode: bump node-sass, webpack and lockfile
Change-Id: I7007b2cdd57ab0120d8e039ae6571649a6e536ca
2021-07-30 07:47:55 +00:00
Egon Elbre
96291a3999 web/{satellite,storagenode}: bump lockfile version
Change-Id: Idc842efb3e49b633e317a0cfe3540f5a05cd81da
2021-07-30 07:47:19 +00:00
Michał Niewrzał
55d7bcc59b satellite/metabase/segmentloop: don't shutdown satellite on loop error
We made decision to avoid satellite shutdown when segment loop
will return error. Loop still can reeturn error but it will be logged
and we will make monitoring/alert around that error.

Change-Id: I6aa8e284406edf644a09d6b1fe00c3155c5430c9
2021-07-30 06:49:10 +00:00
Jeremy Wharton
615aae6bdd web/satellite: Remove activated account page
Error message for attempting to activate an already-activated
account is removed from its own page and incorporated into the
login page in an identical manner to the message that appears
upon successful activation.

Change-Id: I29cd2685a7808fa71d34a439c86a38eb5fc3e199
2021-07-29 14:47:57 +00:00
Michał Niewrzał
ae345320fe satellite/metabase: use db methods that pass context correctly
It turns out that some DB methods are not
passing context to to lower levels even if have
context as a parameter.

Change-Id: I82f4fc1a47c362687a91364d85e4468057e53134
2021-07-29 09:32:31 +00:00
Yingrong Zhao
d3e51353ab satellite/satellitedb/reputation: replace audit_histories table with
reputation

Change-Id: I18417eaa0d146cec876020dc6a358d13992e1d5f
2021-07-28 19:21:16 -04:00
Yingrong Zhao
b35b07081e satellite/nodestats: return default reputation when not found
Change-Id: I0237c13cb9ffe241bf257f8267b98a0e1481c3ef
2021-07-28 17:30:46 -04:00
Yingrong Zhao
c8f4ff4135 satellite/satellitedb: migrate existing data from overlaycache and
audit_histories to reputations table

Change-Id: Ia47e5264b37d2b7585c8486a4e022f4559233bf0
2021-07-28 15:16:52 -04:00
Yingrong Zhao
646ce5b8cc satellite/overlay: remove reputation logic from overlay
Change-Id: I3492860e4537c7a8e4e824ec4c9c8d179134a0c0
2021-07-28 15:15:28 -04:00
Yingrong Zhao
f8914ccce0 satellite/{repair, overlay}: use reputation store in repair
Change-Id: I48db9e68f48239d48621ccc77d33618ecb83ce1a
2021-07-28 13:22:05 -04:00
Yingrong Zhao
58238d850c satellite/{audit, accounting}: use reputation store in tests
Change-Id: I86a8ccf5dcee8d108196a9f67a476fe0ccbd8257
2021-07-28 13:21:55 -04:00
Yingrong Zhao
e91574cee1 satellite/{reputation, gracefulexit}: use reputation store in
gracefulexit

With the effort to move audit related data into reputation store, this
PR updates gracefulexit endpoint to use reputation service to get a
node's audit score

Change-Id: Iad93ea689ad67ff9c57c7be16687e21e715fab7a
2021-07-28 13:21:41 -04:00
Yingrong Zhao
55a77d04bc satellite/satellitedb,private: add initial value on testplanet startup
Currently, reputation table is only populated when a node has been
audited. This is ok in production, however a lot of our tests doesn't
upload any data or trigger audits.
This PR adds an initialization step in testplanet to populate reputation
table with zero value for nodes reputation.

Change-Id: I11b381236669db346dc68a48a6d4a27334a0a8b8
2021-07-28 13:20:32 -04:00
Yingrong Zhao
6c7bf357cd satellite/{reputation,audit,overlay}: replace overlay with reputation
package in audit

This PR implements reputation store and replace overlay in audit service
to use such store for storing node's audit stats.

In order to keep the changeset smaller, most of the changes in this PR is for copying audit logic in overlay to
reputation package. In a following PR, the duplicating code will be
removed from overlay.

Change-Id: I16c12494a0970f44c422b26cf603c1dc489e5bc1
2021-07-28 13:10:48 -04:00
Jeremy Wharton
6bbb5b5cc2 web/satellite: Static error page maintenance fixes
Unifies the stylesheet for 404 and 500 errors. References the
Storj logo rather than embedding it in the HTML. Replaces references
to Storj with Storj DCS. Removes reference to nonexistent font.

Change-Id: If224f30eb0b21171fc8f575de6c4cef98a48c6f5
2021-07-28 16:18:27 +00:00
Jeremy Wharton
49fbb5010f web/satellite: Merge and update 'Forgot Password' page files
Merges the 'Forgot Password' page files into one Vue file and updates
it to be consistent with deployed code. This change is part of an
ongoing effort to remove code from the branding repo.

Change-Id: Ib18b95dd5a3269f26f4f547fc46ac6278c468a54
2021-07-28 14:40:59 +00:00
Moby von Briesen
8f072bdeee cmd/satellite: Skip non-existing users in paid tier conversion
There are some users in our QA satellite which are no longer in Stripe,
and there are some users in Stripe which are not on our QA satellite.
This change allows us to test the paid tier conversion script in QA
despite these problems.

Change-Id: If94c9e882327841d1fd294d75fd302e6a7feee41
2021-07-27 12:53:58 -04:00
Clement Sam
d73b9fff9a satellite/orders: set the expirationDate in CreatePutRepairOrderLimits
In the past ExpirationDate was available inside CreatePutRepairOrderLimits
but this was removed since the metabase segment was missing the ExpiresAt field.
Now ExpiresAt field is available in the metabase segment
and can be set correctly while executing NewSignerRepairPut.

Change-Id: I068c07492ab27bde2c44477bbd32c5872edd024a
2021-07-27 12:44:40 +00:00
Michał Niewrzał
a2b30e42a3 satellite/metainfo: log internal errors
We are missing some logging around errors returned as
Internal.

Change-Id: Ifc41634816648172ff944808d23d1e6a344e41f1
2021-07-27 09:57:59 +00:00
Jeremy Wharton
3720afb81b web/satellite: Update emails to be consistent with deployed code
Update the forgot password, invite, and welcome emails to be
cosistent with deployed code. This change is part of an ongoing
effort to remove code from the branding repo.

Change-Id: I6edb6811904552c1cf7761b96bc9aa879c2eb154
2021-07-26 22:49:28 +00:00
Jeremy Wharton
276ebcd4b6 satellite: Remove duplicate logo in error pages
Remove duplicate instance of the Storj logo in the
40X and 50X error pages.

Change-Id: I74152560d39f34fb93f5d559b9b390c3e83a61ec
2021-07-26 20:30:16 +00:00
Moby von Briesen
149f6f2626 satellite/payments: Implement coupon codes
Full path: satellite/{payments,console},web/satellite

* Adds the ability to apply coupon codes from the billing page in the
satellite UI.
* Flag for coupon code UI is split into two flags - one for the billing
page and one for the signup page. This commit implements the first, but
not the second.
* Update the Stripe dependency to v72, which is necessary to
use Stripe's promo code functionality.

Change-Id: I19d9815c48205932bef68d87d5cb0b000498fa70
2021-07-26 17:15:55 +00:00
Jeremy Wharton
dae6ed7d03 satellite/console: Implement MFA backend
Added MFA passcode and recovery code field for token requests.
Added endpoints for MFA-related activity: enabling MFA,
disabling MFA, generating a new MFA secret key, and
generating new MFA recovery codes.

Change-Id: Ia1443f05d3a2fecaa7f170f56d73c7a4e9b69ad5
2021-07-26 16:37:05 +00:00
Michał Niewrzał
420d2f6275 metabase-orphaned-segments: avoid processing recently created elements
To be sure we are comparing the same set of objects
and segments lets ignore objects and segments created
after processing was started.
Segmets without objects cannot be created in normal
way so we will have them only if we broke something in
the past.

Change-Id: I96c07caf9e5091775d4dc8dfc0fef2b08b87957c
2021-07-26 15:50:20 +00:00
StobR
42b113d92b
web/storagenode: fix ingress chart tooltip
Fixes z-index issue for tooltips when ingress graph is mouseover.
2021-07-26 15:19:40 +03:00
StobR
267a962c3b
web/storagenode: fix egress chart tooltip 2021-07-26 14:51:40 +03:00
Michał Niewrzał
a883d7f582 satellite/repair/checker: fix remote_files_checked metric
While metaloop refactoring we missed metric for all
objects processed by repair checker.

Change-Id: I100f10a36c52e2651923ecaa377261752877d673
2021-07-22 14:48:08 +00:00
Michał Niewrzał
b12d29935a satellite/metabase: remove metaloop package
We moved everything to segment loop so we can now
remove metaloop from code.

Change-Id: I9bd8d2349e5638d7cdad50f2f313f9bd89a8165c
2021-07-22 13:00:45 +00:00
Jeremy Wharton
1535bbe673 satellite/console: Forward friendly registration errors to client
Provides the means to serve an error to the user with a user-friendly
error message (serveCustomJSONError). Auth API uses this when
processing registration attempts.

Previously, the error message was inferred by the client based on
the status code of the response received from the server. However,
if multiple distinct errors fit a certain status code, it was impossible
to correctly interpret the error.

Change-Id: I2f91e9c81ba1a4d14ba67e0b4b531a48800d4799
2021-07-22 11:31:12 +00:00
Michał Niewrzał
237360e765 metabase-orphaned-segments: fix args binding for delete command
As an additon I clarify log message a little bit.

Change-Id: Id2487cc218e3dd62ad5a05b56e8b1d5d070676a7
2021-07-22 11:00:37 +02:00
Fadila Khadar
6d60d412f0 satellite/gracefulexit: use segment loop
Join the segment loop instead of the metainfo loop, to iterate only over segments.

Change-Id: I06259d363b98d4e191f2bf2d82c9b47255ee484a
2021-07-21 15:12:25 +00:00
Fadila Khadar
c4202b9451 satellite/gracefulexit: use graceful_exit_segment_transfer_queue
For being able to use the segment metainfo loop, graceful exit transfers have to include the segment stream_id/position instead of the path. For this, we created a new table graceful_exit_segment_transfer_queue that will replace the graceful_exit_transfer_queue. The table has been created in a previous migration and made accessible through graceful exit db in another one.
This changes makes graceful exit enqueue transfer items for new exiting nodes in the new table.

Change-Id: I7bd00de13e749be521d63ef3b80c168df66b9433
2021-07-21 14:02:20 +00:00
Cameron Ayer
449c873681 satellite/repair/repairer: attempt repair GETs using nodes' last IP and port first
Sometimes we see timeouts from DNS lookups when trying to do
repair GETs. Solution: try using node's last IP and port first.
If we can't connect, retry with DNS lookup.

Change-Id: I59e223aebb436118779fb18378f6e09d072f12be
2021-07-21 13:13:06 +00:00
Fadila Khadar
b0d98b1c1a satellite/gracefulexit: allow use of graceful_exit_segment_transfer_queue
For being able to use the segment metainfo loop, graceful exit transfers have to include the segment stream_id/position instead of the path. For this, we created a new table graceful_exit_segment_transfer_queue that will replace the graceful_exit_transfer_queue. The table has been created in a previous migration.
This change gives access to this table.
Graceful Exit doesn't use the table yet, this will be done in a next change.

Change-Id: I6c09cff4cc45f0529813a8898ddb2d14aadb2cb8
2021-07-21 12:34:44 +00:00
Qweder93
73cdefbc41 munltinode/console: node online statuses extended with StatusUnauthorized and StatusStorageNodeInternal
Change-Id: I12c90169f8959e5eafe2a64bf8b412f7afc48c1c
2021-07-21 15:00:41 +03:00
Michał Niewrzał
f73fcee892 cmd: add tool for orphaned segments
This tool has two commands to execute. One is to 'report'
orphaned segments. Second to 'delete' orphaned
segments.

To find such segments tool is first finding all unique
segment stream ids. As a next step its removing from this
list stream ids of existing objects. What is left is a list of
orphaned segments.

Change-Id: I4a0ae3ad0b10a8d16572bfd22ac92cfa15ca19b3
2021-07-21 10:10:58 +00:00
Cameron Ayer
373ba8fd27 satellite/repair/repairer: metrics for repair bytes uploaded and downloaded
Change-Id: Icb0850692ecc155f6c8169edf1b045b2b546ff48
2021-07-21 09:23:19 +00:00
Cameron Ayer
adc0fbddfa satellite/audit: don't fail nodes for audit if not enough pieces downloaded
In most situations where we would not get enough shares to complete
an audit, something has probably gone wrong like a forgotten delete,
and nodes should not be failed. We have an alert when this occurs.
Check the logs to see what happened. If we decide the nodes should
get audit failures, we can do it manually.

Change-Id: Ib6e408082048d31197c37ebfd7f9031135fc938f
2021-07-20 20:28:18 +00:00
Michał Niewrzał
27a714e8b0 satellite/accounting/tally: use objects iterator instead metaloop
Bucket tally calculation will be removed from metaloop and will
use metabase objects iterator directly.

At the moment only bucket tally needs objects so it make no sense
to implement separate objects loop.

Change-Id: Iee60059fc8b9a1bf64d01cafe9659b69b0e27eb1
2021-07-20 15:52:18 +00:00
Vitalii Shpital
4c7384ca2f web/satellite: removed redundant files splitting
Since we removed login/registration explicit templates and styles from branding repo we don't need file splitting in main repo anymore.
So moved html and css code into components

Change-Id: Ib297e2576f16a7fddd17cc62f6c655adf96a59b6
2021-07-20 12:44:02 +03:00
Vitalii Shpital
2489b78d22 web/satellite: move user MFA secret generation to server-side
It was decided to move user MFA secret generation to server-side and be fetched by http request

Change-Id: I5e31d35fb78d1d9f3280518fe3eb543bb8ed4377
2021-07-19 13:48:09 +00:00
Egon Elbre
5a56021bca satellite/metabase: add intLimitRange
We have quite a bit code duplication in ensuring valid int ranges.

Change-Id: Ib31db8c4b2be6a677625cab5bcdddefb60401fe3
2021-07-19 07:51:16 +00:00
Moby von Briesen
bc2f81c2fa web/satellite: Add beta checkbox to registration page
Only if is-beta-satellite is set to true.
Also cleans up some styles.

Change-Id: I2273936ae2b41a3409fb985b13cc987224a3ba85
2021-07-16 15:55:11 +00:00
Moby von Briesen
7624bdd090 web/satellite: Return server-side error message during registration
It turns out, there are multiple different 400 errors that might be
returned from the server during registration. Rather than display
hardcoded text on the client ("Validation of reCAPTCHA was
unsuccessful"), this change simply displays the error sent by the
server.

As mentioned in the comment, we should eventually do this for all
errors, but that will be a more in-depth change. The purpose of this
commit is to unblock a point release.

Change-Id: Ideca107cc4039a0dabfa0fb02c943da920f7ff4f
2021-07-16 09:43:11 -04:00