Commit Graph

5735 Commits

Author SHA1 Message Date
Michał Niewrzał
260a673327 satellite/metabase: don't delete pieces until query results are received
To optimize memory consumption we where consuming
segment data during processing results from delete
query. Turns out that there is a chance that query will be
rolled-back if something will go wrong while reading
results. In such case its possible to delete pices but
object/segment will be still in DB.
This change removed piece deletion from problematic
place. Pieces are still deleted in batches but are not
limited at the moment. To avoid memory issues object
deletion batch was decreased.

Change-Id: Icb3667220f9c25f64b73cf71d0cf3fdc7e5107c5
2021-08-08 03:43:17 +00:00
Clement Sam
6613133a1c satellite/metabase: return metadata from IterateObjectsAllVersionsWithStatus on demand
Change-Id: I6bb717cf941e2b5fd7c411bd1e5a77fad0835995
2021-08-06 16:20:39 +00:00
Ivan Fraixedes
a60e836baa satellite/admin: Rename endpoints paths entities
Rename the entities in the endpoint paths from singular after we have
agreed that it matches with API design best practices.

Change-Id: I5bdc086c64c6f93e45a13436b65863fc3e8d864e
2021-08-06 10:05:25 +00:00
Ivan Fraixedes
8d75a35e56 satellite/admin: Add new endpoint get list of API Keys
Add a new endpoint to get the list of API keys associated with a
project.

Change-Id: I9b5ed42e226786c03d853bbc8538344b40ee634f
2021-08-06 09:44:06 +00:00
Clement Sam
1f353f3231 segment/{metabase,repair}: change segment created_at column to not accept nulls
This change adds a NOT NULL constraint to the created_at column in the segment table.
All occurrences of CreatedAt as a pointer are changed to non pointer version (metabase, segment loop, etc)

Change-Id: I3efd476ebd1edd3327b69c9223d9edc800e1cc52
2021-08-06 08:16:28 +00:00
Yingrong Zhao
e00b573f8f satellite/overlay: fix UpdateCheckIn comment
Change-Id: I8584895249d7e5be6dbec79974fc9f77b7a1930c
2021-08-06 05:54:00 +00:00
Moby von Briesen
7f02e1f469 satellite/console: Remove config flag for MFA
MFA is complete and we are good to enable it in production. This change
removes the flag that disables MFA by default.

Change-Id: I2f985ae501171bdab505d664b43c8cfc248bad8d
2021-08-05 18:04:36 +00:00
Yingrong Zhao
e4cc965c39 satellite/satellitedb: replace explicit transaction with dbx query for
UpdateReputation

Change-Id: I7c139ededea83d4b58107536c3a031c4f92d6eb4
2021-08-05 17:09:49 +00:00
Egon Elbre
19852a767b web/storagenode: fix lint issues
After migrating to eslint some errors were disabled to make it easier to
migrate.

This enables all the lint rules and treats all warnings as a build
failure. Similarly, CI won't automatically try to fix mistakes.

Change-Id: Iff9c4a59401900fc395cd566dd328f3a9c688a12
2021-08-05 15:45:55 +00:00
Michał Niewrzał
f87ad240ed satellite/metainfo: add DeletePart method
We need a way to delete whole part. This especially
needed for uplink multipart API to do cleanup after
aborted or failed part upload.

Test will be added when uplink part will be merged.

Change-Id: I9ba69a49e1adcdce0f42dd3a76f938fcf931155a
2021-08-05 14:43:30 +00:00
Egon Elbre
65804801ec all: fix mon.Task leak
Change-Id: Ifd58c7ac5631b9c3c750b3f4cc50525167e90709
2021-08-05 14:07:45 +03:00
Yingrong Zhao
ae02f6deda satellite/reputation: return default reputation stats when node is not
found

Change-Id: I587d0ab36ffa0efaf345a6a6e221ae5d2068e1c5
2021-08-04 19:34:54 +00:00
NickolaiYurchenko
cc590343c2 web/satellite: server side encryption acknowledge checkbox added
Change-Id: If57ab6e8fddc34c142d19f754caf1338dfdb0bf6
2021-08-04 16:16:26 +00:00
Egon Elbre
a3e78491b9 web/multinode: fix lint issues
After migrating to eslint some errors were disabled to make it easier to
migrate.

This enables all the lint rules and treats all warnings as a build
failure. Similarly, CI won't automatically try to fix mistakes.

Change-Id: I6d5ef39d476a38dd9a6ab98e3fb94da4c3ab2e57
2021-08-04 17:00:00 +03:00
Egon Elbre
f5ac00f909 web/satellite: switch to eslint, sass, bump deps
tslint has been deprecated so it's nice to switch to eslint.
Currently this uses a minimal eslint, to get things up and running.

node-sass requires C which does not work nicely on all platforms.

Change-Id: I3ca9adf2971475c009e541652b7637c18ad960f4
2021-08-03 15:56:33 +00:00
Clement Sam
f06e7c5f60 segment/{metabase,repair}: add dedicated methods on metabase.Pieces
This change adds dedicated methods on metabase.Pieces to be able to add, remove pieces and also to check duplicates.

Change-Id: I21aaeff40c017c2ebe1cc85a864ae546754769cc
2021-08-03 15:12:03 +00:00
Egon Elbre
80854ea921 web/storagenode: switch to eslint, sass, bump deps
tslint has been deprecated so it's nice to switch to eslint.
Currently this uses a minimal eslint, to get things up and running.

node-sass requires C which does not work nicely on all platforms.

Change-Id: Icf12e375293e052befc2d0daeb23d73697ed6ded
2021-08-03 17:03:38 +03:00
Qweder93
5bb3836312 satellite/metainfo: objectEntryToProtoListItem fills metadata on demand
Added includeMetadata parameter which represents if metadata should be included in response
by default true, in case of new uplink version - ObjectIncludes will be used instead.

Change-Id: I2f8d3b4cc354cd655f8093bbbebe0e3c2ae14e6f
2021-08-03 13:41:11 +00:00
Michał Niewrzał
0d8e7905c1 satellite/repair/checker: don't return error when joining loop
Error from joining loop should not restart satellite. This will be the
same error like for loop itself. In the same way we are handling joining
error for other services that are using segment loop.

Change-Id: Idf1035ef7f78462927bd23989ed8a4ee5826c49e
2021-08-03 12:56:42 +00:00
Egon Elbre
5e24060b25 web/multinode: switch to eslint
tslint has been deprecated so it's nice to switch to eslint.
Currently this uses a minimal eslint, to get things up and running.

Change-Id: If0ca6413b534b0aa15f6130a0a809e002d817356
2021-08-03 13:00:15 +03:00
Egon Elbre
df52e72fe5 web/multinode: fix package versions
Change-Id: I2848f2dbb6a9eb943e2d5d813fefd98b59d312f4
2021-08-02 19:07:50 +03:00
igor gaidaienko
92deef4f34 Revert "storagenode/payouts: historical payouts use satellitesDB instead of trustPool"
This reverts commit 4a98dd40e2.
2021-08-02 13:55:21 +03:00
Vitalii Shpital
42a0b92404 web/satellite: set default NotAfter caveat for object browser access grant
According to FIRE-53 ticket we set default NotAfter caveat for object browser access grant

Change-Id: Ib421605f524cc176d9ffbf94fc95c09bd06fd477
2021-08-02 09:45:50 +00:00
Vitalii Shpital
cc277806d0 web/satellite: regenerate recovery codes popup
Added ability to regenerate MFA recovery codes to account settings.
Appropriate button and popup were added.

Change-Id: Ie7d42e7e13e868f3b0b8b7d5db7eacc9e8206aaf
2021-08-02 09:25:00 +00:00
Michał Niewrzał
be92ff1751 satellite/metabase: fix finishing bucket objects deletion
In case when number of deleted segments in a single
batch was multiplication of opts.DeletePiecesBatchSize it
was possible to finish deletion of objects to early.

Change-Id: I9181d4a3c64053d9df46a11a6e0cd22153c73ee9
2021-08-02 09:00:02 +00:00
Michał Niewrzał
804d221d42 satellite/satellitedb: remove tables unused after metaloop refactoring
Change-Id: I6e6c2ec32e43eb0c24c4ea0522443487634a5164
2021-08-02 08:01:26 +00:00
Michał Niewrzał
011b944382 satellite/metrics: fix metrics for total inline/remote bytes and segments
Change-Id: I567ce127590a4712cab296d28a19838e3a632021
2021-07-30 18:19:51 +02:00
paul cannon
fda316b461 Makefile: update channel name of #team-data
Avoids confusion when metrics need to be changed or updated.

Change-Id: I1d0bf939bd30cd66128eb04ccee5c4133447bfe9
2021-07-30 09:58:57 -05:00
Artur M. Wolff
10d7a63ab0 Makefile: include arm64 (instead of arm) executables in arm64v8 images
Change-Id: Ic9cd5cd7b052efcb75499c338dd60480f960c4fa
2021-07-30 14:01:48 +02:00
Vitalii Shpital
c6c2a1daf9 web/satellite: added MFA login step
Added Enter MFA passcode step to login after first attempt.
Connected client-side and server-side.

Change-Id: I4482afde50172bbf2c598aa01bbd220763b4f6d1
2021-07-30 08:50:19 +00:00
Egon Elbre
2118c89e9a web/multinode: bump node-sass, webpack and lockfile
Change-Id: I7007b2cdd57ab0120d8e039ae6571649a6e536ca
2021-07-30 07:47:55 +00:00
Egon Elbre
96291a3999 web/{satellite,storagenode}: bump lockfile version
Change-Id: Idc842efb3e49b633e317a0cfe3540f5a05cd81da
2021-07-30 07:47:19 +00:00
Michał Niewrzał
55d7bcc59b satellite/metabase/segmentloop: don't shutdown satellite on loop error
We made decision to avoid satellite shutdown when segment loop
will return error. Loop still can reeturn error but it will be logged
and we will make monitoring/alert around that error.

Change-Id: I6aa8e284406edf644a09d6b1fe00c3155c5430c9
2021-07-30 06:49:10 +00:00
Jeremy Wharton
615aae6bdd web/satellite: Remove activated account page
Error message for attempting to activate an already-activated
account is removed from its own page and incorporated into the
login page in an identical manner to the message that appears
upon successful activation.

Change-Id: I29cd2685a7808fa71d34a439c86a38eb5fc3e199
2021-07-29 14:47:57 +00:00
Michał Niewrzał
ae345320fe satellite/metabase: use db methods that pass context correctly
It turns out that some DB methods are not
passing context to to lower levels even if have
context as a parameter.

Change-Id: I82f4fc1a47c362687a91364d85e4468057e53134
2021-07-29 09:32:31 +00:00
Yingrong Zhao
d3e51353ab satellite/satellitedb/reputation: replace audit_histories table with
reputation

Change-Id: I18417eaa0d146cec876020dc6a358d13992e1d5f
2021-07-28 19:21:16 -04:00
Yingrong Zhao
b35b07081e satellite/nodestats: return default reputation when not found
Change-Id: I0237c13cb9ffe241bf257f8267b98a0e1481c3ef
2021-07-28 17:30:46 -04:00
Yingrong Zhao
c8f4ff4135 satellite/satellitedb: migrate existing data from overlaycache and
audit_histories to reputations table

Change-Id: Ia47e5264b37d2b7585c8486a4e022f4559233bf0
2021-07-28 15:16:52 -04:00
Yingrong Zhao
646ce5b8cc satellite/overlay: remove reputation logic from overlay
Change-Id: I3492860e4537c7a8e4e824ec4c9c8d179134a0c0
2021-07-28 15:15:28 -04:00
Yingrong Zhao
f8914ccce0 satellite/{repair, overlay}: use reputation store in repair
Change-Id: I48db9e68f48239d48621ccc77d33618ecb83ce1a
2021-07-28 13:22:05 -04:00
Yingrong Zhao
58238d850c satellite/{audit, accounting}: use reputation store in tests
Change-Id: I86a8ccf5dcee8d108196a9f67a476fe0ccbd8257
2021-07-28 13:21:55 -04:00
Yingrong Zhao
e91574cee1 satellite/{reputation, gracefulexit}: use reputation store in
gracefulexit

With the effort to move audit related data into reputation store, this
PR updates gracefulexit endpoint to use reputation service to get a
node's audit score

Change-Id: Iad93ea689ad67ff9c57c7be16687e21e715fab7a
2021-07-28 13:21:41 -04:00
Yingrong Zhao
55a77d04bc satellite/satellitedb,private: add initial value on testplanet startup
Currently, reputation table is only populated when a node has been
audited. This is ok in production, however a lot of our tests doesn't
upload any data or trigger audits.
This PR adds an initialization step in testplanet to populate reputation
table with zero value for nodes reputation.

Change-Id: I11b381236669db346dc68a48a6d4a27334a0a8b8
2021-07-28 13:20:32 -04:00
Yingrong Zhao
6c7bf357cd satellite/{reputation,audit,overlay}: replace overlay with reputation
package in audit

This PR implements reputation store and replace overlay in audit service
to use such store for storing node's audit stats.

In order to keep the changeset smaller, most of the changes in this PR is for copying audit logic in overlay to
reputation package. In a following PR, the duplicating code will be
removed from overlay.

Change-Id: I16c12494a0970f44c422b26cf603c1dc489e5bc1
2021-07-28 13:10:48 -04:00
Jeremy Wharton
6bbb5b5cc2 web/satellite: Static error page maintenance fixes
Unifies the stylesheet for 404 and 500 errors. References the
Storj logo rather than embedding it in the HTML. Replaces references
to Storj with Storj DCS. Removes reference to nonexistent font.

Change-Id: If224f30eb0b21171fc8f575de6c4cef98a48c6f5
2021-07-28 16:18:27 +00:00
Jeremy Wharton
49fbb5010f web/satellite: Merge and update 'Forgot Password' page files
Merges the 'Forgot Password' page files into one Vue file and updates
it to be consistent with deployed code. This change is part of an
ongoing effort to remove code from the branding repo.

Change-Id: Ib18b95dd5a3269f26f4f547fc46ac6278c468a54
2021-07-28 14:40:59 +00:00
Moby von Briesen
8f072bdeee cmd/satellite: Skip non-existing users in paid tier conversion
There are some users in our QA satellite which are no longer in Stripe,
and there are some users in Stripe which are not on our QA satellite.
This change allows us to test the paid tier conversion script in QA
despite these problems.

Change-Id: If94c9e882327841d1fd294d75fd302e6a7feee41
2021-07-27 12:53:58 -04:00
Clement Sam
d73b9fff9a satellite/orders: set the expirationDate in CreatePutRepairOrderLimits
In the past ExpirationDate was available inside CreatePutRepairOrderLimits
but this was removed since the metabase segment was missing the ExpiresAt field.
Now ExpiresAt field is available in the metabase segment
and can be set correctly while executing NewSignerRepairPut.

Change-Id: I068c07492ab27bde2c44477bbd32c5872edd024a
2021-07-27 12:44:40 +00:00
Michał Niewrzał
a2b30e42a3 satellite/metainfo: log internal errors
We are missing some logging around errors returned as
Internal.

Change-Id: Ifc41634816648172ff944808d23d1e6a344e41f1
2021-07-27 09:57:59 +00:00
Jeremy Wharton
3720afb81b web/satellite: Update emails to be consistent with deployed code
Update the forgot password, invite, and welcome emails to be
cosistent with deployed code. This change is part of an ongoing
effort to remove code from the branding repo.

Change-Id: I6edb6811904552c1cf7761b96bc9aa879c2eb154
2021-07-26 22:49:28 +00:00