Commit Graph

3645 Commits

Author SHA1 Message Date
Moby von Briesen
cdcd3b2687 satellite/console,web/satellite: add feature flag for limit increase req
This change adds a feature flag (default disabled) to enable the ability
to submit a project limit increase request directly from the UI. When
this feature is disabled, the user will be directed to a page to file a
support ticket.

Resolves https://github.com/storj/storj/issues/6480

Change-Id: I5f355dcb1a40e5b694f9623f05fe706ed4d6a528
2023-11-10 08:07:13 +00:00
Wilfred Asomani
98234e76b7 satellite/{payment,console} add legal hold user status
This change introduces a new user status, LegalHold to be used to pause
an account while it is under legal review. It also modifies the
condition whether to generate invoices for a user to apply to this new
status.

Issue: storj/storj-private#492

Change-Id: Idc9a6166cc96178a95e45cf7c0f2e982631ca021
2023-11-09 23:42:26 +00:00
paul cannon
c246d36adf satellite/metabase: tests for versioning in TestIterateObjectsWithStatus
Credit to the tests in TestListObjectsVersioned(), which (since
ListObjects is logically a pretty thin wrapper around
iterateAllVersionsWithStatus()) were able to be translated to tests in
TestIterateObjectsWithStatus with only minor changes.

Change-Id: I6e68675b24fee614e44baddf596703115554014e
2023-11-09 17:25:44 +00:00
Michal Niewrzal
c9430c22b5 satellite/metainfo: wire versioning with commit object operation
Change-Id: Ib19b5bade69dfb3f02256c638cc2f6281befa415
2023-11-09 11:21:58 +00:00
dlamarmorgan
539b32d01e satellite/metainfo: add UseBucketLevelObjectVersioning by project
This small feature flag will give us the ability to test object
versioning for specific projects without enabling it globally.

Change-Id: I78301f071b7b8079dd1bd4a561fce0800ce9f074
2023-11-09 11:21:06 +00:00
paul cannon
dc5ae6f4f6 satellite/gracefulexit: add test for GE through upgrade
This creates an automated test for the situation where a node initiates
graceful exit while TimeBased is off, and then TimeBased is turned on
before the node has completed graceful exit. The node should no longer
try to transfer any more pieces, but should instead sit and wait until
the graceful exit period has elapsed.

Change-Id: Iaf636f9247bc878bc20041221e1a8014c77806ad
2023-11-09 08:58:16 +00:00
Michal Niewrzal
3e1b108b84 satellite/metainfo: return correct version with ListObjects
ListObjects method will now return correct object version.

Change-Id: I663c506079b21f238026bd6ffac2384eae521635
2023-11-09 07:41:22 +00:00
Moby von Briesen
b56a158100 satellite/analytics: Send signup captcha score to analytics
Send signup captcha score (if it exists) to Hubspot and Segment, so that
we can implement logic based on this in the analytics platforms.

Change-Id: Ic4e166639c9ab8e872a212e7466e82433c9ea6d4
2023-11-08 21:56:11 +00:00
JT Olio
98032f2b1f satellite/payments/stripe: another loop variable capture
Change-Id: I9ec7226befb8ebe18c7833b1c1d7ee8661bd9032
2023-11-08 20:43:54 +00:00
Márton Elek
0ef3247d44 satellite/durability: make benchmark even quicker
To make sure that Benchmark tests are good, we run them with -short flag, eg:

```
go test -short -run=BenchmarkDurabilityProcess
```

Durability benchmark already supports this, but we can make it slightly more faster with
using less sgements and pieces during the `-short` run.

Change-Id: I9547ca1e3cd0178eb395a7a388f2e7936a9862d7
2023-11-08 19:00:30 +00:00
Ivan Fraixedes
100519321e
satellite/admin: Allow all operations through Oauth
Allow all the operations when accessing through Oauth, but requires the
authorization token for the ones that we consider that they are
sensitive.

Before these changes, a group of operations weren't available through
Oauth, and people who has access to the authorization token had to
forward the port of the server to their local in order to do them
without Oauth.

These changes shouldn't reduce the security because people who has
access to the authorization token is the same than they can forward the
port and part of those have Oauth access too.

Allowing to perform all the operations through Oauth will improve the
productivity of production owners because they will be able to do all
the administration requests without having to port forward the server.

Change-Id: I6d678abac9f48b9ba5a3c0679ca6b6650df323bb
2023-11-08 18:14:38 +01:00
Márton Elek
23c592adeb satellite/durability: use process level classID cache (instead fork level)
Classifier of durability is sg. like "net:1.3.4.1" or "country:HU".

To make the calculation faster we use arrays instead of maps, which means that we assign a uinique index to all of these strings (classes).

As Egon suggested earlier, we can do this mapping only once (per process), not for each fork.

Not a big deal performance-wise, as we have limited number of forks, which are initialized once per 5-10 hours, but the code is more readable and clean.

Change-Id: Id081846b5d97dae8009aeeecbcc63cb713bed294
2023-11-08 15:22:48 +00:00
paul cannon
a712ee94aa satellite/gracefulexit: suspended nodes fail graceful exit
QA discovered that if your node lost all its data but your audit score
was still healthy, you could call graceful exit, respond with unknown
audit errors when audited, and eventually get suspended, but still get
your held amount back.

We will therefore mark suspended nodes as having failed graceful exit.

Refs: https://github.com/storj/storj-private/issues/485
Change-Id: Id5af18786b574651587cc96bd6a7d0b47c0671a8
2023-11-08 11:26:27 +00:00
paul cannon
084719cde1 satellite/gracefulexit: de-flake TestNodeFailingGracefulExitWithLowOnlineScore
I can't say with certainty yet what caused the two failures I know
about, but I have one theory: the node continuing to check in during the
test skewed the online score towards 1, and using the test default for
GracefulExitDurationInDays meant there were fewer update periods than
expected.

At any rate, it is more correct to pause the graceful exit processing
chore and the contact chore during the test, even if it doesn't end up
solving the problem.

Refs: https://github.com/storj/storj/issues/6401
Change-Id: I06d43d531e0b3344af13878c8d55213349fdcfa3
2023-11-08 10:16:05 +00:00
JT Olio
00cb237a90 satellite/payments/stripe: loop variable capture
Change-Id: Ia2e0cf329f2ff46394b21bad9581e430e0202ead
2023-11-07 19:29:51 -05:00
Ivan Fraixedes
ae945b993a
satellite/admin: Create separate server for new back-office
Create a separate server for implementing the new satellite
administration web app.

This server is in a new package that will implement all the
functionality for the new satellite administration back-end and when it
be completed with all the functionality that the current one offer, it
will replace it.

For now, the new server only exposes the static assets as they were
exposed by the current server.

A main sub-package is added with an example endpoint to scaffold where
we'll define the API through the API generator and to locate the several
generated files.

Change-Id: I172c43b2c180553876ef7ce137cc778b94723451
2023-11-07 15:31:01 +01:00
Márton Elek
015cb94909
satellite/durability: add exemplar and report time to the reported results
Exemplars are representative elements for the stat. For example if a stat min is `30`, we can save one example with that value.

More details about the concept is here: https://grafana.com/docs/grafana/latest/fundamentals/exemplars/

In our context, which save the segment + position in case of min is updated, to make it easier to look after the segment in danger.

Change-Id: I19be482f1ddc7f1711e722c7b17480366d2c8312
2023-11-06 13:22:28 +01:00
dlamarmorgan
d0f4447427
satellite/metabase: add version tests to IteratePendingObjectsByKey
Change-Id: I7335598eab2bac7f12aa93773cceadb195cd8e9e
2023-11-03 12:47:05 -07:00
Vitalii
f6e357be52 satellite/{console, web}: detailed usage report for a single project
Reworked usage report endpoint to return CSV for a single OR all the project user owns.
Added buttons to download usage report CSV for a single project.

Issue:
https://github.com/storj/storj/issues/6154

Change-Id: I55104088180dcf6be49dcde6c9c495f07ba01c5a
2023-11-03 14:58:18 +02:00
Michal Niewrzal
f0f73fc8ae satellite/metainfo: support downloading specific object version
Protobuf definition is ready to support downloading specific version of
object so we just need to wire requested version into metainfo
DownloadObject endpoint.

https://github.com/storj/storj/issues/6221

Change-Id: I3ddc173beb6a6cf30d782dd65c6aa5f88f2cbd44
2023-11-03 11:58:12 +00:00
paul cannon
51c930f532 satellite/repair: extra logging during TestSegmentRepairPlacement
In case we continue to see flaky TestSegmentRepairPlacement, this may
help narrow down the issue.

Change-Id: I34ca70e5bb33eca26e9940e845142121cc946ac0
2023-11-01 20:43:18 +00:00
paul cannon
b621a90745 satellite/repair: promote some Debug log lines to Info
We have frequently wanted to see this sort of information when looking
into issues. Considering that we already log a line at Info every time
we fail to download a piece during repair, it doesn't seem like a very
onerous new burden to log one line per segment saying that it is being
repaired.

Change-Id: I1fa84a985e90473adeb02603e207fc3c7b8592da
2023-11-01 19:52:51 +00:00
Egon Elbre
9e0fff5e22 satellite/gcbf: fix data race in TestGCBFUseRangedLoop
Satellites[0].GCBF is already started when testplanet boots up,
so calling Run separately ends up causing a data race.

Instead create a new instance, that should avoid this issue.

Fixes https://github.com/storj/storj/issues/6435

Change-Id: I6603ef63da7a6ab8bdb952cf5aaca17eb0392e2c
2023-11-01 19:07:20 +02:00
Wilfred Asomani
b3e908f72f satellite/{console,web}: add browser card view config flag
This change adds config flag to toggle whether the browser card view
can be used.

Issue: #6427

Change-Id: I19986ffe9a778b8752549a4d54b318fff0386479
2023-10-31 20:19:57 +00:00
Egon Elbre
f2eb6942c5 satellite/metabase: make the precommit constraint code nicer
This condenses the precommit constraint into a single function, which
allows to cleanup a bunch of logic elsewhere. Also, avoid delete in the
first place when we are not allowed to remove the uncommited object.

This also fixes submitting the metrics and now it submits the correct
metrics.

Change-Id: If91dfa3b19bce5b24ff2a19d7c34b57a200db1dd
2023-10-31 18:10:09 +00:00
Egon Elbre
e5e55ef266 satellite/metabase: move delete object unversioned code
Change-Id: Iec92c91c174b990fa44b210db415da7a4eb650c4
2023-10-31 17:14:16 +00:00
Egon Elbre
46f7e5b7fa all: don't depend on infectious library directly
Change-Id: I0bcc719cc25e1d7ad36a16f65d40220eccba6556
2023-10-31 16:00:47 +02:00
dlamarmorgan
77cac8b465 satellite/metabase: add tests ListObjects
Change-Id: Ia84b560251245fe0050426537751958f2c85e9e1
2023-10-30 20:33:08 +00:00
Wilfred Asomani
513c3cc632 satellite/admin: list users pending deletion
This change adds an endpoint to the admin API and UI to get a list of
users pending deletion and have no unpaid invoice.

Issue: #6410

Change-Id: I906dbf9eee9e7469e45f0c622a891867bf0cc201
2023-10-30 19:11:16 +00:00
Jeremy Wharton
42e1b088c2 satellite/admin/back-office/ui: use composition API
This change migrates Vue components of the new satellite admin web app
from the options API to the composition API.

Change-Id: Ie8c9bcd468f1c0fe0abc9ef0a3724563db096ba9
2023-10-30 16:50:30 +00:00
Michal Niewrzal
67f32bd519 satellite/metainfo: use protbuf ObjectVersion with GetObject request
We need to update metainfo GetObject endpoint to use ObjectVersion
([]byte) field instead of old Version field (int32).

Change-Id: I61663ec8d9f5c731f91346a285048477fb493730
2023-10-30 14:49:18 +00:00
Wilfred Asomani
f7a95e0077 satellite/{payment,console}: add endpoint to add card by pmID
This change introduces a new endpoint that allows adding credit cards
by payment method ID (pmID). The payment method would've already been
created by the frontend using the stripe payment element for example.

Issue: #6436

Change-Id: If9a3f4c98171e36623607968d1a12f29fa7627e9
2023-10-30 13:58:55 +00:00
Egon Elbre
1fa918c255 satellite/metabase: simplify UpdateObjectLastCommitted
Change-Id: Ifec596ab1868baa03688e717adec7d2ab45eafa9
2023-10-28 04:43:43 +00:00
Egon Elbre
55bddb6ce1 satellite/metabase: don't use empty stream id for delete markers
Using a an empty stream id makes it more difficult to target a specific
delete marker. Similarly, we don't want to confuse actual stream id-s
with normal ones. So, we'll create stream id-s where the first few bytes
are 0xFF, but the rest is random.

Change-Id: Ia7fffb0da9a071be2935df99c0846027ee2e03c3
2023-10-28 04:43:32 +00:00
dlamarmorgan
aeec4bd213 satellite/metabase: add tests to UpdateMetadata for versioning
Change-Id: I4503f92b19212e7c4cf1907edd7e529ff75cca3e
2023-10-28 04:43:04 +00:00
Ethan Adams
e65345336f satellite/placement: Add test for new "datacenter" placement tag
Change-Id: Ic9a79bb46256fbe8b1159de2bbf787b4a63c5c8f
2023-10-27 18:15:32 +00:00
Jeremy Wharton
8ebf285081 satellite/admin/back-office/ui: lint files
This change lints files for the new satellite admin web app. An
ESLint config has been added for it that is identical to the one used
for linting satellite web app files.

Change-Id: I66f72fb880d5cbc80b6c080294e4a830b3d28143
2023-10-27 11:23:58 +00:00
Egon Elbre
27724835da satellite/metabase: use database order in ObjectStream.Less
Change-Id: I3d5e00263d78d9128fae6d39a3bc7f0c542039b6
2023-10-27 07:38:51 +00:00
Egon Elbre
a4edbdd005 satellite/metabase: use commit order for objects
This changes metabase behavior such that the latest object
will be the committed object which may result in a new version
assigned to the objects.

Change-Id: I7a731d7db4696caba75fff65107a248569b6111f
2023-10-27 07:38:25 +00:00
Jeremy Wharton
eff1719977 satellite/admin/back-office/ui: speed up build command
This change speeds up the build of the new satellite admin web app
using the same method that 8b0d25c used to speed up the build of the
new satellite web app.

New build time:   23.394s
Old build time: 1m40.930s

Change-Id: Ic6fcfdfc73b78bc26bddc421c608076b23532967
2023-10-26 23:47:18 -05:00
Egon Elbre
c8e4f0099c satellite/metabase: add Prefix status for non-recursive listing
Currently it's awkward to use any of the existing statuses for the
objects in non-recursive listing. Hence, let's add a new one.

Change-Id: I8485e0f858e69998b097e757091991538ca697fa
2023-10-26 15:37:24 +00:00
Michal Niewrzal
20dfcedd97 satellite/metainfo: return full object on commit
We never extended metainfo protocol to return committed object
detailed into and this change is doing it now. Main motivation to
do this now is need for providing object version after upload.

Change-Id: Ib59bdfd9485e4a0091ac02458cc63427cb7159de
2023-10-26 09:46:18 +00:00
Michal Niewrzal
f5d717735b satellite/repair: fix checker and repairer logging
This is fixing two small issues with logging:
* repair checker was logging empty last net values as clumped pieces
but in main logic we stopped classifying it this way
* repairer summary log was showing incorrect number of pieces removed
from segment because list contains duplicated entries

There was no real issue here.

Change-Id: Ifc6a83637d621d628598200cad00cce44fa4cbf9
2023-10-25 22:55:53 +00:00
Vitalii
4ba2703783 satellite/{console, web}: remove AllProjectDashboard feature flag
Removed AllProjectDashboard feature flag.
Removed unused Vue components.
Fixed wrong redirect on reload if pricing packages are disabled.
Fixed wrong redirect on reload if billing features are enabled.

Issue:
https://github.com/storj/storj/issues/6434

Change-Id: I9081a6f737c45fb48da5b23c016a42e23021c4ce
2023-10-25 20:49:47 +00:00
Vitalii
40e43826a9 web/satellite: extend low token balance banner use-case
Updated condition on when this banner should be shown.
Also, added this banner to project dashboard and billing pages.

Issue:
https://github.com/storj/storj/issues/6356
https://github.com/storj/storj/issues/6368

Change-Id: I2f8f587a3c75508df0a9a6e84e1684b3c3904aa7
2023-10-25 19:14:14 +00:00
Michal Niewrzal
4cbdc0342a cmd/tools/segment-verify: add CreatedBefore and CreatedAfter flags
We would like to verify only segments from specific periods of time.

https://github.com/storj/storj/issues/6431

Change-Id: I42610962022bdf6ee36815fe1c157d67792147b8
2023-10-25 18:06:46 +00:00
Márton Elek
bca46736bb satellite/contact: send evenkit entry only with existing fields
Change-Id: Ifdd6c14de5f99c3c9bb8b1e9e2dce8c1e1c3b1ed
2023-10-25 16:16:20 +00:00
Michal Niewrzal
6c2e66fa9e satellite/metabase: small addition to GetObject* tests
Change-Id: I057a432750bbe12b30127c91592747a86d7ca23c
2023-10-25 17:48:30 +03:00
Egon Elbre
504d5c5651 satellite/metabase: add version tests for GetLatestObjectLastSegment
Change-Id: Ia7ed1f6b23bcdc9e83fec288cbf3571b382d5e13
2023-10-25 17:39:57 +03:00
Egon Elbre
a7e1378f89 satellite/metabase: return more information from delete last committed
Change-Id: I2626a100e0c3c41631c9a29b0bf5a7afccc60957
2023-10-25 17:38:57 +03:00