Commit Graph

8769 Commits

Author SHA1 Message Date
Márton Elek
23c592adeb satellite/durability: use process level classID cache (instead fork level)
Classifier of durability is sg. like "net:1.3.4.1" or "country:HU".

To make the calculation faster we use arrays instead of maps, which means that we assign a uinique index to all of these strings (classes).

As Egon suggested earlier, we can do this mapping only once (per process), not for each fork.

Not a big deal performance-wise, as we have limited number of forks, which are initialized once per 5-10 hours, but the code is more readable and clean.

Change-Id: Id081846b5d97dae8009aeeecbcc63cb713bed294
2023-11-08 15:22:48 +00:00
Vitalii
b6e4f4a02d web/satellite: update locked objects wording
Updated locked objects wording to reflect situations when objects were recently deleted.

Change-Id: I9c4e3ca290cce5e211745094b866d568078d4317
2023-11-08 13:09:42 +00:00
paul cannon
a712ee94aa satellite/gracefulexit: suspended nodes fail graceful exit
QA discovered that if your node lost all its data but your audit score
was still healthy, you could call graceful exit, respond with unknown
audit errors when audited, and eventually get suspended, but still get
your held amount back.

We will therefore mark suspended nodes as having failed graceful exit.

Refs: https://github.com/storj/storj-private/issues/485
Change-Id: Id5af18786b574651587cc96bd6a7d0b47c0671a8
2023-11-08 11:26:27 +00:00
paul cannon
084719cde1 satellite/gracefulexit: de-flake TestNodeFailingGracefulExitWithLowOnlineScore
I can't say with certainty yet what caused the two failures I know
about, but I have one theory: the node continuing to check in during the
test skewed the online score towards 1, and using the test default for
GracefulExitDurationInDays meant there were fewer update periods than
expected.

At any rate, it is more correct to pause the graceful exit processing
chore and the contact chore during the test, even if it doesn't end up
solving the problem.

Refs: https://github.com/storj/storj/issues/6401
Change-Id: I06d43d531e0b3344af13878c8d55213349fdcfa3
2023-11-08 10:16:05 +00:00
JT Olio
00cb237a90 satellite/payments/stripe: loop variable capture
Change-Id: Ia2e0cf329f2ff46394b21bad9581e430e0202ead
2023-11-07 19:29:51 -05:00
Ivan Fraixedes
ae945b993a
satellite/admin: Create separate server for new back-office
Create a separate server for implementing the new satellite
administration web app.

This server is in a new package that will implement all the
functionality for the new satellite administration back-end and when it
be completed with all the functionality that the current one offer, it
will replace it.

For now, the new server only exposes the static assets as they were
exposed by the current server.

A main sub-package is added with an example endpoint to scaffold where
we'll define the API through the API generator and to locate the several
generated files.

Change-Id: I172c43b2c180553876ef7ce137cc778b94723451
2023-11-07 15:31:01 +01:00
Wilfred Asomani
14b83bb390 web/satellite/vuetify-poc: add login functionality
This change implements login functionality in the vuetify app.

Issue: #6470

Change-Id: I6079888af14ded6d4886b3fc16108ca410f52982
2023-11-07 10:08:56 +00:00
paul cannon
fd55dad735 storagenode/retain: don't quit on error
It has been noted in the forum that, during a Retain operation, when a
piece can't be deleted, the process never completes. The error is
written to the log, but the completion line "Moved pieces to trash
during retain" never is.

This `return` line is the reason. We should instead continue the loop.

Change-Id: I0f51d34aba0e81ad60a75802069b42dc135ad907
Refs: https://github.com/storj/storj/issues/6482
2023-11-06 10:54:46 -06:00
Márton Elek
015cb94909
satellite/durability: add exemplar and report time to the reported results
Exemplars are representative elements for the stat. For example if a stat min is `30`, we can save one example with that value.

More details about the concept is here: https://grafana.com/docs/grafana/latest/fundamentals/exemplars/

In our context, which save the segment + position in case of min is updated, to make it easier to look after the segment in danger.

Change-Id: I19be482f1ddc7f1711e722c7b17480366d2c8312
2023-11-06 13:22:28 +01:00
Vitalii
86decb1f44 web/satellite/vuetify-poc: add info tooltip for download graph
Added info tooltip for download graph on project dashboard.

Issue:
https://github.com/storj/storj/issues/6461

Change-Id: If405268166fdd529415d7820e4bc87d5f8961ade
2023-11-06 11:25:19 +00:00
dlamarmorgan
d0f4447427
satellite/metabase: add version tests to IteratePendingObjectsByKey
Change-Id: I7335598eab2bac7f12aa93773cceadb195cd8e9e
2023-11-03 12:47:05 -07:00
Wilfred Asomani
81506203c4 web/satellite/vuetify-poc: add image preview to browser card view
This change adds image preview to the browser card view.

Issue: #6427

Change-Id: Iab8110fb3fa70fea29a98d8f96bac9b357dd401d
2023-11-03 18:04:18 +00:00
Wilfred Asomani
52b3ffd8c3 web/satellite/vuetify-poc: add authentication pages
This change copies over the new authentication UIs from the static
vuetify repository.

Change-Id: I06ea7dabbde2a091f524fbcdc8cb1eff08c90a74
2023-11-03 16:37:03 +00:00
Ivan Fraixedes
48877c05cc
private/apigen: Fix TS generator to align with linter
I introduced some subtle linter issues when I added the APIError class
and added the TypeScript mock generator.

This commit addresses them, so the linter doesn't yell about the
TypeScript generated sources.

Change-Id: Icc7dfa4169a228b1a5144d4a292f4350ee5ef9f0
2023-11-03 16:41:54 +01:00
Vitalii
f6e357be52 satellite/{console, web}: detailed usage report for a single project
Reworked usage report endpoint to return CSV for a single OR all the project user owns.
Added buttons to download usage report CSV for a single project.

Issue:
https://github.com/storj/storj/issues/6154

Change-Id: I55104088180dcf6be49dcde6c9c495f07ba01c5a
2023-11-03 14:58:18 +02:00
Michal Niewrzal
f0f73fc8ae satellite/metainfo: support downloading specific object version
Protobuf definition is ready to support downloading specific version of
object so we just need to wire requested version into metainfo
DownloadObject endpoint.

https://github.com/storj/storj/issues/6221

Change-Id: I3ddc173beb6a6cf30d782dd65c6aa5f88f2cbd44
2023-11-03 11:58:12 +00:00
Jeremy Wharton
51fefb2882 web/satellite: statically serve Vuetify theme styles
Vuetify's way of applying themes uses an inline stylesheet. This is
incompatible with our CSP policy, so this change implements a Vite
plugin that writes theme styles to CSS files that are statically
served.

Change-Id: I73e3a032435e46d41248c5181e913a8e04f65881
2023-11-02 23:36:15 +00:00
Ivan Fraixedes
74757ffc1d private/apigen/example: Fix client TS generated file
We introduced some changes in 882c9d64e4
and one reviewer spot an issue which was fixed, but I forgot to
regenerate the files and the generated Typescript client got stale.

This commit regenerate the example code, so the generated code is align
with the current generator version.

Change-Id: Idc023adc9c6e230578f9e77e492126355b006203
2023-11-02 14:24:30 +00:00
Wilfred Asomani
febd2091df web/satellite/vuetify-poc: add browser card view
This change adds an optional card view to the file browser similar to
the all projects card view.

Issue: #6427

Change-Id: I115dea7fdc2e7d0e093a00eb88e46d453c516cd9
2023-11-02 10:46:17 +00:00
Jeremy Wharton
e482e1296e web/satellite/vuetify-poc: fix file preview
This change fixes an issue where clicking a file in the Vuetify UI's
object browser would open an empty file preview dialog.

Change-Id: I5e21cf2e8c57911ac2708110f6ad23f376f86a54
2023-11-02 02:17:42 +00:00
paul cannon
51c930f532 satellite/repair: extra logging during TestSegmentRepairPlacement
In case we continue to see flaky TestSegmentRepairPlacement, this may
help narrow down the issue.

Change-Id: I34ca70e5bb33eca26e9940e845142121cc946ac0
2023-11-01 20:43:18 +00:00
paul cannon
b621a90745 satellite/repair: promote some Debug log lines to Info
We have frequently wanted to see this sort of information when looking
into issues. Considering that we already log a line at Info every time
we fail to download a piece during repair, it doesn't seem like a very
onerous new burden to log one line per segment saying that it is being
repaired.

Change-Id: I1fa84a985e90473adeb02603e207fc3c7b8592da
2023-11-01 19:52:51 +00:00
JT Olio
e81b578606 go.mod: bump storj.io/uplink
Change-Id: I7e63e57b8b582a5bce47c56bbc68fdd8f5f7aa48
2023-11-01 20:43:45 +02:00
Egon Elbre
9f39684799 cmd/storagenode: skip forget-satellite tests
Change-Id: I630bba8690e48f219d23a966796849a97657a49a
2023-11-01 17:24:44 +00:00
Egon Elbre
9e0fff5e22 satellite/gcbf: fix data race in TestGCBFUseRangedLoop
Satellites[0].GCBF is already started when testplanet boots up,
so calling Run separately ends up causing a data race.

Instead create a new instance, that should avoid this issue.

Fixes https://github.com/storj/storj/issues/6435

Change-Id: I6603ef63da7a6ab8bdb952cf5aaca17eb0392e2c
2023-11-01 19:07:20 +02:00
Egon Elbre
0a4a1098d8 go.mod: bump common (and maybe fix quic)
Change-Id: Ib37213ab6d26d2b6adc727e913e3df3e12e9ee9d
2023-11-01 14:42:15 +02:00
Egon Elbre
38ad5a1318 private/testplanet: add STORJ_TEST_DISABLEQUIC environment flag
Currently Windows seems to dislike creating udp servers in a very
fast fashion. Add an environment flag that allows to disable quic,
which is the main culprit.

Individual tests can still override that setting.

Change-Id: I3b30c4aa7fcb148b2894335394fdfae6eaa372bb
2023-11-01 11:39:03 +00:00
Wilfred Asomani
b3e908f72f satellite/{console,web}: add browser card view config flag
This change adds config flag to toggle whether the browser card view
can be used.

Issue: #6427

Change-Id: I19986ffe9a778b8752549a4d54b318fff0386479
2023-10-31 20:19:57 +00:00
Egon Elbre
f2eb6942c5 satellite/metabase: make the precommit constraint code nicer
This condenses the precommit constraint into a single function, which
allows to cleanup a bunch of logic elsewhere. Also, avoid delete in the
first place when we are not allowed to remove the uncommited object.

This also fixes submitting the metrics and now it submits the correct
metrics.

Change-Id: If91dfa3b19bce5b24ff2a19d7c34b57a200db1dd
2023-10-31 18:10:09 +00:00
Egon Elbre
e5e55ef266 satellite/metabase: move delete object unversioned code
Change-Id: Iec92c91c174b990fa44b210db415da7a4eb650c4
2023-10-31 17:14:16 +00:00
Egon Elbre
46f7e5b7fa all: don't depend on infectious library directly
Change-Id: I0bcc719cc25e1d7ad36a16f65d40220eccba6556
2023-10-31 16:00:47 +02:00
dlamarmorgan
77cac8b465 satellite/metabase: add tests ListObjects
Change-Id: Ia84b560251245fe0050426537751958f2c85e9e1
2023-10-30 20:33:08 +00:00
Wilfred Asomani
513c3cc632 satellite/admin: list users pending deletion
This change adds an endpoint to the admin API and UI to get a list of
users pending deletion and have no unpaid invoice.

Issue: #6410

Change-Id: I906dbf9eee9e7469e45f0c622a891867bf0cc201
2023-10-30 19:11:16 +00:00
Jeremy Wharton
405491e8d0 web/satellite{/vuetify-poc}: implement CSV file previewing
This change allows .csv files to be previewed in the file browser.

Resolves #6426

Change-Id: Ib93ebb417b8f69231ed2b36b8258ad91c6a1ba4b
2023-10-30 17:35:48 +00:00
Jeremy Wharton
42e1b088c2 satellite/admin/back-office/ui: use composition API
This change migrates Vue components of the new satellite admin web app
from the options API to the composition API.

Change-Id: Ie8c9bcd468f1c0fe0abc9ef0a3724563db096ba9
2023-10-30 16:50:30 +00:00
Wilfred Asomani
8f59535f95 web/satellite: use stripe payment element
This change uses the recommended stripe payment element to collect card
information instead of the legacy card element currently in use.

Issue: #6436

Change-Id: If931d47430940e0932c845b6eee3e0e23c294fbb
2023-10-30 15:37:30 +00:00
Michal Niewrzal
67f32bd519 satellite/metainfo: use protbuf ObjectVersion with GetObject request
We need to update metainfo GetObject endpoint to use ObjectVersion
([]byte) field instead of old Version field (int32).

Change-Id: I61663ec8d9f5c731f91346a285048477fb493730
2023-10-30 14:49:18 +00:00
Wilfred Asomani
f7a95e0077 satellite/{payment,console}: add endpoint to add card by pmID
This change introduces a new endpoint that allows adding credit cards
by payment method ID (pmID). The payment method would've already been
created by the frontend using the stripe payment element for example.

Issue: #6436

Change-Id: If9a3f4c98171e36623607968d1a12f29fa7627e9
2023-10-30 13:58:55 +00:00
Vitalii
32e67e5fab web/satellite: update low token balance banner copy
Updated the copy for low token balance to correspond with a new use-case.

Change-Id: I1e08a039849c2950cdc4b8dc3b4c9f470461b0dd
2023-10-30 12:49:10 +02:00
Egon Elbre
1fa918c255 satellite/metabase: simplify UpdateObjectLastCommitted
Change-Id: Ifec596ab1868baa03688e717adec7d2ab45eafa9
2023-10-28 04:43:43 +00:00
Egon Elbre
55bddb6ce1 satellite/metabase: don't use empty stream id for delete markers
Using a an empty stream id makes it more difficult to target a specific
delete marker. Similarly, we don't want to confuse actual stream id-s
with normal ones. So, we'll create stream id-s where the first few bytes
are 0xFF, but the rest is random.

Change-Id: Ia7fffb0da9a071be2935df99c0846027ee2e03c3
2023-10-28 04:43:32 +00:00
dlamarmorgan
aeec4bd213 satellite/metabase: add tests to UpdateMetadata for versioning
Change-Id: I4503f92b19212e7c4cf1907edd7e529ff75cca3e
2023-10-28 04:43:04 +00:00
paul cannon
8be7414a52 cmd/tools/segment-verify: fix global race in TestCommandLineTool.
This should stop the failures when TestCommandLineTool is run with
parallel test execution enabled.

Change-Id: Id80d5eacb78fcec886be786ae8b182517b17fbc6
2023-10-27 19:51:24 +00:00
Ethan Adams
e65345336f satellite/placement: Add test for new "datacenter" placement tag
Change-Id: Ic9a79bb46256fbe8b1159de2bbf787b4a63c5c8f
2023-10-27 18:15:32 +00:00
Wilfred Asomani
667b7c6f79 web/satellite/vuetify-poc: fix file preview
This change fixes an issue where clicking on a file from the upload
snackbar would not preview it.

Issue: #6433

Change-Id: I30b0b6098e058d1069b87f99425fac5d6e421fea
2023-10-27 17:18:52 +00:00
Vitalii
539253f646 web/satellite: don't make billing API requests if billing is disabled
Removed billing API calls on all project dashboard if billing is disabled.

Change-Id: Ia3407c811e599f40670b97e1c2ad9a8f661c0e2a
2023-10-27 18:53:14 +03:00
Jeremy Wharton
4f07994bfb Jenkinsfile.premerge: style check satellite admin back office files
This change runs our style checker for assets related to the new
satellite admin web app in the pre-merge build. Additionally, the
testing and style checking for web assets has been parallelized.

Change-Id: I0835673435f7ed8a4704c9373fcf5bb8eb5e7d0a
2023-10-27 09:39:26 -05:00
Jeremy Wharton
8ebf285081 satellite/admin/back-office/ui: lint files
This change lints files for the new satellite admin web app. An
ESLint config has been added for it that is identical to the one used
for linting satellite web app files.

Change-Id: I66f72fb880d5cbc80b6c080294e4a830b3d28143
2023-10-27 11:23:58 +00:00
Egon Elbre
27724835da satellite/metabase: use database order in ObjectStream.Less
Change-Id: I3d5e00263d78d9128fae6d39a3bc7f0c542039b6
2023-10-27 07:38:51 +00:00
Egon Elbre
a4edbdd005 satellite/metabase: use commit order for objects
This changes metabase behavior such that the latest object
will be the committed object which may result in a new version
assigned to the objects.

Change-Id: I7a731d7db4696caba75fff65107a248569b6111f
2023-10-27 07:38:25 +00:00