Credit to the tests in TestListObjectsVersioned(), which (since
ListObjects is logically a pretty thin wrapper around
iterateAllVersionsWithStatus()) were able to be translated to tests in
TestIterateObjectsWithStatus with only minor changes.
Change-Id: I6e68675b24fee614e44baddf596703115554014e
This small feature flag will give us the ability to test object
versioning for specific projects without enabling it globally.
Change-Id: I78301f071b7b8079dd1bd4a561fce0800ce9f074
This creates an automated test for the situation where a node initiates
graceful exit while TimeBased is off, and then TimeBased is turned on
before the node has completed graceful exit. The node should no longer
try to transfer any more pieces, but should instead sit and wait until
the graceful exit period has elapsed.
Change-Id: Iaf636f9247bc878bc20041221e1a8014c77806ad
This change increases the grace period within which test satellites
are expected to respond after being instantiated. Our Jenkins build
pipeline for GitHub was failing because the grace period was too short.
Change-Id: I8bfd7c771660fc39f6eb988f95d706809f936d00
Send signup captcha score (if it exists) to Hubspot and Segment, so that
we can implement logic based on this in the analytics platforms.
Change-Id: Ic4e166639c9ab8e872a212e7466e82433c9ea6d4
Much evident on the storagenode dashboard, the minimum
version shown is a very old version and that is taken
from the deprecated part of version info from the
version control server, and we no longer update the
deprecated part on the server.
This change forces it to use the new sem version, and
checks for the old version if the server is probably
running an old version of the version control system.
Also fixes a bug where the suggested version returned
for all processes is taken from the storagenode part.
Issue: https://github.com/storj/storj/issues/5673
Change-Id: I57b7358c2826a6e25f441dfa9579a1aef50a7e89
WARNING: THIS COMMIT IS ACCIDENTALLY MERGED WITH THE PREVIOUS ONE. BUT I KEEP THE COMMIT MESSAGE FOR IT’S VALUE. CODE IS IN THE PARENT
As Wikipedia defines, bus fator is "the minimum number of team members that have to suddenly
disappear from a project before the project stalls due to lack of knowledgeable or competent personnel."
We use similar definition, but instead of team members, we check the pieces of one segments.
For example, if we have a segment with 10 pieces in US, 8 pieces in DE and 3 pieces in other countries,
traditional bus factor (50) is 2, as we need two contry down (10 + 8) two loose at least 50% of the pieces (10.5).
Loosing just one contry, we will have <50% of th pieces.
In our implementation we are not interested about the 50% threshold, but a configurable value.
As a default we use the repairThreshold - minimalNumber.
With the usual defaults (54 - 29) it's 25. So bus factor can be defined sg. like this:
How many GROUP should disappear from the network to drop the healthy count from repair threshold to the minimum number.
This is critical, as we don't repair anything above the repair threshold, but heatlhy piece number can drop
if the given amount of groups (subnets, countries...) live the network...
Change-Id: I606f091469b45e90f3a9eb8fcff65a834ff27a14
To make sure that Benchmark tests are good, we run them with -short flag, eg:
```
go test -short -run=BenchmarkDurabilityProcess
```
Durability benchmark already supports this, but we can make it slightly more faster with
using less sgements and pieces during the `-short` run.
Change-Id: I9547ca1e3cd0178eb395a7a388f2e7936a9862d7
Allow all the operations when accessing through Oauth, but requires the
authorization token for the ones that we consider that they are
sensitive.
Before these changes, a group of operations weren't available through
Oauth, and people who has access to the authorization token had to
forward the port of the server to their local in order to do them
without Oauth.
These changes shouldn't reduce the security because people who has
access to the authorization token is the same than they can forward the
port and part of those have Oauth access too.
Allowing to perform all the operations through Oauth will improve the
productivity of production owners because they will be able to do all
the administration requests without having to port forward the server.
Change-Id: I6d678abac9f48b9ba5a3c0679ca6b6650df323bb
Classifier of durability is sg. like "net:1.3.4.1" or "country:HU".
To make the calculation faster we use arrays instead of maps, which means that we assign a uinique index to all of these strings (classes).
As Egon suggested earlier, we can do this mapping only once (per process), not for each fork.
Not a big deal performance-wise, as we have limited number of forks, which are initialized once per 5-10 hours, but the code is more readable and clean.
Change-Id: Id081846b5d97dae8009aeeecbcc63cb713bed294
QA discovered that if your node lost all its data but your audit score
was still healthy, you could call graceful exit, respond with unknown
audit errors when audited, and eventually get suspended, but still get
your held amount back.
We will therefore mark suspended nodes as having failed graceful exit.
Refs: https://github.com/storj/storj-private/issues/485
Change-Id: Id5af18786b574651587cc96bd6a7d0b47c0671a8
I can't say with certainty yet what caused the two failures I know
about, but I have one theory: the node continuing to check in during the
test skewed the online score towards 1, and using the test default for
GracefulExitDurationInDays meant there were fewer update periods than
expected.
At any rate, it is more correct to pause the graceful exit processing
chore and the contact chore during the test, even if it doesn't end up
solving the problem.
Refs: https://github.com/storj/storj/issues/6401
Change-Id: I06d43d531e0b3344af13878c8d55213349fdcfa3
Create a separate server for implementing the new satellite
administration web app.
This server is in a new package that will implement all the
functionality for the new satellite administration back-end and when it
be completed with all the functionality that the current one offer, it
will replace it.
For now, the new server only exposes the static assets as they were
exposed by the current server.
A main sub-package is added with an example endpoint to scaffold where
we'll define the API through the API generator and to locate the several
generated files.
Change-Id: I172c43b2c180553876ef7ce137cc778b94723451
It has been noted in the forum that, during a Retain operation, when a
piece can't be deleted, the process never completes. The error is
written to the log, but the completion line "Moved pieces to trash
during retain" never is.
This `return` line is the reason. We should instead continue the loop.
Change-Id: I0f51d34aba0e81ad60a75802069b42dc135ad907
Refs: https://github.com/storj/storj/issues/6482
Exemplars are representative elements for the stat. For example if a stat min is `30`, we can save one example with that value.
More details about the concept is here: https://grafana.com/docs/grafana/latest/fundamentals/exemplars/
In our context, which save the segment + position in case of min is updated, to make it easier to look after the segment in danger.
Change-Id: I19be482f1ddc7f1711e722c7b17480366d2c8312
I introduced some subtle linter issues when I added the APIError class
and added the TypeScript mock generator.
This commit addresses them, so the linter doesn't yell about the
TypeScript generated sources.
Change-Id: Icc7dfa4169a228b1a5144d4a292f4350ee5ef9f0
Reworked usage report endpoint to return CSV for a single OR all the project user owns.
Added buttons to download usage report CSV for a single project.
Issue:
https://github.com/storj/storj/issues/6154
Change-Id: I55104088180dcf6be49dcde6c9c495f07ba01c5a
Protobuf definition is ready to support downloading specific version of
object so we just need to wire requested version into metainfo
DownloadObject endpoint.
https://github.com/storj/storj/issues/6221
Change-Id: I3ddc173beb6a6cf30d782dd65c6aa5f88f2cbd44
Vuetify's way of applying themes uses an inline stylesheet. This is
incompatible with our CSP policy, so this change implements a Vite
plugin that writes theme styles to CSS files that are statically
served.
Change-Id: I73e3a032435e46d41248c5181e913a8e04f65881
We introduced some changes in 882c9d64e4
and one reviewer spot an issue which was fixed, but I forgot to
regenerate the files and the generated Typescript client got stale.
This commit regenerate the example code, so the generated code is align
with the current generator version.
Change-Id: Idc023adc9c6e230578f9e77e492126355b006203
This change adds an optional card view to the file browser similar to
the all projects card view.
Issue: #6427
Change-Id: I115dea7fdc2e7d0e093a00eb88e46d453c516cd9
This change fixes an issue where clicking a file in the Vuetify UI's
object browser would open an empty file preview dialog.
Change-Id: I5e21cf2e8c57911ac2708110f6ad23f376f86a54
We have frequently wanted to see this sort of information when looking
into issues. Considering that we already log a line at Info every time
we fail to download a piece during repair, it doesn't seem like a very
onerous new burden to log one line per segment saying that it is being
repaired.
Change-Id: I1fa84a985e90473adeb02603e207fc3c7b8592da
Satellites[0].GCBF is already started when testplanet boots up,
so calling Run separately ends up causing a data race.
Instead create a new instance, that should avoid this issue.
Fixes https://github.com/storj/storj/issues/6435
Change-Id: I6603ef63da7a6ab8bdb952cf5aaca17eb0392e2c
Currently Windows seems to dislike creating udp servers in a very
fast fashion. Add an environment flag that allows to disable quic,
which is the main culprit.
Individual tests can still override that setting.
Change-Id: I3b30c4aa7fcb148b2894335394fdfae6eaa372bb
This condenses the precommit constraint into a single function, which
allows to cleanup a bunch of logic elsewhere. Also, avoid delete in the
first place when we are not allowed to remove the uncommited object.
This also fixes submitting the metrics and now it submits the correct
metrics.
Change-Id: If91dfa3b19bce5b24ff2a19d7c34b57a200db1dd
This change adds an endpoint to the admin API and UI to get a list of
users pending deletion and have no unpaid invoice.
Issue: #6410
Change-Id: I906dbf9eee9e7469e45f0c622a891867bf0cc201
This change migrates Vue components of the new satellite admin web app
from the options API to the composition API.
Change-Id: Ie8c9bcd468f1c0fe0abc9ef0a3724563db096ba9
This change uses the recommended stripe payment element to collect card
information instead of the legacy card element currently in use.
Issue: #6436
Change-Id: If931d47430940e0932c845b6eee3e0e23c294fbb
We need to update metainfo GetObject endpoint to use ObjectVersion
([]byte) field instead of old Version field (int32).
Change-Id: I61663ec8d9f5c731f91346a285048477fb493730
This change introduces a new endpoint that allows adding credit cards
by payment method ID (pmID). The payment method would've already been
created by the frontend using the stripe payment element for example.
Issue: #6436
Change-Id: If9a3f4c98171e36623607968d1a12f29fa7627e9
