This reverts commit 2b4f347c33.
Reason for revert: not a zero-downtime migration. We need to deploy the dbx changes before completely dropping the column.
Change-Id: Ic9215650242b7848d54f8a5f863a13bc18b60149
This change adds two new admin endpoints to freeze users for legal
review and to remove them from that state
Issue: storj/storj-private#492
Change-Id: I6c8e3ffcb80375e81e78bc6ecc785c1047328cf7
This change adds an endpoint to the back office API that returns
placement IDs and their locations.
References #6503
Change-Id: I20ee1c82dcb647d6d264317beceeb5e70f7a8e87
Implement the authorization that will hook into each endpoint handler
through a wrapping handler for defining the permissions that each
endpoint requires.
Change-Id: I9c8f12b58f48e849e7ea35f372dddce5c9cfc5b5
For convenience of not having to modify the API generator to contemplate
the path prefix that we are adding to the back office server, we define
the path prefix in a constant than the admin server and the definition
of the API uses to adapt the router and the generated code.
Change-Id: Ic557b0e6e88e930e03647835759bb34e06e8bb48
We decided that we won't use seprate table for handling pending
objects. We need to remove related code.
https://github.com/storj/storj/issues/6421
Change-Id: I442b0f58da75409f725e08e2cd83d29ed4f91ec6
Removed MetadataSize, RepairEgress, AuditEgress fields from usage report because they are not relevant.
Issue:
https://github.com/storj/storj/issues/6498
Change-Id: I8ef7d56bd1b01cdb53cec8c67dd47a6a0e7fa184
This change prevents Go code produced by the API generator from
importing its own package. Previously, we tried to prevent self imports
by skipping import paths whose last segment matched the generated Go
code's package name. However, aliased imports circumvented this.
We now require API definitions to define the Go package path so that we
can compare this with the import path directly.
Change-Id: I7ae7ec5e1a342d2f76cd28ff72d4cd7285c2820a
This change wraps account freeze code in DB transactions to prevent
freeze inconsistencies resulting from errors that happen in the process
of freezing accounts.
Change-Id: Ib67fb30dc33248413d3057ceeac5c2f410f551d5
To be compatible with S3 we need to return 'Method Not Allowed' when
delete marker is requested and to do this libuplnk needs to know about
delete marker. It will be returned only if object version will be
specified.
Change-Id: I288da5566c74e1b4951f7cd249dbf34622b92e91
We decided that we won't use seprate table for handling pending
objects. We need to remove related code.
After this change we still needs to remove related code from metabase.
https://github.com/storj/storj/issues/6421
Change-Id: Idaaa8ae08b80bab7114e316849994be171daf948
PathPrefix is the full path of the subrouter passed to the back office
server when it hooks into another server, in this case, the satellite
admin server.
PathPrefix allows to serve the static assets from the root of the
sub-router when the prefix is stripped before accessing them.
There was a bug where the PathPrefix weren't used and a hard-coded path
was used.
Test passed because the back-office server is hooked into the satellite
admin server with a subrouter with path `/back-office/` which matched
the hard-code value, however, it wouldn't work if that path changed or
it is hooked into another server with a different subrouter path,
despite it was set to PathPrefix.
This commit fixes that bug.
Change-Id: Id4a0d86329eb563b008b3fc6f8eb7b51cbfd2e6f
This change extends the account freeze functionality account for legal
freezes as well. This is a freeze event for accounts to be put on hold
for legal review. It also sets the LegalHold status on the affected
user.
Issue: storj/storj-private#492
Change-Id: I8c733269b5cfb647c840379a6bb033da120c8280
This change adds a config flag for whether the stripe payment element
should be used to collect card info.
Change-Id: I301cf69e6f1b64350266e8f2286542b951e216c4
Additional feature flag (onyly for testing) to set versioning enabled
for all new create buckets. We need it until we will have support
for enabling/disabling versioning for bucket on metainfo API.
In addition this change is fixing also two small issues which makes
testing this flag imposible:
* metabase Status list was not aligned with protobuf definition
* object retruned by metainfo API didn't have correct status set in some
cases
Change-Id: I0d63dff6a08efa588c8999af1e17db476943e067
This change replaces with "Storj" all instances of "Storj DCS" or
"Storj OSP".
Issues: storj/storj-private#479
#5069 and #5839
Change-Id: I78d13f2abd8bde5d2cb42d8006016ee87130a566
For some reason minio sometimes validates versionID as UUID. Until
we decide what to do lets align our version format with it.
Change-Id: I6e9832d0adc1d3b6e3f46688b386e0e118219038
This change adds a feature flag (default disabled) to enable the ability
to submit a project limit increase request directly from the UI. When
this feature is disabled, the user will be directed to a page to file a
support ticket.
Resolves https://github.com/storj/storj/issues/6480
Change-Id: I5f355dcb1a40e5b694f9623f05fe706ed4d6a528
This change introduces a new user status, LegalHold to be used to pause
an account while it is under legal review. It also modifies the
condition whether to generate invoices for a user to apply to this new
status.
Issue: storj/storj-private#492
Change-Id: Idc9a6166cc96178a95e45cf7c0f2e982631ca021
Credit to the tests in TestListObjectsVersioned(), which (since
ListObjects is logically a pretty thin wrapper around
iterateAllVersionsWithStatus()) were able to be translated to tests in
TestIterateObjectsWithStatus with only minor changes.
Change-Id: I6e68675b24fee614e44baddf596703115554014e
This small feature flag will give us the ability to test object
versioning for specific projects without enabling it globally.
Change-Id: I78301f071b7b8079dd1bd4a561fce0800ce9f074
This creates an automated test for the situation where a node initiates
graceful exit while TimeBased is off, and then TimeBased is turned on
before the node has completed graceful exit. The node should no longer
try to transfer any more pieces, but should instead sit and wait until
the graceful exit period has elapsed.
Change-Id: Iaf636f9247bc878bc20041221e1a8014c77806ad
Send signup captcha score (if it exists) to Hubspot and Segment, so that
we can implement logic based on this in the analytics platforms.
Change-Id: Ic4e166639c9ab8e872a212e7466e82433c9ea6d4
To make sure that Benchmark tests are good, we run them with -short flag, eg:
```
go test -short -run=BenchmarkDurabilityProcess
```
Durability benchmark already supports this, but we can make it slightly more faster with
using less sgements and pieces during the `-short` run.
Change-Id: I9547ca1e3cd0178eb395a7a388f2e7936a9862d7
Allow all the operations when accessing through Oauth, but requires the
authorization token for the ones that we consider that they are
sensitive.
Before these changes, a group of operations weren't available through
Oauth, and people who has access to the authorization token had to
forward the port of the server to their local in order to do them
without Oauth.
These changes shouldn't reduce the security because people who has
access to the authorization token is the same than they can forward the
port and part of those have Oauth access too.
Allowing to perform all the operations through Oauth will improve the
productivity of production owners because they will be able to do all
the administration requests without having to port forward the server.
Change-Id: I6d678abac9f48b9ba5a3c0679ca6b6650df323bb
Classifier of durability is sg. like "net:1.3.4.1" or "country:HU".
To make the calculation faster we use arrays instead of maps, which means that we assign a uinique index to all of these strings (classes).
As Egon suggested earlier, we can do this mapping only once (per process), not for each fork.
Not a big deal performance-wise, as we have limited number of forks, which are initialized once per 5-10 hours, but the code is more readable and clean.
Change-Id: Id081846b5d97dae8009aeeecbcc63cb713bed294
QA discovered that if your node lost all its data but your audit score
was still healthy, you could call graceful exit, respond with unknown
audit errors when audited, and eventually get suspended, but still get
your held amount back.
We will therefore mark suspended nodes as having failed graceful exit.
Refs: https://github.com/storj/storj-private/issues/485
Change-Id: Id5af18786b574651587cc96bd6a7d0b47c0671a8
I can't say with certainty yet what caused the two failures I know
about, but I have one theory: the node continuing to check in during the
test skewed the online score towards 1, and using the test default for
GracefulExitDurationInDays meant there were fewer update periods than
expected.
At any rate, it is more correct to pause the graceful exit processing
chore and the contact chore during the test, even if it doesn't end up
solving the problem.
Refs: https://github.com/storj/storj/issues/6401
Change-Id: I06d43d531e0b3344af13878c8d55213349fdcfa3
Create a separate server for implementing the new satellite
administration web app.
This server is in a new package that will implement all the
functionality for the new satellite administration back-end and when it
be completed with all the functionality that the current one offer, it
will replace it.
For now, the new server only exposes the static assets as they were
exposed by the current server.
A main sub-package is added with an example endpoint to scaffold where
we'll define the API through the API generator and to locate the several
generated files.
Change-Id: I172c43b2c180553876ef7ce137cc778b94723451
Exemplars are representative elements for the stat. For example if a stat min is `30`, we can save one example with that value.
More details about the concept is here: https://grafana.com/docs/grafana/latest/fundamentals/exemplars/
In our context, which save the segment + position in case of min is updated, to make it easier to look after the segment in danger.
Change-Id: I19be482f1ddc7f1711e722c7b17480366d2c8312
Reworked usage report endpoint to return CSV for a single OR all the project user owns.
Added buttons to download usage report CSV for a single project.
Issue:
https://github.com/storj/storj/issues/6154
Change-Id: I55104088180dcf6be49dcde6c9c495f07ba01c5a
Protobuf definition is ready to support downloading specific version of
object so we just need to wire requested version into metainfo
DownloadObject endpoint.
https://github.com/storj/storj/issues/6221
Change-Id: I3ddc173beb6a6cf30d782dd65c6aa5f88f2cbd44
We have frequently wanted to see this sort of information when looking
into issues. Considering that we already log a line at Info every time
we fail to download a piece during repair, it doesn't seem like a very
onerous new burden to log one line per segment saying that it is being
repaired.
Change-Id: I1fa84a985e90473adeb02603e207fc3c7b8592da
Satellites[0].GCBF is already started when testplanet boots up,
so calling Run separately ends up causing a data race.
Instead create a new instance, that should avoid this issue.
Fixes https://github.com/storj/storj/issues/6435
Change-Id: I6603ef63da7a6ab8bdb952cf5aaca17eb0392e2c
This condenses the precommit constraint into a single function, which
allows to cleanup a bunch of logic elsewhere. Also, avoid delete in the
first place when we are not allowed to remove the uncommited object.
This also fixes submitting the metrics and now it submits the correct
metrics.
Change-Id: If91dfa3b19bce5b24ff2a19d7c34b57a200db1dd
This change adds an endpoint to the admin API and UI to get a list of
users pending deletion and have no unpaid invoice.
Issue: #6410
Change-Id: I906dbf9eee9e7469e45f0c622a891867bf0cc201
This change migrates Vue components of the new satellite admin web app
from the options API to the composition API.
Change-Id: Ie8c9bcd468f1c0fe0abc9ef0a3724563db096ba9
We need to update metainfo GetObject endpoint to use ObjectVersion
([]byte) field instead of old Version field (int32).
Change-Id: I61663ec8d9f5c731f91346a285048477fb493730
This change introduces a new endpoint that allows adding credit cards
by payment method ID (pmID). The payment method would've already been
created by the frontend using the stripe payment element for example.
Issue: #6436
Change-Id: If9a3f4c98171e36623607968d1a12f29fa7627e9
Using a an empty stream id makes it more difficult to target a specific
delete marker. Similarly, we don't want to confuse actual stream id-s
with normal ones. So, we'll create stream id-s where the first few bytes
are 0xFF, but the rest is random.
Change-Id: Ia7fffb0da9a071be2935df99c0846027ee2e03c3
This change lints files for the new satellite admin web app. An
ESLint config has been added for it that is identical to the one used
for linting satellite web app files.
Change-Id: I66f72fb880d5cbc80b6c080294e4a830b3d28143
This changes metabase behavior such that the latest object
will be the committed object which may result in a new version
assigned to the objects.
Change-Id: I7a731d7db4696caba75fff65107a248569b6111f
This change speeds up the build of the new satellite admin web app
using the same method that 8b0d25c used to speed up the build of the
new satellite web app.
New build time: 23.394s
Old build time: 1m40.930s
Change-Id: Ic6fcfdfc73b78bc26bddc421c608076b23532967
Currently it's awkward to use any of the existing statuses for the
objects in non-recursive listing. Hence, let's add a new one.
Change-Id: I8485e0f858e69998b097e757091991538ca697fa
We never extended metainfo protocol to return committed object
detailed into and this change is doing it now. Main motivation to
do this now is need for providing object version after upload.
Change-Id: Ib59bdfd9485e4a0091ac02458cc63427cb7159de
This is fixing two small issues with logging:
* repair checker was logging empty last net values as clumped pieces
but in main logic we stopped classifying it this way
* repairer summary log was showing incorrect number of pieces removed
from segment because list contains duplicated entries
There was no real issue here.
Change-Id: Ifc6a83637d621d628598200cad00cce44fa4cbf9
Removed AllProjectDashboard feature flag.
Removed unused Vue components.
Fixed wrong redirect on reload if pricing packages are disabled.
Fixed wrong redirect on reload if billing features are enabled.
Issue:
https://github.com/storj/storj/issues/6434
Change-Id: I9081a6f737c45fb48da5b23c016a42e23021c4ce
Protobuf definition is ready to support deleting specific version of
object so we just need to wire requested version into metainfo
BeginDeleteObject endpoint.
Dependencies bumped to get latest metainfo protobuf definition.
https://github.com/storj/storj/issues/6221
Change-Id: Ifc3cc0b49d9acdf4f7e57e7184b0f2ae045f9113
A randomly generated UUID can hit something that starts with `00`.
Also fix a tiny mistake in the comment.
Change-Id: I25a8b21e0f9523bc486e5a38b0c3cc9c36515231
Use `(a, b, c) = ($1, $2, $3)` for the object stream location
arguments rather than combining multiple AND queries together.
Using a tuple comparison is shorter and also easier to see which
object is selected.
Change-Id: Iba84b89630d57255023c30e309eb6afaee9ab944
Add the DB schema changes and basic implementation methods to support
bucket level configuration of object versioning.
See bucket-level-versioning-configuration design doc for more details.
Change-Id: I4e920a20a403b3157970a34eb619d827a4007845
Today we got the following error address, even if: `mail.auth-type: nomail`
```
Error: SMTP server address '' couldn't be parsed: missing port in address
```
It's make more sense to print the error message only if the SMTP address (or from address) is expected.
Change-Id: Ie07620099b6aac27630fcfd1cda9921ef4b6060c
This change updates account freeze to set and use the days till
escalation column of the account freezes table.
Issue: #6382
Change-Id: I345798e3d53e5ab4a7653723433fb8affa258212
This change ensures that the user is never prompted to upgrade when
attempting to invite project members if billing features are configured
to be disabled.
Change-Id: I1c49351b00c0e378da24ad080fd1d3b078c97c71
This change adds a flag to the satellite config indicating whether
free tier users should be able to send project invitations.
Change-Id: I9c030c88dbef136ba4a9bf2d8f027a8dcd77fd33
