This is the final change to completely drop the type
column from the nodes table.
All code references are removed and deployed, dbx changes
have been merged and deployed.
Updates https://github.com/storj/storj/issues/5426
Change-Id: I1b76fe92f25e350c1c44730c008f0e9107852509
This change modifies the register endpoint handler to not require name
for signups from the v2 app and adds a new endpoint for completing
account information (e.g. name). This is to support the new signup and
account setup flow of the v2 app.
Issue: #6470
Change-Id: I256e1c804fcdbc8ce05aa82d6bc4b0263f55daa5
The code to get daily storage usage is a little confusing. There is a
portion of the query which looks like it is trying to sum the storage
for buckets by each day, but it doesn't work. There is also a section
which sums the buckets by reading the returned rows, but it assumes that
the rows are ordered by the date, which they are not. This results in
the potential for multiple entries for the same day being created: one
for each bucket. The UI doesn't know about this, so the entries for the
same day overwrite each other and result in the appearance of less
stored data on the graph.
To fix it, don't group by bucket and allow the query to sum the buckets
together.
Change-Id: Id3ec25238e6e6346b93b10f834bdc9c70023f1e1
This change adds a new column to the user table, which will hold a
random code for account activation. And another to hold the signup
request ID as another layer of verification.
Issue: #6428
Change-Id: Icd46cb5d8fc76102264d599aca27686cd8b2e84e
Currently we have large set of nodes that are already disqualified and
we are not sending bloom filters to them. The issue is that we are still
generating filters for them while garbage collection process. Even if
we have only segment with one piece which was stored on this node. This
consumes additional memory and processing powers.
This change is changing logic behind `AllPieceCounts` (renamed to
ActiveNodesPieceCounts) to return piece count for all nodes except disqualified one (even with piece count = 0). With this change we can
modify GC observer to skip nodes that where not returned by
ActiveNodesPieceCounts.
Change-Id: Ic75159135abe535084d8aeee560bb801a4a03e17
This reverts commit 2b4f347c33.
Reason for revert: not a zero-downtime migration. We need to deploy the dbx changes before completely dropping the column.
Change-Id: Ic9215650242b7848d54f8a5f863a13bc18b60149
This change extends the account freeze functionality account for legal
freezes as well. This is a freeze event for accounts to be put on hold
for legal review. It also sets the LegalHold status on the affected
user.
Issue: storj/storj-private#492
Change-Id: I8c733269b5cfb647c840379a6bb033da120c8280
This small feature flag will give us the ability to test object
versioning for specific projects without enabling it globally.
Change-Id: I78301f071b7b8079dd1bd4a561fce0800ce9f074
This change adds an endpoint to the admin API and UI to get a list of
users pending deletion and have no unpaid invoice.
Issue: #6410
Change-Id: I906dbf9eee9e7469e45f0c622a891867bf0cc201
Add the DB schema changes and basic implementation methods to support
bucket level configuration of object versioning.
See bucket-level-versioning-configuration design doc for more details.
Change-Id: I4e920a20a403b3157970a34eb619d827a4007845
Since we no longer need the type column and hopefully,
there are no queries using it, we recreate all indices
that reference the nodes.type column.
Updates https://github.com/storj/storj/issues/5426
Change-Id: I610ccaf474a6f4031e166b79a6d649c4b138e338
Once this is merged and deployed, we can go ahead
and change the db schema.
Note that this will stop updating the `type` column in the nodes
table but it shouldn't be a problem because this column has a
default value set.
Updates https://github.com/storj/storj/issues/5426
Change-Id: I2470ebacbcb0e60cf894617eb69b593227357283
This change adds a new column to the account_freeze_events table,
days_till_escalation, which is the number of days till the freeze event
is escalated. E.g.: 30 days from billing warning event to billing
freeze event.
Issue: #6382
Change-Id: I15cfd6f1208e641d8c380bef61717031befdad73
The overlay.NodeDossier struct only tracks information about a
storagenode, the field is deprecated and no longer needed.
This is a kademlia left-over.
Updates https://github.com/storj/storj/issues/5426
Change-Id: Ie278ffd88d1b9a9fde6c81eb5f0e287bab8c9ef0
Allow user to download detailed usage report from Billing -> Overview screen.
Report is a CSV file containing usage data for all the projects user owns.
Issue:
https://github.com/storj/storj/issues/6154
Change-Id: I3109002bf37b1313652a2be3447aaa7bc6204887
This change extends the account freeze functionality account for
violation freezes as well.
Also, debug level logs in the freeze chore have been changed to info.
It adds an analytics event for when an invoice is found that belongs to
a user frozen for violation.
And finally adds whether a user is frozen for violation to the
/account/freezestatus response.
Issue: https://github.com/storj/storj-private/issues/386
Change-Id: Id8e40282dc8fd8f242da52791ab8ddbbef3da2bc
deleteGeofenceForProject wasn't able to work correctly, because
Console().Projects().Update() declines to update default_placement when
the input value is 0.
This introduces a Console().Projects().UpdateDefaultPlacement() method,
congruent to the method of the same name on Console().Users().
deleteGeofenceForProject now uses this new method, so that specifying a
new placement of 0 will work correctly.
Change-Id: I4589b36707f7e4f1cfdc66543520b0d4205c1a84
This column is no longer used, but it is referenced in dbx. This change
removes those references and adds an exception to the migration test to
disregard this column in schema comparison.
After this change is deployed, we will need to follow up with a
migration to remove the column in a later release, and remove the
exception in the migration test.
Related to https://github.com/storj/storj/issues/5432
Change-Id: I168fb57244b347901d1ed9c7813c1338554ee644
this fixes cases where it's possible to update a user and the
DefaultPlacement field gets overwritten to the zero value.
it also adds UpdateDefaultPlacement which can be used to set
DefaultPlacement directly. This is needed for the geofencing
endpoints in satellite admin to set the DefaultPlacement back
to zero to delete geofencing for a user.
Change-Id: If2c798dabfa6773ed6023fb8257bf00ec7bc2e68
NodeSelection struct is used to make decisions (and assertions) related to node selection.
Usually we don't use email and wallet for placement decision, as they are not reliable.
But there are cases, when we know that the email address is confirmed. Also, it can be used for upper-bound estimations (if same wallet is used for too many pieces in a segment, it's a sign of a risk, even if not all the risks can be detected with this approach, as one owner can use different wallets).
Long story short: let's put wallet and email to the SelectedNode.
Change-Id: I922185e3769d43eb7762b8d60d88ecd3d50991bb
Currently, pending invoice payments that are made using a users token
balance can get stuck in a pending state if the invoice is not able
to be paid appropriately in stripe. This change addresses these stuck
token invoice payments by attempting to transition them to failed
if the invoice cannot be paid.
Change-Id: I2b70a11c97ae5c733d05c918a1082e85bb7f73f3
Current bucket_metainfos table schema was far from optimal. All
operations are using project_id and name to find bucket but primary key
was on id field which is completely not used. Turns out that secondary
index was not as fast as it could be because each select was also joining
primary index which was slowing down whole query.
Primary key is changed to project_id/name. Other indexes are dropped
and id column is now nullable which gives us space for dropping it
completely from schema and code later.
Change-Id: I477056a4243d39e39489f1473de01ded1ed24bbb
This change introduces a new freeze event, ViolationFreeze, for ToS
violations and also prepends other events with Billing to clarify what
they signify.
Issue: https://github.com/storj/storj-private/issues/386
Change-Id: Ieef2abbbb03d06377dc7a73ba5ef0742ada75e8e
Merged bandwidth graph lines to show only allocated-dead for last 3 days and settled for other days.
Issue:
https://github.com/storj/storj/issues/6072
Change-Id: Ic7f03d22ccd82d27ae6e6a85e73e144c9852e33b
API responses containing project information now contain the edge
service URL overrides configured for that project. The overrides are
based on the project's default placement.
References #6188
Change-Id: Ifc3dc74e75c0f5daf0419ac3be184415c65b202e
This reverts commit 31ec421299.
This change made the usages endpoint slower for accounts with large
number of projects.
Change-Id: I95870e95c2bf3bc3050087532fd0d20cbb50748b
The easiest way to get node information WITH node tags is executing two queries:
1. select all nodes
2. select all tags
And we can pair them with a loop, using the in-memory data structures.
But this approach does work only, if we select all nodes, which is true when we use cache (upload, download, repair checker).
But repair process selects only the required nodes, where this approach is suboptimal. (full table scan for all tags, even if we need only tags for a few dozens nodes).
Possible solutions:
1. We can introduce a cache for repair (similar to upload cache)
2. Or we can select both node and tag information with one query (join).
This patch implements the second approach.
Note: repair itself is quite slow (10-20 seconds per segements to repair). With 15 seconds execution time and 3 minutes cache staleness, we would use the cache only 12 times per worker. Probably we don't need cache for now.
https://github.com/storj/storj/issues/6198
Change-Id: I0364d94306e9815a1c280b71e843b8f504e3d870
as GetParticipatingNodes and GetNodes, respectively.
We now want these functions to include offline and suspended nodes as
well, so that we can force immediate repair when pieces are out of
placement or in excluded countries. With that change, the old names no
longer made sense.
Change-Id: Icbcbad43dbde0ca8cbc80a4d17a896bb89b078b7
This change addresses an issue where the /charges endpoint will take a
while to respond due to a project having a large number of buckets.
The method and queries involved have been optimized and benchmarks show
a performance improvement.
test name old ms/op new ms/op
Postgres/sum_all_partner_usages 3.659 1.101
Postgres/individual_partner_usages 3.74 1.299
Cockroach/sum_all_partner_usages 7.201 2.872
Cockroach/individual_partner_usages 7.247 2.852
Issue: https://github.com/storj/storj-private/issues/277
Change-Id: Ia5082a2e1c3e91120a9db7b01c18847fe04574fe
This change fixes an issue where the console DB cleanup chore was never
able to run when using a Cockroach database implementation because of
an inappropriate AS OF SYSTEM TIME clause in the relevant methods.
Resolves#6197
Change-Id: I8456b6df2128678e0eebeb416eb1a955cc9bd706
In some rare cases when two entities are trying to create the same
bucket at the same time it's possible that we will return internal
error instead of `bucket already exists`. It's because we are not
handling correctly DB error about constraint error. This change checks
if while inserting bucket into DB we got constraint error and propagate
correct error to metainfo API.
Change-Id: Ie6fd2c943b864b4ea7d71e4a162e74dc3510e386
This patch fixes the node tag based placement of rangedloop/repairchecker + repair process.
The main change is just adding the node tags for Reliable and KnownReliabel database calls + adding new tests to prove, it works.
https://github.com/storj/storj/issues/6126
Change-Id: I245d654a18c1d61b2c72df49afa0718d0de76da1
