This change fixes data races in tests for the account freeze chore
caused by the chore service being set while in use.
Change-Id: Ibcb94a947c405314deae0d77419e69b768feab40
Once this is merged and deployed, we can go ahead
and change the db schema.
Note that this will stop updating the `type` column in the nodes
table but it shouldn't be a problem because this column has a
default value set.
Updates https://github.com/storj/storj/issues/5426
Change-Id: I2470ebacbcb0e60cf894617eb69b593227357283
This change adds a new column to the account_freeze_events table,
days_till_escalation, which is the number of days till the freeze event
is escalated. E.g.: 30 days from billing warning event to billing
freeze event.
Issue: #6382
Change-Id: I15cfd6f1208e641d8c380bef61717031befdad73
To get better performance, we pre-load all nodealias/node information at the beginning of the segment loop.
It's possible that we receive a new node alias from the segment table what we are not fully aware of (yet).
The easiest solution is just ignoring. New risks/threats can be detected by a new execution cycle.
Change-Id: Ib54f7edc46eedbab6d13b4d651aaac1425994940
Added client side logic to disable billing features depending on config value.
Disabled billing endpoints if billing is disabled.
Issue:
https://github.com/storj/storj-private/issues/464
Change-Id: I6e70dc5e2372953b613ddab9f19cb94f008935ce
The overlay.NodeDossier struct only tracks information about a
storagenode, the field is deprecated and no longer needed.
This is a kademlia left-over.
Updates https://github.com/storj/storj/issues/5426
Change-Id: Ie278ffd88d1b9a9fde6c81eb5f0e287bab8c9ef0
`overlay.NodeDossier` contains information that satellite tracks
about a storagenode.
As part of the work to remove the type column from the nodes
table https://github.com/storj/storj/issues/5426, we would have
to remove the `Type` field in the `overlay.NodeDossier` since it
is only about the storagenode.
The `Local` method on the contact service is also removed because
it is unused. When needed we can create a seperate `SatelliteInfo`
struct which the `Local` method will return.
Change-Id: If0c1a25d9df397a9492bbf1d7f33ba5b6a918878
This change limits payment attempts to
1. Card updates when billing frozen/warned
2. Right before billing freezing a warned account.
Issue: https://github.com/storj/storj-private/issues/457
Change-Id: Ic6d5c649cdac38d5c3b7365e20a4ceb3b6199ee8
This change adds a flag to the satellite config indicating whether
unregistered email addresses should receive project invitation emails.
Change-Id: I0396f25574ddae3f9adaea32a6e7cd15b931bf12
This change prevents multiple project invitation records from being
created from a single API request.
Change-Id: I01268fcc0e2f7b5f24870b032cb53f03c7ad0800
While the index shouldn't be necessary as long as our implementation is
correct, it still provides some additional checks for mistakes in the
implementation.
Change-Id: I7ed71ac99a979e375d7f94c8898e6f83ac623cb6
By using a separate function for deleting the latest object and
fetching the latest version we can simplify some of the code.
However, there can be more performant approaches, such as using
ON CONFLICT for updating the existing object or using select and delete
in the same query in databases that support it.
Change-Id: I52bc3f9fa025f44d05ee010723ffb81f5bd2a2d7
Allow user to download detailed usage report from Billing -> Overview screen.
Report is a CSV file containing usage data for all the projects user owns.
Issue:
https://github.com/storj/storj/issues/6154
Change-Id: I3109002bf37b1313652a2be3447aaa7bc6204887
There are several different object types in a versioned table,
which will determine the exact behaviour.
The object type states are:
* Pending - the object is yet to be committed and is being uploaded.
* Committed - the object has been finished and can be read.
* DeleteMarker - indicates that the object should be treated as not
present when is at the top of the version stack.
There are also versioning states:
* Unversioned - only one unversioned object is allowed per object key.
* Versioned - multiple objects with the same key are allowed.
Change-Id: I65dfa781e8da253a4e5d572b799d53c351196eee
I believe that this change in semantics won't break anything, because
ListVerifySegments is only used by cmd/tools/segment-verify (which only
needs to operate on non-expired segments) and various tests, none of
which expect ListVerifySegments to include expired segments.
Change-Id: I037f43b16bc5750ed914bc32949418e001df1a8c
This change adds user ID rate limiting to the endpoint responsible for
sending project invitations.
Resolves storj-private#462
Change-Id: Icf0be7d7bb7f2765725ba3e152a2195bc02484e2
ClassifySegmentPieces uses custom set implementation instead map.
Side note, for custom set implementation I also checked int8 bit set but
it didn't give better performance so I used simpler implementation.
Benchmark results (compared against part 2 optimization change):
name old time/op new time/op delta
RemoteSegment/healthy_segment-8 21.7µs ± 8% 15.4µs ±16% -29.38% (p=0.008 n=5+5)
name old alloc/op new alloc/op delta
RemoteSegment/healthy_segment-8 7.41kB ± 0% 1.87kB ± 0% -74.83% (p=0.000 n=5+4)
name old allocs/op new allocs/op delta
RemoteSegment/healthy_segment-8 150 ± 0% 130 ± 0% -13.33% (p=0.008 n=5+5)
Change-Id: I21feca9ec6ac0a2558ac5ce8894451c54f69e52d
Given the placement.txt with the content:
```
9:exclude(placement(10))
10:country("DE")
```
Without patch:
```
placement-test --placement=/tmp/placement.txt countrycode=DE
Error: type mismatch: reflect: Call using zero Value argument
```
With the patch:
```
placement-test --placement=/tmp/placement.txt countrycode=DE
Error: Error in line 'exclude(placement(10))' when placement rule is parsed: Placement 10 is referenced before defined. Please define it first!
```
Change-Id: I9ad81016d4a57fdb32f3ff9031b5591f9a7cd2a6
It would appear that we have been making concurrent accesses to
statsCollector for a long, long time (we expect there to be multiple
calls to `Repair()` at the same time on the same instance of
`SegmentRepairer`, up to `config.MaxRepair`, and before this change
there was no sort of synchronization guarding accesses to the
`statsCollector.stats` map.
Refs: https://github.com/storj/storj/issues/6402
Change-Id: I5bcdd13c88913a8d66f6dd906c9037c588960cc9
This change incorporates the UI type (legacy or Vuetify) in the
properties attached to analytics events originating from the satellite
UI.
Resolves#6363
Change-Id: Ie3627bc24e4349407376e28460a5a830d211b47b
This change adds two new admin endpoints to freeze users for ToS
violation and to remove them from that state,
Issue: https://github.com/storj/storj-private/issues/386
Change-Id: I49c922377c9cdb315ce2777fcd35dcad432b0539
Optimizing collecting monkit metrics:
* initialize metrics once at the begining
* avoid using string in map for getting stats structs per redundancy
Benchmark results (compared against part 1 optimization change):
name old time/op new time/op delta
RemoteSegment/Cockroach/healthy_segment-8 31.4µs ± 6% 21.7µs ± 8% -30.73% (p=0.008 n=5+5)
name old alloc/op new alloc/op delta
RemoteSegment/healthy_segment-8 10.2kB ± 0% 7.4kB ± 0% -27.03% (p=0.008 n=5+5)
name old allocs/op new allocs/op delta
RemoteSegment/healthy_segment-8 250 ± 0% 150 ± 0% -40.00% (p=0.008 n=5+5)
Change-Id: Ie09476eb469a4d6c09e52550c8ba92b3b4b34271
This change extends the account freeze functionality account for
violation freezes as well.
Also, debug level logs in the freeze chore have been changed to info.
It adds an analytics event for when an invoice is found that belongs to
a user frozen for violation.
And finally adds whether a user is frozen for violation to the
/account/freezestatus response.
Issue: https://github.com/storj/storj-private/issues/386
Change-Id: Id8e40282dc8fd8f242da52791ab8ddbbef3da2bc
Account freeze excludes accounts who have invoices which are unpaid and
are above a configured value. This change updates that configured value
from $100 to $1000.
Change-Id: Iff381deeb73ef2d8fccaf7a1612a33e8aeb08698
Add billing commands to manually complete or fail storjscan invoice
payments that are stuck in a pending state.
Change-Id: Ia19f0a2597201d9d17aad0889eaedff095d706b9
The API generators rely on the Name and Prefix fields of the
EndpointGroup type to generate code.
Conventional naming code requires using upper or lower case for types,
functions, etc, however requiring the user to set this fields with the
correct casing seems cumbersome for them because they can be adjusted
depending where those values are used on the generated code.
This commit lifts the restriction for the user and adjust the casing of
them according to where they are used.
Change-Id: I700a879d13b4789b4d6ba0519b4d7508061eac73
the existing test was missing a case where deleting the geofencing
should set the placement back to default.
this also changes EveryCountry to DefaultPlacement, as the former is
deprecated.
Change-Id: I29f4f1c3ae1d05e8f5ecaa03a70ac4f17574a475
deleteGeofenceForProject wasn't able to work correctly, because
Console().Projects().Update() declines to update default_placement when
the input value is 0.
This introduces a Console().Projects().UpdateDefaultPlacement() method,
congruent to the method of the same name on Console().Users().
deleteGeofenceForProject now uses this new method, so that specifying a
new placement of 0 will work correctly.
Change-Id: I4589b36707f7e4f1cfdc66543520b0d4205c1a84
This column is no longer used, but it is referenced in dbx. This change
removes those references and adds an exception to the migration test to
disregard this column in schema comparison.
After this change is deployed, we will need to follow up with a
migration to remove the column in a later release, and remove the
exception in the migration test.
Related to https://github.com/storj/storj/issues/5432
Change-Id: I168fb57244b347901d1ed9c7813c1338554ee644
Rename the Endpoint fields MethodName and RequestName because they were
confusing for what they are used.
This commit also adds some validations for these fields values and other
validations for Endpoint and EndpointGroup to avoid generating invalid
code.
It also include some tests for these new validations.
Closes https://github.com/storj/storj/issues/6333
Change-Id: Iaabfc33935517889e3729c8b37be51a55eea366c
this fixes cases where it's possible to update a user and the
DefaultPlacement field gets overwritten to the zero value.
it also adds UpdateDefaultPlacement which can be used to set
DefaultPlacement directly. This is needed for the geofencing
endpoints in satellite admin to set the DefaultPlacement back
to zero to delete geofencing for a user.
Change-Id: If2c798dabfa6773ed6023fb8257bf00ec7bc2e68
NodeSelection struct is used to make decisions (and assertions) related to node selection.
Usually we don't use email and wallet for placement decision, as they are not reliable.
But there are cases, when we know that the email address is confirmed. Also, it can be used for upper-bound estimations (if same wallet is used for too many pieces in a segment, it's a sign of a risk, even if not all the risks can be detected with this approach, as one owner can use different wallets).
Long story short: let's put wallet and email to the SelectedNode.
Change-Id: I922185e3769d43eb7762b8d60d88ecd3d50991bb
These test cases are the parts of the testplan for the Graceful Exit
Revamp which are automateable but not yet automated.
I'm not entirely sure why we have to reject graceful exit from nodes
that are suspended, but implementing that was probably easier than
convincing everybody that it's not necessary.
Refs: https://github.com/storj/storj/issues/6369
Change-Id: I0261b37f7e010d72d84332cde5dd8689f7c41580
Currently, pending invoice payments that are made using a users token
balance can get stuck in a pending state if the invoice is not able
to be paid appropriately in stripe. This change addresses these stuck
token invoice payments by attempting to transition them to failed
if the invoice cannot be paid.
Change-Id: I2b70a11c97ae5c733d05c918a1082e85bb7f73f3
This change modifies wallet payments endpoints to return empty lists
instead of returning a 404 error if a wallet is not found for a user.
Change-Id: Ic765fecbc8183d14f179ce1d510ae512d8e0c4a9
Current bucket_metainfos table schema was far from optimal. All
operations are using project_id and name to find bucket but primary key
was on id field which is completely not used. Turns out that secondary
index was not as fast as it could be because each select was also joining
primary index which was slowing down whole query.
Primary key is changed to project_id/name. Other indexes are dropped
and id column is now nullable which gives us space for dropping it
completely from schema and code later.
Change-Id: I477056a4243d39e39489f1473de01ded1ed24bbb
This change updates the version of the stripe-go dependency from v73 to
v75 in order to improve performance (see stripe/stripe-go#1728).
Resolves#6287
Change-Id: I727ac08dcaa90d7138ba30d907711cc44daf0b7a
This change updates the version of the stripe-go dependency from v72 to
v73. This is part of a process to reach v75, which contains performance
improvements.
References #6287
Change-Id: I95f132378fe05e506f0388a0fe039cb1c2db58e2
Removed code which was responsible for tracking regular user activity in hubspot.
The only direct event we send to Hubspot is 'user created'.
Issue:
https://github.com/storj/storj-private/issues/441
Change-Id: Ia6da30374f91369372af8fe5b5990ec5672fc03b
Fix the API generator to generate valid TypeScript code when using
slices an arrays of any type (base types, struct types, anonymous struct
types, etc.).
Closes https://github.com/storj/storj/issues/6323
Change-Id: I580ae5305c58f65c2e4f4a35d14ca4ee509a9250
With the upcoming versioning changes `BeginObjectExactVersion` makes
only sense for testing. Currently this does not rename the options
struct or move it into `metabasetest`, because it would create a
significant amount of merge/rebase noise.
Change-Id: Iafa2f81a05ae66320bc6a839828217ec94c63e1f
To live count observer is emitting metric with number of segments
already processed. It's handy to see current progress.
Change-Id: Id6766b5ec9b05bce8f6376e9afcae91fcf7df6fd
This change introduces a new freeze event, ViolationFreeze, for ToS
violations and also prepends other events with Billing to clarify what
they signify.
Issue: https://github.com/storj/storj-private/issues/386
Change-Id: Ieef2abbbb03d06377dc7a73ba5ef0742ada75e8e
The yarn.lock file came from the former repository where the back office
UI was developed.
It seems that our build process complains about some dirty state related
to this file. Because we don't use Yarn, we delete the file, hoping to
resolve the build issues.
Change-Id: I5febd8292657289d0fc67e08151c6c8b5ac8b5dc
This change introduces a new user status, PendingDeletion to be used to
mark users before they're actually deleted. It also skips users with
this status or Deleted status when generating invoices.
Issue: https://github.com/storj/storj/issues/6302
Change-Id: I6a80d0ed1fe4f223ae00e0961f18f2f62f9b5213
This change removes the project name and inviter name from the
registration page's invitation text in order to prevent phishing
attempts using these values from succeeding.
Resolves storj-private#431
Change-Id: I08636d712b6b273d484cf0594d395c9d7c02ebfa
This fixes an inconsistency with error returned on copy and move
endpoints to match other endpoints. validateAuth() is already
wrapping the RPC status around the error, so this shouldn't be
doing it again.
This also ensures that rate limit errors for FinishCopyObject and
FinishMoveObject are correctly returned as rpcstatus.ResourceExhausted
so uplink can correctly map these to uplink.ErrTooManyRequests.
Change-Id: I6bf6185b456d6774b99d56cf3d7d8f8aa2afa0e8
The satellite admin API endpoint responsible for returning project
limits now includes the burst limit in its responses.
Resolves#6276
Change-Id: Ibb3f1fdebf2f9ffd62de2d7e7a60d978c25bb22a
This patch finishes the placement aware repair.
We already introduced the parameters to select only the jobs for specific placements, the remaining part is just to configure the exclude/include rules. + a full e2e unit test.
Change-Id: I223ba84e8ab7481a53e5a444596c7a5ae51573c5
This method is sometimes ends with transaction error. Most probably
because it's trying to do full table scan on nodes table which is
heavily used. Adding AOST should help with DB contention.
Change-Id: Ibd4358d28dc26922b60c6b30862f20e7c0662cd1
When the new back office UI sources where copied from former repository
I didn't realize that the .gitignore had the package-lock.json file.
This commit remove the package-lock.json file, so it can be tracked, in
order to have reproducible builds.
The lack of the file caused the build to fail due to `npm ci` requires
it.
Change-Id: Ibe493d0cd5762afe5caabe9b77a333fd6daa5373
Currently, graceful exit is a complicated subsystem that keeps a queue
of all pieces expected to be on a node, and asks the node to transfer
those pieces to other nodes one by one. The complexity of the system
has, unfortunately, led to numerous bugs and unexpected behaviors.
We have decided to remove this entire subsystem and restructure graceful
exit as follows:
* Nodes will signal their intent to exit gracefully
* The satellite will not send any new pieces to gracefully exiting nodes
* Pieces on gracefully exiting nodes will be considered by the repair
subsystem as "retrievable but unhealthy". They will be repaired off of
the exiting node as needed.
* After one month (with an appropriately high online score), the node
will be considered exited, and held amounts for the node will be
released. The repair worker will continue to fetch pieces from the
node as long as the node stays online.
* If, at the end of the month, a node's online score is below a certain
threshold, its graceful exit will fail.
Refs: https://github.com/storj/storj/issues/6042
Change-Id: I52d4e07a4198e9cb2adf5e6cee2cb64d6f9f426b
We set lifecyclestage in Hubspot without Segment now. If Segment tries
to set lifecyclestage, it interferes with the desired behavior in
Hubspot.
Change-Id: I817c0324ecc69529d8ca7f617cb97d2f4e84aee8
Serve the front-end sources of the new back-office through the current
satellite admin server under the path `/back-office`.
The front-end is served in the same way than the current one, which is
through an indicated directory path with a configuration parameter or
embed in the binary when that configuration parameter is empty.
The commit also slightly changes the test that checks serving these
static assets for not targeting the empty file in the build folder.
build folders must remain because of the embed directive.
Change-Id: I3c5af6b75ec944722dbdc4c560d0e7d907a205b8
The repair checker and repair worker both need to determine which pieces
are healthy, which are retrievable, and which should be replaced, but
they have been doing it in different ways in different code, which has
been the cause of bugs. The same term could have very similar but subtly
different meanings between the two, causing much confusion.
With this change, the piece- and node-classification logic is
consolidated into one place within the satellite/repair package, so that
both subsystems can use it. This ought to make decision-making code more
concise and more readable.
The consolidated classification logic has been expanded to create more
sets, so that the decision-making code does not need to do as much
precalculation. It should now be clearer in comments and code that a
piece can belong to multiple sets arbitrarily (except where the
definition of the sets makes this logically impossible), and what the
precise meaning of each set is. These sets include Missing, Suspended,
Clumped, OutOfPlacement, InExcludedCountry, ForcingRepair,
UnhealthyRetrievable, Unhealthy, Retrievable, and Healthy.
Some other side effects of this change:
* CreatePutRepairOrderLimits no longer needs to special-case excluded
countries; it can just create as many order limits as requested (by
way of len(newNodes)).
* The repair checker will now queue a segment for repair when there are
any pieces out of placement. The code calls this "forcing a repair".
* The checker.ReliabilityCache is now accessed by way of a GetNodes()
function similar to the one on the overlay. The classification methods
like MissingPieces(), OutOfPlacementPieces(), and
PiecesNodesLastNetsInOrder() are removed in favor of the
classification logic in satellite/repair/classification.go. This
means the reliability cache no longer needs access to the placement
rules or excluded countries list.
Change-Id: I105109fb94ee126952f07d747c6e11131164fadb
Add the front-end sources of the new back-office.
The front-end doesn't have any business logic, it only has the pages and
the components, so it's purely UI.
The front-end was developed in a separate repository until was
completed.
Change-Id: I382e50789d6b929a67b8a0b887563ef48cb1473d
When we do `satellite run api --placement '...'`, the placement rules are not parsed well.
The problem is based on `viper.AllSettings()`, and the main logic is sg. like this (from a new unit test):
```
r := ConfigurablePlacementRule{}
err := r.Set(p)
require.NoError(t, err)
serialized := r.String()
r2 := ConfigurablePlacementRule{}
err = r2.Set(serialized)
require.NoError(t, err)
require.Equal(t, p, r2.String())
```
All settings evaluates the placement rules in `ConfigurablePlacementRules` and stores the string representation.
The problem is that we don't have proper `String()` implementation (it prints out the structs instead of the original definition.
There are two main solutions for this problem:
1. We can fix the `String()`. When we parse a placement rule, the `String()` method should print out the original definition
2. We can switch to use pure string as configuration parameter, and parse the rules only when required.
I feel that 1 is error prone, we can do it (and in this patch I added a lot of `String()` implementations, but it's hard to be sure that our `String()` logic is inline with the parsing logic.
Therefore I decided to make the configuration value of the placements a string (or a wrapper around string).
That's the main reason why this patch seems to be big, as I updated all the usages.
But the main part is in beginning of the `placement.go` (configuration parsing is not a pflag.Value implementation any more, but a separated step).
And `filter.go`, (a few more String implementation for filters.
https://github.com/storj/storj/issues/6248
Change-Id: I47c762d3514342b76a2e85683b1c891502a0756a
send analytics event if project invite link is clicked and if user
signs up.
github issue: https://github.com/storj/storj/issues/5190
Change-Id: I41eee5e679a84b9ec325815655684a98624d5656
Currently each testplanet test is running ranged loop no matter if
it's used or not. This is small change with some benefits like:
* saves some cpu cycles
* less log entries
* ranged loop won't interfere with other systems
Change have no big impact on tests execration but I believe it's nice to
have.
Change-Id: I731846bf625cac47ed4f3ca3bc1d1a4659bdcce8
There's only one value that BeginExactObject should return and there's
no point in restating that.
Also, use a clearer value for the random object version.
Change-Id: I06b26ad87d64e1b04b48458f624edd630f7f2f1d
This change adds an alternate MFA code recovery endpoint that requires
MFA code to generate codes.
Issue: https://github.com/storj/storj-private/issues/433
Change-Id: I10d922e9ad1ace4300d4bcfea7f48494227f1ff8
We stoped returning lots of errors as is to avoid leaking our internals
but some errors were meanigful for client. Example of such error is
"exceeded maximum number of parts". With this change we are wrapping
some important commit object errors with new ErrFailedPrecondition
error to be able to return it easily to uplink.
Change-Id: Id834b78362ed1920f0c3f6f1c7d9587bfd27e36a
With pending_objects table support enabled we missed passing correctly
expiration time from pending object to committed object. This change
updates commit query to take into account expiration time.
Change-Id: I67146d5b2f7f0bda02925d16275fbc59acb705bd
Merged bandwidth graph lines to show only allocated-dead for last 3 days and settled for other days.
Issue:
https://github.com/storj/storj/issues/6072
Change-Id: Ic7f03d22ccd82d27ae6e6a85e73e144c9852e33b
add descriptions for the endpoint that removes a user from the waning
state.
Issue: https://github.com/storj/storj/issues/6118
Change-Id: I211cd3c41c7fefa295d0db1b9f43f53e33b984e6
To avoid enabling feature for every project at once we would like to
do this partially and control percentage of projects that will have
feature enabled.
https://github.com/storj/storj/issues/6258
Change-Id: Iaac7c42d39da76ed2ecc439847c3b210462befa5
Currently it wasn't quite clear what was a stub version and an actual
version. Use a PendingVersion constant to make this distinction clear.
Also use PendingVersion = NextVersion = 0, that way it's clearer that
the version hasn't been yet determined. DefaultVersion = 1 might imply
that the object will get that version once commited, however that will
entirely depend on whether use-pending-objects is used or versioning is
enabled or not.
Change-Id: I21398141f97035c48c778f23b542266b834c44f1
API responses containing project information now contain the edge
service URL overrides configured for that project. The overrides are
based on the project's default placement.
References #6188
Change-Id: Ifc3dc74e75c0f5daf0419ac3be184415c65b202e
Protobuf definition is read to support getting specific version of
object so we just need to wire requested version into metainfo.GetObject
endpoint.
https://github.com/storj/storj/issues/6221
Change-Id: If4568b82119a6c893788a0a86e598b05ff5951cf
