The Update method of the usersDB takes a console.User as an argument.
To update the columns in the DB, we have to migrate the fields from the
console.Users struct into a special dbx struct. If one of these fields
is left empty, then the zero value of that field's type will be used to
update the respective column.
In most cases where the users.Update method is called, the entire
console.User is apparently retrieved first, fields are updated, then it
is passed to users.Update. This is not the case for
service.UpdateAccount. Because these fields are not populated in the
user struct in UpdateAccount before it is passed into users.Update,
their respective columns in the database are overwritten with zero:
ProjectLimit, ProjectStorageLimit, ProjectBandwidthLimit,
ProjectSegmentLimit, PaidTier, MfaEnabled, MfaRecoveryCodes, MfaSecretKey
Solution: Do what is done in other places which call users.Update. Take
the console.User from the auth context, update the relevant fields on
that, then pass that in.
Change-Id: I3cbd560e8ea5397e5c27711fb40bb3907d987028
segment_copies table has (stream_id, ancestor_stream_id) as primary key,
but it should have only stream_id as primary as a stream_id can only have one ancestor_stream_id.
https://github.com/storj/team-metainfo/issues/84
Change-Id: I0218295e86e449c9dfefa5a21319f3083bf80afd
A user-agent string can contain multiple "products", in the case of
Gateway-MT at least this includes the HTTP client's full user agent.
This means that "other" is often logged even when we know the Storj
product, and sometimes logged more than once per call to "collect".
This makes sure that "other" is only logged if a product isn't
identified, and only logged once.
Change-Id: I8536f7eb32877e36fec97dab7b8d477ccb10f92e
For a thorough explanation of the overall transition, see the message on
commit c053bdbd70.
This change will rename the columns containing gob-encoded big.Floats
and add new columns which will contain the equivalent data in a more
sql-friendly format.
The change should *not* break already-running satellite processes,
because all functionality touching these tables has already been taught
to work with these new columns if it sees any "undefined column" errors.
Change-Id: I229324376533e383c5d05064b8aedad149cf825b
transfer-sh will be set as of https://github.com/dutchcoders/transfer.sh/pull/467.
filezilla needs to be verified and duplicati is set per info from @TopperDEL
comet and orbiter are added as preparation.
Change-Id: I44d730a7b3ba1969068e48c2477b478831799cd1
Part of server-side copy implementation.
Adds new table to metabase for keeping references between
copied segments and its ancestors.
Fixes https://github.com/storj/team-metainfo/issues/84
Change-Id: I436d4b533a3d951bcd752e222e3b721cee7f0844
This change adds some more checks to the deletion process for projects and
users, since we ran into a race condition during invoicing, where projects
have been deleted before the invoicing was finished, leading to missing
references.
This PR changes the logic to block user deletion if we are in exactly that period,
while also allowing the deletion of projects/users on free tier during the month.
Change-Id: Ic0735205e6633762fb7e3c2fa13e744cdfa5ec32
Finished implementing queries for both bandwidth and storage using pgx.Batch.
Fixed CSP styling issue.
Change-Id: I5f9e10abe8096be3115b4e1f6ed3b13f1e7232df
There is a sev-2 issue to add more browser caching.
In this PR I made object map and object preview to be fetched by signed request with non-public credentials using AWS SignatureV4 package.
Change-Id: Ib5013fa6d6af3faa97eed5168c11a13f9629cd87
Before this change we were returning full DB error message.
That can be very confusing for end user. This change is translating
error message into more user frindly version and fixes also DRPC
error status code.
Fixes https://github.com/storj/team-metainfo/issues/76
Change-Id: I29b06ab4ba50a0d14db7a822a2906d95d65ab524
We don't have different version of object than 1 so at the moment
this method is not needed. Also using GetObjectExactVersion
should be slightly more performant.
Change-Id: I78235d8ae22594cc1d6345dabcc915f41cd7797b
We already split main code base, now we need to split test
to reflect new files structure (bucket/object/segment/other).
Fixes https://github.com/storj/team-metainfo/issues/12
Change-Id: Ica1054c4fc7df764483b03f204b4beba094df8e1
We have here two migrations in fact. One is for existing users,
we need to check if its paying user (paid_tier) and set 1M for
them and 150K for others.
Second migration is to set limits for projects depends on owner.
If owner is a paying user (paid_tier) then project should have
1M limit, otherwise it should be 150K. In this case to make
migration faster initially projects table segment_limit was set
to 1M by default. With migration we are selecting all paying
users and we are setting 150K limit for all projects which owners
are not in paying users set.
Initially we had a concern if that query wil lbe quick enough to
be executed during deployment but after investigation CRDB
team confirms that this should take seconds for out DBs.
Fixes https://github.com/storj/team-metainfo/issues/70
Change-Id: I8be06e9f949b68b993e043cc15525e8483bf49ea
Existing tests are checking errors by comparing error message.
We plan to expose storage/segment limit errors as a public uplink
API but before that will happen we need to update tests to handle
both cases as new uplink error will have a bit different error message
than current tested error.
https://github.com/storj/uplink/issues/77
Change-Id: Id2706323c60d050d96752e66e859d4ec051a69b9
Implemented endpoint and query to get bandwidth chart data for new project dashboard.
Connected backend with frontend.
Storage chart data is mocked right now.
Change-Id: Ib24d28614dc74bcc31b81ee3b8aa68b9898fa87b
This is part of change to split metainfo endpoint into smaller
files. It will be grouped by bucket/object/segment/other requests.
Tests will be split as a separate set of changes.
Updates https://github.com/storj/team-metainfo/issues/12
Change-Id: I6c691e4d0e192fe3ad7974d2d0ab5ced0d272f3c
For better performance we should use AOST for getting
data where being up to data it not very crucial. We don't
care about small differences for segment limit calculation.
Change-Id: I9b2d4f2bd15ebc9d1c46bc84dd51a2e9d9231506
This is part of change to split metainfo endpoint into smaller
files. It will be grouped by bucket/object/segment/other requests.
Tests will be split as a separate set of changes.
Change-Id: I9b097dcc8fa889f985b7f4ef5f8f435a1ff0ef95
Additional test case to cover situation when cache value
expired and we need to get this value directly from DB.
Change-Id: I07bda67e19333e09a567104ce70f112fd47a7845
The database table got invalid input and the resulting error
was not checked. This adds updates that contain invalid fields
to trigger different errors.
Change-Id: Iacea32cbef5599aab562c88e4113073596cc9996
We had two problems here. First was how we were handling
errors message from GetProjectSegmentUsage. We were always
returning error, even for ErrKeyNotFound which should be used
to refresh value in cache and not propagated out of method.
Second, we were using wrong value to update cache. We used
current value from cache which obviously it's not what we intend.
Fixes https://github.com/storj/storj/issues/4389
Change-Id: I4d7ca7a1a0a119587db6a5041b44319102ef64f8
This is part of change to split metainfo endpoint into smaller
files. It will be grouped by bucket/object/segment/other requests.
Tests will be split as a separate set of changes.
Change-Id: I5128c84e06c82777fe71460bf5f9a6e26e52a243
Currently the rate limit has kept per satellite api endpoint.
Since we run 9+ api endpoints in production, we do not need
a limit of 1000, since the intention was to allow 1000 total.
This change reduces the effective limit given 9 instances
down to 900, which should be close enough.
Change-Id: Ia579149ccc3a12e8febe0cfd5586b8a39de40f55
We were returning pure non rpc errors in two cases.
This change added loging and correct rpc error as a
return.
Change-Id: I581ceb17dcdc00921dfa3c1057015c3b4d04308d
Users signing up through a url containing a promo code will have that code applied to their stripe account instead of the free tier coupon.
Change-Id: I071041b0934648ef3f5bdb05b6ec97c400f89ae4
A few months ago we removed all references to the contained
column in nodes and reputations
bb21551a9c
and
56fe636123
But we never did the migration to drop the columns.
This commit will finally do that.
Change-Id: I82aa2f257b1fb14a2f1c4c4a1589f80895360ae4
The previous change (59648dc272) ends up removing a lot of characters
for valid non-English names. Instead, only replace URL characters such
as slashes, colons, and periods. Since someone may use these characters
to separate two parts of a name, e.g. Name1/Name2, replace these
characters with a hyphen.
Change-Id: I4cc3d1bdb05d525a83970cf1b42479414c9678e7
When a user is created, but before verification or forgot password email
is sent, remove any special characters in the provided name. This
protects us against certain phishing attacks.
Change-Id: Ieddd3479da20eb80b9f1b56eb86c8f46bca2642c
We need to combine methods from accounting.Service (ExceedsStorageUsage and ExceedsSegmentUsage)
to run checks concurrently.
Resolves https://github.com/storj/team-metainfo/issues/73
Change-Id: I47831bca92457f16cfda789da89dbd460738ac97
Free-tier segment usage limit was defined as 150k, not 140k. This change
is correcting that.
https://github.com/storj/team-metainfo/issues/8
Change-Id: I71ec0961930b19fd09b2b996e01acd406a8dcf8f
Refactoring to do few things:
* move simple validation before validations with DB calls
* combine validation check/update for storage and segment
limits together
Change-Id: I6c2431ba236d4e388791d2e2d01ca7e0dd4439fc
updating project segment usage entry to redis is enabled only if
segment validation is enabled in config value, so no incompatible
changes will be added while deployment.
Change-Id: I1288cb9ff0a8a00f095dc94e20d2f14393e9a613
The following 2 commits added 2 new query parameters to set the `burst`
and `segments` limits for a project and also to new fields to the
response JSON object body to the "get project limits" endpoint:
* c911360eb5
* b7b010adc9
However, the API documentation and the Typescript client API (used by
the UI) weren't updated with them.
Later, the commit dc6128e9e2 updated the
Typescript client API with the `segments` limit but it didn't update the
documentation to reflect it.
This commit brings all things that were missed in those previous
commits.
Change-Id: Iff12cdd4a0d3c448cd73b57a98d171ba468d2c98
We want to know usage statistics for our main tools
like uplink-cli or rclone. Initially we will collect
only usage stats without relation to specific process
e.g. download or upload.
Change-Id: I203b1a6c07ae014e710368f77163f13fdf10763c
Use overlay db node as primary source of truth for reputation status when node
query it's status via Endpoint.Get. If node was was disqualified in overlay reputation
status may be inconsistend with reputation table.
Change-Id: I1bf858a4b020324035847a9215ef29cf5432f6a4
