Added new functionality to query storjscan for all wallet transactions (including pending).
Added new endpoint to query all wallet transactions.
Issue:
https://github.com/storj/storj/issues/5978
Change-Id: Id15fddfc9c95efcaa32aa21403cb177f9297e1ab
Added new observer for billing chore to check user's balance and upgrade their account if balance is more than or equal to needed amount for upgrade.
Added new config value which stands for needed amount of base units of US micro dollars needed to upgrade user.
Issue:
https://github.com/storj/storj/issues/5978
Change-Id: Ic3992cd3114397bfdd9e231ca090ff21ca66648b
This change adds an extra step to the auto freeze chore to attempt
payment before freezing/warning a user.
It also attempts payment after modifying user's cards whether the user
is frozen/warned or not.
Issue: https://github.com/storj/storj-private/issues/341
Change-Id: Ia9c0c5a2d37837bca5153fe720fef61f1385cb15
This feature flag was disabled by default to test it slowly. Its enabled
for some time on one production satellite and test satellites without
any issue. We can enable it by default in code.
Change-Id: If9c36895bbbea12bd4aefa30cb4df912e1729e4c
We are doing full bucket name validation for many requests but
we should do this only while creating bucket. Other requests will be
covered only by basic name length validation. Less strict validation for
other requests will make bucket usable in case of invalid bucket names
in DB (we have such cases from the past).
https://github.com/storj/storj/issues/6044
Change-Id: I3a41050e3637787f788705ef15b5dc4df4d01fc6
Sometimes DownloadSelectionCache doesn't keep up with all node
placement changes we are doing during this test.
Change-Id: Idbda6511e3324b560cee3be85f980bf8d5b9b7ef
This change adds request IDs to requests, logs them as part of audit
logs and sends to the client on error. This is to improve debugging
of customer issues.
Issue: https://github.com/storj/storj/issues/5898
Change-Id: I801514b547d28d810552d91aa7c8502051e552bf
Before this change, if a user creates a bucket with a user_agent attributed then deletes and recreates it, the row in bucket_metainfos
will not have the user_agent. This is because we skip setting the field
in bucket_metainfos if the bucket already exists in value_attributions.
This can be problematic, as we return the bucket's user agent during the
ListBuckets operation, and the client may be expecting this value to be
populated.
This change ensures the bucket table user_agent is set when (re)creating a bucket. To avoid decreasing BeginObject performance, which also
updates attribution, a flag has been added to determine whether to
make sure the buckets table is updated: `forceBucketUpdate`.
Change-Id: Iada2f233b327b292ad9f98c73ea76a1b0113c926
Segments loop implementation is using lots of memory to convert
alias pieces to pieces for each segment while iteration. To improve
situation this change is reusing Pieces between batch pages. This
should signifcantly reduce memory usage for ranged loop executions.
Change-Id: I469188779908facb19ad85c6bb7bc3657111cc9a
This change creates the ability to run a server separate from the
console web server to serve the front end app. You can run it with
`satellite run ui`. Since there are now potentially two servers instead
of one, the UI server has the option to act as a reverse proxy to the
api server for local development by setting `--console.address` to the
console backend address and `--console.backend-reverse-proxy` to the
console backend's http url. Also, a feature flag has been implemented
on the api server to retain the ability to serve the front end app. It
is toggled with `--console.frontend-enable`.
github issue: https://github.com/storj/storj/issues/5843
Change-Id: I0d30451a20636e3184110dbe28c8a2a8a9505804
* Handle failed country code conversion.
* Avoid potential issues with a data-race due to shared slice.
Updates #6028
Change-Id: If7beef2619abd084e1f4109de2d323f834a6090a
Having separate process/pod only for sending bloom filters once a week
is a bit waste. After reconfiguring production settings to use sender in
core we can remove also GC sender peer code.
https://github.com/storj/storj/issues/5979
Change-Id: I6efe3ec073f96545e1f70ad13843f8ccdf923ee8
* Fixes backend to use only a user's owned projects to determine if the
user has hit the project limit
* Makes frontend logic consistent (and simpler) for checking whether to
send user to the "Create Project" modal or the "upgrade account or
request limit increase" modal
Before this change, projects that a user is a member of would be
included in determining whether the user could create a project. Also,
the "create project" button in the projects menu in the navbar of the UI
did not enable a free tier user to create a new project, even if they
had not hit their limits.
Change-Id: Ia776eb627ca37b83f5bc63bed83ee83c9f7cc789
Add a configuration (default true) to exclude users who have made
storjscan payments from being auto-warned/frozen for an unpaid invoice.
This will allow us to reach out to these users and handle warning/freezing
manually. Auto account freeze still handles CC-only users.
Fixes https://github.com/storj/storj/issues/6027
Change-Id: I0c862785dad1c8febfa11100c0d30e621ce3ae9b
This change fixes an issue where a new project member invitation would
silently replace an older one that has the same project ID and email if
the email did not belong to a registered user. Additionally, the
satellite frontend has been updated to display more descriptive error
messages for project member invitations.
Change-Id: I32b582c40c0028b8eedf2aed4b5bfb43501594b4
Currently, we have issue were while counting unhealthy pieces we are
counting twice piece which is in excluded country and is outside segment
placement. This can cause unnecessary repair.
This change is also doing another step to move RepairExcludedCountryCodes
from overlay config into repair package.
Change-Id: I3692f6e0ddb9982af925db42be23d644aec1963f
placement.AllowedCountry is the old way to specify placement, with the new approach we can use a more generic (dynamic method), which can check full node information instead of just the country code.
The 90% of this patch is just search and replace:
* we need to use NodeFilters instead of placement.AllowedCountry
* which means, we need an initialized PlacementRules available everywhere
* which means we need to configure the placement rules
The remaining 10% is the placement.go, where we introduced a new type of configuration (lightweight expression language) to define any kind of placement without code change.
Change-Id: Ie644b0b1840871b0e6bbcf80c6b50a947503d7df
Updated upgrade account modal to show user account free tier limits instead of hardcoded values.
Issue:
https://github.com/storj/storj/issues/5939
Change-Id: I26ffbe2571c5ca4b37f02bec5211bac986bedc6a
All the files in uploadselection are (in fact) related to generic node selection, and used not only for upload,
but for download, repair, etc...
Change-Id: Ie4098318a6f8f0bbf672d432761e87047d3762ab
..instead of using segment_copies and ancestor_stream_id, etc.
This bypasses reference counting entirely, depending on our mark+sweep+
bloomfilter garbage collection strategy to get rid of pieces once they
are no longer part of a segment.
This is only safe to do after we have stopped passing delete requests on
to storage nodes.
Refs: https://github.com/storj/storj/issues/5889
Change-Id: I37bdcffaa752f84fd85045235d6875b3526b5ecc
ReliabilityCache will be now using refactored overlay Reliable method.
This method will provide more info about nodes (e.g. country code) and
with this we are able to add two dedicated methods to classify pieces:
* OutOfPlacementPieces
* PiecesNodesLastNetsInOrder
With those new method we will fix issue where offline but reliable node
won't be checked for clumped pieces and off placement pieces.
https://github.com/storj/storj/issues/5998
Change-Id: I9ffbed9f07f4881c9db3bd0e5f0412f1a418dd82
Currently we are using Reliable to get missing pieces for repair
checker. The issue is that now checker is looking at more things than
just missing pieces (clumped/off, placement pieces) and using only node
ID is not enough. We have issue where we are skipping offline nodes from
clumped and off placement pieces check.
Reliable was refactored to get data (e.g. country, lastNet) about all
reliable nodes. List is split into online and offline. This data will be
cached for quick use by repair checker. It will be also possible to
check nodes metadata like country code or lastNet.
We are also slowly moving `RepairExcludedCountryCodes` config from
overlay to repair which makes more sens for it.
This this first part of changes.
https://github.com/storj/storj/issues/5998
Change-Id: If534342488c0e440affc2894a8fbda6507b8959d
Currently, any attempt to list the api keys associated with a project
from the admin UI results in a 404 NOT FOUND error.
This appears to be because there is no /api/projects/{project}/apiKeys
endpoint registered; it should have a lowercase k.
Change-Id: Ifbe4cd0f9ba12a6e37a0d9f64df91c264ced5558
This change properly encodes email addresses that are used as query
parameters in project invitation-related URLs.
Change-Id: Iaaf7b62b5ac3db3f0b0e000cc06fef8e315400a8
We use two different Node types in `overlay` and `uploadnodeselection` and converting back and forth.
Using the same object would allow us to use a unified node selection interface everywhere.
Change-Id: Ie71e29d60184ee0e5b4547eb54325f09c418f73c
Added functionality to return only settled traffic from project_bandwidth_daily_rollups table for given month.
Updated {projectID}/usage-limits endpoint to return only settled bandwidth used.
This is a possible fix for this issue
https://github.com/storj/storj-private/issues/293
Change-Id: I12516dc898f449c2122e7442b8fbb88309a48ebe
In case some invalid characters in bucket name we need to cast
bucket name to byte array for query argument. This change is
doing this for some missed cases.
Change-Id: I47d0d8e3c85a69bdf63de1137adcd533dcfe50a8
The console DB cleanup chore has been extended to remove expired webapp
session records.
Resolves#5893
Change-Id: I455b4933552cfde86817a2ef8f9879dd7b0a121d
This change allows members without an account to be invited to a
project. The link in the invitation email will redirect these users to
the registration page containing custom text describing the invitation.
Resolves#5353
Change-Id: I6cba91e57c551ca13c7a9ae49150fc1d374cd6b5
At the moment segment repairer is skipping offline nodes in checks like
clumped pieces and off placement pieces. This change is fixing this
problem using new version of KnownReliable method. New method is
returning both online and offline nodes. Provided data can be used to
find clumped and off placement pieces.
We are not using DownloadSelectionCache anymore with segment repairer.
https://github.com/storj/storj/issues/5998
Change-Id: I236a1926e21f13df4cdedc91130352d37ff97e18
The upgrade notification has been updated to adapt to mobile screens
accordance with our designs.
Additionally, an issue where the notification would display "0B free
included" when displayed in the All Projects Dashboard has been fixed.
Change-Id: Ic13b9426ab5d6529c9d7b2ad8446a17da74905b1
By mistake AOST was added to query in deleteInactiveObjectsAndSegments
in DeleteZombieObjects. Delete statement is not supporting it.
Unfortunately unit tests didn't cover this case. This change removes
AOST from mentioned method and it adding AOST cases to unit tests.
Change-Id: Ib7f65134290df08c490c96b7e367d12f497a3373
This change further restricts projects members from modifying project
details by restricting the project edit graphql mutation; making it
check if the user performing the operation is the owner of the project.
Change-Id: Iaf10d16269ddc29437d3d5629db06e20cea3004e
Currently we are using KnownUnreliableOrOffline to get missing pieces
for segment repairer (GetMissingPieces). The issue is that now repairer
is looking at more things than just missing pieces (clumped/off
placement pieces).
KnownReliable was refactored to get data (e.g. country, lastNet) about
all reliable nodes from provided list. List is split into online and
offline. This way we will be able to use results from this method to all
checks: missing pieces, clumped pieces, out of placement pieces.
This this first part of changes to handle different kind of pieces in
segment repairer.
https://github.com/storj/storj/issues/5998
Change-Id: I6cbaf59cff9d6c4346ace75bb814ccd985c0e43e
This change adds row actions (delete,reinvite,copy) to the project
members table. It also adds analytics events for the actions.
Issue: #5762
Also fixes: #5941
Change-Id: I7fb7f88c7bd5ac2ce3e3d00530af4708ff220bd7
A button has been added to the Team page for resending expired project
member invitations. It appears when one or more of such invitations
have been selected.
Additionally, styling for certain search fields and the Team page's
header has been updated to align more closely with our designs.
Resolves#5752
Change-Id: I623fed5f50e60beca2f82136f8771dde5aa684f4
This change prevents project member invitation responses from deleting
expired project invitations. Previously, accepting or declining an
expired invitation cause it to be destroyed.
References #5752
Change-Id: Id3917fb825bffc3e8a262d5b541b907678db1809
This reverts 9c75316 which allowed the satellite console DB cleanup
chore to delete expired project member invitations. We now want such
invitations to be accessible indefinitely.
References #5752
Change-Id: I489a7e19df825dd14376d3d260b70b3eef643e03
The SQL transaction that inserted project invitations relied on the
error result of one of its statements in order to determine whether an
invitation should be updated. This was inappropriate since any errors
returned from a transaction statement should end the transaction
immediately. This change resolves that issue.
Change-Id: I354e430df293054d8583fb4faa5dc1bcf9053836
This change sends new passphrase created event for when passphrase is
created with the method by which it was; entered/generated
Issue: #5918
Change-Id: Ib485b6ff7a968d4c84bf124e14c14c91478f0dfb
Add some basic handling to set cross-origin resource sharing headers for
the satellite UI app handler as well as API endpoints used by the
satellite UI.
This change also removes some no-longer-necessary CORS functionality on
the account registration endpoint. Previously, these CORS headers were
used to enable account registration cross-origin from www.storj.io.
However, we have since removed the ability to sign up via www.storj.io.
With these changes, browsers will prevent any requests to the affected
endpoints, unless the browser is making the request from the same host
as the satellite.
see https://github.com/storj/storj-private/issues/242
Change-Id: Ifd98be4a142a2e61e26392d97242d911e051fe8a
This change registers the "Gallery View Clicked" analytics event on the
satellite's side. Previously, the event reported from the satellite
frontend would be ignored because it was not present in the satellite's
event whitelist.
References #5824
Change-Id: I636ace6a21b2738431d0c1f5adb9a16c4e177188
This reverts commit 676178299f.
Reason for revert:
The new query used by this commit performs a full table scan.
It's been reverted pending a fix for that.
Change-Id: Idc53954459aa6f5a692056232b8674b11d1928ce
Allow a longer encrypted key length to reduce 'key length is too big'
errors in gateway-mt. Gateway is enforcing an unencrypted key length
of 1024 bytes but when encrypted some keys are exceeding the current
limit.
Updates https://github.com/storj/gateway-mt/issues/335
Change-Id: Ib02e2064c42e96b9d59936905832d8dd6068d2c7
Because we are saving all tallies as a single SQL statement we finally
reached maximum message size. With this change we will call SaveTallies multiple times in batches.
https://github.com/storj/storj/issues/5977
Change-Id: I0c7dd27779b1743ede66448fb891e65c361aa3b0
When pieces fail an audit (hard fail, meaning the node acknowledged it
did not have the piece or the piece was corrupted), we will now remove
those pieces from the segment.
Previously, we did not do this, and some node operators were seeing the
same missing piece audited over and over again and losing reputation
every time.
This change will include both verification and reverification audits. It
will also apply to pieces found to be bad during repair, if
repair-to-reputation reporting is enabled.
Change-Id: I0ca7af7e3fecdc0aebbd34fee4be3a0eab53f4f7
Additional test case to cover situation where we are trying to
repair segment with specific placement set. We need to be sure
that segment won't be repaired into nodes that are outside
segment placement, even if that means that repair will fail.
Change-Id: I99d238aa9d9b9606eaf89cd1cf587a2585faee91
the field was not being set, so it would always show false in the
response whether the user was paid tier or not.
Updates storj/gateway-mt#321
Change-Id: I00ae751ac9118fc05b3c22bc6d2cb9600d0faaaf
Some of tally queries are not passing bucket name as byte but as string.
If bucket contains some special characters encoding to bytea can fail.
This change makes sure all parts of tally passes bucket name correctly.
Change-Id: I7330d546b44d86a2e4614c814580e9e5262370ed
Each project member invitation returned from our GraphQL API now
contains a field indicating whether the invitation has expired. This is
required for us to enable functionality in the satellite frontend that
is dependent on this information.
References #5752
Change-Id: I4b71738e7a7373c690de188614f8c95009bc3989
This change removes instances of project invitation deletion due to
expiration because we now want such invitations to be accessible beyond
their expiration date. In the future, project members will be able to
view and resend expired invitations within the Team page in the
satellite frontend.
References #5752
Change-Id: If24a9637945874d719b894a66c06f6e0e9805dfa
After deploying to most of production satellites we want to enable
it by default. No issues found after few weeks of running.
Change-Id: I4d83c65edaa95b0e50eeab6ac5e2c6cbb7206c1e
Implemented UI functionality to allow prod owner to update user's or project's user agent.
Issue:
https://github.com/storj/storj-private/issues/297
Change-Id: Ibe47e735f3c7a9fdf05b70735ebcecbac5c94ce8
Added backend (for now) implementation for updating user's and projects's user_agent using admin API.
Updating both user and project also updates bucket_metainfo and value_attribution tables.
Issue:
https://github.com/storj/storj-private/issues/297
Change-Id: I40244bbaa08b46834c1b1d0720e7d84d0c2a0330
This change makes dial timeout configurable and change it also from
defatul 20s to 5s. Main motivation is that during repair we often loose
lots of time to dial which eventually will fail. New timeout should be
still enough to dial but we will move forward quicker to next node if
that one will fail.
Timeout is also applied directly as context timeout in case we will
use noise of tcp fast open one day.
Change-Id: I021bf459af49b11241e314fa1a7887c81d5214ea
This change adds a new endpoint to verify the validity of an invite
link. It conditionally redirects to the login page with or without
whether the invite is valid.
Related: https://github.com/storj/storj/issues/5741
Change-Id: I587ef8ded67a9ea753e4edec1beeecd39c949922
Now that the table view has been implemented, the all projects dashboard
is ready to be turned on everywhere.
https://github.com/storj/storj/issues/5872
Change-Id: Iead684bf7d326d36d4d323eb63a3ed520602b4dc
This change addresses an issue where the /charges endpoint will take a
while to respond due to a project having a large number of buckets.
The queries responsible for this have been merged into a single query to
get all data needed at a go and potentially improve performance.
Benchmarks indicate that this new way is more performant than the
previous.
name old ms/op new ms/op
Postgres/sum_all_partner_usages 7.534 0.622
Postgres/individual_partner_usages 6.104 0.588
Cockroach/sum_all_partner_usages 14.813 3.057
Cockroach/individual_partner_usages 16.046 2.852
Issue: https://github.com/storj/storj-private/issues/277
Change-Id: Ibb7f867ab6610b3cb1ba203961f7d6aef6bfda4a
This change slightly modifies the logic for serving the vuetify
frontend. After this change, if the config `console.use-vuetify-project`
is set to `true`, the Vuetify UI will be served at the `/vuetifypoc`
prefix. The POC is only project dashboard right now; the existing
login/registration pages must be used, but once the cookie is set, the
POC will work correctly.
Change-Id: I7725f23a0d2b04f274bab36d8be3370116687d1b
We decided that we will stop sending explicit delete requests to nodes
and we will cleanup deleted with GC instead.
https://github.com/storj/storj/issues/5888
Change-Id: I65a308cca6fb17e97e3ba85eb3212584c96a32cd
This change fixes an issue where a formerly warned/frozen user will be
warned again even though they have made payment for the invoice that got
them frozen in the first place. A payment status check is now made
right before a warn/freeze event to make sure the invoice hasn't been
paid already.
Issue: https://github.com/storj/storj/issues/5931
Change-Id: I3f6ac1e224f40107d58dc8f7bdbce58bbbea0196
This change removes the obsolete project member paging code.
Previously, we implemented functionality for including project
invitations in pages of project members. However, the satellite
frontend still expected API responses to use the old paging style, so
the related code could not be removed right away. Now that the frontend
has been updated, this code is no longer necessary.
References #5855
Change-Id: I12fdaaeb869977c4d87a0d50b9a7b11c68552c82
This change adds a new endpoint that uses the new project invite flow's
functionality instead of directly adding users to a project's members.
Issue: https://github.com/storj/storj/issues/5741
Change-Id: I6734f7e95be07086387fb133d6bdfd95e47cf4d9
Added new endpoint, service method and DB query to get all API key names by provided project ID.
Issue:
https://github.com/storj/storj/issues/5693
Change-Id: I62e4e8ae660bd81234b75aa159a472a5aa9d5a48
This change causes users to be served only project member invitations
that have not expired. If expired invitations are encountered during
processing of an invitation listing request, they will be removed.
References #5855
Change-Id: I6f621305f4f0a993953eb40a4dbd2375493f02e3
Add some code to generate a basic markdown file documenting a generated
API. Generate this document for the API in
satellite/console/consoleweb/consoleapi/gen.
The documentation is not completely correct, as it may include some
values in the request body that are not actually usable by the
requester. This can be fixed by making sure all types used within the
generated API are properly annotated with `json` tags.
Issue: https://github.com/storj/storj-private/issues/244
Change-Id: I57b259967fb0db8f548b6598a10c825da15ba723
Project member invitations may now be requested through GraphQL
queries. This is necessary for the satellite frontend to display
invitations in the Team page.
References #5855
Change-Id: Ibc8526ba768fd82c1b1890201004ef0f066df2fc
The console service method responsible for removing members from a
project has been extended to remove project member invitations as well.
This will allow invitations to be deleted through the satellite
frontend.
References #5855
Change-Id: I90ca042cc6fb9a75fcd9b391e317caabb1c828f2
A method has been implemented that allows for paged searching through
project members and project member invitations. In the future, this
will be used to display invitations in the Team page of the satellite
frontend.
References #5855
Change-Id: I0937c425f60f1318e55202bf30b44a33ff695414
This change adds analytics events for accepting and rejecting project
member invitations.
References #5855
Change-Id: I2101eb711312a683ecdb4b7d0c6d7cc3ae09c440
An index has been added on the project_id column of the project_members
satellite database table so that we can retrieve members of a project
without performing a full table scan.
References #5855
Change-Id: I1cc30686f836c8fd1aa319247ce857a2392e7a52
We missed to set placement as a part of selection request. It can case
uploading repaired data out of specified placement.
I will provide test as a separate change.
Change-Id: I4efe67f2d5f545a1d70e831e5d297f0977a4eed1
Built side Vuetify subproject inside web/satellite with limited functinality.
For now it has navigation side bar, simple project dashboard and team page (where you can list/add team members).
Issue:
https://github.com/storj/storj/issues/5854
Change-Id: I9ff3e80b8ace1dc31de6a788174c5ffc19f050f8
* optimize SQL for zombie objects deletion query by reducing some direct
selects to segments table
* set AOST for expired/zombie object deletion (was 0 since now)
https://github.com/storj/storj/issues/5881
Change-Id: I50482151d056a86fe0e31678a463f413d410759d
