There were some dbx-generated queries on the old satellite version that
still reference this column, so we need to add it back until the next
version.
Change-Id: I78b19336d9ca0384936d6cc11f5c50e579b4f2ab
This change includes STORJ bonuses to the list of transactions returned
by the /wallet/payments endpoint.
Issue: https://github.com/storj/storj/issues/5755
Change-Id: Icc95c2cb9dd9fc5ee7a373e68c1cf8a991e1aa58
Methods SelectAllStorageNodesUpload and SelectAllStorageNodesDownload
are not returning full info with overlay.SelectedNode because its
missing CountryCode.
Change-Id: Ie3cb396bf28d7ec4c6ab8927e5bb560236036aa6
Tallies are now created for buckets with no objects. Previously, the
bucket tally collector skipped empty buckets since it created tallies
only using information from the objects table. Methods that used
bucket tallies when calculating usage costs would return incorrect
results because of this.
Change-Id: I0b37fe7159a11cc02a51562000dad9258555d9f9
The last code referencing these columns was removed as of satellite
release v1.79, so it is safe to remove them now.
Resolves https://github.com/storj/storj/issues/5432
Change-Id: I2e9d641b2511a61e0b9482ef0f4955a73c290709
During billing, before invoice creation, check if users are part of a
package plan. If so, and if the package plan is expired, remove unused
credit from the user's balance. If the user has credit in addition to
the package credit, send an analytics event to notify someone to handle
the credit removal manually.
Change-Id: Iad71d791f67c9733f9d9e42f962c64b2780264cc
The project member invitations table has been modified to contain a
column for the ID of the user who sent the invitation. This ID is
required for us to return information about the inviter to the
satellite frontend.
References #5855
Change-Id: I928d987a8db2340f731ca65ce30173d4f90a9837
87d0789691 replaces offset usage with cursors.
But to continue the interation from a specific cursor, we need to iterate over ordered records.
(at least this is what I understood based on the failing tests)
87d0789691
Change-Id: Ic4da3a7c5f03386dd4c373c05102f05871900a3a
This change updates the replacer in satellite/satellitedb/dbx/gen/main.go
to work with an updated dbx.
Change-Id: I08e89d6d27e6f1d435416105fe5f622009add7ad
Stripe invoice project records while listing are causing full table scan
because of OFFSET caluse. This change is refactoring query to list using
cursor.
Change-Id: I6b73b9b2815173d7ef02cf615408778476eb3b7b
We have method which is getting projects owned by specific user but it's
causing full table scan because we don't have index on owner_id column.
Change-Id: Icb71c9ac5b73104a52241ed8ba126c995c10811f
The console DB cleanup chore has been extended to remove old project
member invitation records.
Resolves#5816
Change-Id: Id0a748e40f5acf03b9b903265c653b072846ba19
There is still a reference to partner_id in a query, which we cannot
move until dropping the "not null" constraint for it. This change adds
that migration.
Related to https://github.com/storj/storj/issues/5432
Change-Id: I98802a6e1bd59f3d9214de3db6688d9daf664a70
A chore responsible for purging data from the console DB has been
implemented. Currently, it removes old records for unverified user
accounts. We plan to extend this functionality to include expired
project member invitations in the future.
Resolves#5790
References #5816
Change-Id: I1f3ef62fc96c10a42a383804b3b1d2846d7813f7
This change adds the user's passphrase prompt setting to the
/account/settings endpoints.
Issue: https://github.com/storj/storj/issues/5616
Change-Id: I48d470d49e82096fd090b74da323b279e342546e
This change immplements methods for interacting with the project member
invitations table.
Resolves#5766
Change-Id: I0090c50f9fde5bcdae4ebdaa72cdcaa84d341b54
A table for storing pending project member invitations has been added.
This table is required to satisfy our new project invitation UX revamp.
References #5766
Change-Id: I6f948de66ed5b4dc81532564958ff7f48533cad2
This field is deprecated in favor of UserAgent; Removing these
references is the final step necessary before dropping the columns from
the database.
https: //github.com/storj/storj/issues/5432
Change-Id: I3a6619170dcf382f82dc8eddb73b6547eaf636f0
New SQL queries for GetProjectObjectsSegments turns out to introduce
full table scan. This is fix for this problem.
Change-Id: Ieac22aafeb780168523a97e27c9283c9ac6a24c8
Query to list (with pages) stripe customers were doing full table scan
because Offset clause was used. This refactoring changed listing to
use cursor instead Offset.
Change-Id: I14688e6c533bc932ba0d209a061562f080b4cf54
We avoid putting more than one piece of a segment on the same /24
network (or /64 for ipv6). However, it is possible for multiple pieces
of the same segment to move to the same network over time. Nodes can
change addresses, or segments could be uploaded with dev settings, etc.
We will call such pieces "clumped", as they are clumped into the same
net, and are much more likely to be lost or preserved together.
This change teaches the repair checker to recognize segments which have
clumped pieces, and put them in the repair queue. It also teaches the
repair worker to repair such segments (treating clumped pieces as
"retrievable but unhealthy"; i.e., they will be replaced on new nodes if
possible).
Refs: https://github.com/storj/storj/issues/5391
Change-Id: Iaa9e339fee8f80f4ad39895438e9f18606338908
add ability to sort api keys on the access management page by ascending
or descending name or creation date. Additionally, edit api keys query
when ordering by name to order by lower(name) so names starting with
capital letters are not treated differently from lower case when
ordering.
Change-Id: I81dbb87587a24fb7097313f76bad116b1f20d306
Add columns package_plan and purchased_package_at to stripe_customers
table and add methods to update and select these values from console
service and payments accounts.
Change-Id: I1e89909055cc3054bfb7baa33c9dca3dfdc7336e
Implementing https://github.com/storj/storj/issues/5702 means adding a
bonus billing transaction for each storjscan transaction being recorded.
To do this idempotently, we need to the ability to for both the
storjscan and bonus transaction to be committed together.
This change updates the billing database to allow multiple billing
transactions to be inserted under the same database transaction.
Change-Id: I941864f47fc64d65aab076eec2e96fd04fcc7aac
FileWalker implements methods to walk over pieces in
in a storage directory.
This is just a refactor to separate filewalker functions
from pieces.Store. This is needed to simplify the work
to create a separate filewalker subprocess and reduce the
number of config flags passed to the subprocess.
You might want to check https://review.dev.storj.io/c/storj/storj/+/9773
Change-Id: I4e9567024e54fc7c0bb21a7c27182ef745839fff
We want to eliminate usages of LoopSegmentEntry.Pieces, because it is
costing a lot of cpu time to look up node IDs with every piece of every
segment we read.
In this change, we are eliminating use of LoopSegmentEntry.Pieces in the
node tally observer (both the ranged loop and segments loop variants).
It is not necessary to have a fully resolved nodeID until it is time to
store totals in the database. We can use NodeAliases as the map key
instead, and resolve NodeIDs just before storing totals.
Refs: https://github.com/storj/storj/issues/5622
Change-Id: Iec12aa393072436d7c22cc5a4ae1b63966cbcc18
This handles cases where a user is warned and triggers payment for their
account. Previously, only a frozen account will trigger this payment,
and will be unfrozen on successful payment. Now, accounts in warning
state trigger payments and are removed from that state on successful payment.
Issue: https://github.com/storj/storj/issues/5691
Change-Id: Icc2107f5d256657d176d8b0dd0a43a470eb01277
Previously, our code produced a Go panic when attempting to index an
empty slice of bucket user agent entries. A length check has been
added to ensure that the slice is not accessed beyond its bounds.
Change-Id: I904ca275b934be1b05393a45d99ff415dcafc926
This change adds onboarding_start, onboarding_end and onboarding_step
columns to the user_settings table. the first two are used to determine
if a user should go through onboarding, the last will be used to as the
step a user got to before exiting onboarding without finishing.
Issue: https://github.com/storj/storj/issues/5660
Change-Id: I8070c880d0d2fc22086f24087c962f57c695cc50
Add a combined index on normalized_email,status to improve performance of
common "get user" query used for the satellite UI.
Change-Id: I24a20d7826e0a68a68c2f95b5847eb819921e7c0
Metainfo needs to know rate and burst limit to be able to limit users
requests. We made cache for per project limiter but to make single
instance we need to know about limits. So far we were doing direct DB
call to get rate/burst limit for project but it's generating lots of
DB requests and can be easily cached as we even have project limit cache.
This change extends project limit cache with rate/burst limit and starts
using this change while creating project limiter instance for metainfo.
Because data size kept in project limit cache is quite small this change
also bumps a bit default capacity of the cache.
Fixes https://github.com/storj/storj/issues/5663
Change-Id: Icb42ec1632bfa0c9f74857b559083dcbd054d071
This might be pretty awful, but at least it is a complete and non-flaky
solution.
**Only when using the rollingupgrade test** (which implies a throwaway
satellite and also a PostgreSQL backend), create a trigger on the nodes
table which forces last_net to be equal to last_ip_port always.
Change-Id: I8448cf131e46576d96a414d06780270c7b2b1892
Remove generate-missing-project-salt migration tool code and related
tests. This migration has already been run and this code is no longer
needed.
Issue https://github.com/storj/storj-private/issues/163
Change-Id: I4e36dcd95a07c5305c597113a7fd08148e100ccc
This test involves a satellite with dev defaults (DistinctIP=no) being
upgraded past commit 2522ff09b6, which
means we need to run the dev-defaults-satellite-upgrade migration SQL
to avoid getting DistinctIP=yes behavior (which breaks the tests).
Change-Id: I29fb596d1ffa568dad635d98cfe9abacd3aaa48f
Up to now, we have been implementing the DistinctIP preference with code
in two places:
1. On check-in, the last_net is determined by taking the /24 or /64
(in ResolveIPAndNetwork()) and we store it with the node record.
2. On node selection, a preference parameter defines whether to return
results that are distinct on last_net.
It can be observed that we have never yet had the need to switch from
DistinctIP to !DistinctIP, or from !DistinctIP to DistinctIP, on the
same satellite, and we will probably never need to do so in an automated
way. It can also be observed that this arrangement makes tests more
complicated, because we often have to arrange for test nodes to have IP
addresses in different /24 networks (a particular pain on macOS).
Those two considerations, plus some pending work on the repair framework
that will make repair take last_net into consideration, motivate this
change.
With this change, in the #2 place, we will _always_ return results that
are distinct on last_net. We implement the DistinctIP preference, then,
by making the #1 place (ResolveIPAndNetwork()) more flexible. When
DistinctIP is enabled, last_net will be calculated as it was before. But
when DistinctIP is _off_, last_net can be the same as address (IP and
port). That will effectively implement !DistinctIP because every
record will have a distinct last_net already.
As a side effect, this flexibility will allow us to change the rules
about last_net construction arbitrarily. We can do tests where last_net
is set to the source IP, or to a /30 prefix, or a /16 prefix, etc., and
be able to exercise the production logic without requiring a virtual
network bridge.
This change should be safe to make without any migration code, because
all known production satellite deployments use DistinctIP, and the
associated last_net values will not change for them. They will only
change for satellites with !DistinctIP, which are mostly test
deployments that can be recreated trivially. For those satellites which
are both permanent and !DistinctIP, node selection will suddenly start
acting as though DistinctIP is enabled, until the operator runs a single
SQL update "UPDATE nodes SET last_net = last_ip_port". That can be done
either before or after deploying software with this change.
I also assert that this will not hurt performance for production
deployments. It's true that adding the distinct requirement to node
selection makes things a little slower, but the distinct requirement is
already present for all production deployments, and they will see no
change.
Refs: https://github.com/storj/storj/issues/5391
Change-Id: I0e7e92498c3da768df5b4d5fb213dcd2d4862924
We changed primary key for bucket_bandwidth_rollups table. Now we
need to do some cleanup in places like structs, sorting methods or SQL
queries.
Change-Id: Ida4f874f161356df193379a53507602e04db1668
In the past we set incorrect primary key for bucket_bandwidth_rollups
table and bucket name was first column of key instead of project id.
This change fixes this problem.
We will alter primary key manually for production satellites. That's
why migration code is checking if change was already applied.
https://github.com/storj/storj/issues/5332
Change-Id: I7dd555eb0c2f6fda1c90eaed08fa847b7f273bc7
