This change removes the use of the objects loop for calculating bucket
tallies. It has been superseded by a custom query.
Change-Id: I9ea4633006c9af3ea14d7de40871639e7b687c22
During billing, before invoice creation, check if users are part of a
package plan. If so, and if the package plan is expired, remove unused
credit from the user's balance. If the user has credit in addition to
the package credit, send an analytics event to notify someone to handle
the credit removal manually.
Change-Id: Iad71d791f67c9733f9d9e42f962c64b2780264cc
this is a very old tool built in the very early days
of v3, when we didn't know how the network would be
used. this tool anticipated being able to query remote
nodes for internal state. we don't do that. i don't
think anyone uses this.
Change-Id: Ie1ded3ecbedb09313f2d6fc721039e0f15e4ee85
rather than only logging the last_nets we see in clumpedPieces, this
will run through all the last_nets and log any that have more than one
node. This should have the same outcome, except the counts will be 1
higher (because FindClumpedPieces won't include the first node found in
a clumped network, and this will).
This should be quite a bit faster.
Change-Id: I6a7b2fd387e98963d5295c9ecfde80f2e1ee3b7a
We were using the UploadSelectionCache previously, which does _not_ have
all nodes, or even all online nodes, in it. So all nodes with less than
MinimumVersion, or with less than MinimumDiskSpace, or nodes suspended
for unknown audit errors, or nodes that have started graceful exit, were
all missing, and ended up having empty last_nets. Even with all that,
I'm kind of surprised how many nodes this involved, but using the upload
selection cache was definitely wrong.
This change uses the download selection cache instead, which excludes
nodes only when they are disqualified, gracefully exited (completely),
or offline.
Change-Id: Iaa07c988aa29c1eb05796ac48a6f19d69f5826c1
It seems that the "what pieces are clumped" code does not work right, so
this logic is causing repair overload or other repair failures.
Hide it behind a flag while we figure out what is going on, so that
repair can still work in the meantime.
Change-Id: If83ef7895cba870353a67ab13573193d92fff80b
* Update defaults for gateway credentials URL and linksharing URL to use
storjsatelliteshare.io instead of storjshare.io
* Add new config for "public linksharing URL" and set it to
link.storjshare.io
* Use "private" linksharing URL for actions within the object browser
* Use "public" linksharing URL for sharing files externally
Resolves https://github.com/storj/storj/issues/5805
Change-Id: I2c8fbd04141755b4751dcf4d054253a7ff8d6cf3
Clumped segments (segments with multiple pieces on the same subnet) may
need repair, but the clumped pieces are considered retrievable and we
don't need to call such segments irreparable.
We do want to know where they're coming from, though, if we can, because
we are seeing more than expected.
Change-Id: I41863b243f4bb007ef8929191a3fde1562565ef9
The project member invitations table has been modified to contain a
column for the ID of the user who sent the invitation. This ID is
required for us to return information about the inviter to the
satellite frontend.
References #5855
Change-Id: I928d987a8db2340f731ca65ce30173d4f90a9837
The query for GetNodesNetworkInOrder is causing far too much load on the
database. Since it is not critical that the repair checker have
perfectly up-to-date node network information, we can use a cache
instead.
Change-Id: I07ad45bfdeb46529da093941a06c2da8a00ce878
87d0789691 replaces offset usage with cursors.
But to continue the interation from a specific cursor, we need to iterate over ordered records.
(at least this is what I understood based on the failing tests)
87d0789691
Change-Id: Ic4da3a7c5f03386dd4c373c05102f05871900a3a
We will remove segments loop soon so we need first to move
Segment definition to rangedloop package.
https://github.com/storj/storj/issues/5237
Change-Id: Ibe6aad316ffb7073cc4de166f1f17b87aac07363
This change updates the replacer in satellite/satellitedb/dbx/gen/main.go
to work with an updated dbx.
Change-Id: I08e89d6d27e6f1d435416105fe5f622009add7ad
* Don't use rpcstatus.Unknown as an indicator of dial failure; instead,
GetShare now indicates with a per-share field where a failure happened
(DialFailure, RequestFailure, NoFailure). Use that information in
Verify() to determine how to treat the source node.
* Add a test that replaces a storage node with a black hole, so that
connections there will always time out. Make sure we handle that case
correctly.
Refs: https://github.com/storj/storj/issues/5632
Change-Id: I513a53520fb48b7187d4c4d7e14e01e2cfc0a721
Stripe invoice project records while listing are causing full table scan
because of OFFSET caluse. This change is refactoring query to list using
cursor.
Change-Id: I6b73b9b2815173d7ef02cf615408778476eb3b7b
We have method which is getting projects owned by specific user but it's
causing full table scan because we don't have index on owner_id column.
Change-Id: Icb71c9ac5b73104a52241ed8ba126c995c10811f
The string check previously used to check for constraint errors is now
replaced with dbx.IsConstraintError check.
Change-Id: I553ccd69e3c02b6b54441bd9f929b85a155eaf00
Fix an error that can occur when processing multiple invoices for the same user in a single invoice cycle when the user is paying with Storj tokens.
Change-Id: I54af8c7dde1965d994f687fdfc4e4b5ef4deeb2d
w.Header().Set needs to be called before WriteHeader,
because WriteHeader sends all the headers and calls to
Set won't have any effect afterwards.
Change-Id: Ia6b1c5e2cd54201a6c3980d63de04a0095b2db9a
The console DB cleanup chore has been extended to remove old project
member invitation records.
Resolves#5816
Change-Id: Id0a748e40f5acf03b9b903265c653b072846ba19
There is still a reference to partner_id in a query, which we cannot
move until dropping the "not null" constraint for it. This change adds
that migration.
Related to https://github.com/storj/storj/issues/5432
Change-Id: I98802a6e1bd59f3d9214de3db6688d9daf664a70
A chore responsible for purging data from the console DB has been
implemented. Currently, it removes old records for unverified user
accounts. We plan to extend this functionality to include expired
project member invitations in the future.
Resolves#5790
References #5816
Change-Id: I1f3ef62fc96c10a42a383804b3b1d2846d7813f7
This change makes the error thrown when adding an existing member to a
project readable.
Issue: https://github.com/storj/storj/issues/5840
Change-Id: I4269495f9b7b09c77fbf1af1fc605e5c95bd7cbf
This change adds the user's passphrase prompt setting to the
/account/settings endpoints.
Issue: https://github.com/storj/storj/issues/5616
Change-Id: I48d470d49e82096fd090b74da323b279e342546e
Allow a longer encrypted key length to reduce 'key length is too big'
errors in gateway-mt. Gateway is enforcing an unencrypted key length
of 1024 bytes but when encrypted some keys are exceeding the current
limit.
updates https://github.com/storj/gateway-mt/issues/335
Change-Id: I861a2313e558c9f7d39569d21c7a3d429c83575c
Ensure that the value of "pricing packages enabled" flag on frontend is
the same as what is configured on the backend.
Change-Id: Id78771800a4973ebd3ad4e22f1953f6f71c75dd4
The "object not found" included an additional prefix "metabase:", which
broke uplink error message detection.
Ideally, changing an internal error message shouldn't break the uplink.
Change-Id: I5ce7789cc11742d3435af1ec555bc96927f1bedc
Current observer used with ranged loop is using massive amount of
memory because each range is generating separate set of bloom filters.
Each bloom filter can be up to 2MB of memory. That's a lot.
This change is initial change to reduce used memory by sharing bloom
filters between ranges and just synchronize access to them. This
implementation is rather simple and even naive but maybe it will be
enough without doing something more complex.
https://github.com/storj/storj/issues/5803
Change-Id: Ie62d19276aa9023076b1c97f712b788bce963cbe
Remove the command to apply free tier coupons from the generate invoices
command. Applying free tier coupons should instead be done outside the
normal time window for invoice generation to save time during the
invoicing cycle.
Change-Id: If8fecc558411d5a6fff9d5689143d72f3b709e55
This is refactor/cleanup change before I will start working on adding
separate GC observer with optimized memory consumption.
https://github.com/storj/storj/issues/5803
Change-Id: I854cb3797802a32942c25f2765dbb72be88bacbd
This change immplements methods for interacting with the project member
invitations table.
Resolves#5766
Change-Id: I0090c50f9fde5bcdae4ebdaa72cdcaa84d341b54
A table for storing pending project member invitations has been added.
This table is required to satisfy our new project invitation UX revamp.
References #5766
Change-Id: I6f948de66ed5b4dc81532564958ff7f48533cad2
Add a new billing command that will convert stripe customer balances into invoice items so that the charges can be processed normally by the invoicing workflow.
Change-Id: Iaa8350e7aca80a0f14e94eb8ef8b7d6ce0b5b3b8
Fixed nil pointer dereference panic.
Updated naming conventions so that PUT request and GET response bodies are the same (bandwidth, storage and segment).
Allowed usage of notations like 150GB, 2TB for storage and bandwidth limits.
Updated tests.
Issue:
https://github.com/storj/storj/issues/5674
Change-Id: I7ac27c00721a9b4bf507afa34cb05c4475a809ad
This change adds more tests to the autofreeze chore and the freeze
service according to the testplan linked in the issue below.
Issue: https://github.com/storj/storj/issues/5738
Change-Id: Ib2afaa283961b2e7ef6fb6e5613ee083ac7d79eb
This field is deprecated in favor of UserAgent; Removing these
references is the final step necessary before dropping the columns from
the database.
https: //github.com/storj/storj/issues/5432
Change-Id: I3a6619170dcf382f82dc8eddb73b6547eaf636f0
Currently when error.html is not available, we should still start the
server and fallback to some simpler implementation. This template does
not require any external assets.
Change-Id: I76b660db988987e1e9ebadd966f60e149f26ff24
lrucache is now using time2 package and we can make expiration
test without using time.Sleep.
https://github.com/storj/storj/issues/5788
Change-Id: I48f2693c3db78fcf4e30e618bb3304be3625100c
Remove the extra prepended '0x' from the wallet address.
Remove prepended '0x' from the transaction ID.
Change-Id: Id215536915fba62cc348aa2c3356ecc7898d68a2
This change separates hubspot form submission for personal and business
accounts, with new company name and storage needs fields.
Issue: https://github.com/storj/storj-private/issues/220
Change-Id: Ieb0fb64f87614c7327dc5f894140fb8a54ededa0
New SQL queries for GetProjectObjectsSegments turns out to introduce
full table scan. This is fix for this problem.
Change-Id: Ieac22aafeb780168523a97e27c9283c9ac6a24c8
This change reverses behavior added by 45d5a93 that made the server
return a 500 status code when the index.html file for the satellite
frontend couldn't be loaded. The presence of this file was previously
intentionally optional.
Change-Id: I875a171a37b735c3523eb5b13d83f084f1781053
Previously, we evaluated index.html as a template in order to insert
frontend config values into meta tags. Now that the frontend fetches
its config through the satellite API, this is no longer necessary.
Resolves#5494
Change-Id: Ic98507c5e16cd80317bd9c31d4b55abda0dd7e34
The test for the admin API's OAuth authorization behaviour has been
modified to use a random available port given by the system rather than
a hardcoded one. This prevents the test from accidentally using a port
that is already in use.
Change-Id: Iae017b2f397ae53f1a006bae1d0578d2ddfd0875
References to the meta tag config values in Vue components have been
modified to instead refer to the frontend config fetched through the
satellite API.
References #5494
Change-Id: I00ecf81d4a0ba6bd07c827cecb2c689d923d67c0
The project charge information used by the satellite frontend to
compute project cost estimates has been updated to account for
configured egress discounts.
Resolvesstorj/storj-private#215
Change-Id: Ic90b015d65f5bea104ac96fb0cea545b3f9f1f8f
Query to list (with pages) stripe customers were doing full table scan
because Offset clause was used. This refactoring changed listing to
use cursor instead Offset.
Change-Id: I14688e6c533bc932ba0d209a061562f080b4cf54
Invoicing has been modified to account for an egress discount ratio,
which specifies the fraction of the amount of data stored that egress
usage should be discounted.
The egress discount ratio measures discounted egress units per storage
unit-months, so a ratio of 0.5 with 2 MB-months of storage would
discount 1 MB of egress.
Resolvesstorj/storj-private#224
Change-Id: I43d7d6719391c303712c082709aef77249c65f62
This change allows for specifying the ratio of free egress per unit
of storage within a price override configuration.
References storj/storj-private#215
References storj/storj-private#224
Change-Id: Ib1c79f77ec8bb11dd5b2f9dace13800b0b3ce942
No component has referenced this page since 9dab10e and we do not
anticipate this changing, so this page can be safely removed.
Resolves#5768
Change-Id: I57acb5e4d0977d74df46aaf67606a19ec0f10bcf
We automatically start a chore to check whether the blobstore is
writeable and readable, however, we don't want to fail the tests due to
that reason. Usually we want to test some other failure.
There probably should be some nicer way to achieve this, but this is an
easier fix.
Change-Id: I77ada75329f88d3ea52edd2022e811e337c5255a
- only applies to storjscan transactions
- applies a 10% bonus by default
- bonus transactions have a distinct source "type" to allow for
filtering on the frontend
Fixes: https://github.com/storj/storj/issues/5702
Change-Id: I32d65f776c58bcb41227ff5bc77a8e4cb62a9add
This another account endpoint; patch /auth/account/settings. to handle
changing a user's settings, including their session timeout config.
Issue: https://github.com/storj/storj/issues/5560
Change-Id: I747b4e919cf7cef7c867ac9d282837ef51bed67e
We avoid putting more than one piece of a segment on the same /24
network (or /64 for ipv6). However, it is possible for multiple pieces
of the same segment to move to the same network over time. Nodes can
change addresses, or segments could be uploaded with dev settings, etc.
We will call such pieces "clumped", as they are clumped into the same
net, and are much more likely to be lost or preserved together.
This change teaches the repair checker to recognize segments which have
clumped pieces, and put them in the repair queue. It also teaches the
repair worker to repair such segments (treating clumped pieces as
"retrievable but unhealthy"; i.e., they will be replaced on new nodes if
possible).
Refs: https://github.com/storj/storj/issues/5391
Change-Id: Iaa9e339fee8f80f4ad39895438e9f18606338908
We would like to have ability to limit burst uploads to the single
object (the same location). This change we are limiting such upload to
one per second.
Change-Id: Ib9351df1017cbc07d7fc2f846c2dbdbfcd3a360c
The blobstore implementation is entirely related to storagenode, so the
rightful place is together with the storagenode implementation.
Fixes https://github.com/storj/storj/issues/5754
Change-Id: Ie6637b0262cf37af6c3e558556c7604d9dc3613d
add ability to sort api keys on the access management page by ascending
or descending name or creation date. Additionally, edit api keys query
when ordering by name to order by lower(name) so names starting with
capital letters are not treated differently from lower case when
ordering.
Change-Id: I81dbb87587a24fb7097313f76bad116b1f20d306
With this change we are replacing parsing code with existing go-redis
util.
We also switch redis client to version 9.
Change-Id: Ie4a651e3ae6960e68958c690873925d319b70e10
The following tests should be made less flaky by this change:
- TestFailedDataRepair
- TestOfflineNodeDataRepair
- TestUnknownErrorDataRepair
- TestMissingPieceDataRepair_Succeed
- TestMissingPieceDataRepair
- TestCorruptDataRepair_Succeed
- TestCorruptDataRepair_Failed
This follows on to a change in commit 6bb64796. Nearly all tests in the
repair suite used to rely on events happening in a certain order. After
some of our performance work, those things no longer happen in that
expected order every time. This caused much flakiness.
The fix in 6bb64796 was sufficient for the tests operating directly on
an `*ECRepairer` instance, but not for the tests that make use of the
repairer by way of the repair queue and the repair worker. These tests
needed a different way to indicate the number of expected failures. This
change provides that different way.
Refs: https://github.com/storj/storj/issues/5736
Refs: https://github.com/storj/storj/issues/5718
Refs: https://github.com/storj/storj/issues/5715
Refs: https://github.com/storj/storj/issues/5609
Change-Id: Iddcf5be3a3ace7ad35fddb513ab53dd3f2f0eb0e
Components related to project usage costs have been updated to show
different estimations for each partner, and the satellite has been
updated to send the client the information it needs to do this.
Previously, project costs in the satellite frontend were estimated
using only the price model corresponding to the partner that the user
registered with. This caused users who had a project containing
differently-attributed buckets to see an incorrect price estimation.
Resolvesstorj/storj-private#186
Change-Id: I2531643bc49f24fcb2e5f87e528b552285b6ff20
This combines the ListStreamPositions and GetSegmentByPosition
calls with a ListSegments call that now knows how to return
only the segments within a Range, just like ListStreamPositions.
It would theoretically be possible to also include the
GetObjectLastCommitted call by having it do one of three
queries based on the incoming request range, but that would
mean duplicating the data for the object in every single
row that is returned for each segment in the range.
One gross thing that ListSegments has to do now is update the
first segment returned with the information from any ancestor
segment because GetSegmentByPosition used to do that. It only
updates the first segment so that it doesn't do O(N) database
queries. It seems difficult to have it do a single query to
update all of the segments at once. I'm not certain this change
should be merged on this basis alone.
This change has made me think a couple of things should happen:
1. Server side copy with ancestor segments strikes again
making the code less clear and potentially more buggy
or inefficient for a rare case (empirically <0.1%)
2. The download code requests individual segments from
the satellite lazily as part of its download which
requires the satellite telling it the locations of
all of the segments which requires the satellite
querying the locations of all of the segments. Instead
the download RPC could return the orders for all of
the segments for a range and the download code could
issue N download calls rather than 1 download call and
N get segment calls. I believe both sides of the code
paths would be simpler and more efficient this way.
3. In looking at the timing information for downloads when
testing this, we really need to focus on getting the
auth key and bandwidth limit verification times down.
Here's the timing I saw:
- 42ms: validate auth
- 52ms: bandwidth usage checking
- 14ms: get object info
- 26ms: get segment position info
- 26ms: getting the first segment full info
- 20ms: unaccounted for by spans
- 6ms: creating the orders
This change will remove 26ms, but there's a good 90ms
in just validation. With improved semantics hitting the
database only once and improved validation, a download
rpc taking ~30ms seems doable compared to our current
~200ms.
Change-Id: I4109dba082eaedb79e634c61dbf86efa93ab1222
Get bucket was returning a "bad request" HTTP status code when the
bucket doesn't exists.
We have to return HTTP "Not found" status code.
Change-Id: If717d99276b02a1e59a9b71ebc909bd6d8d9390b
Instead of granting a coupon when purchasing a package, grant credit.
This changes paymentsconfig.PackagePlan to use credit amount rather than
coupon ID. Add additional check to see if a paid invoice with the
description exists. If so, don't create and pay another invoice.
Change-Id: I81df24984c519c773db5fc8e9070bd7797070ec2
Add and implement interface to manage customer balances. Adds ability to
add credit to a user's balance, list balance transactions, and get the
balance.
Change-Id: I7fd65d07868bb2b7489d1141a5e9049514d6984e
Invoicing-related payment service methods have been modified to send
Stripe API requests in parallel.
Additionally, randomness has been added to the Stripe backend wrapper's
exponential backoff strategy in order to reduce the effects of the
thundering herd problem, which arises when executing many simultaneous
API calls.
Resolves#5156
Change-Id: I568f933284f4229ef41c155377ca0cc33f0eb5a4
Add columns package_plan and purchased_package_at to stripe_customers
table and add methods to update and select these values from console
service and payments accounts.
Change-Id: I1e89909055cc3054bfb7baa33c9dca3dfdc7336e
Implementing https://github.com/storj/storj/issues/5702 means adding a
bonus billing transaction for each storjscan transaction being recorded.
To do this idempotently, we need to the ability to for both the
storjscan and bonus transaction to be committed together.
This change updates the billing database to allow multiple billing
transactions to be inserted under the same database transaction.
Change-Id: I941864f47fc64d65aab076eec2e96fd04fcc7aac
A row in the new `user_settings` table does not always exist for a user,
even if they have been around for a while.
Since `user_settings` is now what defines the state of a user's
onboarding flow, prior to this fix, even old users would receive the
onboarding flow again.
This change appropriately updates `user_settings` for users who already
have projects, and thus have already gone through the onboarding flow. A
brand new user will still be navigated to the beginning of onboarding.
Change-Id: Ie745d280f6b8094ec60c200c2dca8d018d51f7d1
FileWalker implements methods to walk over pieces in
in a storage directory.
This is just a refactor to separate filewalker functions
from pieces.Store. This is needed to simplify the work
to create a separate filewalker subprocess and reduce the
number of config flags passed to the subprocess.
You might want to check https://review.dev.storj.io/c/storj/storj/+/9773
Change-Id: I4e9567024e54fc7c0bb21a7c27182ef745839fff
By accident query to get latest table stats was ordered by 'row_count'
column instead 'create'. We need latest stats so we need ordering by
creation time.
Change-Id: I9d0a0edda8bab59c3d96b7a15cd6502ed51633fc
This flag was in general one time switch to enable versions internally.
New we can remove it as it makes code more complex.
Change-Id: I740b6e8fae80d5fac51d9425793b02678357490e
We want to eliminate usages of LoopSegmentEntry.Pieces, because it is
costing a lot of cpu time to look up node IDs with every piece of every
segment we read.
In this change, we are eliminating use of LoopSegmentEntry.Pieces in the
node tally observer (both the ranged loop and segments loop variants).
It is not necessary to have a fully resolved nodeID until it is time to
store totals in the database. We can use NodeAliases as the map key
instead, and resolve NodeIDs just before storing totals.
Refs: https://github.com/storj/storj/issues/5622
Change-Id: Iec12aa393072436d7c22cc5a4ae1b63966cbcc18
The default price threshold has been changed to 10000 cents. The help
text has also been modified to clarify that this value is in cents.
Change-Id: I3b605056c860a3afef7202950180bc8b863fe16a
A debug log has been added to the autofreeze chore to show numbers of
invoices, users involved, users frozen, users warned and users bypassed
due to larger than config amount owed.
Issue: https://github.com/storj/storj/issues/5700
Change-Id: I4e628606fe234b6d239b76167a4781ca9e276308
This handles cases where a user is warned and triggers payment for their
account. Previously, only a frozen account will trigger this payment,
and will be unfrozen on successful payment. Now, accounts in warning
state trigger payments and are removed from that state on successful payment.
Issue: https://github.com/storj/storj/issues/5691
Change-Id: Icc2107f5d256657d176d8b0dd0a43a470eb01277
Previously, our code produced a Go panic when attempting to index an
empty slice of bucket user agent entries. A length check has been
added to ensure that the slice is not accessed beyond its bounds.
Change-Id: I904ca275b934be1b05393a45d99ff415dcafc926
This change adds an endpoint that gets a user's settings. It will
create a new settings entry if no settings exists. There's also a new
endpoint to change a user's onboarding status.
Issue: https://github.com/storj/storj/issues/5661
Change-Id: I9941bb9d61994af46244003f3ef4fcfe7d36918e
This change adds onboarding_start, onboarding_end and onboarding_step
columns to the user_settings table. the first two are used to determine
if a user should go through onboarding, the last will be used to as the
step a user got to before exiting onboarding without finishing.
Issue: https://github.com/storj/storj/issues/5660
Change-Id: I8070c880d0d2fc22086f24087c962f57c695cc50
A Stripe backend implementation has been added that uses an exponential
backoff strategy to retry failed API calls. This behavior can be
configured in the satellite config.
References #5156
Change-Id: I16ff21a39775ea331c442457f976be0c95a7b695
We have a specific issue that a user uploaded a file to a bucket
geo-fenced to EU and one of the pieces appeared to be on a node in the
US. The country code of this node is set to SE (Sweden) in the satellite
DB. It turns out that some time ago MaxMind changed the country code of
this node's IP from Sweden to US, but this change hasn't been reflected
in the satellite's database.
So far the satellite updates the country code of a node only if its IP
changes. It was assumed that if the IP does not change, its country code
shouldn't change too. This turned to be a wrong assumption.
With this change, the satellite will look up the MaxMindDB on every
check-in to see if the country code of the node's IP has changed.
Change-Id: Icdf659b09be9fc6ad14601902032b35ba5ea78c4
Add a combined index on normalized_email,status to improve performance of
common "get user" query used for the satellite UI.
Change-Id: I24a20d7826e0a68a68c2f95b5847eb819921e7c0
This commit ensures that all invocations of Stripe library methods
include a context. This allows us to control the timeout and
cancellation of the underlying HTTP requests made by the Stripe
library.
References #5156
Change-Id: I8ddb317f3f2cbb06cfab869fbebdaf2ad78b7999
Metainfo needs to know rate and burst limit to be able to limit users
requests. We made cache for per project limiter but to make single
instance we need to know about limits. So far we were doing direct DB
call to get rate/burst limit for project but it's generating lots of
DB requests and can be easily cached as we even have project limit cache.
This change extends project limit cache with rate/burst limit and starts
using this change while creating project limiter instance for metainfo.
Because data size kept in project limit cache is quite small this change
also bumps a bit default capacity of the cache.
Fixes https://github.com/storj/storj/issues/5663
Change-Id: Icb42ec1632bfa0c9f74857b559083dcbd054d071
We just fixed case were project limit cache was not used properly. This
is test case to cover that fix.
Change-Id: Iee467f0a46836860a14ab6238a9842ffbf54ed4c
This might be pretty awful, but at least it is a complete and non-flaky
solution.
**Only when using the rollingupgrade test** (which implies a throwaway
satellite and also a PostgreSQL backend), create a trigger on the nodes
table which forces last_net to be equal to last_ip_port always.
Change-Id: I8448cf131e46576d96a414d06780270c7b2b1892
Currently, to get number of entries in segments table we are doing
heavy SELECT count(*) operation. For biggest satellite it's taking
25min now. We are using this method to get stat before and after
segments loop so it adds almost 1h to overall loop time.
With current version of crdb we are using this additional code won't be
used because global configuration for stats refresh rate is inaccurate
for such large table like `segments`. Soon we should be able to upgrade
crdb and be able to adjust refresh rate per table and configure it to
satisfy defined threshold.
https://github.com/storj/storj/issues/5544
Change-Id: I05cfd9154f08894d2bc56bf716b436d1b03b87f1
A field has been added to the coupon struct indicating whether it is
associated with a partner's pricing package. This is required to
alter the appearance of partner coupons in the satellite frontend.
References storj/storj-private#172
Change-Id: Ie48ae3902aaa108abf9a399242a0cd98cb53d1c3
This change sends an event to segment for when a user is unfrozen.
It also moves freeze and warning event triggers from the autofreeze
chore to the account freeze service.
Change-Id: I5c0522b921b7baf52d6db5eb7ef841c08644a461
This change reworks the allowedAuthorization function to check what
groups the user is a part of to determine if authorization should be
granted. By wrapping each handler with withAuth, we can specify the
allowed groups for each api method individually.
github issue: https://github.com/storj/storj/issues/5565
Change-Id: I1804dda04d5b16d19e93bd7199fb3fc89fca1294
Remove generate-missing-project-salt migration tool code and related
tests. This migration has already been run and this code is no longer
needed.
Issue https://github.com/storj/storj-private/issues/163
Change-Id: I4e36dcd95a07c5305c597113a7fd08148e100ccc
It looks that at some point we broke how project limits cache is used
and we were missing cache in most critical paths (upload/download).
This is fix for this issue.
I also adjusted cache methods naming.
Change-Id: Ic98372779a39365d0920fe3943f1f7a68b064173
This change adds a card to the billing overview page, which shows the
user's token balance from coinpayments.
Issue: https://github.com/storj/storj-private/issues/151
Change-Id: I11e295b48791b32b745cb7a11c5b4aad6b56618e
This test involves a satellite with dev defaults (DistinctIP=no) being
upgraded past commit 2522ff09b6, which
means we need to run the dev-defaults-satellite-upgrade migration SQL
to avoid getting DistinctIP=yes behavior (which breaks the tests).
Change-Id: I29fb596d1ffa568dad635d98cfe9abacd3aaa48f
Only API peer needs access to order DB (and rollups cache) because it's
only place where we are creating orders for PUT and GET operations. For
other peers like auditor and repairer we can set noop implementation to
reduce number of dependencies needed for them.
Change-Id: Ic32d1879f0b97ffc4516f401898e31e95ae892e4
It was surprising that `satellite auditor` complained about SMTP mail settings, even if it's not supposed to sending any mail.
Looks like we can remove the mail service dependency, as it's not a hard requirement for overlay.Service.
Change-Id: I29a52eeff3f967ddb2d74a09458dc0ee2f051bd7
I tried to configure a satellite service and got this error:
```
DEBUG process/exec_conf.go:408 Unrecoverable error {"error": "missing port in address"}
```
It took some time to realize that I forgot to set the SMTPServerAddress.
This patch makes it easier to detect similar problem (detailed error message), and makes SMTP parameters optional if no real mail sending is used (simulated or nomail)
Change-Id: I32535a7c8d6529e19e4d919806f42ba430d074a5
Up to now, we have been implementing the DistinctIP preference with code
in two places:
1. On check-in, the last_net is determined by taking the /24 or /64
(in ResolveIPAndNetwork()) and we store it with the node record.
2. On node selection, a preference parameter defines whether to return
results that are distinct on last_net.
It can be observed that we have never yet had the need to switch from
DistinctIP to !DistinctIP, or from !DistinctIP to DistinctIP, on the
same satellite, and we will probably never need to do so in an automated
way. It can also be observed that this arrangement makes tests more
complicated, because we often have to arrange for test nodes to have IP
addresses in different /24 networks (a particular pain on macOS).
Those two considerations, plus some pending work on the repair framework
that will make repair take last_net into consideration, motivate this
change.
With this change, in the #2 place, we will _always_ return results that
are distinct on last_net. We implement the DistinctIP preference, then,
by making the #1 place (ResolveIPAndNetwork()) more flexible. When
DistinctIP is enabled, last_net will be calculated as it was before. But
when DistinctIP is _off_, last_net can be the same as address (IP and
port). That will effectively implement !DistinctIP because every
record will have a distinct last_net already.
As a side effect, this flexibility will allow us to change the rules
about last_net construction arbitrarily. We can do tests where last_net
is set to the source IP, or to a /30 prefix, or a /16 prefix, etc., and
be able to exercise the production logic without requiring a virtual
network bridge.
This change should be safe to make without any migration code, because
all known production satellite deployments use DistinctIP, and the
associated last_net values will not change for them. They will only
change for satellites with !DistinctIP, which are mostly test
deployments that can be recreated trivially. For those satellites which
are both permanent and !DistinctIP, node selection will suddenly start
acting as though DistinctIP is enabled, until the operator runs a single
SQL update "UPDATE nodes SET last_net = last_ip_port". That can be done
either before or after deploying software with this change.
I also assert that this will not hurt performance for production
deployments. It's true that adding the distinct requirement to node
selection makes things a little slower, but the distinct requirement is
already present for all production deployments, and they will see no
change.
Refs: https://github.com/storj/storj/issues/5391
Change-Id: I0e7e92498c3da768df5b4d5fb213dcd2d4862924
Segments loop have build-in sanity check to verify if number of segments
processed by loop is roughly fine. We want to have the same verification
for ranged loop.
https://github.com/storj/storj/issues/5544
Change-Id: Ia19edc0fb4aa8dc45993498a8e6a4eb5928485e9
If the session_minutes column in user_settings is not null, use this
value to determine the expiration when creating or refreshing a session.
github issue: https://github.com/storj/storj/issues/5487
Change-Id: I1f777f7ffcd4018b0c40f0b26c8a50de85a31909
We should return public project id instead of regular one when quering project charges so that it is consistent on a client side.
This is a fix for an issue with displaying project name on a Billing screen.
Issue:
https://github.com/storj/storj/issues/5641
Change-Id: Ic0c544dbe7369aa88f3a7c97a2f5de5bc854e1fd
We changed primary key for bucket_bandwidth_rollups table. Now we
need to do some cleanup in places like structs, sorting methods or SQL
queries.
Change-Id: Ida4f874f161356df193379a53507602e04db1668
The tests were using global variables for keeping the mock state, which
was indexed by the satellite ID. However, the satellite ID-s are
deterministic and it's possible for two tests end up using the same
mocks.
Instead make the mock creation not depend on the satellite ID and
instead require it being configured via paymentsconfig.
This fixes TestAutoFreezeChore failure.
Change-Id: I531d3550a934fbb36cff2973be96fd43b7edc44a
In the past we set incorrect primary key for bucket_bandwidth_rollups
table and bucket name was first column of key instead of project id.
This change fixes this problem.
We will alter primary key manually for production satellites. That's
why migration code is checking if change was already applied.
https://github.com/storj/storj/issues/5332
Change-Id: I7dd555eb0c2f6fda1c90eaed08fa847b7f273bc7
An endpoint has been added that returns whether a pricing package is
available for a user to purchase. This will be used to conditionally
skip the pricing plan selection step of the onboarding tour.
Change-Id: I8c02a4e474e5f0f80778453b7daf674c8da64306
If a user changes password while a password recovery token exists,
delete the token so the reset password email doesn't work.
github issue: https://github.com/storj/storj-private/issues/149
Change-Id: Ibb0c13076e182c46501c1d7ac92d3fecc7d5ee58
Add passphrase_prompt column to user_settings table to store a boolean
whether a user would like to be prompted to enter a passphrase when
entering a new project.
Issue https://github.com/storj/storj/issues/5613
Change-Id: I5d086df277aff458453343c7c38a379e7b1b8bf9
Earlier we made a change to not cancel flushing orders when flushing
was triggered by orders endpoint method but we missed a case
where it can be also triggered (and canceled) by metainfo endpoints
method. This change moves ignoring context cancellation deeper.
Change-Id: Id43176f552efc3167345783f73aab885411ac247
This change adds a new chore that will check for failed invoices and
potentially freeze corresponding accounts.
It makes slight modifications to stripemock.go and invoices.go (adding
stripe CustomerID to the Invoice struct).
Issue: https://github.com/storj/storj-private/issues/140
Change-Id: I161f4037881222003bd231559c75f43360509894
A pricing plan selection step for users with a recognized partner has
been added to the beginning of the onboarding tour. Once visited, users
have the option of purchasing the pricing plan associated with their
partner or proceeding as a paid or free tier user.
Resolves storj-private#118
Resolves storj-private#126
Change-Id: I3b423194d96deaf87cf9807a766bf4d04fbcf86d
add new config to the satellite admin: --admin.groups.limit-update.
This can be used as an alternate means of authentication if the request
is coming from the oauth proxy.
Change-Id: Ic2de13862e6414244b060c66a0f2bed72097cbad
Commit fb59974 disabled usage price overrides because of a failing
test. This change reenables it while resolving the issue that caused
the test to fail.
The previous version of the test passed Gerrit verification and was
merged, but it failed for the primary Jenkins pipeline after merge.
This is due to a difference in how the Jenkins build runs Cockroach
and Postgres for each pipeline.
This commit rewrites the test to be safe for concurrent execution by
ensuring any mutable variables are defined within each test so that
shared state across tests is reduced.
Change-Id: Ia4566c9cd2d698afdb2caa4b7e2808b17e18de4e
It was possible to get into a situation where successfulPieces =
es.RequiredCount(), errorCount < minFailures, and inProgress == 0 (when
the succeeding gets all completed before the failures), whereupon the
last goroutine in the limiter would sit and wait forever for another
goroutine to finish.
This change corrects the handling of that situation.
As an aside, this is really pretty confusing code and we should think
about redoing the whole function.
Change-Id: Ifa3d3ad92bc755e563fd06b2aa01ef6147075a69
Also, fix nodevents iota usage. Type is directly used in the database
hence we should avoid iota, because it's easy to mess-up the constant
values.
Change-Id: I32bf6d0502fbf3d18660dab5b1dd0b5627b444df
Add new purchase-package endpoint to Server. The endpoint can be enabled
or disabled by a new config, --console.pricing-packages-enabled.
The purchase-package endpoint applies a coupon and adds and charges a
credit card if user's useragent is a partner with a configured package
plan.
github issue: https://github.com/storj/storj-private/issues/125
Change-Id: I0d6ccccd6874ddba360c45f338fd1c44f95e135a
Project usage price overriding has been removed because it produces
incorrect results when tested. It should not be re-implemented until
the issues it causes are resolved.
Change-Id: Ic92eff374c9af4fea3bf32782a72303a7978b055
The update user API function was defining a parameter that the API
calling function doesn't receive, so all the values were shifted by
1 after this extra parameter and the calling function was sending these
values to the next one updating the fields that the user was not
expecting.
The API server doesn't allow to update this removed parameter, so it
was likely a bug caused by copy-pasta.
Change-Id: I52a2b66abc1f6e5cb0070dc6fd230c30d1133dcf
A new struct containing configuration values for the satellite frontend
is populated and served to the client. In the future, this struct will
be used to configure the frontend rather than using Go templates and
meta tags to do so.
References #5494
Change-Id: Ie2c8a140d53b20fab159d34f2e3767e5dbd43bdb
This code is essentially replacement for eestream.CalcPieceSize. To call
eestream.CalcPieceSize we need eestream.RedundancyStrategy which is not
trivial to get as it requires infectious.FEC. For example infectious.FEC
creation is visible on GE loop observer CPU profile because we were
doing this for each segment in DB.
New method was added to storj.Redundancy and here we are just wiring it
with metabase Segment.
BenchmarkSegmentPieceSize
BenchmarkSegmentPieceSize/eestream.CalcPieceSize
BenchmarkSegmentPieceSize/eestream.CalcPieceSize-8 5822 189189 ns/op 9776 B/op 8 allocs/op
BenchmarkSegmentPieceSize/segment.PieceSize
BenchmarkSegmentPieceSize/segment.PieceSize-8 94721329 11.49 ns/op 0 B/op 0 allocs/op
Change-Id: I5a8b4237aedd1424c54ed0af448061a236b00295
Purchase collects a payment from user using the specified price,
description, and payment method. This is implemented at the lower layers
using the payments.Invoices interface. There are also additions to
stripecoinpayments stripemock to simulate errors returned from stripe
invoices New and Pay. If an invoice exists with the same description, if
it is a draft, it is deleted and a new one is created and paid. If it is
open, pay it and don't create a new one.
Change-Id: Ic3147434bc44a0777ecbedb3a4ed4c768eb02ea3
This change removes the trailing slash from the account activation and
password recovery URLs, making them consistent with the rest. The URLs'
previous forms are still supported, however, in order to not invalidate
emails containing them.
Resolvesstorj/customer-issues#491
Change-Id: Ie774a87698d8e9edd1836611968fc3911c6cc56f
If a node is already disqualified or exited, don't bother going through
the audit reputation code. If there's an additional reputation update to
be made based on the automatic offline audit these nodes would get, the
overlay DB rejects it anyway.
Change-Id: I48ba151aa640b32f87e1285521be465d1f692a78
This is first attempt to use AliasPieces inastead Pieces with
segments/range loop. So far we were always using Pieces
which are always converted from AliasPieces for easy use.
Side effect is that using NodeID with loop observers is heavy
e.g. we are using maps which behaves slower with NodeIDs.
We are starting with audit observer because it's easy to change
it as in feact it doesn't need access to real NodeID at all. We just
need to reference node in some way and this way is NodeAlias.
Results of BenchmarkRemoteSegment:
name old time/op new time/op delta
RemoteSegment/Cockroach/multiple_segments-8 1.79µs ± 6% 0.03µs ± 4% -98.29% (p=0.008 n=5+5)
name old alloc/op new alloc/op delta
RemoteSegment/Cockroach/multiple_segments-8 0.00B 0.00B ~ (all equal)
name old allocs/op new allocs/op delta
RemoteSegment/Cockroach/multiple_segments-8 0.00 0.00 ~ (all equal)
Change-Id: Ib7fc87e568a4d3a9af27b5e3b644ea68ab6db7aa
Additional elements added:
* monkit metric for observers methods like Start/Fork/Join/Finish to
be able to check how much time those methods are taking
* few more logs e.g. entries with processed range
* segmentsProcessed metric to be able to check loop progress
Change-Id: I65dd51f7f5c4bdbb4014fbf04e5b6b10bdb035ec
Add notifications for free account limits for segment usage
and update to follow the figma designs.
Issue: https://github.com/storj/storj/issues/5482
Change-Id: I8a2fe38d609d53e09bf5074484cedc343223bffd
stripecoinpayments.mockStripeClient implements the
stripecoinpayments.StripeClient interface which is used to interact with
stripe's APIs in production. We use it in tests to simulate API calls to
stripe. This change introduces new stripecoinpayments package level
constants: TestPaymentMethodsNewFailure and
TestPaymentMethodsAttachFailure. These can be passed as part of the
arguments their respective stripe mock methods to signal to the method
that it should return an error. This is useful so that we can test how
layers on top of the StripeClient interface handle errors.
Change-Id: Ib64c08ba1e91f31e755b66a1ad563c3b6e77f6f0
Add new console service payments methods ApplyCoupon and
ApplyFreeTierCoupon. ApplyCoupon applies a coupon to an account based on
the coupon ID passed to it. ApplyFreeTierCoupon applies the satellite
configured free tier coupon to the account.
Change-Id: Ic221092278553a79207ac2a0c9229c374d76c881
Several tests using `(*ECRepairer).Get()` have begun to exhibit flaky
results. The tests are expecting to see failures in certain cases, but
the failures are not present. It appears that the cause of this is that,
sometimes, the fastest good nodes are able to satisfy the repairer
(providing RequiredCount pieces) before the repairer is able to identify
the problem scenario we have laid out.
In this commit, we add an argument to `(*ECRepairer).Get()` which
specifies how many failure results are expected. In normal/production
conditions, this parameter will be 0, meaning Get need not wait for
any errors and should only report those that arrived while waiting for
RequiredCount pieces (the existing behavior). But in these tests, we can
request that Get() wait for enough results to see the errors we are
expecting.
Refs: https://github.com/storj/storj/issues/5593
Change-Id: I2920edb6b5a344491786aab794d1be6372c07cf8
We are supposed to wait for some amount of time after a timed-out audit
before retrying the audit on the contained node. We are also supposed to
wait for some amount of time before subsequent retries, if they are
necessary. The test added here tries to assure that those delays happen,
as far as it is possible to assure that a delay will happen in computer
code.
The previous behavior of the system was, in fact, to carry out
Reverifies as soon as a worker could retrieve the job from the
reverification queue. That's not a very major problem, as subsequent
retries do have a delay and the node does get several retries. Still, it
was not ideal, and this test exposed that mismatch with expectations, so
this commit includes a minor change to effect that pause between verify
and the first reverify.
Refs: https://github.com/storj/storj/issues/5499
Change-Id: I83bb79c166a458ba59a2db2d17c85eca43ca90f0
While working on fixing listing for committed objects we didn't fix
the same case for pending objects. For case were we have many
pending objects under different locations we need to set cursor
version to highest value to avoid duplicates.
For case where we have many pending objects under the same location
we will need to make a separate fix.
https://github.com/storj/storj/issues/5570
Change-Id: Id5c8eb728868e8e1177fdbcf65a493142be4eaf0
Adds a feature flag for the new all projects dashboard. It defaults to false.
Issue: https://github.com/storj/storj/issues/5514
Change-Id: I160904eccae7d30e05b734e69600725702b16aca
Create an endpoint in the Admin Api to be able to update a user’s limits
for all existing and new projects. Also added a GET endpoint to return
user's limits.
Fixes: https://github.com/storj/storj/issues/5395
Change-Id: I2c093dc08ebf79a4318391e63a37da4d2b403547
We have an issue where object can appear in two different listing pages.
It's because protobuf listing cursor doesn't have version included and
now we can have internally versions higher than 1. On satellite side
version 1 was always used as a default cursor version.
As a workaround for existing implementation of libuplink library we will
use always maximum version for listing cursor on satellite side.
Fixing protobuf and libuplink implementation will happen later.
https://github.com/storj/storj/issues/5570
Change-Id: Ibd27b174556c9d8b8bd60fab8cff7862fd11e994
This change adds support for project public id to the bucket-names and
usage-report endpoints.
Issue: https://github.com/storj/storj/issues/5578
Change-Id: I2429ebebe52dfc8217fc40f4691e7bc473b805fb
Peer for generating bloom filters will be able to use ranged loop.
As an addition some cleanup were made:
* remove unused parts of GC BF peer (identity, version control)
* added missing Close method for ranged loop service
* some additional tests added
https://github.com/storj/storj/issues/5545
Change-Id: I9a3d85f5fffd2ebc7f2bf7ed024220117ab2be29
This modification introduce support of the new "desired node" field of download segment/object.
This can be used to request more nodes than the suggested minimum. It can be used to achieve better performance in exchange of using more bandwidth. (more parallel downloads).
Change-Id: Ia167d6979e6d70a597c85070a4ccd1c3a573e406
Most of our (~integration) tests based on testplanet runner.
However running testplanet for each test make the testing process slow.
It seems to be better to use real unit tests (without db dependency) when it's possible.
This patch makes small modification to make it possible to test orders.Service with real unit test.
As the existing unit test of `service.go` is isolated with `_test` package name, it's moved to an `_integration_test.go` file to make place for the unit test.
Change-Id: Ia69f26a34e2c48d230d8d36c2040dd02a60455a6
