Only API peer needs access to order DB (and rollups cache) because it's
only place where we are creating orders for PUT and GET operations. For
other peers like auditor and repairer we can set noop implementation to
reduce number of dependencies needed for them.
Change-Id: Ic32d1879f0b97ffc4516f401898e31e95ae892e4
It was surprising that `satellite auditor` complained about SMTP mail settings, even if it's not supposed to sending any mail.
Looks like we can remove the mail service dependency, as it's not a hard requirement for overlay.Service.
Change-Id: I29a52eeff3f967ddb2d74a09458dc0ee2f051bd7
I tried to configure a satellite service and got this error:
```
DEBUG process/exec_conf.go:408 Unrecoverable error {"error": "missing port in address"}
```
It took some time to realize that I forgot to set the SMTPServerAddress.
This patch makes it easier to detect similar problem (detailed error message), and makes SMTP parameters optional if no real mail sending is used (simulated or nomail)
Change-Id: I32535a7c8d6529e19e4d919806f42ba430d074a5
Up to now, we have been implementing the DistinctIP preference with code
in two places:
1. On check-in, the last_net is determined by taking the /24 or /64
(in ResolveIPAndNetwork()) and we store it with the node record.
2. On node selection, a preference parameter defines whether to return
results that are distinct on last_net.
It can be observed that we have never yet had the need to switch from
DistinctIP to !DistinctIP, or from !DistinctIP to DistinctIP, on the
same satellite, and we will probably never need to do so in an automated
way. It can also be observed that this arrangement makes tests more
complicated, because we often have to arrange for test nodes to have IP
addresses in different /24 networks (a particular pain on macOS).
Those two considerations, plus some pending work on the repair framework
that will make repair take last_net into consideration, motivate this
change.
With this change, in the #2 place, we will _always_ return results that
are distinct on last_net. We implement the DistinctIP preference, then,
by making the #1 place (ResolveIPAndNetwork()) more flexible. When
DistinctIP is enabled, last_net will be calculated as it was before. But
when DistinctIP is _off_, last_net can be the same as address (IP and
port). That will effectively implement !DistinctIP because every
record will have a distinct last_net already.
As a side effect, this flexibility will allow us to change the rules
about last_net construction arbitrarily. We can do tests where last_net
is set to the source IP, or to a /30 prefix, or a /16 prefix, etc., and
be able to exercise the production logic without requiring a virtual
network bridge.
This change should be safe to make without any migration code, because
all known production satellite deployments use DistinctIP, and the
associated last_net values will not change for them. They will only
change for satellites with !DistinctIP, which are mostly test
deployments that can be recreated trivially. For those satellites which
are both permanent and !DistinctIP, node selection will suddenly start
acting as though DistinctIP is enabled, until the operator runs a single
SQL update "UPDATE nodes SET last_net = last_ip_port". That can be done
either before or after deploying software with this change.
I also assert that this will not hurt performance for production
deployments. It's true that adding the distinct requirement to node
selection makes things a little slower, but the distinct requirement is
already present for all production deployments, and they will see no
change.
Refs: https://github.com/storj/storj/issues/5391
Change-Id: I0e7e92498c3da768df5b4d5fb213dcd2d4862924
Segments loop have build-in sanity check to verify if number of segments
processed by loop is roughly fine. We want to have the same verification
for ranged loop.
https://github.com/storj/storj/issues/5544
Change-Id: Ia19edc0fb4aa8dc45993498a8e6a4eb5928485e9
If the session_minutes column in user_settings is not null, use this
value to determine the expiration when creating or refreshing a session.
github issue: https://github.com/storj/storj/issues/5487
Change-Id: I1f777f7ffcd4018b0c40f0b26c8a50de85a31909
We should return public project id instead of regular one when quering project charges so that it is consistent on a client side.
This is a fix for an issue with displaying project name on a Billing screen.
Issue:
https://github.com/storj/storj/issues/5641
Change-Id: Ic0c544dbe7369aa88f3a7c97a2f5de5bc854e1fd
We changed primary key for bucket_bandwidth_rollups table. Now we
need to do some cleanup in places like structs, sorting methods or SQL
queries.
Change-Id: Ida4f874f161356df193379a53507602e04db1668
The tests were using global variables for keeping the mock state, which
was indexed by the satellite ID. However, the satellite ID-s are
deterministic and it's possible for two tests end up using the same
mocks.
Instead make the mock creation not depend on the satellite ID and
instead require it being configured via paymentsconfig.
This fixes TestAutoFreezeChore failure.
Change-Id: I531d3550a934fbb36cff2973be96fd43b7edc44a
In the past we set incorrect primary key for bucket_bandwidth_rollups
table and bucket name was first column of key instead of project id.
This change fixes this problem.
We will alter primary key manually for production satellites. That's
why migration code is checking if change was already applied.
https://github.com/storj/storj/issues/5332
Change-Id: I7dd555eb0c2f6fda1c90eaed08fa847b7f273bc7
An endpoint has been added that returns whether a pricing package is
available for a user to purchase. This will be used to conditionally
skip the pricing plan selection step of the onboarding tour.
Change-Id: I8c02a4e474e5f0f80778453b7daf674c8da64306
If a user changes password while a password recovery token exists,
delete the token so the reset password email doesn't work.
github issue: https://github.com/storj/storj-private/issues/149
Change-Id: Ibb0c13076e182c46501c1d7ac92d3fecc7d5ee58
Add passphrase_prompt column to user_settings table to store a boolean
whether a user would like to be prompted to enter a passphrase when
entering a new project.
Issue https://github.com/storj/storj/issues/5613
Change-Id: I5d086df277aff458453343c7c38a379e7b1b8bf9
Earlier we made a change to not cancel flushing orders when flushing
was triggered by orders endpoint method but we missed a case
where it can be also triggered (and canceled) by metainfo endpoints
method. This change moves ignoring context cancellation deeper.
Change-Id: Id43176f552efc3167345783f73aab885411ac247
This change adds a new chore that will check for failed invoices and
potentially freeze corresponding accounts.
It makes slight modifications to stripemock.go and invoices.go (adding
stripe CustomerID to the Invoice struct).
Issue: https://github.com/storj/storj-private/issues/140
Change-Id: I161f4037881222003bd231559c75f43360509894
A pricing plan selection step for users with a recognized partner has
been added to the beginning of the onboarding tour. Once visited, users
have the option of purchasing the pricing plan associated with their
partner or proceeding as a paid or free tier user.
Resolves storj-private#118
Resolves storj-private#126
Change-Id: I3b423194d96deaf87cf9807a766bf4d04fbcf86d
add new config to the satellite admin: --admin.groups.limit-update.
This can be used as an alternate means of authentication if the request
is coming from the oauth proxy.
Change-Id: Ic2de13862e6414244b060c66a0f2bed72097cbad
Commit fb59974 disabled usage price overrides because of a failing
test. This change reenables it while resolving the issue that caused
the test to fail.
The previous version of the test passed Gerrit verification and was
merged, but it failed for the primary Jenkins pipeline after merge.
This is due to a difference in how the Jenkins build runs Cockroach
and Postgres for each pipeline.
This commit rewrites the test to be safe for concurrent execution by
ensuring any mutable variables are defined within each test so that
shared state across tests is reduced.
Change-Id: Ia4566c9cd2d698afdb2caa4b7e2808b17e18de4e
It was possible to get into a situation where successfulPieces =
es.RequiredCount(), errorCount < minFailures, and inProgress == 0 (when
the succeeding gets all completed before the failures), whereupon the
last goroutine in the limiter would sit and wait forever for another
goroutine to finish.
This change corrects the handling of that situation.
As an aside, this is really pretty confusing code and we should think
about redoing the whole function.
Change-Id: Ifa3d3ad92bc755e563fd06b2aa01ef6147075a69
Also, fix nodevents iota usage. Type is directly used in the database
hence we should avoid iota, because it's easy to mess-up the constant
values.
Change-Id: I32bf6d0502fbf3d18660dab5b1dd0b5627b444df
Add new purchase-package endpoint to Server. The endpoint can be enabled
or disabled by a new config, --console.pricing-packages-enabled.
The purchase-package endpoint applies a coupon and adds and charges a
credit card if user's useragent is a partner with a configured package
plan.
github issue: https://github.com/storj/storj-private/issues/125
Change-Id: I0d6ccccd6874ddba360c45f338fd1c44f95e135a
Project usage price overriding has been removed because it produces
incorrect results when tested. It should not be re-implemented until
the issues it causes are resolved.
Change-Id: Ic92eff374c9af4fea3bf32782a72303a7978b055
The update user API function was defining a parameter that the API
calling function doesn't receive, so all the values were shifted by
1 after this extra parameter and the calling function was sending these
values to the next one updating the fields that the user was not
expecting.
The API server doesn't allow to update this removed parameter, so it
was likely a bug caused by copy-pasta.
Change-Id: I52a2b66abc1f6e5cb0070dc6fd230c30d1133dcf
A new struct containing configuration values for the satellite frontend
is populated and served to the client. In the future, this struct will
be used to configure the frontend rather than using Go templates and
meta tags to do so.
References #5494
Change-Id: Ie2c8a140d53b20fab159d34f2e3767e5dbd43bdb
This code is essentially replacement for eestream.CalcPieceSize. To call
eestream.CalcPieceSize we need eestream.RedundancyStrategy which is not
trivial to get as it requires infectious.FEC. For example infectious.FEC
creation is visible on GE loop observer CPU profile because we were
doing this for each segment in DB.
New method was added to storj.Redundancy and here we are just wiring it
with metabase Segment.
BenchmarkSegmentPieceSize
BenchmarkSegmentPieceSize/eestream.CalcPieceSize
BenchmarkSegmentPieceSize/eestream.CalcPieceSize-8 5822 189189 ns/op 9776 B/op 8 allocs/op
BenchmarkSegmentPieceSize/segment.PieceSize
BenchmarkSegmentPieceSize/segment.PieceSize-8 94721329 11.49 ns/op 0 B/op 0 allocs/op
Change-Id: I5a8b4237aedd1424c54ed0af448061a236b00295
Purchase collects a payment from user using the specified price,
description, and payment method. This is implemented at the lower layers
using the payments.Invoices interface. There are also additions to
stripecoinpayments stripemock to simulate errors returned from stripe
invoices New and Pay. If an invoice exists with the same description, if
it is a draft, it is deleted and a new one is created and paid. If it is
open, pay it and don't create a new one.
Change-Id: Ic3147434bc44a0777ecbedb3a4ed4c768eb02ea3
This change removes the trailing slash from the account activation and
password recovery URLs, making them consistent with the rest. The URLs'
previous forms are still supported, however, in order to not invalidate
emails containing them.
Resolvesstorj/customer-issues#491
Change-Id: Ie774a87698d8e9edd1836611968fc3911c6cc56f
If a node is already disqualified or exited, don't bother going through
the audit reputation code. If there's an additional reputation update to
be made based on the automatic offline audit these nodes would get, the
overlay DB rejects it anyway.
Change-Id: I48ba151aa640b32f87e1285521be465d1f692a78
This is first attempt to use AliasPieces inastead Pieces with
segments/range loop. So far we were always using Pieces
which are always converted from AliasPieces for easy use.
Side effect is that using NodeID with loop observers is heavy
e.g. we are using maps which behaves slower with NodeIDs.
We are starting with audit observer because it's easy to change
it as in feact it doesn't need access to real NodeID at all. We just
need to reference node in some way and this way is NodeAlias.
Results of BenchmarkRemoteSegment:
name old time/op new time/op delta
RemoteSegment/Cockroach/multiple_segments-8 1.79µs ± 6% 0.03µs ± 4% -98.29% (p=0.008 n=5+5)
name old alloc/op new alloc/op delta
RemoteSegment/Cockroach/multiple_segments-8 0.00B 0.00B ~ (all equal)
name old allocs/op new allocs/op delta
RemoteSegment/Cockroach/multiple_segments-8 0.00 0.00 ~ (all equal)
Change-Id: Ib7fc87e568a4d3a9af27b5e3b644ea68ab6db7aa
Additional elements added:
* monkit metric for observers methods like Start/Fork/Join/Finish to
be able to check how much time those methods are taking
* few more logs e.g. entries with processed range
* segmentsProcessed metric to be able to check loop progress
Change-Id: I65dd51f7f5c4bdbb4014fbf04e5b6b10bdb035ec
