Metainfo needs to know rate and burst limit to be able to limit users
requests. We made cache for per project limiter but to make single
instance we need to know about limits. So far we were doing direct DB
call to get rate/burst limit for project but it's generating lots of
DB requests and can be easily cached as we even have project limit cache.
This change extends project limit cache with rate/burst limit and starts
using this change while creating project limiter instance for metainfo.
Because data size kept in project limit cache is quite small this change
also bumps a bit default capacity of the cache.
Fixes https://github.com/storj/storj/issues/5663
Change-Id: Icb42ec1632bfa0c9f74857b559083dcbd054d071
We just fixed case were project limit cache was not used properly. This
is test case to cover that fix.
Change-Id: Iee467f0a46836860a14ab6238a9842ffbf54ed4c
We have lots of direct DB requests to get API keys. It should be handled
by cache but default value is very low at the moment.
Fixes https://github.com/storj/storj/issues/5665
Change-Id: I214ebebd6e397cacff80b2f36dc4a2eea388f93d
This might be pretty awful, but at least it is a complete and non-flaky
solution.
**Only when using the rollingupgrade test** (which implies a throwaway
satellite and also a PostgreSQL backend), create a trigger on the nodes
table which forces last_net to be equal to last_ip_port always.
Change-Id: I8448cf131e46576d96a414d06780270c7b2b1892
Currently, to get number of entries in segments table we are doing
heavy SELECT count(*) operation. For biggest satellite it's taking
25min now. We are using this method to get stat before and after
segments loop so it adds almost 1h to overall loop time.
With current version of crdb we are using this additional code won't be
used because global configuration for stats refresh rate is inaccurate
for such large table like `segments`. Soon we should be able to upgrade
crdb and be able to adjust refresh rate per table and configure it to
satisfy defined threshold.
https://github.com/storj/storj/issues/5544
Change-Id: I05cfd9154f08894d2bc56bf716b436d1b03b87f1
We do regular testing with executing uplink. But sometimes the recorded execution time showed spikes.
Would be nice to know the reason of the spikes (just internet blip, or something what we should be worried about).
We can collect distributed traces, but it's not easy to find the right trace in Jaeger.
* We can provide a random trace-id, but it should be persisted / processed
* We can also save standard output and use `--trace-verbose` which prints out the used trace id, but it's also complicated to collect all of them in a DB
Would be nice to attach additional metadata to traces to make sure that we can filter all traces of one specific kind of test.
This patch provides this feature:
* It always adds hostname to the trace (if you opt-in to distributed tracing, which is turned off by default)
* Additional tags can be defined with CLI flag
Tags can be used to find the right trace in Jaeger (or in Elastic search backend of Jaeger).
Change-Id: I08f10023bbebd783f812cfca95ac6237360ac2b0
The Coupons area of the Billing page has been updated to match our
designs.
Resolvesstorj/storj-private#172
Change-Id: I0f3d7f3818f47a1bbc6a2dba930ab429f83f92c2
A field has been added to the coupon struct indicating whether it is
associated with a partner's pricing package. This is required to
alter the appearance of partner coupons in the satellite frontend.
References storj/storj-private#172
Change-Id: Ie48ae3902aaa108abf9a399242a0cd98cb53d1c3
This change sends an event to segment for when a user is unfrozen.
It also moves freeze and warning event triggers from the autofreeze
chore to the account freeze service.
Change-Id: I5c0522b921b7baf52d6db5eb7ef841c08644a461
A package has been written to generate a TypeScript file containing
classes corresponding to the frontend config. These classes will be
used across the frontend once we no longer reference meta tag values
for configuration.
References #5494
Change-Id: If425035892773167ac6d9fbfae8140cab79fbb70
The code responsible for generating TypeScript classes has been
separated from the rest of the TypeScript generation code so that other
packages may take advantage of this functionality.
References #5494
Change-Id: I97eabd430bd6a5f748eafaf8b1d783977e75e660
This change reworks the allowedAuthorization function to check what
groups the user is a part of to determine if authorization should be
granted. By wrapping each handler with withAuth, we can specify the
allowed groups for each api method individually.
github issue: https://github.com/storj/storj/issues/5565
Change-Id: I1804dda04d5b16d19e93bd7199fb3fc89fca1294
This changes the icon, and text of the "Details" dropdown item to
"Preview". It also fixes text alignment in the dropdown menu.
Issue: https://github.com/storj/storj/issues/5643
Change-Id: Ib7ab3bc6fd98b9319ec3c3af5644aee6d1942335
Remove generate-missing-project-salt migration tool code and related
tests. This migration has already been run and this code is no longer
needed.
Issue https://github.com/storj/storj-private/issues/163
Change-Id: I4e36dcd95a07c5305c597113a7fd08148e100ccc
It looks that at some point we broke how project limits cache is used
and we were missing cache in most critical paths (upload/download).
This is fix for this issue.
I also adjusted cache methods naming.
Change-Id: Ic98372779a39365d0920fe3943f1f7a68b064173
This change adds a card to the billing overview page, which shows the
user's token balance from coinpayments.
Issue: https://github.com/storj/storj-private/issues/151
Change-Id: I11e295b48791b32b745cb7a11c5b4aad6b56618e
This test involves a satellite with dev defaults (DistinctIP=no) being
upgraded past commit 2522ff09b6, which
means we need to run the dev-defaults-satellite-upgrade migration SQL
to avoid getting DistinctIP=yes behavior (which breaks the tests).
Change-Id: I29fb596d1ffa568dad635d98cfe9abacd3aaa48f
The styling of pricing plan components has been updated to match our
designs.
References storj/storj-private#172
Change-Id: I61a2536267c41292abd9e4860b5bd45fafb61408
This change implements a new account settings page for the all projects
dashboard.
Issue: https://github.com/storj/storj/issues/5514
Change-Id: Id777cd5c1efe3fa4b40234771ae2a99cc5cb9dd3
This change implements onboarding for the all projects dashboard.
Issue: https://github.com/storj/storj/issues/5514
Change-Id: If8a48c21e6df264c84d362d17bf01d770d9ca768
Added truncating for long names in access grant and project member tables.
Added title to all table items so that full value is displayed on hover.
Issue:
https://github.com/storj/storj/issues/5130
Change-Id: I7f81d88c8ff4db606bdfeb15406ba3ee3fcec968
* testsuite/playwright: copys files to storj repo
This PR copies all the playwright ui tests to the storj repository.
* CI: Jenkinsfile.ui
Jenkinsfile.ui builds all necessary items for running storj binaries, installs all items for playwright ui tests and executes the test. Note that we aren't running regular unit tests and we are not linting the code as this is occuring in the public instance. Furthermore, we still need to implement the starting of storj services as well as typescript linters for the playwright tests.
* testsuite: playwright ui tests
This PR copies all playwright ui tests files from the qa repo.
* testsuite: playwright ui tests
This PR copies all playwright ui tests files from the qa repo.
Only API peer needs access to order DB (and rollups cache) because it's
only place where we are creating orders for PUT and GET operations. For
other peers like auditor and repairer we can set noop implementation to
reduce number of dependencies needed for them.
Change-Id: Ic32d1879f0b97ffc4516f401898e31e95ae892e4
It was surprising that `satellite auditor` complained about SMTP mail settings, even if it's not supposed to sending any mail.
Looks like we can remove the mail service dependency, as it's not a hard requirement for overlay.Service.
Change-Id: I29a52eeff3f967ddb2d74a09458dc0ee2f051bd7
I tried to configure a satellite service and got this error:
```
DEBUG process/exec_conf.go:408 Unrecoverable error {"error": "missing port in address"}
```
It took some time to realize that I forgot to set the SMTPServerAddress.
This patch makes it easier to detect similar problem (detailed error message), and makes SMTP parameters optional if no real mail sending is used (simulated or nomail)
Change-Id: I32535a7c8d6529e19e4d919806f42ba430d074a5
Up to now, we have been implementing the DistinctIP preference with code
in two places:
1. On check-in, the last_net is determined by taking the /24 or /64
(in ResolveIPAndNetwork()) and we store it with the node record.
2. On node selection, a preference parameter defines whether to return
results that are distinct on last_net.
It can be observed that we have never yet had the need to switch from
DistinctIP to !DistinctIP, or from !DistinctIP to DistinctIP, on the
same satellite, and we will probably never need to do so in an automated
way. It can also be observed that this arrangement makes tests more
complicated, because we often have to arrange for test nodes to have IP
addresses in different /24 networks (a particular pain on macOS).
Those two considerations, plus some pending work on the repair framework
that will make repair take last_net into consideration, motivate this
change.
With this change, in the #2 place, we will _always_ return results that
are distinct on last_net. We implement the DistinctIP preference, then,
by making the #1 place (ResolveIPAndNetwork()) more flexible. When
DistinctIP is enabled, last_net will be calculated as it was before. But
when DistinctIP is _off_, last_net can be the same as address (IP and
port). That will effectively implement !DistinctIP because every
record will have a distinct last_net already.
As a side effect, this flexibility will allow us to change the rules
about last_net construction arbitrarily. We can do tests where last_net
is set to the source IP, or to a /30 prefix, or a /16 prefix, etc., and
be able to exercise the production logic without requiring a virtual
network bridge.
This change should be safe to make without any migration code, because
all known production satellite deployments use DistinctIP, and the
associated last_net values will not change for them. They will only
change for satellites with !DistinctIP, which are mostly test
deployments that can be recreated trivially. For those satellites which
are both permanent and !DistinctIP, node selection will suddenly start
acting as though DistinctIP is enabled, until the operator runs a single
SQL update "UPDATE nodes SET last_net = last_ip_port". That can be done
either before or after deploying software with this change.
I also assert that this will not hurt performance for production
deployments. It's true that adding the distinct requirement to node
selection makes things a little slower, but the distinct requirement is
already present for all production deployments, and they will see no
change.
Refs: https://github.com/storj/storj/issues/5391
Change-Id: I0e7e92498c3da768df5b4d5fb213dcd2d4862924
Segments loop have build-in sanity check to verify if number of segments
processed by loop is roughly fine. We want to have the same verification
for ranged loop.
https://github.com/storj/storj/issues/5544
Change-Id: Ia19edc0fb4aa8dc45993498a8e6a4eb5928485e9
If the session_minutes column in user_settings is not null, use this
value to determine the expiration when creating or refreshing a session.
github issue: https://github.com/storj/storj/issues/5487
Change-Id: I1f777f7ffcd4018b0c40f0b26c8a50de85a31909
We should return public project id instead of regular one when quering project charges so that it is consistent on a client side.
This is a fix for an issue with displaying project name on a Billing screen.
Issue:
https://github.com/storj/storj/issues/5641
Change-Id: Ic0c544dbe7369aa88f3a7c97a2f5de5bc854e1fd
quic is still configurable based on the quic rollout
environment variables in storj.io/common. this stops
using a method removed in:
https://review.dev.storj.io/c/storj/uplink/+/9815
Change-Id: Ibfe28cfb19e5672630970b9e2c8c6ac0c98d4822
We decided to remove sanitizing dependency and restrict v-html with 2 exceptions for web/satellite (we control the content in those cases)
Issue:
https://github.com/storj/storj-private/issues/148
Change-Id: Ic6b2e894d20e7f0553f759ede51845a10831e890
Divide single error event into 2 separate events.
Those are list objects event and change route event.
Issue:
https://github.com/storj/storj-private/issues/164
Change-Id: I3efe0f2d14fceb5841b6d0ff6db685540351281a
We changed primary key for bucket_bandwidth_rollups table. Now we
need to do some cleanup in places like structs, sorting methods or SQL
queries.
Change-Id: Ida4f874f161356df193379a53507602e04db1668
