Commit Graph

772 Commits

Author SHA1 Message Date
Egon Elbre
ff22fc7ddd all: fix deprecated ioutil commands
Change-Id: I59db35116ec7215a1b8e2ae7dbd319fa099adfac
2022-10-11 15:27:29 +00:00
dlamarmorgan
d39caf7674 cmd/satellite: change invoicing flags to iso8601
Update input parameters for invoicing commands to use iso8601 standard formatting.

Change-Id: I3b4a7df268e8f1a5cd51b0dab03ed38e13a76554
2022-10-07 07:58:23 -07:00
Igor
96d4968041
use python2 for draft release (#5216) 2022-09-29 16:22:08 +03:00
Fadila Khadar
02924d0ded satellite/metainfo: temporary feature flag for listing query testing
Fixes: https://github.com/storj/storj/issues/5144

Change-Id: I7650f4d5dd0378e2246339e79710a695996a845c
2022-09-28 17:46:52 +00:00
Igor
352e937813
scripts: add changelog to release draft (#4791)
* scripts: add changelog to release draft

* scripts: fix run command

* rename variable

* rename variable

* fix command

* add comment

* rename variable

Co-authored-by: Stefan Benten <mail@stefan-benten.de>
2022-09-28 19:17:40 +03:00
dlamarmorgan
88328ab240 satellite/payments/billing/billing.go: change default loop interval
Change the default loop interval for querying for new payments and adding them into the billing table from 1 minute to 15 seconds.

Change-Id: I26cf4a764cbe1de4c9b839ad60352374d8231522
2022-09-27 11:42:28 -07:00
dlamarmorgan
0b468aca56 satellite/payments/storjscan/storjscan.go: change default block confirmations
Change the default number of required block confirmations for a payment to be confirmed from 12 to 15.

Change-Id: I44c258134c293e7691623bc00c504130aa69a96a
2022-09-23 14:30:05 +00:00
Artur M. Wolff
fe85fefcc5 ci: add check-downgrades checks
Change-Id: Idfe81897dc701c24c0f81b0021cb1d640e8224ac
2022-09-23 11:15:27 +00:00
Michal Niewrzal
4e71dd302a satellite/{metainfo,metabase}: prepare feature flag MultipleVersions
We will introduce new logic for creating new objects (BeginObject).
Instead of using single version internally (1) we will be selecting first
available version during object creation. Because we need to be sure
that everything is wired up correctly we need a feature flag to be
able to control if new feature is enabled.

Change-Id: If0f8496397130811f43bf9db9fdcc2b30cd2e4ca
2022-09-21 13:07:13 +00:00
Erik van Velzen
e6b5501f9b satellite/gc/sender: new service to send retain filters
Implement a new service to read retain filter from a bucket and
send them out to storagenodes.

This allows the retain filters to be generated by a separate command on
a backup of the database.

Paralellism (setting ConcurrentSends) and end-to-end garbage collection
tests will be restored in a subsequent commit.

Solves https://github.com/storj/team-metainfo/issues/121

Change-Id: Iaf8a33fbf6987676cc3cf74a18a8078916fe673d
2022-09-20 11:49:40 +00:00
Artur M. Wolff
5150ee5001 mod: bump storj.io/private
Change-Id: I534aeedead1f1a2c5099e80b49fa5a8ea599fac8
2022-09-19 13:09:31 +00:00
Michal Niewrzal
a3d9630336 scripts: move tests to separate folders
Doing some cleanup in "scripts" folder. All integration like tests are
moved under "test" directory (integration, bc, redis) and bash scripts
are adjusted to reflect new location.

As an addition "scripts/install-awscli.sh" was deleted as it was not
used.

Change-Id: I152905c4258f471a71f2d0e8731d91bb075e99c1
2022-09-16 11:10:18 +00:00
Michal Niewrzal
158eb2381e satellite/gc/bloomfilter: uploading bloom filters
We would like to have separate process/command to collect bloom
filters from source different than production DBs. Such process will
use segment loop to build bloom filters for all storage nodes and
will send it to Storj bucket.

This change add main logic to new service. After collecting all bloom
filters with segment loop and piece tracker all filters are marshaled
and packed into zip files. Each zip contains up to "ZipBatchSize" bloom
filters and it's uploaded to specified in configuration bucket.

All uploaded objects have specified expiration time to not delete them
manually.

Updates https://github.com/storj/team-metainfo/issues/120

Change-Id: I2b6bc02a7dd7c3a639e75810fd013ae4afdc80a2
2022-09-12 08:33:53 +00:00
Michal Niewrzal
68f6d93f29 satellite/gc/bloomfilter: add service to collect bloom filters
We would like to have separate process/command to collect bloom
filters from source different than production DBs. Such process will
use segment loop to build bloom filters for all storage nodes and
will send it to Storj bucket. This this initial change to add such
service. Added service is joining segment loop and collects all
bloom filters.

Sending bloom filters to the bucket will be added as a subsequent
change.

Updates https://github.com/storj/team-metainfo/issues/120

Change-Id: I2551723605afa41bec84826b0c647cd1f61f3b14
2022-09-02 08:10:46 +00:00
Jeremy Wharton
3f26cc599f satellite/console,web/satellite: invalidate sessions after inactivity
Sessions now expire after a much shorter amount of time, requiring
clients to issue API requests for session extension. This is handled
behind the scenes as the user interacts with the page, but once session
expiration is imminent, a modal appears which informs the user of his
inactivity and presents him with the choice of loging out or preserving
his session.

Change-Id: I68008d45859c814a835d65d882ad5ad2199d618e
2022-08-23 15:51:05 +00:00
paul cannon
0dcc0a9ee0 satellite/reputation: reconfigure lambda and alpha
This is in response to community feedback that our existing reputation
calculation is too likely to disqualify storage nodes unfairly with
extreme swings up and down.

For details and analysis, please see the data_loss_vs_dq_chance_sim.py
tool, the "tuning reputation further.ipynb" Jupyter notebook in the
storj/datascience repository, and the discussion at

    https://forum.storj.io/t/tuning-audit-scoring/14084

In brief: changing the lambda and initial-alpha parameters in this way
causes the swings in reputation to be smaller and less likely to put a
node past the disqualification threshold unfairly.

Note: this change will cause a one-time reset of all (non-disqualified)
node reputations, because the new initial alpha value of 1000 is
dramatically different, and the disqualification threshold is going to
be much higher.

Change-Id: Id6dc4ba8fde1be3db4255b72282207bab5491ca3
2022-08-17 18:52:53 +00:00
dlamarmorgan
f596f72f44 satellite/payments/storjscan: add billing update to chore
Update chore to add confirmed transactions to the billing table.

Change-Id: I3c9a98c2ddc96f8a8905250376a1f5490d810277
2022-08-16 11:18:15 -07:00
Vitalii
0ec28ca623 web/satellite: add token funds modal
Created new modal which shows user their native STORJ token wallet address.
There are QR and copy buttons.
It will be used only in new billing screen.

Change-Id: Icef3c8668c548b779c07fe2b85eb5761cd1221a3
2022-08-15 14:17:30 +03:00
Márton Elek
96b88e286e build: fix conditional run of web builds
Jenkins doesn't do a very good job with identifying what has been changed.

While it has a syntax to defined patterns, it compares the current build with the previous build (in case of git-verify it can be a totally different branch) instead of checking the HEAD commit.

This patch introduces shell scripts to do this better:
 * It doesn't depend on Jenkins any more
 * It can be executed locally
 * It can detect web changes properly (see the relation change as an example).

Change-Id: I9d37775e3818c08c4aa96ffb78f84d57f28a2c95
2022-08-11 14:34:44 +00:00
Moby von Briesen
3f6bf64af9 satellite/console: Default NewProjectDashboard to "true"
We have enabled the new project dashboard in production. Change the
default to true so that we do not need an explicit configuration in
prod.

Change-Id: I0f93773965283e7b0682f6586685224281cbf78c
2022-08-06 01:00:31 +00:00
Vitalii
ad37ea4518 satellite/{web, console}: login captcha implemented
Implemented Recaptcha and Hcaptcha for login screen.
Slightly refactored registration page implementation.
Made 2 different login/registration captcha configs on server side to easily swap between captchas independently.

Issue: https://github.com/storj/storj/issues/4982

Change-Id: I362bd5db2d59010e90a22301893bc3e1d860293a
2022-08-03 23:02:27 +00:00
Jeff Wendling
f343b856dd go.mod: bump common/private/monkit/monkit-jaeger
this includes all of the fixes identified for end to end
tracing to finally work.

Change-Id: Ia433a2cf9130dc43b6810379af32f5c10f9d3888
2022-07-27 16:17:00 +00:00
Qweder93
2b24632610 {satellite/metainfo, satellite/accounting}: remove ValidateSegmentLimit config value and check
removed segment limit validation and checks in metainfo endpoint and accounting/projectusage
since feature is live and has always has segment limitation now

Resolves: https://github.com/storj/storj/issues/4470

Change-Id: I8cf87cbbc40ac61262f9f05e52573d3ae6410611
2022-07-27 15:36:53 +00:00
Egon Elbre
82fede2132 satellite/satellitedb: batch delete storage node tallies
Currently we have a significant number of tallies that need to be
deleted together. Add a limit (by default 10k) to how many will
be deleted at the same time.

Change-Id: If530383f19b4d3bb83ed5fe956610a2e52f130a1
2022-07-20 15:42:35 +03:00
paul cannon
2f20bbf4d8 satellite/reputation: add a reputation write cache
This should lower the amount of database load coming from
reputation updates.

Change-Id: Iaacfb81480075261da77c5cc93e08b24f69f8949
2022-07-14 21:40:16 +00:00
Yaroslav Vorobiov
de5951331a satellite: wire storjscan chore to core process
Adds storjscan chore and storjscan client to satellite chore.
Storjscan chore interval added to storjscan config.

Change-Id: I13bb8d83b14ca90f4735c9a10c361ad24a56b256
2022-07-14 15:07:52 +00:00
Vitalii
69dc9a4731 satellite/console: added new email which is sent on unknown password reset
Added new email html template.
It is sent when user tries to reset password with unknown or unverified account.
Made a couple of minor config changes.

Issue: https://github.com/storj/storj/issues/4913

Change-Id: I730f48b3478e302d1e38e1f8a27c75f66a8ba6fd
2022-07-14 14:32:59 +00:00
littleskunk
cf750716ea
satellite/console: enable new access grant flow (#4934) 2022-06-30 00:06:39 +02:00
JT Olio
18666b078a metabase: support disabling serverside copy when already enabled
Change-Id: I99b0b5597537f69fbdd8cab479e059ba4bf310bd
2022-06-24 17:56:43 +00:00
Márton Elek
204367b74d
build: premerge/verify Jenkinsfile (subsets of Jenkinsifle.public)
Change-Id: I33c65a37ecd8fcb89729b2a8db3906c6b8890ca3
2022-06-23 09:15:43 +02:00
Ivan Fraixedes
fc4b0e41f0 scripts: Change Storagenode arm32v6 to arm32v5
We don't build "multinode" nor "storagenode" docker images for armv6
architecture, instead we build for armv5.

Fix the script that publish the manifest for those images for a specific
tag to use armv5 for not failing when executing.

Change-Id: I7d859d8718240e1cd0dae6489e7e5c3b4068ff6e
2022-06-21 11:54:38 +00:00
Jeremy Wharton
58c5d44f44 satellite/console: integrate sessions into satellite UI
This change integrates the session management database functionality
with the web application. Claim-based authentication has been removed
in favor of session token-based authentication.

Change-Id: I62a4f5354a3ed8ca80272814aad2448f901eab1b
2022-06-13 08:02:02 +00:00
JT Olio
098ee71e03 test-sim-versions: write what bash step is executing
Change-Id: If1231f69601c1774bc6ce116f45105cc1b2f4e22
2022-06-10 19:33:03 +00:00
Yaroslav Vorobiov
62774768d9 satellite/payments/storjscan: update config and wrap service errors
Change-Id: I4f75375e9499119cbcce78c5e77d1950e4bae38b
2022-06-07 14:21:42 +00:00
Paul Willoughby
911cc1e163 satellite/contact: reject privateIPs in PingMe and CheckIn endpoints
prevent network enumeration by rejecting privateIPs in PingMe and
Checkin endpoints

Closes storj/storj-private#32

Change-Id: I63f00483ff4128ebd5fa9b7b8da826a5706748c9
2022-06-07 08:09:14 +00:00
Vitalii
ba58530089 {satellite}/web,console,testsuite: remove old navigation structure
Removed old satellite UI navigation structure.
Removed old feature flag.

Change-Id: Ic998886cf2e30ebd44e67a20fc53888103fe4b8d
2022-06-06 16:43:05 +03:00
dlamarmorgan
270204f352 satellite/{payments/storjscan,satellitedb}: Add wallet implementation
Add storjscan wallets implementation to the satellite. The wallets interface allows you to add and claim new wallets as called by the API. The storjscan specific implementation of this interface uses a wallets DB to associate the user to a wallet address, as well as a storjscan client to request and associate new wallets to the satellite.

Change-Id: I54081edb5545d4e3ee07cf1cce3d3e87cc00c4a1
2022-06-03 11:45:47 +00:00
cl-mitch
cbaca8b17e
web/satellite added new billing screen feature flag (#4836)
New feature flag implemented to enable the new billing flow.

Co-authored-by: Maximillian von Briesen <mobyvb@gmail.com>
2022-06-02 09:30:27 -05:00
littleskunk
6cdd250019
satellite/console: enable new object flow / onboarding (#4851) 2022-06-01 14:04:03 +02:00
littleskunk
3d1b469f35
satellite/console: disable account activation reminder email (#4844)
* satellite/console: disable account activation reminder email
2022-05-31 17:36:40 +02:00
Stefan Benten
8673583582 satellite/admin: fix console config handling
An older change plummed the full console config as subconfig of
the admin api configuration in. This bloated the generated satellite
configuration unnecessarily while also allow for confusion/mistakes.

Change-Id: Icf49cc1f147711e37e85f6eac1143fab8ddf1659
2022-05-27 22:26:06 +00:00
Ivan Fraixedes
54aa3adbf7 scripts: Fix lint to show out on error
`os/exec.Cmd.CombineOutput` runs the command, hence, it cannot be used
after calling the `Run` method.

Because `CombineOutput` already runs the command, we can use it directly
instead of `Run`.

Without this change if the command returns an non-zero code we get an
error because of the command already started and we don't get the
output.

Example removing a copyright notice from one file and running the linter
(only showing the affected printed line)

Without this fix

  2022/05/26 15:48:40 [/storj check-copyright] error  exec: already started

With this fix

  2022/05/26 16:22:40 [/storj check-copyright] error missing copyright certificate/doc.go: %!w(<nil>)

NOTE the `%!w(<nil)` is a bug in the check-copyright linter.

Change-Id: I40b64842028399b92a8982bfb143e1f87f92467b
2022-05-27 09:32:48 +00:00
Mya
b74340b1d6 scripts/lint: fix timing of lint tasks
Change-Id: I068cad315fd05ee235413d8230d454507b0a60e4
2022-05-25 13:24:03 -05:00
Mya
9153f37055 Makefile: run lint locally in docker
Our linting process depends heavily on custom tools and linting
configuration. To help improve this process for running on local
developer machines, we can run the various tasks in our existing CI
container.

Change-Id: I60407686ce9233cc4f16e3724c5e8d44367aa200
2022-05-25 12:30:15 -05:00
Cameron
87f6a3dcda {web/satelliite, satellite/console}: logo redirects to homepage
logo redirects to homepage on login, signup, forgot password, reset
password, and activate account pages

Change-Id: I992aeae197004d620addd8d515cae1c1ca80a778
2022-05-23 16:26:19 -04:00
NickolaiYurchenko
99237d5c78 web/satellite: new bucket creation flow
old bucket creation flow removed
new flow added
name and passphrase splitted into separate views
demo bucket will not be created automatically
bucket creation progress bar added

Change-Id: I2a1d7d77c3038caaafb3c06bdb0ac5dd1ad17599
2022-05-22 11:55:55 +00:00
Michał Niewrzał
f9a3f19443 scripts: add 'uplink cp' tests for stdin/stdout
Uplink can upload from stdin and download to stdout. We had
such tests for old binary but now we were missing it.

Change-Id: I5110a9f531f5cc21277fa53611995fb5b556ff16
2022-05-18 08:42:04 +00:00
Malcolm Bouzi
087e57d037 web/satellite: create chore that will resend verification emails to unverified users
We want to remind unverified users to verify their emails:
once after 24 hours has passed and again after 5 days has passed.

Add mailservice.Service to satellite core because it is needed by the
chore for sending emails. To add the mailservice.Service to the core,
we create a helper function in satellite/peer.go to avoid duplicating
the code in both api.go and core.go. In addition to the chore, this
change adds methods to users.DB to get unverified users in need of
reminder.

Change-Id: I4e515bdf43f922788b4f965b2efb34fa32288bd1
2022-05-18 08:08:33 +00:00
JT Olio
5fb9ee3cfa uplink, satellite: use bgp hostnames
Change-Id: I58f5011d3019f8267fa8cbd3096b2cfe42eb5f8b
2022-05-16 18:14:37 +00:00
Cameron
0633aca607 satellite/console: create new consoleauth service
We want to send email verification reminders to users from the satellite
core, but some of the functionality required to do so exists in the
satellite console service. We could simply import the console service
into the core to achieve this, but the service requires a lot of
dependencies that would go unused just to be able to send these emails.

Instead, we break out the needed functionality into a new service which
can be imported separately by the console service and the future email
chore.

The consoleauth service creates, signs, and checks the expiration of auth
tokens.

Change-Id: I2ad794b7fd256f8af24c1a8d73a203d508069078
2022-05-13 16:27:07 +00:00
Moby von Briesen
763bfc0913 satellite/console,web/satellite: Implement hCaptcha
Adds a new configuration for hcaptcha enabled, secretkey, and sitekey.
If both reCAPTCHA and hCaptcha are configured as "enabled", reCAPTCHA
will be used.

Change-Id: I73cc6e133d8da3555e0ed8b2b377cf9eb263e6dc
2022-05-13 14:57:45 +00:00
Erik van Velzen
db1cc8ca95 satellite/repair/checker: buffer repair queue
Integrate previous changes. Speed up the segment loop by batch inserting
into repair queue.

Change-Id: Ib9f4962d91960d21bad298f7771345b0dd270276
2022-05-12 16:28:05 +00:00
Vitalii
dedccbd2e4 satellite/console, web/satellite: limit failed login attempts
Added account locking on 3 or more login attempts.
Includes both password and MFA failed attempts on login.
Unlock account on successful password reset.

Change-Id: If4899b40ab4a77d531c1f18bfe22cee2cffa72e0
2022-05-11 14:49:11 +00:00
Qweder93
384118ee1e satellite/metabase: use pgutil.NullTimestampTZArray
This code cleanup to use helper DB method.

Change-Id: Idc438234cb4f7da23d6ade2d69f5d889f15e9009
2022-05-03 19:30:21 +00:00
hovex023
58f957bd1d
web/satellite: Add new access grants flow (#4765)
* Added new feature Flag for new Access Grant Flow. 

* Added 3 cards to access grant view for S3, CLI and Access grant to replace old header

* Added new formatting, text and Icon for Access Grant Delete Popup modal
2022-04-29 10:31:52 -05:00
nadimhq
dd6aac9b39
satellite/metainfo: enable server-side copy in production (#4708)
these changes enable server-side copy in production
2022-04-26 09:58:16 -04:00
Moby von Briesen
ed5ebb2527 satellite: Rename "acct mgmt api" to "rest api"
"REST API" is a more accurate descriptor of the generated API in the
console package than "account management API". The generated API is very
flexible and will allow us to implement many more endpoints outside the
scope of "account management", and "account management" is not very well
defined to begin with.

Change-Id: Ie87faeaa3c743ef4371eaf0edd2826303d592da7
2022-04-25 18:51:46 +00:00
Michał Niewrzał
6188c4c0dd satellite/metainfo: enable segment limit by default
Change-Id: I26cc0bbd97d6cc2ffc96213788f261efb749d028
2022-04-14 06:12:09 +00:00
Qweder93
2a7b20e8e4 cmd/uplink: integrate server-side copy with uplink cp command
Resolves https://github.com/storj/storj/issues/4486

Change-Id: I42ac2ad2e1a05df4a83606f1990b639f08791403
2022-03-31 09:25:29 +00:00
Egon Elbre
4a6c2e2a9c satellite/admin/ui: avoid needing an additional copy
This also fixes the build order. Unfortunately we need
to ensure that the web frontends are built before installing
Go binaries.

Fixes https://github.com/storj/storj/issues/4654

Change-Id: I5d1c83125fd3d1a454d3400b2cbdd44bd3f2250c
2022-03-25 11:42:22 +02:00
Cameron
84b522bc06 satellite/console: create account management api keys service
We are in the process of creating an api to allow users to manage their
accounts programmatically. We would like to use api keys for
authorization. We were originally going to create an entirely new table
for these api keys, but seeing as we already have 2 other tables for
keys/tokens, api_keys and oauth_tokens, we thought it might be better to
use one of these. We're using oauth_tokens.

We create a new oidc.OAuthTokenKind for account management api keys:
KindAccountManagementTokenV0. We made the key versioned because we
likely want to improve the implementation in the future, but we want to
get something functional out the door ASAP because the account management
api feature is highly desired.

Add a new method to oidc.OAuthTokens interface for revoking v0 account
management api keys, RevokeAccountManagementTokenV0. Add update method
to dbx implementation to allow updating the expiration. We will revoke
these keys by setting the expiration to 0 so they are expired.

Change-Id: Ideb8ae04b23aa55d5825b064b5e43e32eadc1fba
2022-03-23 17:02:20 +00:00
Egon Elbre
0d2d59f884 all: fix linting issues
Change-Id: Idfc93948e59a181321d79b365e638d63e256a16f
2022-03-21 15:26:42 +00:00
Stefan Benten
2a10b2fd14
scripts/draft-release: add sleep to avoid race condition on uploading the binaries (#4647)
It seems that the github API is a little slow/laggy with regards to propagation of whether a release tag has been made or not.
This sleep should fix it and avoid having to retrigger builds.
2022-03-21 15:24:20 +01:00
littleskunk
6fbc56bd06
satellite/payment: enable segment fee (#4628) 2022-03-18 14:25:14 +01:00
Mya
4a110b266e satellite/console: added oidc endpoints
This change adds endpoints for supporting OpenID Connect (OIDC) and
OAuth requests. This allows application developers to easily
develop apps with Storj using common mechanisms for authentication
and authorization.

Change-Id: I2a76d48bd1241367aa2d1e3309f6f65d6d6ea4dc
2022-03-16 12:01:26 +00:00
dlamarmorgan
715077d04a satellite/{console,satellitedb}: increase project limit on paid tier upgrade
Update the user to the default paid tier project limit, which is currently 3 projects, when the user upgrades to a paid account.

Change-Id: I95b19d62cebc7d878b716355f2ebcaf0b51ca3f7
2022-03-14 16:10:23 +00:00
Michał Niewrzał
39ac90835f satellite/satellitedb: change default segment limit to 100M for paying users
We decided that we want to have segment limit for paying users high
enough to not have to change it too often.

Fixes https://github.com/storj/storj/issues/4590

Change-Id: Ic1c38bf3e2fcc000548ff4c7e7004647b39fbecf
2022-03-04 13:18:14 +00:00
Moby von Briesen
b2d342aa9b satellite/overlay: Add ability to exclude country codes on upload
Create global config to specify a list of country codes that should be
excluded from node selection during uploads.

This exclusion is not implemented when the upload selection cache is
disabled.

Change-Id: Ic41e8b4f18857a11045668eac23107da99668a72
2022-03-03 16:58:48 +00:00
NickolaiYurchenko
64176aaca4 web/satellite: registration success redirect from config
This change allows us to send newly registered users to a configured URL
to help us track user conversions for marketing campaigns.
Brave conversions continue to be tracked using the /signup-success page
within the satellite app.

Change-Id: I9b451947ce0f39d3c99b233cb4b806d361151823
2022-03-03 10:13:21 -05:00
Fadila Khadar
e776c65172 satellite/checker: pieces in excluded countries are not healthy
Add a RepairExcludedCountryCodes config flag for overlay for providing a list of country codes to exclude nodes from target repair selection.

Mark segments with less than repairThreshold pieces in countries not in the RepairExcludedCountryCodes as not healthy.
With this change, the repair process is not affected. The segment will be removed from the repair queue by the repairer.

Another change will handle the logic at the repairer level.

Fixes https://github.com/storj/team-metainfo/issues/95

Change-Id: I9231b32de117a116488de055a3e94efcabb46e81
2022-03-02 09:59:09 +00:00
Vitalii Shpital
ba6956db0f console/server, apigen: feature flag for new generated console api
Added a feture flag which will be used to indicate if new generated console api is used.
Fixed some comments from previous PR.

Change-Id: Ice31c998b0b347028a491c971a648fd1269bfd49
2022-02-28 23:00:12 +00:00
Michał Niewrzał
fbe2680500 satellite/metainfo: add feature flag for server side copy code
We would like to disable in production those parts of code
which are now mixed with new server-side copy logic.

Change-Id: Iff50682bc9545207330f58dd19b5eee53d404d7f
2022-02-24 10:43:49 +00:00
Jeff Wendling
ccb847a329 scripts/tests: write old uplink config yaml directly
sometimes these scripts want to have an access imported
after it has been potentially modified by having the
satellite address and node id added. it used to use the
uplink command to do this, but the cli api for that
has changed. rather than try to have the script detect
which uplink version is in use and call the right thing
it can always write out a valid yaml file and depend on
the new cli migrating it.

Change-Id: Ib82819699333f5f29e00117b99bfb10640033b94
2022-02-22 15:26:06 -05:00
Jeff Wendling
863b01bf09 testversions: skip some downloads for older uplinks
uplink command versions >= 1.48.0 always do multipart
uploads which cannot be downloaded by any of the
versions in the test < v1.27.6. so skip those tests.

Change-Id: I9644afbd14bfce9facfd87644d132f7d66367d62
2022-02-22 17:24:07 +00:00
Stefan Benten
ec2bd50bb4
scripts/deploy-storagenode.sh: adding multinode image to deployment
In order to tag the latest release for the multinode image, it makes
the most sense to do it at the same time as we release the storagenode
image.

Change-Id: I2d63c1f93858354ad1f9a4fce0ce45a8fda2716f
2022-02-21 14:37:04 +01:00
Jeff Wendling
0a860a2e89 scripts: fix builds for new uplink command
the new uplink command expects to be able to migrate
and so we need to specify the --legacy-config-dir
flag sometimes as well. but unfortunately, the old
uplink will error if it gets a flag it doesn't
understand, so we have to set it as an env var.
but we can't use the env var for the --config-dir
flag all the time because the old uplink doesn't
look for that env var.

Change-Id: I019315192c0e6c348814527794342d823a5f9ec3
2022-02-16 21:40:06 +00:00
NickolaiYurchenko
70c8ccb435 web/satellite: inactivity timer to dashboard added
added InactivityTimerEnabled flag to enable/disable feature
added InactivityTimerDelay to configure delay time in seconds
default timer set up to 10 minutes
reset dom events: keypress, mouseover, mousedown, touchmove

Change-Id: Idb66067c2902b2cdbe1a972225319c8abff97927
2022-02-16 07:16:00 +00:00
Jeff Wendling
9061dd309f cmd/uplinkng: become cmd/uplink
Change-Id: If426c32219d32044d715ab6dfa9718807f32cb9f
2022-02-09 17:02:21 +00:00
Vitalii Shpital
b3e1be37ff satellite/projectaccounting: query to get daily project usage by date range
Finished implementing queries for both bandwidth and storage using pgx.Batch.
Fixed CSP styling issue.

Change-Id: I5f9e10abe8096be3115b4e1f6ed3b13f1e7232df
2022-02-04 22:32:37 +00:00
Jeff Wendling
b7ca2289e0 cmd/uplinkng: improvements to prepare for doc update
* improve setup wizard
* added access grant file support
* improved consistency across commands
* a couple bug fixes
* added access import and export

Change-Id: I30ad9d4771f15430904a503a4d465bc40be471b5
2022-01-31 17:37:54 +00:00
Egon Elbre
64c8de6ea5 mod: use vendored base58
Change-Id: I5aa29515928848c862500330218cc094618638d7
2022-01-31 15:54:33 +02:00
Stefan Benten
e7e39fe628
satellite/metainfo: reduce default user rate limit to 100
Currently the rate limit has kept per satellite api endpoint.
Since we run 9+ api endpoints in production, we do not need
a limit of 1000, since the intention was to allow 1000 total.
This change reduces the effective limit given 9 instances
down to 900, which should be close enough.

Change-Id: Ia579149ccc3a12e8febe0cfd5586b8a39de40f55
2022-01-21 23:37:59 +01:00
Michał Niewrzał
270c6e24f2 satellite/console: fix default limit for free-tier segment usage limit
Free-tier segment usage limit was defined as 150k, not 140k. This change
is correcting that.

https://github.com/storj/team-metainfo/issues/8

Change-Id: I71ec0961930b19fd09b2b996e01acd406a8dcf8f
2022-01-19 09:21:13 +00:00
Qweder93
8be655c487 satellite/metainfo: add segment limit validation, caching
We want to be able to limit the number of segments per project for users.
To limit this we need to check limit value associated with project
and value of used segments already in BeginMoveObject, BeginMoveSegment
and increment cache segments usage after each CommitSegment call.

Resolves https://github.com/storj/team-metainfo/issues/1

Change-Id: I6290e67c095a174b9d101c4521802d9bfe0453b8
2022-01-10 14:33:26 +02:00
Egon Elbre
04d04c0306 web/satellite: remove old object browser
Change-Id: Ief16a3887444df093c644831d86a9b3e865e5867
2022-01-05 16:29:19 +02:00
Malcolm Bouzi
6a059b7f9d web/satellite: remove all references to the old onboarding flow
Change-Id: I1f42fabdb407a7eb9c34f0341d332e66c3ce288d
2022-01-04 18:40:08 +00:00
Jeff Wendling
4426470173 cmd/uplinkng: rename parallelism flags, small fixes
this makes the flags match rclone nomenclature

fixes test-uplinkng to use the temporary config dir
instead of the machine default, and clean up some.

bumps clingy so that the command errors when an unknown
command is specified.

also fixes some printfs in share to use clingy stdout.
it still does some external actions that should be
passed through a ulext.External for mocking, but
that's ok for now.

Change-Id: Icc231e7e26393541c312396fec907b640b97718e
2021-12-22 13:13:30 +00:00
Egon Elbre
5f39f20766 web/: add check for change to eslint import
Change-Id: Idf94c8e0c92d34e840db549e9e704eb3373762ac
2021-12-21 15:59:23 +00:00
Stefan Benten
e2481fb510 scripts,Jenkinsfile,Makefile: upload binaries to drafted github release
This change adds a script and the needed build logic to create a draft github release and then upload the created binaries to it in order to make the actual publishing a lot less error prone.
The logic is currently that it only does this for all github tags, but not for every main build/push. This is handled by the checks in the script itself.

Change-Id: Ie172a8e4a97200de901a26a055aa5a8a54b60a2a
2021-12-21 13:55:38 +00:00
Michał Niewrzał
87a9292127 satellite/metainfo/zombiedeletion: enable chore by default
We enabled this chore on different satellites
for testing. Works fine so we can enable this
by default for all configurations.

Change-Id: I987639685d8de5c7e5798adca30fe26bdac9e1d1
2021-12-21 13:26:18 +01:00
Michał Niewrzał
3f0babffe8 satellite/{console,satellitedb}: add project segment limit to user
All limits we have for projects have also parent limits stored
with user data. New created project is first taking limits from
owner (user) limits.
This change is extending users table with project_segment_limit
column and adds functionality to get and set value for this
column.

Change-Id: Iff5e36c62b517652390b649fc05992475916ecff
2021-12-16 08:46:01 +00:00
Jeremy Wharton
9d13c649a2 satellite/{console,satellitedb}: Forbid creating users with used email
This change disallows creation of users possessing the same email.
If a user attempts to create an account with an email address
that's already used - whether it belongs to an active account or not -
he will be notified of unsuccessful account creation. If he attempts to
log in using an email address belonging to an inactive account,
he will be presented with a link allowing him to re-send the
verification email. Attempting to register with an email address
belonging to an existing account triggers a password reset email.

Change-Id: Iefd8c3bef00ecb1dd9e8504594607aa0dca7d82e
2021-12-15 17:24:35 +00:00
Jeff Wendling
9615bd191f cmd/uplinkng: update clingy for better boolean support
this allows commands like

    uplinkng cp -r sj://foo sj://bar

to work correctly, rather than complain that sj://foo is
not a boolean.

Change-Id: I003e47aabb85566bc2b454851cf55043b17ee7ea
2021-12-14 09:41:38 +00:00
prerna-parashar
a41758bba5
Track user events through HubSpot events API (#4300)
Track user events through HubSpot events API
2021-12-09 12:52:51 -08:00
Stefano Fornari
af3edf2b78
scripts/release.sh: making clear if builing a production or development version (#4304)
Co-authored-by: Stefan Benten <mail@stefan-benten.de>
2021-12-09 12:38:23 +01:00
Igor
4daaa9503e
satellite/console: enable object flow by default (#4298) 2021-11-29 16:43:37 +02:00
Michał Niewrzał
24cf7e8ea6 cmd/uplinkng: add mv command
Add ability to move files and objects.

Change-Id: I4929da730984c06aa578678b1d8c8e9b4aceade8
2021-11-22 09:07:24 +00:00
Vitalii Shpital
3de7f8d5af satellite/console, web/satellite: feature flag for new project dashboard
Added a feature flag to switch between old and new project dashboard

Change-Id: Ibb1b04ccbddb43e3f41116e3513f917deba95699
2021-11-17 16:02:47 +02:00
andriikotko
09568b3e2b
satellite/console: change default feature flag (#4274)
* satellite/console: change default feature flag
2021-11-16 11:38:42 +02:00
dlamarmorgan
44b1ca6b97 satellite/{console,satellitedb}: move project limits from config file to DB to keep limits on a per user basis
To allow for changing limits for new users, while leaving existing users limits as they are, we must store the project limits for each user. We currently store the limit for the number of projects a user can create in the user DB table. This change would also store the project bandwidth and storage limits in the same table.

Change-Id: If8d79b39de020b969f3445ef2fcc370e51d706c6
2021-11-11 15:10:00 +00:00
Ivan Fraixedes
4e67ea007c satellite/admin: Serve static UI assets
Change the satellite Admin HTTP server for:

* Embedding the UI assets into the Go binary.
* Serve the UI assets from the embedded file system or from a specific
  directory path through a configuration flag, without requiring
  authentication but keeping the authentication verification for the API
  endpoints.
* Add tests to verify that the UI assets are served without
  authentication.

Change-Id: I9003ac96f1ec585a189b67fc1cb315905403d557
2021-11-10 18:47:46 +02:00
Mya
bf51c286d9
satellite/geoip: update node check-in to associate a country code
Resolves https://github.com/storj/storj/issues/4247

Change-Id: Idfd71bf1795d48ca3c686066bbdb95b9c6594f00
2021-11-10 16:44:41 +01:00
Vitalii Shpital
3c683998f5 satellite/console, web/satellite: feature flag for new objects flow
Added a feature flag to switch between old and new objects flow

Change-Id: Id1802f9fb1230928445f16e4d0a62aecdc91c038
2021-11-08 11:45:22 +00:00
Clement Sam
16a334020f cmd/uplinkng: add ranged download
This change adds the ability to download byte ranges
to uplinkng.

Extended the uplinkng Filesystem interface with Stat
method and an OpenOptions struct as parameter for the
Open method.

Also added a few tests for the ranged download

Change-Id: I89a7276a75c51a4b22d7a450f15b3eb18ba838d4
2021-11-03 19:51:25 +00:00
Egon Elbre
d043b9facd satellite/console: load index.html from disk when developing
This eases development by not needing to restart satellite after each
change.

Change-Id: I1063b7f431f675fb21e82dcfc2420310a9806422
2021-11-03 18:19:40 +02:00
Artur M. Wolff
785eb93cd7 ci: install gateway@latest instead of @main
Gateway-ST frequent release cycle has been resurrected, which means it's
safer to use the latest release tag in the storj repository's CI now.

Change-Id: I9df1c789a9b9418ba7cceaec9cfec3cc6c448284
2021-11-01 12:16:38 +00:00
mya
9749f9756c
cmd/uplinkng: port expanded option to show additional metadata (#4229) 2021-10-28 11:51:33 -05:00
Egon Elbre
edb8d656de satellite/metainfo: adjust piecedeletion timeouts
Currently slower storagenodes can slow down deletion queue.
To make piece deletion faster reduce the maximum time spent in
either dialing or piece deletion requests.

With this change:
* dial timeout is 3s
* request timeout is 15s
* fail threshold is set to 10min

Similarly, we'll mark storage node as failed when the timeout occurs.
The timeout usually indicates that the storagenode is overwhelmed.
Garbage collection will ensure that the pieces get deleted eventually.

Change-Id: Iec5de699f5917905f5807140e2c3252088c6399b
2021-10-28 13:37:01 +03:00
Yingrong Zhao
d441c8da15 satellite: use segment count for billing
Change-Id: Iafaf41cb6cde4cb2abdac3f75ce0fead838cb758
2021-10-27 19:21:50 +00:00
dlamarmorgan
01cfbde56b satellite/console: increase free tier project bandwidth and storage from 50 GB to 150 GB and reduce free tier max projects from 3 to 1
To make our free tier limits more clear, we will reduce the number of projects allowed from 3 to 1, and increase the storage and bandwidth limit of the free tier from 50 Gb to 150 GB. The total allotments across all projects for a given user are unchanged, just reduced to a single project.

Change-Id: Ic8dddb135f2b83a3f36e2b9fdcb477e351ec137b
2021-10-25 17:34:19 +00:00
Clement Sam
85b49bb27c cmd/uplink: add ranged download to uplink cli
Change-Id: Ib274df024a8ffc5db2d5c99f8f363efa3b43723f
2021-10-25 13:13:33 +00:00
Qweder93
50baefa10e satellite/metabase: limit maximum number of parts and size
Multipart upload limits added. Last part has no size limit.
Max number of parts: 10000, min part size: 5 MiB

Change-Id: Ic2262ce25f989b34d92f662bde720d4c4d0dc93d
2021-10-22 10:16:37 +00:00
Monty Anderson
3989107031 web/satellite: add feature flag for new browser
Change-Id: If3773b9df7f79d4c5e094140448d60ec060cdaeb
2021-10-22 03:49:08 +00:00
crawter
9ed826eabb cmd/uplink/mv: moving files between folders
https://storjlabs.atlassian.net/browse/PG-305

we should extend method move of cmd/uplink/cmd/mv.go
if both parameters end with slash -
i should list all files and call move method in loop

parameters:
uplink mv sj://bucket/a/prefix/ sj://new-bucket/a/new-prefix/

Change-Id: Ic24c2af83153ea60ec74393e65736af094877151
2021-10-20 18:16:08 +00:00
Vitalii Shpital
0009447773 satellite/console, web/satellite: feature flag for new navigation structure
Change-Id: I6f1fd43ea8154714187c7547ae93de0efe3482e6
2021-10-20 13:51:25 +00:00
littleskunk
016ac75052
satellite/console: Enable new onboarding workflow (#4222) 2021-10-15 21:07:35 +02:00
igor gaidaienko
5b4a9070b0 scripts/testversions: reduced the number of version to test 2021-10-12 13:28:46 +03:00
Jeremy Wharton
3b751a35c5 satellite/{payments,satellitedb}: Remove custom coupon implementation
Removes database tables and functionality related to our custom
coupon implementation because it has been superseded by the Stripe
coupon and promo code system. Requires implementations of the
payments Invoices interface to return coupon usages along with
invoices.

Change-Id: Iac52d2ff64afca8cc4dbb2d1f20e6ad4b39ddfde
2021-10-11 19:47:00 +00:00
Michał Niewrzał
0209bc6ef7 cmd/uplink: add mv command
New command for cli to move object to different
location.

uplink mv sj://bucket/your-object sj://bucket/moved-object

Change-Id: I85a4961aa59f250819954e78f20363ac3c570938
2021-09-29 19:10:58 +00:00
Michał Niewrzał
5b66136312 cmd/uplinkng: fix mb command
Make bucket command was using full location
specified in command line instead only bucket name.

As an addition change contains basic integration tests
with storj-sim.

Change-Id: Ie3b5283468b7fbde0b1333f01dc4fc2a2952e1a1
2021-09-29 15:22:03 +00:00
Artur M. Wolff
cc9b845a83 ci: switch back to pulling gateway@main
Switch back to pulling gateway@main for continuous integration because
https://review.dev.storj.io/c/storj/gateway/+/5834 was merged. This
change is a partial revert of
https://review.dev.storj.io/c/storj/storj/+/5832.

Change-Id: I0a87f6f18a9a863fe92003b76c830335f9253ced
2021-09-28 15:01:04 +00:00
Egon Elbre
8ef03b0967 ci: cleanup ws before build, fix gateway install
Jenkins uses the same folder for different PR-s. Sometimes other runs do
not cleanup after themselves (e.g. timeout), hence add a cleanWs step to
ensure we delete files in the workspace.

gateway-st introduced a replace directive in go.mod, which does not work
with go install. Hardcode to the last version without the directive.
Using this fix to unblock ci builds.

Change-Id: I5e5d75bf47e30a5a8b6d835867c0c9176f25e08a
2021-09-23 15:57:47 +03:00
Vitalii Shpital
8b91c55ec4 web/satellite, satellite/console: return old onboarding flow with feature flag
Added new feature flag for switching between different flows.
Got back old onboarding flow.

Change-Id: Idbf23ba8ff36900a47e3b4ebeb34b78351202e29
2021-09-21 11:37:30 +00:00
Egon Elbre
6e660cecdd Jenkinsfile: test cross-compile and bump deps
Change-Id: I47091de2e80bd96fcada616f75e1db07a59fb0c1
2021-09-16 18:59:31 +03:00
crawter
e1a4195a35 satellite/metainfo/endpoint: MaxObjectKey length validation for BeginObject and CommitObject added
Change-Id: I6a766415a84b1e3c44ce65052ffa8ed4511d00dd
2021-09-13 15:07:16 +03:00
Michał Niewrzał
03d638bbb3 cmd/uplink: add parallel upload
Use multipart upload to upload single object in parts in
parallel. Its using parallelism flag added earlier.

Change-Id: I45b531a5db43c86f0112a5e3bb4a83bc1d65650f
2021-09-07 15:12:09 +00:00
Michał Niewrzał
211a630982 cmd/uplink: add parallelism flag for single object download
Adds support for new uplink method DownloadObjectAt which
gives ability to download single object in parallel.

Change-Id: I8388653429992b0d24c383d17d7e90904203fe77
2021-08-26 13:13:38 +00:00
Michał Niewrzał
b7555980c4 satellite/metainfo: add zombiedeletion chore
This chore will be deleting zombie objects and it's segments.

Change-Id: Ia5d92dd6c3f52e178d4d43d9603732310761e027
2021-08-24 08:22:06 +00:00
Jeremy Wharton
6a6cc28fc1 satellite/console,private/web: Rate limit coupon code application
Rate limits application of coupon codes by user ID to prevent
brute forcing. Refactors the rate limiter to allow limiting based
on arbitrary criteria and not just by IP.

Change-Id: I99d6749bd5b5e47d7e1aeb0314e363a8e7259dba
2021-08-23 17:00:31 +00:00
Moby von Briesen
7f02e1f469 satellite/console: Remove config flag for MFA
MFA is complete and we are good to enable it in production. This change
removes the flag that disables MFA by default.

Change-Id: I2f985ae501171bdab505d664b43c8cfc248bad8d
2021-08-05 18:04:36 +00:00
Yingrong Zhao
646ce5b8cc satellite/overlay: remove reputation logic from overlay
Change-Id: I3492860e4537c7a8e4e824ec4c9c8d179134a0c0
2021-07-28 15:15:28 -04:00
Yingrong Zhao
6c7bf357cd satellite/{reputation,audit,overlay}: replace overlay with reputation
package in audit

This PR implements reputation store and replace overlay in audit service
to use such store for storing node's audit stats.

In order to keep the changeset smaller, most of the changes in this PR is for copying audit logic in overlay to
reputation package. In a following PR, the duplicating code will be
removed from overlay.

Change-Id: I16c12494a0970f44c422b26cf603c1dc489e5bc1
2021-07-28 13:10:48 -04:00
Moby von Briesen
149f6f2626 satellite/payments: Implement coupon codes
Full path: satellite/{payments,console},web/satellite

* Adds the ability to apply coupon codes from the billing page in the
satellite UI.
* Flag for coupon code UI is split into two flags - one for the billing
page and one for the signup page. This commit implements the first, but
not the second.
* Update the Stripe dependency to v72, which is necessary to
use Stripe's promo code functionality.

Change-Id: I19d9815c48205932bef68d87d5cb0b000498fa70
2021-07-26 17:15:55 +00:00
Michał Niewrzał
b12d29935a satellite/metabase: remove metaloop package
We moved everything to segment loop so we can now
remove metaloop from code.

Change-Id: I9bd8d2349e5638d7cdad50f2f313f9bd89a8165c
2021-07-22 13:00:45 +00:00
Michał Niewrzał
27a714e8b0 satellite/accounting/tally: use objects iterator instead metaloop
Bucket tally calculation will be removed from metaloop and will
use metabase objects iterator directly.

At the moment only bucket tally needs objects so it make no sense
to implement separate objects loop.

Change-Id: Iee60059fc8b9a1bf64d01cafe9659b69b0e27eb1
2021-07-20 15:52:18 +00:00
Vitalii Shpital
8855c0dff7 web/satellite: added MFA feature flag, updated client-side api and Vuex store module
Added feature flag for MFA
Added new client-side api call to enable MFA returning secret
Updated users Vuex module to include new API call

Change-Id: Ia9e10f68c4a7da39b4f7c1073e657c2de98fb0db
2021-07-12 10:12:59 +00:00
Jeremy Wharton
a5f6bb9cc0 satellite/console: Add reCAPTCHA verification step to registration
The user must complete a reCAPTCHA in order to register.
ReCAPTCHA verification failure results in rejection of the
registration attempt.

Change-Id: I34ba7db414d756fd1aaebdc3d19cccbfc7fc1ea3
2021-07-07 21:34:07 +00:00
Moby von Briesen
e36001b7cf satellite/console: Implement paid tier
When a user adds a credit card, switch them to the paid tier and update
their projects with new bandwidth/storage limits. New projects for the
paid tier user will also have the updated limits.

The new limits are:
* storage per project - 50 GB free/25 TB paid
* bandwidth per project - 50 GB free/100 TB paid

Change-Id: I7d6467d077e8bb2bbe4bcf88ab8d75490f83165e
2021-07-06 14:30:12 +00:00
Moby von Briesen
4e95d27033 web,satellite: Remove paywall-related functionality
Because of our free/paid tier plan, we do not need a paywall anymore. We
have not used it in a while, but still have leftover code laying around.

Change-Id: Iaea8c39faf042a2f7a6b837727bb135c8bdf2907
2021-06-29 02:47:48 +02:00
Michał Niewrzał
4267a958d4 satellite/satellitedb: use AS OF SYSTEM TIME for GetProjectBandwidth query
Adding AS OF SYSTEM TIME to query that is calculating project bandiwdth.
As an addition method for setting interval is added as test doesn't
work well with default interval.

Change-Id: Id1e15be4f6afff13b9dc2b7f595e2edb6de28db9
2021-06-28 15:19:47 +00:00
Jeremy Wharton
8a070e7c25 satellite/overlay: Ignore unnecessary check-ins
This prevents the database from being contacted unnecessarily,
reducing load.

Change-Id: Ib2420f68a20636ec35eb3dd3df8e02bd5341b419
2021-06-22 09:00:41 +00:00
Egon Elbre
ca64e55281 satellite/gc: remove skip first
We used this to reduce initial load on the core to avoid OOM. However,
this is not a problem anymore with garbage collection running
separately.

Change-Id: Ifd62c822a74974bc21a5913199334469a4bc0130
2021-06-21 18:30:38 +00:00
Egon Elbre
9640cc2c06 satellite/metabase/segmentloop: verify processed count
This adds verification for the processed count and before and after
segment/objects table counts.

This adds new flag:

  metainfo.segment-loop.suspicious-processed-ratio: 0.03

This defaults to 3%, which at 100M segments is 3M segments.

Change-Id: I5ee03e913ddc4e67e94010ced126a2a9ea51f41b
2021-06-17 19:46:55 +03:00
Egon Elbre
341033dda1 satellite/metabase/metaloop: verify processed count
This adds verification for the processed count and before and after
segment/objects table counts.

This adds new flag:

  metainfo.loop.suspicious-processed-ratio: 0.03

This defaults to 3%, which at 100M objects is 3M objects.

Change-Id: Ife5522ecc97bcc5a55667f36868a0f1fc8e4c561
2021-06-17 15:23:53 +03:00
Michał Niewrzał
a93e47514a satellite: remove irreparabledb
This is part of metaloop refactoring. We plan to remove
irreparable at some point but there was not time for it.
Now instead refatoring it for segmentloop its just easier
to drop it.

Later we still need to drop table with migration step.

Change-Id: I270e77f119273d39a1ecdcf5e1c37a5662a29ab4
2021-06-17 07:20:15 +00:00
Egon Elbre
9b2607d6ba satellite: remove garbage collection option from core
We don't run it anywhere in this configuration, so it's not worthwhile
to keep it that way.

Change-Id: I88afb8bb3eb3843801b15454408f10d1353596cb
2021-06-15 21:07:02 +03:00
Egon Elbre
f3a52d1da5 satellite/metabase/segmentloop: limit max interval
Ensure that we don't query too far in the history, which slows things
down.

Change-Id: Ia77aa522f7f4c5d43629d51bb9a51a49fab6fa14
2021-06-10 17:06:33 +00:00
Egon Elbre
347f5f87e0 satellite/metabase/metaloop: limit as of system time
Currently we did not limit the "as of system time" for iterating over
objects table. Using just an interval would cause problems with the
tests. That could be overcome skipping that interval for tests
altogether, however, we should probably test those more to ensure that
GC stays working as intended.

This is a safer code, however, maybe not as straigthforward as it could
be.

Change-Id: I374f77783b2af42bb6da846735ceea20a7ce5e60
2021-06-07 13:01:06 +00:00
JT Olio
da9ca0c650 testplanet/satellite: reduce the number of places default values need to be configured
Satellites set their configuration values to default values using
cfgstruct, however, it turns out our tests don't test these values
at all! Instead, they have a completely separate definition system
that is easy to forget about.

As is to be expected, these values have drifted, and it appears
in a few cases test planet is testing unreasonable values that we
won't see in production, or perhaps worse, features enabled in
production were missed and weren't enabled in testplanet.

This change makes it so all values are configured the same,
systematic way, so it's easy to see when test values are different
than dev values or release values, and it's less hard to forget
to enable features in testplanet.

In terms of reviewing, this change should be actually fairly
easy to review, considering private/testplanet/satellite.go keeps
the current config system and the new one and confirms that they
result in identical configurations, so you can be certain that
nothing was missed and the config is all correct.
You can also check the config lock to see what actual config
values changed.

Change-Id: I6715d0794887f577e21742afcf56fd2b9d12170e
2021-06-01 22:14:17 +00:00
Michał Niewrzał
053e58b683 satellite/metabase: add segmentloop service
We want to move some of current metainfo loop observers to
segment loop. This change adds new service, similar to metainfo
loop but which is iterating only over segments.

Change-Id: I67f7f461781723a4476e2b83377f31736d7c4870
2021-06-01 11:15:07 +00:00