Commit Graph

666 Commits

Author SHA1 Message Date
dlamarmorgan
0b468aca56 satellite/payments/storjscan/storjscan.go: change default block confirmations
Change the default number of required block confirmations for a payment to be confirmed from 12 to 15.

Change-Id: I44c258134c293e7691623bc00c504130aa69a96a
2022-09-23 14:30:05 +00:00
Artur M. Wolff
fe85fefcc5 ci: add check-downgrades checks
Change-Id: Idfe81897dc701c24c0f81b0021cb1d640e8224ac
2022-09-23 11:15:27 +00:00
Michal Niewrzal
4e71dd302a satellite/{metainfo,metabase}: prepare feature flag MultipleVersions
We will introduce new logic for creating new objects (BeginObject).
Instead of using single version internally (1) we will be selecting first
available version during object creation. Because we need to be sure
that everything is wired up correctly we need a feature flag to be
able to control if new feature is enabled.

Change-Id: If0f8496397130811f43bf9db9fdcc2b30cd2e4ca
2022-09-21 13:07:13 +00:00
Erik van Velzen
e6b5501f9b satellite/gc/sender: new service to send retain filters
Implement a new service to read retain filter from a bucket and
send them out to storagenodes.

This allows the retain filters to be generated by a separate command on
a backup of the database.

Paralellism (setting ConcurrentSends) and end-to-end garbage collection
tests will be restored in a subsequent commit.

Solves https://github.com/storj/team-metainfo/issues/121

Change-Id: Iaf8a33fbf6987676cc3cf74a18a8078916fe673d
2022-09-20 11:49:40 +00:00
Artur M. Wolff
5150ee5001 mod: bump storj.io/private
Change-Id: I534aeedead1f1a2c5099e80b49fa5a8ea599fac8
2022-09-19 13:09:31 +00:00
Michal Niewrzal
a3d9630336 scripts: move tests to separate folders
Doing some cleanup in "scripts" folder. All integration like tests are
moved under "test" directory (integration, bc, redis) and bash scripts
are adjusted to reflect new location.

As an addition "scripts/install-awscli.sh" was deleted as it was not
used.

Change-Id: I152905c4258f471a71f2d0e8731d91bb075e99c1
2022-09-16 11:10:18 +00:00
Michal Niewrzal
158eb2381e satellite/gc/bloomfilter: uploading bloom filters
We would like to have separate process/command to collect bloom
filters from source different than production DBs. Such process will
use segment loop to build bloom filters for all storage nodes and
will send it to Storj bucket.

This change add main logic to new service. After collecting all bloom
filters with segment loop and piece tracker all filters are marshaled
and packed into zip files. Each zip contains up to "ZipBatchSize" bloom
filters and it's uploaded to specified in configuration bucket.

All uploaded objects have specified expiration time to not delete them
manually.

Updates https://github.com/storj/team-metainfo/issues/120

Change-Id: I2b6bc02a7dd7c3a639e75810fd013ae4afdc80a2
2022-09-12 08:33:53 +00:00
Michal Niewrzal
68f6d93f29 satellite/gc/bloomfilter: add service to collect bloom filters
We would like to have separate process/command to collect bloom
filters from source different than production DBs. Such process will
use segment loop to build bloom filters for all storage nodes and
will send it to Storj bucket. This this initial change to add such
service. Added service is joining segment loop and collects all
bloom filters.

Sending bloom filters to the bucket will be added as a subsequent
change.

Updates https://github.com/storj/team-metainfo/issues/120

Change-Id: I2551723605afa41bec84826b0c647cd1f61f3b14
2022-09-02 08:10:46 +00:00
Jeremy Wharton
3f26cc599f satellite/console,web/satellite: invalidate sessions after inactivity
Sessions now expire after a much shorter amount of time, requiring
clients to issue API requests for session extension. This is handled
behind the scenes as the user interacts with the page, but once session
expiration is imminent, a modal appears which informs the user of his
inactivity and presents him with the choice of loging out or preserving
his session.

Change-Id: I68008d45859c814a835d65d882ad5ad2199d618e
2022-08-23 15:51:05 +00:00
paul cannon
0dcc0a9ee0 satellite/reputation: reconfigure lambda and alpha
This is in response to community feedback that our existing reputation
calculation is too likely to disqualify storage nodes unfairly with
extreme swings up and down.

For details and analysis, please see the data_loss_vs_dq_chance_sim.py
tool, the "tuning reputation further.ipynb" Jupyter notebook in the
storj/datascience repository, and the discussion at

    https://forum.storj.io/t/tuning-audit-scoring/14084

In brief: changing the lambda and initial-alpha parameters in this way
causes the swings in reputation to be smaller and less likely to put a
node past the disqualification threshold unfairly.

Note: this change will cause a one-time reset of all (non-disqualified)
node reputations, because the new initial alpha value of 1000 is
dramatically different, and the disqualification threshold is going to
be much higher.

Change-Id: Id6dc4ba8fde1be3db4255b72282207bab5491ca3
2022-08-17 18:52:53 +00:00
dlamarmorgan
f596f72f44 satellite/payments/storjscan: add billing update to chore
Update chore to add confirmed transactions to the billing table.

Change-Id: I3c9a98c2ddc96f8a8905250376a1f5490d810277
2022-08-16 11:18:15 -07:00
Vitalii
0ec28ca623 web/satellite: add token funds modal
Created new modal which shows user their native STORJ token wallet address.
There are QR and copy buttons.
It will be used only in new billing screen.

Change-Id: Icef3c8668c548b779c07fe2b85eb5761cd1221a3
2022-08-15 14:17:30 +03:00
Márton Elek
96b88e286e build: fix conditional run of web builds
Jenkins doesn't do a very good job with identifying what has been changed.

While it has a syntax to defined patterns, it compares the current build with the previous build (in case of git-verify it can be a totally different branch) instead of checking the HEAD commit.

This patch introduces shell scripts to do this better:
 * It doesn't depend on Jenkins any more
 * It can be executed locally
 * It can detect web changes properly (see the relation change as an example).

Change-Id: I9d37775e3818c08c4aa96ffb78f84d57f28a2c95
2022-08-11 14:34:44 +00:00
Moby von Briesen
3f6bf64af9 satellite/console: Default NewProjectDashboard to "true"
We have enabled the new project dashboard in production. Change the
default to true so that we do not need an explicit configuration in
prod.

Change-Id: I0f93773965283e7b0682f6586685224281cbf78c
2022-08-06 01:00:31 +00:00
Vitalii
ad37ea4518 satellite/{web, console}: login captcha implemented
Implemented Recaptcha and Hcaptcha for login screen.
Slightly refactored registration page implementation.
Made 2 different login/registration captcha configs on server side to easily swap between captchas independently.

Issue: https://github.com/storj/storj/issues/4982

Change-Id: I362bd5db2d59010e90a22301893bc3e1d860293a
2022-08-03 23:02:27 +00:00
Jeff Wendling
f343b856dd go.mod: bump common/private/monkit/monkit-jaeger
this includes all of the fixes identified for end to end
tracing to finally work.

Change-Id: Ia433a2cf9130dc43b6810379af32f5c10f9d3888
2022-07-27 16:17:00 +00:00
Qweder93
2b24632610 {satellite/metainfo, satellite/accounting}: remove ValidateSegmentLimit config value and check
removed segment limit validation and checks in metainfo endpoint and accounting/projectusage
since feature is live and has always has segment limitation now

Resolves: https://github.com/storj/storj/issues/4470

Change-Id: I8cf87cbbc40ac61262f9f05e52573d3ae6410611
2022-07-27 15:36:53 +00:00
Egon Elbre
82fede2132 satellite/satellitedb: batch delete storage node tallies
Currently we have a significant number of tallies that need to be
deleted together. Add a limit (by default 10k) to how many will
be deleted at the same time.

Change-Id: If530383f19b4d3bb83ed5fe956610a2e52f130a1
2022-07-20 15:42:35 +03:00
paul cannon
2f20bbf4d8 satellite/reputation: add a reputation write cache
This should lower the amount of database load coming from
reputation updates.

Change-Id: Iaacfb81480075261da77c5cc93e08b24f69f8949
2022-07-14 21:40:16 +00:00
Yaroslav Vorobiov
de5951331a satellite: wire storjscan chore to core process
Adds storjscan chore and storjscan client to satellite chore.
Storjscan chore interval added to storjscan config.

Change-Id: I13bb8d83b14ca90f4735c9a10c361ad24a56b256
2022-07-14 15:07:52 +00:00
Vitalii
69dc9a4731 satellite/console: added new email which is sent on unknown password reset
Added new email html template.
It is sent when user tries to reset password with unknown or unverified account.
Made a couple of minor config changes.

Issue: https://github.com/storj/storj/issues/4913

Change-Id: I730f48b3478e302d1e38e1f8a27c75f66a8ba6fd
2022-07-14 14:32:59 +00:00
littleskunk
cf750716ea
satellite/console: enable new access grant flow (#4934) 2022-06-30 00:06:39 +02:00
JT Olio
18666b078a metabase: support disabling serverside copy when already enabled
Change-Id: I99b0b5597537f69fbdd8cab479e059ba4bf310bd
2022-06-24 17:56:43 +00:00
Márton Elek
204367b74d
build: premerge/verify Jenkinsfile (subsets of Jenkinsifle.public)
Change-Id: I33c65a37ecd8fcb89729b2a8db3906c6b8890ca3
2022-06-23 09:15:43 +02:00
Ivan Fraixedes
fc4b0e41f0 scripts: Change Storagenode arm32v6 to arm32v5
We don't build "multinode" nor "storagenode" docker images for armv6
architecture, instead we build for armv5.

Fix the script that publish the manifest for those images for a specific
tag to use armv5 for not failing when executing.

Change-Id: I7d859d8718240e1cd0dae6489e7e5c3b4068ff6e
2022-06-21 11:54:38 +00:00
Jeremy Wharton
58c5d44f44 satellite/console: integrate sessions into satellite UI
This change integrates the session management database functionality
with the web application. Claim-based authentication has been removed
in favor of session token-based authentication.

Change-Id: I62a4f5354a3ed8ca80272814aad2448f901eab1b
2022-06-13 08:02:02 +00:00
JT Olio
098ee71e03 test-sim-versions: write what bash step is executing
Change-Id: If1231f69601c1774bc6ce116f45105cc1b2f4e22
2022-06-10 19:33:03 +00:00
Yaroslav Vorobiov
62774768d9 satellite/payments/storjscan: update config and wrap service errors
Change-Id: I4f75375e9499119cbcce78c5e77d1950e4bae38b
2022-06-07 14:21:42 +00:00
Paul Willoughby
911cc1e163 satellite/contact: reject privateIPs in PingMe and CheckIn endpoints
prevent network enumeration by rejecting privateIPs in PingMe and
Checkin endpoints

Closes storj/storj-private#32

Change-Id: I63f00483ff4128ebd5fa9b7b8da826a5706748c9
2022-06-07 08:09:14 +00:00
Vitalii
ba58530089 {satellite}/web,console,testsuite: remove old navigation structure
Removed old satellite UI navigation structure.
Removed old feature flag.

Change-Id: Ic998886cf2e30ebd44e67a20fc53888103fe4b8d
2022-06-06 16:43:05 +03:00
dlamarmorgan
270204f352 satellite/{payments/storjscan,satellitedb}: Add wallet implementation
Add storjscan wallets implementation to the satellite. The wallets interface allows you to add and claim new wallets as called by the API. The storjscan specific implementation of this interface uses a wallets DB to associate the user to a wallet address, as well as a storjscan client to request and associate new wallets to the satellite.

Change-Id: I54081edb5545d4e3ee07cf1cce3d3e87cc00c4a1
2022-06-03 11:45:47 +00:00
cl-mitch
cbaca8b17e
web/satellite added new billing screen feature flag (#4836)
New feature flag implemented to enable the new billing flow.

Co-authored-by: Maximillian von Briesen <mobyvb@gmail.com>
2022-06-02 09:30:27 -05:00
littleskunk
6cdd250019
satellite/console: enable new object flow / onboarding (#4851) 2022-06-01 14:04:03 +02:00
littleskunk
3d1b469f35
satellite/console: disable account activation reminder email (#4844)
* satellite/console: disable account activation reminder email
2022-05-31 17:36:40 +02:00
Stefan Benten
8673583582 satellite/admin: fix console config handling
An older change plummed the full console config as subconfig of
the admin api configuration in. This bloated the generated satellite
configuration unnecessarily while also allow for confusion/mistakes.

Change-Id: Icf49cc1f147711e37e85f6eac1143fab8ddf1659
2022-05-27 22:26:06 +00:00
Ivan Fraixedes
54aa3adbf7 scripts: Fix lint to show out on error
`os/exec.Cmd.CombineOutput` runs the command, hence, it cannot be used
after calling the `Run` method.

Because `CombineOutput` already runs the command, we can use it directly
instead of `Run`.

Without this change if the command returns an non-zero code we get an
error because of the command already started and we don't get the
output.

Example removing a copyright notice from one file and running the linter
(only showing the affected printed line)

Without this fix

  2022/05/26 15:48:40 [/storj check-copyright] error  exec: already started

With this fix

  2022/05/26 16:22:40 [/storj check-copyright] error missing copyright certificate/doc.go: %!w(<nil>)

NOTE the `%!w(<nil)` is a bug in the check-copyright linter.

Change-Id: I40b64842028399b92a8982bfb143e1f87f92467b
2022-05-27 09:32:48 +00:00
Mya
b74340b1d6 scripts/lint: fix timing of lint tasks
Change-Id: I068cad315fd05ee235413d8230d454507b0a60e4
2022-05-25 13:24:03 -05:00
Mya
9153f37055 Makefile: run lint locally in docker
Our linting process depends heavily on custom tools and linting
configuration. To help improve this process for running on local
developer machines, we can run the various tasks in our existing CI
container.

Change-Id: I60407686ce9233cc4f16e3724c5e8d44367aa200
2022-05-25 12:30:15 -05:00
Cameron
87f6a3dcda {web/satelliite, satellite/console}: logo redirects to homepage
logo redirects to homepage on login, signup, forgot password, reset
password, and activate account pages

Change-Id: I992aeae197004d620addd8d515cae1c1ca80a778
2022-05-23 16:26:19 -04:00
NickolaiYurchenko
99237d5c78 web/satellite: new bucket creation flow
old bucket creation flow removed
new flow added
name and passphrase splitted into separate views
demo bucket will not be created automatically
bucket creation progress bar added

Change-Id: I2a1d7d77c3038caaafb3c06bdb0ac5dd1ad17599
2022-05-22 11:55:55 +00:00
Michał Niewrzał
f9a3f19443 scripts: add 'uplink cp' tests for stdin/stdout
Uplink can upload from stdin and download to stdout. We had
such tests for old binary but now we were missing it.

Change-Id: I5110a9f531f5cc21277fa53611995fb5b556ff16
2022-05-18 08:42:04 +00:00
Malcolm Bouzi
087e57d037 web/satellite: create chore that will resend verification emails to unverified users
We want to remind unverified users to verify their emails:
once after 24 hours has passed and again after 5 days has passed.

Add mailservice.Service to satellite core because it is needed by the
chore for sending emails. To add the mailservice.Service to the core,
we create a helper function in satellite/peer.go to avoid duplicating
the code in both api.go and core.go. In addition to the chore, this
change adds methods to users.DB to get unverified users in need of
reminder.

Change-Id: I4e515bdf43f922788b4f965b2efb34fa32288bd1
2022-05-18 08:08:33 +00:00
JT Olio
5fb9ee3cfa uplink, satellite: use bgp hostnames
Change-Id: I58f5011d3019f8267fa8cbd3096b2cfe42eb5f8b
2022-05-16 18:14:37 +00:00
Cameron
0633aca607 satellite/console: create new consoleauth service
We want to send email verification reminders to users from the satellite
core, but some of the functionality required to do so exists in the
satellite console service. We could simply import the console service
into the core to achieve this, but the service requires a lot of
dependencies that would go unused just to be able to send these emails.

Instead, we break out the needed functionality into a new service which
can be imported separately by the console service and the future email
chore.

The consoleauth service creates, signs, and checks the expiration of auth
tokens.

Change-Id: I2ad794b7fd256f8af24c1a8d73a203d508069078
2022-05-13 16:27:07 +00:00
Moby von Briesen
763bfc0913 satellite/console,web/satellite: Implement hCaptcha
Adds a new configuration for hcaptcha enabled, secretkey, and sitekey.
If both reCAPTCHA and hCaptcha are configured as "enabled", reCAPTCHA
will be used.

Change-Id: I73cc6e133d8da3555e0ed8b2b377cf9eb263e6dc
2022-05-13 14:57:45 +00:00
Erik van Velzen
db1cc8ca95 satellite/repair/checker: buffer repair queue
Integrate previous changes. Speed up the segment loop by batch inserting
into repair queue.

Change-Id: Ib9f4962d91960d21bad298f7771345b0dd270276
2022-05-12 16:28:05 +00:00
Vitalii
dedccbd2e4 satellite/console, web/satellite: limit failed login attempts
Added account locking on 3 or more login attempts.
Includes both password and MFA failed attempts on login.
Unlock account on successful password reset.

Change-Id: If4899b40ab4a77d531c1f18bfe22cee2cffa72e0
2022-05-11 14:49:11 +00:00
Qweder93
384118ee1e satellite/metabase: use pgutil.NullTimestampTZArray
This code cleanup to use helper DB method.

Change-Id: Idc438234cb4f7da23d6ade2d69f5d889f15e9009
2022-05-03 19:30:21 +00:00
hovex023
58f957bd1d
web/satellite: Add new access grants flow (#4765)
* Added new feature Flag for new Access Grant Flow. 

* Added 3 cards to access grant view for S3, CLI and Access grant to replace old header

* Added new formatting, text and Icon for Access Grant Delete Popup modal
2022-04-29 10:31:52 -05:00
nadimhq
dd6aac9b39
satellite/metainfo: enable server-side copy in production (#4708)
these changes enable server-side copy in production
2022-04-26 09:58:16 -04:00