We would like to have separate process/command to collect bloom
filters from source different than production DBs. Such process will
use segment loop to build bloom filters for all storage nodes and
will send it to Storj bucket.
This change add main logic to new service. After collecting all bloom
filters with segment loop and piece tracker all filters are marshaled
and packed into zip files. Each zip contains up to "ZipBatchSize" bloom
filters and it's uploaded to specified in configuration bucket.
All uploaded objects have specified expiration time to not delete them
manually.
Updates https://github.com/storj/team-metainfo/issues/120
Change-Id: I2b6bc02a7dd7c3a639e75810fd013ae4afdc80a2
We would like to have separate process/command to collect bloom
filters from source different than production DBs. Such process will
use segment loop to build bloom filters for all storage nodes and
will send it to Storj bucket. This this initial change to add such
service. Added service is joining segment loop and collects all
bloom filters.
Sending bloom filters to the bucket will be added as a subsequent
change.
Updates https://github.com/storj/team-metainfo/issues/120
Change-Id: I2551723605afa41bec84826b0c647cd1f61f3b14
Sessions now expire after a much shorter amount of time, requiring
clients to issue API requests for session extension. This is handled
behind the scenes as the user interacts with the page, but once session
expiration is imminent, a modal appears which informs the user of his
inactivity and presents him with the choice of loging out or preserving
his session.
Change-Id: I68008d45859c814a835d65d882ad5ad2199d618e
This is in response to community feedback that our existing reputation
calculation is too likely to disqualify storage nodes unfairly with
extreme swings up and down.
For details and analysis, please see the data_loss_vs_dq_chance_sim.py
tool, the "tuning reputation further.ipynb" Jupyter notebook in the
storj/datascience repository, and the discussion at
https://forum.storj.io/t/tuning-audit-scoring/14084
In brief: changing the lambda and initial-alpha parameters in this way
causes the swings in reputation to be smaller and less likely to put a
node past the disqualification threshold unfairly.
Note: this change will cause a one-time reset of all (non-disqualified)
node reputations, because the new initial alpha value of 1000 is
dramatically different, and the disqualification threshold is going to
be much higher.
Change-Id: Id6dc4ba8fde1be3db4255b72282207bab5491ca3
Created new modal which shows user their native STORJ token wallet address.
There are QR and copy buttons.
It will be used only in new billing screen.
Change-Id: Icef3c8668c548b779c07fe2b85eb5761cd1221a3
Jenkins doesn't do a very good job with identifying what has been changed.
While it has a syntax to defined patterns, it compares the current build with the previous build (in case of git-verify it can be a totally different branch) instead of checking the HEAD commit.
This patch introduces shell scripts to do this better:
* It doesn't depend on Jenkins any more
* It can be executed locally
* It can detect web changes properly (see the relation change as an example).
Change-Id: I9d37775e3818c08c4aa96ffb78f84d57f28a2c95
We have enabled the new project dashboard in production. Change the
default to true so that we do not need an explicit configuration in
prod.
Change-Id: I0f93773965283e7b0682f6586685224281cbf78c
Implemented Recaptcha and Hcaptcha for login screen.
Slightly refactored registration page implementation.
Made 2 different login/registration captcha configs on server side to easily swap between captchas independently.
Issue: https://github.com/storj/storj/issues/4982
Change-Id: I362bd5db2d59010e90a22301893bc3e1d860293a
removed segment limit validation and checks in metainfo endpoint and accounting/projectusage
since feature is live and has always has segment limitation now
Resolves: https://github.com/storj/storj/issues/4470
Change-Id: I8cf87cbbc40ac61262f9f05e52573d3ae6410611
Currently we have a significant number of tallies that need to be
deleted together. Add a limit (by default 10k) to how many will
be deleted at the same time.
Change-Id: If530383f19b4d3bb83ed5fe956610a2e52f130a1
Added new email html template.
It is sent when user tries to reset password with unknown or unverified account.
Made a couple of minor config changes.
Issue: https://github.com/storj/storj/issues/4913
Change-Id: I730f48b3478e302d1e38e1f8a27c75f66a8ba6fd
We don't build "multinode" nor "storagenode" docker images for armv6
architecture, instead we build for armv5.
Fix the script that publish the manifest for those images for a specific
tag to use armv5 for not failing when executing.
Change-Id: I7d859d8718240e1cd0dae6489e7e5c3b4068ff6e
This change integrates the session management database functionality
with the web application. Claim-based authentication has been removed
in favor of session token-based authentication.
Change-Id: I62a4f5354a3ed8ca80272814aad2448f901eab1b
prevent network enumeration by rejecting privateIPs in PingMe and
Checkin endpoints
Closesstorj/storj-private#32
Change-Id: I63f00483ff4128ebd5fa9b7b8da826a5706748c9
Add storjscan wallets implementation to the satellite. The wallets interface allows you to add and claim new wallets as called by the API. The storjscan specific implementation of this interface uses a wallets DB to associate the user to a wallet address, as well as a storjscan client to request and associate new wallets to the satellite.
Change-Id: I54081edb5545d4e3ee07cf1cce3d3e87cc00c4a1
An older change plummed the full console config as subconfig of
the admin api configuration in. This bloated the generated satellite
configuration unnecessarily while also allow for confusion/mistakes.
Change-Id: Icf49cc1f147711e37e85f6eac1143fab8ddf1659
`os/exec.Cmd.CombineOutput` runs the command, hence, it cannot be used
after calling the `Run` method.
Because `CombineOutput` already runs the command, we can use it directly
instead of `Run`.
Without this change if the command returns an non-zero code we get an
error because of the command already started and we don't get the
output.
Example removing a copyright notice from one file and running the linter
(only showing the affected printed line)
Without this fix
2022/05/26 15:48:40 [/storj check-copyright] error exec: already started
With this fix
2022/05/26 16:22:40 [/storj check-copyright] error missing copyright certificate/doc.go: %!w(<nil>)
NOTE the `%!w(<nil)` is a bug in the check-copyright linter.
Change-Id: I40b64842028399b92a8982bfb143e1f87f92467b
Our linting process depends heavily on custom tools and linting
configuration. To help improve this process for running on local
developer machines, we can run the various tasks in our existing CI
container.
Change-Id: I60407686ce9233cc4f16e3724c5e8d44367aa200
logo redirects to homepage on login, signup, forgot password, reset
password, and activate account pages
Change-Id: I992aeae197004d620addd8d515cae1c1ca80a778
old bucket creation flow removed
new flow added
name and passphrase splitted into separate views
demo bucket will not be created automatically
bucket creation progress bar added
Change-Id: I2a1d7d77c3038caaafb3c06bdb0ac5dd1ad17599
Uplink can upload from stdin and download to stdout. We had
such tests for old binary but now we were missing it.
Change-Id: I5110a9f531f5cc21277fa53611995fb5b556ff16
We want to remind unverified users to verify their emails:
once after 24 hours has passed and again after 5 days has passed.
Add mailservice.Service to satellite core because it is needed by the
chore for sending emails. To add the mailservice.Service to the core,
we create a helper function in satellite/peer.go to avoid duplicating
the code in both api.go and core.go. In addition to the chore, this
change adds methods to users.DB to get unverified users in need of
reminder.
Change-Id: I4e515bdf43f922788b4f965b2efb34fa32288bd1
We want to send email verification reminders to users from the satellite
core, but some of the functionality required to do so exists in the
satellite console service. We could simply import the console service
into the core to achieve this, but the service requires a lot of
dependencies that would go unused just to be able to send these emails.
Instead, we break out the needed functionality into a new service which
can be imported separately by the console service and the future email
chore.
The consoleauth service creates, signs, and checks the expiration of auth
tokens.
Change-Id: I2ad794b7fd256f8af24c1a8d73a203d508069078
Adds a new configuration for hcaptcha enabled, secretkey, and sitekey.
If both reCAPTCHA and hCaptcha are configured as "enabled", reCAPTCHA
will be used.
Change-Id: I73cc6e133d8da3555e0ed8b2b377cf9eb263e6dc
Added account locking on 3 or more login attempts.
Includes both password and MFA failed attempts on login.
Unlock account on successful password reset.
Change-Id: If4899b40ab4a77d531c1f18bfe22cee2cffa72e0
* Added new feature Flag for new Access Grant Flow.
* Added 3 cards to access grant view for S3, CLI and Access grant to replace old header
* Added new formatting, text and Icon for Access Grant Delete Popup modal
"REST API" is a more accurate descriptor of the generated API in the
console package than "account management API". The generated API is very
flexible and will allow us to implement many more endpoints outside the
scope of "account management", and "account management" is not very well
defined to begin with.
Change-Id: Ie87faeaa3c743ef4371eaf0edd2826303d592da7
This also fixes the build order. Unfortunately we need
to ensure that the web frontends are built before installing
Go binaries.
Fixes https://github.com/storj/storj/issues/4654
Change-Id: I5d1c83125fd3d1a454d3400b2cbdd44bd3f2250c
We are in the process of creating an api to allow users to manage their
accounts programmatically. We would like to use api keys for
authorization. We were originally going to create an entirely new table
for these api keys, but seeing as we already have 2 other tables for
keys/tokens, api_keys and oauth_tokens, we thought it might be better to
use one of these. We're using oauth_tokens.
We create a new oidc.OAuthTokenKind for account management api keys:
KindAccountManagementTokenV0. We made the key versioned because we
likely want to improve the implementation in the future, but we want to
get something functional out the door ASAP because the account management
api feature is highly desired.
Add a new method to oidc.OAuthTokens interface for revoking v0 account
management api keys, RevokeAccountManagementTokenV0. Add update method
to dbx implementation to allow updating the expiration. We will revoke
these keys by setting the expiration to 0 so they are expired.
Change-Id: Ideb8ae04b23aa55d5825b064b5e43e32eadc1fba
