Commit Graph

843 Commits

Author SHA1 Message Date
Michal Niewrzal
67ad792d1a satellite/rangedloop: migrate segments verification from segment loop
Segments loop have build-in sanity check to verify if number of segments
processed by loop is roughly fine. We want to have the same verification
for ranged loop.

https://github.com/storj/storj/issues/5544

Change-Id: Ia19edc0fb4aa8dc45993498a8e6a4eb5928485e9
2023-03-08 17:00:11 +00:00
Erik van Velzen
464ceb1c0e satellite/gc: improve comments
Change-Id: I9e71c9bee3447f78365ba1593e4a4ef55b28356f
2023-03-08 13:15:13 +00:00
Sembeth
0408997e6c
cmd/multinode: Removes dependency on deprecated identity-dir flag, code and documentation. (#5646)
* The Multinode Dashboard no longer requires the identity files.

* Removed deprecated code that used the identity-dir.
2023-03-08 13:56:15 +01:00
Vitalii
a01d47b3ed satellite:{console, web}: remove old project dashboard
Removed old project dashboard along with a feature flag.

Change-Id: I9292b19caf101a84042d8c3006c27bc44db8d6b5
2023-03-07 13:34:59 +02:00
Wilfred Asomani
faeea88265 satellite/{db,analytics,payments}: add chore for auto account freeze
This change adds a new chore that will check for failed invoices and
potentially freeze corresponding accounts.
It makes slight modifications to stripemock.go and invoices.go (adding
stripe CustomerID to the Invoice struct).

Issue: https://github.com/storj/storj-private/issues/140

Change-Id: I161f4037881222003bd231559c75f43360509894
2023-03-01 09:31:27 +00:00
Cameron
b24ce98944 satellite/admin: add Groups to config
add new config to the satellite admin: --admin.groups.limit-update.
This can be used as an alternate means of authentication if the request
is coming from the oauth proxy.

Change-Id: Ic2de13862e6414244b060c66a0f2bed72097cbad
2023-02-27 16:07:20 +00:00
Cameron
8842985571 satellite/console/consoleweb: create purchase-package endpoint
Add new purchase-package endpoint to Server. The endpoint can be enabled
or disabled by a new config, --console.pricing-packages-enabled.
The purchase-package endpoint applies a coupon and adds and charges a
credit card if user's useragent is a partner with a configured package
plan.

github issue: https://github.com/storj/storj-private/issues/125

Change-Id: I0d6ccccd6874ddba360c45f338fd1c44f95e135a
2023-02-23 15:42:36 -05:00
Michal Niewrzal
8aa6110835 tests/backwardcompatibility: use noquic tag for release binaries
Older releases are not compiling with latest Go version if quic is used.
We need to add noquic tag to be able to compile older release with
latest Go version.

Change-Id: Id5768fcaa5c1f7cf3e6fbb633e7ca60309b7a37c
2023-02-21 13:26:46 +01:00
JT Olio
77bf88e916 satellite/overlay: check node difficulty before entering database
closes https://github.com/storj/storj/issues/5568

Change-Id: Id413637c2678e7a7cf8dbf414e082c687c8e8a39
2023-02-15 17:46:25 +00:00
Wilfred Asomani
9138e84fb1 satellite/{web,console}: feature flag for all projects dashboard
Adds a feature flag for the new all projects dashboard. It defaults to false.

Issue: https://github.com/storj/storj/issues/5514

Change-Id: I160904eccae7d30e05b734e69600725702b16aca
2023-02-15 15:53:31 +00:00
Egon Elbre
a8caea8b9c satellite/satellitedb: remove gob migration exceptions
Change-Id: I2958e2145edaa501d510314f9bfd0e5280ea82ec
2023-02-10 15:05:18 +02:00
Vitalii
65e3cfb9c6 satellite/{web, console}: feature flag for new access grant flow
Added a feature flag for new create access grant flow.
Also added some initial setup.

Change-Id: I7f738181c8a83f5a724f9e562427445cae146b6f
2023-02-09 00:35:35 +00:00
paul cannon
d6f8be1ec6 satellite/audit: add ContainmentSyncChore
We will be needing an infrequent chore to check which nodes are in the
reverify queue and synchronize that set with the 'contained' field in
the nodes db, since it is easily possible for them to get out of sync.
(We can't require that the reverification queue table be in the same
database as the nodes table, so maintaining consistency with SQL
transactions is out. Plus, even if they were in the same database, using
such SQL transactions to maintain consistency would be slow and
unwieldy.)

This commit adds the actual chore.

Refs: https://github.com/storj/storj/issues/5431
Change-Id: Id78b40bf69fae1ac39010e3b553315db8a1472bd
2023-02-07 01:18:49 +00:00
Michal Niewrzal
0b519fbd05 satellite/satellitedb: create index for interval_day for project_bandwidth_daily_rollup
To improve deletion of old entries in project_bandwidth_daily_rollup
we need index on `interval_day` column which is used to find those old
entries.

As an addition we are changing interval how often deletion is executed
from 7 to 1 day. We would like to have smaller portion of data to
delete.

Fixes https://github.com/storj/storj/issues/5465

Change-Id: Ie18ebe859887b93d6e4e6065a61fb9214c7ad27a
2023-02-03 10:02:58 +00:00
Egon Elbre
3976a2fd1d go.mod: bump common to latest
Temporarily allow larger wasm size, while we migrate to picobuf.

Change-Id: I55916f47486465e4223cbd79f412ab2876fee042
2023-02-01 12:33:37 +02:00
Vitalii
5377b9c314 satellite/{web, console}: removed old project level passphrase flow
Removed old behavior and bucket creation flow.

Issue:
https://github.com/storj/storj/issues/5521

Change-Id: I1a443911215cd1f611069fd9e6202598066b80b2
2023-01-31 23:02:44 +00:00
JT Olio
2a641b60d3 private/server: support tcp fastopen
we are not enabling it on client-side code yet, but
it will be hard to test this in the wild without
server-side support.

this is phase 2 of tcp fast open support. see
https://forum.storj.io/t/two-new-blueprints-design-drafts-seeking-feedback-replacing-tls-with-noise-and-tcp-fastopen/21053/12
for more details

Change-Id: I20362be4c49abab90afcc9b6572ef9f79816345b
2023-01-30 13:30:43 -05:00
Michal Niewrzal
bb2ac4279a satellite/metainfo: enable multiple versions fix by default
Change-Id: I6cc7ba928e59cc8b8fa50f2ab19ec5418dc76507
2023-01-26 09:35:20 +00:00
Andrew Harding
e7b35381f2 mod: bump common and uplink deps
Change-Id: Ie2d87f517b57a8760f50ac5c9f1b8b0448f5b3b5
2023-01-25 13:34:47 -07:00
JT Olio
e40191afd6 storj: upgrade to use latest storj/common NodeAddress
Change-Id: I5987391bcfe5f6dfd7b525698c337a4cbda9b76e
2023-01-25 01:37:26 +00:00
Qweder93
d6a948f59d satellite/repair : implemented ranged loop observer
implemented observer and partial, created new structures to keep mon
metrics remain in same way as in segment loop

Change-Id: I209c126096c84b94d4717332e56238266f6cd004
2023-01-23 14:23:03 +00:00
Cameron
7ffa9ef914 satellite/payments: add config for partner package plans
Create a config to specify one-time prices and corresponding coupon
ids for partners.

github issue: https://github.com/storj/storj-private/issues/118

Change-Id: I67b26e7208b12ba8f0e6dc1b164dd9545b09cac0
2023-01-20 02:33:36 +00:00
Vitalii
65a090d111 satellite/console: enable project level passphrase by default
Enable project level passphrase by default.

Issue:
https://github.com/storj/storj/issues/5456

Change-Id: I8c08f2d81a3ce1b5e0134172b7440309b850c5c2
2023-01-18 15:32:29 +02:00
Yaroslav Vorobiov
5644fb1a7e satellite/accounting/nodetally: add ranged loop
Add node tally ranged loop observer and partial.
Add node tally randed observer to range loop peer.
Add config flag to select which loop to use for node tally.
Update satellite core to use segement/ranged loop based on a flag.
Duplicate existing node tally test but using ranged loop.

Change-Id: I6786f1a16933463fab5f79601bf438203a7a5f9e
2023-01-17 13:50:18 +01:00
Qweder93
8c69ee62fc {cmd/storj-sim, satellite/rangedloop}: added rangedloop to storj-sim, removed identity
added in storj-sim rangedloop for each satellite, to verify it works for metrics oveserver,
removed identity from rangedloop peer as we never use it, added logs on service run, added loop
to service instead of endless for loop, interval value to config

Closes: https://github.com/storj/storj/issues/5414

Change-Id: Ibc3b06071b68feda4a35b45da2bbe36e22a02fc8
2023-01-05 11:29:00 +00:00
Andrew Harding
5362dff94b satellite/gc/bloomfilter: implement rangedloop observer
https://github.com/storj/storj/issues/5235

Change-Id: Iffe8f682adfa46e48e47976bf838326e7125ff80
2023-01-03 09:46:02 -07:00
Andrew Harding
590d44301c satellite/audit: implement rangedloop observer
This change implements the ranged loop observer to replace the audit
chore that builds the audit queue.

The strategy employed by this change is to use a collector for each
segment range to  build separate per-node segment reservoirs that are
then merge them during the join step.

In previous observer migrations, there were only a handful of tests so
the strategy was to duplicate them. In this package, there are dozens
of tests that utilize the chore. To reduce code churn and maintenance
burden until the chore is removed, this change introduces a helper that
runs tests under both the chore and observer, providing a pair of
functions that can be used to pause or run the queueing function.

https://github.com/storj/storj/issues/5232

Change-Id: I8bb4b4e55cf98b1aac9f26307e3a9a355cb3f506
2023-01-03 08:52:01 -07:00
Wilfred Asomani
ea00213b45 console/userinfo: implement get userinfo
This change implements the get userinfo endpoint stubbed in #5358

Issue: https://github.com/storj/storj/issues/5363

Change-Id: I8d8deb0be1375395bbf3f52e7718990bc77f00b5
2022-12-23 21:03:54 +00:00
Vitalii
af238e2ef9 web/satellite: removed old create access grant flow
Removed old flow and feature flag.

Issue:
https://github.com/storj/storj/issues/5407

Change-Id: I9dec18eb7d8c7912ead87188789466db5f59a5ca
2022-12-23 15:14:02 +00:00
Andrew Harding
4241e6bf5f satellite/gracefulexit: implement rangedloop observer
The tests are forked from the chore tests with slight adaptations for
being run against the ranged loop. I also moved a benchmark for the
database from chore_test.go to db_test.go.

The pathcollector is reused as a rangedloop.Partial.

https://github.com/storj/storj/issues/5234

Change-Id: I56182031d133812a9f4d4a433c01b9150af39f31
2022-12-22 10:47:10 -07:00
Wilfred Asomani
e598c2b3b1 console/userinfo: stub userinfo endpoint
This change stubs userinfo endpoint from storj/common/pb/userinfo.proto.
It also adds config for allowed peers, and a method for verifying peers.

Issue: https://github.com/storj/storj/issues/5358

Change-Id: I057a0e873a9e9b3b9ad0bba69305f0d708bd9b9e
2022-12-20 06:10:08 +00:00
Michal Niewrzal
5110803102 storagenode/piecestore: add Exists endpoint
Adds new method Exists which can be used to verify which
requested piece ids exists on storage node. Will verify only pieces
which belongs to the satellite that used that endpoint.

Minum WASM size was increased a bit.

https://github.com/storj/storj/issues/5415

Change-Id: Ia5f9cadeb526541b2776a8973eb7d50133ad8636
2022-12-17 04:08:26 +00:00
Andrew Harding
99206fca04 satellite/payments/stripecoinpayments: skip line item generation for records with no usage
This change updates the stripecoinpayments service to optionally skip
generating line items for payments records that have no egress, storage,
or segments for the billing period.

This results in a reduction from 4 to 1 Stripe API calls for customers
who have no usage. The final API call is the attempt to generate an
invoice on stripe, which expectedly fails because there are no unapplied line
items. Removing that final API call would require some additional
queries and is out of scope for this change.

This functionality is behind the
`payments.stripe-coin-payments.skip-empty-invoices` feature flag.

https://github.com/storj/storj/issues/5381

Change-Id: Id184969a4c79047c40502336d69c51388ab03bf8
2022-12-16 17:02:50 +00:00
Andrew Harding
1cb2eb4c3b satellite/rangedloop: wire up metrics observer
Final touches on https://github.com/storj/storj/issues/5236.

Change-Id: I2259ec4e7825d20db9efb36beb42d6309dee55ba
2022-12-12 19:06:23 +00:00
Jeremy Wharton
ba7d2c2dbe satellite/payments/stripecoinpayments: add config for price overrides
This change adds configuration flags for defining partner-specific
project usage price overrides.

Resolves https://github.com/storj/storj-private/issues/61

Change-Id: Ia535ac22576382211d045f9ff2c9b983a07e86f3
2022-12-09 15:33:27 +00:00
Erik van Velzen
ff6d640fca
satellite/metabase/rangedloop: minimal loop (#5334)
Minimal implementation of the ranged (=threaded) segment loop
service, to improve performance over the existing loop.

Has tests with a an inmemory segment database
and example observer.

Does not have yet: database link, observer duration tracking,
suspicious processed ratio guard, rate limiting, minimum execution
interval per observer, etc.

Part of https://github.com/storj/storj/issues/5223

Change-Id: I08ffb392c3539e380f4e7b4f1afd56c4c394668d
2022-12-08 15:27:21 +01:00
Cameron
907c911f57 satellite/console/emailreminders: set Enable config default to true
Enable email verification reminders by default

Change-Id: I2f8143e2805ccfd2ffccfbc019591859326499c5
2022-12-08 11:02:43 +00:00
Michal Niewrzal
4544eee72b Revert "satellite/metainfo: enable metainfo.multiple-versions flag by default"
This reverts commit f0ce8996c3.

We need to revert it until https://github.com/storj/storj/issues/5373 is
fixed.

Change-Id: Ibb22af100014724d1910d4871d8f4e159fdea391
2022-12-07 19:43:20 +00:00
Wilfred Asomani
c1ed5c06e8 satellite/{admin,ui}: implement changes for oauth2 proxy
We want to put the OAuth2 proxy, https://github.com/oauth2-proxy/oauth2-proxy, in front of the satellite admin ui.
This change implements the changes required/necessary for this to work.

Issue: https://github.com/storj/storj/issues/5072

Change-Id: I6da0df090cc6f0c18f1bf41e48ae082493f53f20
2022-12-06 09:44:04 +00:00
Cameron
4a3c2cb7d0 satellite/nodeevents: implement customer.io nodeevents.Notifier
The CustomerioNotifier sends node event data to customer.io.

Change-Id: I40a942d6666ae5723c6acb84b11638497f4423f8
2022-12-02 16:50:49 +00:00
Yaroslav Vorobiov
bb1e86c790 satellite: remove unused coinpayments code and chores
issue: https://github.com/storj/storj/issues/4824

Change-Id: I2e3e63151d1def96270279719f6eceda0acba66c
2022-11-30 16:24:48 +00:00
Wilfred Asomani
2442ba415f satellite/{web,console}: token links expiry changes
This change reduces the token links expiry time from 24h to 30m and improves the UI to promt users of the expiration.

see: https://github.com/storj/storj-private/issues/17

Change-Id: Iac00f5740fa84069937fdf9bd30a739b6db2a9e0
2022-11-29 21:44:42 +00:00
paul cannon
b612ded9af satellite/audit: help performance of pushing to audit queue
The audit chore will be pushing a large number of segments to be
audited, and the db might choke on that large insert when under load.

This change divides the insert up into batches, which can be sized
however is optimal for the backing database. It also arranges for
segments to be inserted in the order of the primary key, which helps
performance on some systems.

Refs: https://github.com/storj/storj/issues/5228

Change-Id: I941f580f690d681b80c86faf4abca2995e37135d
2022-11-29 15:37:49 +00:00
Moby von Briesen
3501656e98 satellite/repair: Add flag to allow disabling reputation updates
Reputation updates during repair currently consumes a lot of database
resources. Sometimes increasing the rate of repair is more important
than auditing a node based on whether they have or don't have the
correct piece during repair. This is the job of the audit service.

This commit is to implement an intermediate solution from this issue: https://github.com/storj/storj/issues/5089
This commit does not address the more in-depth fix discussed here: https://github.com/storj/storj/issues/4939

Change-Id: I4163b18d78a96fadf5265789fd73c8aa8def0e9f
2022-11-24 08:31:11 -05:00
Michal Niewrzal
f0ce8996c3 satellite/metainfo: enable metainfo.multiple-versions flag by default
We tested new upload flow (with multiple versions) to fix inconsistency
while uploading object on QA/EUN1/SLC. Now we would like to enable it
for all satellites by default. Tests required small adjustments.

Fixes https://github.com/storj/storj/issues/5283

Change-Id: I0d53c041abebc0d182ba5a88bb1dac906c29caf0
2022-11-23 17:05:22 +00:00
paul cannon
8b494f3740 satellite/audit: use db for auditor queue
As part of the effort of splitting out the auditor workers to their own
process, we are transitioning the communication between the auditor
chore and the verification workers to a queue implemented in the
database, rather than the sequence of in-memory queues we used to use.

This logical database is safely partitionable from the rest of
satelliteDB.

Refs: https://github.com/storj/storj/issues/5251

Change-Id: I6cd31ac5265423271fbafe6127a86172c5cb53dc
2022-11-22 14:04:00 +00:00
Michal Niewrzal
d5eea2db61 satellite/accounting: use custom query for bucket tally by default
We added alternative way to calculate bucket tallies for accounting and
now it's tested and we will enable it by default.

CollectBucketTallies was extended to support overriding current time
to be able to test handling expired objects.

Change-Id: I738b99a33fd2e086245f92d874c1cbb806e834c0
2022-11-22 10:23:40 +00:00
Egon Elbre
954d703533 ci: use check-cross-compile tool
Change-Id: I2b858222c33eb0c9d1081ba8853a852ee14b32bd
2022-11-22 09:38:46 +00:00
Clement Sam
3378215adf satellite/orders: decrease order expiration time to 24hours
Closes https://github.com/storj/storj/issues/5202

Change-Id: I55d1a84c46dd610eeb00dd79df8f4f7e699499a0
2022-11-21 14:52:32 +00:00
Cameron
87660bd9b3 satellite/overlay/offlinenodes: insert offline nodes into node events
Add a new chore to periodically insert nodes who are offline and
have not gotten an offline email in a certain amount of time into node
events

Change-Id: I658b385bb777b0240c98092946a93d65bee94abc
2022-11-18 12:10:06 -05:00
Cameron
57be07f60a satellite/nodeevents: add Chore
Create NodeEvents Chore on satellite core to read nodeevents DB and
notify node operators on node events. The chore sends notifications
grouped by email and event type: it selects the oldest entry in
nodeevents.DB and also any other event with the same email and event
type no matter how old it is. The oldest entry of a group must exist for
a minimum amount of time before that group can be selected, however.
This minimum amount of time is a configurable value:
--node-events.selection-wait-period. This wait period allows us to
combine events of the same time and same email address into a singular
email.

Change-Id: I8b444aa324d2dae265cc27d9e9e85faef79195d8
2022-11-18 12:00:14 +00:00
Jeremy Wharton
e339f1a785 satellite/console: enable session timeout by default
This change causes the session inactivity timer to be enabled unless
expressly specified otherwise.

Change-Id: I85b4014394afac2feb21f383cac414cddb09ca8f
2022-11-17 17:41:15 +00:00
Moby von Briesen
776f1128b3 web/satellite: project level passphrase vuex logic
Added new feature flag.
Reworked vuex logic to work properly with project level passphrase.
Implemented new simple set project level passphrase modal.

Issue:
https://github.com/storj/storj/issues/5280

Change-Id: I6a15e90ee9fa7aa8a09c67022466787090120f9c
2022-11-17 16:53:25 +00:00
Cameron
d856569935 satellite/overlay: node software update email cooldown config
Change-Id: I792eef4a570e38e94f79a3f73c204b65f86ab541
2022-11-11 20:14:25 +00:00
Wilfred Asomani
9a303b00bf satellite/analytics: replace hubspot api with oauth access token
Hubspot is migrating from using API keys for authentication to OAuth.
This change migrates our Hubspot integration to use OAuth tokens.
It modifies the EnqueueCreateUser code to not send empty HubspotUTK to hubspot, and to return error for failed requests.

see: https://developers.hubspot.com/changelog/upcoming-api-key-sunset

Change-Id: I422f00e3e3caeff3ff3d08ddec059502b9addaee
2022-11-08 16:10:32 +00:00
Igor
ec6a79af85
scripts/changelog.py: a few fixes (#5293) 2022-11-08 16:44:28 +02:00
Moby von Briesen
cc858f4e91 satellite/console: Turn new billing screens on by default
Set default value for console.new-billing-screen to true.

Change-Id: I0f7f9663af4b4dc384d2bda946b4853593151cfa
2022-11-04 19:41:54 +00:00
Márton Elek
603bc019e1 build: use specific tag for nightly deployments
This patch is required to fix the nightly deployment:

 * We need to use the exact docker image tag what we built earlier
 * Migration should be full instead of snapshot (snapshot couldn't update existing, but older dbs)

Change-Id: Id2a2070638072a7b0021326326b0d53533817168
2022-11-02 17:57:20 +00:00
Michal Niewrzal
d21bbab2b2 satellite: fix metabase configuration wiring
New flag 'MultipleVersions' was not correctly passed from metainfo
configuration to metabase configuration. Because configuration was
set correctly for unit tests we didn't catch it and issue was found
while testing on QA satellite.

This change reduce number of places where new metabase flags needs
to be propagated from metainfo configuration to avoid problems with
setting new flags in the future.

Fixes https://github.com/storj/storj/issues/5274

Change-Id: I74bc122649febefd87f665be2fba628f6bfd9044
2022-11-02 15:17:34 +00:00
Egon Elbre
947cdf0fb4 ci: fixes for new Go version
We need to make exceptions for older uplink versions, because it does
not compile with newer Go versions due to quic dependency.

Change-Id: I3e073694f0942029c56740f0689088058ee068c3
2022-11-02 15:17:46 +02:00
Qweder93
5e5d6ecf6c satellite/{accounting, buckets}: added feature that allows to use custom SQL query instead of object loop for buckets accounting
since amount of objects is growing and looping through all of them
starts taking lot of time, we are switching for SQL query to do it
in chunks of tallies per bucket. 2nd part of issue fix.

Closes https://github.com/storj/team-metainfo/issues/125

Change-Id: Ia26bcac0a7e2c6503df9ebbf4817a636841d3284
2022-11-02 09:01:33 +00:00
Ethan
4efde65c9e satellite/gc: Optionally run the GC bloomfilter process once, instead of in a loop
The current deployment strategy requires that the GC bloomfilter generation process executes only once and exits.

Change-Id: I952991f126596aa165d1f2e9fce6f8548c21bdba
2022-11-01 18:19:40 +00:00
Egon Elbre
aeb645d32b all: replace deprecated ioutil
Change-Id: I60b0bbf5b68b066e2d44b8b99438594d600a3c2d
2022-10-31 15:50:41 +00:00
Igor
982b9a1844
scripts/draft-release.sh: add missing quotes 2022-10-31 14:15:49 +02:00
Wilfred Asomani
2dc2669e22 console/abTesting: add support for AB testing
This change adds support for AB testing using flagship.io

Change-Id: I3e12f5d6cd7248d69adc2c684e4bcff2aadda1df
2022-10-27 10:57:12 +00:00
Márton Elek
29aac75d80 build: provides earthfile for nightly build
Earthly is a build tool, it uses buildkitd to create reproducable and highly cacheable builds.

It is used by a new experimental nightly build to easily create storj-up images. (but can be used for any ad-hoc storj-up cluster to create the images).

To make the nightly more robust, I would prefer to commit the helper files (today I do a rebase every time, but sometimes it fails).

More detailed information about Earthly can be found at https://earthly.dev or https://www.youtube.com/watch?v=nChpMEdOaCQ

Change-Id: I683601e0558aca53b45ed3819c46c909534f8b15
2022-10-27 09:25:17 +00:00
Egon Elbre
bcbf30a333 ci: make go compatibility check parallel
Change-Id: I40162548305dcd181cfb702f72830d2c0a887cf4
2022-10-20 15:41:03 +00:00
Erik van Velzen
2ed18a0ed6 satellite/metainfo/piecedeletion: configurable threshold
The threshold of piece deletions from the nodes during CommitObject
when overriding an existing object seemed to cause a race condition in
tests.

This change makes the threshold configurable so we can set it to maximum
so CommitObject waits until all pieces are removed from the nodes in the
test.

Change-Id: Idf6b52e71d0082a1cd87ad99a2edded6892d02a8
2022-10-20 16:58:05 +02:00
Vitalii
149b59069f satellite/{web, console}: removed old object flow
Removed old flow along with a feature flag

Issue:
https://github.com/storj/storj/issues/5166

Change-Id: I85cfabbf94e910c9d5efb01ef850f6b888a13f71
2022-10-20 16:27:50 +03:00
Cameron
a52f766273 satellite/overlay: add email-sending functionality to overlay service
We want to send emails to SNOs. Node status changes go through the
overlay service, so it's a good place to add the mail service.
Add the mailservice.Service, satellite address, and satellite name to
overlay service. Also add feature flag --overlay.send-node-emails

Change-Id: I3bd2cb3bf22f9724954ce2374f8b651b902b3a24
2022-10-13 18:01:05 +00:00
Márton Elek
ea106f8894 go.mod: bump storj/private
Change-Id: I087b39c267c6492c4a5c85f470631b2f206cdf39
2022-10-13 11:23:57 +00:00
Egon Elbre
ff22fc7ddd all: fix deprecated ioutil commands
Change-Id: I59db35116ec7215a1b8e2ae7dbd319fa099adfac
2022-10-11 15:27:29 +00:00
dlamarmorgan
d39caf7674 cmd/satellite: change invoicing flags to iso8601
Update input parameters for invoicing commands to use iso8601 standard formatting.

Change-Id: I3b4a7df268e8f1a5cd51b0dab03ed38e13a76554
2022-10-07 07:58:23 -07:00
Igor
96d4968041
use python2 for draft release (#5216) 2022-09-29 16:22:08 +03:00
Fadila Khadar
02924d0ded satellite/metainfo: temporary feature flag for listing query testing
Fixes: https://github.com/storj/storj/issues/5144

Change-Id: I7650f4d5dd0378e2246339e79710a695996a845c
2022-09-28 17:46:52 +00:00
Igor
352e937813
scripts: add changelog to release draft (#4791)
* scripts: add changelog to release draft

* scripts: fix run command

* rename variable

* rename variable

* fix command

* add comment

* rename variable

Co-authored-by: Stefan Benten <mail@stefan-benten.de>
2022-09-28 19:17:40 +03:00
dlamarmorgan
88328ab240 satellite/payments/billing/billing.go: change default loop interval
Change the default loop interval for querying for new payments and adding them into the billing table from 1 minute to 15 seconds.

Change-Id: I26cf4a764cbe1de4c9b839ad60352374d8231522
2022-09-27 11:42:28 -07:00
dlamarmorgan
0b468aca56 satellite/payments/storjscan/storjscan.go: change default block confirmations
Change the default number of required block confirmations for a payment to be confirmed from 12 to 15.

Change-Id: I44c258134c293e7691623bc00c504130aa69a96a
2022-09-23 14:30:05 +00:00
Artur M. Wolff
fe85fefcc5 ci: add check-downgrades checks
Change-Id: Idfe81897dc701c24c0f81b0021cb1d640e8224ac
2022-09-23 11:15:27 +00:00
Michal Niewrzal
4e71dd302a satellite/{metainfo,metabase}: prepare feature flag MultipleVersions
We will introduce new logic for creating new objects (BeginObject).
Instead of using single version internally (1) we will be selecting first
available version during object creation. Because we need to be sure
that everything is wired up correctly we need a feature flag to be
able to control if new feature is enabled.

Change-Id: If0f8496397130811f43bf9db9fdcc2b30cd2e4ca
2022-09-21 13:07:13 +00:00
Erik van Velzen
e6b5501f9b satellite/gc/sender: new service to send retain filters
Implement a new service to read retain filter from a bucket and
send them out to storagenodes.

This allows the retain filters to be generated by a separate command on
a backup of the database.

Paralellism (setting ConcurrentSends) and end-to-end garbage collection
tests will be restored in a subsequent commit.

Solves https://github.com/storj/team-metainfo/issues/121

Change-Id: Iaf8a33fbf6987676cc3cf74a18a8078916fe673d
2022-09-20 11:49:40 +00:00
Artur M. Wolff
5150ee5001 mod: bump storj.io/private
Change-Id: I534aeedead1f1a2c5099e80b49fa5a8ea599fac8
2022-09-19 13:09:31 +00:00
Michal Niewrzal
a3d9630336 scripts: move tests to separate folders
Doing some cleanup in "scripts" folder. All integration like tests are
moved under "test" directory (integration, bc, redis) and bash scripts
are adjusted to reflect new location.

As an addition "scripts/install-awscli.sh" was deleted as it was not
used.

Change-Id: I152905c4258f471a71f2d0e8731d91bb075e99c1
2022-09-16 11:10:18 +00:00
Michal Niewrzal
158eb2381e satellite/gc/bloomfilter: uploading bloom filters
We would like to have separate process/command to collect bloom
filters from source different than production DBs. Such process will
use segment loop to build bloom filters for all storage nodes and
will send it to Storj bucket.

This change add main logic to new service. After collecting all bloom
filters with segment loop and piece tracker all filters are marshaled
and packed into zip files. Each zip contains up to "ZipBatchSize" bloom
filters and it's uploaded to specified in configuration bucket.

All uploaded objects have specified expiration time to not delete them
manually.

Updates https://github.com/storj/team-metainfo/issues/120

Change-Id: I2b6bc02a7dd7c3a639e75810fd013ae4afdc80a2
2022-09-12 08:33:53 +00:00
Michal Niewrzal
68f6d93f29 satellite/gc/bloomfilter: add service to collect bloom filters
We would like to have separate process/command to collect bloom
filters from source different than production DBs. Such process will
use segment loop to build bloom filters for all storage nodes and
will send it to Storj bucket. This this initial change to add such
service. Added service is joining segment loop and collects all
bloom filters.

Sending bloom filters to the bucket will be added as a subsequent
change.

Updates https://github.com/storj/team-metainfo/issues/120

Change-Id: I2551723605afa41bec84826b0c647cd1f61f3b14
2022-09-02 08:10:46 +00:00
Jeremy Wharton
3f26cc599f satellite/console,web/satellite: invalidate sessions after inactivity
Sessions now expire after a much shorter amount of time, requiring
clients to issue API requests for session extension. This is handled
behind the scenes as the user interacts with the page, but once session
expiration is imminent, a modal appears which informs the user of his
inactivity and presents him with the choice of loging out or preserving
his session.

Change-Id: I68008d45859c814a835d65d882ad5ad2199d618e
2022-08-23 15:51:05 +00:00
paul cannon
0dcc0a9ee0 satellite/reputation: reconfigure lambda and alpha
This is in response to community feedback that our existing reputation
calculation is too likely to disqualify storage nodes unfairly with
extreme swings up and down.

For details and analysis, please see the data_loss_vs_dq_chance_sim.py
tool, the "tuning reputation further.ipynb" Jupyter notebook in the
storj/datascience repository, and the discussion at

    https://forum.storj.io/t/tuning-audit-scoring/14084

In brief: changing the lambda and initial-alpha parameters in this way
causes the swings in reputation to be smaller and less likely to put a
node past the disqualification threshold unfairly.

Note: this change will cause a one-time reset of all (non-disqualified)
node reputations, because the new initial alpha value of 1000 is
dramatically different, and the disqualification threshold is going to
be much higher.

Change-Id: Id6dc4ba8fde1be3db4255b72282207bab5491ca3
2022-08-17 18:52:53 +00:00
dlamarmorgan
f596f72f44 satellite/payments/storjscan: add billing update to chore
Update chore to add confirmed transactions to the billing table.

Change-Id: I3c9a98c2ddc96f8a8905250376a1f5490d810277
2022-08-16 11:18:15 -07:00
Vitalii
0ec28ca623 web/satellite: add token funds modal
Created new modal which shows user their native STORJ token wallet address.
There are QR and copy buttons.
It will be used only in new billing screen.

Change-Id: Icef3c8668c548b779c07fe2b85eb5761cd1221a3
2022-08-15 14:17:30 +03:00
Márton Elek
96b88e286e build: fix conditional run of web builds
Jenkins doesn't do a very good job with identifying what has been changed.

While it has a syntax to defined patterns, it compares the current build with the previous build (in case of git-verify it can be a totally different branch) instead of checking the HEAD commit.

This patch introduces shell scripts to do this better:
 * It doesn't depend on Jenkins any more
 * It can be executed locally
 * It can detect web changes properly (see the relation change as an example).

Change-Id: I9d37775e3818c08c4aa96ffb78f84d57f28a2c95
2022-08-11 14:34:44 +00:00
Moby von Briesen
3f6bf64af9 satellite/console: Default NewProjectDashboard to "true"
We have enabled the new project dashboard in production. Change the
default to true so that we do not need an explicit configuration in
prod.

Change-Id: I0f93773965283e7b0682f6586685224281cbf78c
2022-08-06 01:00:31 +00:00
Vitalii
ad37ea4518 satellite/{web, console}: login captcha implemented
Implemented Recaptcha and Hcaptcha for login screen.
Slightly refactored registration page implementation.
Made 2 different login/registration captcha configs on server side to easily swap between captchas independently.

Issue: https://github.com/storj/storj/issues/4982

Change-Id: I362bd5db2d59010e90a22301893bc3e1d860293a
2022-08-03 23:02:27 +00:00
Jeff Wendling
f343b856dd go.mod: bump common/private/monkit/monkit-jaeger
this includes all of the fixes identified for end to end
tracing to finally work.

Change-Id: Ia433a2cf9130dc43b6810379af32f5c10f9d3888
2022-07-27 16:17:00 +00:00
Qweder93
2b24632610 {satellite/metainfo, satellite/accounting}: remove ValidateSegmentLimit config value and check
removed segment limit validation and checks in metainfo endpoint and accounting/projectusage
since feature is live and has always has segment limitation now

Resolves: https://github.com/storj/storj/issues/4470

Change-Id: I8cf87cbbc40ac61262f9f05e52573d3ae6410611
2022-07-27 15:36:53 +00:00
Egon Elbre
82fede2132 satellite/satellitedb: batch delete storage node tallies
Currently we have a significant number of tallies that need to be
deleted together. Add a limit (by default 10k) to how many will
be deleted at the same time.

Change-Id: If530383f19b4d3bb83ed5fe956610a2e52f130a1
2022-07-20 15:42:35 +03:00
paul cannon
2f20bbf4d8 satellite/reputation: add a reputation write cache
This should lower the amount of database load coming from
reputation updates.

Change-Id: Iaacfb81480075261da77c5cc93e08b24f69f8949
2022-07-14 21:40:16 +00:00
Yaroslav Vorobiov
de5951331a satellite: wire storjscan chore to core process
Adds storjscan chore and storjscan client to satellite chore.
Storjscan chore interval added to storjscan config.

Change-Id: I13bb8d83b14ca90f4735c9a10c361ad24a56b256
2022-07-14 15:07:52 +00:00
Vitalii
69dc9a4731 satellite/console: added new email which is sent on unknown password reset
Added new email html template.
It is sent when user tries to reset password with unknown or unverified account.
Made a couple of minor config changes.

Issue: https://github.com/storj/storj/issues/4913

Change-Id: I730f48b3478e302d1e38e1f8a27c75f66a8ba6fd
2022-07-14 14:32:59 +00:00
littleskunk
cf750716ea
satellite/console: enable new access grant flow (#4934) 2022-06-30 00:06:39 +02:00
JT Olio
18666b078a metabase: support disabling serverside copy when already enabled
Change-Id: I99b0b5597537f69fbdd8cab479e059ba4bf310bd
2022-06-24 17:56:43 +00:00
Márton Elek
204367b74d
build: premerge/verify Jenkinsfile (subsets of Jenkinsifle.public)
Change-Id: I33c65a37ecd8fcb89729b2a8db3906c6b8890ca3
2022-06-23 09:15:43 +02:00
Ivan Fraixedes
fc4b0e41f0 scripts: Change Storagenode arm32v6 to arm32v5
We don't build "multinode" nor "storagenode" docker images for armv6
architecture, instead we build for armv5.

Fix the script that publish the manifest for those images for a specific
tag to use armv5 for not failing when executing.

Change-Id: I7d859d8718240e1cd0dae6489e7e5c3b4068ff6e
2022-06-21 11:54:38 +00:00
Jeremy Wharton
58c5d44f44 satellite/console: integrate sessions into satellite UI
This change integrates the session management database functionality
with the web application. Claim-based authentication has been removed
in favor of session token-based authentication.

Change-Id: I62a4f5354a3ed8ca80272814aad2448f901eab1b
2022-06-13 08:02:02 +00:00
JT Olio
098ee71e03 test-sim-versions: write what bash step is executing
Change-Id: If1231f69601c1774bc6ce116f45105cc1b2f4e22
2022-06-10 19:33:03 +00:00
Yaroslav Vorobiov
62774768d9 satellite/payments/storjscan: update config and wrap service errors
Change-Id: I4f75375e9499119cbcce78c5e77d1950e4bae38b
2022-06-07 14:21:42 +00:00
Paul Willoughby
911cc1e163 satellite/contact: reject privateIPs in PingMe and CheckIn endpoints
prevent network enumeration by rejecting privateIPs in PingMe and
Checkin endpoints

Closes storj/storj-private#32

Change-Id: I63f00483ff4128ebd5fa9b7b8da826a5706748c9
2022-06-07 08:09:14 +00:00
Vitalii
ba58530089 {satellite}/web,console,testsuite: remove old navigation structure
Removed old satellite UI navigation structure.
Removed old feature flag.

Change-Id: Ic998886cf2e30ebd44e67a20fc53888103fe4b8d
2022-06-06 16:43:05 +03:00
dlamarmorgan
270204f352 satellite/{payments/storjscan,satellitedb}: Add wallet implementation
Add storjscan wallets implementation to the satellite. The wallets interface allows you to add and claim new wallets as called by the API. The storjscan specific implementation of this interface uses a wallets DB to associate the user to a wallet address, as well as a storjscan client to request and associate new wallets to the satellite.

Change-Id: I54081edb5545d4e3ee07cf1cce3d3e87cc00c4a1
2022-06-03 11:45:47 +00:00
cl-mitch
cbaca8b17e
web/satellite added new billing screen feature flag (#4836)
New feature flag implemented to enable the new billing flow.

Co-authored-by: Maximillian von Briesen <mobyvb@gmail.com>
2022-06-02 09:30:27 -05:00
littleskunk
6cdd250019
satellite/console: enable new object flow / onboarding (#4851) 2022-06-01 14:04:03 +02:00
littleskunk
3d1b469f35
satellite/console: disable account activation reminder email (#4844)
* satellite/console: disable account activation reminder email
2022-05-31 17:36:40 +02:00
Stefan Benten
8673583582 satellite/admin: fix console config handling
An older change plummed the full console config as subconfig of
the admin api configuration in. This bloated the generated satellite
configuration unnecessarily while also allow for confusion/mistakes.

Change-Id: Icf49cc1f147711e37e85f6eac1143fab8ddf1659
2022-05-27 22:26:06 +00:00
Ivan Fraixedes
54aa3adbf7 scripts: Fix lint to show out on error
`os/exec.Cmd.CombineOutput` runs the command, hence, it cannot be used
after calling the `Run` method.

Because `CombineOutput` already runs the command, we can use it directly
instead of `Run`.

Without this change if the command returns an non-zero code we get an
error because of the command already started and we don't get the
output.

Example removing a copyright notice from one file and running the linter
(only showing the affected printed line)

Without this fix

  2022/05/26 15:48:40 [/storj check-copyright] error  exec: already started

With this fix

  2022/05/26 16:22:40 [/storj check-copyright] error missing copyright certificate/doc.go: %!w(<nil>)

NOTE the `%!w(<nil)` is a bug in the check-copyright linter.

Change-Id: I40b64842028399b92a8982bfb143e1f87f92467b
2022-05-27 09:32:48 +00:00
Mya
b74340b1d6 scripts/lint: fix timing of lint tasks
Change-Id: I068cad315fd05ee235413d8230d454507b0a60e4
2022-05-25 13:24:03 -05:00
Mya
9153f37055 Makefile: run lint locally in docker
Our linting process depends heavily on custom tools and linting
configuration. To help improve this process for running on local
developer machines, we can run the various tasks in our existing CI
container.

Change-Id: I60407686ce9233cc4f16e3724c5e8d44367aa200
2022-05-25 12:30:15 -05:00
Cameron
87f6a3dcda {web/satelliite, satellite/console}: logo redirects to homepage
logo redirects to homepage on login, signup, forgot password, reset
password, and activate account pages

Change-Id: I992aeae197004d620addd8d515cae1c1ca80a778
2022-05-23 16:26:19 -04:00
NickolaiYurchenko
99237d5c78 web/satellite: new bucket creation flow
old bucket creation flow removed
new flow added
name and passphrase splitted into separate views
demo bucket will not be created automatically
bucket creation progress bar added

Change-Id: I2a1d7d77c3038caaafb3c06bdb0ac5dd1ad17599
2022-05-22 11:55:55 +00:00
Michał Niewrzał
f9a3f19443 scripts: add 'uplink cp' tests for stdin/stdout
Uplink can upload from stdin and download to stdout. We had
such tests for old binary but now we were missing it.

Change-Id: I5110a9f531f5cc21277fa53611995fb5b556ff16
2022-05-18 08:42:04 +00:00
Malcolm Bouzi
087e57d037 web/satellite: create chore that will resend verification emails to unverified users
We want to remind unverified users to verify their emails:
once after 24 hours has passed and again after 5 days has passed.

Add mailservice.Service to satellite core because it is needed by the
chore for sending emails. To add the mailservice.Service to the core,
we create a helper function in satellite/peer.go to avoid duplicating
the code in both api.go and core.go. In addition to the chore, this
change adds methods to users.DB to get unverified users in need of
reminder.

Change-Id: I4e515bdf43f922788b4f965b2efb34fa32288bd1
2022-05-18 08:08:33 +00:00
JT Olio
5fb9ee3cfa uplink, satellite: use bgp hostnames
Change-Id: I58f5011d3019f8267fa8cbd3096b2cfe42eb5f8b
2022-05-16 18:14:37 +00:00
Cameron
0633aca607 satellite/console: create new consoleauth service
We want to send email verification reminders to users from the satellite
core, but some of the functionality required to do so exists in the
satellite console service. We could simply import the console service
into the core to achieve this, but the service requires a lot of
dependencies that would go unused just to be able to send these emails.

Instead, we break out the needed functionality into a new service which
can be imported separately by the console service and the future email
chore.

The consoleauth service creates, signs, and checks the expiration of auth
tokens.

Change-Id: I2ad794b7fd256f8af24c1a8d73a203d508069078
2022-05-13 16:27:07 +00:00
Moby von Briesen
763bfc0913 satellite/console,web/satellite: Implement hCaptcha
Adds a new configuration for hcaptcha enabled, secretkey, and sitekey.
If both reCAPTCHA and hCaptcha are configured as "enabled", reCAPTCHA
will be used.

Change-Id: I73cc6e133d8da3555e0ed8b2b377cf9eb263e6dc
2022-05-13 14:57:45 +00:00
Erik van Velzen
db1cc8ca95 satellite/repair/checker: buffer repair queue
Integrate previous changes. Speed up the segment loop by batch inserting
into repair queue.

Change-Id: Ib9f4962d91960d21bad298f7771345b0dd270276
2022-05-12 16:28:05 +00:00
Vitalii
dedccbd2e4 satellite/console, web/satellite: limit failed login attempts
Added account locking on 3 or more login attempts.
Includes both password and MFA failed attempts on login.
Unlock account on successful password reset.

Change-Id: If4899b40ab4a77d531c1f18bfe22cee2cffa72e0
2022-05-11 14:49:11 +00:00
Qweder93
384118ee1e satellite/metabase: use pgutil.NullTimestampTZArray
This code cleanup to use helper DB method.

Change-Id: Idc438234cb4f7da23d6ade2d69f5d889f15e9009
2022-05-03 19:30:21 +00:00
hovex023
58f957bd1d
web/satellite: Add new access grants flow (#4765)
* Added new feature Flag for new Access Grant Flow. 

* Added 3 cards to access grant view for S3, CLI and Access grant to replace old header

* Added new formatting, text and Icon for Access Grant Delete Popup modal
2022-04-29 10:31:52 -05:00
nadimhq
dd6aac9b39
satellite/metainfo: enable server-side copy in production (#4708)
these changes enable server-side copy in production
2022-04-26 09:58:16 -04:00
Moby von Briesen
ed5ebb2527 satellite: Rename "acct mgmt api" to "rest api"
"REST API" is a more accurate descriptor of the generated API in the
console package than "account management API". The generated API is very
flexible and will allow us to implement many more endpoints outside the
scope of "account management", and "account management" is not very well
defined to begin with.

Change-Id: Ie87faeaa3c743ef4371eaf0edd2826303d592da7
2022-04-25 18:51:46 +00:00
Michał Niewrzał
6188c4c0dd satellite/metainfo: enable segment limit by default
Change-Id: I26cc0bbd97d6cc2ffc96213788f261efb749d028
2022-04-14 06:12:09 +00:00
Qweder93
2a7b20e8e4 cmd/uplink: integrate server-side copy with uplink cp command
Resolves https://github.com/storj/storj/issues/4486

Change-Id: I42ac2ad2e1a05df4a83606f1990b639f08791403
2022-03-31 09:25:29 +00:00
Egon Elbre
4a6c2e2a9c satellite/admin/ui: avoid needing an additional copy
This also fixes the build order. Unfortunately we need
to ensure that the web frontends are built before installing
Go binaries.

Fixes https://github.com/storj/storj/issues/4654

Change-Id: I5d1c83125fd3d1a454d3400b2cbdd44bd3f2250c
2022-03-25 11:42:22 +02:00
Cameron
84b522bc06 satellite/console: create account management api keys service
We are in the process of creating an api to allow users to manage their
accounts programmatically. We would like to use api keys for
authorization. We were originally going to create an entirely new table
for these api keys, but seeing as we already have 2 other tables for
keys/tokens, api_keys and oauth_tokens, we thought it might be better to
use one of these. We're using oauth_tokens.

We create a new oidc.OAuthTokenKind for account management api keys:
KindAccountManagementTokenV0. We made the key versioned because we
likely want to improve the implementation in the future, but we want to
get something functional out the door ASAP because the account management
api feature is highly desired.

Add a new method to oidc.OAuthTokens interface for revoking v0 account
management api keys, RevokeAccountManagementTokenV0. Add update method
to dbx implementation to allow updating the expiration. We will revoke
these keys by setting the expiration to 0 so they are expired.

Change-Id: Ideb8ae04b23aa55d5825b064b5e43e32eadc1fba
2022-03-23 17:02:20 +00:00
Egon Elbre
0d2d59f884 all: fix linting issues
Change-Id: Idfc93948e59a181321d79b365e638d63e256a16f
2022-03-21 15:26:42 +00:00
Stefan Benten
2a10b2fd14
scripts/draft-release: add sleep to avoid race condition on uploading the binaries (#4647)
It seems that the github API is a little slow/laggy with regards to propagation of whether a release tag has been made or not.
This sleep should fix it and avoid having to retrigger builds.
2022-03-21 15:24:20 +01:00
littleskunk
6fbc56bd06
satellite/payment: enable segment fee (#4628) 2022-03-18 14:25:14 +01:00
Mya
4a110b266e satellite/console: added oidc endpoints
This change adds endpoints for supporting OpenID Connect (OIDC) and
OAuth requests. This allows application developers to easily
develop apps with Storj using common mechanisms for authentication
and authorization.

Change-Id: I2a76d48bd1241367aa2d1e3309f6f65d6d6ea4dc
2022-03-16 12:01:26 +00:00
dlamarmorgan
715077d04a satellite/{console,satellitedb}: increase project limit on paid tier upgrade
Update the user to the default paid tier project limit, which is currently 3 projects, when the user upgrades to a paid account.

Change-Id: I95b19d62cebc7d878b716355f2ebcaf0b51ca3f7
2022-03-14 16:10:23 +00:00
Michał Niewrzał
39ac90835f satellite/satellitedb: change default segment limit to 100M for paying users
We decided that we want to have segment limit for paying users high
enough to not have to change it too often.

Fixes https://github.com/storj/storj/issues/4590

Change-Id: Ic1c38bf3e2fcc000548ff4c7e7004647b39fbecf
2022-03-04 13:18:14 +00:00
Moby von Briesen
b2d342aa9b satellite/overlay: Add ability to exclude country codes on upload
Create global config to specify a list of country codes that should be
excluded from node selection during uploads.

This exclusion is not implemented when the upload selection cache is
disabled.

Change-Id: Ic41e8b4f18857a11045668eac23107da99668a72
2022-03-03 16:58:48 +00:00
NickolaiYurchenko
64176aaca4 web/satellite: registration success redirect from config
This change allows us to send newly registered users to a configured URL
to help us track user conversions for marketing campaigns.
Brave conversions continue to be tracked using the /signup-success page
within the satellite app.

Change-Id: I9b451947ce0f39d3c99b233cb4b806d361151823
2022-03-03 10:13:21 -05:00
Fadila Khadar
e776c65172 satellite/checker: pieces in excluded countries are not healthy
Add a RepairExcludedCountryCodes config flag for overlay for providing a list of country codes to exclude nodes from target repair selection.

Mark segments with less than repairThreshold pieces in countries not in the RepairExcludedCountryCodes as not healthy.
With this change, the repair process is not affected. The segment will be removed from the repair queue by the repairer.

Another change will handle the logic at the repairer level.

Fixes https://github.com/storj/team-metainfo/issues/95

Change-Id: I9231b32de117a116488de055a3e94efcabb46e81
2022-03-02 09:59:09 +00:00
Vitalii Shpital
ba6956db0f console/server, apigen: feature flag for new generated console api
Added a feture flag which will be used to indicate if new generated console api is used.
Fixed some comments from previous PR.

Change-Id: Ice31c998b0b347028a491c971a648fd1269bfd49
2022-02-28 23:00:12 +00:00
Michał Niewrzał
fbe2680500 satellite/metainfo: add feature flag for server side copy code
We would like to disable in production those parts of code
which are now mixed with new server-side copy logic.

Change-Id: Iff50682bc9545207330f58dd19b5eee53d404d7f
2022-02-24 10:43:49 +00:00
Jeff Wendling
ccb847a329 scripts/tests: write old uplink config yaml directly
sometimes these scripts want to have an access imported
after it has been potentially modified by having the
satellite address and node id added. it used to use the
uplink command to do this, but the cli api for that
has changed. rather than try to have the script detect
which uplink version is in use and call the right thing
it can always write out a valid yaml file and depend on
the new cli migrating it.

Change-Id: Ib82819699333f5f29e00117b99bfb10640033b94
2022-02-22 15:26:06 -05:00
Jeff Wendling
863b01bf09 testversions: skip some downloads for older uplinks
uplink command versions >= 1.48.0 always do multipart
uploads which cannot be downloaded by any of the
versions in the test < v1.27.6. so skip those tests.

Change-Id: I9644afbd14bfce9facfd87644d132f7d66367d62
2022-02-22 17:24:07 +00:00
Stefan Benten
ec2bd50bb4
scripts/deploy-storagenode.sh: adding multinode image to deployment
In order to tag the latest release for the multinode image, it makes
the most sense to do it at the same time as we release the storagenode
image.

Change-Id: I2d63c1f93858354ad1f9a4fce0ce45a8fda2716f
2022-02-21 14:37:04 +01:00
Jeff Wendling
0a860a2e89 scripts: fix builds for new uplink command
the new uplink command expects to be able to migrate
and so we need to specify the --legacy-config-dir
flag sometimes as well. but unfortunately, the old
uplink will error if it gets a flag it doesn't
understand, so we have to set it as an env var.
but we can't use the env var for the --config-dir
flag all the time because the old uplink doesn't
look for that env var.

Change-Id: I019315192c0e6c348814527794342d823a5f9ec3
2022-02-16 21:40:06 +00:00
NickolaiYurchenko
70c8ccb435 web/satellite: inactivity timer to dashboard added
added InactivityTimerEnabled flag to enable/disable feature
added InactivityTimerDelay to configure delay time in seconds
default timer set up to 10 minutes
reset dom events: keypress, mouseover, mousedown, touchmove

Change-Id: Idb66067c2902b2cdbe1a972225319c8abff97927
2022-02-16 07:16:00 +00:00
Jeff Wendling
9061dd309f cmd/uplinkng: become cmd/uplink
Change-Id: If426c32219d32044d715ab6dfa9718807f32cb9f
2022-02-09 17:02:21 +00:00
Vitalii Shpital
b3e1be37ff satellite/projectaccounting: query to get daily project usage by date range
Finished implementing queries for both bandwidth and storage using pgx.Batch.
Fixed CSP styling issue.

Change-Id: I5f9e10abe8096be3115b4e1f6ed3b13f1e7232df
2022-02-04 22:32:37 +00:00