Commit Graph

140 Commits

Author SHA1 Message Date
Sembeth
0408997e6c
cmd/multinode: Removes dependency on deprecated identity-dir flag, code and documentation. (#5646)
* The Multinode Dashboard no longer requires the identity files.

* Removed deprecated code that used the identity-dir.
2023-03-08 13:56:15 +01:00
Jeremy Wharton
3fa31c2077 satellite/console/consoleweb: remove trailing slash from URLs
This change removes the trailing slash from the account activation and
password recovery URLs, making them consistent with the rest. The URLs'
previous forms are still supported, however, in order to not invalidate
emails containing them.

Resolves storj/customer-issues#491

Change-Id: Ie774a87698d8e9edd1836611968fc3911c6cc56f
2023-02-21 19:15:36 +00:00
Qweder93
8c69ee62fc {cmd/storj-sim, satellite/rangedloop}: added rangedloop to storj-sim, removed identity
added in storj-sim rangedloop for each satellite, to verify it works for metrics oveserver,
removed identity from rangedloop peer as we never use it, added logs on service run, added loop
to service instead of endless for loop, interval value to config

Closes: https://github.com/storj/storj/issues/5414

Change-Id: Ibc3b06071b68feda4a35b45da2bbe36e22a02fc8
2023-01-05 11:29:00 +00:00
Egon Elbre
ff22fc7ddd all: fix deprecated ioutil commands
Change-Id: I59db35116ec7215a1b8e2ae7dbd319fa099adfac
2022-10-11 15:27:29 +00:00
Jeremy Wharton
3f26cc599f satellite/console,web/satellite: invalidate sessions after inactivity
Sessions now expire after a much shorter amount of time, requiring
clients to issue API requests for session extension. This is handled
behind the scenes as the user interacts with the page, but once session
expiration is imminent, a modal appears which informs the user of his
inactivity and presents him with the choice of loging out or preserving
his session.

Change-Id: I68008d45859c814a835d65d882ad5ad2199d618e
2022-08-23 15:51:05 +00:00
Egon Elbre
1cb3cbaecf cmd/storj-sim: ignore context canceled and flush output
It's possible that content was not being flushed from processes.

For now, ignore other process failures under storj-sim network test.
Once we get other processes stable, we can repropagate the error.

Change-Id: I01ed572d7c779ab6451124f1e24e3d1168b3ea79
2022-08-02 20:48:46 +03:00
Ivan Fraixedes
aa913e63db cmd/storj-sim: Fix path to admin UI assets folder
The admin UI assets aren't inside of the `web` directory they are
directly in the `satellite` one.

The invalid path provoked that storj-sim generated a satellite
configuration with an invalid path to the Admin UI static assets
provoking that it didn't load the UI by default.

Change-Id: I49fb289377f51634057173690fbd8cf863ca9a9d
2022-05-31 18:54:54 +00:00
Paul Willoughby
036d5a8619 cmd/storj-sim: fix printing node url
Fix printing satellite url to help configure authservice.
  storj-sim network env SATELLITE_0_URL

Change-Id: I786ab6723f837bbc35f1228eb73a82427d2f9250
2022-05-25 00:09:17 -06:00
Egon Elbre
4791ba5d50 satellite/{mailservice,oidc}: improvements to debugging
Things that make debugging easier.
* Added logging to automatic link clicking to make it obvious, when it
  fails.
* Added monitoring to oidc.
* Made dbx create calls noreturn for oauth_*

Change-Id: I37397b4e84ce5bfd82954aed9c38fdfd52595f24
2022-05-11 19:59:42 +00:00
Egon Elbre
64c8de6ea5 mod: use vendored base58
Change-Id: I5aa29515928848c862500330218cc094618638d7
2022-01-31 15:54:33 +02:00
Egon Elbre
6a69c66477 cmd/storj-sim: support printing node url
Add support for node url-s with PEER_URL to help configuring
authservice and gateway.

   storj-sim network env SATELLITE_0_URL

Change-Id: I8b595b398007730662f100a4e6b3529cc0a7512a
2021-11-03 16:11:41 +02:00
Jeremy Wharton
a5f6bb9cc0 satellite/console: Add reCAPTCHA verification step to registration
The user must complete a reCAPTCHA in order to register.
ReCAPTCHA verification failure results in rejection of the
registration attempt.

Change-Id: I34ba7db414d756fd1aaebdc3d19cccbfc7fc1ea3
2021-07-07 21:34:07 +00:00
Egon Elbre
10372afbe4 ci: fix lint errors
Change-Id: Ib5893440807811f77175ccd347aa3f8ca9cccbdf
2021-05-17 13:37:31 +00:00
Yaroslav Vorobiov
c08ca361d8 cmd/storj-sim: add multinode process
Change-Id: I7e6c8db045daa87c5d89f49e58fc82453d4024c6
2021-05-14 01:40:13 +03:00
Egon Elbre
a2e20c93ae private/dbutil: use dbutil and tagsql from storj.io/private
Initially we duplicated the code to avoid large scale changes to
the packages. Now we are past metainfo refactor we can remove the
duplication.

Change-Id: I9d0b2756cc6e2a2f4d576afa408a15273a7e1cef
2021-04-23 14:36:52 +03:00
Yingrong Zhao
3b49d3cddf satellite: remove referral program related code
This PR removes all back-end related referral program code including the
marketing portal.

We will have a separate PR for front-end code and database migration to
drop `offers` and `usercredits` table

Change-Id: If59f952cddfe0558a7dc03a0eac7cc1081517f88
2021-02-08 13:52:50 +00:00
Egon Elbre
c44368c489 cmd/storj-sim: fix port assignment for multiple satellites
Currently first satellite GC would've conflicted with second satellites
public RPC port. Instead assign "satellite workers" a new peer class.

Change-Id: Id6bdaa17243556482e88da708c5147149788f6be
2021-01-22 14:08:39 +02:00
Egon Elbre
590bc529d2 cmd/storj-sim: fix local binary lookup
Change-Id: I6404fd05119c2a7113e4b0d7593a555327bef35f
2021-01-06 16:26:22 +02:00
Egon Elbre
b33d7a318e cmd/storj-sim: add -failfast
Fail all the processes immediately when one of the processes fails. This
is to make it more obvious that one of them has failed.

To disable failfast, use `-failfast=false`.

Change-Id: I2bbedf12fb653e42739d00273aa9ae515d34eda6
2021-01-06 16:26:22 +02:00
Egon Elbre
9cb4466eb0 cmd/storj-sim: use dev setup by default for consistency
Fixes bug when using release binaries together with storj-sim.

Change-Id: I077bedc1486ac85aa1f04fcc0ed4098cd313f2fc
2021-01-05 13:47:30 +02:00
Stefan Benten
7f1871b8f1 all: switch from master to main 2020-12-28 22:59:06 +01:00
Jessica Grebenschikov
d961437889 satellite/orders: remove the config IncludeEncryptedMetadata
Since the Satellite now requires the order encryption functionality (since serial_number table is deprecated) to properly function, we can remove the config flag to turn on/off the feature.

Change-Id: Ie973f72a9a05a81cef9e53dc9c99d22c940c2488
2020-12-18 10:39:29 -08:00
Michal Niewrzal
b1712cc93b cmd/storj-sim: update default storj-sim access with real node id
Currently node id in access grant is '1' and it cannot be parsed to
valid node id. This change update access grant satellite address with
randomly generated node id.

Change-Id: Id1684ac71509bc5a8177b069a914355be3c72d43
2020-12-17 18:31:43 +00:00
Jeff Wendling
0e83233700 storj-sim: add node id to default access
Change-Id: I59874fe8d73a832d04a5597c98d05971a74d2164
2020-12-17 09:38:23 +00:00
Egon Elbre
5b6fcfb725 cmd/storj-sim: fix 32bit code
Change-Id: Ieec74f2a8ed1a207e7deff0f9072c1243cb1e3be
2020-12-09 09:49:33 +02:00
Jessica Grebenschikov
b261110352 satellite/orders: get bucketID from encrypted metadata in order instead of serial_numbers table
We want to stop using the serial_numbers table in satelliteDB. One of the last places using the serial_numbers table is when storagenodes settle orders, we look up the bucket name and project ID from the serial number from the serial_numbers table.

Now that we have support to add encrypted metadata into the OrderLimit, this PR makes use of that and now attempts to read the project ID and bucket name from the encrypted orderLimit metadata instead of from the serial_numbers table. For backwards compatibility and to ensure no errors, we will still fallback to the old way of getting that info from the serial_numbers table, but this will be removed in the next release as long as there are no errors.

All processes that create orderLimits must have an orders.encryption-keys set. The services that create orderLimits (and thus need to encrypt the order metadata) are the satellite apiProcess, the repair process, audit service (core process), and graceful exit (core process). Only the satellite api process decrypts the order metadata when storagenodes settle orders. This means that the same encryption key needs to be provided in the config for the satellite api process, repair process, and the core process like so:
orders.include-encrypted-metadata=true
orders.encryption-keys="<"encryptionKeyID>=<encryptionKey>"

Change-Id: Ie2c037971713d6fbf69d697bfad7f8b672eedd66
2020-12-01 15:29:32 +00:00
Egon Elbre
afc9545ff1 cmd/storj-sim: add "tool wait-for <address>"
For coordinating with other processes it can be useful to wait until
another process is accepting requests on an address.

Change-Id: Id623ed815149f14f9f0344e2f396ab70fc4dec6a
2020-11-16 20:38:56 +00:00
Yingrong Zhao
8fd841b910 scripts/tests/testversions: fix installation during setup
This PR fixes below issues:
1. remove concurrent installation for various versions
    We were doing this to decrease the amount of execution time the versions test.
    However, it's returning incorrect exit code when there's an
    installation failure.
    Right now, we are only installing two versions of `storj-sim` and
    the rest are only doing uplink cli installation. The performance of
    this test should be hugely impacted by the setup step now.
2. only remove release settings instead of deleting the entire file
    uplink CLI is referrencing `private/version` package. Therefore, we
    cannot delete it
3. add back `GATEWAY_0_API_KEY` in storj-sim
    In order to set up older version of uplink cli, we need access to
    the gate way api key.

Change-Id: Ia3c37c197bd007b6e1f7c2bd71adde42181d46f0
2020-11-10 20:38:49 +00:00
Jessica Grebenschikov
99c88efbbf scripts/tests: fix gateway tests
Change-Id: I9a23ef08794043ad615066ae5929df9ff3a02d69
2020-10-27 08:21:28 -07:00
Kaloyan Raev
830817ec0d cmd/storj-sim: run gateway without --access flag
This makes it possible to remove of this obsolete flag from the
multi-tenant gateway.

As a consequence, displaying the GATEWAY_0_ACCESS env var will always
require a running storj-sim. Until now, it was required only the first
time. Then the value was stored in the 'access' config. But this is now
not possible anymore.

The changes in StripeMock are required to fix failures in integration
tests. StripeMock is in-memory and its data does not survive restarts of
storj-sim. The second and following starts of storj-sim had invalid
state of StripeMock, which failed requests that were required to
populate the GATEWAY_0_ACCESS env var. The changes in StripeMock makes
it repopulate the Stripe customers from the database.

Change-Id: I981a208172b76577f12ecdaae485f5ae4ea269bc
2020-10-13 14:45:04 +00:00
Egon Elbre
2268cc1df3 all: fix linter complaints
Change-Id: Ia01404dbb6bdd19a146fa10ff7302e08f87a8c95
2020-10-13 15:59:01 +03:00
Kaloyan Raev
bd177bff03 cmd/storj-sim: cleanup gateway setup
Remove usage of --non-interactive flag. It is not provided (and
necessary) by the multitenant S3 gateway anymore.

ACCESS_KEY and SECRET_KEY env vars are not provided anymore as they are
not generated by the multitenant S3 gateway.

Change-Id: I3ecfb92110e31ae13977de3899dad273daae6c1e
2020-10-06 14:22:47 +00:00
paul cannon
5b9fd4e50a cmd/storj-sim: fix prefix writer infinite loop bug
Fix case where line is larger than maxline and writer ended up in an
infinite loop trying to buffer more data.

Change-Id: I243da738b331279d6bf27255778b5798e7f37f95
2020-09-11 12:33:04 +00:00
stefanbenten
086a3d5348 satellite/{payments,admin}: add deletion of user creditcards on account deletion
Change-Id: I38bf7e3995846150268f7b88a70f75b0ac871b62
2020-08-27 10:18:19 +00:00
Egon Elbre
94a09ce20b all: add missing dots
Change-Id: I93b86c9fb3398c5d3c9121b8859dad1c615fa23a
2020-08-11 17:50:01 +03:00
Egon Elbre
080ba47a06 all: fix dots
Change-Id: I6a419c62700c568254ff67ae5b73efed2fc98aa2
2020-07-16 14:58:28 +00:00
Yingrong Zhao
175f048aa7 scripts;cmd/storj-sim: include satellite node id in satellite address
Expose satellite ID from storj-sim so we can have access to it when
changing satellite address.

Change-Id: Ife816f4d35eae2d0bc9f9ad592fe75d73d93d9ff
2020-06-05 09:44:19 -04:00
Michal Niewrzal
48e67a29a9 cmd/storj-sim: remove old libuplink from storj-sim
Change-Id: I146a2d7947d1f358e7bdd8acba68eb3fee68539d
2020-05-29 08:39:37 +00:00
Michal Niewrzal
ac375d37bc satellite/payments: remove mockpayments and add Stripe client mock
instead

Change-Id: If3496f6abc16da90d2b43fa0c5be356847a39507
2020-05-19 09:35:37 +02:00
Egon Elbre
ec589a8289 all: fix comments about grpc
Change-Id: Id830fbe2d44f083c88765561b6c07c5689afe5bd
2020-05-11 13:05:34 +03:00
Kaloyan Raev
b5433322b4 cmd/storj-sim: enable open user registration
Change-Id: I695664279f42209245a82cc5bf103d037b844c32
2020-05-05 13:03:02 +00:00
Yingrong Zhao
c5309a3f91 cmd/uplink: set sample rate for tracing to be 1 when tracing is enabled
when tracing is enabled, we should also set sampling rate to
a non-zero value. For now, we will set it to 1.
Uplink CLI users should be able to override it with the sample
flag.

Change-Id: I8bcf514fb14c2a1c4349b7957dd24ec23e4a85e5
2020-04-28 20:15:28 +00:00
Egon Elbre
d3ce845f82 satellite: revert log lines used to figure out node id
Currently storj-sim relies on the log lines to be exactly the same,
when they change it cannot find the necessary information from log.

Change-Id: Ia039915ef3375a7cf60f107b2c05c958de15b6d5
2020-04-15 17:07:56 +03:00
Kaloyan Raev
a2ce836761 remove sugar logging
Change-Id: I6b6ca9704837cb3f5f5449ba7f55661487814d9f
2020-04-15 12:37:47 +00:00
Egon Elbre
1024bf9ce1 all: simplify uuid usage
Instead of uuid.Parse, use uuid.FromString.
This removes a bunch of pointer management logic.

Change-Id: Id25bd174eb43c71d00b450158a198abafd8958f2
2020-04-02 13:45:19 +00:00
Egon Elbre
0a69da4ff1 all: switch to storj.io/common/uuid
Change-Id: I178a0a8dac691e57bce317b91411292fb3c40c9f
2020-03-31 19:16:41 +03:00
Michal Niewrzal
e258de518c cmd/storj-sim: remove identity dir from gateway configuration
We are not generating identity for gateway during setup anymore, its
generated on-fly by libuplink. With this change we can remove
`--identity-dir` from gateway.

Change-Id: I63298115d4399564b2f29541b8dc16e3b3acdcf2
2020-03-23 14:11:51 +00:00
Michal Niewrzal
d994584e88 cmd/storj-sim: remove removed flags from storj-sim gateway setup
Change-Id: I27d585740e8441b41bce5010480cc8e4f5deb9ce
2020-03-20 12:09:55 +01:00
Jessica Grebenschikov
5142874144 satellite/gc: move garbage collection to its own process
Change-Id: I7235aa83f7c641e31c62ba9d42192b2232dca4a5
2020-03-18 16:44:01 +00:00
Jennifer Johnson
1c1750e6be removes bandwidth limiting
On satellite, remove all references to free_bandwidth column in nodes table.
On storage node, remove references to AllocatedBandwidth and MinimumBandwidth and mark as deprecated.

Protobuf message, NodeCapacity, is left intact for backwards compatibility.
Once this is released to all satellites, we can drop the column from the DB.

Change-Id: I2ff6c6537fc9008a0c5588e951afea58ede85838
2020-03-04 14:04:00 +00:00