Commit Graph

134 Commits

Author SHA1 Message Date
Ivan Fraixedes
aa913e63db cmd/storj-sim: Fix path to admin UI assets folder
The admin UI assets aren't inside of the `web` directory they are
directly in the `satellite` one.

The invalid path provoked that storj-sim generated a satellite
configuration with an invalid path to the Admin UI static assets
provoking that it didn't load the UI by default.

Change-Id: I49fb289377f51634057173690fbd8cf863ca9a9d
2022-05-31 18:54:54 +00:00
Paul Willoughby
036d5a8619 cmd/storj-sim: fix printing node url
Fix printing satellite url to help configure authservice.
  storj-sim network env SATELLITE_0_URL

Change-Id: I786ab6723f837bbc35f1228eb73a82427d2f9250
2022-05-25 00:09:17 -06:00
Egon Elbre
4791ba5d50 satellite/{mailservice,oidc}: improvements to debugging
Things that make debugging easier.
* Added logging to automatic link clicking to make it obvious, when it
  fails.
* Added monitoring to oidc.
* Made dbx create calls noreturn for oauth_*

Change-Id: I37397b4e84ce5bfd82954aed9c38fdfd52595f24
2022-05-11 19:59:42 +00:00
Egon Elbre
64c8de6ea5 mod: use vendored base58
Change-Id: I5aa29515928848c862500330218cc094618638d7
2022-01-31 15:54:33 +02:00
Egon Elbre
6a69c66477 cmd/storj-sim: support printing node url
Add support for node url-s with PEER_URL to help configuring
authservice and gateway.

   storj-sim network env SATELLITE_0_URL

Change-Id: I8b595b398007730662f100a4e6b3529cc0a7512a
2021-11-03 16:11:41 +02:00
Jeremy Wharton
a5f6bb9cc0 satellite/console: Add reCAPTCHA verification step to registration
The user must complete a reCAPTCHA in order to register.
ReCAPTCHA verification failure results in rejection of the
registration attempt.

Change-Id: I34ba7db414d756fd1aaebdc3d19cccbfc7fc1ea3
2021-07-07 21:34:07 +00:00
Egon Elbre
10372afbe4 ci: fix lint errors
Change-Id: Ib5893440807811f77175ccd347aa3f8ca9cccbdf
2021-05-17 13:37:31 +00:00
Yaroslav Vorobiov
c08ca361d8 cmd/storj-sim: add multinode process
Change-Id: I7e6c8db045daa87c5d89f49e58fc82453d4024c6
2021-05-14 01:40:13 +03:00
Egon Elbre
a2e20c93ae private/dbutil: use dbutil and tagsql from storj.io/private
Initially we duplicated the code to avoid large scale changes to
the packages. Now we are past metainfo refactor we can remove the
duplication.

Change-Id: I9d0b2756cc6e2a2f4d576afa408a15273a7e1cef
2021-04-23 14:36:52 +03:00
Yingrong Zhao
3b49d3cddf satellite: remove referral program related code
This PR removes all back-end related referral program code including the
marketing portal.

We will have a separate PR for front-end code and database migration to
drop `offers` and `usercredits` table

Change-Id: If59f952cddfe0558a7dc03a0eac7cc1081517f88
2021-02-08 13:52:50 +00:00
Egon Elbre
c44368c489 cmd/storj-sim: fix port assignment for multiple satellites
Currently first satellite GC would've conflicted with second satellites
public RPC port. Instead assign "satellite workers" a new peer class.

Change-Id: Id6bdaa17243556482e88da708c5147149788f6be
2021-01-22 14:08:39 +02:00
Egon Elbre
590bc529d2 cmd/storj-sim: fix local binary lookup
Change-Id: I6404fd05119c2a7113e4b0d7593a555327bef35f
2021-01-06 16:26:22 +02:00
Egon Elbre
b33d7a318e cmd/storj-sim: add -failfast
Fail all the processes immediately when one of the processes fails. This
is to make it more obvious that one of them has failed.

To disable failfast, use `-failfast=false`.

Change-Id: I2bbedf12fb653e42739d00273aa9ae515d34eda6
2021-01-06 16:26:22 +02:00
Egon Elbre
9cb4466eb0 cmd/storj-sim: use dev setup by default for consistency
Fixes bug when using release binaries together with storj-sim.

Change-Id: I077bedc1486ac85aa1f04fcc0ed4098cd313f2fc
2021-01-05 13:47:30 +02:00
Stefan Benten
7f1871b8f1 all: switch from master to main 2020-12-28 22:59:06 +01:00
Jessica Grebenschikov
d961437889 satellite/orders: remove the config IncludeEncryptedMetadata
Since the Satellite now requires the order encryption functionality (since serial_number table is deprecated) to properly function, we can remove the config flag to turn on/off the feature.

Change-Id: Ie973f72a9a05a81cef9e53dc9c99d22c940c2488
2020-12-18 10:39:29 -08:00
Michal Niewrzal
b1712cc93b cmd/storj-sim: update default storj-sim access with real node id
Currently node id in access grant is '1' and it cannot be parsed to
valid node id. This change update access grant satellite address with
randomly generated node id.

Change-Id: Id1684ac71509bc5a8177b069a914355be3c72d43
2020-12-17 18:31:43 +00:00
Jeff Wendling
0e83233700 storj-sim: add node id to default access
Change-Id: I59874fe8d73a832d04a5597c98d05971a74d2164
2020-12-17 09:38:23 +00:00
Egon Elbre
5b6fcfb725 cmd/storj-sim: fix 32bit code
Change-Id: Ieec74f2a8ed1a207e7deff0f9072c1243cb1e3be
2020-12-09 09:49:33 +02:00
Jessica Grebenschikov
b261110352 satellite/orders: get bucketID from encrypted metadata in order instead of serial_numbers table
We want to stop using the serial_numbers table in satelliteDB. One of the last places using the serial_numbers table is when storagenodes settle orders, we look up the bucket name and project ID from the serial number from the serial_numbers table.

Now that we have support to add encrypted metadata into the OrderLimit, this PR makes use of that and now attempts to read the project ID and bucket name from the encrypted orderLimit metadata instead of from the serial_numbers table. For backwards compatibility and to ensure no errors, we will still fallback to the old way of getting that info from the serial_numbers table, but this will be removed in the next release as long as there are no errors.

All processes that create orderLimits must have an orders.encryption-keys set. The services that create orderLimits (and thus need to encrypt the order metadata) are the satellite apiProcess, the repair process, audit service (core process), and graceful exit (core process). Only the satellite api process decrypts the order metadata when storagenodes settle orders. This means that the same encryption key needs to be provided in the config for the satellite api process, repair process, and the core process like so:
orders.include-encrypted-metadata=true
orders.encryption-keys="<"encryptionKeyID>=<encryptionKey>"

Change-Id: Ie2c037971713d6fbf69d697bfad7f8b672eedd66
2020-12-01 15:29:32 +00:00
Egon Elbre
afc9545ff1 cmd/storj-sim: add "tool wait-for <address>"
For coordinating with other processes it can be useful to wait until
another process is accepting requests on an address.

Change-Id: Id623ed815149f14f9f0344e2f396ab70fc4dec6a
2020-11-16 20:38:56 +00:00
Yingrong Zhao
8fd841b910 scripts/tests/testversions: fix installation during setup
This PR fixes below issues:
1. remove concurrent installation for various versions
    We were doing this to decrease the amount of execution time the versions test.
    However, it's returning incorrect exit code when there's an
    installation failure.
    Right now, we are only installing two versions of `storj-sim` and
    the rest are only doing uplink cli installation. The performance of
    this test should be hugely impacted by the setup step now.
2. only remove release settings instead of deleting the entire file
    uplink CLI is referrencing `private/version` package. Therefore, we
    cannot delete it
3. add back `GATEWAY_0_API_KEY` in storj-sim
    In order to set up older version of uplink cli, we need access to
    the gate way api key.

Change-Id: Ia3c37c197bd007b6e1f7c2bd71adde42181d46f0
2020-11-10 20:38:49 +00:00
Jessica Grebenschikov
99c88efbbf scripts/tests: fix gateway tests
Change-Id: I9a23ef08794043ad615066ae5929df9ff3a02d69
2020-10-27 08:21:28 -07:00
Kaloyan Raev
830817ec0d cmd/storj-sim: run gateway without --access flag
This makes it possible to remove of this obsolete flag from the
multi-tenant gateway.

As a consequence, displaying the GATEWAY_0_ACCESS env var will always
require a running storj-sim. Until now, it was required only the first
time. Then the value was stored in the 'access' config. But this is now
not possible anymore.

The changes in StripeMock are required to fix failures in integration
tests. StripeMock is in-memory and its data does not survive restarts of
storj-sim. The second and following starts of storj-sim had invalid
state of StripeMock, which failed requests that were required to
populate the GATEWAY_0_ACCESS env var. The changes in StripeMock makes
it repopulate the Stripe customers from the database.

Change-Id: I981a208172b76577f12ecdaae485f5ae4ea269bc
2020-10-13 14:45:04 +00:00
Egon Elbre
2268cc1df3 all: fix linter complaints
Change-Id: Ia01404dbb6bdd19a146fa10ff7302e08f87a8c95
2020-10-13 15:59:01 +03:00
Kaloyan Raev
bd177bff03 cmd/storj-sim: cleanup gateway setup
Remove usage of --non-interactive flag. It is not provided (and
necessary) by the multitenant S3 gateway anymore.

ACCESS_KEY and SECRET_KEY env vars are not provided anymore as they are
not generated by the multitenant S3 gateway.

Change-Id: I3ecfb92110e31ae13977de3899dad273daae6c1e
2020-10-06 14:22:47 +00:00
paul cannon
5b9fd4e50a cmd/storj-sim: fix prefix writer infinite loop bug
Fix case where line is larger than maxline and writer ended up in an
infinite loop trying to buffer more data.

Change-Id: I243da738b331279d6bf27255778b5798e7f37f95
2020-09-11 12:33:04 +00:00
stefanbenten
086a3d5348 satellite/{payments,admin}: add deletion of user creditcards on account deletion
Change-Id: I38bf7e3995846150268f7b88a70f75b0ac871b62
2020-08-27 10:18:19 +00:00
Egon Elbre
94a09ce20b all: add missing dots
Change-Id: I93b86c9fb3398c5d3c9121b8859dad1c615fa23a
2020-08-11 17:50:01 +03:00
Egon Elbre
080ba47a06 all: fix dots
Change-Id: I6a419c62700c568254ff67ae5b73efed2fc98aa2
2020-07-16 14:58:28 +00:00
Yingrong Zhao
175f048aa7 scripts;cmd/storj-sim: include satellite node id in satellite address
Expose satellite ID from storj-sim so we can have access to it when
changing satellite address.

Change-Id: Ife816f4d35eae2d0bc9f9ad592fe75d73d93d9ff
2020-06-05 09:44:19 -04:00
Michal Niewrzal
48e67a29a9 cmd/storj-sim: remove old libuplink from storj-sim
Change-Id: I146a2d7947d1f358e7bdd8acba68eb3fee68539d
2020-05-29 08:39:37 +00:00
Michal Niewrzal
ac375d37bc satellite/payments: remove mockpayments and add Stripe client mock
instead

Change-Id: If3496f6abc16da90d2b43fa0c5be356847a39507
2020-05-19 09:35:37 +02:00
Egon Elbre
ec589a8289 all: fix comments about grpc
Change-Id: Id830fbe2d44f083c88765561b6c07c5689afe5bd
2020-05-11 13:05:34 +03:00
Kaloyan Raev
b5433322b4 cmd/storj-sim: enable open user registration
Change-Id: I695664279f42209245a82cc5bf103d037b844c32
2020-05-05 13:03:02 +00:00
Yingrong Zhao
c5309a3f91 cmd/uplink: set sample rate for tracing to be 1 when tracing is enabled
when tracing is enabled, we should also set sampling rate to
a non-zero value. For now, we will set it to 1.
Uplink CLI users should be able to override it with the sample
flag.

Change-Id: I8bcf514fb14c2a1c4349b7957dd24ec23e4a85e5
2020-04-28 20:15:28 +00:00
Egon Elbre
d3ce845f82 satellite: revert log lines used to figure out node id
Currently storj-sim relies on the log lines to be exactly the same,
when they change it cannot find the necessary information from log.

Change-Id: Ia039915ef3375a7cf60f107b2c05c958de15b6d5
2020-04-15 17:07:56 +03:00
Kaloyan Raev
a2ce836761 remove sugar logging
Change-Id: I6b6ca9704837cb3f5f5449ba7f55661487814d9f
2020-04-15 12:37:47 +00:00
Egon Elbre
1024bf9ce1 all: simplify uuid usage
Instead of uuid.Parse, use uuid.FromString.
This removes a bunch of pointer management logic.

Change-Id: Id25bd174eb43c71d00b450158a198abafd8958f2
2020-04-02 13:45:19 +00:00
Egon Elbre
0a69da4ff1 all: switch to storj.io/common/uuid
Change-Id: I178a0a8dac691e57bce317b91411292fb3c40c9f
2020-03-31 19:16:41 +03:00
Michal Niewrzal
e258de518c cmd/storj-sim: remove identity dir from gateway configuration
We are not generating identity for gateway during setup anymore, its
generated on-fly by libuplink. With this change we can remove
`--identity-dir` from gateway.

Change-Id: I63298115d4399564b2f29541b8dc16e3b3acdcf2
2020-03-23 14:11:51 +00:00
Michal Niewrzal
d994584e88 cmd/storj-sim: remove removed flags from storj-sim gateway setup
Change-Id: I27d585740e8441b41bce5010480cc8e4f5deb9ce
2020-03-20 12:09:55 +01:00
Jessica Grebenschikov
5142874144 satellite/gc: move garbage collection to its own process
Change-Id: I7235aa83f7c641e31c62ba9d42192b2232dca4a5
2020-03-18 16:44:01 +00:00
Jennifer Johnson
1c1750e6be removes bandwidth limiting
On satellite, remove all references to free_bandwidth column in nodes table.
On storage node, remove references to AllocatedBandwidth and MinimumBandwidth and mark as deprecated.

Protobuf message, NodeCapacity, is left intact for backwards compatibility.
Once this is released to all satellites, we can drop the column from the DB.

Change-Id: I2ff6c6537fc9008a0c5588e951afea58ede85838
2020-03-04 14:04:00 +00:00
Egon Elbre
decb2ec69a private/processgroup: moved to storj.io/common/processgroup
Change-Id: I1ec0bb440dda757d8f9a6f564a0084dde2f9cc84
2020-03-03 10:50:33 +00:00
Michal Niewrzal
54e38b8986 pkg/miniogw: gateway implementation with new libuplink
Change-Id: I170c3a68cfeea33b528eeb27e6aecb126ecb0365
2020-02-21 16:20:38 +01:00
Michal Niewrzal
cea4c25f53 mod: bump common and uplink version
Change-Id: Ia063d33c087dd91a46c008e154b078f11fa21527
2020-02-12 14:33:54 +00:00
Egon Elbre
dbf46c4aa7 satellite/admin: administrative endpoint
Admin server allows creating basic REST and html API-s
for different administrative tasks.

Change-Id: I3dc1786abe1c87350eed60ec90e48130f44e63cf
2020-02-12 12:12:50 +02:00
Egon Elbre
429f08b4f0 satellite: add Admin peer
This peer will contain our administrative panels.
It's completely separated from our other satellite
processes because it allows better control for restricting
access to it.

Change-Id: Ifca473bee82ff6c680b346918ba32b835a7a6847
2020-02-11 16:15:33 +00:00
Egon Elbre
7d62b1cf3b cmd/storj-sim: don't wait for process to start indefinitely
In case the endpoint doesn't start, it might end up indefinitely
waiting for it to come up stalling jenkins.

Change-Id: Ib10bf1a25461e7532ec56ca705178bc9a7f85d12
2020-02-11 16:15:18 +00:00