Commit Graph

1217 Commits

Author SHA1 Message Date
Cameron
98fed4bc30 {satellite/console,web/satellite}: get project salt from satellite
Add getSalt to projects api. Add action, GET_SALT, on Store
Projects module to make the api request and return the salt
string everywhere in the web app that generates an access grant.
The Wasm code which is used to create the access grant has been
changed to decode the salt as a base64 encoded string. The names
of the function calls in the changed Wasm code have also been
changed to ensure that access grant creation fails if JS access
grant worker code and Wasm code are not the same version.

https://github.com/storj/storj-private/issues/64

Change-Id: Ia2bc4cbadad84b066ca1882b042a3f0bb13c783a
2022-10-12 19:06:27 +00:00
Egon Elbre
dd60318147 cmd/tools/segment-verify: use resolved ip
Change-Id: I3662aaea3ff8721c415c038b2b5324d165b60975
2022-10-12 12:43:11 +00:00
Clement Sam
d800b51dd9 cmd/multinode: generate identity when not provided
Closes https://github.com/storj/storj/issues/4974

Change-Id: I7a07d846be7ac8f8f7d7e9ad61511ff84d2ab400
2022-10-12 12:01:14 +00:00
JT Olio
9c09a8dbd5 cmd/uplink: don't send TODO as the instance id
there's not really anything better to send. uplinks have
rotating node ids on each startup, so that's not right
here. i don't think anyone will use instance id for
uplinks so let's just fold and send nothing.

Change-Id: I2511605e95eba1816d662d385b28d5feab8c4eb0
2022-10-12 06:56:19 +00:00
Egon Elbre
ff22fc7ddd all: fix deprecated ioutil commands
Change-Id: I59db35116ec7215a1b8e2ae7dbd319fa099adfac
2022-10-11 15:27:29 +00:00
Egon Elbre
142a04f208 cmd/tools/segment-verify: add connection pool
Change-Id: If0f85edbf99438ac41c23fc7107fdab926288cc2
2022-10-11 09:06:44 +03:00
Egon Elbre
8916f2ee92 cmd/tools/segment-verify: allow ignoring specific nodes
This adds a new flag that allows to ignore some nodes completely.

Change-Id: I203d25f931262c809037e25e9c37e9a89bf47026
2022-10-10 20:14:38 +03:00
Egon Elbre
9e50d837e3 cmd/tools/segment-verify: add tool for summarizing log
Change-Id: I3177ab71dfd25e11adfedce32a530d83dda63bd6
2022-10-10 20:02:50 +03:00
Egon Elbre
5f01dad3a3 cmd/tools/segment-verify: add total progress indicator
Change-Id: Ib729abf6adbeba8d94e08c7e11497c6d5ddd5ec2
2022-10-10 20:02:30 +03:00
JT Olio
d632f23950 cmd/uplink: add eventkit
Change-Id: If109c8f7de257b77794e45599487ad2c46f2c3ec
2022-10-07 12:08:08 -04:00
dlamarmorgan
d39caf7674 cmd/satellite: change invoicing flags to iso8601
Update input parameters for invoicing commands to use iso8601 standard formatting.

Change-Id: I3b4a7df268e8f1a5cd51b0dab03ed38e13a76554
2022-10-07 07:58:23 -07:00
Egon Elbre
ea4b3023d9 cmd/tools/segment-verify: fix piece id derivation
Change-Id: Ib27fd8630e1e5a90060dff2a09c51f488960177f
2022-10-06 13:43:08 +00:00
Egon Elbre
c8506cdda3 satellite/metabase,cmd/tools/segment-verify: simplify interface
Change-Id: Icdd445b1713bc26cee3b3a125b68b0cde0739837
2022-10-06 13:42:00 +00:00
Márton Elek
f61507a125 satellite: include 'unsafe' in the new migration options
Change-Id: I9d30c6ece430aad420332df159d0d41c6688f9d6
2022-10-06 12:05:14 +00:00
Egon Elbre
c1817ab743 cmd/tools/segment-verify: a few fixes
The flags weren't properly loading from config.

The code assumed that every node that's online for downloading also have
data uploaded to them -- which is not true.

Change-Id: Ifd65a47b9eca5b4841231928244fab17acbde6fb
2022-10-05 15:51:38 +00:00
Márton Elek
7f6f7e1246
satellite: optional migration for integration tests
This patch addresses the following issues:

 1. Running full migration in cockroachdb is quite slow. We already have an approach for unit tests to start from the latest snapshot. This patch makes it possible to use it for integrations tests.
 2. Migration requires executing a separated command which makes it hard to run application in containerized test environments (like storj-up) or from IDE. This patch introduces a hidden flag to run migration.
 3. Test user creation is painful. We do it with calling GraphQL + admin API. Providing an option with testuser makes the integration tests significant more simple (especially as the projectID -> access grant can be predictable)

Change-Id: I61010728727b490ea6aac32620f2da0484966727
2022-10-04 11:35:03 +02:00
dlamarmorgan
85f9dad225 satellite/payments/stripecoinpayments/service.go: add date to pay invoices command
Add an extra parameter to the pay-invoices command that can be used to restrict which invoices will have a payment attempted in stripe. The parameter should be of the form MM/DD/YYY and any invoices created on or after the date will have token balances applied and be processed for payment according to stripe subscriptions settings.

Change-Id: I5da5070d3ac97f45c05c02f2849254bdc44413c3
2022-10-03 21:28:37 -07:00
Jeremy Wharton
4ab8031171 cmd/satellite: combine draft invoice generation commands
This change introduces the generate-invoices satellite billing
command whose functionality is equivalent to running
apply-free-coupons, prepare-invoice-records,
create-project-invoice-items, and create-invoices in order.
Invoice finalization must still be performed separately.

Change-Id: Ia3d80b95eef1f2776c38bd730ed731e42ec4c35e
2022-10-03 21:19:44 -05:00
Egon Elbre
4c374a2357 cmd/tools/segment-verify: add tiny readme
Change-Id: Ia314c615f8b7fdb13e2b5f81c1be82ec686ca819
2022-10-03 16:01:24 +00:00
Michal Niewrzal
a97cd97789 satellite/orders: remove unused service dependency
Orders service doesn't need buckets service anymore.

Change-Id: I27853cda87e82b528f53667e4b4866801f7bfb62
2022-09-28 08:56:36 +00:00
Jeff Wendling
929642b675 cmd/uplink: remove io/ioutil
it has been deprecated since go1.16.

Change-Id: I8c15b24176e5a4687fe297d569505e84fc5affda
2022-09-27 15:53:07 +00:00
Jeff Wendling
fa4af92392 cmd/uplink: improve cp behavior
allow multiple source paths and a single destination path.
this makes commands like `uplink cp foo* sj://bucket` work
as expected.

require at least one remote path when copying. this ensures
that users don't accidentally overwrite their local files
with other local files, which is almost never what they wanted
because they would just use cp.

Change-Id: I28948f4ff735d29db06de81fc8c2a15b9f4ee3f5
2022-09-27 10:23:41 +00:00
Egon Elbre
8069973dee cmd/tools/segment-verify: add failure tests
* Fix an invalid slice index calculation.

Change-Id: I7f1b85edc46df362697aa132b967d5d23f9d5522
2022-09-26 19:38:16 +00:00
Erik van Velzen
9e2a57130a cmd/uplink: restore explicit share expiry
Due to a programming error it was possible to "share" without an expiry
implicitly. This pollutes the auth service database.

fixes https://github.com/storj/storj/issues/5188

Change-Id: I04a345662c26948c6be6c1ae6bee3b5a583bebc4
2022-09-23 11:57:08 +00:00
Egon Elbre
f98d551c9b cmd/tools/segment-verify: test service
Change-Id: Ibd83960c18123e8f29e22089007dc32c8d532240
2022-09-22 17:23:02 +00:00
Egon Elbre
0bfaadcc6c cmd/tools/segment-verify: fixes and more tests
* Disallow too large listing limit, which would cause a lot of memory to
  be consumed.
* Fix throttling logic and add a test.
* Fix read error handling; depending on the concurrency it can return
  the NotFound status either in the Read or Close.

Change-Id: I778f04a5961988b2480df5c7faaa22393fc5d760
2022-09-22 10:32:30 +00:00
Egon Elbre
0e99f7a8cf cmd/tools/segment-verify: add loading of priority nodes
Change-Id: Idcc41469ea5f71eab1b9dccbe0f14da537386a17
2022-09-21 14:56:13 +00:00
Egon Elbre
8b527f2d12 cmd/tools/segment-verify: add throttling
Change-Id: Ia0b4ec255adc90d874f4366b80799414a1a94700
2022-09-21 14:52:51 +00:00
Egon Elbre
cf50696745 cmd/tools/segment-verify: wire up overlay logic
Change-Id: I0a4c737a8b0995a1c3e3adeac728fe833d0ce684
2022-09-19 11:32:18 +03:00
Egon Elbre
0809ae73cf cmd/tools/segment-verify: add main
Change-Id: Ib7161a0f44d447f9ddb9be83f6673587a0bd7712
2022-09-19 10:36:57 +03:00
Jennifer Johnson
8529a169ee cmd/tools/segment-verify: add verifier
Change-Id: I4cc1fbcf964c4a9a37cf80322f6f99dd956f3d7b
2022-09-19 10:36:57 +03:00
dlamarmorgan
a3a3ffd123 satellite/payments/stripecoinpayments: update invoicing to use credit note
Rather than using Invoice Items to account for storjscan token payments, credit notes will be used and applied to the users finalized invoice. This credit note will reduce the amount due of the users invoice based on the amount of storj token balance the user has on the satellite. Applying credit notes to a finalized invoice also requires that the invoice not be automatically paid when finalized. Therefore, a new command (pay-invoices) was added to initiate payment for users invoices.

Change-Id: Ie539375a10e842e3cb64bf0140834bbab0774f54
2022-09-16 13:40:38 +00:00
Egon Elbre
9b520b2114 satellite/metabase: expose ConvertNodesToAliases and ConvertAliasesToNodes
They are needed for segment-verify tool.

Also rename some of the conversion methods to make clear,
which of them have side-effects.

Change-Id: Ie9a0952548e9ed5068c7a30c2fd2134b07139bca
2022-09-15 13:56:10 +00:00
Egon Elbre
cd81c5bd58 cmd/tools/segment-verify: add csv writer
Change-Id: I9306d7a6927f4dacca9623d7bd57f8560404db3e
2022-09-15 13:28:21 +00:00
Egon Elbre
507b099d44 cmd/tools/segment-verify: add monitoring / error
Change-Id: I6fd0369719ddf176a98208348560004a4134f810
2022-09-14 18:20:48 +00:00
Egon Elbre
6127f465dc cmd/tools/segment-verify: add logic for iterating over segments
This adds parts for:
1. iterating over the segments
2. using an interface for writing the segments
3. stubs for handling deleted segments

Change-Id: I76a17cac6deb0b6c042a8ab7c4155a890db9da84
2022-09-14 18:20:31 +00:00
JT Olio
3b7d8e8798 cmd/tools/convert-node-id: tool for node id conversions
Change-Id: I3d18a051131135dfdbfd9c54002efbc853e4004d
2022-09-14 11:13:50 +03:00
Márton Elek
26444194c2 uplink: fix distributed tracing reporter
This patch fix the beavior of the distributed tracing reporter.

1. For developer build we don't append the date
 * We don't need to separate service instances in jaeger (search by trace ID)
 * It's usually 0000-00-000 anyway as release.sh is not used for dev builds

2. Tracing ID MUST be unique
 * Instead of trusting the user to set a unique value (how can they do it?), we generate a random number

3. To make it possible to find the trace, there is a new flag to print out the generated tracing ID

4. Monkit `remoteTrace` call is replaced with normal monkit Task.
 * remoteTrace call assumes that we have a parent span in an other service (which is already sent to the server)
 * Here we must send out the parent span, as this is the beginning of the trace

5. We properly close the Jaeger UDP collector, and we wait until remaining messages are sent out

Change-Id: Iabf5abf25f4f20881188f88edcbadca95ac74927
2022-09-13 14:36:41 +00:00
Egon Elbre
8459a3faa2 cmd/tools/segment-verify: add processing of a collection of segments
This adds logic to put segments into batches and retry them.

Change-Id: I86450e2f5ab13edb3ae9a5d5b9417e0cced2892b
2022-09-12 18:29:04 +03:00
Egon Elbre
3467fd4b7f cmd/tools/segment-verify: implement batch creation
Implements creating roughly load-balanced set of batched
that can be used to make multiple requests.

Change-Id: I349b276176dcb8ba9163e7e06a94509d73fa5ddc
2022-09-12 18:29:01 +03:00
Márton Elek
1646a0ac9e cmd/uplink: experimental environment variable to change piece hash algorithm
Change-Id: I7420919146c64d29ff1023f7498867d3d30892d0
2022-09-08 10:41:04 +00:00
Egon Elbre
731916ffa4 cmd/inspector: move command back from cmd/tools/
Some of the makefiles require it to be in cmd/inspector.
This was initially moved in 381c1e1257.

Change-Id: I850f3bfa1d66e66fbd1d9069e353984a27a7c58c
2022-09-07 09:35:12 +00:00
Egon Elbre
381c1e1257 cmd/tools: move tooling to a separate folder
This helps to cleanup the cmd folder a bit.

Change-Id: I24025c3dbfd35966325d7d5aaa95cd9a1176a8b7
2022-09-02 18:25:36 +03:00
Cameron
7933e0c4c7 cmd: add command to migrate project public_ids
This is to update all projects to have a public_id if they do not have
one.

github issue: https://github.com/storj/storj/issues/4861

Change-Id: Icfa42b62e15ca75d3c04a0aab48a3c3b0f3a9d6e
2022-08-31 13:14:17 +00:00
Michal Niewrzal
2ae1db53b6 cmd/satellite: add command for collecting GC bloom filters
We would like to have separate process/command to collect bloom
filters from source different than production DBs. Such process will
use segment loop to build bloom filters for all storage nodes and
will send it to Storj bucket. This change is extending satellite binary
with appropriete command.

New GC service for collecting bloom filter will be a subsequent
change.

Updates https://github.com/storj/team-metainfo/issues/120

Change-Id: Ibc03e119c340919cf468fc1f5a4f3d187bb3a5a1
2022-08-31 11:40:36 +00:00
Márton Elek
ea1408f7a8 go.mod: bump clingy dependency
As a reminder: latest clingy removed the requirement of having custom context (which made the usage of context.WithValue harder) and uses simple context instead.

Clingy saves the stdin/stdout/stderr to the context (earlier to separated context type) to make it available for unit testing.

Change-Id: I8896574f4670721de43a577cd4b35952e3b5d00e
2022-08-31 10:24:27 +00:00
JT Olio
e621190b99 satellitedb: randomize nodes for restore from trash
Change-Id: I030bd9093ec13a882edc7e0d325b9b19c201e135
2022-08-26 17:19:41 +00:00
JT Olio
12ef68bdf4 satellitedb: restore-trash shouldn't bother with nodes we've never talked to
Change-Id: I3c631e92a9a2670c52c9fa23b2b1baf67d3c9a9c
2022-08-25 16:14:29 +00:00
Clement Sam
ea0124a183 cmd/multinode: handle JSON input with BOM to add command
Decode the JSON input string to its corresponding unicode
decoding if the special byte order mark is present.

Fixes https://github.com/storj/storj/issues/4950

Change-Id: If91bac22590c89b35c58bf54f6d3bdb8a67d7a4f
2022-08-25 03:05:49 +00:00
Márton Elek
3d79c5f3a1 storagenode/docker: allow to configure updater with separated env variables
We couldn't use environment variables safely to configure storagenode, since we introduced the embedded updater.

For example STORJ_DEBUG_ADDR=localhost:11111 would try to set debug port 11111 for both the storagenode and storagenode-updated, causing port conflict.

This small change enables to configure storagenode-updater with STORJUPDATER_... environment variables.

Tested with creating custom image and installing to my own storage node.

Change-Id: I6b0a601a4dc63d2d1ff3c191ae89981434e55c30
2022-08-24 06:50:30 +00:00