Compare commits

...

758 Commits

Author SHA1 Message Date
5546e07191 HACK: prebuild storagenode gui 2023-12-08 17:18:13 +00:00
Andrii Kotko
61a64a6ed8
release v1.94.1 2023-12-08 16:59:29 +02:00
Tome Boshevski
5767191bfc web/satellite/vuetify-poc: ui updates and improvements (#6576)
Updating common components and dialogs.
Updating the new browser card view.
Updating the file preview.
Updating usage per project.
Updating tooltips and messaging.
Icons update.
Themes and styling updates.
2023-12-08 16:55:09 +02:00
Tome Boshevski
88a78262b1 web/satellite: update preview types and icons (#6577)
Add .md .json .xml as text preview types.
Add .webp format for image preview.
Add .ogv format for video preview.
Add .aac .flac for audio preview.

Add all these plus more extensions to show the correct icons in the browser.
2023-12-08 16:55:09 +02:00
dlamarmorgan
d07dba00e3 dbx/satellitedb: update default_versioning field for projects
Update the default versioning fields to align with the values of
the buckets versioning field.

Change-Id: Ia8657f31733bfd4a3e45bbfa25500ddcd3461913
2023-12-08 16:55:09 +02:00
Márton Elek
9414e3270d
satellite/durability: remove unclassified measurements + reduce memory
Small adjustments in durability observer:
 * use local map for class names instead of struct level var (not used anywhere else, earlier GC)
 * continue / skip when the class is "unclassified" / 0.
 * `classes["unclassified"] = 0`: doesn't really matter, but this is more true

Change-Id: Ib927bab52d5502ad67ecc8570fda218fbfb6a95b
2023-12-07 16:57:10 +01:00
Moby von Briesen
c87f380e2e satellite/repair: Fix flaky test
Don't use global monkit.Default in Queue Stat test, or it can fail due
to concurrently-executing tests.

Change-Id: I061d626f26220705c8dd0de17ac7e14c81831d7f
2023-12-05 16:19:30 -05:00
Vitalii
5094100e21 web/satellite: fix indentation for TS logic
Added new eslint rule to fix indentation for typescript logic.

Change-Id: I08ad94411c579ac6c4687217a29c3b7dc61b0d76
2023-12-05 20:17:10 +00:00
Vitalii
61a9d63d76 web/{satellite, vuetify}: rework locked objects logic
When a user enters the encryption passphrase when trying to enter a bucket, only display a "you may have files uploaded with a different passphrase" message when: 0 objects can be listed with the provided passphrase, but there are >0 objects in the bucket.

When a user enters the encryption passphrase when trying to enter a bucket, and >0 objects can be listed with that passphrase, proceed immediately with opening the bucket and displaying the files: do not show any indication in the file list view that there might be "locked files".

Issue:
https://github.com/storj/storj-private/issues/516

Change-Id: I2ae3809971867e5b69b804192a380a6919ed0108
2023-12-05 19:27:04 +00:00
Michal Niewrzal
45cfaa8743 satellite/metainfo: set internalpb.StreamID Versioned field
To have the same UploadID returned from BeginUpload, ListObjects
and ListPendingObjectStreams we need to set StreamID.Versioned field
everywhere.

Change-Id: I1328d9c476767559b8feb7c5bcd5afb154f7cee3
2023-12-05 14:46:56 +00:00
Vitalii
c5e44072a8 web/satellite/vuetify-poc: prevent page reload on folder create
Remove default page reload behavior on form submit during folder creation.

Issue:
https://github.com/storj/storj/issues/6559

Change-Id: I6d5c433adb4046238f958761b4d1f27696cfc459
2023-12-05 10:22:43 +00:00
Vitalii
b31e417546 satellite/payments: use stripe idempotency on create/update invoice items
Add idempotency key for every create/update invoice item request.
Key consists of public project ID, line item description and formatted invoicing period.
This should eliminate the possibility of making duplicate create/update actions.
Also, with this change, we mark a project record as consumed only after the appropriate invoice items are created or the record itself is skipped.

Issue:
https://github.com/storj/storj/issues/6501

Change-Id: I7506a8f3886d7f575bcc0facc3f107513352a312
2023-12-05 11:16:53 +02:00
Márton Elek
504c72f29d satellite/repair: run chore which updates repair_queue stat
Change-Id: I18c9e79e700ac690c3aa78ee0df0cf6089acd2ac
2023-12-05 04:33:46 +00:00
Wilfred Asomani
0590eadc17 satellite/{db,console,billing}: fix legal/violation freeze
This change fixes the behavior of legal freeze; it now does not allow
lists and deletes for users in that freeze. It also fixes an issue
where users who have been legal/violation frozen will have their limits
upgraded, partially exiting freeze status when they pay with STORJ
tokens.

Issue: storj/storj-private/#517
storj/storj-private/#515

Change-Id: I6fa2b6353159984883c60a2888617da1ee51ce0a
2023-12-04 21:19:44 +00:00
Ivan Fraixedes
e1c12674c5 satellite/admin: Fix API key delete by name
We couldn't delete API keys by name whose name contained slashes because
Gorilla Mux router interpreted the as path separator and didn't resolve
to the right endpoint.

To fix the issue the name is sent as a query parameter rather than as a
path parameter.

Change-Id: Ica67d6b9f047d7c33a5350457afc822cb8d4c7a1
2023-12-04 20:03:30 +00:00
Clement Sam
8ec1a8de7d satellite/satellitedb: drop nodes.type column
This is the final change to completely drop the type
column from the nodes table.

All code references are removed and deployed, dbx changes
have been merged and deployed.

Updates https://github.com/storj/storj/issues/5426

Change-Id: I1b76fe92f25e350c1c44730c008f0e9107852509
2023-12-04 17:31:33 +00:00
Michal Niewrzal
a3f84bcb0c satellite/gc/bloomfilter: reduce number of filters in single zip
Current value is too big and can cause memory issues. With 500 bloom
filters where single filter can go up to 2MB we can end up with 1GB zip
which is more problematic if we don't want to download it before
processing.

Change-Id: I4b983afbcac39c718afbbeb764a30d32c4e80c08
2023-12-04 16:09:24 +00:00
Wilfred Asomani
5d775e61b1 satellite/{console/web}: add v2 signup flow flag
This change adds a new flag to toggle whether to use the v2 app signup
flow. If disabled, the v2 signup page will redirect to the current
app's signup page.

Issue: #6552

Change-Id: I0d48ca713dc77cb2879630613f92123baced3ca3
2023-12-04 15:18:37 +00:00
Márton Elek
9e6c421cc5 satellite/durability: report bus_factor with the new reports
New instance based satellites are reporting durability alerts like `class=last_net, value=192.168.0.0`.

Bus factor will be reported as `class=bus_factor value=last_net`.

Change-Id: I0130866fa8e50b3d2876f580a78383bb1c609159
2023-12-04 14:06:20 +00:00
Cameron
5c79f59b79 web/satellite: remove VChart tooltip elements on unmounted
We use a custom plugin for the chart tooltips. If a user navigates away
from the page while a tooltip is rendered, the element doesn't get
removed. To fix it, when the chart is unmounted check for the tooltip
elements and remove them if they exist.

issue: https://github.com/storj/storj/issues/6344

Change-Id: I42122ef5fad3a1ee7d16ed554881f8c3f79f2b26
2023-12-04 12:42:21 +00:00
Vitalii
7b3770ed17 web/satellite/vuetify-poc: hide 'Add coupon' button when coupon features are disabled
Hide button if feature flag is set to false.

Issue:
https://github.com/storj/storj/issues/6547

Change-Id: I0e13bacfe6a8651ebc573abc3f23a749821d63a5
2023-12-04 10:38:13 +00:00
Vitalii
775ee860a5 web/satellite/vuetify-poc: remove billing information tab
Removed billing information tab because it's non-functional.

Issue:
https://github.com/storj/storj/issues/6546

Change-Id: Ieb9a691dcc5015863108e11895bb75536fb310b2
2023-12-04 10:36:59 +00:00
Vitalii
207dad4362 cmd/satellite: bind new billing command to config
This is a fix for invoice generation flow (method 2).
New command wasn't bind to config.
Method 1 still works as expected.

Change-Id: I481a0646f3f8f6f877ee8efab3622dd525d8c22a
2023-12-01 23:49:20 +00:00
Vitalii
0c913ab548 satellite/{console, web}: show banner for bot accounts
Display a banner on the dashboard of all projects for bot accounts, with a prompt to submit a support ticket.

Issue:
https://github.com/storj/storj-private/issues/503

Change-Id: Id5f2aae0394b4fdca968a4de93bda19d6b0eb37a
2023-12-01 23:01:11 +00:00
JT Olio
591971b4dc satellite/metainfo: reduce database hits for segment creation
Change-Id: I48a748b48daefa95f1dfbf6a0d75e65a568ee36a
2023-12-01 20:28:48 +00:00
Ivan Fraixedes
5d492a9e01 satellite/admin/back-office: Don't send project ID
The project ID must never be sent out from the server because old
projects used that ID as salt for creating access grants.

We must always expose the public ID and the server must retrieve the ID
when the public ID is sent in to the server and needs it for performing
some DB queries.

Change-Id: I4a9d94049ef84f578b62907827d6c09dfd1db4db
2023-12-01 19:07:42 +00:00
Wilfred Asomani
94bbda81dc web/satellite/vuetify-poc: add delete action to file preview
This change adds a menu to the file preview dialog with an action to
delete files.

Issue: #6553

Change-Id: I3b282696077b884df4171a85dee6a62dc9fb0513
2023-12-01 18:18:57 +00:00
Wilfred Asomani
4b660e58fa web/satellite/vuetify-poc: add partner branding to signup
This change adds partner branding to the signup page.

Issue: #6528

Change-Id: Idbed1818b30e80a818311ee4a73d6660a3aadc6d
2023-12-01 17:11:25 +00:00
JT Olio
8a42cad6a1 cmd/tools/convert-node-id: show node id for an id.cert
Change-Id: Iae392398dac3d4f45ef829b25b27a2a7a69b0932
2023-12-01 15:24:39 +00:00
Vitalii
b776382f18 satellite/admin: functionality to disable account bot restriction
Added new admin functionality to disable account bot restriction by activating it.
This must be used for accounts with status PendingBotVerification.

Note: PendingBotVerification status is automatically applied for accounts with high captcha score right after activation.

Issue:
https://github.com/storj/storj-private/issues/503

Change-Id: I4c9ee834075a7abaf221ac054a455a8d27debf40
2023-12-01 13:09:49 +00:00
Vitalii
594b0933f1 satellite/console: PendingBotVerification status for users with high captcha score
Added new captcha score cutoff threshold config value (default is 0.8).
Added new user status PendingBotVerification which is applied right after account activation if signup captcha score is above threshold.
Restricted project creation/joining if user's status is PendingBotVerification.

Issue:
https://github.com/storj/storj-private/issues/503

Change-Id: I9fa9932ffad48ea4f5ce8235178bd4af45a1bc48
2023-12-01 15:09:01 +02:00
Ivan Fraixedes
fb31761bad satellite/admin/back-office: Add auth middleware
Create an API generator middleware for being able to hook the new
satellite admin authorization in the endpoints.

The commit fixes a bug found in the API generator that caused that
fields of types of the same package of the generated code where wrongly
added. Concretely:

- The package matching was missing in the function middlewareFields,
  hence it was generating code that referenced types with the package
  name.
- middlewareFields function was not adding the pointer symbol (*) when
  the type was from the same package where the generated code is
  written.

There is also an accidental enhancement in the API generator because I
thought that the bug commented above corresponded to it, rather than
removing it, I though that was worthwhile to keep it because it was
already implemented. This enhancement allows to use fields in the
middleware with packages whose last path part contains `-` or `.`, using
a package rename in the import statement.

Change-Id: Ie98b303226a8e8845e494f25054867f95a283aa0
2023-12-01 00:29:49 +00:00
JT Olio
0467903b52 cmd/uplink: don't show throughput during copy, since it's wrong
we can bring it back once we understand this issue:
https://github.com/vbauerster/mpb/issues/135

Change-Id: I2cbdcfc260786e1ab08d1badc78770a792785650
2023-11-30 21:20:50 +00:00
Wilfred Asomani
8568e49578 satellite/{web,console}: remove upload modal flag
This change removes the config flag for the new upload modal.

Issue: #6515

Change-Id: I69612c6730f933d6320cb3c05995900eac98ac91
2023-11-30 19:43:24 +00:00
JT Olio
e03091dd51 go.mod: bump storj.io/{common,uplink}
Change-Id: Ifb7c44e57643e1e3cfab5f9f70f1bfa5224c5025
2023-11-30 11:24:19 -05:00
Egon Elbre
a44ed9457e satellite/metabase: allow creating delete markers for missing objects
Change-Id: Ic8f6f80c234478b3e7b3bc4e88fbb5df9a783456
2023-11-30 16:33:42 +02:00
Wilfred Asomani
50c9f4c85a web/satellite/vuetify-poc: add account setup dialog
This change copies the account setup dialog from the static v2 repo and adds functionality to it

Issue: #6481

Change-Id: I0cacec3edf054c1945df7593b73e3c0f0f96677c
2023-11-30 13:15:31 +00:00
Ivan Fraixedes
ea022ede46 satellite/admin/back-office: Add method for auth middleware
Add a new method to the Authorizer to use it with the API midleware that
we are going to implement for injecting it into the handler generated by
the API generator.

This new method will reduce the lines of code to generate and avoiding
errors that are more difficult to test in generated code.

The commit deletes the Middleware method because we won't used due to
the API generator doesn't support "standard" middlewares and allows
their customization via code generated and injected inside the handler
base logic generated by the API generator.

Change-Id: Ie427eb2eea94797913e2c357cf097ecf1e2e63ef
2023-11-30 11:47:34 +00:00
Wilfred Asomani
a3a7df91e3 web/satellite/vuetify-poc: add signup functionality
This change implements functionality for signup and account activation.

Issue: #6481

Change-Id: Ide8f743dc3996c8b2a23a494f8d8500e1af658bc
2023-11-30 10:13:04 +00:00
Cameron
26b6ab8a3a web/satellite: only fill charts with zero if day is complete
Before this change we would fill all days on the chart with zero if no
data existed. This led to concerns when users saw their stored data
value drop to zero. Instead, we will leave incomplete days blank if
there is no usage data. This includes today and future days.

issue: https://github.com/storj/customer-issues/issues/1157

Change-Id: I4b9def392b89684fe63f1503274e868a240eb353
2023-11-30 08:28:10 +00:00
Moby von Briesen
dcc3245954 satellite/console: Update project members endpoint
We don't need to send all the information that the endpoint currently
provides - name and email is sufficient.

Change-Id: I088d61edfc022a4a40c96e1faebd48808af5b00f
2023-11-29 17:34:35 -05:00
Wilfred Asomani
f749b8ff51 satellite/{db,console}: support v2 app account set up
This change modifies the register endpoint handler to not require name
for signups from the v2 app and adds a new endpoint for completing
account information (e.g. name). This is to support the new signup and
account setup flow of the v2 app.

Issue: #6470

Change-Id: I256e1c804fcdbc8ce05aa82d6bc4b0263f55daa5
2023-11-29 20:59:55 +00:00
Márton Elek
1ba314428f go.mod: make bump-dependencies (uplink, common, ...)
It requires more work at this time, as https://review.dev.storj.io/c/storj/private/+/11066 modified the way how we configure the debug port.

Change-Id: I4a7cc999e13fd3514064a515b21885bb4d39ff16
2023-11-29 16:55:41 +00:00
dlamarmorgan
fd13cf1058 dbx/satellitedb: add default_versioning field for projects
Change-Id: I294a85c244976493b490a58f17536f06f6c01306
2023-11-29 15:29:14 +00:00
Wilfred Asomani
ab65572af0 web/satellite: implement account activation with code
This change implements account activation using OTP code. Based on
whether this activation method is activated, signup will require
    the user to fill the activation code to activate their account.

Issue: #6428

Change-Id: Ifdd3e34ddb8ab299dd24b183f9c484613b2e8888
2023-11-29 13:29:50 +00:00
Wilfred Asomani
116d8cbea1 satellite/console: implement account activation with code
This change implements account activation using OTP code. Based on
whether this activation method is activated, signup will generate a
6-digit code and send it via email. A new endpoint is added to validate
this code and activate the user's account and log them in.

Issue: #6428

Change-Id: Ia78bb123258021bce78ab9e98dce2c900328057a
2023-11-29 13:06:37 +00:00
Vitalii
1546732afc web/satellite/vuetify-poc: set input value length limits
Ensure that text inputs have length limits, like in production and more.

Issue:
https://github.com/storj/storj/issues/6536

Change-Id: I2f216cc57d6bcb49b88c34ce53bed58327b0cdc6
2023-11-29 11:57:05 +00:00
Márton Elek
20a3045e1a satellite/durability: use fixed number of pieces in integration test
Test was flaky because we asserted if we have 15 classes:

6 email (for each used (!!!) nodes)
6 last_net (for each used (!!!) nodes)
1 wallet
1 country ("HU")
1 empty value

But there was a very low chance to use only 5 nodes, out of the 6 (RS.Success=5, RS.Total=6).

In that specific case, we had only 12 classes, as we didn't see all the used emails, as we iterated over the used nodes only (and one node was not used).

https://github.com/storj/storj/issues/6549

Change-Id: I66882d5fa9b0d5f5b2397ea856494037972d4b81
2023-11-29 11:04:20 +00:00
Moby von Briesen
03a8e7c81a satellite/analytics: improve error logging
This inserts additional information provided in error messages from
Hubspot into the Go error that is eventually logged out.
Before, we would see the a generic "sending event failed" log.
With this change, we will see more detailed information in the log, such
as a list of required fields that were not submitted.

Change-Id: I24da0646bca62f459377abe6281403020fb54c49
2023-11-29 09:49:25 +00:00
Jeremy Wharton
4822b18472 satellite/admin/back-office/ui: implement view user functionality
This change adds a search field to the new admin UI through which user
email addresses may be submitted. If the email belongs to a verified
user, the client will be redirected to the Account Details page which
is populated with the user's information.

Resolves #6469
Resolves #6475

Change-Id: Icbf3cb3f8374f2764e73a523f111c5ecf3d06569
2023-11-28 16:31:37 +00:00
Egon Elbre
7186525d5c satellite/metabase: test zombie deletion with versioning
Change-Id: I495745f84ea0a7c36763d2889170c4068347dcb7
2023-11-28 12:12:26 +00:00
Ivan Fraixedes
6885ca27fb private/apigen: Inject middleware code after param parsing
Inject the middleware code after the query and path parameters parsing
because in the case that a middleware needs to execute a more expensive
logic, it isn't executed if there are malformed or missing parameters.

Change-Id: I883ade8ee42d25accb153c78d9f583d883730d6a
2023-11-27 23:02:44 +00:00
Vitalii
5e3cab29a2 satellite/{cmd, payments}: add optional command flag to toggle invoice items aggregation
Added new optional '--aggregate' flag for billing.generate-invoices and billing.prepare-invoice-records commands to toggle invoice items aggregation.
Added new explicit command billing.create-aggregated-project-invoice-items which should be used after preparing aggregated invoice records (in cases when invoice generation happens step-by-step).

Change-Id: I04fc0110be5263edb959306d5314a4a1a8eec3ba
2023-11-27 22:11:15 +00:00
Jeremy Wharton
d2819522c6 satellite/admin/back-office: add endpoint to get users by email
This change adds an endpoint to the back office API that returns user
info based on email address.

References #6503

Change-Id: Ib48d30b0b6c6862887b3f8114f50538b3deca57b
2023-11-27 21:05:12 +00:00
Michal Niewrzal
220920edb9 satellite/metainfo: return NotFound status for (Get|Set)BucketVersioning
We need to return NotFound rpc status when bucket doesn't exist.

Change-Id: I3abdd588e2af288618b05513884bc0c545bcbacf
2023-11-27 19:45:45 +00:00
Ivan Fraixedes
c2788ab6ae satellite/admin/back-office: Add README with API guidelines
Add a README document with some general API guidelines that we wrote in
a Slack canvas to discuss them.

Change-Id: Iec933edeb7622b78a98155512b25267d12879837
2023-11-27 18:42:14 +00:00
Ivan Fraixedes
269dd5602e private/apigen: Don't print empty description & version in docs
If the API description or version is empty, don't print out in the docs
a placeholder for them.

In commit bcde51138d a space between the
placeholder and description was removed because it was printing a
trailing space if the description was empty, but that introduced another
bug, which is not adding the space when the description is empty.

This commit fixes the bug, no printing the placeholder if the
description is empty, otherwise print it with the corresponding blank
space in between the placeholder and the description.

Change-Id: I19c92c3dd7dfb38b8b2a43821bafc726ffad59d7
2023-11-27 16:53:33 +00:00
Egon Elbre
4620d6d4cc satellite/accounting/tally: simplify code
Change-Id: I50bdecad5d85e07fdc0dfbd11bb03f8f6bbba56e
2023-11-27 17:01:47 +02:00
Egon Elbre
89dad05c65 satellite/metabase: don't create delete markers for pending
Fix a case where it was possible to create a delete marker when only
pending object was present.

The solution is not pretty, but we have a TODO note to cleanup the
precommit code, so let's fix the bug first.

Change-Id: I0ab66d745443c9dccbf29ef32389dd912b2d9caf
2023-11-23 11:13:21 +00:00
Ivan Fraixedes
adcd810e37 private/apigen: Allow to customize handlers logic
The API generator doesn't have a way to customize each Go handler
endpoint unless that the Go generator is modified.

This commit adds a way to customize each endpoint injecting instances of
types that implement an interface (Middleware) that return the code to inject.

To show how it works, the commit get rid of the 2 fields that we used to
customize the authentication request with the logic that the
satellite/console/consoleweb/consoleapi needs and replace the hardcoded
customization using this new way to customize handlers.

This new way should allow to hook the satellite/admin/back-office
authorization into the handlers using a Middleware implementation.

Change-Id: I894aa0026b30fa2f4a5604a6c34c22e0ed582e2b
2023-11-23 06:57:40 +00:00
Cameron
479fbb628c satellite/satellitedb: fix bucket sum for daily usage
The code to get daily storage usage is a little confusing. There is a
portion of the query which looks like it is trying to sum the storage
for buckets by each day, but it doesn't work. There is also a section
which sums the buckets by reading the returned rows, but it assumes that
the rows are ordered by the date, which they are not. This results in
the potential for multiple entries for the same day being created: one
for each bucket. The UI doesn't know about this, so the entries for the
same day overwrite each other and result in the appearance of less
stored data on the graph.

To fix it, don't group by bucket and allow the query to sum the buckets
together.

Change-Id: Id3ec25238e6e6346b93b10f834bdc9c70023f1e1
2023-11-22 16:25:46 -05:00
Igor
fb894b3720
scripts/draft-release.sh: improve changelog script (#6508)
- Improved commit message parsing for better format handling
- Added argument validation and user prompts for interactive mode
- Implemented enhanced error handling and logging
2023-11-22 19:26:03 +01:00
andriikotko
3c2596e3e2
mainteiners.md: update release process description (#6430)
Co-authored-by: Antonio Franco <antonio@storj.io>
2023-11-22 17:01:04 +01:00
Vitalii
6b1c62d7b2 satellite/{payments, db}: aggregate invoice items if many projects
Implemented invoice items aggregation if projects count is more than 83 for a single invoice.

Change-Id: I6bce81e537eaaddd9297a85718b594047436964a
2023-11-22 11:47:48 +00:00
Wilfred Asomani
f09f352628 satellite/{console,emails}: add activation code email
This change adds an email template for sending activation codes.

Issue: #6428

Change-Id: Ief56a2e32ac0a335561572b9573215da24d9146b
2023-11-22 10:01:27 +00:00
Wilfred Asomani
15b90661f4 satellite/{console,web}: add activation code config flag
This change adds a config flag for whether signup activation code
should be used.

Issue: #6428

Change-Id: I2b2b1e76f6013eab6438aefba848af0a4bd62544
2023-11-22 09:58:35 +00:00
Wilfred Asomani
26574fb2bd satellite/db: add account activation columns
This change adds a new column to the user table, which will hold a
random code for account activation. And another to hold the signup
request ID as another layer of verification.

Issue: #6428

Change-Id: Icd46cb5d8fc76102264d599aca27686cd8b2e84e
2023-11-22 09:57:52 +00:00
Michal Niewrzal
573ce712f2 satellite/bloomfilter: don't create BF for disqualified nodes
Currently we have large set of nodes that are already disqualified and
we are not sending bloom filters to them. The issue is that we are still
generating filters for them while garbage collection process. Even if
we have only segment with one piece which was stored on this node. This
consumes additional memory and processing powers.

This change is changing logic behind `AllPieceCounts` (renamed to
ActiveNodesPieceCounts) to return piece count for all nodes except disqualified one (even with piece count = 0). With this change we can
modify GC observer to skip nodes that where not returned by
ActiveNodesPieceCounts.

Change-Id: Ic75159135abe535084d8aeee560bb801a4a03e17
2023-11-22 07:07:51 +00:00
Egon Elbre
07cb8dc677 satelite/metabase: tiny todo cleanups
Change-Id: I655176e50202b0a363e4dce6299c02545b491244
2023-11-21 19:38:54 +00:00
Egon Elbre
11b083c611 satellite/metabase: test CommitObject.DisallowDelete
Change-Id: I4f408dd23ba2875a0aa46fb4dbd494c3c7dbc0af
2023-11-21 18:47:05 +00:00
Jeremy Wharton
587fa8fdff private/apigen: use correct TS type for nillable fields
This change gives the proper type to TS class fields generated from
nillable Go struct fields. Previously, Go struct fields having a nil
representation ([]Type, *Type, etc.) were translated into TypeScript as
"Type | undefined". This isn't correct because these fields, when nil,
are given the value "null" when marshalled into JSON. This change fixes
this issue by giving these fields the type "Type | null".

Change-Id: I5a1a83eb3810a3cba10895bb2f0f75ca5fd7d1b5
2023-11-21 17:58:07 +00:00
Márton Elek
0f4f1ddde8 satellite/durability: use single classifier per observer instance
the new bus_factor calculation doesn't make sense with different classes, as we have overlaps.

For example: it can detect a risk if we loose one country and one different subnet (with possible overlap).

It's better to calculate the stat and bus_factor per class (net, country, ...).

It also makes it easier to measure execution time per class.

Change-Id: I7d4d5f7cb811cd50c5831077b43e001908aab96b
2023-11-21 17:08:34 +00:00
Márton Elek
7c25e3733a build: move Run Versions test to a separated job
Run Versions test is intermittent and slow.

We need to rewrite it with storj-up, preferable with a container where
 old versions are cached, instead of rebuilding them everytime
 (which is -- IMHO -- the root cause of the intermittency).

As we discussed on the chat, we can separate the runs, until we prioritize this work.

Change-Id: I7fdf6fdb20625fd76d6334be0d0afe72af1b734a
2023-11-21 15:21:19 +00:00
Michal Niewrzal
51ba901737 satellite/metainfo: (Get|Download)Object returns MethodNotAllowed on delete marker
For S3 compatibility we should not allow returning pure delete
marker object. If metabase returns delete marker metainfo will
return MethodNotAllowed rpc status.

https://github.com/storj/storj/issues/6522

Change-Id: I89804b2bd22da0e5beec8f106e74b74733e19a52
2023-11-21 13:27:54 +00:00
Márton Elek
f2eca99bde satellite/repair: stat method to the repair queue to return with statistics
Change-Id: I2e07b116df9b282978a794423bd38803e2778755
2023-11-21 11:37:49 +00:00
JT Olio
2d8cca49b2 cmd/uplink: support writing monkit traces to file
Change-Id: I759d1687cf890e21cb260a5377c7153bfd0eb486
2023-11-20 22:07:00 +00:00
Clement Sam
b8b5afba73 satellite/satellitedb: remove dbx references to nodes.type column
Updates https://github.com/storj/storj/issues/5426

Change-Id: I36f12f354d9f2d575e4bd99b0520892f846bdc19
2023-11-20 18:08:37 +00:00
Márton Elek
0fdacfed8f satellite/durability: observer must reset between executions
Change-Id: I8f5b951beba513b219c4bb5680658f5e8b54538d
2023-11-20 16:56:43 +00:00
Jeremy Wharton
866780bcde web/satellite: fix papa parse worker plugin issue in watch mode
This change fixes an issue that caused the Papa Parse worker plugin to
break during Vite's watch mode.

Change-Id: I84d03d38e2d0d6fe4b7536706638640d3ed259a6
2023-11-20 16:07:51 +00:00
Vitalii
6d22ec21d0 web/satellite/vuetify-poc: buckets limit card for project dashboard
Show buckets limit card on project dashboard instead of coupon card if user is in paid tier or is not the owner of currently selected project.

Issue:
https://github.com/storj/storj/issues/6492

Change-Id: I033bbcced1b0fb77436607847cd9f10830330009
2023-11-20 14:26:51 +00:00
Clement Sam
f5164c78cd Revert "satellite/satellitedb: drop type column on nodes table"
This reverts commit 2b4f347c33.

Reason for revert: not a zero-downtime migration. We need to deploy the dbx changes before completely dropping the column.

Change-Id: Ic9215650242b7848d54f8a5f863a13bc18b60149
2023-11-20 13:36:44 +00:00
Wilfred Asomani
33fb21c8e0 satellite/admin: add endpoints to legal freeze/unfreeze users
This change adds two new admin endpoints to freeze users for legal
review and to remove them from that state

Issue: storj/storj-private#492

Change-Id: I6c8e3ffcb80375e81e78bc6ecc785c1047328cf7
2023-11-20 11:23:23 +00:00
Clement Sam
2b4f347c33 satellite/satellitedb: drop type column on nodes table
Updates https://github.com/storj/storj/issues/5426

Change-Id: If554f30124f234d1c50b2e8059e3d1dfec389ebb
2023-11-20 10:09:17 +00:00
Jeremy Wharton
0c591fa25a private/apigen: handle omitempty JSON option in structs
This change makes the TypeScript API code generator properly handle
struct fields with the "omitempty" option in the JSON struct tag.

Change-Id: I9b22ce33a8b8c39c115ec827a8e5b7e85d856f83
2023-11-18 07:09:42 -06:00
dlamarmorgan
cbc82690d7 satellite: implement metainfo get/set versioning endpoint
Change-Id: Ic43ea8419e7e7092dd191c8ed9f6e4eb4bdff20b
2023-11-18 01:26:34 +00:00
Egon Elbre
13d02d9d11 satellite/metabase: disallow moving delete markers
Change-Id: I5427ce2cf6783f83e77fcd8ca2fe7b98f78a945b
2023-11-17 21:56:33 +00:00
Michal Niewrzal
26a04a5929 satellite/metainfo: use bucket versioning state for listing
Currently we need to use different methods for listing objects
depends on versioning state.

Change-Id: I5747a699ba2dafcfc384216e4be0b662c8ae95fa
2023-11-17 21:07:58 +00:00
nerdatwork
bae3f7f73d
web/multinode: Show Storj tokens on Etherscan (#6483)
Currently the "View on Etherscan" link doesn't show Storj tokens on Etherscan like it should on MND (Multi Node Dashboard).
2023-11-17 17:57:53 +01:00
Ivan Fraixedes
bcde51138d private/apigen: Remove trailing space in doc generator
Remove a trailing space that the doc generator was adding.

Change-Id: I5c24478f79726afbaa5c6772d569b35cafe7c288
2023-11-17 11:44:50 +00:00
Ivan Fraixedes
ed67ce33bb
satellite/console: Clarify return params of interface method
Name the return parameters in a User interface method to clarify what
they are.

Change-Id: I6a5c97659c8167df89b38016fcccbc8eb70a09c5
2023-11-17 11:45:52 +01:00
Márton Elek
257bdbac32 satellitedb/repair: update placement when InsertBatch updates records
Change-Id: If974ff5d57abbe5bd16ce4cb6643d8a12314fe12
2023-11-16 18:55:18 +00:00
Vitalii
e474b9728c web/satellite: fix redundant redirect on first login
Not partnered users were redirected to onboarding pricing plan step and back to all projects dashboard in case when they have not finished onboarding. That caused tons of redundant API requests to be done.

Change-Id: I454a8cc29730943b84a351416b78c4281c47f6ca
2023-11-16 18:05:46 +00:00
Jeremy Wharton
1ea81c8887 satellite/admin/back-office: add endpoint to get placement info
This change adds an endpoint to the back office API that returns
placement IDs and their locations.

References #6503

Change-Id: I20ee1c82dcb647d6d264317beceeb5e70f7a8e87
2023-11-16 13:15:32 +00:00
Ivan Fraixedes
e67691d51c private/apigen: Remove support for anonymous types
I made several commits to add support to the API generator for
anonymous types because the example was using them.

Later, I was told that we don't need to support them, so the example
should have never used anonymous types.

Supporting anonymous types has added complexity to the API generator and
we are finding bugs in corner cases whose fixes lead to adding more
complexity and some fixes doesn't seem to have a neat solution without
doing a big refactoring.

We decided to reduce the side effects that supporting anonymous type has
brought just removing it.

This commit remove the support for anonymous types and reflect that in
the example.

Change-Id: I2f8c87a0db0e229971ab1bef46cca16fee924191
2023-11-16 12:29:21 +00:00
Ivan Fraixedes
418673f7a2 satellite/admin/back-office: Implement authorization
Implement the authorization that will hook into each endpoint handler
through a wrapping handler for defining the permissions that each
endpoint requires.

Change-Id: I9c8f12b58f48e849e7ea35f372dddce5c9cfc5b5
2023-11-16 11:37:55 +00:00
Igor
720b75ad73
docs/testplan: add object versioning testplan (#6444)
* docs/testplan: add object versioning testplan
2023-11-16 11:46:25 +01:00
Ivan Fraixedes
359c09b57f satellite/admin/back-office: Specify router path prefix
For convenience of not having to modify the API generator to contemplate
the path prefix that we are adding to the back office server, we define
the path prefix in a constant than the admin server and the definition
of the API uses to adapt the router and the generated code.

Change-Id: Ic557b0e6e88e930e03647835759bb34e06e8bb48
2023-11-15 20:28:52 -06:00
Jeremy Wharton
032faefa4b private/apigen: fix URL construction in generated TypeScript code
This change fixes an incorrect invocation of the URL object constructor
in generated TypeScript HTTP clients.

Change-Id: I9011bc535f2096374d20b74b401d4cc38a0451fb
2023-11-16 01:12:58 +00:00
Vitalii
cfb7f55220 satellite/console: update project usage-limits endpoint to also return buckets count/limit
Extended {projectID}/usage-limits endpoint to also return buckets count/limit.

Issue:
https://github.com/storj/storj/issues/6492

Change-Id: Ibc8956a2c10c9f383324e049f4b093410cbea899
2023-11-16 00:08:59 +00:00
Michal Niewrzal
6834c04539 satellite/metabase: drop pending objects table support
We decided that we won't use seprate table for handling pending
objects. We need to remove related code.

https://github.com/storj/storj/issues/6421

Change-Id: I442b0f58da75409f725e08e2cd83d29ed4f91ec6
2023-11-15 23:19:46 +00:00
Vitalii
cd9518a3c3 satellite/{accounting, console}: remove unnecessary fields from usage report
Removed MetadataSize, RepairEgress, AuditEgress fields from usage report because they are not relevant.

Issue:
https://github.com/storj/storj/issues/6498

Change-Id: I8ef7d56bd1b01cdb53cec8c67dd47a6a0e7fa184
2023-11-15 22:31:29 +00:00
Jeremy Wharton
a52934ef4d private/apigen: prevent self imports
This change prevents Go code produced by the API generator from
importing its own package. Previously, we tried to prevent self imports
by skipping import paths whose last segment matched the generated Go
code's package name. However, aliased imports circumvented this.
We now require API definitions to define the Go package path so that we
can compare this with the import path directly.

Change-Id: I7ae7ec5e1a342d2f76cd28ff72d4cd7285c2820a
2023-11-15 21:41:20 +00:00
Wilfred Asomani
24370964ab satellite/{console/payment}: wrap freeze code in transactions
This change wraps account freeze code in DB transactions to prevent
freeze inconsistencies resulting from errors that happen in the process
of freezing accounts.

Change-Id: Ib67fb30dc33248413d3057ceeac5c2f410f551d5
2023-11-15 20:40:54 +00:00
Egon Elbre
1105deba48 storagenode/blobstore/filestore: fix flaky TestTrashAndRestore
The result sorting was not correct causing occasional failures.

Fixes https://github.com/storj/storj/issues/6511

Change-Id: I09242846eda7c40198e7375ce218cb7026f9e7e4
2023-11-15 19:52:37 +00:00
dlamarmorgan
25262d62f3 satellite/metabase: add version tests to FinishMoveObject
Change-Id: I7603e0ff7819ad781a8ef5acb87e5efd9edc0b7f
2023-11-15 18:21:04 +00:00
Jeremy Wharton
f1ce0ab6d0 private/apigen/example: use hardcoded time
This change makes the example API definition use a hardcoded timestamp
rather than the current time so that files aren't unnecessarily
changed each time the API code is generated.

Change-Id: I72e4ec98d29345b9f16ca2ca38a91e593f83ea87
2023-11-15 16:52:18 +00:00
dlamarmorgan
2ed05314fc satellite/metabase: add tests FinishCopyObject
Change-Id: I94a6071c797857e8d5ba50d069500d8d1f10185f
2023-11-15 14:39:55 +00:00
Michal Niewrzal
90ed7ca070 satellite/metainfo: allow returning delete marker with GetObject
To be compatible with S3 we need to return 'Method Not Allowed' when
delete marker is requested and to do this libuplnk needs to know about
delete marker. It will be returned only if object version will be
specified.

Change-Id: I288da5566c74e1b4951f7cd249dbf34622b92e91
2023-11-15 12:18:38 +00:00
Michal Niewrzal
d2083281c9 satellite/metainfo: drop pending objects table support
We decided that we won't use seprate table for handling pending
objects. We need to remove related code.

After this change we still needs to remove related code from metabase.

https://github.com/storj/storj/issues/6421

Change-Id: Idaaa8ae08b80bab7114e316849994be171daf948
2023-11-15 10:55:08 +01:00
Jeremy Wharton
40d67065ba web/satellite: statically serve Papa Parse worker
Papa Parse, the library we use to parse CSV files in the satellite UI,
uses a blob URL for its worker. This isn't allowed by our content
security policy, so this change implements a Vite plugin that writes
the worker code to a file that is statically served.

Change-Id: I0ce58c37b86953a71b7433b789b72fbd8ede313d
2023-11-14 18:24:38 +00:00
JT Olio
9930b86791 go.mod: bump storj.io/common
Change-Id: Ifae770c2eb8dd398e3e049240afa14714c605cba
2023-11-14 16:58:51 +00:00
Ivan Fraixedes
e39f395cf1 satellite/admin/back-office: Use PathPrefix value
PathPrefix is the full path of the subrouter passed to the back office
server when it hooks into another server, in this case, the satellite
admin server.

PathPrefix allows to serve the static assets from the root of the
sub-router when the prefix is stripped before accessing them.

There was a bug where the PathPrefix weren't used and a hard-coded path
was used.

Test passed because the back-office server is hooked into the satellite
admin server with a subrouter with path `/back-office/` which matched
the hard-code value, however, it wouldn't work if that path changed or
it is hooked into another server with a different subrouter path,
despite it was set to PathPrefix.

This commit fixes that bug.

Change-Id: Id4a0d86329eb563b008b3fc6f8eb7b51cbfd2e6f
2023-11-14 16:05:04 +00:00
Vitalii
fe890ff535 web/satellite/vuetify-poc: add low token balance banner
Added low token balance banner to vuetify app with the same logic as in main app.

Issue:
https://github.com/storj/storj/issues/6460

Change-Id: Ifa9af8e2179ec3a6601b5053575990b86cc8f0b5
2023-11-14 16:43:47 +02:00
Wilfred Asomani
cd8e9bd044 satellite/{payment,console,analytics} extend freeze functionality for legal freeze
This change extends the account freeze functionality account for legal
freezes as well. This is a freeze event for accounts to be put on hold
for legal review. It also sets the LegalHold status on the affected
user.

Issue: storj/storj-private#492

Change-Id: I8c733269b5cfb647c840379a6bb033da120c8280
2023-11-14 11:09:25 +00:00
Igor
a053b210a5
scripts/draft-release.sh: ignore warning (#6496) 2023-11-13 14:20:24 +01:00
Wilfred Asomani
443a7b220e web/satellite: fix storj token card flexible height
This change fixes an issue where the storj token cards will extend
their height to match their parent, causing them to get taller when the
add card form is showing.

Change-Id: I22219c8e537bda6555c8c3bf813f9649b664639a
2023-11-13 10:53:27 +00:00
Wilfred Asomani
dc28dadbc2 satellite/{web,console}: add config for payment element
This change adds a config flag for whether the stripe payment element
should be used to collect card info.

Change-Id: I301cf69e6f1b64350266e8f2286542b951e216c4
2023-11-10 23:18:12 +00:00
Vitalii
7c7123a156 web/satellite/vuetify-poc: updated warning state copy for enter bucket dialog
Updated copy to be the same as in main app.

Change-Id: I856248db2965837449aa51eb6058f28268a93d5a
2023-11-10 20:08:07 +00:00
Michal Niewrzal
b96e7401c3 satellite/metainfo: add TestEnableBucketVersioning flag
Additional feature flag (onyly for testing) to set versioning enabled
for all new create buckets. We need it until we will have support
for enabling/disabling versioning for bucket on metainfo API.

In addition this change is fixing also two small issues which makes
testing this flag imposible:
* metabase Status list was not aligned with protobuf definition
* object retruned by metainfo API didn't have correct status set in some
cases

Change-Id: I0d63dff6a08efa588c8999af1e17db476943e067
2023-11-10 19:19:30 +00:00
Wilfred Asomani
e9fd430d01 storj: remove instances of DCS and OSP
This change replaces with "Storj" all instances of "Storj DCS" or
"Storj OSP".

Issues: storj/storj-private#479
#5069 and #5839

Change-Id: I78d13f2abd8bde5d2cb42d8006016ee87130a566
2023-11-10 18:29:48 +00:00
Michal Niewrzal
e592138c51 satellite/metainfo: align object version format with minio
For some reason minio sometimes validates versionID as UUID. Until
we decide what to do lets align our version format with it.

Change-Id: I6e9832d0adc1d3b6e3f46688b386e0e118219038
2023-11-10 16:52:19 +00:00
Vitalii
f2ccb910df web/satellite: fix low token balance banner's show condition
Fixed the condition when low token balance is shown.
The bug was related to account balance being returned in dollars instead of cents and this was not handled by 'sum' method.

Change-Id: I35f642302d877c9b1ed9f6f1d03d6091f2942a0b
2023-11-10 17:08:35 +02:00
Moby von Briesen
cdcd3b2687 satellite/console,web/satellite: add feature flag for limit increase req
This change adds a feature flag (default disabled) to enable the ability
to submit a project limit increase request directly from the UI. When
this feature is disabled, the user will be directed to a page to file a
support ticket.

Resolves https://github.com/storj/storj/issues/6480

Change-Id: I5f355dcb1a40e5b694f9623f05fe706ed4d6a528
2023-11-10 08:07:13 +00:00
Vitalii
147076a696 web/satellite/vuetify-poc: notifications for getting close to limits
Added notification banners on project dashboard when user is close to or reached some particular project limit.
Implementation is similar to main app notifications.

Issue:
https://github.com/storj/storj/issues/6459

Change-Id: Ifaf14facabd0b57f45431c874cfd6fcc1e991282
2023-11-10 02:31:25 +00:00
Wilfred Asomani
98234e76b7 satellite/{payment,console} add legal hold user status
This change introduces a new user status, LegalHold to be used to pause
an account while it is under legal review. It also modifies the
condition whether to generate invoices for a user to apply to this new
status.

Issue: storj/storj-private#492

Change-Id: Idc9a6166cc96178a95e45cf7c0f2e982631ca021
2023-11-09 23:42:26 +00:00
Wilfred Asomani
76594466ef web/satellite/vuetify-poc: add pagination to browser card view
This change adds pagination, sort and search to the object browser card
view.

Issue: #6427

Change-Id: I4cc42f3062a7cbd06d9f253e4864478f0b430fc8
2023-11-09 22:54:51 +00:00
Tome Boshevski
83ea3772e2 web/satellite/vuetify-poc: ui ux improvements
This change improves the satellite v2 user interface and overall
experience.

Change-Id: I702a6aa6b0b1a676bb2c404ab54f0ecef3b1ad7a
2023-11-09 20:25:34 +00:00
Tome Boshevski
07e2a9af06 web/satellite: update charts styling
This change updates the styling of the satellite UI dashboard charts
to prepare for V2.

Change-Id: Ic329963a6b825e3e4aaecab0696080a152a41c5f
2023-11-09 13:31:50 -06:00
paul cannon
c246d36adf satellite/metabase: tests for versioning in TestIterateObjectsWithStatus
Credit to the tests in TestListObjectsVersioned(), which (since
ListObjects is logically a pretty thin wrapper around
iterateAllVersionsWithStatus()) were able to be translated to tests in
TestIterateObjectsWithStatus with only minor changes.

Change-Id: I6e68675b24fee614e44baddf596703115554014e
2023-11-09 17:25:44 +00:00
Michal Niewrzal
c9430c22b5 satellite/metainfo: wire versioning with commit object operation
Change-Id: Ib19b5bade69dfb3f02256c638cc2f6281befa415
2023-11-09 11:21:58 +00:00
dlamarmorgan
539b32d01e satellite/metainfo: add UseBucketLevelObjectVersioning by project
This small feature flag will give us the ability to test object
versioning for specific projects without enabling it globally.

Change-Id: I78301f071b7b8079dd1bd4a561fce0800ce9f074
2023-11-09 11:21:06 +00:00
paul cannon
dc5ae6f4f6 satellite/gracefulexit: add test for GE through upgrade
This creates an automated test for the situation where a node initiates
graceful exit while TimeBased is off, and then TimeBased is turned on
before the node has completed graceful exit. The node should no longer
try to transfer any more pieces, but should instead sit and wait until
the graceful exit period has elapsed.

Change-Id: Iaf636f9247bc878bc20041221e1a8014c77806ad
2023-11-09 08:58:16 +00:00
Michal Niewrzal
3e1b108b84 satellite/metainfo: return correct version with ListObjects
ListObjects method will now return correct object version.

Change-Id: I663c506079b21f238026bd6ffac2384eae521635
2023-11-09 07:41:22 +00:00
Jeremy Wharton
404bddd2a4 cmd/storj-sim: increase satellite contact grace period
This change increases the grace period within which test satellites
are expected to respond after being instantiated. Our Jenkins build
pipeline for GitHub was failing because the grace period was too short.

Change-Id: I8bfd7c771660fc39f6eb988f95d706809f936d00
2023-11-08 22:46:41 +00:00
Moby von Briesen
b56a158100 satellite/analytics: Send signup captcha score to analytics
Send signup captcha score (if it exists) to Hubspot and Segment, so that
we can implement logic based on this in the analytics platforms.

Change-Id: Ic4e166639c9ab8e872a212e7466e82433c9ea6d4
2023-11-08 21:56:11 +00:00
Clement Sam
e27381f3af private/version: use minimum key in new sem version
Much evident on the storagenode dashboard, the minimum
version shown is a very old version and that is taken
from the deprecated part of version info from the
version control server, and we no longer update the
deprecated part on the server.

This change forces it to use the new sem version, and
checks for the old version if the server is probably
running an old version of the version control system.

Also fixes a bug where the suggested version returned
for all processes is taken from the storagenode part.

Issue: https://github.com/storj/storj/issues/5673

Change-Id: I57b7358c2826a6e25f441dfa9579a1aef50a7e89
2023-11-08 21:08:01 +00:00
JT Olio
98032f2b1f satellite/payments/stripe: another loop variable capture
Change-Id: I9ec7226befb8ebe18c7833b1c1d7ee8661bd9032
2023-11-08 20:43:54 +00:00
Márton Elek
7d8f0845fd
satellite/durability: calculate busfactor for classes
WARNING: THIS COMMIT IS ACCIDENTALLY MERGED WITH THE PREVIOUS ONE. BUT I KEEP THE COMMIT MESSAGE FOR IT’S VALUE. CODE IS IN THE PARENT

As Wikipedia defines, bus fator is "the minimum number of team members that have to suddenly
disappear from a project before the project stalls due to lack of knowledgeable or competent personnel."

We use similar definition, but instead of team members, we check the pieces of one segments.

For example, if we have a segment with 10 pieces in US, 8 pieces in DE and 3 pieces in other countries,
traditional bus factor (50) is 2, as we need two contry down (10 + 8) two loose at least 50% of the pieces (10.5).
Loosing just one contry, we will have <50% of th pieces.

In our implementation we are not interested about the 50% threshold, but a configurable value.

As a default we use the repairThreshold - minimalNumber.

With the usual defaults (54 - 29) it's 25. So bus factor can be defined sg. like this:

How many GROUP should disappear from the network to drop the healthy count from repair threshold to the minimum number.

This is critical, as we don't repair anything above the repair threshold, but heatlhy piece number can drop
if the given amount of groups (subnets, countries...) live the network...

Change-Id: I606f091469b45e90f3a9eb8fcff65a834ff27a14
2023-11-08 21:17:54 +01:00
Márton Elek
0ef3247d44 satellite/durability: make benchmark even quicker
To make sure that Benchmark tests are good, we run them with -short flag, eg:

```
go test -short -run=BenchmarkDurabilityProcess
```

Durability benchmark already supports this, but we can make it slightly more faster with
using less sgements and pieces during the `-short` run.

Change-Id: I9547ca1e3cd0178eb395a7a388f2e7936a9862d7
2023-11-08 19:00:30 +00:00
Ivan Fraixedes
100519321e
satellite/admin: Allow all operations through Oauth
Allow all the operations when accessing through Oauth, but requires the
authorization token for the ones that we consider that they are
sensitive.

Before these changes, a group of operations weren't available through
Oauth, and people who has access to the authorization token had to
forward the port of the server to their local in order to do them
without Oauth.

These changes shouldn't reduce the security because people who has
access to the authorization token is the same than they can forward the
port and part of those have Oauth access too.

Allowing to perform all the operations through Oauth will improve the
productivity of production owners because they will be able to do all
the administration requests without having to port forward the server.

Change-Id: I6d678abac9f48b9ba5a3c0679ca6b6650df323bb
2023-11-08 18:14:38 +01:00
Márton Elek
23c592adeb satellite/durability: use process level classID cache (instead fork level)
Classifier of durability is sg. like "net:1.3.4.1" or "country:HU".

To make the calculation faster we use arrays instead of maps, which means that we assign a uinique index to all of these strings (classes).

As Egon suggested earlier, we can do this mapping only once (per process), not for each fork.

Not a big deal performance-wise, as we have limited number of forks, which are initialized once per 5-10 hours, but the code is more readable and clean.

Change-Id: Id081846b5d97dae8009aeeecbcc63cb713bed294
2023-11-08 15:22:48 +00:00
Vitalii
b6e4f4a02d web/satellite: update locked objects wording
Updated locked objects wording to reflect situations when objects were recently deleted.

Change-Id: I9c4e3ca290cce5e211745094b866d568078d4317
2023-11-08 13:09:42 +00:00
paul cannon
a712ee94aa satellite/gracefulexit: suspended nodes fail graceful exit
QA discovered that if your node lost all its data but your audit score
was still healthy, you could call graceful exit, respond with unknown
audit errors when audited, and eventually get suspended, but still get
your held amount back.

We will therefore mark suspended nodes as having failed graceful exit.

Refs: https://github.com/storj/storj-private/issues/485
Change-Id: Id5af18786b574651587cc96bd6a7d0b47c0671a8
2023-11-08 11:26:27 +00:00
paul cannon
084719cde1 satellite/gracefulexit: de-flake TestNodeFailingGracefulExitWithLowOnlineScore
I can't say with certainty yet what caused the two failures I know
about, but I have one theory: the node continuing to check in during the
test skewed the online score towards 1, and using the test default for
GracefulExitDurationInDays meant there were fewer update periods than
expected.

At any rate, it is more correct to pause the graceful exit processing
chore and the contact chore during the test, even if it doesn't end up
solving the problem.

Refs: https://github.com/storj/storj/issues/6401
Change-Id: I06d43d531e0b3344af13878c8d55213349fdcfa3
2023-11-08 10:16:05 +00:00
JT Olio
00cb237a90 satellite/payments/stripe: loop variable capture
Change-Id: Ia2e0cf329f2ff46394b21bad9581e430e0202ead
2023-11-07 19:29:51 -05:00
Ivan Fraixedes
ae945b993a
satellite/admin: Create separate server for new back-office
Create a separate server for implementing the new satellite
administration web app.

This server is in a new package that will implement all the
functionality for the new satellite administration back-end and when it
be completed with all the functionality that the current one offer, it
will replace it.

For now, the new server only exposes the static assets as they were
exposed by the current server.

A main sub-package is added with an example endpoint to scaffold where
we'll define the API through the API generator and to locate the several
generated files.

Change-Id: I172c43b2c180553876ef7ce137cc778b94723451
2023-11-07 15:31:01 +01:00
Wilfred Asomani
14b83bb390 web/satellite/vuetify-poc: add login functionality
This change implements login functionality in the vuetify app.

Issue: #6470

Change-Id: I6079888af14ded6d4886b3fc16108ca410f52982
2023-11-07 10:08:56 +00:00
paul cannon
fd55dad735 storagenode/retain: don't quit on error
It has been noted in the forum that, during a Retain operation, when a
piece can't be deleted, the process never completes. The error is
written to the log, but the completion line "Moved pieces to trash
during retain" never is.

This `return` line is the reason. We should instead continue the loop.

Change-Id: I0f51d34aba0e81ad60a75802069b42dc135ad907
Refs: https://github.com/storj/storj/issues/6482
2023-11-06 10:54:46 -06:00
Márton Elek
015cb94909
satellite/durability: add exemplar and report time to the reported results
Exemplars are representative elements for the stat. For example if a stat min is `30`, we can save one example with that value.

More details about the concept is here: https://grafana.com/docs/grafana/latest/fundamentals/exemplars/

In our context, which save the segment + position in case of min is updated, to make it easier to look after the segment in danger.

Change-Id: I19be482f1ddc7f1711e722c7b17480366d2c8312
2023-11-06 13:22:28 +01:00
Vitalii
86decb1f44 web/satellite/vuetify-poc: add info tooltip for download graph
Added info tooltip for download graph on project dashboard.

Issue:
https://github.com/storj/storj/issues/6461

Change-Id: If405268166fdd529415d7820e4bc87d5f8961ade
2023-11-06 11:25:19 +00:00
dlamarmorgan
d0f4447427
satellite/metabase: add version tests to IteratePendingObjectsByKey
Change-Id: I7335598eab2bac7f12aa93773cceadb195cd8e9e
2023-11-03 12:47:05 -07:00
Wilfred Asomani
81506203c4 web/satellite/vuetify-poc: add image preview to browser card view
This change adds image preview to the browser card view.

Issue: #6427

Change-Id: Iab8110fb3fa70fea29a98d8f96bac9b357dd401d
2023-11-03 18:04:18 +00:00
Wilfred Asomani
52b3ffd8c3 web/satellite/vuetify-poc: add authentication pages
This change copies over the new authentication UIs from the static
vuetify repository.

Change-Id: I06ea7dabbde2a091f524fbcdc8cb1eff08c90a74
2023-11-03 16:37:03 +00:00
Ivan Fraixedes
48877c05cc
private/apigen: Fix TS generator to align with linter
I introduced some subtle linter issues when I added the APIError class
and added the TypeScript mock generator.

This commit addresses them, so the linter doesn't yell about the
TypeScript generated sources.

Change-Id: Icc7dfa4169a228b1a5144d4a292f4350ee5ef9f0
2023-11-03 16:41:54 +01:00
Vitalii
f6e357be52 satellite/{console, web}: detailed usage report for a single project
Reworked usage report endpoint to return CSV for a single OR all the project user owns.
Added buttons to download usage report CSV for a single project.

Issue:
https://github.com/storj/storj/issues/6154

Change-Id: I55104088180dcf6be49dcde6c9c495f07ba01c5a
2023-11-03 14:58:18 +02:00
Michal Niewrzal
f0f73fc8ae satellite/metainfo: support downloading specific object version
Protobuf definition is ready to support downloading specific version of
object so we just need to wire requested version into metainfo
DownloadObject endpoint.

https://github.com/storj/storj/issues/6221

Change-Id: I3ddc173beb6a6cf30d782dd65c6aa5f88f2cbd44
2023-11-03 11:58:12 +00:00
Jeremy Wharton
51fefb2882 web/satellite: statically serve Vuetify theme styles
Vuetify's way of applying themes uses an inline stylesheet. This is
incompatible with our CSP policy, so this change implements a Vite
plugin that writes theme styles to CSS files that are statically
served.

Change-Id: I73e3a032435e46d41248c5181e913a8e04f65881
2023-11-02 23:36:15 +00:00
Ivan Fraixedes
74757ffc1d private/apigen/example: Fix client TS generated file
We introduced some changes in 882c9d64e4
and one reviewer spot an issue which was fixed, but I forgot to
regenerate the files and the generated Typescript client got stale.

This commit regenerate the example code, so the generated code is align
with the current generator version.

Change-Id: Idc023adc9c6e230578f9e77e492126355b006203
2023-11-02 14:24:30 +00:00
Wilfred Asomani
febd2091df web/satellite/vuetify-poc: add browser card view
This change adds an optional card view to the file browser similar to
the all projects card view.

Issue: #6427

Change-Id: I115dea7fdc2e7d0e093a00eb88e46d453c516cd9
2023-11-02 10:46:17 +00:00
Jeremy Wharton
e482e1296e web/satellite/vuetify-poc: fix file preview
This change fixes an issue where clicking a file in the Vuetify UI's
object browser would open an empty file preview dialog.

Change-Id: I5e21cf2e8c57911ac2708110f6ad23f376f86a54
2023-11-02 02:17:42 +00:00
paul cannon
51c930f532 satellite/repair: extra logging during TestSegmentRepairPlacement
In case we continue to see flaky TestSegmentRepairPlacement, this may
help narrow down the issue.

Change-Id: I34ca70e5bb33eca26e9940e845142121cc946ac0
2023-11-01 20:43:18 +00:00
paul cannon
b621a90745 satellite/repair: promote some Debug log lines to Info
We have frequently wanted to see this sort of information when looking
into issues. Considering that we already log a line at Info every time
we fail to download a piece during repair, it doesn't seem like a very
onerous new burden to log one line per segment saying that it is being
repaired.

Change-Id: I1fa84a985e90473adeb02603e207fc3c7b8592da
2023-11-01 19:52:51 +00:00
JT Olio
e81b578606 go.mod: bump storj.io/uplink
Change-Id: I7e63e57b8b582a5bce47c56bbc68fdd8f5f7aa48
2023-11-01 20:43:45 +02:00
Egon Elbre
9f39684799 cmd/storagenode: skip forget-satellite tests
Change-Id: I630bba8690e48f219d23a966796849a97657a49a
2023-11-01 17:24:44 +00:00
Egon Elbre
9e0fff5e22 satellite/gcbf: fix data race in TestGCBFUseRangedLoop
Satellites[0].GCBF is already started when testplanet boots up,
so calling Run separately ends up causing a data race.

Instead create a new instance, that should avoid this issue.

Fixes https://github.com/storj/storj/issues/6435

Change-Id: I6603ef63da7a6ab8bdb952cf5aaca17eb0392e2c
2023-11-01 19:07:20 +02:00
Egon Elbre
0a4a1098d8 go.mod: bump common (and maybe fix quic)
Change-Id: Ib37213ab6d26d2b6adc727e913e3df3e12e9ee9d
2023-11-01 14:42:15 +02:00
Egon Elbre
38ad5a1318 private/testplanet: add STORJ_TEST_DISABLEQUIC environment flag
Currently Windows seems to dislike creating udp servers in a very
fast fashion. Add an environment flag that allows to disable quic,
which is the main culprit.

Individual tests can still override that setting.

Change-Id: I3b30c4aa7fcb148b2894335394fdfae6eaa372bb
2023-11-01 11:39:03 +00:00
Wilfred Asomani
b3e908f72f satellite/{console,web}: add browser card view config flag
This change adds config flag to toggle whether the browser card view
can be used.

Issue: #6427

Change-Id: I19986ffe9a778b8752549a4d54b318fff0386479
2023-10-31 20:19:57 +00:00
Egon Elbre
f2eb6942c5 satellite/metabase: make the precommit constraint code nicer
This condenses the precommit constraint into a single function, which
allows to cleanup a bunch of logic elsewhere. Also, avoid delete in the
first place when we are not allowed to remove the uncommited object.

This also fixes submitting the metrics and now it submits the correct
metrics.

Change-Id: If91dfa3b19bce5b24ff2a19d7c34b57a200db1dd
2023-10-31 18:10:09 +00:00
Egon Elbre
e5e55ef266 satellite/metabase: move delete object unversioned code
Change-Id: Iec92c91c174b990fa44b210db415da7a4eb650c4
2023-10-31 17:14:16 +00:00
Egon Elbre
46f7e5b7fa all: don't depend on infectious library directly
Change-Id: I0bcc719cc25e1d7ad36a16f65d40220eccba6556
2023-10-31 16:00:47 +02:00
dlamarmorgan
77cac8b465 satellite/metabase: add tests ListObjects
Change-Id: Ia84b560251245fe0050426537751958f2c85e9e1
2023-10-30 20:33:08 +00:00
Wilfred Asomani
513c3cc632 satellite/admin: list users pending deletion
This change adds an endpoint to the admin API and UI to get a list of
users pending deletion and have no unpaid invoice.

Issue: #6410

Change-Id: I906dbf9eee9e7469e45f0c622a891867bf0cc201
2023-10-30 19:11:16 +00:00
Jeremy Wharton
405491e8d0 web/satellite{/vuetify-poc}: implement CSV file previewing
This change allows .csv files to be previewed in the file browser.

Resolves #6426

Change-Id: Ib93ebb417b8f69231ed2b36b8258ad91c6a1ba4b
2023-10-30 17:35:48 +00:00
Jeremy Wharton
42e1b088c2 satellite/admin/back-office/ui: use composition API
This change migrates Vue components of the new satellite admin web app
from the options API to the composition API.

Change-Id: Ie8c9bcd468f1c0fe0abc9ef0a3724563db096ba9
2023-10-30 16:50:30 +00:00
Wilfred Asomani
8f59535f95 web/satellite: use stripe payment element
This change uses the recommended stripe payment element to collect card
information instead of the legacy card element currently in use.

Issue: #6436

Change-Id: If931d47430940e0932c845b6eee3e0e23c294fbb
2023-10-30 15:37:30 +00:00
Michal Niewrzal
67f32bd519 satellite/metainfo: use protbuf ObjectVersion with GetObject request
We need to update metainfo GetObject endpoint to use ObjectVersion
([]byte) field instead of old Version field (int32).

Change-Id: I61663ec8d9f5c731f91346a285048477fb493730
2023-10-30 14:49:18 +00:00
Wilfred Asomani
f7a95e0077 satellite/{payment,console}: add endpoint to add card by pmID
This change introduces a new endpoint that allows adding credit cards
by payment method ID (pmID). The payment method would've already been
created by the frontend using the stripe payment element for example.

Issue: #6436

Change-Id: If9a3f4c98171e36623607968d1a12f29fa7627e9
2023-10-30 13:58:55 +00:00
Vitalii
32e67e5fab web/satellite: update low token balance banner copy
Updated the copy for low token balance to correspond with a new use-case.

Change-Id: I1e08a039849c2950cdc4b8dc3b4c9f470461b0dd
2023-10-30 12:49:10 +02:00
Egon Elbre
1fa918c255 satellite/metabase: simplify UpdateObjectLastCommitted
Change-Id: Ifec596ab1868baa03688e717adec7d2ab45eafa9
2023-10-28 04:43:43 +00:00
Egon Elbre
55bddb6ce1 satellite/metabase: don't use empty stream id for delete markers
Using a an empty stream id makes it more difficult to target a specific
delete marker. Similarly, we don't want to confuse actual stream id-s
with normal ones. So, we'll create stream id-s where the first few bytes
are 0xFF, but the rest is random.

Change-Id: Ia7fffb0da9a071be2935df99c0846027ee2e03c3
2023-10-28 04:43:32 +00:00
dlamarmorgan
aeec4bd213 satellite/metabase: add tests to UpdateMetadata for versioning
Change-Id: I4503f92b19212e7c4cf1907edd7e529ff75cca3e
2023-10-28 04:43:04 +00:00
paul cannon
8be7414a52 cmd/tools/segment-verify: fix global race in TestCommandLineTool.
This should stop the failures when TestCommandLineTool is run with
parallel test execution enabled.

Change-Id: Id80d5eacb78fcec886be786ae8b182517b17fbc6
2023-10-27 19:51:24 +00:00
Ethan Adams
e65345336f satellite/placement: Add test for new "datacenter" placement tag
Change-Id: Ic9a79bb46256fbe8b1159de2bbf787b4a63c5c8f
2023-10-27 18:15:32 +00:00
Wilfred Asomani
667b7c6f79 web/satellite/vuetify-poc: fix file preview
This change fixes an issue where clicking on a file from the upload
snackbar would not preview it.

Issue: #6433

Change-Id: I30b0b6098e058d1069b87f99425fac5d6e421fea
2023-10-27 17:18:52 +00:00
Vitalii
539253f646 web/satellite: don't make billing API requests if billing is disabled
Removed billing API calls on all project dashboard if billing is disabled.

Change-Id: Ia3407c811e599f40670b97e1c2ad9a8f661c0e2a
2023-10-27 18:53:14 +03:00
Jeremy Wharton
4f07994bfb Jenkinsfile.premerge: style check satellite admin back office files
This change runs our style checker for assets related to the new
satellite admin web app in the pre-merge build. Additionally, the
testing and style checking for web assets has been parallelized.

Change-Id: I0835673435f7ed8a4704c9373fcf5bb8eb5e7d0a
2023-10-27 09:39:26 -05:00
Jeremy Wharton
8ebf285081 satellite/admin/back-office/ui: lint files
This change lints files for the new satellite admin web app. An
ESLint config has been added for it that is identical to the one used
for linting satellite web app files.

Change-Id: I66f72fb880d5cbc80b6c080294e4a830b3d28143
2023-10-27 11:23:58 +00:00
Egon Elbre
27724835da satellite/metabase: use database order in ObjectStream.Less
Change-Id: I3d5e00263d78d9128fae6d39a3bc7f0c542039b6
2023-10-27 07:38:51 +00:00
Egon Elbre
a4edbdd005 satellite/metabase: use commit order for objects
This changes metabase behavior such that the latest object
will be the committed object which may result in a new version
assigned to the objects.

Change-Id: I7a731d7db4696caba75fff65107a248569b6111f
2023-10-27 07:38:25 +00:00
Jeremy Wharton
eff1719977 satellite/admin/back-office/ui: speed up build command
This change speeds up the build of the new satellite admin web app
using the same method that 8b0d25c used to speed up the build of the
new satellite web app.

New build time:   23.394s
Old build time: 1m40.930s

Change-Id: Ic6fcfdfc73b78bc26bddc421c608076b23532967
2023-10-26 23:47:18 -05:00
Cameron
1aadc0974d web/satellite: project limit increase request
create modal to allow pro users to request project limit increase when
trying to create a project if they have reached the project limit.

github issue: https://github.com/storj/storj/issues/6298

Change-Id: I1799028e742c55197fa5d944c242053cf4dc3a2c
2023-10-26 14:24:52 -04:00
Egon Elbre
c8e4f0099c satellite/metabase: add Prefix status for non-recursive listing
Currently it's awkward to use any of the existing statuses for the
objects in non-recursive listing. Hence, let's add a new one.

Change-Id: I8485e0f858e69998b097e757091991538ca697fa
2023-10-26 15:37:24 +00:00
Vitalii
8b6c60f26b cmd/satellite: copy wasm files into app/static/wasm subfolder
Fixed docker command to copy files to app/static/wasm subfolder instead of app/static itself.

Change-Id: I50e605b9224d2c643cd5f6ae62a3610ed2a06752
2023-10-26 14:12:35 +00:00
Michal Niewrzal
20dfcedd97 satellite/metainfo: return full object on commit
We never extended metainfo protocol to return committed object
detailed into and this change is doing it now. Main motivation to
do this now is need for providing object version after upload.

Change-Id: Ib59bdfd9485e4a0091ac02458cc63427cb7159de
2023-10-26 09:46:18 +00:00
Michal Niewrzal
f5d717735b satellite/repair: fix checker and repairer logging
This is fixing two small issues with logging:
* repair checker was logging empty last net values as clumped pieces
but in main logic we stopped classifying it this way
* repairer summary log was showing incorrect number of pieces removed
from segment because list contains duplicated entries

There was no real issue here.

Change-Id: Ifc6a83637d621d628598200cad00cce44fa4cbf9
2023-10-25 22:55:53 +00:00
Wilfred Asomani
3ba452b301 web/satellite: fix signup logo alignment
This change fixes the alignment of the Storj logo on the registration
page on Safari.

Issue: #6429

Change-Id: I70a5c28a939c6a8975eb23ea1f409643b6da334a
2023-10-25 22:13:52 +00:00
Jeremy Wharton
ff16d2fa02 web/satellite: implement .txt preview
This change allows .txt files to be previewed in the legacy UI's
file browser.

References #6426

Change-Id: If0267695f07e6ea1738377527827c1e386fb668f
2023-10-25 21:32:22 +00:00
Vitalii
4ba2703783 satellite/{console, web}: remove AllProjectDashboard feature flag
Removed AllProjectDashboard feature flag.
Removed unused Vue components.
Fixed wrong redirect on reload if pricing packages are disabled.
Fixed wrong redirect on reload if billing features are enabled.

Issue:
https://github.com/storj/storj/issues/6434

Change-Id: I9081a6f737c45fb48da5b23c016a42e23021c4ce
2023-10-25 20:49:47 +00:00
Jeremy Wharton
f319af5a35 web/satellite/vuetify-poc: implement .txt preview
This change allows .txt files to be previewed in the Vuetify UI's
file browser.

References #6426

Change-Id: Ib84ca562d47a413af17890af160542da65425016
2023-10-25 20:06:26 +00:00
Vitalii
40e43826a9 web/satellite: extend low token balance banner use-case
Updated condition on when this banner should be shown.
Also, added this banner to project dashboard and billing pages.

Issue:
https://github.com/storj/storj/issues/6356
https://github.com/storj/storj/issues/6368

Change-Id: I2f8f587a3c75508df0a9a6e84e1684b3c3904aa7
2023-10-25 19:14:14 +00:00
Michal Niewrzal
4cbdc0342a cmd/tools/segment-verify: add CreatedBefore and CreatedAfter flags
We would like to verify only segments from specific periods of time.

https://github.com/storj/storj/issues/6431

Change-Id: I42610962022bdf6ee36815fe1c157d67792147b8
2023-10-25 18:06:46 +00:00
Márton Elek
bca46736bb satellite/contact: send evenkit entry only with existing fields
Change-Id: Ifdd6c14de5f99c3c9bb8b1e9e2dce8c1e1c3b1ed
2023-10-25 16:16:20 +00:00
Michal Niewrzal
6c2e66fa9e satellite/metabase: small addition to GetObject* tests
Change-Id: I057a432750bbe12b30127c91592747a86d7ca23c
2023-10-25 17:48:30 +03:00
Egon Elbre
504d5c5651 satellite/metabase: add version tests for GetLatestObjectLastSegment
Change-Id: Ia7ed1f6b23bcdc9e83fec288cbf3571b382d5e13
2023-10-25 17:39:57 +03:00
Egon Elbre
a7e1378f89 satellite/metabase: return more information from delete last committed
Change-Id: I2626a100e0c3c41631c9a29b0bf5a7afccc60957
2023-10-25 17:38:57 +03:00
Egon Elbre
97c98d72e4 satellite/metabase: adjust code for iteration
Change-Id: Id3d4efe228a6f2d3642a639ef66a30e178ca001a
2023-10-25 13:44:36 +00:00
Michal Niewrzal
988ebbaf8d satellite/metainfo: support deleting specific object version
Protobuf definition is ready to support deleting specific version of
object so we just need to wire requested version into metainfo
BeginDeleteObject endpoint.

Dependencies bumped to get latest metainfo protobuf definition.

https://github.com/storj/storj/issues/6221

Change-Id: Ifc3cc0b49d9acdf4f7e57e7184b0f2ae045f9113
2023-10-25 12:47:31 +00:00
Ivan Fraixedes
9338f3f088
private/apigen: Add a TypeScript client mock generator
Implement a TypeScript client mock generator to generate mocks with
static data specified in the API definition.

Change-Id: I11419f4bbf72576fcd829f9d4acd8471034ca571
2023-10-25 14:43:01 +02:00
Ivan Fraixedes
882c9d64e4
private/apigen: Create class for API errors (TypeScript)
Create a class that inherits the Javascript Error class to use it when
throwing errors on the TypeScript client when the ok property of the
Fetch API Response object is false, which means that the server
responded with a status code less than 200 or greater than 299.

The APIError class contains apart of the message, the response status
code which may be useful for the caller to decide what to do, for
example, it may decide to retry the call on certain response status
errors.

Change-Id: Ic48466b40cbf134a27d5c92a4af9f93232e84fca
2023-10-25 14:43:00 +02:00
paul cannon
39c2bb9e4b cmd/tools/segment-verify: add test for whole command line tool
Clarify and expand some tests, and add a large test that tries to cover
the whole segment-verify stack.

Trying to reproduce a problem we saw in production where the Found count
in the output csv increases with each entry, eventually reaching the
thousands, instead of representing the actual number of pieces found for
each entry.

Change-Id: I65e342fad7dc1c350830fd0c8ce75a87a01d495c
2023-10-25 10:21:53 +00:00
Egon Elbre
6939c7ec25 satellite/metabase: simplify delete code
Change-Id: Iaa89421423bce836bfb23ccd17adbf90b79599ef
2023-10-25 10:17:04 +00:00
Egon Elbre
ca9ab74df8 satellite/metabase/metabasetest: condense creation logic
Change-Id: I7f82c34fe5aab2a49e4c8feeacddc8fa0f8809e5
2023-10-25 10:14:47 +00:00
Egon Elbre
25c4e4eec1 satellite/metabase: move CollectBucketTallies
Change-Id: Ia7a4bac91b02c006513f3cf9b9266053d60e90e4
2023-10-25 10:13:43 +00:00
Egon Elbre
d98498d17f satellite/metabase: adjust UpdateMetadata for versioning
Change-Id: Id18908c28384696bf234a8f879c5b5e550ddb8cf
2023-10-25 09:11:05 +00:00
Egon Elbre
080f58acfe satellite/metabase: adjust FinishMoveObject for versioning
Change-Id: Ib63c0d953f7b9f52a456b275f61cac166a93de12
2023-10-25 08:03:01 +00:00
Egon Elbre
a1a8c258d5 satellite/metabase: adjust FinishCopyObject for versioning
Change-Id: Ia2089238040624937b208c142d118dcbf5aa3432
2023-10-25 00:00:40 +00:00
Egon Elbre
97ac27942c satellite/metabase: fix flaky TestIteratePendingObjects
A randomly generated UUID can hit something that starts with `00`.

Also fix a tiny mistake in the comment.

Change-Id: I25a8b21e0f9523bc486e5a38b0c3cc9c36515231
2023-10-24 23:18:59 +00:00
Egon Elbre
7239a99505 satellite/metabase: make queries clearer
Use `(a, b, c) = ($1, $2, $3)` for the object stream location
arguments rather than combining multiple AND queries together.
Using a tuple comparison is shorter and also easier to see which
object is selected.

Change-Id: Iba84b89630d57255023c30e309eb6afaee9ab944
2023-10-24 22:36:32 +00:00
Egon Elbre
7de1178836 satellite/metabase: adjust ListObjects
Change-Id: Id435388cd1447cacccff7897c4b0d1a58cd67591
2023-10-24 21:54:46 +00:00
Vitalii
c79629e4da web/satellite/vuetify-poc: add keyboard controls for gallery view
Added mappings for left/right arrow keys and escape in gallery view.

Issue:
https://github.com/storj/storj/issues/6424

Change-Id: I995060dcee6a3c4b3f05f28415c81f83f6fe89c3
2023-10-24 19:24:31 +00:00
dlamarmorgan
fe9afad8cd
satellite/{buckets,satellitedb}: add versioning to buckets.DB
Add the DB schema changes and basic implementation methods to support
bucket level configuration of object versioning.

See bucket-level-versioning-configuration design doc for more details.

Change-Id: I4e920a20a403b3157970a34eb619d827a4007845
2023-10-24 11:16:31 -07:00
Márton Elek
4bf5d75a8a satellite/overlay: dedicated error type for placement parsing
Change-Id: Ifd78341bf4760db784b20bf80cf22018088873c7
2023-10-24 11:41:19 +00:00
Márton Elek
d64e2167ce satellite: do not validate smtp from / host when emails are disabled
Today we got the following error address, even if: `mail.auth-type: nomail`

```
Error: SMTP server address '' couldn't be parsed: missing port in address
```

It's make more sense to print the error message only if the SMTP address (or from address) is expected.

Change-Id: Ie07620099b6aac27630fcfd1cda9921ef4b6060c
2023-10-24 10:59:49 +00:00
Egon Elbre
1891f6501d satellite/metabase: make name clearer in iterator
Change-Id: I49f1539f23aa6fa05a02467e4cfa166cbf4e8968
2023-10-23 23:22:35 +00:00
Egon Elbre
0c7ad88857 satellite/metabase: adjust other Get queries
Change-Id: I6445328ad03e0b8cbcbad33b76d9ef30c41a804e
2023-10-23 22:40:03 +00:00
Egon Elbre
ff9013b8ab satellite/metabase: adjust some Get queries
Change-Id: I758ac42ce0a388c5a71a1b45502286800e3b07b8
2023-10-23 21:57:47 +00:00
Egon Elbre
aed664a78d satellite/metabase: adjust DeleteObjectLastCommitted for versioning
Change-Id: I7545af21fb098867ec43ba331ea6eaf9073db4b3
2023-10-23 21:15:49 +00:00
Wilfred Asomani
0f538093af satellite/{accountfreeze,console}: use days till escalation values
This change updates account freeze to set and use the days till
escalation column of the account freezes table.

Issue: #6382

Change-Id: I345798e3d53e5ab4a7653723433fb8affa258212
2023-10-23 20:34:07 +00:00
JT Olio
e469ee6cce overlaycache: don't return weird responses when certified nodes are disqualified
Change-Id: Ic63c65c4cb5637f966520ef996018e55c27d1f4f
2023-10-23 17:01:33 +00:00
Tome Boshevski
998babcfae
web/satellite: ui improvements (#6418)
Responsiveness and styling ui updates related to signup, login, forgot password pages, and common components.
2023-10-23 18:23:26 +02:00
dlamarmorgan
a014af45eb satellite/metabase: add tests CommitObjectWithSegments.Versioned
Change-Id: Ied838e6bee4e2d8254708d21c90edb68bdf05916
2023-10-23 11:48:33 +00:00
Egon Elbre
d3429fafd0 satellite/metabase: add CommitObjectWithSegments.Versioned
Change-Id: I4e1fc82b45cca94ff0b48d5b8d9deb6e13d0957b
2023-10-23 14:46:56 +03:00
Jeremy Wharton
1d5ea2d35c web/satellite: allow free users to invite if billing features disabled
This change ensures that the user is never prompted to upgrade when
attempting to invite project members if billing features are configured
to be disabled.

Change-Id: I1c49351b00c0e378da24ad080fd1d3b078c97c71
2023-10-20 21:49:36 +00:00
Jeremy Wharton
a6222afdd0 satellite/console,web/satellite: configure whether free user can invite
This change adds a flag to the satellite config indicating whether
free tier users should be able to send project invitations.

Change-Id: I9c030c88dbef136ba4a9bf2d8f027a8dcd77fd33
2023-10-20 21:06:19 +00:00
Moby von Briesen
e5fd061e70 satellite/satellitedb: Drop value_attributions.partner_id
Removes final instance of legacy "partner ID" column.

Resolves https://github.com/storj/storj/issues/5432

Change-Id: I89657c8f9f366c79a5439632f870a47db590ff99
2023-10-20 20:25:13 +00:00
Jeremy Wharton
405c46d0f6 web/satellite/vuetify-poc: add page loading bar
This change adds a loading bar in the Vuetify UI's app bar that is
active whenever a page is being loaded.

Resolves #6412

Change-Id: I0a4e148d843c7a3bace63410e8802fde3c7e92eb
2023-10-20 18:49:23 +00:00
paul cannon
b2c6ec0091 satellite/metabase: change versions column to INT8
This will allow further progress towards S3-compatible object
versioning.

Refs: https://github.com/storj/storj/issues/6352
Change-Id: I0b3aa93fcacd1f9d91a667d619d6cb41fba602a9
2023-10-20 17:42:23 +00:00
Clement Sam
67bd36ae4d satellite/satellitedb: remove nodes.type from indices on nodes table
Since we no longer need the type column and hopefully,
there are no queries using it, we recreate all indices
that reference the nodes.type column.

Updates https://github.com/storj/storj/issues/5426

Change-Id: I610ccaf474a6f4031e166b79a6d649c4b138e338
2023-10-20 14:48:36 +00:00
Vitalii
979675374b cmd/satellite: copy full wasm folder instead of explicit files
Copy full wasm folder to /app/static/ instead of explicit files because access.wasm file name is dynamic now (includes hash).

Change-Id: I5e8cebccdf5afdf024174b6bdc169fc4923b9ed1
2023-10-20 15:19:22 +03:00
Egon Elbre
74e4368cf1 satellite/metabase: add tests CommitObject.Versioned
Change-Id: Ie26e8806cfea0d0ba6ce58d290d329e04754f20a
2023-10-20 00:02:00 +00:00
Egon Elbre
a23d9d20aa satellite/metabase: add CommitObject.Versioned
This allows to commit versioned objects.

Change-Id: I7ae100e508a23899392ba40084198617fe3e4e0c
2023-10-19 23:20:48 +00:00
Márton Elek
be5302d9cc segment-verify: synchronize refreshing of the nodeAliasMap
This part can be called from multiple goroutines, therefore we should bw prepared for concurrent run.

Change-Id: I7acf1a29bdb51427d3d03f501b58b190dcf08412
2023-10-19 21:01:56 +00:00
Márton Elek
c031b2ad75 segment-verify: stat of reused segment should be set to 0
There are 3 different ways to execute segment verify.

When the bucket based segment list is used, the code tries to reuse Segments objects.

But without resetting the stat, it will create bad results.

(This is not the case of the other type of runs, as there we create arrays in each loop)

Change-Id: Ie2d52c7e44088a85d4a3ce541da1c5ff767591d6
2023-10-19 20:21:51 +00:00
Jeremy Wharton
0c1306780e satellite/payments/accountfreeze: fix race in test
This change fixes data races in tests for the account freeze chore
caused by the chore service being set while in use.

Change-Id: Ibcb94a947c405314deae0d77419e69b768feab40
2023-10-19 19:41:39 +00:00
Clement Sam
ab57cd30b0 satellite: remove all code references to nodes.type column
Once this is merged and deployed, we can go ahead
and change the db schema.

Note that this will stop updating the `type` column in the nodes
table but it shouldn't be a problem because this column has a
default value set.

Updates https://github.com/storj/storj/issues/5426

Change-Id: I2470ebacbcb0e60cf894617eb69b593227357283
2023-10-19 19:00:33 +00:00
Wilfred Asomani
be025a40cc satellite/{db,console}: add days column to account freeze table
This change adds a new column to the account_freeze_events table,
days_till_escalation, which is the number of days till the freeze event
is escalated. E.g.: 30 days from billing warning event to billing
freeze event.

Issue: #6382

Change-Id: I15cfd6f1208e641d8c380bef61717031befdad73
2023-10-19 18:18:44 +00:00
Vitalii
b0a52f4b51 web/satellite/vuetify-poc: enable/disable billing features depending on config value
Added client side logic to disable billing features depending on config value.

Issue:
https://github.com/storj/storj-private/issues/464

Change-Id: I80ead8c91a39a387a1651efc700ca2d2341b6e0f
2023-10-19 17:19:47 +00:00
Márton Elek
5c49ba1d85 satellite/durability: ignore information from new nodes
To get better performance, we pre-load all nodealias/node information at the beginning of the segment loop.

It's possible that we receive a new node alias from the segment table what we are not fully aware of (yet).

The easiest solution is just ignoring. New risks/threats can be detected by a new execution cycle.

Change-Id: Ib54f7edc46eedbab6d13b4d651aaac1425994940
2023-10-19 16:38:39 +00:00
Márton Elek
a63a69dfd9 satellite/nodeselection: support OR in placement definition
Change-Id: Icc7fd465b28c0c6f09f50c4ab8bffbcc77631dbd
2023-10-19 15:21:15 +00:00
Ivan Fraixedes
3e71ea555d private/apigen: Remove stale TODOs & improve doc
Remove some stale TODO comments and make a method documentation more
specific.

Change-Id: I21b9df0f22a09698014518b5a675eb115d1ba4a4
2023-10-19 12:42:58 +00:00
Márton Elek
188aa3011b satellite/repair/checker: report checker_segment_off_placement_count per placement
Change-Id: Ic1639899f8f0b55c4ef8fe246e7efc0a5d9a2bc1
2023-10-19 11:59:56 +00:00
Vitalii
07c382914c scripts/wasm, worker: fixed wasm module caching issue
Included hash in wasm file.
Added a manifest file that contains the wasm file name for worker access.
Reworked worker setup to query the manifest file without caching, ensuring the correct wasm file name is always retrieved.
The worker will first attempt to retrieve the cached wasm file, but will refetch it with a cache reload if an error occurs.

Change-Id: Ic4ef68e502b318a29243bf275c041863ec1275ee
2023-10-19 12:55:06 +03:00
Jeremy Wharton
4e0ffd1a11 web/satellite/vuetify-poc: improve keyboard navigation for sidebars
This change improves keyboard navigation for the Vuetify UI's
navigation sidebars. Navigation items can now be focused with the tab
key and selected with the enter or space key.

Resolves #6411

Change-Id: I6416efbee74209e089abbccd0e1a7f1c0f4b80cb
2023-10-18 21:44:10 +00:00
Vitalii
6ae28e2306 satellite/{web,console}: enable/disable billing features depending on config value
Added client side logic to disable billing features depending on config value.
Disabled billing endpoints if billing is disabled.

Issue:
https://github.com/storj/storj-private/issues/464

Change-Id: I6e70dc5e2372953b613ddab9f19cb94f008935ce
2023-10-18 21:00:43 +00:00
Clement Sam
bce022ea7a satellite/overlay: remove Type field from NodeDossier
The overlay.NodeDossier struct only tracks information about a
storagenode, the field is deprecated and no longer needed.
This is a kademlia left-over.

Updates https://github.com/storj/storj/issues/5426

Change-Id: Ie278ffd88d1b9a9fde6c81eb5f0e287bab8c9ef0
2023-10-18 18:21:26 +00:00
Clement Sam
0d144ee5af satellite/contact: remove dependency on overlay.NodeDossier
`overlay.NodeDossier` contains information that satellite tracks
about a storagenode.
As part of the work to remove the type column from the nodes
table https://github.com/storj/storj/issues/5426, we would have
to remove the `Type` field in the `overlay.NodeDossier` since it
is only about the storagenode.

The `Local` method on the contact service is also removed because
it is unused. When needed we can create a seperate `SatelliteInfo`
struct which the `Local` method will return.

Change-Id: If0c1a25d9df397a9492bbf1d7f33ba5b6a918878
2023-10-18 18:11:36 +00:00
Wilfred Asomani
32b7b80666 satellite/{console,accountfreeze}: reduce payment retries
This change limits payment attempts to
1. Card updates when billing frozen/warned
2. Right before billing freezing a warned account.

Issue: https://github.com/storj/storj-private/issues/457

Change-Id: Ic6d5c649cdac38d5c3b7365e20a4ceb3b6199ee8
2023-10-18 15:54:46 +00:00
Jeremy Wharton
6d03b92ea6 private/apigen: fix TS class generation from collection of named struct
This change fixes an issue where the TypeScript API code generator
would silently fail to generate class definitions for named structs
used as the element type of array or slice struct fields.

This issue caused invalid TypeScript code to be generated from the
frontend config struct because it contained a slice of partnered
satellite structs. The TypeScript class definition of the frontend
config referenced the nonexistent partnered satellite class.
The satellite frontend config uses, so this issue caused the

Change-Id: Idfcb6ec1bbc603a43033ee4afb5b421b7454855c
2023-10-17 22:45:07 -05:00
Jeremy Wharton
f8b59a50ff satellite/console: configure sending invites to unregistered emails
This change adds a flag to the satellite config indicating whether
unregistered email addresses should receive project invitation emails.

Change-Id: I0396f25574ddae3f9adaea32a6e7cd15b931bf12
2023-10-17 22:42:57 -05:00
Jeremy Wharton
24ae79345b web/satellite{/vuetify-poc}: show upgrade dialog when trying to invite
This change displays a dialog prompting free tier users to upgrade when
the button to invite project members is clicked.

Also, the Create New Project dialog in the Vuetify UI now opens the
upgrade dialog when its Upgrade button is clicked.

Change-Id: I6e233bd15fd14a486a3e9008bbc6fba3e669d67e
2023-10-17 22:40:34 -05:00
Vitalii
4721d2bd4e satellite/console: feature flag for billing features
Change-Id: I27623b2a9cd100ef38dda34bb48e332d616a478c
2023-10-17 22:56:29 +00:00
Moby von Briesen
e0cf6f1e8b satellite/console,web/satellite: Change Vuetify app prefix to "v2"
Change-Id: Ib0e14bd2f5056c6300a640e678413de767e8ee6d
2023-10-17 22:12:46 +00:00
Jeremy Wharton
2cf4784b20 satellite/console: make project invites exclusive to paid tier
This change makes the project member invitation feature exclusive to
the paid tier.

Change-Id: I13c967c8381d49b2d131e15799ad48487b0f6c74
2023-10-17 09:38:29 -05:00
Jeremy Wharton
524e074a8c satellite/console,web/satellite: disallow creating multiple new invites
This change prevents multiple project invitation records from being
created from a single API request.

Change-Id: I01268fcc0e2f7b5f24870b032cb53f03c7ad0800
2023-10-17 09:05:00 -05:00
Egon Elbre
b2d2a8a744 satellite/metabase: add unique unversioned constraint for tests
While the index shouldn't be necessary as long as our implementation is
correct, it still provides some additional checks for mistakes in the
implementation.

Change-Id: I7ed71ac99a979e375d7f94c8898e6f83ac623cb6
2023-10-17 12:08:50 +00:00
Vitalii
45fdc64300 web/satellite/vuetify-poc: download total usage report
Added CTA to billing -> overview screen for user to download total usage report

Issue:
https://github.com/storj/storj/issues/6154

Change-Id: I465c955486e6fa9bf922d56798b2338c4bd2d73f
2023-10-17 11:26:36 +00:00
Michal Niewrzal
d7af97c919 satellite/satellitedb: move IterateBucketLocations sql to dbx
This is attempt to move query back to dbx.

It also removes one unused method.

Change-Id: I8182dd8ecf794cdf0cb3158c36cc00810fc683df
2023-10-17 10:30:51 +00:00
Egon Elbre
7ba8a627bc satellite/metabase: add deleteObjectUnversionedCommitted
By using a separate function for deleting the latest object and
fetching the latest version we can simplify some of the code.

However, there can be more performant approaches, such as using
ON CONFLICT for updating the existing object or using select and delete
in the same query in databases that support it.

Change-Id: I52bc3f9fa025f44d05ee010723ffb81f5bd2a2d7
2023-10-17 11:44:35 +03:00
JT Olio
2578580e21 docs/blueprints: fix deletes and server side copy
Change-Id: I13dc3580bbfcba4dd97d1a3d073505f0e5688ab3
2023-10-17 07:52:03 +00:00
Vitalii
e3713fddb8 satellite/{console, web}: added detailed usage report
Allow user to download detailed usage report from Billing -> Overview screen.
Report is a CSV file containing usage data for all the projects user owns.

Issue:
https://github.com/storj/storj/issues/6154

Change-Id: I3109002bf37b1313652a2be3447aaa7bc6204887
2023-10-16 21:54:18 +00:00
Egon Elbre
41e16bc398 satellite/metabase: versioning, add new statuses
There are several different object types in a versioned table,
which will determine the exact behaviour.

The object type states are:
* Pending   - the object is yet to be committed and is being uploaded.
* Committed - the object has been finished and can be read.
* DeleteMarker - indicates that the object should be treated as not
  present when is at the top of the version stack.

There are also versioning states:
* Unversioned - only one unversioned object is allowed per object key.
* Versioned - multiple objects with the same key are allowed.

Change-Id: I65dfa781e8da253a4e5d572b799d53c351196eee
2023-10-16 17:50:18 +00:00
paul cannon
58f75502f3 satellite/metabase: exclude expired segments from ListVerifySegments results
I believe that this change in semantics won't break anything, because
ListVerifySegments is only used by cmd/tools/segment-verify (which only
needs to operate on non-expired segments) and various tests, none of
which expect ListVerifySegments to include expired segments.

Change-Id: I037f43b16bc5750ed914bc32949418e001df1a8c
2023-10-16 16:58:01 +00:00
paul cannon
23c5d6c287 cmd/tools/segment-verify: improve logging around common problems
Change-Id: I4f684745df708627f135baee619d17788bc8d63e
2023-10-16 16:14:23 +00:00
Michal Niewrzal
281edfa585 satellite/satellitedb: add index to bucket_storage_tallies table
Additional index on 'interval_start' to support data analysis.

Change-Id: I373787871da594aa1b4ebd66cb684484221eb4d5
2023-10-16 15:09:24 +00:00
Jeremy Wharton
d200726436 satellite/console: rate limit project invitation endpoint
This change adds user ID rate limiting to the endpoint responsible for
sending project invitations.

Resolves storj-private#462

Change-Id: Icf0be7d7bb7f2765725ba3e152a2195bc02484e2
2023-10-16 14:24:33 +00:00
Vitalii
902cf5898f web/satellite/vuetify-poc: remove required rule on S3 credentials info step
Remove required rule on S3 credentials info step when creating new one.

Issue:
https://github.com/storj/storj/issues/6400

Change-Id: Ie59267bb881e6dfa769ae0cdda83ff369ec920cd
2023-10-16 13:07:07 +00:00
Michal Niewrzal
0eaf43120b satellite/repair/checker: optimize processing, part 3
ClassifySegmentPieces uses custom set implementation instead map.

Side note, for custom set implementation I also checked int8 bit set but
it didn't give better performance so I used simpler implementation.

Benchmark results (compared against part 2 optimization change):
name                                       old time/op    new time/op    delta
RemoteSegment/healthy_segment-8    21.7µs ± 8%    15.4µs ±16%  -29.38%  (p=0.008 n=5+5)

name                                       old alloc/op   new alloc/op   delta
RemoteSegment/healthy_segment-8    7.41kB ± 0%    1.87kB ± 0%  -74.83%  (p=0.000 n=5+4)

name                                       old allocs/op  new allocs/op  delta
RemoteSegment/healthy_segment-8       150 ± 0%       130 ± 0%  -13.33%  (p=0.008 n=5+5)

Change-Id: I21feca9ec6ac0a2558ac5ce8894451c54f69e52d
2023-10-16 12:06:16 +00:00
Márton Elek
c3fbac2e7a satellite/overlay/placement: better error message for placement parsing
Given the placement.txt with the content:

```
9:exclude(placement(10))
10:country("DE")
```

Without patch:

```
placement-test --placement=/tmp/placement.txt countrycode=DE
Error: type mismatch: reflect: Call using zero Value argument
```

With the patch:

```
placement-test --placement=/tmp/placement.txt countrycode=DE
Error: Error in line 'exclude(placement(10))' when placement rule is parsed: Placement 10 is referenced before defined. Please define it first!
```

Change-Id: I9ad81016d4a57fdb32f3ff9031b5591f9a7cd2a6
2023-10-16 09:28:45 +00:00
Ivan Fraixedes
d8376a2a24 Earthfile: Use same NodeJS version as Makefile
Makefile defines the NodeJS version to use for building front-end
sources.

Earthfile must use the same version for guaranteeing the same results.

Change-Id: I74d5629a2a4489392d2a3303a03a7ccf012f8b24
2023-10-16 08:04:21 +00:00
Vitalii
356eb43a9f satellite/console: make Etherscan URL more configurable
make the block explorer URL (e.g. https://etherscan.io) configurable (default to using https://etherscan.io/)

Issue:
https://github.com/storj/storj/issues/6357

Change-Id: I1c9f14eb4653763318fd96c242130a9c0a7d24c5
2023-10-14 01:58:43 +00:00
paul cannon
ee33cb1289 satellite/repair: protect concurrent access to statsCollector
It would appear that we have been making concurrent accesses to
statsCollector for a long, long time (we expect there to be multiple
calls to `Repair()` at the same time on the same instance of
`SegmentRepairer`, up to `config.MaxRepair`, and before this change
there was no sort of synchronization guarding accesses to the
`statsCollector.stats` map.

Refs: https://github.com/storj/storj/issues/6402
Change-Id: I5bcdd13c88913a8d66f6dd906c9037c588960cc9
2023-10-13 09:12:00 -05:00
Vitalii
7381b5e508 web/satellite: fix object count calculation inside folders
Moved .file_placeholder decrement logic to be outside continuation tokens loop so that object count is decremented only once.

Change-Id: Ie19657753b501df9344dc84d7be2bfb731933faa
2023-10-13 12:27:58 +00:00
Jeremy Wharton
ad13cb2bf5 web/satellite, satellite/analytics: add UI type to analytics properties
This change incorporates the UI type (legacy or Vuetify) in the
properties attached to analytics events originating from the satellite
UI.

Resolves #6363

Change-Id: Ie3627bc24e4349407376e28460a5a830d211b47b
2023-10-13 11:43:54 +00:00
Wilfred Asomani
c9421d11e7 satellite/admin: add endpoints to violation freeze/unfreeze users
This change adds two new admin endpoints to freeze users for ToS
violation and to remove them from that state,

Issue: https://github.com/storj/storj-private/issues/386

Change-Id: I49c922377c9cdb315ce2777fcd35dcad432b0539
2023-10-13 10:43:44 +00:00
Wilfred Asomani
594e63f13a satellite/accountfreeze: mark billing frozen users for deletion
This change modifies the billing freeze chore to set pending deletion
status to users who are still frozen after a grace period.
It also modifies the chore to skip already deleted users.

Issue: https://github.com/storj/storj/issues/6303
https://github.com/storj/storj-private/issues/453

Change-Id: I4d0e7dd904463e99424372dc9ac81b71c0bc6e28
2023-10-12 20:25:23 +00:00
Wilfred Asomani
ee2b6e66de web/satellite/vuetify-poc: improve team page
This change makes minor improvements to the vuetify team page.

Issue: https://github.com/storj/storj/issues/6376

Change-Id: Iedd0e1f47d8075f13b424da65d719057eade14ed
2023-10-12 17:49:55 +00:00
Márton Elek
db3578d9ba satellite: durability rangeloop observer for monitoring risks
Change-Id: I92805fcc6e7c1bbe0f42bbf849d22f9908fedadb
2023-10-12 16:32:30 +00:00
Tome Boshevski
863c96b771
web/satellite: update signup pages plus logo (#6409)
* web/satellite: update signup pages plus logo

Update designs of the signup, login, forgot password, reset password, confirm email pages. Make Business as default on signup page per marketing request. Plus update the logo to remove DCS.
2023-10-12 18:05:44 +02:00
Vitalii
f41e117918 web/satellite/vuetify-poc: fix object browser pagination inside folders
Reflects the same changes made for main app.
https://review.dev.storj.io/c/storj/storj/+/11313

Change-Id: I8e0a2a64b1adfdf411c3044890e85928d592259d
2023-10-12 14:35:09 +00:00
Clement Sam
db7c6d38e5 storagenode/orders: archive unsent order for untrusted satellite
The order service still tries to settle orders at all instances even
when the satellite is marked as untrusted by the trust service, which
will always fail because the trust cache no longer has record of the
URL of the satellite, and it will keep retrying.

This leaves a lot of "satellite is untrusted" errors in the logs.
There has been several complaints on the forum because this was
happening a lot for the stefanlite and I expect it will be the
same issue for the decommisioned satellites US2 and EUN-1 once
the forget-satellite command is run to clean up the satellite.

This change allows the order service to archive unsent orders available
for any untrusted satellite, and will not attempt to settle the order.

https://github.com/storj/storj/issues/6262

Change-Id: If0f7f1783587cd18fab8917d45948f22df5b1dcf
2023-10-12 13:48:07 +00:00
Jeremy Wharton
4e0f062cb5 web/satellite/vuetify-poc: support PDF previewing
This change allows supported platforms to preview PDFs in the object
browser.

Resolves #6397

Change-Id: I6078914f0ddf5f620514e27e51d24ba8c11a2786
2023-10-12 12:00:59 +00:00
Vitalii
7038ddef7f web/satellite/vuetify-poc: fixed styling of stripe input
Made stripe input look more fancy on Payment Methods tab of Billing screen.

Issue:
https://github.com/storj/storj/issues/6399

Change-Id: I69497191b5c7a93905cdd4660ef733f17b4a2854
2023-10-12 13:44:15 +03:00
Michal Niewrzal
e3e303754b satellite/repair/checker: optimize processing, part 2
Optimizing collecting monkit metrics:
* initialize metrics once at the begining
* avoid using string in map for getting stats structs per redundancy

Benchmark results (compared against part 1 optimization change):
name                                       old time/op    new time/op    delta
RemoteSegment/Cockroach/healthy_segment-8    31.4µs ± 6%    21.7µs ± 8%  -30.73%  (p=0.008 n=5+5)

name                                       old alloc/op   new alloc/op   delta
RemoteSegment/healthy_segment-8    10.2kB ± 0%     7.4kB ± 0%  -27.03%  (p=0.008 n=5+5)

name                                       old allocs/op  new allocs/op  delta
RemoteSegment/healthy_segment-8       250 ± 0%       150 ± 0%  -40.00%  (p=0.008 n=5+5)

Change-Id: Ie09476eb469a4d6c09e52550c8ba92b3b4b34271
2023-10-12 10:02:53 +02:00
Michal Niewrzal
de4559d862 satellite/repair/checker: optimize processing, part 1
Optimization by reusing more slices.

Benchmark result:
name                                       old time/op    new time/op    delta
RemoteSegment/healthy_segment-8    33.2µs ± 1%    31.4µs ± 6%   -5.49%  (p=0.032 n=4+5)

name                                       old alloc/op   new alloc/op   delta
RemoteSegment/healthy_segment-8    15.9kB ± 0%    10.2kB ± 0%  -35.92%  (p=0.008 n=5+5)

name                                       old allocs/op  new allocs/op  delta
RemoteSegment/healthy_segment-8       280 ± 0%       250 ± 0%  -10.71%  (p=0.008 n=5+5)

Change-Id: I60462169285462dee6cd16d4f4ce1f30fb6cdfdf
2023-10-11 15:50:29 +00:00
Artur M. Wolff
3019471514 {Jenkinsfile, Makefile}: update Go to 1.21.3
Change-Id: Ia014ff4fa622903431789d0dc7e93044015ecaa9
2023-10-11 10:47:04 +02:00
Márton Elek
9186365507 storagenode/pieces: more granular io and hashing statistic
This patch adds two new monkit metric:
 * piece_writer_io: the sum of the time, which is spent with io.Write during a piece upload (excluding the fs sync of the commit)
 * piece_writer_hash: the sum of the time, which is spent with hashing

The second is especially important. My storagenode (hosted on a cloud server) spend ~30 ms on hasing data, piece_write_io time is usually 5ms for me.

These metrics can help us to identify the reason of slownes on storagenode sides.

Both of these depend on the size of the piece. To make it more meaningfull without exploding the cardinality, I created a few size categories and classified the pieces based on these. Measurements shows that it can provide usefull results (>2MB uploads are usually 23-28 ms).

Change-Id: Ifa1c205a490046655bcc34891003e7b43ed9c0bc
2023-10-11 07:33:04 +00:00
Wilfred Asomani
6308da2cc0 satellite/{payment,console,analytics} extend freeze functionality for violation freeze
This change extends the account freeze functionality account for
violation freezes as well.
Also, debug level logs in the freeze chore have been changed to info.
It adds an analytics event for when an invoice is found that belongs to
a user frozen for violation.
And finally adds whether a user is frozen for violation to the
/account/freezestatus response.

Issue: https://github.com/storj/storj-private/issues/386

Change-Id: Id8e40282dc8fd8f242da52791ab8ddbbef3da2bc
2023-10-10 18:39:29 +00:00
Tome Boshevski
211659b9b0
web/satellite: update and optimize error page (#6391)
Use 3x smaller size background image and update styles. Update button.
2023-10-10 18:02:27 +02:00
Ivan Fraixedes
386a978310 private/apigen: Validate param types
The Go generator only supports certain types as query and path
parameters and it panics on any an unsupported type.

The Document and TypeScript generator don't have any validation for
them. TypeScript generator generates code that compiles, however, it
won't work properly with all the types not supported by the Go
generator.

Because it doesn't make sense that some types may work on the TypeScript
generator, while the Go generator doesn't, doing the validation in the
Param constructor is better because it reports the issue without having
to run the Go generator and it gives a more understanding panic message.

TypeScript generator generates code that works properly with all the
types supported by the Go generator, hence, there isn't any change int
he TypeScript generator in this commit.

Change-Id: I03085283942a98341726a1560f511a46540df9f5
2023-10-10 12:37:20 +00:00
Vitalii
0db898b0a8 web/satellite/vuetify-poc: tiny changes for project dashboard
Download card: added 'per month' to Limit value.
Coupon card: replaced 'Limit' label with 'Included free usage'.
Also, fixed chart tooltip positioning.

Issue:
https://github.com/storj/storj/issues/6378

Change-Id: I6a4eb544cecaeabab6856acf6e272784f2a1c802
2023-10-10 11:41:28 +00:00
Jeremy Wharton
b6b9cccb72 web/satellite/vuetify-poc: object browser improvements
- Object preview can be opened through the object row actions menu.
- Previewed objects no longer shift vertically when transitioning.
- Previewed objects no longer display the next object's preview when
  transitioning.
- The download notification text has been changed from "Success" to
  "Download Started".
- Clicking object browser breadcrumbs no longer redirects to the
  all projects dashboard.
- Upload progress snackbar:
  - The expanded and collapsed widths are now the same.
  - Clicking an item opens the object preview for it.
  - The "Uploading" tooltip position has been moved to the left so that
    it doesn't block the cancel button.

Resolves #6379

Change-Id: Ic1f5cc7948ffa62dc0bce488b61f6d5e121c77b9
2023-10-10 08:32:27 +00:00
Jeremy Wharton
99ba88ae2f web/satellite/vuetify-poc: allow adding coupons
This change allows users to apply coupon codes via the billing
page's overview section.

Resolves #6393

Change-Id: I6f973f5293362efa8cd76149dc1b43468b49b2c9
2023-10-09 22:53:10 +00:00
Jeremy Wharton
dd3779a623 web/satellite/vuetify-poc: invite through all projects dashboard
This change allows project owners to invite members through the all
projects dashboard.

Resolves #6394

Change-Id: Id36d21432ab7b18532679e900d3e00c52fa21fc9
2023-10-09 20:32:54 +00:00
Cameron
0cc04208db satellite/console: add limit increase errors to getUserErrorMessage
Change-Id: Icc84b8d09d3c1edc3ae216b71812442d67e3b5c9
2023-10-09 17:44:48 +00:00
Vitalii
d60e0f0036 web/satellite: fix locked files calculation when pagination is enabled
The problem is that we always have to refetch objects with List command without Delimiter provided to count in objects inside folders.
This change doesn't fix the main problem for recently deleted objects.

Change-Id: Ia64579745999301c285358869e283dff09399f41
2023-10-09 13:02:16 +00:00
Sean Harvey
1c0b23a8d0
web/satellite: ensure "s/" prefix in path for linksharing urls
currently we are generated some linksharing urls with and without
the "s/" in the path. Make sure we're using "s/" for both cases
to avoid a redirect with the backwards compatible style.

Change-Id: If04a41325ebbece23d4e7f2c0431bd4905596bce
2023-10-09 13:00:52 +13:00
Vitalii
013ddbc57f web/satellite: fixed balance inconsistency for billing tabs
Made overview tab always fetch account balance on mount.
Made payment methods tab always fetch wallet (which includes wallet balance) on mount.
Account and Wallet balances are queried from transactionsDB so there is no inconsistency on backend side.

Issue:
https://github.com/storj/storj/issues/6093

Change-Id: Ifb63d20040acf03d60ad08eac51b0f20b5b7365d
2023-10-06 19:31:10 +00:00
Moby von Briesen
3216674c19 satellite/payments/accountfreeze: Increase exclusion threshold
Account freeze excludes accounts who have invoices which are unpaid and
are above a configured value. This change updates that configured value
from $100 to $1000.

Change-Id: Iff381deeb73ef2d8fccaf7a1612a33e8aeb08698
2023-10-06 14:39:20 +00:00
Jeremy Wharton
3ca626a878 web/satellite: send more analytics events for onboarding selection
This change causes analytics events to be sent when the user has chosen
to continue onboarding in the browser or to skip onboarding.
Previously, only the the selection to onboard in the CLI flow was
tracked.

Change-Id: I44bdac6b8704a67382d8655767cdaea691191cbe
2023-10-06 13:26:19 +00:00
Vitalii
60bf9531af web/satellite/vuetify-poc: add client-side sorting for buckets table
Sorting works only if buckets fit on single page (can be changed by modifing limit value).
We use custom sorting approach to sort by correct value type and not by string.
We don't use custom-key-sort prop of VDataTableServer component because it doesn't work (probably expects it to be done on backend side because VDataTable component's prop works as expected).

Issue:
https://github.com/storj/storj/issues/6342

Change-Id: I0cfbdf432e255f530457c89253a7f29b8e1cbc30
2023-10-06 12:45:42 +00:00
dlamarmorgan
7f02b73b5d cmd/satellite/main: complete/fail pending payments cmd
Add billing commands to manually complete or fail storjscan invoice
payments that are stuck in a pending state.

Change-Id: Ia19f0a2597201d9d17aad0889eaedff095d706b9
2023-10-06 01:23:34 +00:00
Wilfred Asomani
65f14ec3f5 web/satellite: prevent row action for project owner
This change fixes an issue where users might be able to remove project
owners from a project buy clicking the "invisible" row action button on
the project owner row.

Issue: https://github.com/storj/storj/issues/6345

Change-Id: Ib6f689d65f23539cc3660f72878d6d50fc502e2c
2023-10-05 20:13:21 +00:00
Clement Sam
d432a7197a storagenode/piecestore: notify if download piece was restored from trash
Updates https://github.com/storj/storj/issues/6146

Change-Id: Iece285eb5ecb6898b29096416ab10e43338480b0
2023-10-05 18:50:01 +00:00
Cameron
e072b37a86 satellite/console: add endpoint to request project limit increase
create endpoint to allow pro users to request project limit increase.

github issue: https://github.com/storj/storj/issues/6298

Change-Id: I96c3dff8bf0906904d199fc2c7ee738f3e6b04a3
2023-10-05 17:21:32 +00:00
Jeremy Wharton
22ad017f12 web/satellite/vuetify-poc: implement browser onboarding flow
This change implements the browser onboarding flow in the Vuetify
project.

Resolves #6334

Change-Id: I68ff214730cb0a39ea5f9af47d9ecbe3051f2005
2023-10-05 16:34:41 +00:00
Clement Sam
cfbb5dac14 go.mod: update storj.io/common
Change-Id: Ibdfd962b8be8407a687753d0648149a5f98a44b2
2023-10-05 15:52:06 +00:00
Clement Sam
05901aa303 cmd/tools/tag-signer: fail for comma separated tags
This change allows tag-signer to fail when key-value pairs
provided as arguments are comma-separated.
However, for cases where a value is expected to contain a comma,
we validate the value only if --confirm flag is specified

Resolves https://github.com/storj/storj/issues/6336

Change-Id: Ib6a100ee3adf529f44c8b3ca620a3c0b4f953a17
2023-10-05 13:57:53 +00:00
Jeremy Wharton
5d286399f3 web/satellite/vuetify-poc: improve the share dialog
This change makes several minor improvements to the Share dialog:
- The "Copy" button has been renamed to "Copy Link".
- A copy icon button has been added to the linksharing URL text field.
- An alert has been added notifying that anyone with the link will be
  able to see the shared data.

Resolves #6381

Change-Id: Idf4c2b2963d7174173c1fa479c90a4fb6c1bf24c
2023-10-05 12:30:21 +00:00
Egon Elbre
27a13efb17 satellite/console/consoleweb/consoleapi: fix defer in loop
Change-Id: Ifdedad8a5556be47b5f99dc8d8a8ff67904e2688
2023-10-05 12:55:36 +03:00
Egon Elbre
800bba7a0e go.mod: bump uplink
This brings in 32bit arm bug fix.

Change-Id: I61b72abf2366643f58b1e670d2e02f4072aee0b5
2023-10-05 08:51:11 +00:00
Ivan Fraixedes
99c4359062 private/apigen: Avoid clashes of types names
The TypeScript generator create types of anonymous structs. When those
anonymous structs are in endpoints with the same name, but in different
endpoint groups, the generator panic because it create different types
with the same names.

This commit fixes that problem through using the endpoint group prefix
to create the types with different names.

Change-Id: Ibe87532609ce824b80951326f9777ed5b0cc2f7a
2023-10-05 07:30:09 +00:00
Ivan Fraixedes
3193ff9155 private/apigen: Don't force casing for API group name/prefix
The API generators rely on the Name and Prefix fields of the
EndpointGroup type to generate code.

Conventional naming code requires using upper or lower case for types,
functions, etc, however requiring the user to set this fields with the
correct casing seems cumbersome for them because they can be adjusted
depending where those values are used on the generated code.

This commit lifts the restriction for the user and adjust the casing of
them according to where they are used.

Change-Id: I700a879d13b4789b4d6ba0519b4d7508061eac73
2023-10-05 06:48:50 +00:00
Sean Harvey
7b50ece931
satellite/admin: add missing test assert to project geofence tests
the existing test was missing a case where deleting the geofencing
should set the placement back to default.

this also changes EveryCountry to DefaultPlacement, as the former is
deprecated.

Change-Id: I29f4f1c3ae1d05e8f5ecaa03a70ac4f17574a475
2023-10-05 18:21:48 +13:00
Vitalii
95958a3c5c web/satellite/vuetify-poc: emit success notification on access grant delete
Show success notification after access grant deletion.

Issue:
https://github.com/storj/storj/issues/6364

Change-Id: Idebc4210ff5efbaab0c71ad519d34de4e1803420
2023-10-04 20:21:26 +00:00
Ethan Adams
6961d2e19d satellite/email: Add support for unauthenticated, cleartext SMTP connections
Change-Id: I11b4852122764c1ede188ca40d5edb14f2c4ee72
2023-10-04 16:55:55 +00:00
paul cannon
02899dfae2 satellite/{admin,console,satellitedb}: fix geofence removal
deleteGeofenceForProject wasn't able to work correctly, because
Console().Projects().Update() declines to update default_placement when
the input value is 0.

This introduces a Console().Projects().UpdateDefaultPlacement() method,
congruent to the method of the same name on Console().Users().
deleteGeofenceForProject now uses this new method, so that specifying a
new placement of 0 will work correctly.

Change-Id: I4589b36707f7e4f1cfdc66543520b0d4205c1a84
2023-10-04 16:14:29 +00:00
Moby von Briesen
a2acf359ad satellite/satellitedb: Remove dbx references to partner_id
This column is no longer used, but it is referenced in dbx. This change
removes those references and adds an exception to the migration test to
disregard this column in schema comparison.

After this change is deployed, we will need to follow up with a
migration to remove the column in a later release, and remove the
exception in the migration test.

Related to https://github.com/storj/storj/issues/5432

Change-Id: I168fb57244b347901d1ed9c7813c1338554ee644
2023-10-04 15:33:44 +00:00
paul cannon
c33475f63e storagenode/gracefulexit: fix flaky test (hopefully)
If the storagenode chore is left running and it has a chance to check in
again after we move time forward (line 139), then the satellite will
mark it as having finished GE before we check which nodes are still in
GE (line 149).

Change-Id: I350e1ef2e943f758d44132aaddd05fe248b30f3e
2023-10-04 09:53:51 -05:00
Ivan Fraixedes
47f344927d
private/apigen: Fix TS generator to apply naming conventions
Fix the TypeScript generator to generate code using the common
TypeScript conventions.

Closes https://github.com/storj/storj/issues/6360

Change-Id: I244896feea389670eca0df95d3ac58e25d600e14
2023-10-04 13:47:33 +02:00
Ivan Fraixedes
71c9547de5
private/apigen: Don't allow certain types for Request/Response
We cannot map certain types to TypeScript, hence we verify them in the
Endpoint.Validate method.

Even some types could be somehow mapped, we don't want to add more
complexity or allow types that don't make sense to be for a request or
response.

Change-Id: I51ecee286e637b1160e967d77f9ce6c7403ddfdc
2023-10-04 13:46:54 +02:00
Ivan Fraixedes
956109a097
private/apigen: Rename Endpoint fields
Rename the Endpoint fields MethodName and RequestName because they were
confusing for what they are used.

This commit also adds some validations for these fields values and other
validations for Endpoint and EndpointGroup to avoid generating invalid
code.

It also include some tests for these new validations.

Closes https://github.com/storj/storj/issues/6333

Change-Id: Iaabfc33935517889e3729c8b37be51a55eea366c
2023-10-04 13:46:54 +02:00
Ivan Fraixedes
ac86eb397a
private/apigen: Fix bug TypeScript optional class members
The TypesScript generator generates classes and checks for Go struct
field if the struct type is of a nullable type, however, it should check
if the field types is nullable because it doesn't make sense to check
that struct field for each field nor to assign to the field if it's
nullable or not depending on it.

Change-Id: Ia22a609a17752f520233c006cba17685fe142b32
2023-10-04 13:46:08 +02:00
Sean Harvey
bc7f621073
satellite/satellitedb: fix DefaultPlacement overwritten on user
this fixes cases where it's possible to update a user and the
DefaultPlacement field gets overwritten to the zero value.

it also adds UpdateDefaultPlacement which can be used to set
DefaultPlacement directly. This is needed for the geofencing
endpoints in satellite admin to set the DefaultPlacement back
to zero to delete geofencing for a user.

Change-Id: If2c798dabfa6773ed6023fb8257bf00ec7bc2e68
2023-10-04 15:17:35 +13:00
Márton Elek
6304046e80 satellite/nodeselection: read email + wallet from db to SelectedNode
NodeSelection struct is used to make decisions (and assertions) related to node selection.

Usually we don't use email and wallet for placement decision, as they are not reliable.

But there are cases, when we know that the email address is confirmed. Also, it can be used for upper-bound estimations (if same wallet is used for too many pieces in a segment, it's a sign of a risk, even if not all the risks can be detected with this approach, as one owner can use different wallets).

Long story short: let's put wallet and email to the SelectedNode.

Change-Id: I922185e3769d43eb7762b8d60d88ecd3d50991bb
2023-10-03 18:15:56 +00:00
paul cannon
a06735c1b6 satellite/gracefulexit: add missing test cases
These test cases are the parts of the testplan for the Graceful Exit
Revamp which are automateable but not yet automated.

I'm not entirely sure why we have to reject graceful exit from nodes
that are suspended, but implementing that was probably easier than
convincing everybody that it's not necessary.

Refs: https://github.com/storj/storj/issues/6369
Change-Id: I0261b37f7e010d72d84332cde5dd8689f7c41580
2023-10-03 17:35:20 +00:00
Clement Sam
a2c162db9b storagenode/trust: ensure trust pool updates satellite status on Refresh
Fixes https://github.com/storj/storj/issues/6261

Change-Id: Ic01ce423156058dd4676fb073c0de3d768991d0e
2023-10-03 16:54:00 +00:00
dlamarmorgan
8a1bedd367 satellite/payments/{billing,stripe}: handle pending invoice payments
Currently, pending invoice payments that are made using a users token
balance can get stuck in a pending state if the invoice is not able
to be paid appropriately in stripe. This change addresses these stuck
token invoice payments by attempting to transition them to failed
if the invoice cannot be paid.

Change-Id: I2b70a11c97ae5c733d05c918a1082e85bb7f73f3
2023-10-03 16:12:39 +00:00
Wilfred Asomani
2e87df380d satellite/{web,console}: return empty payments list on no wallet error
This change modifies wallet payments endpoints to return empty lists
instead of returning a 404 error if a wallet is not found for a user.

Change-Id: Ic765fecbc8183d14f179ce1d510ae512d8e0c4a9
2023-10-03 14:53:17 +00:00
Tome Boshevski
512e063a79
web/satellite: ui updates (#6366)
* web/satellite: ui updates

Updating global styles, theme, ui components, charts design, actions and dialogs.
Added new icons and replaced current to use icon components with consistent width.
Changed project card with new details button.
Updated the same button for actions across the tables and the actions dialog that it shows.
Changed the account menu with the account type displayed on top.
Add notification in invite team members dialog to explain passphrase.

* web/satellite: ui updates lint fix
2023-10-03 10:51:27 -04:00
Vitalii
487ac17870 web/satellite/vuetify-poc: updates for dashboard limit cards
Updated segment card CTA behavior.
Updated coupon card title, used label and CTA behavior.

Issue:
https://github.com/storj/storj/issues/6343

Change-Id: I86f242208cd4f17ab9d5d61967b22ff740c4edd4
2023-10-03 13:05:23 +00:00
Michal Niewrzal
2c76c1e6ae satellite/satellitedb: alter bucket_metainfos primary key
Current bucket_metainfos table schema was far from optimal. All
operations are using project_id and name to find bucket but primary key
was on id field which is completely not used. Turns out that secondary
index was not as fast as it could be because each select was also joining
primary index which was slowing down whole query.

Primary key is changed to project_id/name. Other indexes are dropped
and id column is now nullable which gives us space for dropping it
completely from schema and code later.

Change-Id: I477056a4243d39e39489f1473de01ded1ed24bbb
2023-10-03 12:23:00 +00:00
Vitalii
eddfacc2e9 web/satellite/vuetify-poc: remove project member feature
Implemented remove project member functionality.
Also, fixed project members search/pagination/sort functionality to work through backend.

Issue:
https://github.com/storj/storj/issues/6327

Change-Id: I0a8df1578a8c7ab9b7d6ce8e2687a3a02cf6be57
2023-10-03 11:41:51 +00:00
Jeremy Wharton
fd835859d5 go.mod: bump stripe-go from v73 to v75
This change updates the version of the stripe-go dependency from v73 to
v75 in order to improve performance (see stripe/stripe-go#1728).

Resolves #6287

Change-Id: I727ac08dcaa90d7138ba30d907711cc44daf0b7a
2023-10-03 10:12:29 +00:00
Vitalii
a7c246badc web/satellite: reworked delete bucket modal
Reworked delete bucket modal:
- updated styling
- added bucket name label which is supposed to be deleted

Issue:
https://github.com/storj/storj/issues/5550

Change-Id: I0d254d45a11878c86847020e2f8c172c2dfbec98
2023-10-03 12:31:30 +03:00
Jeremy Wharton
2f19636eb3 go.mod,satellite/payments/stripe: bump stripe-go from v72 to v73
This change updates the version of the stripe-go dependency from v72 to
v73. This is part of a process to reach v75, which contains performance
improvements.

References #6287

Change-Id: I95f132378fe05e506f0388a0fe039cb1c2db58e2
2023-10-03 07:38:07 +00:00
Cameron
4d5335dbe9 web/satellite: use project limits for upgrade modal
use project limits instead of user limits for upgrade modal in case the
project limits were increased manually.

issue: https://github.com/storj/storj-private/issues/435

Change-Id: Ib70386b390527e96a78461354996ce9a74caa61b
2023-10-03 01:44:34 +00:00
Vitalii
25ffa3ad11 satellite/analytics: stop sending normal account activity events directly to Hubspot
Removed code which was responsible for tracking regular user activity in hubspot.
The only direct event we send to Hubspot is 'user created'.

Issue:
https://github.com/storj/storj-private/issues/441

Change-Id: Ia6da30374f91369372af8fe5b5990ec5672fc03b
2023-10-02 22:48:15 +00:00
Vitalii
5295afb2da web/satellite: use modal instead of old create project view
Use newer modal instead of old create project view.
Also, created new composable to handle create project click.

Issue:
https://github.com/storj/storj/issues/6318

Change-Id: I50fce95924c5511c4a31e8f6e7ad271d3ff7081c
2023-10-02 22:07:12 +00:00
Ivan Fraixedes
9d7ef17a26
private/apigen: Fix code generation for slices & arrays
Fix the API generator to generate valid TypeScript code when using
slices an arrays of any type (base types, struct types, anonymous struct
types, etc.).

Closes https://github.com/storj/storj/issues/6323

Change-Id: I580ae5305c58f65c2e4f4a35d14ca4ee509a9250
2023-10-02 17:05:35 +02:00
Igor
bde3e48842
docs/testplan: Adding a testplan for GE revamp (#6355) 2023-10-02 16:56:09 +02:00
Egon Elbre
c6436e1500 satellite/metabase: rename BeginObjectExactVersion method
With the upcoming versioning changes `BeginObjectExactVersion` makes
only sense for testing. Currently this does not rename the options
struct or move it into `metabasetest`, because it would create a
significant amount of merge/rebase noise.

Change-Id: Iafa2f81a05ae66320bc6a839828217ec94c63e1f
2023-10-02 16:17:13 +03:00
Michal Niewrzal
bec981aa7a gc-bf: add live count observer to loop
To live count observer is emitting metric with number of segments
already processed. It's handy to see current progress.

Change-Id: Id6766b5ec9b05bce8f6376e9afcae91fcf7df6fd
2023-10-02 12:35:27 +00:00
Wilfred Asomani
33fe731620 satellite/{payment,console} rename freeze events and add violation event
This change introduces a new freeze event, ViolationFreeze, for ToS
violations and also prepends other events with Billing to clarify what
they signify.

Issue: https://github.com/storj/storj-private/issues/386

Change-Id: Ieef2abbbb03d06377dc7a73ba5ef0742ada75e8e
2023-09-29 16:52:23 +00:00
Jeremy Wharton
4dbf26e153 web/satellite/vuetify-poc: shorten project ID in URLs
This change shortens the project ID path segment in Vuetify URLs in
order to make the URLs more aesthetically pleasing and allow users to
see more of the URL in the address bar. The ID path segment is now 11
characters long instead of the previous 36, but in rare cases where a
user is a member of multiple projects with the same ID prefix, it
expands to preserve uniqueness.

Resolves #6308

Change-Id: I25a51d05b72d2cc701c0aa2cd3a6d070080c4b1e
2023-09-29 16:11:21 +00:00
Wilfred Asomani
b069b9b038 web/satellite: patch listing token history issue
This change patches an issue where a user who has not claimed a wallet
would see the error "Can not list token payment history" on the all
projects dashboard.

Issue: https://github.com/storj/storj/issues/6358

Change-Id: I0783fae2c4441be4495b9c8bd82cf6dbe6eea557
2023-09-29 13:57:25 +00:00
Vitalii
c34202f5e2 web/satellite/vuetify-poc: localize big numbers
Localized number formatting (e.g. dots, commas, spaces)

Issue:
https://github.com/storj/storj/issues/6335

Change-Id: Ia1f6ba781fe0089803717aa0e2a9eb5da6ab2946
2023-09-29 12:11:03 +00:00
Ivan Fraixedes
1f400a2750
satellite/admin/back-office/ui: Delete yarn.lock file
The yarn.lock file came from the former repository where the back office
UI was developed.

It seems that our build process complains about some dirty state related
to this file. Because we don't use Yarn, we delete the file, hoping to
resolve the build issues.

Change-Id: I5febd8292657289d0fc67e08151c6c8b5ac8b5dc
2023-09-29 12:48:17 +02:00
Wilfred Asomani
46ee1c1414 satellite/{payment,console} add pending deletion user status
This change introduces a new user status, PendingDeletion to be used to
mark users before they're actually deleted. It also skips users with
this status or Deleted status when generating invoices.

Issue: https://github.com/storj/storj/issues/6302

Change-Id: I6a80d0ed1fe4f223ae00e0961f18f2f62f9b5213
2023-09-28 22:23:46 +00:00
Tome Boshevski
a14a18185b
Update logo in README.md (#6354)
Update logo to the newest one, and also add a dark mode version of the logo.
2023-09-28 19:44:11 +02:00
Jeremy Wharton
ae91fa3ce4 {satellite/console,web/satellite}: change invite text in register page
This change removes the project name and inviter name from the
registration page's invitation text in order to prevent phishing
attempts using these values from succeeding.

Resolves storj-private#431

Change-Id: I08636d712b6b273d484cf0594d395c9d7c02ebfa
2023-09-28 16:19:22 +00:00
Jeremy Wharton
9587e09c78 cmd/satellite: copy Vuetify distribution folder to satellite
This change ensures that files generated by building the Vuetify
project are copied to the satellite container.

References #6251

Change-Id: If56fe754d51f1487a3b3c2cf98c40e3010539121
2023-09-28 10:21:58 -05:00
Jeremy Wharton
e5f6be2f02 web/satellite/vuetify-poc: prepend icons to dashboard card titles
This change adds icons before the titles of cards in the project
dashboard.

Resolves #6294

Change-Id: I65a0e3be5433bd156568a76a875552ace8905a91
2023-09-28 09:18:55 -05:00
Michal Niewrzal
4737e912f0 satellite/metabase: simplify main commit object query
Simplifying over complicated SQL query for committing object while
upload.

Change-Id: I99b918192c292288e6b9397d0f9ccf9b81838f4f
2023-09-28 13:20:41 +00:00
Márton Elek
b28439be24 cmd/tools/placement-test: cli to test placement configuration
Change-Id: I7308fbf8fcd740fc136e87d9c2c08eaeb461a106
2023-09-28 10:01:53 +00:00
Wilfred Asomani
41799ef86f web/satellite/vuetify-poc: add pricing plan steps
This change adds the ability to upgrade using a custom pricing plan.

Issue: https://github.com/storj/storj/issues/6288

Change-Id: I866de25e47cb315d107201b1ccaca2cbdad6cf3c
2023-09-28 07:31:21 +00:00
Sean Harvey
31d42bb136
satellite/metainfo: fix inconsistency with auth check error
This fixes an inconsistency with error returned on copy and move
endpoints to match other endpoints. validateAuth() is already
wrapping the RPC status around the error, so this shouldn't be
doing it again.

This also ensures that rate limit errors for FinishCopyObject and
FinishMoveObject are correctly returned as rpcstatus.ResourceExhausted
so uplink can correctly map these to uplink.ErrTooManyRequests.

Change-Id: I6bf6185b456d6774b99d56cf3d7d8f8aa2afa0e8
2023-09-28 18:38:53 +13:00
Jeremy Wharton
b3d12a2436 web/satellite/vuetify-poc: include project and route name in page title
This change adds the name of the current route and the name of the
currently-selected project to the webpage title.

Resolves #6331

Change-Id: Ia42f6a5f9411e92a8217b21757ff7581bac2c3be
2023-09-27 22:52:07 +00:00
Jeremy Wharton
f40954c7b6 satellite/admin: return burst limit in API response
The satellite admin API endpoint responsible for returning project
limits now includes the burst limit in its responses.

Resolves #6276

Change-Id: Ibb3f1fdebf2f9ffd62de2d7e7a60d978c25bb22a
2023-09-27 22:11:00 +00:00
Jeremy Wharton
1d1f8811a8 web/satellite: update wording of limit increase success notification
This change updates the wording of the notification that appears after
successfully submitting a project limit increase request. The
notification now indicates how long it will take for the new limit to
be applied.

Resolves #6291

Change-Id: Ia034e5576a3a04246e38a0583b1febb832794afc
2023-09-27 21:13:47 +00:00
Wilfred Asomani
a5c1d9aa19 web/satellite/vuetify-poc: add card and token option steps
This change adds the option to upgrade using credit card or tokens.

Issue: https://github.com/storj/storj/issues/6288

Change-Id: Ic0141c49ec4cf6311d381c4941cfa95371d62e94
2023-09-27 18:25:31 +00:00
Wilfred Asomani
f3dbeed239 web/satellite/vuetify-poc: add upgrade options step
This change adds the option step to choose between adding a card or
adding Storj tokens.

Issue: https://github.com/storj/storj/issues/6288

Change-Id: If3324912c02b84f47e49eb06e04f54ba1fbf0ca4
2023-09-27 18:00:22 +00:00
Wilfred Asomani
1e3da9f276 web/satellite/vuetify-poc: add upgrade account dialog
This change adds the account upgrade dialog with the first information
step. It allows a user to toggle on this dialog from the account
dropdown or the dashboard.

Issue: https://github.com/storj/storj/issues/6288
https://github.com/storj/storj/issues/6292

Change-Id: Ide87612994c999759150c8aa85ead3866e9df1f5
2023-09-27 17:58:51 +00:00
Moby von Briesen
c14e4b1eb4 satellite/console: update CSP
Include *.storjsatelliteshare.io in the `connect-src` portion of the CSP
for the satellite UI.

Change-Id: Ic8c3d0cf892a3275866634cae3e9260d925e1c3e
2023-09-27 15:49:48 +00:00
Márton Elek
58b98bc335 satellite/repair: repair is configurable to work only on included/excluded placements
This patch finishes the placement aware repair.

We already introduced the parameters to select only the jobs for specific placements, the remaining part is just to configure the exclude/include rules. + a full e2e unit test.

Change-Id: I223ba84e8ab7481a53e5a444596c7a5ae51573c5
2023-09-27 14:54:06 +00:00
Vitalii
63645205c0 web/satellite: fix object browser pagination for the folders
Fix pagination for the folders which are not on the first page.
Also, fixed object count calculation inside folders.

Issue:
https://github.com/storj/customer-issues/issues/1055

Change-Id: I1d0fbb8856f13be6fb20698315a7e4d20b4affd9
2023-09-27 14:12:02 +00:00
Michal Niewrzal
e1215d5da8 satellite/overlay: add AOST to GetParticipatingNodes method
This method is sometimes ends with transaction error. Most probably
because it's trying to do full table scan on nodes table which is
heavily used. Adding AOST should help with DB contention.

Change-Id: Ibd4358d28dc26922b60c6b30862f20e7c0662cd1
2023-09-27 12:00:10 +00:00
Ivan Fraixedes
2a8e5aecfd
satellite/admin/back-office/ui: Don't ignore package-lock.json
When the new back office UI sources where copied from former repository
I didn't realize that the .gitignore had the package-lock.json file.

This commit remove the package-lock.json file, so it can be tracked, in
order to have reproducible builds.

The lack of the file caused the build to fail due to `npm ci` requires
it.

Change-Id: Ibe493d0cd5762afe5caabe9b77a333fd6daa5373
2023-09-27 13:17:30 +02:00
paul cannon
72189330fd satellite/gracefulexit: revamp graceful exit
Currently, graceful exit is a complicated subsystem that keeps a queue
of all pieces expected to be on a node, and asks the node to transfer
those pieces to other nodes one by one. The complexity of the system
has, unfortunately, led to numerous bugs and unexpected behaviors.

We have decided to remove this entire subsystem and restructure graceful
exit as follows:

* Nodes will signal their intent to exit gracefully
* The satellite will not send any new pieces to gracefully exiting nodes
* Pieces on gracefully exiting nodes will be considered by the repair
  subsystem as "retrievable but unhealthy". They will be repaired off of
  the exiting node as needed.
* After one month (with an appropriately high online score), the node
  will be considered exited, and held amounts for the node will be
  released. The repair worker will continue to fetch pieces from the
  node as long as the node stays online.
* If, at the end of the month, a node's online score is below a certain
  threshold, its graceful exit will fail.

Refs: https://github.com/storj/storj/issues/6042
Change-Id: I52d4e07a4198e9cb2adf5e6cee2cb64d6f9f426b
2023-09-27 08:40:01 +00:00
Jeremy Wharton
3d3785a605 web/satellite/vuetify-poc: prevent inappropriate sidebar closure
This change prevents the navigation sidebar from closing when an item
in the My Account dropdown menu is clicked and the display size is
larger than medium.

Resolves #6332

Change-Id: Id37c3d8ee7179805cfecbd3eac9257130e9acc5b
2023-09-26 19:31:12 +00:00
Moby von Briesen
f63f3f19ee satellite/analytics: Do not set lifecyclestage from Segment
We set lifecyclestage in Hubspot without Segment now. If Segment tries
to set lifecyclestage, it interferes with the desired behavior in
Hubspot.

Change-Id: I817c0324ecc69529d8ca7f617cb97d2f4e84aee8
2023-09-26 17:10:22 +00:00
Márton Elek
0affe03007 cmd/tools/tag-signer: make the output less noisy
This patch removes the following lines from the output (with disable tracing + set the log level to warn):

```
2023-09-25T15:30:58+02:00	INFO	process/tracing.go:73	Anonymized tracing enabled
2023-09-25T15:30:58+02:00	DEBUG	tracing collector	monkit-jaeger@v0.0.0-20220915074555-d100d7589f41/udp.go:128	started
2023-09-25T15:30:58+02:00	DEBUG	process/debug.go:37	debug server listening on 127.0.0.1:34803
```

Change-Id: Iccbf4fc3bde9436e0571943d0d85c51ebc766ef9
2023-09-26 16:30:48 +00:00
Vitalii
05a276ecc7 web/satellite/vuetify-poc: add buckets table to project dashboard
Added buckets table to the bottom of project dashboard view

Issue:
https://github.com/storj/storj/issues/6322

Change-Id: I059bf60631096956d55522da7d18cb8a9eaedc93
2023-09-26 15:49:47 +00:00
Vitalii
f5af0f2268 web/satellite/vuetify-poc: add geographic distribution dialog
Added geographic distribution dialog to object browser preview.

Issue:
https://github.com/storj/storj/issues/6321

Change-Id: Ia3d89a3c27ce8100a3a0a4e5fff91f0c41d0595e
2023-09-26 15:49:06 +00:00
Vitalii
f78cde93ac web/satellite/vuetify-poc: make project table search field be consistent with other search fields
Made search field styling be consistent with other search fields.

Issue:
https://github.com/storj/storj/issues/6320

Change-Id: I21e383cef522a9f76b437a8f9977eab72987766c
2023-09-26 15:09:41 +00:00
Cameron
8689f609d7 web/satellite: vuetify upload progress panel
vuetify snackbar with expansion panels displaying upload statuses.

https://github.com/storj/storj/issues/6257

Change-Id: Ife01616f5a07a4987153ef85331ff71f53b8cf78
2023-09-26 14:04:40 +00:00
Ivan Fraixedes
6555a68fa9 satellite/admin: Serve back-office static UI
Serve the front-end sources of the new back-office through the current
satellite admin server under the path `/back-office`.

The front-end is served in the same way than the current one, which is
through an indicated directory path with a configuration parameter or
embed in the binary when that configuration parameter is empty.

The commit also slightly changes the test that checks serving these
static assets for not targeting the empty file in the build folder.

build folders must remain because of the embed directive.

Change-Id: I3c5af6b75ec944722dbdc4c560d0e7d907a205b8
2023-09-26 13:18:29 +00:00
Ivan Fraixedes
48d7be7eab
private/apigen: Panic types defined in main package
The API generator was generating invalid code when types were defined in
a main package because the generated Go code was defining in import from
it.

This commit update the Go generator to panic with a explicit error
message if that situation happens.

The commit also add a new endpoint to the example with a named types
(i.e. no anonymous) to show that the Generator works fine with them.

Change-Id: Ieddd89c67048de50516f7ac7787d602660dc4a54
2023-09-26 14:04:33 +02:00
Ivan Fraixedes
7ab7ac49c8
private/apigen: Remove TypeScript errors from generated code
Remove the "duplicate identifier: 'path'" TypeScript errors from the
generated code.

Change-Id: I6b411a5cee720e2a16353034627954616c480f8a
2023-09-26 14:04:33 +02:00
Ivan Fraixedes
00484429d6
private/apigen: Generate valid TypeScript code with anonymous types
The API generator didn't generate valid TypeScript code when using
Go anonymous types.

This commit fixes that issue creating names for anonymous types.

Change-Id: Ice0748d8650686e3d3979523b8f218dc20eade5a
2023-09-26 14:04:33 +02:00
Ivan Fraixedes
a9901cc7d0
private/apigen: Add validations to improve usage
Add a few validations to panic with a nicer message or abort rather than
generating invalid code.

Also improve the panic message wrapping a standard error with errs2 at
the time that it's returned.

Change-Id: I1393933eb5f0bc3f86646bf4d0acfc64626efbe0
2023-09-26 14:04:33 +02:00
Ivan Fraixedes
822a13570e
private/apigen: Document Endpoint fields
Add document comments to tall the fields of the Endpoint struct.

Change-Id: Ibcd534f26474fc73fb378a9f30ce7d9401217557
2023-09-26 14:04:32 +02:00
Wilfred Asomani
d67bb4f2c5 web/satellite/vuetify-poc: show file delete confirmation dialog
This change extends the folder deletion modal to work for objects as
well.

Issue: https://github.com/storj/storj/issues/6299

Change-Id: I13e9ffa508c802480c0e3ed2ac630fa693b66fc7
2023-09-26 11:00:16 +00:00
Vitalii
4cbfa28e10 web/satellite/vuetify-poc: use 'demo-bucket' placeholder instead of prefilled value
Use placeholder instead of prefilled value for bucket creation dialog.

Issue:
https://github.com/storj/storj/issues/6319

Change-Id: I86c25926034adbe93a58df56e0bf18b60b41e568
2023-09-26 09:59:10 +00:00
Vitalii
4a9d5edbfc web/satellite/vuetify-poc: another round of small improvements
break text of download notification (e.g. when you download from preview)
hide right/left buttons in object preview if there are not next/previous items
add counter property to project name in "project create". Set max length so that user cannot type additional characters

Issue:
https://github.com/storj/storj/issues/6268

Change-Id: Icff95427a5c73c2fb5bb014ff09150283cc49e83
2023-09-26 11:37:21 +03:00
Ivan Fraixedes
9a06de9058 private/apigen: Add test for Types.All
Add a test for Types.All method because we may need to adjust the logic
in future commits and we want to detect what has changed for good or
bad.

Change-Id: I1db4bf67db3c87513cb9aeb7b942c6c132fc4dd1
2023-09-25 21:15:06 +00:00
Antonio Franco
fe0b5743b0
Af testplan/private cloud (#6265)
* doc(testplan): storj private cloud

* Update storj-private-cloud-testplan.md

this commit pushes the content to the template file and removes unused sections.
2023-09-25 18:44:39 +02:00
paul cannon
1b8bd6c082 satellite/repair: unify repair logic
The repair checker and repair worker both need to determine which pieces
are healthy, which are retrievable, and which should be replaced, but
they have been doing it in different ways in different code, which has
been the cause of bugs. The same term could have very similar but subtly
different meanings between the two, causing much confusion.

With this change, the piece- and node-classification logic is
consolidated into one place within the satellite/repair package, so that
both subsystems can use it. This ought to make decision-making code more
concise and more readable.

The consolidated classification logic has been expanded to create more
sets, so that the decision-making code does not need to do as much
precalculation. It should now be clearer in comments and code that a
piece can belong to multiple sets arbitrarily (except where the
definition of the sets makes this logically impossible), and what the
precise meaning of each set is. These sets include Missing, Suspended,
Clumped, OutOfPlacement, InExcludedCountry, ForcingRepair,
UnhealthyRetrievable, Unhealthy, Retrievable, and Healthy.

Some other side effects of this change:

* CreatePutRepairOrderLimits no longer needs to special-case excluded
  countries; it can just create as many order limits as requested (by
  way of len(newNodes)).
* The repair checker will now queue a segment for repair when there are
  any pieces out of placement. The code calls this "forcing a repair".
* The checker.ReliabilityCache is now accessed by way of a GetNodes()
  function similar to the one on the overlay. The classification methods
  like MissingPieces(), OutOfPlacementPieces(), and
  PiecesNodesLastNetsInOrder() are removed in favor of the
  classification logic in satellite/repair/classification.go. This
  means the reliability cache no longer needs access to the placement
  rules or excluded countries list.

Change-Id: I105109fb94ee126952f07d747c6e11131164fadb
2023-09-25 09:42:08 -05:00
Márton Elek
c44e3d78d8 satellite/satellitedb: repairqueue.Select uses placement constraints
Change-Id: I59739926f8f6c5eaca3199369d4c5d88a9c08be8
2023-09-25 10:14:25 +00:00
Márton Elek
b4fdc49194 satellite/repair/checker: persist placement information to the queue
Change-Id: I51c7fd5a2a38f9f6620c16eddaed3b4915ffd792
2023-09-25 09:33:46 +00:00
Márton Elek
18d5caad7e satellite/satellitedb: write/read placement information to/from repairqueue
Change-Id: Ie58f129feae7898850905940f94643605dcf56ae
2023-09-25 08:52:56 +00:00
Vitalii
7d0b8f6f8c web/satellite/vuetify-poc: center loader in file preview
Vertically centered loader and no preview states for gallery view.

Issue:
https://github.com/storj/storj/issues/6312

Change-Id: Icce4aff6d0927ee36e3a94886edf4cc31dd379bf
2023-09-23 05:11:51 +00:00
Vitalii
5689083393 web/satellite/vuetify-poc: fix limits values formatting on project dashboard
Fixed regular expression which should search and remove insignificant trailing zeros of limit values.

Issue:
https://github.com/storj/storj/issues/6311

Change-Id: I267c779e406a1933d43f09b497470bad9f8ab71c
2023-09-22 21:16:54 +00:00
Vitalii
52d337496f web/satellite: change 'Bandwidth' and 'Egress' labels to 'Download' on project dashboard/settings
Updated label to be 'Download' instead of 'Bandwidth' or 'Egress'

Issue:
https://github.com/storj/storj/issues/6310
https://github.com/storj/storj/issues/6315

Change-Id: Id075882e8dd1e243ca516907d9db420d6d066437
2023-09-22 20:36:22 +00:00
Vitalii
a43f17d9a3 web/satellite/vuetify-poc: use 'Files' header instead of 'Objects' for buckets table
Updated table header to be 'Files' instead of 'Objects'

Issue:
https://github.com/storj/storj/issues/6309

Change-Id: I65058b473481e8b7218db15767c5cb7e2e0c2d8d
2023-09-22 19:01:10 +00:00
Ivan Fraixedes
8381483f79
satellite/admin: Add back-office UI sources
Add the front-end sources of the new back-office.

The front-end doesn't have any business logic, it only has the pages and
the components, so it's purely UI.

The front-end was developed in a separate repository until was
completed.

Change-Id: I382e50789d6b929a67b8a0b887563ef48cb1473d
2023-09-22 18:02:55 +02:00
Dan Willoughby
e563de6c81 web/satellite: Update segment and free tier doc links
4 links updated

Change-Id: I71111079b58542e8c7e4240e2cb55414a7b35934
2023-09-22 11:33:58 +00:00
Jeremy Wharton
5f4cd92cc5 web/satellite/vuetify-poc: clear S3 data when switching projects
This change fixes an issue where the S3 client data was not cleared
when switching projects. This would cause errors to appear when
entering the bucket of a project you switched to.

Resolves #6295

Change-Id: Ib9da43ddf1d38eed6ca26ba73a24e38815617b3e
2023-09-22 00:49:22 -05:00
Jeremy Wharton
e92acca937 web/satellite/vuetify-poc: show loader when creating a folder
This change makes the New Folder dialog indicate its loading status.

Resolves #6307

Change-Id: I37c87ce78f5dfa0d2594e874f248c451fb0d710f
2023-09-21 23:08:24 +00:00
Jeremy Wharton
0205a08c20 web/satellite/vuetify-poc: use image icon for SVGs in object browser
This change makes SVG file entries in the object browser table use the
image icon. Previously, SVGs used the icon for unknown file types.

Resolves #6306

Change-Id: Ic2a8b6154dc222292f3048d967a0420d5872acd0
2023-09-21 22:27:45 +00:00
Jeremy Wharton
6a4abb7f14 web/satellite/vuetify-poc: make project stat cards clickable
When clicked, cards containing project stats will redirect the user to
the card's respective page.

Resolves #6305

Change-Id: I6a598ad2a8a6ab79f48f559eced55f8f8257a518
2023-09-21 21:47:22 +00:00
Jeremy Wharton
15c0c675b1 web/satellite/vuetify-poc: update "Learn More" links in page subtitles
This change updates the "Learn More" links in the Buckets page,
the Team page, and the project dashboard.

Resolves #6293

Change-Id: I2d9c0ca9b8bbd2991869648d231a8069868efdc0
2023-09-21 21:06:57 +00:00
Jeremy Wharton
123309c648 web/satellite: update vuetify dependency
This change updates the Vuetify dependency from 3.3.12 to 3.3.17.

Change-Id: Id570e28c105a10fb2baca25367a4a3e53ce4bf52
2023-09-21 15:55:40 +00:00
Márton Elek
98921f9faa satellite/overlay: fix placement selection config parsing
When we do `satellite run api --placement '...'`, the placement rules are not parsed well.

The problem is based on `viper.AllSettings()`, and the main logic is sg. like this (from a new unit test):

```
		r := ConfigurablePlacementRule{}
		err := r.Set(p)
		require.NoError(t, err)
		serialized := r.String()

		r2 := ConfigurablePlacementRule{}
		err = r2.Set(serialized)
		require.NoError(t, err)

		require.Equal(t, p, r2.String())
```

All settings evaluates the placement rules in `ConfigurablePlacementRules` and stores the string representation.

The problem is that we don't have proper `String()` implementation (it prints out the structs instead of the original definition.

There are two main solutions for this problem:

 1. We can fix the `String()`. When we parse a placement rule, the `String()` method should print out the original definition
 2. We can switch to use pure string as configuration parameter, and parse the rules only when required.

I feel that 1 is error prone, we can do it (and in this patch I added a lot of `String()` implementations, but it's hard to be sure that our `String()` logic is inline with the parsing logic.

Therefore I decided to make the configuration value of the placements a string (or a wrapper around string).

That's the main reason why this patch seems to be big, as I updated all the usages.

But the main part is in beginning of the `placement.go` (configuration parsing is not a pflag.Value implementation any more, but a separated step).

And `filter.go`, (a few more String implementation for filters.

https://github.com/storj/storj/issues/6248

Change-Id: I47c762d3514342b76a2e85683b1c891502a0756a
2023-09-21 14:31:41 +00:00
Jeremy Wharton
950672ca6c web/satellite/vuetify-poc: implement Manage Passphrase dialog
This change implements the Manage Passphrase dialog in the Vuetify
project. Within it, users can create, switch, or clear the current
passphrase.

Resolves #6284

Change-Id: I2ca87e62b59c0cefd13bf36005c9301b35f12255
2023-09-20 18:36:21 -05:00
Jeremy Wharton
471111122b web/satellite/vuetify-poc: implement Delete Access dialog
This change implements the Delete Access dialog for deleting access
grants. It is triggered by clicking Delete in the actions menu of
access grant table rows.

Resolves #6300

Change-Id: I288f9a88c62e57390f039e41ca6770d2942a9058
2023-09-20 22:08:30 +00:00
Cameron
22261146be cmd/uplink: don't output stacktrace with unknown access
A stacktrace is unnecessary for this error, so don't print one.

https://github.com/storj/storj/issues/5910

Change-Id: Id9e5d9d188042cc7ac0dba571138d5f3e331a9d6
2023-09-20 10:06:47 +00:00
Ivan Fraixedes
926076bffd private/apigen/example: Add description to endpoint
Add the description field to the endpoint to show where it ends up when
generating code.

Change-Id: I415f088dbf795656ed1ef042ed41ebf39a517692
2023-09-19 18:31:20 +00:00
Wilfred Asomani
e599df03a8 satellite/console: add more cross-user api tests
This change adds more endpoints to the cross-user api test.

Issue: https://github.com/storj/storj/issues/6246

Change-Id: I4c3128c3a932c713b10499a8909836a599b79458
2023-09-19 16:50:31 +00:00
Cameron
a2d37bc69a {satellite/console, web/satellite}: send analytics if invitee signs up
send analytics event if project invite link is clicked and if user
signs up.

github issue: https://github.com/storj/storj/issues/5190

Change-Id: I41eee5e679a84b9ec325815655684a98624d5656
2023-09-19 14:50:44 +00:00
Vitalii
81d49ada06 web/satellite: slightly refactored billing modals
Slightly refactored html and css code (it's still weird) of billing modals.
Probably there is no reason to rework it entirely since we're going to migrate to Vuetify project.

Issue:
https://github.com/storj/storj/issues/5220

Change-Id: I25b4b0cdb9d4d24ef3d1f615f2a3471b2d3e727d
2023-09-19 14:01:31 +00:00
Wilfred Asomani
0a063fdeb3 web/satellite/vuetify-poc: require MFA code to generate MFA recovery codes
This change uses the code protected MFA code generation endpoint. It
requires a code from the user before generating new recovery codes.

Issue: https://github.com/storj/storj-private/issues/433

Change-Id: I38c7c6f543a1d0c68aa1c2e9092e76fed2448467
2023-09-19 08:28:10 +00:00
Márton Elek
5bbd477a58 go.mod: bump dependencies (uplink,common,monkit)
Change-Id: I72e17c3acbabd36ffb0a0adbcccec4c252ba710c
2023-09-19 00:35:07 +00:00
Jeremy Wharton
3104a830ae web/satellite/vuetify-poc: unmock list of owned and shared projects
This change shows real projects in the Project dropdown of the project
navigation sidebar, replacing mock data.

Resolves #6283

Change-Id: Id8eef6cc02b5b773f89a00d41d5335ae2feb1729
2023-09-18 23:53:43 +00:00
Michal Niewrzal
5d0934e4d9 satellite/metabase/rangedloop: disable ranged loop for tests
Currently each testplanet test is running ranged loop no matter if
it's used or not. This is small change with some benefits like:
* saves some cpu cycles
* less log entries
* ranged loop won't interfere with other systems

Change have no big impact on tests execration but I believe it's nice to
have.

Change-Id: I731846bf625cac47ed4f3ca3bc1d1a4659bdcce8
2023-09-18 22:51:10 +00:00
Jeremy Wharton
95d87f5a22 web/satellite/vuetify-poc: remove Browse from project nav sidebar
This change removes the Browse item from the project navigation
sidebar. This item didn't do anything, and we have no plans to use it.

Resolves #6281

Change-Id: Ifed6a563e38585955b0bf180c2896d6d224cb7dc
2023-09-18 21:58:26 +00:00
Andrew Harding
fd679c329c private/server: FreeBSD TCP fastopen support detection
Detects whether or not TCP fastopen is supported by running sysctl to
grab the current value of net.inet.tcp.fastopen.server_enable.

If not enabled, directs the operator to use sysctl to set it temporarily
and /etc/sysctl.conf to set it on-boot.

Setting the socket option always works whether it is enabled or not.

Verified that fastopen works on FreeBSD 13. Did not attempt on earlier
versions but support has been there since FreeBSD 10.

Change-Id: I2e0c457558a6fa7b7a1b18bc3c6684aff50b81a2
2023-09-18 21:17:42 +00:00
Jeremy Wharton
ad5c2e171a web/satellite/vuetify-poc: add tier indicator
This change indicates the user's tier in the My Account dropdown button
of the Vuetify project.

Resolves #6278

Change-Id: I0db6bfe8e03720b87ff77e947f785031eed7e528
2023-09-18 15:25:53 -05:00
Egon Elbre
c48f58e968 satellite/metabase: simplify BeginExactObject
There's only one value that BeginExactObject should return and there's
no point in restating that.

Also, use a clearer value for the random object version.

Change-Id: I06b26ad87d64e1b04b48458f624edd630f7f2f1d
2023-09-18 19:26:49 +00:00
Jeremy Wharton
109c0d5e37 web/satellite/vuetify-poc: make Resources dropdown items functional
This change adds functionality to the items in the project navigation
sidebar's Resources dropdown menu.

Resolves #6282

Change-Id: I64e0d472592ec07545034adff6a3c2122fcc113c
2023-09-18 18:47:05 +00:00
Jeremy Wharton
012e1fbc06 cmd/uplink: use configured user agent
This change fixes an issue where the configured user agent wasn't used
by Uplink commands.

Resolves storj/customer-issues#999

Change-Id: I2d9f38308eddad7c471a100c968082783c05a3b3
2023-09-18 18:00:34 +00:00
Wilfred Asomani
f42548ac1c web/satellite: require MFA code to generate MFA recovery codes
This change uses the code protected MFA code generation endpoint. It
requires a code from the user before generating new recovery codes.

Issue: https://github.com/storj/storj-private/issues/433

Change-Id: I248649567a4800374b84ee512a79195ea2c44652
2023-09-18 16:54:20 +00:00
Wilfred Asomani
8ad0bc5e61 satellite/console: add alt code protected MFA recovery endpoint
This change adds an alternate MFA code recovery endpoint that requires
MFA code to generate codes.

Issue: https://github.com/storj/storj-private/issues/433

Change-Id: I10d922e9ad1ace4300d4bcfea7f48494227f1ff8
2023-09-18 16:07:43 +00:00
Wilfred Asomani
1a8913e7a0 web/satellite/vuetify-poc: improve navigation drawer on small screens
This change improves the behaviour of the navigation drawer. It is now
able to automatically close on resize to smaller screen sizes and vice
versa, and also by close by clicking outside the drawer.

Issue: https://github.com/storj/storj/issues/6266

Change-Id: I1aee465a546abbf1369c48f6827b058523c5da21
2023-09-18 15:02:22 +00:00
Andrew Harding
e4d6829971 private/server: Windows TCP fast-open support detection
Implements tryEnableFastOpen by creating a localhost socket and enabling
the TCP_FASTOPEN socket option. On builds where TCP_FASTOPEN isn't
available, setting the socket option fails and tryEnableFastOpen returns
false.

This was verified on a supporting build (latest Windows 10) and an
unsupporting build (Windows 8.1, couldn't find an ISO for an older
Windows 10 build).

Change-Id: I497117dc2f04acdd2b0cc836e20d12d69076b939
2023-09-18 14:15:30 +00:00
Vitalii
ecc527ad3b web/satellite: low STORJ balance notification
Added a new banner to the dashboard of all projects. This banner is displayed when a user meets the following conditions: they have no credit cards on file, they have a payment history with tokens, and their estimated charges are higher than their current balance.

Issue:
https://github.com/storj/storj/issues/6234

Change-Id: I1f90ae81032d459111b111d23ce2e1d8119e649d
2023-09-18 16:30:59 +03:00
Egon Elbre
bc517cae2f go.mod: bump common
This brings in common/grant that doesn't depend on protobuf anymore.
This ends up causing the console wasm bundle from ~11MB to ~4.7MB.

Change-Id: I145dcb9239952a7a9e352c8793c111acb61ff0cc
2023-09-15 17:43:44 +00:00
Jeremy Wharton
8a0d1e8b55 web/satellite/vuetify-poc: show region in My Account dropdown
This change displays the satellite's region in the My Account dropdown
menu of the Vuetify project.

Resolves #6279

Change-Id: Ib7d739c0104fb20ad7ee23234c66813b7f37b3a5
2023-09-15 16:31:21 +00:00
Wilfred Asomani
1efc0ceaa5 web/satellite: load usage and charges separately
This change separates the loading of usage and charges from other API
calls since that may take long. This will allow the other data to be
displayed while usage and charges are waited on.

Issue: https://github.com/storj/storj/issues/6259

Change-Id: I4a8d8f911baf432d6f1e9eee49176480197ae3ca
2023-09-15 15:50:02 +00:00
Jeremy Wharton
789b37c21f web/satellite/vuetify-poc: remove notification on close button click
This change fixes an issue where a notification would not be completely
removed after clicking the close button, causing it to invisibly prevent
elements beneath it from being clicked. Also, the file preview dialog no
longer closes when a notification is clicked.

Resolves #6280

Change-Id: I135aa2e77ddc1ec845101209f26dde5e48f10bd6
2023-09-15 15:10:48 +00:00
Vitalii
bd36749f7d web/satellite: improve bucket list on small screens
Bucket list in vuetify never requires horizontal scrolling to view the entire row.
Removed non-essential columns from the bucket list when the screen size is small.
Bucket name is the only "essential" piece of information that must always be shown.

Issue:
https://github.com/storj/storj/issues/6232

Change-Id: Id4bee3100f6d4ca112670d2f68bd63ff2dc266e9
2023-09-15 17:28:40 +03:00
Michal Niewrzal
975d953cb8 satellite/metabase: custom error for commit object
We stoped returning lots of errors as is to avoid leaking our internals
but some errors were meanigful for client. Example of such error is
"exceeded maximum number of parts". With this change we are wrapping
some important commit object errors with new ErrFailedPrecondition
error to be able to return it easily to uplink.

Change-Id: Id834b78362ed1920f0c3f6f1c7d9587bfd27e36a
2023-09-15 09:52:42 +00:00
Márton Elek
f4fe983b1e satellite/{placement,nodeselection}: introduce empty() and notEmpty() for tag value selection
It helps to implement rules like `tag("nodeid","select",notEmpty())

Change-Id: If7a4532eacc0e4e670ffe81d504aab9d5b34302f
2023-09-14 19:30:29 +00:00
Vitalii
92a69c7de4 web/satellite: add loader to object browser table
This is a fix based on early feedback from QA team.
Added loader to object browser table so that user can't change pages while request is still in progress because it breaks pagination.

Change-Id: I5cc2ff057955478b3c745c169d520e1a639eff92
2023-09-14 18:50:51 +00:00
Wilfred Asomani
bd48a5cbe6 web/satellite: allow limit increase request
This change implements a modal to request limit increases.

Issue: https://github.com/storj/storj/issues/6233

Change-Id: I40ace89a79c65751547d804e8d190e866217d379
2023-09-14 18:08:56 +00:00
Jeremy Wharton
0f3ff66485 web/satellite: show error for rate limited MFA logins
This change fixes an issue where errors were not displayed for login
attempts that failed due to rate limiting.

Change-Id: Ia3c7fccf434ad62bb252f4215676b1f32903ac53
2023-09-14 15:20:32 +00:00
Michal Niewrzal
881137539c satellite/metabase: commit object is not respecting expiration time
With pending_objects table support enabled we missed passing correctly
expiration time from pending object to committed object. This change
updates commit query to take into account expiration time.

Change-Id: I67146d5b2f7f0bda02925d16275fbc59acb705bd
2023-09-14 16:25:28 +02:00
Vitalii
8f27425284 satellite/{projectaccounting, web}: merge settled and allocated lines for bandwidth graph
Merged bandwidth graph lines to show only allocated-dead for last 3 days and settled for other days.

Issue:
https://github.com/storj/storj/issues/6072

Change-Id: Ic7f03d22ccd82d27ae6e6a85e73e144c9852e33b
2023-09-14 15:39:56 +03:00
Jeremy Wharton
ec8f3b4528 web/satellite/vuetify-poc: allow for sharing files and folders
This change allows linksharing URLs to be generated for files and
folders within the Vuetify project's file browser.

Resolves #6111

Change-Id: I8cbe81b33cb5e35de0c34bba8ccc9175c727bd94
2023-09-14 11:23:08 +00:00
Michal Niewrzal
8f1682941e mod: bump storj/uplink to v1.12.0
Change-Id: Ia58de16d4ac43c7b888000ad8ca85a979496a2cb
2023-09-14 09:20:38 +02:00
Wilfred Asomani
ccb9b7ae8e satellite/web/vuetify-poc: make tables consistent
This change makes tables in the vuetify app more consistent. Also
clearing search has been fixed for tables whose data would not populate
after search has been cleared.

Issue: https://github.com/storj/storj/issues/6267

Change-Id: I053d9e5f23662774c60d67a29f814a2c1c3067ed
2023-09-13 23:15:28 +00:00
Clement Sam
f14fabc90a cmd/storagenode: add forget-satellite subcommand
This change adds a new forget-satellite sub-command to
the storagenode CLI which cleans up untrusted satellite
data.

Issue: https://github.com/storj/storj/issues/6068
Change-Id: Iafa109fdc98afdba7582f568a61c22222da65f02
2023-09-13 19:06:55 +00:00
Wilfred Asomani
dcf3f25f93 satellite/admin: update README
add descriptions for the endpoint that removes a user from the waning
state.

Issue: https://github.com/storj/storj/issues/6118

Change-Id: I211cd3c41c7fefa295d0db1b9f43f53e33b984e6
2023-09-13 17:50:28 +00:00
Jeremy Wharton
7d8b231aaf cmd/satellite: build Vuetify project in Dockerfile
This change updates the satellite Dockerfile to build the Vuetify
project.

References #6251

Change-Id: I699360c0f7eb7a8abdd0bc523ee74910a04fecd3
2023-09-13 12:43:58 +00:00
Michal Niewrzal
fb04a22088 satellite/metainfo: flag to rollout pending_objects table support
To avoid enabling feature for every project at once we would like to
do this partially and control percentage of projects that will have
feature enabled.

https://github.com/storj/storj/issues/6258

Change-Id: Iaac7c42d39da76ed2ecc439847c3b210462befa5
2023-09-13 10:33:23 +00:00
Egon Elbre
d91ee440ba satellite/metabase: use constant for pending version
Currently it wasn't quite clear what was a stub version and an actual
version. Use a PendingVersion constant to make this distinction clear.

Also use PendingVersion = NextVersion = 0, that way it's clearer that
the version hasn't been yet determined. DefaultVersion = 1 might imply
that the object will get that version once commited, however that will
entirely depend on whether use-pending-objects is used or versioning is
enabled or not.

Change-Id: I21398141f97035c48c778f23b542266b834c44f1
2023-09-12 18:01:12 +00:00
Jeremy Wharton
4f8697568d web/satellite: use edge service URL overrides
This change makes the satellite frontend use edge service URL overrides
if they have been configured for a project.

Resolves #6188
Resolves #6190

Change-Id: I4c8fb3f5f00f450fb8cd139383972ab622234fb0
2023-09-12 12:19:20 -05:00
Jeremy Wharton
c8f4f5210d satellite/console: return edge URL overrides in project info responses
API responses containing project information now contain the edge
service URL overrides configured for that project. The overrides are
based on the project's default placement.

References #6188

Change-Id: Ifc3dc74e75c0f5daf0419ac3be184415c65b202e
2023-09-12 12:10:18 -05:00
Clement Sam
89d682f49f satellite/metainfo: prevent internal DB errors in Public API
Resolves https://github.com/storj/storj/issues/6081

Change-Id: I0e530db39947138dcafc1b6bd1710ff1ca96b8c5
2023-09-12 15:12:44 +00:00
Michal Niewrzal
e21978f11a satellite/metainfo: support getting specific object version
Protobuf definition is read to support getting specific version of
object so we just need to wire requested version into metainfo.GetObject
endpoint.

https://github.com/storj/storj/issues/6221

Change-Id: If4568b82119a6c893788a0a86e598b05ff5951cf
2023-09-12 14:27:40 +00:00
Vitalii
4e4da7be6d web/satellite/vuetify-poc: always show details container under validatable inputs
Inputs that have some custom validation messaging attached should not use hide-details=auto to prevent 'jumpy' visual experience.

Issue:
https://github.com/storj/storj/issues/6230

Change-Id: Ia90e122516eb853a3908c0f57634971243fb38b3
2023-09-12 13:47:22 +00:00
Jeremy Wharton
088496efdf web/satellite/vuetify-poc: allow for deleting files and folders
This change allows files and folders to be deleted from within the
Vuetify project's file browser.

Resolves #6106

Change-Id: I0d7b0528b08333aeec29917c4ebef6ea966ac1fa
2023-09-12 13:08:09 +00:00
Egon Elbre
28ee6f024c satellite: don't use fmt.Print in tests
Change-Id: Ia10450240ad075c9d78614adff9164f292fb1fa0
2023-09-12 15:11:02 +03:00
Márton Elek
afa5c54a35 satellite/satellitedb: add placement column to repair_queue
It makes it possible to run dedicated repair worker for different placement definitions.

https://github.com/storj/storj-private/issues/400

Change-Id: I376da867da5dbb4ab392d5f86c766f7543c32ee6
2023-09-12 11:29:18 +00:00
Vitalii
2cf233ac23 web/satellite/vuetify-poc: hide nav sidebar by default for screens less than 1280px
On screen sizes where the sidebar overlaps with the content of the page, the sidebar should be collapsed by default.

Issue:
https://github.com/storj/storj/issues/6229

Change-Id: Ia0a91acd95519de27f9ff8f1ee90c6b8e7932266
2023-09-12 13:48:53 +03:00
Cameron
f9ab2f0de7 satellite/console: prefix oidc paths with /api/v0/
add /api/v0/ prefix to oidc paths to route requests to the api
pod with split UI infrastructure.

Change-Id: I1e7e691487ab1d4e84434d204d10b7f944ae8873
2023-09-11 18:12:39 +00:00
Egon Elbre
487f64e164 satellite/satellitedb,multinode/multinodedb: update to latest dbx
Change-Id: I500df6d0541706c3960d4560721c3783d0d049ff
2023-09-11 17:21:02 +00:00
Egon Elbre
8edb9c5f98 multinode/multinodedb: don't generate rx
Change-Id: Ica1a2bb1ab42e881daa951ab61b1dd99e877aff5
2023-09-11 16:40:52 +00:00
Egon Elbre
87bfb3b02b satellite/satellitedb: don't generate rx
I'm not sure what Rx is, however, we aren't using it --
so let's remove it for now.

Change-Id: I9caacbc150479f93945477101528a4fd60ea865f
2023-09-11 16:00:15 +00:00
Egon Elbre
3e73d414d1 satellite/console/consoleweb: initialize mime lazily
Change-Id: I80b78edcf057acef9b5a599cb77308baddc07692
2023-09-11 15:19:56 +00:00
Michal Niewrzal
0a3ee6ff8a satellite/metabase: remove old object segments on overwrite
While adding support for pending_objects table one case was missed.
When we are uploading object to location where old objects exists
we are not removing old object segments at all. Old object is
overwritten with new object metadata but segments remains without
corresponding object. This fix removes all existing committed objects
(with it's segments) before committing new object.

https://github.com/storj/storj/issues/6255

Change-Id: Id657840edf763fd6aec8191788d819191b074fb7
2023-09-11 14:16:47 +00:00
Vitalii
c31fb9c1cf satellite/payments: restrict addition of duplicate credit cards
By this change we don't allow users to add credit cards that are already bind to their account.
We still allow the same CC number but with a different expiration date.

Issue:
https://github.com/storj/storj/issues/5597

Change-Id: Ifeb0cc5ae0c2f0f7596af4dead70ae7d20d30613
2023-09-11 13:24:43 +03:00
Kaloyan Raev
3119b614ae cmd/uplink: --max-object-ttl flag for share and access restrict commands
Context: https://github.com/storj/storj/issues/6249

Change-Id: Ic65a1d8aef61f1a88752a7b12a23fb854dac8f6d
2023-09-11 08:14:55 +00:00
Vitalii
9254dd2208 web/satellite/vuetify-poc: enable object browser pagination
Enabled object browser pagination for vuetify app.
Also fixed some small bug when returning to first page.

Issue:
https://github.com/storj/storj/issues/5595

Change-Id: I8b5e90a4cd7d7a79a8beeb292b7374db3f93d700
2023-09-09 00:36:55 +00:00
Wilfred Asomani
2bf4113821 web/satellite/veutify-poc: create folder
This change adds the ability to create folders in the file browser.

Issue: https://github.com/storj/storj/issues/6105

Change-Id: I0dae0f9874b571cfd0ae79b2b994b58149d70aa3
2023-09-08 23:57:13 +00:00
Wilfred Asomani
54379fc0ee satellite/{console,analytics}: allow limit increase request
This change adds a  new endpoint that submits limit increase requests
to segment.

Issue: https://github.com/storj/storj/issues/6233

Change-Id: Ie4f70aef31079acbe2f24771b3ea359d5769eb95
2023-09-08 23:17:38 +00:00
Wilfred Asomani
cd7d9cf079 satellite/{console,payments}: unfreeze user on token payment
This change unfreezes/unwarns users in either state after successful
payment with tokens.

Change-Id: I7494d6a33cf9383576f42af695df522d8f409e03
2023-09-08 12:51:01 +00:00
Cameron
c52554a2b9 web/satellite: vuetify upload file and folder
add functionality to upload files and folders in object browser

issue: https://github.com/storj/storj/issues/6101
issue: https://github.com/storj/storj/issues/6102

Change-Id: I4aa86b89adc051b91b0d7fde69dd7375b2a3f370
2023-09-08 11:47:01 +00:00
Kaloyan Raev
4e499fb9bf satellite/metainfo: respect MaxObjectTTL in BeginObject
If MaxObjectTTL is set in the API key, BeginObject will use it for the
object expiration time, unless an explicit ExpireAt is available in the
request.

Context: https://github.com/storj/storj/issues/6249

Change-Id: I2adf57d979a9c68eec3a787f3739d2f1dbec1f7e
2023-09-08 09:30:48 +00:00
Moby von Briesen
8d1a765fd6 satellite/console: Partially revert change to remove graphql
This partially reverts commit 516241e406.

Endpoints are added to the backend, as there are some customers who may
use these endpoints, even though they are no longer necessary for the
satellite UI.

Change-Id: I52a99912d9eacf269fbb2ddca603e53c4af6d6bf
2023-09-07 20:50:24 +00:00
Wilfred Asomani
754bf5f8af Revert "satellite/db: optimize project usage query"
This reverts commit 31ec421299.

This change made the usages endpoint slower for accounts with large
number of projects.

Change-Id: I95870e95c2bf3bc3050087532fd0d20cbb50748b
2023-09-07 19:27:08 +00:00
Michal Niewrzal
df037564d7 satellite/zombiedeletion: remove inactive uploads from pending_objects
With zombie deletion chore we are removing inactive pending objects from
objects table but new we need also to do this for pending_objects table.

https://github.com/storj/storj/issues/6050

Change-Id: Ia29116c103673a1d9e10c2f16654022572210a8a
2023-09-07 18:47:29 +00:00
Márton Elek
ad87d1de74
satellite/satellitedb/overlaycache: fill node tags with join for limited number of nodes
The easiest way to get node information WITH node tags is executing two queries:

 1. select all nodes
 2. select all tags

And we can pair them with a loop, using the in-memory data structures.

But this approach does work only, if we select all nodes, which is true when we use cache (upload, download, repair checker).

But repair process selects only the required nodes, where this approach is suboptimal. (full table scan for all tags, even if we need only tags for a few dozens nodes).

Possible solutions:

 1. We can introduce a cache for repair (similar to upload cache)
 2. Or we can select both node and tag information with one query (join).

This patch implements the second approach.

Note: repair itself is quite slow (10-20 seconds per segements to repair). With 15 seconds execution time and 3 minutes cache staleness, we would use the cache only 12 times per worker. Probably we don't need cache for now.

https://github.com/storj/storj/issues/6198

Change-Id: I0364d94306e9815a1c280b71e843b8f504e3d870
2023-09-07 19:27:53 +02:00
Kaloyan Raev
82b108de69 satellite/console/consolewasm: no direct cast to grant.Permission
A new field is introduced to grant.Permission in storj.io/common. Having
a direct cast here leads to compilation problems when bumping
storj.io/uplink to the latest storj.io/common. Avoiding the direct cast
resolves the issue.

Context: https://github.com/storj/storj/issues/6249

Change-Id: I3b9bc14ebcce8e192e218c621b996300753b8de4
2023-09-07 13:27:53 +03:00
Moby von Briesen
6195b8cd52 satellite/admin: support more options for passing project ID
This change does two things:
* allow using either public ID or private ID to do project-related
  requests in admin UI
* allow passing a UUID string not containing dashes (i.e. a pure hex
  string) in order to do project-related requests in admin UI

Change-Id: I4807a5d7252a48f4a09e3966c406645d55c856e2
2023-09-07 08:53:41 +00:00
Dan Willoughby
091c72319a web/satellite: update welcome email links
Change-Id: Ic918a7ec30db1753d780643492704fc10aafeb99
2023-09-06 15:18:51 -06:00
Wilfred Asomani
7311d08139 web/satellite/vuetify-poc: upload via drag and drop
This change adds the drag-drop upload feature to the vuetify app.

Issue: https://github.com/storj/storj/issues/6104

Change-Id: I177e33a677d94db9ef95a31e32da853a46a7dc51
2023-09-06 19:20:50 +00:00
Vitalii
623b989973 web/satellite/vuetify-poc: unmock STORJ txs table on billing screen
Show real STORJ token transactions on billing screen in vuetify app.

Issue:
https://github.com/storj/storj/issues/6098

Change-Id: I1d7c2a613fefbf68c7ce3b8f62ec7ee992885bc4
2023-09-06 18:31:05 +03:00
Vitalii
6e1fd12930 web/satellite: paginate ListObjects request to show more than 1000 objects
With this change, we are able to fetch all objects to show in the object browser.
AWS SDK V3 provides paginator functionality to automatically make additional requests for every MaxKeys value (we use 500 objects at a time).
By initial request we fetch first 500 objects and save continuation tokens for the rest of the object batches.
Also, we save currently active (fetched) object range.
If user tries to open a page with objects which are out of currently active range then we look for needed continuation token and fetch needed objects batch.
Added a feature flag for this funtionality.

Issue:
https://github.com/storj/storj/issues/5595

Change-Id: If63e3c2ddaac3ea9f2bc1dc63cb49007f897e3e2
2023-09-06 12:47:15 +00:00
Márton Elek
f40baf8629
go.mod: bump dependencies (private,uplink,common)
Change-Id: I6c55735b45cadaf36697eff53e78b5b09afe9dea
2023-09-06 13:28:22 +02:00
Clement Sam
9ab934e2ae storagenode/piecestore: implement trash recovery for download requests
This change allows a node to look for a piece in the trash when
serving a download request.

If the piece is found in the trash, it restores it to the blobs
directory and continue to serve the request as expected.

Resolves https://github.com/storj/storj/issues/6145

Change-Id: Ibfa3c0b4954875fa977bc995fc4dd2705ca3ce42
2023-09-05 23:04:21 +00:00
Wilfred Asomani
3f1ea4a0b9 web/satellite/vuetify-poc: list invoices
This change lists invoices on the vuetifypoc billing history tab.
It also removes invoice filtering on the main app.

Issue: https://github.com/storj/storj/issues/6099

Change-Id: Id4cc2db003a0208775ddaefc87abf26f4b05106c
2023-09-05 15:01:16 +00:00
Wilfred Asomani
775db4aa3c satellite/payments: filter out draft invoices
This change filters the list of invoices sent to the frontend to only
contain open or paid invoices.

Change-Id: I9ac2640a7587a76b0baf46d941f799e742aa2b3b
2023-09-05 14:22:30 +00:00
Jeremy Wharton
4e3e31a425 satellite/console/consoleweb/consoleapi: refactor api requests in tests
This change refactors the way requests are sent in console API tests,
placing identical logic in a dedicated function to reduce code
duplication.

Change-Id: I7a5ac42d8d68a3fd9a9f8b9d61775659234e883f
2023-09-05 11:17:02 +00:00
Wilfred Asomani
2d18f43b6a web/satellite/vuetify-poc: show file previews
This change enables files to be previewed.

Issue: https://github.com/storj/storj/issues/6108

Change-Id: I3045950281822620de96b86e0180eb0d24e2d629
2023-09-05 10:35:31 +00:00
Márton Elek
e2006d821c satellite/overlay: change Reliable and KnownReliable
as GetParticipatingNodes and GetNodes, respectively.

We now want these functions to include offline and suspended nodes as
well, so that we can force immediate repair when pieces are out of
placement or in excluded countries. With that change, the old names no
longer made sense.

Change-Id: Icbcbad43dbde0ca8cbc80a4d17a896bb89b078b7
2023-09-02 23:34:50 +00:00
Cameron
6896241933 satellite/console: fix unverified invite email
When checking if invited user is unverified, initialize oldest
with unverified row rather than empty User, because empty User
CreatedAt is zero, so no real user could be created earlier.

Change-Id: I74dd8f7fc82951cbb61071632a74b1a9443b41fe
2023-09-01 15:11:58 -04:00
Michal Niewrzal
c9591e9754 satellite/metainfo: better metabase errors handling
Some errors were returned as metabase errors, not pure drpc
errors because of how rpcstatus.Code method is working. Status
code was returned for errors like metabase context canceled but
we would like to not leak our internals to the client.

Change-Id: I3f0194755f8d7359b1e3d342fa3be3d984019ecb
2023-09-01 11:32:10 +00:00
Moby von Briesen
b4c95a3b28 private/apigen/tsgen: Set query params better
The old way did not properly handle escaping, e.g. if the value of a
query param contained `&` or `=` inside it. By using
url.searchParams.set, we can safely add these types of arguments to the
path.

Change-Id: I62d3883b14f9d5a517e4a3d58f019014b46fd1b4
2023-08-31 20:09:29 -04:00
Jeremy Wharton
28d498f91d web/satellite/vuetify-poc: allow file browser downloads
This change allows files to be downloaded from within the file browser
of the Vuetify project.

Resolves #6107

Change-Id: I0ac0384711baccb99c0a6d382fe96f318290789b
2023-08-31 22:41:52 +00:00
Jeremy Wharton
b671641a28 satellite/console: update CSP to include storjapi.io
This change updates our content security policy to include the domain
storjapi.io and all of its subdomains.

References #6188

Change-Id: I6f3073bc32aa99626c54caf00bf07d2253ccbb8f
2023-08-31 22:02:45 +00:00
Márton Elek
c202929413
satellite/nodeselection: rename (NodeFilter).MatchInclude to Match
As I learned, the `Include` supposed to communicate that some internal change also "included" to the filters during the check -> filters might be stateful.

But it's not the case any more after 552242387, where we removed the only one stateful filter.

Change-Id: I7c36ddadb2defbfa3b6b67bcc115e4427ba9e083
2023-08-31 16:17:52 +02:00
Wilfred Asomani
dcc4bd0d10 satellite/{console,payments}: freeze/warn storjscan users
This change enables the freezing/warning of users who use storjscan.

Issue: https://github.com/storj/storj/issues/6164

Change-Id: I7b00ee09d6527b3818b72326e9065c82ef5a2ac8
2023-08-31 13:22:21 +00:00
Márton Elek
ca0ea50cba satellite/overlay: remove/deprecate NodeSelectionCache.Disabled
Once uppon a time, at the dawn of the implementation of Storj, when all the nodes are read from the database directly, every time.

After a while --  due to performance reasons -- it has been changed for upload and download: where all the nodes are read for a short period of time, and used from memory.

This is the version which was improved recently to support advanced node selections using placement.

But stil we have an old configuration value `service.config.NodeSelectionCache.Disabled`, and the db based implementation: `service.FindStorageNodesWithPreferences(ctx, req, &service.config.Node)`.

For safety, we need to remove this option, to make sure that we use the cache, which has the advanced features.

This patch was supposed to be a very small one (just removing a method and a config: https://review.dev.storj.io/c/storj/storj/+/11074/1/satellite/overlay/service.go), but it turned out that we need to update a lot of unit tests.

These unit tests used the old implementation (which is not used in production any more).

The tests which used both implementation are just updated to use only the new one
The tests which used only the old implementation are refactored (but keeping the test cases).
Using real unit tests (without DB, working on OSX, fast)

Closes https://github.com/storj/storj/issues/6217

Change-Id: I023f92c7e34235665cf8474513e67b2fcc4763eb
2023-08-31 09:46:29 +00:00
Jeremy Wharton
00194f54a2 web/satellite: show limit notifications to paid tier users
This change causes paid tier users to see notifications in the project
dashboard when their usage is approaching or has reached their maximum
or custom usage limits.

Change-Id: I7b68fcdd7d62797b6b26869e109cfb0b193fdddb
2023-08-31 07:31:09 +00:00
Wilfred Asomani
31ec421299 satellite/db: optimize project usage query
This change addresses an issue where the /charges endpoint will take a
while to respond due to a project having a large number of buckets.
The method and queries involved have been optimized and benchmarks show
a performance improvement.

test name                            old ms/op  new ms/op
Postgres/sum_all_partner_usages          3.659      1.101
Postgres/individual_partner_usages        3.74      1.299
Cockroach/sum_all_partner_usages         7.201      2.872
Cockroach/individual_partner_usages      7.247      2.852

Issue: https://github.com/storj/storj-private/issues/277

Change-Id: Ia5082a2e1c3e91120a9db7b01c18847fe04574fe
2023-08-30 22:08:11 +00:00
Jeremy Wharton
0c9c37875d web/satellite/vuetify-poc: unmock file browser table entries
This change shows real file entries in the file browser table,
replacing the mock data. Sorting, searching, and folder navigation have
been implemented.

Resolves #6199

Change-Id: I7360879d2e26605489c20f9d094c3f231fee49cd
2023-08-30 21:12:55 +00:00
Jeremy Wharton
ebfbbca1be web/satellite/vuetify-poc: pull latest changes from internal repo
The changes as of storj/vuetify-storj@2312936 have been pulled into the
Vuetify project.

Change-Id: Ia45d3d0832727d3ca7b9ee9b3dce5949a79a5d31
2023-08-30 20:34:13 +00:00
Wilfred Asomani
fecaa6a71a web/satellite/vuetify-poc: enable claiming wallets
This change adds the ability to claim STORJ wallets and display token
balance.

Issue: https://github.com/storj/storj/issues/6096

Change-Id: Ifc89b586c0e3ed876905ff0a5b270e718cbb689c
2023-08-30 14:33:35 +00:00
Michal Niewrzal
67371c43bd satellite/metainfo: enable UsePendingObjectsTable by project
This small feature will give us ability to test pending_objects table
without enabling it globally.

Change-Id: I802f45987ad329f94adfc0f02957c802b21d8251
2023-08-30 15:44:29 +02:00
Michal Niewrzal
780c0e0b35 satellite/metainfo: adjust ListPendingObjectStreams to pending_objects
table

New method IteratePendingObjectsByKeyNew is used to provide results for
metainfo.ListPendingObjectStreams. This endpoint is used to list
pending objects with the same object key. In this case to support
both tables (objects, pending_objects) we need to do one query per table
and merge results.

Because existing metainfo protobuf API is missing some fields to have
proper listing cursor we are not able to make ListPendingObjectStreams
correct for returning more than single page. We need to fix it
separately.

With this change also turns out that approach to merge results from
listing objects for ListObjects method was wrong and this change is also
fixing this problem.

Handling both tables will be removed at some point and only
pending_objects will be used to look for results.

Part of https://github.com/storj/storj/issues/6047

Change-Id: I8a88a6f885ad529704e6c032f1d97926123c2909
2023-08-30 13:35:54 +00:00
Wilfred Asomani
6e3da022e0 web/satellite: add pagination to billing history
This change adds pagination to the billing history table. It uses the
new invoice-history endpoint since we only list invoices in this table.

Issue: https://github.com/storj/storj/issues/5479

Change-Id: I192d58503434203808a23a7c18e8d1feb6afc73f
2023-08-30 03:19:31 +00:00
Vitalii
614d213432 web/satellite: fix duplicate satellite selection on login and forgot password pages
Remove duplicate satellite in selection dropdown on login and forgot password pages.

Issue:
https://github.com/storj/storj/issues/6121

Change-Id: I81bd0c2c29c695ed9ed7f0d648311448cdc9bf26
2023-08-30 01:17:14 +00:00
Cameron
a5b1c0432f satellite/console: send unverified user activation link in project invite
When an unverified user is sent a project invitation it contains a
registration link currently. Instead, send an activation link.

github issue: https://github.com/storj/storj/issues/6033

Change-Id: I54b88de8347a2532f7a85372c0c5e4df4bf4eb38
2023-08-29 12:54:09 -04:00
Vitalii
d6e0987dd9 web/satellite: added info message to sharing modals
Add notification explaining that the share will be public to anyone with the link.
Remove the cancel button as it may be interpreted as cancelling or revoking the shared access.
Make the copy link full width block button.

Issue:
https://github.com/storj/storj/issues/6213

Change-Id: I7b9580b3d8135802c36af8e68f46630b499ab110
2023-08-29 12:55:26 +03:00
Wilfred Asomani
6219aba40c satellite/{console,consoleweb,consoleapi}: add new endpoint for paged invoices
This change adds a new endpoint for listing invoices for billing history
This endpoint will replace the billing-history endpoint used on the
front end since were only interested in listing invoices.

Issue: https://github.com/storj/storj/issues/5479

Change-Id: I4730f5dc497245c6730e60b7f9986554479d1d3b
2023-08-28 23:24:12 +00:00
Vitalii
f0829d5961 web/satellite: use stripe as ES module
Start using @stripe/stripe-js lib (ES module) instead of regular stripe dependency.
Use strict typing for stripe commands/events.
This lib makes us able to modify stripe input styling in the future.

Change-Id: Iaba4f32a42e87edc85a4fbad82e5107c21bf19b6
2023-08-28 22:45:52 +00:00
Wilfred Asomani
f1e8cdfe3e web/satellite/vuetify-poc: add project usages to billing
This change adds project usage and cost per project to the vuetify app.

Issue: https://github.com/storj/storj/issues/6117

Change-Id: I8921aacb6bb24b41794008100ea6e52deed76b60
2023-08-28 22:05:00 +00:00
Vitalii
4964ca5ffb web/satellite: don't show CTAs on limit cards if user is not project owner
Hide CTAs on limit cards if user is not the owner of selected project.
We do this because user can't update limits if they are not the owner of the project.

Issue:
https://github.com/storj/storj/issues/6214

Change-Id: Ib6b564cd6afc1b4bed08ee9b26108f803f0ffb89
2023-08-28 21:16:42 +00:00
Michal Niewrzal
c010e37374 satellite/metainfo: adjust ListObjects to use pending_objects table
Adjust metainfo.ListObjects method to use IteratePendingObjects to
support new pending_objects table. New method will be used only when
we are listing pending objects.

Because until objects table will be free from pending objects we can
have results in both tables we are merging listing results. This also
means that in some (rare?) cases we may return more results than
specified listing limit. This situation is temporary.

Part of https://github.com/storj/storj/issues/6047

Change-Id: I06389145e5d916c532dfdbd3dcc9ef68ef70e515
2023-08-28 16:22:54 +00:00
Egon Elbre
273ebd61d7 private/apigen/example: make it nicer
Change-Id: I1bd779090a902ed2b99b3993dc7cf61fc250f10f
2023-08-28 15:13:46 +00:00
Ivan Fraixedes
2d8f396eeb private/apigen: Make API base path configurable
Previously the base path for the API was hardcoded to `/api` and the
specified version.

This was not obvious that the generated code was setting that base path
and it was not flexible for serving the API under a different path than
`/api`.

We will likely need to set a different base path if we pretend to serve
the new back office API that we are going to implement alongside the
current admin API until the new back office is fully implemented and
verified that works properly.

This commit also fix add the base path of the endpoints to the
documentation because it was even more confusing for somebody that wants
to use the API having to find out them through looking to the generated
code.

Change-Id: I6efab6b6f3d295129d6f42f7fbba8c2dc19725f4
2023-08-28 14:35:01 +00:00
Cameron
ec42fdae6d web/satellite: add bucket details dialog
issue https://github.com/storj/storj/issues/6115

Change-Id: I4dd3b93bd43f4871cd28ab595bace9e8ce59b7c2
2023-08-28 13:09:03 +00:00
Michal Niewrzal
b26df035f9 satellite/metainfo: tests for new GetBucketLocation method
Change-Id: I809ebab51606aa9e55dff3c40ef2e865caf06924
2023-08-28 11:49:23 +00:00
Artur M. Wolff
37d6df23fa satellite: implement metainfo.GetBucketLocation endpoint
Updates storj/storj-private#408
Updates storj/storj-private#409

Change-Id: Idaaca74b4a5c9c7907d095e0a3a5f29e52843ce6
2023-08-28 13:48:07 +02:00
Márton Elek
5c12a3406d satellite/nodeselection: improve annotation composability
We would like to make it easier to accept multiple annotations.

Examples:
```
country("GB") && annotation(...)
annotated(annotated(X,...),...)
```

Change-Id: I92e622e8b985b314dadddf83b17976c245eb2069
2023-08-28 09:27:04 +00:00
Jeremy Wharton
2cdc1a973f satellite/console: fix project creation race condition
This change fixes an issue where the project limit could be exceeded if
multiple project creation requests were sent sufficiently close to one
another. This could also be used to bypass project name duplication
checking.

Change-Id: I61cde7abaf25dedc5601c6870275de9938d7b949
2023-08-25 17:18:34 +00:00
Ivan Fraixedes
d34c1b6825 private/apigen: Generate docs & typescript for example
Generate the documentation and the typescript code for the example of
that we have to generate an API through the API generator.

The JSON Go struct field tags are needed because the typescript
generator require them, otherwise it panics.

The test had to be adjusted according to match the tags so Go consider
the structs the same type, otherwise, it doesn't compile.

Change-Id: I3e4ebff9174885c50ca2058b86b7ec60e025945c
2023-08-25 12:30:21 +00:00
Moby von Briesen
d76f059c55 private/apigen: Update generated doc links
Make the link more human-friendly - to contain the text of the group and
endpoint names.

Also link back to list of endpoints from each endpoint.

Change-Id: Ia3e2ebe20b58b5f60ecefe9d35fb8fd90dd4b4d7
2023-08-25 11:51:57 +00:00
Jeremy Wharton
61fe95c44a web/satellite/vuetify-poc: update access grant creation dialog
This change updates the access grant creation dialog to align with our
new designs. It also fixes an issue that would occur if the web worker
was initialized after the dialog was mounted.

Resolves #6169

Change-Id: Ic557766e6fcf57cc79c72e670a0e83c7eb2834ba
2023-08-25 09:11:58 +00:00
Wilfred Asomani
e44365d265 web/satellite: use fontsource for font loading
This change updates the vuetify app and the production app to load
fonts using the fontsource package.

Issue: https://github.com/storj/storj/issues/6200

Change-Id: I4b91a4d0dfcfc42f9f71ac03b31d1ef74c53e15d
2023-08-24 23:08:41 +00:00
Wilfred Asomani
36f8eeb272 web/satellite: fix broken preview link
This change fixes an issue where the gallery view will use wrong links
display object previews because they were encoded multiple times.

Issue: https://github.com/storj/customer-issues/issues/961

Change-Id: I498878e9beb927b812e40d4c7e5ae812cfa3554c
2023-08-24 22:30:03 +00:00
Jeremy Wharton
84e75d5994 satellite/satellitedb: remove subquery AOST in console db cleanup funcs
This change fixes an issue where the console DB cleanup chore was never
able to run when using a Cockroach database implementation because of
an inappropriate AS OF SYSTEM TIME clause in the relevant methods.

Resolves #6197

Change-Id: I8456b6df2128678e0eebeb416eb1a955cc9bd706
2023-08-24 21:51:55 +00:00
Wilfred Asomani
fe9f69a757 satellite/{consoleweb,consoleapi}: add cross user api tests
This change adds tests to ensure critical endpoints are not able to be
called by users for other users. It asserts that if cases like that
do happen, a 401 response will be sent.

Issue: https://github.com/storj/storj-private/issues/407

Change-Id: I70097a80f691a7d0fcb0bc5dbce8291144177720
2023-08-24 20:16:20 +00:00
Moby von Briesen
5c155752d2 satellite/analytics: Update lifecyclestage for personal users
Personal users, like business users, should now be classified with
a lifecycle stage of PQL ("product qualified lead") instead of "other"

Change-Id: Iff5139043da1c8e75559302320ff9ca43ea956e5
2023-08-24 09:47:26 -04:00
Jeremy Wharton
0070cd322a web/satellite/vuetify-poc: fix bucket deletion dialog
This change fixes an issue where the bucket deletion dialog wasn't
functional due to an invalid property value. This change also fixes an
issue where access grant worker errors were reported as being caused by
the bucket deletion dialog even when it wasn't open.

Change-Id: If2c2713857c4cc5c3c7ae60e431f5034e78c4c5f
2023-08-24 09:53:03 +00:00
Moby von Briesen
7fbabbcf16 private/apigen: Add list of endpoints with links
This change makes it easier for someone reading the documentation to see
a full list of supported endpoints, and have direct links to the
details.

Change-Id: I46e2f809cfa2760845898eaa3d99db9066d435ef
2023-08-24 02:45:46 +00:00
Sean Harvey
a0c69568b5 Makefile: add sha256sums checksum file with release binaries
Change-Id: I518e075445288f238104b801f89837e2d8d0d781
2023-08-23 20:43:12 +00:00
Moby von Briesen
5340a351b7 satellite/.../consoleapi/gen: Update README
Remove outdated information from the generated API readme, and add a
link to the generated documentation.

Change-Id: Icc098c81f235464344895d2195444044831aac63
2023-08-23 19:49:35 +00:00
Ivan Fraixedes
b0d072270b private/apigen: Add comment about panics
Add a comment in 2 methods about they panic as it's specified for the
`MustWriteDocs`.

Change-Id: I15d5fa420af37318d828bc633bed62988f8b207b
2023-08-23 18:50:37 +00:00
Michal Niewrzal
95a5cfe647 satellite/buckets: handle bucket exists better
In some rare cases when two entities are trying to create the same
bucket at the same time it's possible that we will return internal
error instead of `bucket already exists`. It's because we are not
handling correctly DB error about constraint error. This change checks
if while inserting bucket into DB we got constraint error and propagate
correct error to metainfo API.

Change-Id: Ie6fd2c943b864b4ea7d71e4a162e74dc3510e386
2023-08-23 14:40:31 +00:00
Márton Elek
84ea80c1fd satellite/repair/checker: respect autoExcludeSubnet anntation in checker rangedloop
This patch is a oneliner: rangedloop checker should check the subnets only if it's not turned off with placement annotation.
(see in satellite/repair/checker/observer.go).

But I didn't find any unit test to cover that part, so I had to write one, and I prefered to write it as a unit test not an integration test, which requires a mock repair queue (observer_unit_test.go mock.go).

Because it's small change, I also included a small change: creating a elper method to check if AutoExcludeSubnet annotation is defined

Change-Id: I2666b937074ab57f603b356408ef108cd55bd6fd
2023-08-23 13:45:09 +00:00
Márton Elek
4ccce11893
satellite/overlay: improve realistic placement rule test
10 --> node tag inclusion in raw format
11 --> same, but using same subnet is enabled
12 --> same as 11 but with US restrictions

Change-Id: I20792689e0caf5fe190f566a770d70c3b3824793
2023-08-23 13:56:35 +02:00
Jeremy Wharton
973b54365e web/satellite/vuetify-poc: add project creation dialog
This change implements a dialog box through which users can create
projects.

Resolves #6172

Change-Id: I99311f43ef49dc04bb852a6736771b42f26a3276
2023-08-22 19:45:35 +00:00
Moby von Briesen
b66fc6dcdb web/satellite: Update build script to build vuetify app
Also copy the necessary vuetify dist directory in the `Earthfile` so
that this app can be easily deployed to staging environments.

Change-Id: I8d91c52bb8f7c31fc3764efe60a071e21b399b46
2023-08-22 13:55:10 +00:00
Wilfred Asomani
516241e406 cmd,satellite: remove Graphql code and dependencies
This change removes unused GraphQL code. It also updates storj sim code
to use the GraphQL replacement HTTP endpoints and removes the GraphQL
dependency.

Issue: https://github.com/storj/storj/issues/6142

Change-Id: Ie502553706c4b1282cd883a9275ea7332b8fc92d
2023-08-22 12:23:14 +00:00
Wilfred Asomani
d10ce19f50 web/satellite: remove Graphql dependency
This change removes unused GraphQL code and dependencies.

Issue: https://github.com/storj/storj/issues/6142

Change-Id: Ib2346c7a034f26b46af1bc6113e007b6adcbdd51
2023-08-21 22:17:41 +00:00
Paul Willoughby
fdd4be80bf satellite/metainfo: increase default MaxEncryptedObjectKeyLength
Allow a longer encrypted key length to reduce 'key length is too big'
errors in gateway-mt.  Gateway is enforcing an unencrypted key length
of 1024 bytes but when encrypted some keys are exceeding the current
limit.

Updates https://github.com/storj/gateway-mt/issues/335

Change-Id: I38a0fbb0843fd782aeadca85f9a202821421b5a2
2023-08-21 14:03:16 -06:00
Wilfred Asomani
9cf9721abe web/satellite: use create project http endpoint
This change uses the new POST projects endpoint in place of the GraphQL
createProject query.

Issue: https://github.com/storj/storj/issues/6195

Change-Id: I1776b8b0e8656d2f5fa4219df020615bcc0f2543
2023-08-21 17:30:38 +00:00
Jeremy Wharton
7f9317aa48 satellite/console: fix account creation race condition
This change fixes an issue where multiple unverified users with the
same email address could be created if registration requests were
sent sufficiently close to one another.

Resolves #6156

Change-Id: If8b1a145bcab842ace718119183de59947430463
2023-08-21 16:52:11 +00:00
Márton Elek
b2780b028d satellite/gracefulexit: use placement when gracefulexit pick new nodes
It's quite straightforward change, and AFAIK graceful exit will be decommissioned very soon.

Therefore I didn't create big unit tests, yet. But I can be convinced to invest more time.

Change-Id: Ia588e516d7af5171fa47f9bab100edd3bf2b2cf9
2023-08-21 15:40:05 +00:00
Michal Niewrzal
005fb19a7b satellite/metabase: adjust BucketEmpty to use pending_objects table
Extends metabase.BucketEmpty logic to check also pending_objects
table for any entry.

https://github.com/storj/storj/issues/6057

Change-Id: Ia26c272de24a983b308a0b692e6bd5800487eb98
2023-08-21 15:01:59 +00:00
Michal Niewrzal
16588033fd satellite/metabase: delete bucket deletes also from pending_objects
While deleting bucket we need also to delete pending objects from
pending_objects table.

Part of https://github.com/storj/storj/issues/6048

Change-Id: Icc83eaecf8388704e0b6329c397e8028debcf672
2023-08-21 14:05:13 +00:00
Jeremy Wharton
ffa50f0758 web/satellite/vuetify-poc: add bucket sharing option to buckets table
This change allows buckets to be shared from the Buckets page through
a dropdown menu in the buckets table.

Resolves #6116

Change-Id: I92055a3ea174d786f2ef960124aafcbd3b2139c4
2023-08-21 11:26:40 +00:00
Michal Niewrzal
ae2cba1d23 satellite/metabase: add IteratePendingObjectsByKeyNew method
New metabase method IteratePendingObjectsByKeyNew to iterate
over entries in pending_objects table with the same object key.
Implementation and tests are mostly copy of code for
IteratePendingObjectsByKey. Main difference is that pending_objects
table have StreamID column part of primary key instead Version.

Method will be used to support new table in
metainfo.ListPendingObjectStreams request.

After full transition to pending_objects table we should remove 'New'
suffix from methods names.

Part of https://github.com/storj/storj/issues/6047

Change-Id: Ifc1ecbc534f8510fbd70c4ec676cf2bf8abb94cb
2023-08-21 08:08:03 +00:00
Michal Niewrzal
929dc80091 satellite/metabase: add IteratePendingObjects method
New metabase method IteratePendingObjects to iterate over entries in
pending_objects table. Implementation and tests are mostly copy of
code that we are using to iterate over objects table. Main difference
is that pending_objects table have StreamID column part of primary
key instead Version. Also structure of pending object is smaller
than the one from object table but it's a detail.

Method will be used to support new table in metainfo.ListObjects
request.

Next step will be to port rest of iterator implementation to support
pending_objects table in metainfo.ListPendingObjectStreams.

Part of https://github.com/storj/storj/issues/6047

Change-Id: Ia578182f88840539f3668d4a242953e061eace02
2023-08-21 08:07:16 +00:00
Michal Niewrzal
5a8ef89824 satellite/{metainfo,metabase}: delete from pending_objects table
We are deleting pending objects while aborting multipart upload. We are
using metainfo BeginDeleteObject to do that. This change starts using
DeletePendingObjectNew to delete entry from pending_objects table when
request indicates that object is in this table.

Part of https://github.com/storj/storj/issues/6048

Change-Id: I4478a9c13c8e3db48dc5de3087ef03d1b4c47a5c
2023-08-21 08:06:23 +00:00
Wilfred Asomani
33c0a82fb7 satellite/console: add create project http endpoint
This change adds an HTTP endpoint for creating projects, to be used in
place of the GraphQL version.

Issue: https://github.com/storj/storj/issues/6195

Change-Id: I0377353418df7c152db6a935e99a3ea7ab4ce625
2023-08-21 06:58:03 +00:00
Cameron
6f47e178e2 web/satellite: add "do not share link" to emails
Change-Id: I94da487b622f083d5cf893d6dd90ad30076f6466
2023-08-18 23:16:13 +00:00
Vitalii
d0c3a59f44 web/satellite: differentiate password and passphrase autocomplete inputs
Added unique autocomplete values for password and passphrase inputs to make browser native autocomplete feature work correctly.

Issue:
https://github.com/storj/storj/issues/6151

Change-Id: I38e1bfed46beb47b3be1a620ce2a4996359feafc
2023-08-18 17:29:20 +00:00
Wilfred Asomani
f46280c93b web/satellite/vuetify-poc: add open bucket passphrase
This change requires a passphrase before opening a bucket and after
opening the objects page without going through the buckets page. It also shows if there are
objects stored with different passphrase.

Issues: https://github.com/storj/storj/issues/6100
https://github.com/storj/storj/issues/6110

Change-Id: Ica509fd6db968ef9c232a05b3ee39e0ab5f746e1
2023-08-18 10:54:05 +00:00
Jeremy Wharton
2b278bb05f web/satellite/vuetify-poc: unmock total cost, token balance, coupon
This change replaces mock data for the total cost, STORJ token balance,
and coupon information in the Vuetify project's billing overview with
real data.

Resolves #6095

Change-Id: Iad1c5141065f8761d7d13140d42871018fa9aca4
2023-08-18 10:14:57 +00:00
Márton Elek
f0afe0d2ea satelite/repairer: ignore declumping when subnet filtering is turned off with filter annotation
+ restoring the functionality of repairer.doPlacementCheck

Change-Id: I75521f2da280758345face07eeea661765717318
2023-08-18 09:35:38 +00:00
Márton Elek
5522423871 satellite/nodeselection: remove AutoExcludeSubnet filter
It's statefull, therefore it can hit naive users. (NodeFilters couldn't be reused for more than one iterations).

But looks like we don't need it, as `SelectBySubnet` doest the same job.

Change-Id: Ie85b7f9c2bd9a47293f4e3b359f8b619215c7649
2023-08-18 08:31:00 +00:00
Jeremy Wharton
db13e3ef15 web/satellite: warn when indenting with tabs
This change adds a rule to the ESLint config to warn when tabs are used
instead of spaces for indentation.

Change-Id: I068d559211e0f9e40abaea86de839a409b405674
2023-08-17 20:45:47 +00:00
Jeremy Wharton
c706df5218 web/satellite/vuetify-poc: make project invite titles unclickable
Users with a pending invitation for a project could be redirected to
that project's dashboard if its title in the projects table was
clicked. This type of redirection is only intended for projects that
the user is a member of, as navigating into any other project will send
the user back to the all projects dashboard. This change fixes this
issue, making invitation titles in the projects table unclickable.

Change-Id: I6ef6a72d92e8eb576d49d6268373af340116ee2f
2023-08-17 20:07:16 +00:00
Wilfred Asomani
ea38b9f78e web/satellite/vuetify-poc: fix vuetify form reloads
This change fixes an issue where the vuetify app will reload when
"enter" is pressed in a form. It also adds a success notification on
successfully enabling 2FA.

Change-Id: I23a66122dce4f1bd2ff21a5cd6ae0f1b22be9395
2023-08-17 19:28:11 +00:00
Jeremy Wharton
957d8d6ca0 web/satellite/vuetify-poc: add bucket deletion option to buckets table
This change allows buckets to be deleted from the Buckets page through
a dropdown menu in the buckets table. Before deletion, users are
prompted to confirm their decision in a dialog box.

Revoles #6114

Change-Id: Ie3a8bfbf94a29c4bc67b4dacbace6abe7e1bc2eb
2023-08-17 18:48:54 +00:00
Jeremy Wharton
f2e607c70f web/satellite/vuetify-poc: fix project navigation sidebar paths
This change resolves an issue where the buttons in the navigation
sidebar would not navigate to the correct page when clicked if the
current path was not a sibling of the intended page's path.

Change-Id: I400aa05954f532eabbad98f9fa36f594f08a6c10
2023-08-17 16:52:31 +00:00
Márton Elek
b218002752 satellite/overlay/placement: improve placement configurability with &&, placement, region and ! support in country
This patch makes it easier to configure existing placement rules only with string.

 1. placement(n) rule can be used to reuse earlier definitions
 2 .&& can be used in addition to all(n1,n2)
 3. country(c) accepts exclusions (like '!RU'), regions ('EU','EEA'), all and none

See the 'full example' unit test, which uses all of these, in a realistic example.

https://github.com/storj/storj/issues/6126

Change-Id: Ica76f016ebd002eb7ea8103d4258bacd6a6d77bf
2023-08-17 16:12:53 +00:00
Egon Elbre
b5e1e5a9e2 private/apigen: use typescript in docs markdown
Fixes https://github.com/storj/storj/issues/6187

Change-Id: I34016513f29a9538343f0909951ab1a4605bb585
2023-08-17 15:33:21 +00:00
Vitalii
5abed63f53 web/satellite: update estimated charges info message
Updated message to be more clear, and changed "estimated charges" to
"estimated usage". This should make this card in the billing UI less
confusing.
Updated styling.

Issue:
https://github.com/storj/storj/issues/6150

Change-Id: I5de4d72dfd437d4f6bda882f5b779b2217e84f98
2023-08-16 15:11:59 -04:00
Márton Elek
9ddc8b4ca3 satellite/repair: piecescheck.OutOfPlacementPiecesSet should not contain offline nodes
When we check the availability of the pieces, we do:

```
result.NumUnhealthyRetrievable = len(result.ClumpedPiecesSet) + len(result.OutOfPlacementPiecesSet)
// + some magic if there are overlaps between them
numHealthy := len(pieces) - len(piecesCheck.MissingPiecesSet) - piecesCheck.NumUnhealthyRetrievable
```

This works only if OutOfPlacementPieceSet doesn't contain the offline nodes (which are already included in MissingPieceSet).

But `result.OutOfPlacementPieces.Set` should include all the nodes (even offline), as in case of lucky conditions, we are able to remove those pieces from DB.

The solution is to remove all offline nodes from `NumUnhealthyRetrievable`.

Change-Id: I90baa0396352dd040e1e1516314b3271f8712034
2023-08-16 17:35:10 +00:00
Jeremy Wharton
80186ecc67 web/satellite/vuetify-poc: add project settings page
This change implements the Project Settings page in the Vuetify
project. It allows users to view and change a project's name,
description, storage limit, and bandwidth limit.

Resolves #6176

Change-Id: Ibcc56d2bb7c71e818390e69eec197508e3551803
2023-08-16 11:55:01 -05:00
Márton Elek
de7aabc8c9 satellite/{repair,rangedloop,overlay}: fix node tag placement selection for repair
This patch fixes the node tag based placement of rangedloop/repairchecker + repair process.

The main change is just adding the node tags for Reliable and KnownReliabel database calls + adding new tests to prove, it works.

https://github.com/storj/storj/issues/6126

Change-Id: I245d654a18c1d61b2c72df49afa0718d0de76da1
2023-08-16 15:45:41 +00:00
Egon Elbre
a3067b7b3b storagenode/monitor: ignore shutdown errors
This fixes some flakyness in tests.

Change-Id: I535232c5d80827d6d72c73d61134f3b2806b5db9
2023-08-16 11:53:58 +00:00
Jeremy Wharton
93ce4a0e49 web/satellite/vuetify-poc: fix offscreen notifications
This change fixes an issue where notifications would display above the
viewport if the page was scrolled down. Now, notifications are fixed to
the top-right corner of the viewport.

Change-Id: I4d55b7d149b889e7b41e2f245ff8547e998877fc
2023-08-16 10:11:28 +00:00
Wilfred Asomani
d0f5f06159 web/satellite/vuetify-poc: add create bucket dialog
This change adds the ability to create a bucket to the vuetify poc.

Issue: https://github.com/storj/storj/issues/6112

Change-Id: Ib56432e2ad09c6673c9903a49537199882fd1cf3
2023-08-16 09:25:33 +00:00
Márton Elek
da08117fcd satellite/~placement: do not ignore placement check for placement=0
There are cases when we would like to override the default placement=0 rule.

For example when we would like to exclude tagged nodes from the selection (by default).

Therefore we couldn't use a shortcut any more, we should always check the placement rules, even if we use placement=0.

TODO: we need to update common, and rename `EveryCountry` to `DefaultPlacement`, just to avoid confusion.

https://github.com/storj/storj/issues/6126

Change-Id: Iba6c655bd623e04351ea7ff91fd741785dc193e4
2023-08-16 07:06:56 +00:00
Jeremy Wharton
baef654197 web/satellite/vuetify-poc: fix visibility of input details
This change resolves an issue that 37a027a introduced wherein each
input component's details area was indiscriminately hidden. Now,
details are only hidden when there is no content to be displayed.

References #6170

Change-Id: I8ccbef3f64a54c400a2a2e9222618eb5015c8dd8
2023-08-15 19:45:53 +00:00
Márton Elek
c08792f066 satellite/overlay: implement an exclude filter for placement configuration
https://github.com/storj/storj/issues/6126

Change-Id: I05215b5d46bec958001cc020edf1fa97b00d3299
2023-08-15 17:29:29 +00:00
Márton Elek
0e17b1018c satellite/{nodeselection,overlay}: support annotations on node filters
Change-Id: I844d8a25042750aae189175842113e2f052d5b17
2023-08-15 16:49:57 +00:00
Jeff Wendling
b70fb2f87f cmd/uplink: fix progress bar crash
the progress bar was being set to inconsistent lengths
multiple times, causing a crash. this fixes that by
only setting the progress bar length once to the length
of the full object. it avoids a round trip by doing so
only after it has gotten the first read handle from the
source, so the length information is cached.

Change-Id: I112d7c79016e54ba3794e96c6174cc01b8baedb4
2023-08-15 13:10:03 +00:00
Egon Elbre
d701f9f081 Makefile: use storjlabs/ci:slim
We moved the slim build to the same image name and used the tag
for distinguishing the different builds.

Change-Id: I82e235ba4db6a0b76b1edccab3b13d006c6c9328
2023-08-15 09:49:27 +00:00
Jeff Wendling
1cbad0fcab cmd/uplink: add back parallelism
for very large machines (>10Gbit) it is still useful
to have parallelism for uploads because we're actually
bound by getting new pieces from the satellite, so doing
that in parallel provides a big win.

this change adds back that flag to exist for uploads, and
removes the backwards compatibility code for the flag with
the maximum-concurrent-pieces as they are now independent.

the upload code parallelism story is now this:

    - each object is a transfer
    - each transfer happens in N parts (size dynamically
      chosen to avoid having >10000 parts)
    - each part can happen in parallel up to the limit
      specified
    - each parallel part can have up to the limit of
      max concurrent pieces and segments

this change also changes some defaults to be better.

    - the connection pool capacity now takes into acount
      transfers, parallelism and max concurrent pieces
    - the default smallest part size is 1GiB to allow the
      new upload code path to upload multiple segments

Change-Id: Iff6709ae73425fbc2858ed360faa2d3ece297c2d
2023-08-14 20:28:58 -04:00
Jeremy Wharton
03690daa35 web/satellite/vuetify-poc: add session timeout and refresh
This change allows sessions within the Vuetify project to be refreshed.
In addition, dialogs appear to inform the user when a session is about
to expire and when it has expired.

Resolves #6147

Change-Id: I53d5508825aa9992e4fed8ce7b957d949ff28e2d
2023-08-14 16:16:51 +00:00
littleskunk
792bb113bc
satellite/console: Enable gallery view and limits area by default (#6177) 2023-08-14 17:05:19 +02:00
Moby von Briesen
b9206b1844 satellite/console: support hosting Vuetify POC on subdomain
This change allows you to host the vuetify app on <x>.example.com where
the main app is hosted on example.com. A configuration is added to
specify an exact subdomain for cookies. For example, if my production
app is hosted on us1.storj.io and my vuetify app is hosted on
vuetify.us1.storj.io, the cookie domain should be set to ".us1.storj.io"
so that any authentication cookie is accessible to lower-level
subdomains.

Since the vuetify app does not currently support login/signup on its
own, it is still required to first login to the main satellite UI, then
navigate to the Vuetify app after the session cookie is set.

If the "vuetifypoc" prefix is not desirable when using subdomain hosting
for vuetify, the VITE_VUETIFY_PREFIX variable can be modified in
web/satellite/.env before running `npm run build-vuetify`. For now, we
should keep this prefix because it makes developing on the vuetify app
significantly easier if subdomains are not being used.

Issue: https://github.com/storj/storj/issues/6144

Change-Id: Iba1a5737892c8ee8f38148a17b94e3222f8798e6
2023-08-14 13:15:41 +00:00
Márton Elek
ceb7b7375c satellite/geoip: exclude nodes with represented_contry from geofencing
Change-Id: I80b8e5d98f46559b158a26c47fff0586b97aff79
2023-08-14 12:36:33 +00:00
Wilfred Asomani
0fd7f2958f web/satellite/vuetify-poc: add functionality to add credit cards
This implements functionality to add cards to users' accounts, to the
vuetify POC.

Issue: https://github.com/storj/storj/issues/6097

Change-Id: Ie56e85e74fd44de6839e6a985877843b730a3f5f
2023-08-14 09:45:53 +00:00
Cameron
df60380793 web/satellite: use project members delete http endpoint
issue: https://github.com/storj/storj/issues/6136

Change-Id: I717755721dc29714ad161fd311f7226a69acc163
2023-08-11 13:48:35 -04:00
Cameron
969599b60b satellite/console: delete project members endpoint
This commit adds a new endpoint on the console api to delete project
members and invitations.

issue: https://github.com/storj/storj/issues/6136

Change-Id: I980bb97afd1ed2ed8f0f27cc2e8dc1d80d7eef05
2023-08-11 13:48:33 -04:00
Vitalii
d7d196fe61 web/satellite: small vuetify fixes
Removed selection/checkboxes on team table.
Made search bars on searchable tables clearable.
Navbar toggle works on small screen sizes.
Removed notifications section from the settings page.

Issue:
https://github.com/storj/storj/issues/6148

Change-Id: I531088cf0db04fd2979d39f92bb4342e0d9551a6
2023-08-11 16:14:39 +00:00
Jeremy Wharton
03b45162d9 web/satellite/vuetify-poc: update alert styling
The custom styling for the VAlert component as of
storj/vuetify-storj@71c509a has been pulled into the Vuetify project.

References #6170

Change-Id: Ia36fd936ddc33c7687bcf7b2cdd6f789d0bc0a92
2023-08-11 10:17:16 -05:00
Moby von Briesen
dbc7443c9d web/satellite: Set part size back to 64mb
Reverts this commit to make part size dynamic:
a9d979e4d7

There are issues with client-side memory usage using the dynamic sizing.

Change-Id: Iba18af1e424622b7b3bf57413322332d8e1aa63a
2023-08-11 12:53:10 +00:00
Vitalii
8b0d25cde1 web/satellite/vuetify: speed up build commands
Fixed the way we use vuetify settings. Seems like the problem was related to SASS variables.
Partially used this approach https://github.com/vuetifyjs/vuetify/issues/8169#issuecomment-1495643012

Also, bumped vite and vuetify deps.

New build time for me:
real    0m28.629s
user    0m41.237s
sys     0m3.753s

Issue:
https://github.com/storj/storj/issues/6143

Change-Id: I94f8ee77057b85faf4d1a10bed7b43ba8ea929f1
2023-08-11 12:13:08 +00:00
Jeremy Wharton
75f2152ae3 web/satellite: modularize session timeout code
This change adds a Vue composable and wrapper component for reusing
session timeout code. In the future, the composable will be used to
implement session timeout in the Vuetify project.

References #6147

Change-Id: Ibd049a8a8041007319798ac4187a6ed6487b591f
2023-08-10 20:55:57 -05:00
Vitalii
6c035a70af web/satellite: modify 'add STORJ token' behavior for free-tier users
When a user is in the free tier and attempts to add STORJ from billing screen, open the "upgrade" modal instead of the simpler "add STORJ tokens" modal.
The simpler modal is still used for Pro users.

Issue:
https://github.com/storj/storj/issues/6165

Change-Id: I261d67e1c4d569f7058da828bcb145a64136c231
2023-08-10 19:11:05 +00:00
Egon Elbre
0335697664 satellite/accounting/live: ensure we don't panic when we get nil
Change-Id: I99b6b94d37a9856e8a705679d117b42d16326e81
2023-08-10 19:41:57 +03:00
Egon Elbre
1f261bcc70 go.mod: bump lang to 1.19 and common
Change-Id: I8d91f97d786456da29ebe89a78412c50efbb8ccc
2023-08-10 18:41:15 +03:00
Jess Stingray
6f01a81648
Downgrade "context canceled" errors to Info instead of Error (#6166) 2023-08-10 09:33:49 +02:00
Vitalii
e226606ce8 web/satellite: updates for large file upload
Universal "large file notification" removed.
Threshold for user-triggered large file notification is 5gb instead of 1gb.
Large file notification appears as "info" instead of "warning".
Display large file notification on failed uploads >1gb.

Issue:
https://github.com/storj/storj/issues/6149

Change-Id: I6ae6ba50e45c65f058cbb91a5f0fbda79a49a812
2023-08-09 23:39:15 +00:00
Wilfred Asomani
6b2fb9dfc4 web/satellite: use force delete bucket
This change modifies delete bucket requests to optin for force delete
using the x-minio-force-delete header. This solves an issue where
buckets deletes will be slow for buckets that have many objects. This
change is to make the UI emulate uplink rb --force since it has better
performance.

Issue: https://github.com/storj/customer-issues/issues/930

Change-Id: I0a74c1a201e74b07eb30b917adf78ef865ce2003
2023-08-09 23:05:50 +00:00
Vitalii
7d149dca0f web/satellite: use delete API keys http endpoint
This change uses the new /delete-by-ids endpoint in place of the GraphQL query.

Issue:
https://github.com/storj/storj/issues/6140

Change-Id: Ia78016e68fca296367e650b7a919130e662ee8f6
2023-08-09 22:32:47 +00:00
Wilfred Asomani
a88711319c web/satellite: use update project http endpoint
This change uses the update project HTTP endpoint in place of the
GraphQL alternative.

Issue: https://github.com/storj/storj/issues/6134

Change-Id: I7d8203c20d8c0c1dcdddb7d1aaf472fc66568a3d
2023-08-09 16:41:47 +00:00
Wilfred Asomani
5bdb7bc3f0 satellite/console: add update project http endpoint
This change adds an endpoint update projects, to be used in place of
the graphql alternative.

Issue: https://github.com/storj/storj/issues/6134

Change-Id: I26c04f4400f71721cbddb7f64405e6c9b78edb4d
2023-08-09 16:07:51 +00:00
Jeremy Wharton
81163321ad web/satellite/vuetify-poc: remove project description from nav button
This change removes the project description from the subtitle of the
topmost item in the project navigation area, replacing it with the
project name and inserting "Project" where the name used to be.

References #6170

Change-Id: I07a04ebba54828df41f4d241aa69ec6b3d9e0cef
2023-08-09 14:35:13 +00:00
Jeremy Wharton
37a027af67 web/satellite/vuetify-poc: hide input message areas with no content
The message area beneath input components has been hidden by default so
that it doesn't occupy space when it isn't needed.

References #6170

Change-Id: I16b078cddb07708169b96e53db283aedb290f9f4
2023-08-09 14:01:52 +00:00
Jeremy Wharton
eace6e37b8 web/satellite/vuetify-poc: fix dialog close button position
This change fixes the position of the close button in dialogs where
there was no spacing between the button and the edges of the dialog
header.

References #6170

Change-Id: Id3f798953319c1703d7bb6868757aecd02ec09f5
2023-08-09 13:28:59 +00:00
Cameron
c931234e46 web/satellite: use project members http endpoint instead of graphql
issue: https://github.com/storj/storj/issues/6137

Change-Id: I1f9b0d74fa4796c4dbee3b80618e87583b0f3f35
2023-08-09 12:56:01 +00:00
Jeremy Wharton
ea94bc7f6d web/satellite/vuetify-poc: add access grant creation flow
This change implements the access grant creation flow in the Vuetify
project.

Resolves #6061

Change-Id: I233088cbfcfe458936410899531389e290f276d4
2023-08-09 12:23:24 +00:00
Egon Elbre
dc41978743 all: fix golangci failures
Change-Id: I07421388d53c837e35a4727cead26fc21c324d04
2023-08-09 11:44:44 +03:00
Clement Sam
9e3d54fec4 satellite/admin: extend API to allow setting and deleting account level geofence
Issue: https://github.com/storj/storj-private/issues/357
Change-Id: I04589e18214e7090ccd686fd531066d942afa6ed
2023-08-09 03:34:37 +00:00
Jeremy Wharton
9e00d495c4 web/satellite: use get buckets http endpoint instead of graphql
This change causes the satellite frontend to use the HTTP endpoint for
retrieving bucket usage totals rather than its GraphQL counterpart.

Resolves #6141

Change-Id: I9a42080c8344aa617a910a8782dc61101a6734c8
2023-08-08 21:59:25 +00:00
Jeremy Wharton
a00ec7af40 satellite/console: create http endpoint for getting bucket usage totals
This change introduces an HTTP endpoint for retrieving bucket usage
totals. In the future, this will replace its GraphQL counterpart.

References #6141

Change-Id: Ic6a0069a7e58b90dc2b6c55f164393f036c6acf4
2023-08-08 21:26:17 +00:00
Cameron
683119b835 satellite/console: get project members endpoint
This commit adds a new endpoint on the satellite console api to get
project members and invitations.

issue: https://github.com/storj/storj/issues/6137

Change-Id: I66cb064eeaffb1c34878462b3e6b3be8f3629f4e
2023-08-08 14:10:29 -04:00
Michal Niewrzal
887209bc24 satellite/metabase: fix CommitObject query for postgres
Query was not working for postgres because types were not correctly set.

Change-Id: Ic898aa37b71a4754e4ebe82085f3563fc954616a
2023-08-08 15:31:44 +00:00
Wilfred Asomani
55b23d2bdb web/satellite: use get owned projects http endpoint
This change uses the new /projects/paged endpoint to get user's owned
projects in place of the OwnedProjects graphQL query.

Issue: https://github.com/storj/storj/issues/6135

Change-Id: I1e8dfdefdc7de51595637577f7d8bdce8e969652
2023-08-08 14:03:18 +00:00
Wilfred Asomani
34e1caa55a satellite: add get user paged projects http endpoint
This change adds an endpoint to get a user's projects, similar to
the OwnedProjects GraphQL query.
The console.ProjectInfo struct has been renamed to UpsertProjectInfo
to more accurately reflect its use.

Issue: https://github.com/storj/storj/issues/6135

Change-Id: I802fe4694a5cc75a9df2b565476f6e6f473431d4
2023-08-08 14:02:53 +00:00
Wilfred Asomani
b21041d6f9 web/satellite/vuetify: add recovery MFA codes dialog
Added regenerate MFA codes dialog and functionality.

Issue: https://github.com/storj/storj/issues/6091

Change-Id: I5139e83077adb6deea1ad57c0ebe4e84a88e770c
2023-08-08 13:09:01 +00:00
Michal Niewrzal
7be844351d satellite/metainfo: remove ServerSideCopyDuplicateMetadata
https://github.com/storj/storj/issues/5891

Change-Id: Ib5440169107acca6e832c2280e1ad12dfd380f28
2023-08-08 12:15:10 +00:00
Michal Niewrzal
9550b5f4a5 satellite/metabase: drop object deletion code for copy with references
With this change we are removing code responsible for deleting objects
and supporting server side copies created with references. In practice
we are restoring delete queries that we had before server side copy
implementation (with small exception, see bellow).

From deletion queries we are also removing parts with segment metadata
as result because we are not longer sending explicit delete requests to
storage nodes.

https://github.com/storj/storj/issues/5891

Change-Id: Iee4e7a9688cff27a60fb95d60dd233d996f41c85
2023-08-08 11:31:12 +00:00
Michal Niewrzal
03f8ad323d satellite/metabase: remove segment_copies support from ListSegments
We don't need to support segment copies with references anymore.
We migrated to copies where all metadata are copied from original
segment to copy.

https://github.com/storj/storj/issues/5891

Change-Id: Ic91dc21b0386ddf5c51aea45530024cd463e8ba9
2023-08-08 11:21:08 +00:00
Vitalii
6c3300c522 satellite/console: add delete api keys http endpoint
This change adds an endpoint to delete API keys, similar to GraphQL mutation.

Issue:
https://github.com/storj/storj/issues/6140

Change-Id: Ia4a808222a057a199d803d8ea6b944c411a4dc8d
2023-08-08 10:46:54 +00:00
Vitalii
5345eadffa web/satellite: use create API key http endpoint
This change uses the new /create endpoint in place of the GraphQL query.

Issue:
https://github.com/storj/storj/issues/6139

Change-Id: Ic7db9407377b742ab87c887a74a37eaec6a8dbb6
2023-08-08 10:46:25 +00:00
Michal Niewrzal
7f249ab7ca cmd/tools: remove migrate-segment-copies tool
Migration was done. We can remove tool now.

https://github.com/storj/storj/issues/5891

Change-Id: I5d56bad1ac680cd77dabfcf271788e100a6a435b
2023-08-08 10:00:40 +00:00
Michal Niewrzal
a5cbec7b3b satellite/metabase: drop bucket deletion code for copy with references
With this change we are removing code responsible to handle deleting
bucket and supporting server side copies created with references. In practice we are restoring delete queries that we had before server side copy implementation (with small exception, see bellow).

From deletion queries we are also removing parts with segment metadata
as result because we are not longer sending explicit delete requests to
storage nodes.

https://github.com/storj/storj/issues/5891

Change-Id: If866d9f3a1b01e9ebd9b49c4740a6425ba06dd43
2023-08-08 09:07:44 +00:00
Wilfred Asomani
00531fe2b0 web/satellite/vuetify: add disable MFA dialog
Added disable MFA dialog and functionality.

Issue: https://github.com/storj/storj/issues/6091

Change-Id: I24cf863b2a04cc147894ab0640e1e1491acc430d
2023-08-08 07:57:01 +00:00
Vitalii
185ebe3dcf web/satellite/vuetify: added notifications
Added notifications to Vuetify app.
Populated existing functionality with notifications.

Issue:
https://github.com/storj/storj/issues/6087

Change-Id: I8339c372bb32fbf1e0ea136c92383494c129b4b6
2023-08-08 07:14:52 +00:00
Vitalii
f57bc81ce7 satellite/console: add create api key http endpoint
This change adds an endpoint to create new API key, similar to GraphQL mutation.

Issue:
https://github.com/storj/storj/issues/6139

Change-Id: I2b35d680fa8e019666c811ad3bdf16201e3b8946
2023-08-07 23:58:38 +00:00
Vitalii
fa4f5a6ae9 web/satellite: use get project api keys http endpoint
This change uses the new /list-paged endpoint in place of the GraphQL query.

Issue:
https://github.com/storj/storj/issues/6138

Change-Id: I68c35437f3f2e18898783e2fd523cfdfb3f10de3
2023-08-07 18:59:30 +00:00
Vitalii
256bd18120 satellite/console: add get project api keys http endpoint
This change adds an endpoint to get paged project API keys, similar to GraphQL query.

Issue:
https://github.com/storj/storj/issues/6138

Change-Id: I5dea9e4ac61e798cc8a2e56a2755d722c1b66bfa
2023-08-07 18:26:31 +00:00
Vitalii
a4f7f0634d web/satellite/vuetify: added analytics events for currently implemented functionality
Populated already implemented functionality with analytics events.

Issue:
https://github.com/storj/storj/issues/6122

Change-Id: I1e250ca02debc9bedbf78024f74e0dbfa9dfbcb9
2023-08-07 17:52:39 +00:00
Vitalii
e2e437dd95 web/satellite: use analytics pinia module instead of direct API requests
Start using analytics pinia module instead of making direct API requests from components.
We do this to not mix transport layer logic with view logic.

Issue:
https://github.com/storj/storj/issues/6122

Change-Id: Idb1902aec0635df4f0e682ba50bcc97b240ac4a9
2023-08-07 17:18:10 +00:00
Wilfred Asomani
e5bcb8b209 web/satellite: use get user projects http endpoint
This change uses the new /projects endpoint in place of the GraphQL
MyProjects query.

Issue: https://github.com/storj/storj/issues/6132

Change-Id: Ie4ae69dd6def75c4b8c627aaf55c31914cf69ce5
2023-08-07 16:42:41 +00:00
Wilfred Asomani
70f6b60d91 satellite/console: add get user projects http endpoint
This change adds an endpoint to get a user's projects, similar to
the MyProjects GraphQL query.

Issue: https://github.com/storj/storj/issues/6132

Change-Id: I91feb5a1ee8c1231a8a5e6de9b8dc5b256f857c5
2023-08-07 16:05:09 +00:00
Vitalii
d8d3bb5033 web/satellite/vuetify: added update session timeout dialog
Added update session timeout dialog for vuetify app and wired it up with the backend

Issue:
https://github.com/storj/storj/issues/6090

Change-Id: Icfd981c8d24a87b199e5e1eeb599f6369e6ee5c4
2023-08-07 15:31:13 +00:00
paul cannon
6e46a926bb satellite/nodeselection: expand SelectedNode
In the repair subsystem, it is necessary to acquire several extra
properties of nodes that are holding pieces of things or may be
selected to hold pieces. We need to know if a node is 'online' (the
definition of "online" may change somewhat depending on the situation),
if a node is in the process of graceful exit, and whether a node is
suspended. We can't just filter out nodes with all of these properties,
because sometimes we need to know properties about nodes even when the
nodes are suspended or gracefully exiting.

I thought the best way to do this was to add fields to SelectedNode,
and (to avoid any confusion) arrange for the added fields to be
populated wherever SelectedNode is returned, whether or not the new
fields are necessarily going to be used.

If people would rather I use a separate type from SelectedNode, I can do
that instead.

Change-Id: I7804a0e0a15cfe34c8ff47a227175ea5862a4ebc
2023-08-07 12:44:49 +00:00
Márton Elek
0b02a48a10
satellite/nodeselection: SelectBySubnet should use placement filters for all nodes
Current node selection logic (in case of using SelectBySubnet):

 1. selects one subnet randomly
 2. selects one node randomly from the subnet
 3. applies the placement NodeFilters to the node and ignore it, if doesn't match

This logic is wrong:

 1. Imagine that we have a subnet with two DE and one GB nodes.
 2. We would like to select DE nodes
 2. In case of GB node is selected (randomly) in step2, step3 will ignore the subnet, even if there are good (DE) nodes in there.

Change-Id: I7673f52c89b46e0cc7b20a9b74137dc689d6c17e
2023-08-04 10:48:15 +02:00
Michal Niewrzal
03c52f184e satellite/metabase: adjust CommitObject to use pending_objects table
Change is adjusting CommitObject to use `pending_objects` table to
commit object.

Satellite stream id is used to determine if we need to use
`pending_objects` or `objects` table during commit.

General goal is to support both tables until `objects` table will be
free from pending objects.

Part of https://github.com/storj/storj/issues/6046

Change-Id: I2ebe0cd6b446727c98c8e210d4d00504dd0dacb6
2023-08-03 10:21:22 +00:00
Márton Elek
2ed08922d9 satellite/api: configuration option to set additional node tag authorities
Change-Id: Iba387e37cb586ce1378668c99beb3b4db8a9064d
2023-08-03 08:34:02 +00:00
Wilfred Asomani
034542db8f web/satellite/vuetify: add enable MFA dialog
Added enable MFA dialog and functionality.

Issue: https://github.com/storj/storj/issues/6091

Change-Id: Idf2f27937549b9bb709ddcc70ffa4611beea7b78
2023-08-02 17:31:02 +00:00
Michal Niewrzal
f40805763e satellite/metabase: adjust segment commit to use pending_objects table
Change is adjusting CommitSegment to check pending object existence in
`pending_objects` or `objects` table.

Satellite stream id is used to determine if we need to use
`pending_objects` or `objects` table.

General goal is to support both tables until `objects` table will be
free from pending objects. Whenever it will be needed code will be
supporting both tables at once.

Part of https://github.com/storj/storj/issues/6046

Change-Id: I954444a53b4733ae6fc909420573242b02746787
2023-08-02 16:56:25 +00:00
Cameron
fac522d8dd web/satellite: implement vuetify poc project dashboard
issue: https://github.com/storj/storj/issues/6060

Change-Id: I40497a39afacd4787468e1259225ddd75e78295b
2023-08-02 16:22:22 +00:00
Michal Niewrzal
7b2006a883 satellite/metabase: adjust BeginSegment to use pending_objects table
Change is adjusting BeginSegment to check pending object existence in
`pending_objects` or `objects` table.

Satellite stream id is used to determine if we need to use
`pending_objects` or `objects` table.

General goal is to support both tables until `objects` table will be
free from pending objects. Whenever it will be needed code will be
supporting both tables at once.

Part of https://github.com/storj/storj/issues/6046

Change-Id: I08aaa605c23d82695fde352fdbd0a7fd11f46bb5
2023-08-02 15:30:23 +00:00
Michal Niewrzal
cebf255d64 satellite/metabase: adjust BeginObjectNextVersion to use pending_objects
Change is adjusting BeginObjectNextVersion to create pending object in
`pending_objects` or `objects` table depends on configuration. This is
first change to move pending objects from objects table.

General goal is to support both tables until `objects` table will be
free from pending objects. Whenever it will be needed code will be
supporting both tables at once.

To be able to decide if we need to use `pending_objects` table or
`objects` table we extend satellite stream id to keep that information
for later use.

BeginObjectExactVersion will be not adjusted because at the moment it's
used only in tests.

Part of https://github.com/storj/storj/issues/6046

Change-Id: Ibf21965f63cca5e1775469994a29f1fd1261af4e
2023-08-02 14:42:26 +00:00
Jeremy Wharton
391a63f9fa web/satellite: bump Vuetify to v3.3.10
This change updates the Vuetify dependency version so that we may use
new components.

Change-Id: Ib4ae4db3b50c267179503ff866077379d52b6d07
2023-08-02 13:46:20 +00:00
Vitalii
2d2f1b858e web/satellite: show pending transactions during token upgrade flow
Made UI updates to reflect pending token payments during upgrade account flow.
So pending transactions with confirmations count are displayed on Add STORJ Tokens step of upgrade flow.
While this modal is open we make a request to storjscan once in 20 seconds to get recent confirmations count.
When all transactions become confirmed we show success view where a sum of STORJ tokens received is displayed.

Issue:
https://github.com/storj/storj/issues/5978

Change-Id: Icfdc1e5080ed58cea1822cb7d85551ba8064c636
2023-08-02 15:39:25 +03:00
Márton Elek
63c8cfe4c3 satellite/nodeselection: remove CountryCodeExclude
We don't need it any more, as CountryCode uses location.Set, which supports exclusion (`Without`).

Change-Id: Ie311ae19fefa0bc9a0161496af1233ef4a6607df
2023-08-02 09:48:36 +00:00
Márton Elek
f7b39aaed4 satellite/nodeselection: remove stats/size from nodeselection state
stats/size/count is not used by any production code, and it's not required, as we can assert the state with other checks.

real motivation: next commits will make the Selector of the State configurable, therefore we won't have one single Stat, it depends on the request parameters.

(we plan to support both network and id based randomization)

Change-Id: I631828fc0046d2fef5b7a674fc0268a0446e9655
2023-08-01 18:29:41 +00:00
Wilfred Asomani
7c65c0cea5 satellite/{db,console,payments}: unfreeze user with no failed invoices
This change extends the autofreeze chore to go through users who have
been warned/frozen to check if they have no failed invoices. If they do
not, this extension unwarns/unfreezes them.

Issue: https://github.com/storj/storj/issues/6077

Change-Id: I570b1d4b2e29574bd8b9ae37eb2d4fb41d178336
2023-08-01 17:54:39 +00:00
JT Olio
e78658d174 go.mod: use latest maxmind-golang library
Change-Id: I6f06785a096d7d05b175d468482db658b7436a87
2023-08-01 15:22:35 +00:00
Michal Niewrzal
65aa9c11bc satellite/gc/sender: adjust some defaults
Change-Id: I94f5a18bc2d4218e834259b2ba22f3578745c975
2023-08-01 14:27:44 +00:00
Vitalii
59034ca094 web/satellite: upgraded aws-sdk dependencies to resolve vulnerabilities
Upgraded package-lock version to correspond to node v18+.
Upgraded aws-sdk dependencies to resolve vulnerabilities.
Also, fixed typing errors in object browser pinia module.

Change-Id: I35e6e219e66f98ca167ccb4ac57ad07ac99efff1
2023-08-01 13:41:15 +00:00
Vitalii
d40091a3cb web/satellite/vuetify: change name dialog
Added change name dialog for vuetify app and wired it up with backend.

Issue:
https://github.com/storj/storj/issues/6088

Change-Id: I37af595c9e48034d59ead051918bbb4ec5e6ff31
2023-08-01 14:26:16 +03:00
Wilfred Asomani
af69e20cc8 web/satellite: fix errant pagination
This change fixes an issue where changing the size of a page will also
change the current page. It uses a more accurate calculation to
determine if the new size and current page will result in a page index
error.

Issue: https://github.com/storj/storj/issues/6094

Change-Id: Ie62f79191eb34ccd75ccd42b923b8866fe4e4d7f
2023-07-31 22:43:52 +00:00
Vitalii
ceef4b8362 web/satellite/vuetify: added change password dialog
Added change password dialog for vuetify app and wired it up with backend.

Issue:
https://github.com/storj/storj/issues/6089

Change-Id: Ib1dffa947b65c299b278a48ed1491a623895a0bd
2023-07-31 17:03:26 +03:00
Márton Elek
6f002f4220
satellite/overlay: NR placement should exclude nodes without geofencing information
https://github.com/storj/storj-private/issues/378

Change-Id: If2af02083496e5a8eefe27beabb406388ee50644
2023-07-31 09:55:54 +02:00
Vitalii
ae5e742a12 web/satellite: update wording for upgrade account banner
Updated wording because we don't want to dissuade customers from upgrading and using Storj by implying that there is a ceiling to what they can store and egress with Storj.

Issue:
https://github.com/storj/storj/issues/6040

Change-Id: I8a08bea47b698e66e023d1bfcaa5ef09dae79192
2023-07-28 09:46:49 +00:00
Vitalii
62f52c829a web/satellite/vuetify: add charts to project dashboard
Imported chart components from main project to vuetify app project dashboard with slight updates.

Issue:
https://github.com/storj/storj/issues/6071

Change-Id: I51254ea60aab59c7a841784b5dac7d41446e1df4
2023-07-27 23:16:40 +00:00
Vitalii
fcbb37fb66 web/satellite: fix build-watch
Build-watch is broken again since we don't use NODE_ENV variable anymore.
Adjusted code to work with 'mode' setting.

Change-Id: I08b19ef59b39e6b14ce05d7340f032ee8377c49c
2023-07-27 21:36:18 +00:00
Wilfred Asomani
c934974652 satellite/{console,web}: update error handling
This change updates how API errors are handled and sent.

Change-Id: Ia4c71eeb6f2d009a47b59ce77a23f70b8b10f6dc
2023-07-27 21:01:01 +00:00
Vitalii
2c56599ca0 web/satellite: revert uploads queue size
Revert upload parallelism queue size back to 4 segments at a time.
Increasing this number causes some memory issues for some users.

Issue:
https://github.com/storj/storj/issues/6120

Change-Id: Ic5bcd1f13ee625fdc2613ee23d6945c905f03142
2023-07-27 19:57:11 +03:00
Clement Sam
cc12a48c24 satellite/admin: extend admin API to allow setting and deleting geofence for projects
Issue: https://github.com/storj/storj-private/issues/357
Change-Id: Ib59319581641f1f5da71c629143e12f11eb04925
2023-07-27 11:40:26 +00:00
Michal Niewrzal
4cc167a6bd satellite/metabase: use better label for ignoring FTS queries
For some test queries we are using workaround to filter them out from
full table scan detection. To avoid confustion what is this all about
we are changing label to be more descriptive.

Change-Id: I41a744e8faf400e3e8de7e416d8f4242f9093fce
2023-07-26 20:01:38 +00:00
Michal Niewrzal
3d9c217627 satellite/metabase: add pending_objects table
This change adds only schema definition of pending_objects table and
small amount of supporting code which will be useful for testing later.

With this table we would like to achieve two major things:
* simplify `objects` table, before we will start working on object
versioning
* gain performance by removing need to filter `objects` results with `status` column, which is not indexed and we would like to avoid that

https://github.com/storj/storj/issues/6045

Change-Id: I6097ce1c644a8a3dad13185915fe01989ad41d90
2023-07-26 20:00:58 +00:00
Moby von Briesen
28737f5c62 web/satellite: Implement "sign out" in vuetify poc
Implements the "sign out" button in the vuetify app to clear all app
data and navigate to login.

For now, there is a hard refresh after the router navigates to login,
becuase the vuetify poc does not have its own login page to route to.
The hard refresh allows the user to be presented with the main app's
login page after signing out.

Change-Id: Idb1c1e6af8511ea5db6533e46b96b9c15693379f
2023-07-26 13:49:30 -04:00
Moby von Briesen
7487809476 web/satellite: unmock list buckets in vuetify app
This change implements the table on the buckets view of the vuetify app.
Searching and pagination are implemented, but sorting functionality
needs to be added in a separate change - we need to extend the
bucketsStore/backend functionality in order to support sorting.

Resolves https://github.com/storj/storj/issues/6062

Change-Id: I4e92e6facbd7b21f4ec081e41f7e568ddb3cea29
2023-07-26 13:49:30 -04:00
Michal Niewrzal
16c5b3c340 mod: bump uplink to v1.11.0
Change-Id: I88922421f250a261b78227a409f4879ae5221140
2023-07-26 15:01:18 +00:00
Vitalii
6d94d6a681 satellite/{console, accounting}: fix project bandwidth calculation which is shown on project dashboard
Another try to fix calculation of used bandwidth which is displayed on Project Dashboard.
This change sums up allocated-dead traffic for the last 3 days and settled traffic for the period which is earlier than 3 days ago.

Issue:
https://github.com/storj/storj-private/issues/293

Change-Id: I91e652eba69f81bd21e0d053ac170e2b926b3cb4
2023-07-26 06:14:48 +00:00
Vitalii
5db0fd8846 web/satellite: wire up vuetify access page to backend
Added logic to list, sort, search and paginate access grants table for Vuetify POC

Change-Id: I215a9ad4f894b6ac985cb6a3059fdcf007e3d914
2023-07-24 23:17:19 +00:00
Jeremy Wharton
28711e30ad web/satellite/vuetify-poc: add project ID to project-specific routes
A project ID path parameter has been incorporated into project-specific
routes. Previously, project IDs were stored only in local storage,
which made it impossible to bookmark or share links to pages within a
particular project.

Resolves #6076

Change-Id: I4f46f58a95dfc9b67d95fdb1c6a65b50fcafe02b
2023-07-24 22:25:29 +00:00
Jeremy Wharton
5a03e29fca web/satellite/vuetify-poc: unmock all projects dashboard data
The all projects dashboard of the Vuetify project has been updated to
display real project information rather than mock data. Additionally,
projects may now be selected and project invitations replied to through
the all projects dashboard.

Resolves #6036
Resolves #6038

Change-Id: I8aef0dc07c172e2bc8e2f7eb26a3d205bd56067f
2023-07-24 21:39:28 +00:00
Jeremy Wharton
9086078ac7 web/satellite: remove glob patterns from linting commands
The glob patterns in the linting commands have been removed in favor of
a simpler way of expressing which files should be linted.

Change-Id: I150a01725642b4bc445e2e157ddf86e50e3911a2
2023-07-24 10:33:07 -05:00
Michal Niewrzal
73a279235a satellite/accounting/live: get project totals in batches
After infrastructure changes redis instance is not neccessay close
to core instance (where tally is calculated) and round trips to get
data from redis can be very costly. From less than hour calculation can take few hours for larger satellite.

This change combines 'segment' and 'storage' usage requests into
batches to reduce latency impact on tally calculation.

https://github.com/storj/storj/issues/5800

Change-Id: I87e57ec09e88fd167060a4ed51dc8b0274a095c5
2023-07-24 14:13:04 +00:00
Jeremy Wharton
f30e0986b6 web/satellite/vuetify-poc: fix app store name conflict
The name of the Vuetify Pinia app store has been renamed so that it
does not conflict with the regular app store. Previously, whichever
store was created first would override the other due to them having
identical names.

Change-Id: I11cb3b70b7385320accd30644cb3ad6cfad7d00e
2023-07-22 20:51:50 -05:00
Cameron
b16c8ba2e4 web/satellite: add indeterminate progress bar
Display indeterminate progress bars in upload modal if progress is less
than 1.

Change-Id: Icdad8e59914985f3ed8fd25dd01dba7e9ff88cf0
2023-07-21 18:11:18 -04:00
Wilfred Asomani
8ed4c573db satellite/admin: add endpoint to unwarn user
This change enables the admin UI to remove the warning status of users.

resolves: storj-private/issues/342

Change-Id: Ib960ffb33fdabc045884ce7fa2c55c3553db0fb0
2023-07-21 17:10:09 +00:00
Vitalii
4ac8320f3c web/satellite: update wording for upgrade account token flow
Update wording to reflect latest auto upgrade account flow.

new wording:
Send more than $10 in STORJ Tokens to the following deposit address to upgrade to a Pro account.
Your account will be upgraded after your transaction receives 15 confirmations.
If your account is not automatically upgraded, please fill out this limit increase request form.

Issue:
https://github.com/storj/storj-private/issues/367

Change-Id: I46fcb9243722313b98e5c54e8194d6152f7e1631
2023-07-21 15:27:31 +00:00
Michal Niewrzal
d9525a0f27 cmd/tools/migrate-segment-copies: fix placement scan from DB
Placement can be null in DB and we need adjust scanning this column
from DB.

Additionally this change sets application name for DB connection.

Change-Id: I3c7d6294f4a3e5e441160b2fd4aeafffe705ec76
2023-07-21 13:12:08 +00:00
Vitalii
6e4044b245 satellite/payments: slightly refactor user upgrade observer of billing chore
Resolves post-merge comments from here
https://review.dev.storj.io/c/storj/storj/+/10780

Reworked some definitions. Added checks and comments.

Change-Id: I19c63804ad1f30a1ffd8cb87e96f43deed20a685
2023-07-21 03:43:42 +00:00
Wilfred Asomani
2c934d1cfd satellite/console: trigger invoice payment on card remove
This change triggers payment when a card is removed.

Issue: https://github.com/storj/storj/issues/6056

Change-Id: Ia1f2b49609ec9d111cbffb0757bec0b513abb79d
2023-07-20 15:46:50 +00:00
Egon Elbre
dc1509ee42 Makefile,web: bump to node@18
Change-Id: Ib50e18ecda9520a802a5a565e4dfe034ae41e98a
2023-07-20 16:44:13 +03:00
Artur M. Wolff
d45bc879c3 {Jenkinsfile, Makefile}: update Go to 1.20.6
Change-Id: I0a9b483ddcf2e0322840b1eb61982a21cea88e72
2023-07-20 09:20:30 +00:00
Egon Elbre
a058b7e982 web/satellite: fix npm for windows
`NODE_ENV=development xyz` approach does not work with Windows.

Change-Id: I3f0c4a6a2e9566df4923d5e6bc317c7d09573f18
2023-07-20 07:59:03 +00:00
Egon Elbre
7d1031feda satellite/accounting: fix duplicate code-block
Change-Id: Ic6de7337a5ff9a2fec73aa3d8a74ba5b9076a8c4
2023-07-20 07:58:53 +00:00
Jeremy Wharton
ca263c05bb web/satellite/vuetify-poc: go to all projects dashboard on logo click
This change navigates users to the all projects dashboard when the logo
in the navigation bar has been clicked.

Change-Id: Id63b616d12eea510b52f715f3003838541d2ab63
2023-07-19 12:06:29 -05:00
Clement Sam
4cb85186b2 storagenode/pieces: enable lazyfilewalker by default
Resolves https://github.com/storj/storj/issues/5861

Change-Id: I20e0a5b8a15ca966cbccd71369322a517a2c2130
2023-07-19 15:25:20 +00:00
paul cannon
9a871bf3bc go.mod: bump storj.io/common
In particular, to get commit 9db74ed9 in for the next storagenode build.

Change-Id: If07283bc72d66dfbd0ee91bc1cc8b2ce015871cd
2023-07-19 09:50:06 -05:00
Jeremy Wharton
afae5b578e web/satellite/vuetify-poc: allow navigation drawer to be toggled
This change allows the navigation drawer to be toggled by clicking the
the hamburger menu in the navigation bar. The hamburger menu has been
hidden in pages not containing a navigation drawer.

Resolves #6034

Change-Id: I48cfee1f48964c500c07f09f188c7077442261ab
2023-07-19 12:09:05 +00:00
Vitalii
5317135416 satellite/payments: fix config value for auto upgrade user tier flow
Fixed config value which indicates how many base units of US micro dollars are needed to auto upgrade user to paid tier.

Change-Id: I22821ac22fc3eaeeea21c6dec4e6912025df63aa
2023-07-19 10:43:07 +00:00
Wilfred Asomani
7cc873a62a satellite/payments: prevent removing other users' cards
This change patches a loophole that allowed accounts to remove cards
that belong to other users.

Closes #storj/storj-private#326

Change-Id: I33e9efe5c9cdb03aa48ad4c6b1d3283c396a7890
2023-07-19 09:44:06 +00:00
Michal Niewrzal
31bb6d54c7 cmd/tools: add tool to migrate segment copies metadata
We need migrate all existing segment copies to contain all the same
metadata as original segment. So far we were not duplicating stored
pieces but we are changing this behavior right now. We will use this
tool after enabling new way of doing server side copies.

Fixes https://github.com/storj/storj/issues/5890

Change-Id: Ia9ca12486f3c527abd28949eb438d1c4c7138d55
2023-07-18 15:12:51 +00:00
Egon Elbre
23631dc8bb satellite/accounting: fix TestProjectSegmentLimit*
Tally ensures that live accounting has the latest information,
however, when there are concurrent updates to live-accounting
it may by off by a few segments. Disable tally for those tests.

Change-Id: I6fa8a1794334bba093e18f29cb76e7b8d1244979
2023-07-18 14:26:05 +00:00
Egon Elbre
b1e7d70a86 satellite/payments/billing: fix test
`time.Now()` can return the same value when called sequentially.

Change-Id: I800c7696b919ad073e4558fb51c8d2eb4a04f05e
2023-07-18 16:25:42 +03:00
Vitalii
583ad54d86 satellite/{payments, console}: added functionality to get wallet's transactions (including pending)
Added new functionality to query storjscan for all wallet transactions (including pending).
Added new endpoint to query all wallet transactions.

Issue:
https://github.com/storj/storj/issues/5978

Change-Id: Id15fddfc9c95efcaa32aa21403cb177f9297e1ab
2023-07-18 11:09:29 +00:00
Vitalii
2ee0195eba satellite/payments: extend billing chore functionality to upgrade user
Added new observer for billing chore to check user's balance and upgrade their account if balance is more than or equal to needed amount for upgrade.
Added new config value which stands for needed amount of base units of US micro dollars needed to upgrade user.

Issue:
https://github.com/storj/storj/issues/5978

Change-Id: Ic3992cd3114397bfdd9e231ca090ff21ca66648b
2023-07-18 13:15:02 +03:00
Michal Niewrzal
0303920da7 satellite/metainfo: remove unused method
Change-Id: I08e307e6909cdc46951c5f3112d77a685e67fe2e
2023-07-18 08:45:29 +00:00
Jeremy Wharton
df9a6e968e web/satellite: lint Vuetify files
This change enables linting for the Vuetify proof of concept code and
fixes the linting errors that were detected. Additionally, it migrates
the Vuetify components to the composition API.

Change-Id: Id8cc083954e3f4cb66a00ad2715a96c8747b592c
2023-07-17 20:32:59 +00:00
dlamarmorgan
abe1463a73 payments/stripe/invoices: add token payment to overdue invoice payment
Add an attempt to pay overdue invoices via storj token if the user has
a token balance.

Change-Id: I819b89e7cf9cdb7deb9a51eab5ca684b54418218
2023-07-17 12:59:33 -07:00
dlamarmorgan
c96c83e805 satellite/payments/stripe/service: add manual payment with token command
Add the ability to pay an individual users open invoices using their
storj token balance.

Change-Id: I6115f2b033fd77f109ded6f55b1f35fc77c71ff1
2023-07-17 19:24:36 +00:00
dlamarmorgan
0f4371e84c scripts/tests/{backwardcompatibility,integrations}: add test scripts
Change-Id: Ib83cd0f083bab7f560a200fd95e62e5b21e60c27
2023-07-17 18:39:18 +00:00
Wilfred Asomani
0a8115b149 satellite/{console,payments}: fix handling for autofreeze flow
This change adds an extra step to the auto freeze chore to attempt
payment before freezing/warning a user.
It also attempts payment after modifying user's cards whether the user
is frozen/warned or not.

Issue: https://github.com/storj/storj-private/issues/341

Change-Id: Ia9c0c5a2d37837bca5153fe720fef61f1385cb15
2023-07-17 17:37:11 +00:00
Michal Niewrzal
47a4d4986d satellite/repair: enable declumping by default
This feature flag was disabled by default to test it slowly. Its enabled
for some time on one production satellite and test satellites without
any issue. We can enable it by default in code.

Change-Id: If9c36895bbbea12bd4aefa30cb4df912e1729e4c
2023-07-17 15:02:35 +00:00
Michal Niewrzal
5272fd8497 satellite/metainfo: do full bucket validation only on create
We are doing full bucket name validation for many requests but
we should do this only while creating bucket. Other requests will be
covered only by basic name length validation. Less strict validation for
other requests will make bucket usable in case of invalid bucket names
in DB (we have such cases from the past).

https://github.com/storj/storj/issues/6044

Change-Id: I3a41050e3637787f788705ef15b5dc4df4d01fc6
2023-07-17 16:15:33 +02:00
Jeremy Wharton
95761908b5 web/satellite: update Vuetify proof of concept
The changes as of storj/vuetify-storj@c801fe6 have been pulled into the
Vuetify proof of concept.

Change-Id: I3db208836cff21287052615d36258fcf2d4c6169
2023-07-14 12:46:50 +00:00
Michal Niewrzal
5234727886 satellite/repair/repairer: fix flaky TestSegmentRepairPlacement
Sometimes DownloadSelectionCache doesn't keep up with all node
placement changes we are doing during this test.

Change-Id: Idbda6511e3324b560cee3be85f980bf8d5b9b7ef
2023-07-14 10:10:40 +00:00
Tomasz Melcer
5a1c3f7f19
storage/reputation: logging changes to node scores (#5877)
Useful for monitoring storage nodes using log-parsing tools, like swatchdog.

Co-authored-by: Clement Sam <clementsam75@gmail.com>
2023-07-13 17:03:18 +02:00
Wilfred Asomani
4ee647a951 satellite: add request id to requests
This change adds request IDs to requests, logs them as part of audit
logs and sends to the client on error. This is to improve debugging
of customer issues.

Issue: https://github.com/storj/storj/issues/5898

Change-Id: I801514b547d28d810552d91aa7c8502051e552bf
2023-07-13 09:22:43 +00:00
Cameron
e8fcdc10a4 satellite/metainfo: set user_agent in bucket_metainfos on bucket recreation
Before this change, if a user creates a bucket with a user_agent attributed then deletes and recreates it, the row in bucket_metainfos
will not have the user_agent. This is because we skip setting the field
in bucket_metainfos if the bucket already exists in value_attributions.
This can be problematic, as we return the bucket's user agent during the
ListBuckets operation, and the client may be expecting this value to be
populated.

This change ensures the bucket table user_agent is set when (re)creating a bucket. To avoid decreasing BeginObject performance, which also
updates attribution, a flag has been added to determine whether to
make sure the buckets table is updated: `forceBucketUpdate`.

Change-Id: Iada2f233b327b292ad9f98c73ea76a1b0113c926
2023-07-12 21:48:05 +00:00
Michal Niewrzal
99128ab551 satellite/metabase: reuse Pieces while looping segments
Segments loop implementation is using lots of memory to convert
alias pieces to pieces for each segment while iteration. To improve
situation this change is reusing Pieces between batch pages. This
should signifcantly reduce memory usage for ranged loop executions.

Change-Id: I469188779908facb19ad85c6bb7bc3657111cc9a
2023-07-12 09:29:34 +00:00
Jeremy Wharton
062ca285a0 web/satellite: add sharing option to dropdown in buckets page
This change adds a sharing option to the dropdown menu for bucket rows
in the Buckets page.

Resolves #5964

Change-Id: Ife0eb8f6cabbe85eaedae1d94d97694f3c677a3e
2023-07-11 22:03:42 +00:00
Egon Elbre
465941b345 satellite/{nodeselection,overlay}: use location.Set
location.Set is faster for comparisons.

Updates #6028

Change-Id: I764eb5cafc507f908e4168b16a7994cc7721ce4d
2023-07-11 17:16:30 +00:00
Cameron
7e03ccfa46 satellite/console: optional separate web app server
This change creates the ability to run a server separate from the
console web server to serve the front end app. You can run it with
`satellite run ui`. Since there are now potentially two servers instead
of one, the UI server has the option to act as a reverse proxy to the
api server for local development by setting `--console.address` to the
console backend address and `--console.backend-reverse-proxy` to the
console backend's http url. Also, a feature flag has been implemented
on the api server to retain the ability to serve the front end app. It
is toggled with `--console.frontend-enable`.

github issue: https://github.com/storj/storj/issues/5843

Change-Id: I0d30451a20636e3184110dbe28c8a2a8a9505804
2023-07-11 12:17:35 -04:00
Egon Elbre
9370bc4580 satellite/{nodeselection,overlay}: bump common and fix some potential issues
* Handle failed country code conversion.
* Avoid potential issues with a data-race due to shared slice.

Updates #6028

Change-Id: If7beef2619abd084e1f4109de2d323f834a6090a
2023-07-11 11:13:41 +00:00
Michal Niewrzal
1f92e7acda satellite: move GC sender to Core peer
Having separate process/pod only for sending bloom filters once a week
is a bit waste. After reconfiguring production settings to use sender in
core we can remove also GC sender peer code.

https://github.com/storj/storj/issues/5979

Change-Id: I6efe3ec073f96545e1f70ad13843f8ccdf923ee8
2023-07-11 10:31:35 +00:00
Vitalii
a9d979e4d7 web/satellite: update default multipart upload part size
Updated multipart upload part size to be 64MB or higher depending on file size.
Increased queue size from 4 to 5 (5 parts being uploaded at a time) because theoretically it can decrease uploading time, right?

Issue:
https://github.com/storj/storj/issues/5851

Change-Id: Ida5661fa0ed6bc5a0651afc05b5feb7e77791efe
2023-07-10 20:07:51 +03:00
Moby von Briesen
4108aa72ba satellite/console,web/satellite: Fix project limit checking
* Fixes backend to use only a user's owned projects to determine if the
  user has hit the project limit
* Makes frontend logic consistent (and simpler) for checking whether to
  send user to the "Create Project" modal or the "upgrade account or
  request limit increase" modal

Before this change, projects that a user is a member of would be
included in determining whether the user could create a project. Also,
the "create project" button in the projects menu in the navbar of the UI
did not enable a free tier user to create a new project, even if they
had not hit their limits.

Change-Id: Ia776eb627ca37b83f5bc63bed83ee83c9f7cc789
2023-07-10 15:51:00 +00:00
Jeremy Wharton
4e876fbdba web/satellite: update upload modal
The upload modal has been updated to more closely match our designs.
- A button has been added to cancel all in-progress uploads.
- The status text has been fixed to display the proper file count.
- Clicking completed items in the upload modal previews them.

Resolves #5973

Change-Id: Iaee5fe05be14b3a6f2de1a9c807eca5137c7d643
2023-07-10 15:12:31 +00:00
Moby von Briesen
bd4d57c604 satellite/payments: Exclude users who pay via storjscan from autofreeze
Add a configuration (default true) to exclude users who have made
storjscan payments from being auto-warned/frozen for an unpaid invoice.
This will allow us to reach out to these users and handle warning/freezing
manually. Auto account freeze still handles CC-only users.

Fixes https://github.com/storj/storj/issues/6027

Change-Id: I0c862785dad1c8febfa11100c0d30e621ce3ae9b
2023-07-10 13:39:01 +00:00
Jeremy Wharton
c79d1b0d2f {satellite/console,web/satellite}: show error for project invite dupes
This change fixes an issue where a new project member invitation would
silently replace an older one that has the same project ID and email if
the email did not belong to a registered user. Additionally, the
satellite frontend has been updated to display more descriptive error
messages for project member invitations.

Change-Id: I32b582c40c0028b8eedf2aed4b5bfb43501594b4
2023-07-10 12:56:02 +00:00
Jeremy Wharton
fbda13c752 {satellite/console,web/satellite}: trim emails when inviting members
This change trims whitespace from email addresses in project member
invitation requests.

Change-Id: Idd9116820897bf29f3eeba8cf95770b1aa14690c
2023-07-10 12:22:07 +00:00
Jeremy Wharton
0f9a0ba9cd web/satellite: fix project switch when removing project members
This change fixes an issue where the currently selected project would
be switched when removing project members.

Change-Id: I8138b4229eb7933d25a2fe84e5aa0b5846fc79b8
2023-07-10 11:46:49 +00:00
JT Olio
73d65fce9a cmd/satellite/billing: don't fail the overall process if an individual invoice fails
Change-Id: I36591a717ef97bdb417cc6d9218e22b2f91f249b
2023-07-10 11:13:23 +00:00
Michal Niewrzal
1d62dc63f5 satellite/repair/repairer: fix NumHealthyInExcludedCountries calculation
Currently, we have issue were while counting unhealthy pieces we are
counting twice piece which is in excluded country and is outside segment
placement. This can cause unnecessary repair.

This change is also doing another step to move RepairExcludedCountryCodes
from overlay config into repair package.

Change-Id: I3692f6e0ddb9982af925db42be23d644aec1963f
2023-07-10 12:01:19 +02:00
Igor
05f30740f5
docs/testplan: add project cowbell testplan (#6001) 2023-07-10 11:23:55 +02:00
Márton Elek
97a89c3476 satellite: switch to use nodefilters instead of old placement.AllowedCountry
placement.AllowedCountry is the old way to specify placement, with the new approach we can use a more generic (dynamic method), which can check full node information instead of just the country code.

The 90% of this patch is just search and replace:

 * we need to use NodeFilters instead of placement.AllowedCountry
 * which means, we need an initialized PlacementRules available everywhere
 * which means we need to configure the placement rules

The remaining 10% is the placement.go, where we introduced a new type of configuration (lightweight expression language) to define any kind of placement without code change.

Change-Id: Ie644b0b1840871b0e6bbcf80c6b50a947503d7df
2023-07-07 16:55:45 +00:00
Wilfred Asomani
e0b5476e78 web/satellite: fix long table content overflow
This change fixes an issue where long text content in the common table
will overflow and breaks layout.

Change-Id: I30f6e08488410359e0a97995f8d769b272b56c72
2023-07-07 12:24:37 +00:00
Jeremy Wharton
074457fa4e web/satellite: add folder sharing
This change allows users to generate links for sharing folders.
Previously, users were only able to do this with files and buckets.

Resolves #5644

Change-Id: I16dd8270337e3561b6bda895b46d3cc9be5f8041
2023-07-07 10:03:06 +00:00
Vitalii
5fc6eaab17 satellite/{console, web}: display accurate legacy free tier information in upgrade modal
Updated upgrade account modal to show user account free tier limits instead of hardcoded values.

Issue:
https://github.com/storj/storj/issues/5939

Change-Id: I26ffbe2571c5ca4b37f02bec5211bac986bedc6a
2023-07-07 09:23:36 +00:00
Márton Elek
70cdca5d3c
satellite: move satellite/nodeselection/uploadselection => satellite/nodeselection
All the files in uploadselection are (in fact) related to generic node selection, and used not only for upload,
but for download, repair, etc...

Change-Id: Ie4098318a6f8f0bbf672d432761e87047d3762ab
2023-07-07 10:32:03 +02:00
Márton Elek
8b4387a498 satellite/satellitedb: add tag information to nodes selected for upload/downloads
Change-Id: I0fa7daebcf83f7949726e5fffe68e0bdc6fd1d7a
2023-07-07 07:54:16 +00:00
Vitalii
ced8657caa web/satellite: removed unused images
Change-Id: Ifd9c691c69f50a8f346ac4ac2fa1433b00ea81b9
2023-07-06 20:31:24 +00:00
Vitalii
ece0cc5785 web/satellite: fix bottom spacing for all pages
Fixed bottom spacing for all pages. Basically removed bottom padding override in dashboard wrapper.
Removed a couple of unused components and icons.
Made pagination size changer dropdown to appear on top of selector to not extend page height.

This change fixes pagination issue on Team page:
https://github.com/storj/storj/issues/6012

Change-Id: I707dd1bf9d61b05742b7f97a757a2a8f5f9f93fd
2023-07-06 19:57:43 +00:00
JT Olio
a85c080509 docs/blueprints: certified nodes
Change-Id: I7c670d740e4c3d1035dee145ed65aaed4cbaba0c
2023-07-06 14:07:46 -04:00
paul cannon
a4d68b9b7e satellite/metabase: server-side copy copies metadata
..instead of using segment_copies and ancestor_stream_id, etc.

This bypasses reference counting entirely, depending on our mark+sweep+
bloomfilter garbage collection strategy to get rid of pieces once they
are no longer part of a segment.

This is only safe to do after we have stopped passing delete requests on
to storage nodes.

Refs: https://github.com/storj/storj/issues/5889
Change-Id: I37bdcffaa752f84fd85045235d6875b3526b5ecc
2023-07-06 14:40:59 +00:00
Márton Elek
ddf1f1c340 satellite/{nodeselection,overlay}: NodeFilters for dynamic placement implementations
Change-Id: Ica3a7b535fa6736cd8fb12066e615b70e1fa65d6
2023-07-06 12:08:01 +00:00
Vitalii
e3d2f09988 web/satellite: add support link to upgrade account STORJ token flow
Added support link to upgrade account STORJ token flow to tell user that they have to fill in another form to upgrade with tokens only.

Issue:
https://github.com/storj/storj/issues/5985

Change-Id: Ifb852b883c6bf092d5eec588e823925a8ea661c9
2023-07-06 08:38:18 +00:00
Ethan Adams
f819b6a210 satellite/entrypoint: Ignore unset variable errors while checking for VALID_EXECUTABLE
Otherwise core and ranged loop fail at startup with "/entrypoint: line 94: $1: unbound variable"

Change-Id: I45a318038cd937c11f6a00d506c339ba69ea07bf
2023-07-05 20:18:38 +00:00
Márton Elek
1525324384 satellite/uploadselection: avoid String conversation of location during node selection
Converting location to String is not free, better to avoid it.

81cb588c23/storj/location/countrycode.go (L32)

Thanks to Egon, who reported this issue.

See also: https://review.dev.storj.io/c/storj/common/+/10732

Change-Id: Ife348cffa59c020b46914a68be231c6eb75f06c9
2023-07-05 19:22:12 +00:00
Michal Niewrzal
2c3464081f satellite/metainfo: fix bucket name validation
Change-Id: Ifa400ec855ee978ff001fa3736a8a4c1c53fd18c
2023-07-05 14:42:31 +00:00
Márton Elek
6a3802de4f satellite,storagenode: propagate node tags with NodeCheckin
Change-Id: Ib1a602a8cf81204efa001b5d338914ea4218c39b
2023-07-05 13:45:42 +00:00
Clement Sam
a740f96f75 storagenode/pieces/lazyfilewalker: test zapwrapper
This add tests to the zapwrapper package and also adds a test
to verify the issue in https://github.com/storj/storj/issues/6006

Change-Id: Iec3f568e72683af71e1718017109a1ed52794b0b
2023-07-05 12:33:00 +00:00
Clement Sam
7ac2031cac web/multinode: fix wrong free disk space in allocation on dashboard
There are many case where the keywords `free` and `available`
are confused in their usage.

For most cases, `free` space is the amount of free space left
on the whole disk, and not just in allocation while
`available` space is the amount of free space left in the
allocated disk space.

What the user/sno wants to see is not the free space but the
available space. To the SNO, free space is the free space
left in the allocated disk space.

Because of this confusion, the multinode dashboard displays
the `free` disk space instead of the free space in the
allocated disk space https://github.com/storj/storj/issues/5248
While the storagenode dashboard shows the correct free space
in the allocation.

This change fixes the wrong free disk space. I also added a
few comments to make a distinction between the `free`
and `available` fields in the `DiskSpace*` structs.

Change-Id: I11b372ca53a5ac05dc3f79834c18f85ebec11855
2023-07-05 11:24:24 +00:00
Michal Niewrzal
21c1e66a85 satellite/overlay: refactor ReliabilityCache to keep more data
ReliabilityCache will be now using refactored overlay Reliable method.
This method will provide more info about nodes (e.g. country code) and
with this we are able to add two dedicated methods to classify pieces:
* OutOfPlacementPieces
* PiecesNodesLastNetsInOrder

With those new method we will fix issue where offline but reliable node
won't be checked for clumped pieces and off placement pieces.

https://github.com/storj/storj/issues/5998

Change-Id: I9ffbed9f07f4881c9db3bd0e5f0412f1a418dd82
2023-07-05 11:19:10 +02:00
Michal Niewrzal
f2cd7b0928 satellite/overlay: refactor Reliable to be used with repair checker
Currently we are using Reliable to get missing pieces for repair
checker. The issue is that now checker is looking at more things than
just missing pieces (clumped/off, placement pieces) and using only node
ID is not enough. We have issue where we are skipping offline nodes from
clumped and off placement pieces check.

Reliable was refactored to get data (e.g. country, lastNet) about all
reliable nodes. List is split into online and offline. This data will be
cached for quick use by repair checker. It will be also possible to
check nodes metadata like country code or lastNet.

We are also slowly moving `RepairExcludedCountryCodes` config from
overlay to repair which makes more sens for it.

This this first part of changes.

https://github.com/storj/storj/issues/5998

Change-Id: If534342488c0e440affc2894a8fbda6507b8959d
2023-07-05 10:56:31 +02:00
Márton Elek
500b6244f8
satellite/satellitedb: create table for node tags
Change-Id: I884bb740974e6b8241aa6b85faf266b85fe892d4
2023-07-05 09:38:53 +02:00
Clement Sam
1851d103f9 go.mod: bump storj.io/private
Updates https://github.com/storj/storj/issues/6006

Change-Id: I1f549d5642213e420f3e5d0df4ef972c77add713
2023-07-03 23:59:15 +00:00
paul cannon
032546219c satellite/admin: fix spelling of list-apikeys endpoint
Currently, any attempt to list the api keys associated with a project
from the admin UI results in a 404 NOT FOUND error.

This appears to be because there is no /api/projects/{project}/apiKeys
endpoint registered; it should have a lowercase k.

Change-Id: Ifbe4cd0f9ba12a6e37a0d9f64df91c264ced5558
2023-07-03 21:03:53 +00:00
Jeremy Wharton
1173877167 {satellite/console,web/satellite}: encode email in project invite URLs
This change properly encodes email addresses that are used as query
parameters in project invitation-related URLs.

Change-Id: Iaaf7b62b5ac3db3f0b0e000cc06fef8e315400a8
2023-07-03 18:07:19 +00:00
Vitalii
cb41c51692 web/satellite: abort request to get object count in 10 seconds
When entering passphrase to open bucket we make a ListObjectsV2Command request to figure out if there are objects encrypted with different passphrase.
If there are a lot of objects then this request takes forever to process.
By this change I added 10 seconds timeout for that request to not block user.

Issue:
https://github.com/storj/storj/issues/5954

Change-Id: I5243fba68d0b56dff2fb2b3a608a5e71860724c2
2023-07-03 17:33:11 +00:00
Márton Elek
d38b8fa2c4 satellite/nodeselection: use the same Node object from overlay and nodeselection
We use two different Node types in `overlay` and `uploadnodeselection` and converting back and forth.

Using the same object would allow us to use a unified node selection interface everywhere.

Change-Id: Ie71e29d60184ee0e5b4547eb54325f09c418f73c
2023-07-03 16:59:33 +00:00
Márton Elek
20a47034a5
cmd/tools: tag-signer utility to create signed node tags
Change-Id: I2983d688a109325a02fcd060ca1a2d4eb8e9e931
2023-07-03 18:10:08 +02:00
Márton Elek
01e33e7753 cmd/satellite: make satellite docker image compatible with storj-up
This patch makes satellite container images compatible with storj-up.

Which means that any official release can be easily tested locally.

It means that we need some binaries (like storj-up, dlv) and shall fragments part of the production image, but I think the risk is very low and the benefit is very high.

This is the first towards to unify all the images and make it possible to test/run the same components everywhere (storj-up/nigttly/prod).

https://github.com/storj/storj/issues/5946

Change-Id: Ib53b6213d94f93a793a841dedfe32cc59ef69b72
2023-07-03 15:02:27 +00:00
Márton Elek
8482b37c14
go.mod: bump to use latest common
Change-Id: Ie31e7779e86d13ca3c8acaacfe6ed1e23f2c9740
2023-07-03 13:32:35 +02:00
Wilfred Asomani
f131047f1a web/satellite: match projects table with the designs
This change updates the projects table on all projects dashboard to
more closely match the designs.

Change-Id: I547a83352fba8c3ad7958802db7b38b342b383e8
2023-06-30 12:05:47 +00:00
Vitalii
8d8f6734de satellite/{db, accounting}: added functionality to query settled bandwidth for given project
Added functionality to return only settled traffic from project_bandwidth_daily_rollups table for given month.
Updated {projectID}/usage-limits endpoint to return only settled bandwidth used.

This is a possible fix for this issue
https://github.com/storj/storj-private/issues/293

Change-Id: I12516dc898f449c2122e7442b8fbb88309a48ebe
2023-06-30 13:24:16 +03:00
Jeremy Wharton
c006126d54 web/satellite: add Back button to Bucket Details page
A button has been added to the Bucket Details page that returns the
user to the file browser.

Change-Id: Ic2868b1fc9e3b2b0e9785728dc7a116c828eced8
2023-06-30 08:53:36 +00:00
1234 changed files with 96883 additions and 34989 deletions

View File

@ -1,6 +1,6 @@
--- ---
name: "\U0001F41B Bug report" name: "\U0001F41B Bug report"
about: Bugs encountered while using Storj DCS or running a storage node. about: Bugs encountered while using Storj or running a storage node.
title: '' title: ''
labels: Bug labels: Bug
assignees: '' assignees: ''

View File

@ -3,7 +3,7 @@ FROM golang:1.19
WORKDIR /go/storj WORKDIR /go/storj
multinode-web: multinode-web:
FROM node:18 FROM node:18.17
WORKDIR /build WORKDIR /build
COPY web/multinode . COPY web/multinode .
RUN ./build.sh RUN ./build.sh
@ -21,7 +21,7 @@ wasm:
SAVE ARTIFACT release/earthly/wasm wasm AS LOCAL web/satellite/static/wasm SAVE ARTIFACT release/earthly/wasm wasm AS LOCAL web/satellite/static/wasm
storagenode-web: storagenode-web:
FROM node:18 FROM node:18.17
WORKDIR /build WORKDIR /build
COPY web/storagenode . COPY web/storagenode .
RUN ./build.sh RUN ./build.sh
@ -29,16 +29,17 @@ storagenode-web:
SAVE ARTIFACT static AS LOCAL web/storagenode/static SAVE ARTIFACT static AS LOCAL web/storagenode/static
satellite-web: satellite-web:
FROM node:18 FROM node:18.17
WORKDIR /build WORKDIR /build
COPY web/satellite . COPY web/satellite .
RUN ./build.sh RUN ./build.sh
COPY +wasm/wasm static/wasm COPY +wasm/wasm static/wasm
SAVE ARTIFACT dist AS LOCAL web/satellite/dist SAVE ARTIFACT dist AS LOCAL web/satellite/dist
SAVE ARTIFACT dist_vuetify_poc AS LOCAL web/satellite/dist_vuetify_poc
SAVE ARTIFACT static AS LOCAL web/satellite/static SAVE ARTIFACT static AS LOCAL web/satellite/static
satellite-admin: satellite-admin:
FROM node:16 FROM node:18.17
WORKDIR /build WORKDIR /build
COPY satellite/admin/ui . COPY satellite/admin/ui .
RUN ./build.sh RUN ./build.sh
@ -119,6 +120,7 @@ build-tagged-image:
FROM img.dev.storj.io/storjup/base:20230208-1 FROM img.dev.storj.io/storjup/base:20230208-1
COPY +multinode-web/dist /var/lib/storj/storj/web/multinode/dist COPY +multinode-web/dist /var/lib/storj/storj/web/multinode/dist
COPY +satellite-web/dist /var/lib/storj/storj/web/satellite/dist COPY +satellite-web/dist /var/lib/storj/storj/web/satellite/dist
COPY +satellite-web/dist_vuetify_poc /var/lib/storj/storj/web/satellite/dist_vuetify_poc
COPY +satellite-admin/build /app/satellite-admin/ COPY +satellite-admin/build /app/satellite-admin/
COPY +satellite-web/static /var/lib/storj/storj/web/satellite/static COPY +satellite-web/static /var/lib/storj/storj/web/satellite/static
COPY +storagenode-web/dist /var/lib/storj/storj/web/storagenode/dist COPY +storagenode-web/dist /var/lib/storj/storj/web/storagenode/dist

37
Jenkinsfile vendored
View File

@ -10,41 +10,6 @@ node('node') {
echo "Current build result: ${currentBuild.result}" echo "Current build result: ${currentBuild.result}"
} }
if (env.BRANCH_NAME == "main") {
stage('Run Versions Test') {
lastStage = env.STAGE_NAME
try {
echo "Running Versions test"
env.STORJ_SIM_POSTGRES = 'postgres://postgres@postgres:5432/teststorj?sslmode=disable'
env.STORJ_SIM_REDIS = 'redis:6379'
echo "STORJ_SIM_POSTGRES: $STORJ_SIM_POSTGRES"
echo "STORJ_SIM_REDIS: $STORJ_SIM_REDIS"
sh 'docker run --rm -d -e POSTGRES_HOST_AUTH_METHOD=trust --name postgres-$BUILD_NUMBER postgres:12.3'
sh 'docker run --rm -d --name redis-$BUILD_NUMBER redis:latest'
sh '''until $(docker logs postgres-$BUILD_NUMBER | grep "database system is ready to accept connections" > /dev/null)
do printf '.'
sleep 5
done
'''
sh 'docker exec postgres-$BUILD_NUMBER createdb -U postgres teststorj'
// fetch the remote main branch
sh 'git fetch --no-tags --progress -- https://github.com/storj/storj.git +refs/heads/main:refs/remotes/origin/main'
sh 'docker run -u $(id -u):$(id -g) --rm -i -v $PWD:$PWD -w $PWD --entrypoint $PWD/scripts/tests/testversions/test-sim-versions.sh -e STORJ_SIM_POSTGRES -e STORJ_SIM_REDIS --link redis-$BUILD_NUMBER:redis --link postgres-$BUILD_NUMBER:postgres storjlabs/golang:1.20.3'
}
catch(err){
throw err
}
finally {
sh 'docker stop postgres-$BUILD_NUMBER || true'
sh 'docker rm postgres-$BUILD_NUMBER || true'
sh 'docker stop redis-$BUILD_NUMBER || true'
sh 'docker rm redis-$BUILD_NUMBER || true'
}
}
}
stage('Run Rolling Upgrade Test') { stage('Run Rolling Upgrade Test') {
lastStage = env.STAGE_NAME lastStage = env.STAGE_NAME
@ -69,7 +34,7 @@ node('node') {
sh 'docker exec postgres-$BUILD_NUMBER createdb -U postgres teststorj' sh 'docker exec postgres-$BUILD_NUMBER createdb -U postgres teststorj'
// fetch the remote main branch // fetch the remote main branch
sh 'git fetch --no-tags --progress -- https://github.com/storj/storj.git +refs/heads/main:refs/remotes/origin/main' sh 'git fetch --no-tags --progress -- https://github.com/storj/storj.git +refs/heads/main:refs/remotes/origin/main'
sh 'docker run -u $(id -u):$(id -g) --rm -i -v $PWD:$PWD -w $PWD --entrypoint $PWD/scripts/tests/rollingupgrade/test-sim-rolling-upgrade.sh -e BRANCH_NAME -e STORJ_SIM_POSTGRES -e STORJ_SIM_REDIS -e STORJ_MIGRATION_DB --link redis-$BUILD_NUMBER:redis --link postgres-$BUILD_NUMBER:postgres storjlabs/golang:1.20.3' sh 'docker run -u $(id -u):$(id -g) --rm -i -v $PWD:$PWD -w $PWD --entrypoint $PWD/scripts/tests/rollingupgrade/test-sim-rolling-upgrade.sh -e BRANCH_NAME -e STORJ_SIM_POSTGRES -e STORJ_SIM_REDIS -e STORJ_MIGRATION_DB --link redis-$BUILD_NUMBER:redis --link postgres-$BUILD_NUMBER:postgres storjlabs/golang:1.21.3'
} }
catch(err){ catch(err){
throw err throw err

View File

@ -229,50 +229,64 @@ pipeline {
} }
} }
stage('wasm npm') { stage('Test Web') {
steps { parallel {
dir(".build") { stage('wasm npm') {
sh 'cp -r ../satellite/console/wasm/tests/ .' steps {
sh 'cd tests && cp "$(go env GOROOT)/misc/wasm/wasm_exec.js" .' dir(".build") {
sh 'cd tests && npm install && npm run test' sh 'cp -r ../satellite/console/wasm/tests/ .'
sh 'cd tests && cp "$(go env GOROOT)/misc/wasm/wasm_exec.js" .'
sh 'cd tests && npm install && npm run test'
}
}
} }
}
}
stage('web/satellite') { stage('web/satellite') {
steps { steps {
dir("web/satellite") { dir("web/satellite") {
sh 'npm run lint-ci' sh 'npm run lint-ci'
sh script: 'npm audit', returnStatus: true sh script: 'npm audit', returnStatus: true
sh 'npm run test' sh 'npm run test'
}
}
} }
}
}
stage('web/storagenode') { stage('web/storagenode') {
steps { steps {
dir("web/storagenode") { dir("web/storagenode") {
sh 'npm run lint-ci' sh 'npm run lint-ci'
sh script: 'npm audit', returnStatus: true sh script: 'npm audit', returnStatus: true
sh 'npm run test' sh 'npm run test'
}
}
} }
}
}
stage('web/multinode') { stage('web/multinode') {
steps { steps {
dir("web/multinode") { dir("web/multinode") {
sh 'npm run lint-ci' sh 'npm run lint-ci'
sh script: 'npm audit', returnStatus: true sh script: 'npm audit', returnStatus: true
sh 'npm run test' sh 'npm run test'
}
}
} }
}
}
stage('satellite/admin/ui') { stage('satellite/admin/ui') {
steps { steps {
dir("satellite/admin/ui") { dir("satellite/admin/ui") {
sh script: 'npm audit', returnStatus: true sh script: 'npm audit', returnStatus: true
}
}
}
stage('satellite/admin/back-office/ui') {
steps {
dir("satellite/admin/back-office/ui") {
sh 'npm install --prefer-offline --no-audit --loglevel verbose'
sh 'npm run lint-ci'
sh script: 'npm audit', returnStatus: true
}
}
} }
} }
} }

View File

@ -125,6 +125,7 @@ pipeline {
sh 'check-atomic-align ./...' sh 'check-atomic-align ./...'
sh 'check-monkit ./...' sh 'check-monkit ./...'
sh 'check-errs ./...' sh 'check-errs ./...'
sh 'check-deferloop ./...'
sh 'staticcheck ./...' sh 'staticcheck ./...'
sh 'golangci-lint --config /go/ci/.golangci.yml -j=2 run' sh 'golangci-lint --config /go/ci/.golangci.yml -j=2 run'
sh 'check-downgrades' sh 'check-downgrades'

64
Jenkinsfile.versions Normal file
View File

@ -0,0 +1,64 @@
def lastStage = ''
node('node') {
properties([disableConcurrentBuilds()])
try {
currentBuild.result = "SUCCESS"
stage('Checkout') {
lastStage = env.STAGE_NAME
checkout scm
echo "Current build result: ${currentBuild.result}"
}
stage('Run Versions Test') {
lastStage = env.STAGE_NAME
try {
echo "Running Versions test"
env.STORJ_SIM_POSTGRES = 'postgres://postgres@postgres:5432/teststorj?sslmode=disable'
env.STORJ_SIM_REDIS = 'redis:6379'
echo "STORJ_SIM_POSTGRES: $STORJ_SIM_POSTGRES"
echo "STORJ_SIM_REDIS: $STORJ_SIM_REDIS"
sh 'docker run --rm -d -e POSTGRES_HOST_AUTH_METHOD=trust --name postgres-$BUILD_NUMBER postgres:12.3'
sh 'docker run --rm -d --name redis-$BUILD_NUMBER redis:latest'
sh '''until $(docker logs postgres-$BUILD_NUMBER | grep "database system is ready to accept connections" > /dev/null)
do printf '.'
sleep 5
done
'''
sh 'docker exec postgres-$BUILD_NUMBER createdb -U postgres teststorj'
// fetch the remote main branch
sh 'git fetch --no-tags --progress -- https://github.com/storj/storj.git +refs/heads/main:refs/remotes/origin/main'
sh 'docker run -u $(id -u):$(id -g) --rm -i -v $PWD:$PWD -w $PWD --entrypoint $PWD/scripts/tests/testversions/test-sim-versions.sh -e STORJ_SIM_POSTGRES -e STORJ_SIM_REDIS --link redis-$BUILD_NUMBER:redis --link postgres-$BUILD_NUMBER:postgres storjlabs/golang:1.21.3'
}
catch(err){
throw err
}
finally {
sh 'docker stop postgres-$BUILD_NUMBER || true'
sh 'docker rm postgres-$BUILD_NUMBER || true'
sh 'docker stop redis-$BUILD_NUMBER || true'
sh 'docker rm redis-$BUILD_NUMBER || true'
}
}
}
catch (err) {
echo "Caught errors! ${err}"
echo "Setting build result to FAILURE"
currentBuild.result = "FAILURE"
slackSend color: 'danger', message: "@build-team ${env.BRANCH_NAME} build failed during stage ${lastStage} ${env.BUILD_URL}"
throw err
}
finally {
stage('Cleanup') {
deleteDir()
}
}
}

View File

@ -33,7 +33,7 @@ Here we need to post changes for each topic(storj-sim, Uplink, Sattelite, Storag
Then its time to cut the release branch: Then its time to cut the release branch:
`git checkout -b v1.3` - will create and checkout branch v1.3 `git checkout -b v1.3` - will create and checkout branch v1.3
`git push origin v1.3`- will push release branch to the repo `git push origin v1.3`- will push release branch to the repo\
Also we need to cut same release branch on tardigrade-satellite-theme repo Also we need to cut same release branch on tardigrade-satellite-theme repo
`git checkout -b v1.3` - will create and checkout branch v1.3 `git checkout -b v1.3` - will create and checkout branch v1.3
`git push origin v1.3`- will push release branch to the repo `git push origin v1.3`- will push release branch to the repo
@ -42,15 +42,22 @@ The next step is to create tag for `storj` repo using `tag-release.sh` which is
Example: Example:
`./scripts/tag-release.sh v1.3.0-rc` `./scripts/tag-release.sh v1.3.0-rc`
`git push origin v1.3.0-rc` `git push origin v1.3.0-rc`
Then verify that the Jenkins job of the build Storj V3 for such tag and branch has finished successfully. Then verify that the Jenkins job of the build Storj V3 for such tag and branch has finished successfully.\
Pay attention to tardigrade-satellite-theme job - it should be successfully finished as well.
## How to cherry pick ## How to cherry pick
If you need to cherry-pick something after the release branch has been created then you need to create point release. If you need to cherry-pick something after the release branch has been created then you need to create point release.
Make sure that you have the latest changes, checkout the release branch and execute cherry-pick: Make sure that you have the latest changes, checkout the release branch and execute cherry-pick:
`git cherry-pick <your commit hash>` ```
You need to create pull request to the release branch with that commit. After the pull request will be approved and merged you should create new release tag: git fetch
git checkout -b <xxx>/cherry-pick-v1.xx
git cherry-pick <your commit hash>
```
You need push and create pull request to the release branch with that commit.
`git push origin <xxx>/cherry-pick-v1.xx`
After the pull request will be approved, pass all tests and merged you should create new release tag:
`./scripts/tag-release.sh v1.3.1` `./scripts/tag-release.sh v1.3.1`
and push the tag to the repo: and push the tag to the repo:
`git push origin v1.3.1` `git push origin v1.3.1`
@ -64,10 +71,25 @@ git push origin release-v1.3
``` ```
Update Jenkins job. Update Jenkins job.
## Revert from release
If revert needed we proceed with next flow:
Ask developer to fix problem and push commit to main branch. After that cherry-pick fix to the release branch.
Why we do use this flow but not revert from the release branch? It's to prevent situation to fix bug in the main.
## Where to find the release binaries ## Where to find the release binaries
After Jenkins job for this release finished it will automaticaly post this tag on [GitHub release page](https://github.com/storj/storj/releases). The status will be `Draft`. After Jenkins job for this release finished it will automaticaly post this tag on [GitHub release page](https://github.com/storj/storj/releases). The status will be `Draft`.
Update this tag with changelog that you previosly created. Update this tag with changelog that you previously created.\
For now changelog is generated automatically, but binaries for darwin not. Darwin binaries should be generated manually and added to tag.\
Add New Contributors list to the release. To generate it:
`git shortlog -sn release-v1.2 | cut -f 2 > ../old.txt && git shortlog -sn release-v1.3 | cut -f 2 > ../new.txt && grep -Fxv -f ../old.txt ../new.txt`
Note, to run this command current and previous release should be on your local machine.
## Setting the 'Latest' release version
After 100% storagenodes rollout is finished -> new release should be set as 'Latest'.
## Which tests do we want to execute ## Which tests do we want to execute
Everything that could break production. Everything that could break production.

View File

@ -1,8 +1,8 @@
GO_VERSION ?= 1.20.3 GO_VERSION ?= 1.21.3
GOOS ?= linux GOOS ?= linux
GOARCH ?= amd64 GOARCH ?= amd64
GOPATH ?= $(shell go env GOPATH) GOPATH ?= $(shell go env GOPATH)
NODE_VERSION ?= 16.11.1 NODE_VERSION ?= 18.17.0
COMPOSE_PROJECT_NAME := ${TAG}-$(shell git rev-parse --abbrev-ref HEAD) COMPOSE_PROJECT_NAME := ${TAG}-$(shell git rev-parse --abbrev-ref HEAD)
BRANCH_NAME ?= $(shell git rev-parse --abbrev-ref HEAD | sed "s!/!-!g") BRANCH_NAME ?= $(shell git rev-parse --abbrev-ref HEAD | sed "s!/!-!g")
GIT_TAG := $(shell git rev-parse --short HEAD) GIT_TAG := $(shell git rev-parse --short HEAD)
@ -73,6 +73,8 @@ build-multinode-npm:
cd web/multinode && npm ci cd web/multinode && npm ci
build-satellite-admin-npm: build-satellite-admin-npm:
cd satellite/admin/ui && npm ci cd satellite/admin/ui && npm ci
# Temporary until the new back-office replaces the current admin API & UI
cd satellite/admin/back-office/ui && npm ci
##@ Simulator ##@ Simulator
@ -126,7 +128,7 @@ lint:
-v ${GOPATH}/pkg:/go/pkg \ -v ${GOPATH}/pkg:/go/pkg \
-v ${PWD}:/storj \ -v ${PWD}:/storj \
-w /storj \ -w /storj \
storjlabs/ci-slim \ storjlabs/ci:slim \
make .lint LINT_TARGET="$(LINT_TARGET)" make .lint LINT_TARGET="$(LINT_TARGET)"
.PHONY: .lint/testsuite/ui .PHONY: .lint/testsuite/ui
@ -286,6 +288,14 @@ satellite-admin-ui:
-u $(shell id -u):$(shell id -g) \ -u $(shell id -u):$(shell id -g) \
node:${NODE_VERSION} \ node:${NODE_VERSION} \
/bin/bash -c "npm ci && npm run build" /bin/bash -c "npm ci && npm run build"
# Temporary until the new back-office replaces the current admin API & UI
docker run --rm -i \
--mount type=bind,src="${PWD}",dst=/go/src/storj.io/storj \
-w /go/src/storj.io/storj/satellite/admin/back-office/ui \
-e HOME=/tmp \
-u $(shell id -u):$(shell id -g) \
node:${NODE_VERSION} \
/bin/bash -c "npm ci && npm run build"
.PHONY: satellite-wasm .PHONY: satellite-wasm
satellite-wasm: satellite-wasm:
@ -464,7 +474,9 @@ binaries-upload: ## Upload binaries to Google Storage (jenkins)
zip -r "$${zipname}.zip" "$${filename}" \ zip -r "$${zipname}.zip" "$${filename}" \
; fi \ ; fi \
; done ; done
cd "release/${TAG}"; gsutil -m cp -r *.zip "gs://storj-v3-alpha-builds/${TAG}/" cd "release/${TAG}" \
&& sha256sum *.zip > sha256sums \
&& gsutil -m cp -r *.zip sha256sums "gs://storj-v3-alpha-builds/${TAG}/"
.PHONY: draft-release .PHONY: draft-release
draft-release: draft-release:

View File

@ -4,7 +4,11 @@
[![Go Doc](https://img.shields.io/badge/godoc-reference-blue.svg?style=flat-square)](https://pkg.go.dev/storj.io/storj) [![Go Doc](https://img.shields.io/badge/godoc-reference-blue.svg?style=flat-square)](https://pkg.go.dev/storj.io/storj)
[![Coverage Status](https://img.shields.io/badge/coverage-master-green.svg)](https://build.dev.storj.io/job/storj/job/main/cobertura) [![Coverage Status](https://img.shields.io/badge/coverage-master-green.svg)](https://build.dev.storj.io/job/storj/job/main/cobertura)
<img src="https://github.com/storj/storj/raw/main/resources/logo.png" width="100"> <picture>
<source media="(prefers-color-scheme: dark)" srcset="https://github.com/storj/.github/assets/3217669/15b2f86d-e585-430f-83f8-67cccda07f73">
<source media="(prefers-color-scheme: light)" srcset="https://github.com/storj/.github/assets/3217669/de7657b7-0497-4b72-8d71-99bf210164dc">
<img alt="Storj logo" src="https://github.com/storj/.github/assets/3217669/de7657b7-0497-4b72-8d71-99bf210164dc" height="100">
</picture>
Storj is building a distributed cloud storage network. Storj is building a distributed cloud storage network.
[Check out our white paper for more info!](https://storj.io/storj.pdf) [Check out our white paper for more info!](https://storj.io/storj.pdf)

View File

@ -62,8 +62,6 @@ func RunCommand(runCfg *Config) *cobra.Command {
ctx, _ := process.Ctx(cmd) ctx, _ := process.Ctx(cmd)
log := zap.L() log := zap.L()
runCfg.Debug.Address = *process.DebugAddrFlag
identity, err := runCfg.Identity.Load() identity, err := runCfg.Identity.Load()
if err != nil { if err != nil {
log.Error("failed to load identity.", zap.Error(err)) log.Error("failed to load identity.", zap.Error(err))

View File

@ -134,8 +134,6 @@ func cmdRun(cmd *cobra.Command, args []string) (err error) {
ctx, _ := process.Ctx(cmd) ctx, _ := process.Ctx(cmd)
log := zap.L() log := zap.L()
runCfg.Debug.Address = *process.DebugAddrFlag
identity, err := getIdentity(ctx, &runCfg) identity, err := getIdentity(ctx, &runCfg)
if err != nil { if err != nil {
log.Error("failed to load identity", zap.Error(err)) log.Error("failed to load identity", zap.Error(err))

View File

@ -1,34 +1,51 @@
ARG DOCKER_ARCH ARG DOCKER_ARCH
# Satellite UI static asset generation # Satellite UI static asset generation
FROM node:16.11.1 as ui FROM node:18.17.0 as ui
WORKDIR /app WORKDIR /app
COPY web/satellite/ /app COPY web/satellite/ /app
# Need to clean up (or ignore) local folders like node_modules, etc... # Need to clean up (or ignore) local folders like node_modules, etc...
RUN npm install RUN npm install
RUN npm run build RUN npm run build
RUN npm run build-vuetify
# Fetch ca-certificates file for arch independent builds below # Fetch ca-certificates file for arch independent builds below
FROM debian:buster-slim as ca-cert FROM debian:buster-slim as ca-cert
RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates
RUN update-ca-certificates RUN update-ca-certificates
# Install storj-up helper (for local/dev runs)
FROM --platform=$TARGETPLATFORM golang:1.19 AS storjup
RUN --mount=type=cache,target=/root/.cache/go-build \
--mount=type=cache,target=/go/pkg/mod \
go install storj.io/storj-up@latest
# Install dlv (for local/dev runs)
FROM --platform=$TARGETPLATFORM golang:1.19 AS dlv
RUN --mount=type=cache,target=/root/.cache/go-build \
--mount=type=cache,target=/go/pkg/mod \
go install github.com/go-delve/delve/cmd/dlv@latest
FROM ${DOCKER_ARCH:-amd64}/debian:buster-slim FROM ${DOCKER_ARCH:-amd64}/debian:buster-slim
ARG TAG ARG TAG
ARG GOARCH ARG GOARCH
ENV GOARCH ${GOARCH} ENV GOARCH ${GOARCH}
ENV CONF_PATH=/root/.local/share/storj/satellite \ ENV CONF_PATH=/root/.local/share/storj/satellite \
STORJ_CONSOLE_STATIC_DIR=/app \ STORJ_CONSOLE_STATIC_DIR=/app \
STORJ_MAIL_TEMPLATE_PATH=/app/static/emails \
STORJ_CONSOLE_ADDRESS=0.0.0.0:10100 STORJ_CONSOLE_ADDRESS=0.0.0.0:10100
ENV PATH=$PATH:/app
EXPOSE 7777 EXPOSE 7777
EXPOSE 10100 EXPOSE 10100
WORKDIR /app WORKDIR /app
COPY --from=ui /app/static /app/static COPY --from=ui /app/static /app/static
COPY --from=ui /app/dist /app/dist COPY --from=ui /app/dist /app/dist
COPY --from=ui /app/dist_vuetify_poc /app/dist_vuetify_poc
COPY --from=ca-cert /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt COPY --from=ca-cert /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
COPY release/${TAG}/wasm/access.wasm /app/static/wasm/ COPY release/${TAG}/wasm /app/static/wasm
COPY release/${TAG}/wasm/wasm_exec.js /app/static/wasm/
COPY release/${TAG}/wasm/access.wasm.br /app/static/wasm/
COPY release/${TAG}/wasm/wasm_exec.js.br /app/static/wasm/
COPY release/${TAG}/satellite_linux_${GOARCH:-amd64} /app/satellite COPY release/${TAG}/satellite_linux_${GOARCH:-amd64} /app/satellite
COPY --from=storjup /go/bin/storj-up /usr/local/bin/storj-up
COPY --from=dlv /go/bin/dlv /usr/local/bin/dlv
# test identities for quick-start
COPY --from=img.dev.storj.io/storjup/base:20230607-1 /var/lib/storj/identities /var/lib/storj/identities
COPY cmd/satellite/entrypoint /entrypoint COPY cmd/satellite/entrypoint /entrypoint
ENTRYPOINT ["/entrypoint"] ENTRYPOINT ["/entrypoint"]

View File

@ -11,6 +11,8 @@ import (
"storj.io/private/process" "storj.io/private/process"
"storj.io/private/version" "storj.io/private/version"
"storj.io/storj/satellite" "storj.io/storj/satellite"
"storj.io/storj/satellite/accounting"
"storj.io/storj/satellite/accounting/live"
"storj.io/storj/satellite/metabase" "storj.io/storj/satellite/metabase"
"storj.io/storj/satellite/satellitedb" "storj.io/storj/satellite/satellitedb"
) )
@ -19,8 +21,6 @@ func cmdAdminRun(cmd *cobra.Command, args []string) (err error) {
ctx, _ := process.Ctx(cmd) ctx, _ := process.Ctx(cmd)
log := zap.L() log := zap.L()
runCfg.Debug.Address = *process.DebugAddrFlag
identity, err := runCfg.Identity.Load() identity, err := runCfg.Identity.Load()
if err != nil { if err != nil {
log.Error("Failed to load identity.", zap.Error(err)) log.Error("Failed to load identity.", zap.Error(err))
@ -47,7 +47,21 @@ func cmdAdminRun(cmd *cobra.Command, args []string) (err error) {
err = errs.Combine(err, metabaseDB.Close()) err = errs.Combine(err, metabaseDB.Close())
}() }()
peer, err := satellite.NewAdmin(log, identity, db, metabaseDB, version.Build, &runCfg.Config, process.AtomicLevel(cmd)) accountingCache, err := live.OpenCache(ctx, log.Named("live-accounting"), runCfg.LiveAccounting)
if err != nil {
if !accounting.ErrSystemOrNetError.Has(err) || accountingCache == nil {
return errs.New("Error instantiating live accounting cache: %w", err)
}
log.Warn("Unable to connect to live accounting cache. Verify connection",
zap.Error(err),
)
}
defer func() {
err = errs.Combine(err, accountingCache.Close())
}()
peer, err := satellite.NewAdmin(log, identity, db, metabaseDB, accountingCache, version.Build, &runCfg.Config, process.AtomicLevel(cmd))
if err != nil { if err != nil {
return err return err
} }

View File

@ -26,8 +26,6 @@ func cmdAPIRun(cmd *cobra.Command, args []string) (err error) {
ctx, _ := process.Ctx(cmd) ctx, _ := process.Ctx(cmd)
log := zap.L() log := zap.L()
runCfg.Debug.Address = *process.DebugAddrFlag
identity, err := runCfg.Identity.Load() identity, err := runCfg.Identity.Load()
if err != nil { if err != nil {
log.Error("Failed to load identity.", zap.Error(err)) log.Error("Failed to load identity.", zap.Error(err))

View File

@ -21,8 +21,6 @@ func cmdAuditorRun(cmd *cobra.Command, args []string) (err error) {
ctx, _ := process.Ctx(cmd) ctx, _ := process.Ctx(cmd)
log := zap.L() log := zap.L()
runCfg.Debug.Address = *process.DebugAddrFlag
identity, err := runCfg.Identity.Load() identity, err := runCfg.Identity.Load()
if err != nil { if err != nil {
log.Error("Failed to load identity.", zap.Error(err)) log.Error("Failed to load identity.", zap.Error(err))

View File

@ -1,6 +1,7 @@
#!/bin/bash #!/bin/bash
set -euo pipefail set -euo pipefail
## production helpers
SETUP_PARAMS="" SETUP_PARAMS=""
if [ -n "${IDENTITY_ADDR:-}" ]; then if [ -n "${IDENTITY_ADDR:-}" ]; then
@ -21,6 +22,10 @@ if [ "${SATELLITE_API:-}" = "true" ]; then
exec ./satellite run api $RUN_PARAMS "$@" exec ./satellite run api $RUN_PARAMS "$@"
fi fi
if [ "${SATELLITE_UI:-}" = "true" ]; then
exec ./satellite run ui $RUN_PARAMS "$@"
fi
if [ "${SATELLITE_GC:-}" = "true" ]; then if [ "${SATELLITE_GC:-}" = "true" ]; then
exec ./satellite run garbage-collection $RUN_PARAMS "$@" exec ./satellite run garbage-collection $RUN_PARAMS "$@"
fi fi
@ -37,4 +42,63 @@ if [ "${SATELLITE_AUDITOR:-}" = "true" ]; then
exec ./satellite run auditor $RUN_PARAMS "$@" exec ./satellite run auditor $RUN_PARAMS "$@"
fi fi
exec ./satellite run $RUN_PARAMS "$@" ## storj-up helpers
if [ "${STORJUP_ROLE:-""}" ]; then
if [ "${STORJ_IDENTITY_DIR:-""}" ]; then
#Generate identity if missing
if [ ! -f "$STORJ_IDENTITY_DIR/identity.key" ]; then
if [ "$STORJ_USE_PREDEFINED_IDENTITY" ]; then
# use predictable, pre-generated identity
mkdir -p $(dirname $STORJ_IDENTITY_DIR)
cp -r /var/lib/storj/identities/$STORJ_USE_PREDEFINED_IDENTITY $STORJ_IDENTITY_DIR
else
identity --identity-dir $STORJ_IDENTITY_DIR --difficulty 8 create .
fi
fi
fi
if [ "${STORJ_WAIT_FOR_DB:-""}" ]; then
storj-up util wait-for-port cockroach:26257
storj-up util wait-for-port redis:6379
fi
if [ "${STORJUP_ROLE:-""}" == "satellite-api" ]; then
mkdir -p /var/lib/storj/.local
#only migrate first time
if [ ! -f "/var/lib/storj/.local/migrated" ]; then
satellite run migration --identity-dir $STORJ_IDENTITY_DIR
touch /var/lib/storj/.local/migrated
fi
fi
# default config generated without arguments is misleading
rm /root/.local/share/storj/satellite/config.yaml
mkdir -p /var/lib/storj/.local/share/storj/satellite || true
if [ "${GO_DLV:-""}" ]; then
echo "Starting with go dlv"
#absolute file path is required
CMD=$(which $1)
shift
/usr/local/bin/dlv --listen=:2345 --headless=true --api-version=2 --accept-multiclient exec --check-go-version=false -- $CMD "$@"
exit $?
fi
fi
# for backward compatibility reason, we use argument as command, only if it's an executable (and use it as satellite flags oterwise)
set +eo nounset
which "$1" > /dev/null
VALID_EXECUTABLE=$?
set -eo nounset
if [ $VALID_EXECUTABLE -eq 0 ]; then
# this is a full command (what storj-up uses)
exec "$@"
else
# legacy, run-only parameters
exec ./satellite run $RUN_PARAMS "$@"
fi

View File

@ -20,8 +20,6 @@ func cmdGCBloomFilterRun(cmd *cobra.Command, args []string) (err error) {
ctx, _ := process.Ctx(cmd) ctx, _ := process.Ctx(cmd)
log := zap.L() log := zap.L()
runCfg.Debug.Address = *process.DebugAddrFlag
db, err := satellitedb.Open(ctx, log.Named("db"), runCfg.Database, satellitedb.Options{ApplicationName: "satellite-gc-bloomfilter"}) db, err := satellitedb.Open(ctx, log.Named("db"), runCfg.Database, satellitedb.Options{ApplicationName: "satellite-gc-bloomfilter"})
if err != nil { if err != nil {
return errs.New("Error starting master database on satellite GC: %+v", err) return errs.New("Error starting master database on satellite GC: %+v", err)

View File

@ -20,8 +20,6 @@ func cmdGCRun(cmd *cobra.Command, args []string) (err error) {
ctx, _ := process.Ctx(cmd) ctx, _ := process.Ctx(cmd)
log := zap.L() log := zap.L()
runCfg.Debug.Address = *process.DebugAddrFlag
identity, err := runCfg.Identity.Load() identity, err := runCfg.Identity.Load()
if err != nil { if err != nil {
log.Error("Failed to load identity.", zap.Error(err)) log.Error("Failed to load identity.", zap.Error(err))

View File

@ -27,7 +27,7 @@ import (
) )
// generateGracefulExitCSV creates a report with graceful exit data for exiting or exited nodes in a given period. // generateGracefulExitCSV creates a report with graceful exit data for exiting or exited nodes in a given period.
func generateGracefulExitCSV(ctx context.Context, completed bool, start time.Time, end time.Time, output io.Writer) error { func generateGracefulExitCSV(ctx context.Context, timeBased bool, completed bool, start time.Time, end time.Time, output io.Writer) error {
db, err := satellitedb.Open(ctx, zap.L().Named("db"), reportsGracefulExitCfg.Database, satellitedb.Options{ApplicationName: "satellite-gracefulexit"}) db, err := satellitedb.Open(ctx, zap.L().Named("db"), reportsGracefulExitCfg.Database, satellitedb.Options{ApplicationName: "satellite-gracefulexit"})
if err != nil { if err != nil {
return errs.New("error connecting to master database on satellite: %+v", err) return errs.New("error connecting to master database on satellite: %+v", err)
@ -67,11 +67,14 @@ func generateGracefulExitCSV(ctx context.Context, completed bool, start time.Tim
if err != nil { if err != nil {
return err return err
} }
exitProgress, err := db.GracefulExit().GetProgress(ctx, id) exitProgress := &gracefulexit.Progress{}
if gracefulexit.ErrNodeNotFound.Has(err) { if !timeBased {
exitProgress = &gracefulexit.Progress{} exitProgress, err = db.GracefulExit().GetProgress(ctx, id)
} else if err != nil { if gracefulexit.ErrNodeNotFound.Has(err) {
return err exitProgress = &gracefulexit.Progress{}
} else if err != nil {
return err
}
} }
exitStatus := node.ExitStatus exitStatus := node.ExitStatus

View File

@ -40,7 +40,7 @@ import (
"storj.io/storj/satellite/accounting/live" "storj.io/storj/satellite/accounting/live"
"storj.io/storj/satellite/compensation" "storj.io/storj/satellite/compensation"
"storj.io/storj/satellite/metabase" "storj.io/storj/satellite/metabase"
"storj.io/storj/satellite/overlay" "storj.io/storj/satellite/nodeselection"
"storj.io/storj/satellite/payments/stripe" "storj.io/storj/satellite/payments/stripe"
"storj.io/storj/satellite/satellitedb" "storj.io/storj/satellite/satellitedb"
) )
@ -100,6 +100,11 @@ var (
Short: "Run the satellite API", Short: "Run the satellite API",
RunE: cmdAPIRun, RunE: cmdAPIRun,
} }
runUICmd = &cobra.Command{
Use: "ui",
Short: "Run the satellite UI",
RunE: cmdUIRun,
}
runRepairerCmd = &cobra.Command{ runRepairerCmd = &cobra.Command{
Use: "repair", Use: "repair",
Short: "Run the repair service", Short: "Run the repair service",
@ -221,6 +226,9 @@ var (
Long: "Creates stripe invoice line items for stripe customer balances obtained from past invoices and other miscellaneous charges.", Long: "Creates stripe invoice line items for stripe customer balances obtained from past invoices and other miscellaneous charges.",
RunE: cmdCreateCustomerBalanceInvoiceItems, RunE: cmdCreateCustomerBalanceInvoiceItems,
} }
aggregate = false
prepareCustomerInvoiceRecordsCmd = &cobra.Command{ prepareCustomerInvoiceRecordsCmd = &cobra.Command{
Use: "prepare-invoice-records [period]", Use: "prepare-invoice-records [period]",
Short: "Prepares invoice project records", Short: "Prepares invoice project records",
@ -235,6 +243,13 @@ var (
Args: cobra.ExactArgs(1), Args: cobra.ExactArgs(1),
RunE: cmdCreateCustomerProjectInvoiceItems, RunE: cmdCreateCustomerProjectInvoiceItems,
} }
createCustomerAggregatedProjectInvoiceItemsCmd = &cobra.Command{
Use: "create-aggregated-project-invoice-items [period]",
Short: "Creates aggregated stripe invoice line items for project charges",
Long: "Creates aggregated stripe invoice line items for not consumed project records.",
Args: cobra.ExactArgs(1),
RunE: cmdCreateAggregatedCustomerProjectInvoiceItems,
}
createCustomerInvoicesCmd = &cobra.Command{ createCustomerInvoicesCmd = &cobra.Command{
Use: "create-invoices [period]", Use: "create-invoices [period]",
Short: "Creates stripe invoices from pending invoice items", Short: "Creates stripe invoices from pending invoice items",
@ -255,12 +270,33 @@ var (
Long: "Finalizes all draft stripe invoices known to satellite's stripe account.", Long: "Finalizes all draft stripe invoices known to satellite's stripe account.",
RunE: cmdFinalizeCustomerInvoices, RunE: cmdFinalizeCustomerInvoices,
} }
payCustomerInvoicesCmd = &cobra.Command{ payInvoicesWithTokenCmd = &cobra.Command{
Use: "pay-customer-invoices",
Short: "pay open finalized invoices for customer",
Long: "attempts payment on any open finalized invoices for a specific user.",
Args: cobra.ExactArgs(1),
RunE: cmdPayCustomerInvoices,
}
payAllInvoicesCmd = &cobra.Command{
Use: "pay-invoices", Use: "pay-invoices",
Short: "pay finalized invoices", Short: "pay finalized invoices",
Long: "attempts payment on all open finalized invoices according to subscriptions settings.", Long: "attempts payment on all open finalized invoices according to subscriptions settings.",
Args: cobra.ExactArgs(1), Args: cobra.ExactArgs(1),
RunE: cmdPayCustomerInvoices, RunE: cmdPayAllInvoices,
}
failPendingInvoiceTokenPaymentCmd = &cobra.Command{
Use: "fail-token-payment",
Short: "fail pending invoice token payment",
Long: "attempts to transition the token invoice payments that are stuck in a pending state to failed.",
Args: cobra.ExactArgs(1),
RunE: cmdFailPendingInvoiceTokenPayments,
}
completePendingInvoiceTokenPaymentCmd = &cobra.Command{
Use: "complete-token-payment",
Short: "complete pending invoice token payment",
Long: "attempts to transition the token invoice payments that are stuck in a pending state to complete.",
Args: cobra.ExactArgs(1),
RunE: cmdCompletePendingInvoiceTokenPayments,
} }
stripeCustomerCmd = &cobra.Command{ stripeCustomerCmd = &cobra.Command{
Use: "ensure-stripe-customer", Use: "ensure-stripe-customer",
@ -342,6 +378,7 @@ var (
Database string `help:"satellite database connection string" releaseDefault:"postgres://" devDefault:"postgres://"` Database string `help:"satellite database connection string" releaseDefault:"postgres://" devDefault:"postgres://"`
Output string `help:"destination of report output" default:""` Output string `help:"destination of report output" default:""`
Completed bool `help:"whether to output (initiated and completed) or (initiated and not completed)" default:"false"` Completed bool `help:"whether to output (initiated and completed) or (initiated and not completed)" default:"false"`
TimeBased bool `help:"whether the satellite is using time-based graceful exit (and thus, whether to include piece transfer progress in output)" default:"false"`
} }
reportsVerifyGracefulExitReceiptCfg struct { reportsVerifyGracefulExitReceiptCfg struct {
} }
@ -366,6 +403,7 @@ func init() {
rootCmd.AddCommand(runCmd) rootCmd.AddCommand(runCmd)
runCmd.AddCommand(runMigrationCmd) runCmd.AddCommand(runMigrationCmd)
runCmd.AddCommand(runAPICmd) runCmd.AddCommand(runAPICmd)
runCmd.AddCommand(runUICmd)
runCmd.AddCommand(runAdminCmd) runCmd.AddCommand(runAdminCmd)
runCmd.AddCommand(runRepairerCmd) runCmd.AddCommand(runRepairerCmd)
runCmd.AddCommand(runAuditorCmd) runCmd.AddCommand(runAuditorCmd)
@ -394,16 +432,23 @@ func init() {
billingCmd.AddCommand(setInvoiceStatusCmd) billingCmd.AddCommand(setInvoiceStatusCmd)
billingCmd.AddCommand(createCustomerBalanceInvoiceItemsCmd) billingCmd.AddCommand(createCustomerBalanceInvoiceItemsCmd)
billingCmd.AddCommand(prepareCustomerInvoiceRecordsCmd) billingCmd.AddCommand(prepareCustomerInvoiceRecordsCmd)
prepareCustomerInvoiceRecordsCmd.Flags().BoolVar(&aggregate, "aggregate", false, "Used to enable creation of to be aggregated project records in case users have many projects (more than 83).")
billingCmd.AddCommand(createCustomerProjectInvoiceItemsCmd) billingCmd.AddCommand(createCustomerProjectInvoiceItemsCmd)
billingCmd.AddCommand(createCustomerAggregatedProjectInvoiceItemsCmd)
billingCmd.AddCommand(createCustomerInvoicesCmd) billingCmd.AddCommand(createCustomerInvoicesCmd)
billingCmd.AddCommand(generateCustomerInvoicesCmd) billingCmd.AddCommand(generateCustomerInvoicesCmd)
generateCustomerInvoicesCmd.Flags().BoolVar(&aggregate, "aggregate", false, "Used to enable invoice items aggregation in case users have many projects (more than 83).")
billingCmd.AddCommand(finalizeCustomerInvoicesCmd) billingCmd.AddCommand(finalizeCustomerInvoicesCmd)
billingCmd.AddCommand(payCustomerInvoicesCmd) billingCmd.AddCommand(payInvoicesWithTokenCmd)
billingCmd.AddCommand(payAllInvoicesCmd)
billingCmd.AddCommand(failPendingInvoiceTokenPaymentCmd)
billingCmd.AddCommand(completePendingInvoiceTokenPaymentCmd)
billingCmd.AddCommand(stripeCustomerCmd) billingCmd.AddCommand(stripeCustomerCmd)
consistencyCmd.AddCommand(consistencyGECleanupCmd) consistencyCmd.AddCommand(consistencyGECleanupCmd)
process.Bind(runCmd, &runCfg, defaults, cfgstruct.ConfDir(confDir), cfgstruct.IdentityDir(identityDir)) process.Bind(runCmd, &runCfg, defaults, cfgstruct.ConfDir(confDir), cfgstruct.IdentityDir(identityDir))
process.Bind(runMigrationCmd, &runCfg, defaults, cfgstruct.ConfDir(confDir), cfgstruct.IdentityDir(identityDir)) process.Bind(runMigrationCmd, &runCfg, defaults, cfgstruct.ConfDir(confDir), cfgstruct.IdentityDir(identityDir))
process.Bind(runAPICmd, &runCfg, defaults, cfgstruct.ConfDir(confDir), cfgstruct.IdentityDir(identityDir)) process.Bind(runAPICmd, &runCfg, defaults, cfgstruct.ConfDir(confDir), cfgstruct.IdentityDir(identityDir))
process.Bind(runUICmd, &runCfg, defaults, cfgstruct.ConfDir(confDir), cfgstruct.IdentityDir(identityDir))
process.Bind(runAdminCmd, &runCfg, defaults, cfgstruct.ConfDir(confDir), cfgstruct.IdentityDir(identityDir)) process.Bind(runAdminCmd, &runCfg, defaults, cfgstruct.ConfDir(confDir), cfgstruct.IdentityDir(identityDir))
process.Bind(runRepairerCmd, &runCfg, defaults, cfgstruct.ConfDir(confDir), cfgstruct.IdentityDir(identityDir)) process.Bind(runRepairerCmd, &runCfg, defaults, cfgstruct.ConfDir(confDir), cfgstruct.IdentityDir(identityDir))
process.Bind(runAuditorCmd, &runCfg, defaults, cfgstruct.ConfDir(confDir), cfgstruct.IdentityDir(identityDir)) process.Bind(runAuditorCmd, &runCfg, defaults, cfgstruct.ConfDir(confDir), cfgstruct.IdentityDir(identityDir))
@ -429,10 +474,14 @@ func init() {
process.Bind(createCustomerBalanceInvoiceItemsCmd, &runCfg, defaults, cfgstruct.ConfDir(confDir), cfgstruct.IdentityDir(identityDir)) process.Bind(createCustomerBalanceInvoiceItemsCmd, &runCfg, defaults, cfgstruct.ConfDir(confDir), cfgstruct.IdentityDir(identityDir))
process.Bind(prepareCustomerInvoiceRecordsCmd, &runCfg, defaults, cfgstruct.ConfDir(confDir), cfgstruct.IdentityDir(identityDir)) process.Bind(prepareCustomerInvoiceRecordsCmd, &runCfg, defaults, cfgstruct.ConfDir(confDir), cfgstruct.IdentityDir(identityDir))
process.Bind(createCustomerProjectInvoiceItemsCmd, &runCfg, defaults, cfgstruct.ConfDir(confDir), cfgstruct.IdentityDir(identityDir)) process.Bind(createCustomerProjectInvoiceItemsCmd, &runCfg, defaults, cfgstruct.ConfDir(confDir), cfgstruct.IdentityDir(identityDir))
process.Bind(createCustomerAggregatedProjectInvoiceItemsCmd, &runCfg, defaults, cfgstruct.ConfDir(confDir), cfgstruct.IdentityDir(identityDir))
process.Bind(createCustomerInvoicesCmd, &runCfg, defaults, cfgstruct.ConfDir(confDir), cfgstruct.IdentityDir(identityDir)) process.Bind(createCustomerInvoicesCmd, &runCfg, defaults, cfgstruct.ConfDir(confDir), cfgstruct.IdentityDir(identityDir))
process.Bind(generateCustomerInvoicesCmd, &runCfg, defaults, cfgstruct.ConfDir(confDir), cfgstruct.IdentityDir(identityDir)) process.Bind(generateCustomerInvoicesCmd, &runCfg, defaults, cfgstruct.ConfDir(confDir), cfgstruct.IdentityDir(identityDir))
process.Bind(finalizeCustomerInvoicesCmd, &runCfg, defaults, cfgstruct.ConfDir(confDir), cfgstruct.IdentityDir(identityDir)) process.Bind(finalizeCustomerInvoicesCmd, &runCfg, defaults, cfgstruct.ConfDir(confDir), cfgstruct.IdentityDir(identityDir))
process.Bind(payCustomerInvoicesCmd, &runCfg, defaults, cfgstruct.ConfDir(confDir), cfgstruct.IdentityDir(identityDir)) process.Bind(payInvoicesWithTokenCmd, &runCfg, defaults, cfgstruct.ConfDir(confDir), cfgstruct.IdentityDir(identityDir))
process.Bind(payAllInvoicesCmd, &runCfg, defaults, cfgstruct.ConfDir(confDir), cfgstruct.IdentityDir(identityDir))
process.Bind(failPendingInvoiceTokenPaymentCmd, &runCfg, defaults, cfgstruct.ConfDir(confDir), cfgstruct.IdentityDir(identityDir))
process.Bind(completePendingInvoiceTokenPaymentCmd, &runCfg, defaults, cfgstruct.ConfDir(confDir), cfgstruct.IdentityDir(identityDir))
process.Bind(stripeCustomerCmd, &runCfg, defaults, cfgstruct.ConfDir(confDir), cfgstruct.IdentityDir(identityDir)) process.Bind(stripeCustomerCmd, &runCfg, defaults, cfgstruct.ConfDir(confDir), cfgstruct.IdentityDir(identityDir))
process.Bind(consistencyGECleanupCmd, &consistencyGECleanupCfg, defaults, cfgstruct.ConfDir(confDir), cfgstruct.IdentityDir(identityDir)) process.Bind(consistencyGECleanupCmd, &consistencyGECleanupCfg, defaults, cfgstruct.ConfDir(confDir), cfgstruct.IdentityDir(identityDir))
process.Bind(fixLastNetsCmd, &runCfg, defaults, cfgstruct.ConfDir(confDir), cfgstruct.IdentityDir(identityDir)) process.Bind(fixLastNetsCmd, &runCfg, defaults, cfgstruct.ConfDir(confDir), cfgstruct.IdentityDir(identityDir))
@ -448,8 +497,6 @@ func cmdRun(cmd *cobra.Command, args []string) (err error) {
ctx, _ := process.Ctx(cmd) ctx, _ := process.Ctx(cmd)
log := zap.L() log := zap.L()
runCfg.Debug.Address = *process.DebugAddrFlag
identity, err := runCfg.Identity.Load() identity, err := runCfg.Identity.Load()
if err != nil { if err != nil {
log.Error("Failed to load identity.", zap.Error(err)) log.Error("Failed to load identity.", zap.Error(err))
@ -644,7 +691,7 @@ func cmdReportsGracefulExit(cmd *cobra.Command, args []string) (err error) {
// send output to stdout // send output to stdout
if reportsGracefulExitCfg.Output == "" { if reportsGracefulExitCfg.Output == "" {
return generateGracefulExitCSV(ctx, reportsGracefulExitCfg.Completed, start, end, os.Stdout) return generateGracefulExitCSV(ctx, reportsGracefulExitCfg.TimeBased, reportsGracefulExitCfg.Completed, start, end, os.Stdout)
} }
// send output to file // send output to file
@ -657,7 +704,7 @@ func cmdReportsGracefulExit(cmd *cobra.Command, args []string) (err error) {
err = errs.Combine(err, file.Close()) err = errs.Combine(err, file.Close())
}() }()
return generateGracefulExitCSV(ctx, reportsGracefulExitCfg.Completed, start, end, file) return generateGracefulExitCSV(ctx, reportsGracefulExitCfg.TimeBased, reportsGracefulExitCfg.Completed, start, end, file)
} }
func cmdNodeUsage(cmd *cobra.Command, args []string) (err error) { func cmdNodeUsage(cmd *cobra.Command, args []string) (err error) {
@ -808,7 +855,7 @@ func cmdPrepareCustomerInvoiceRecords(cmd *cobra.Command, args []string) (err er
} }
return runBillingCmd(ctx, func(ctx context.Context, payments *stripe.Service, _ satellite.DB) error { return runBillingCmd(ctx, func(ctx context.Context, payments *stripe.Service, _ satellite.DB) error {
return payments.PrepareInvoiceProjectRecords(ctx, periodStart) return payments.PrepareInvoiceProjectRecords(ctx, periodStart, aggregate)
}) })
} }
@ -825,6 +872,19 @@ func cmdCreateCustomerProjectInvoiceItems(cmd *cobra.Command, args []string) (er
}) })
} }
func cmdCreateAggregatedCustomerProjectInvoiceItems(cmd *cobra.Command, args []string) (err error) {
ctx, _ := process.Ctx(cmd)
periodStart, err := parseYearMonth(args[0])
if err != nil {
return err
}
return runBillingCmd(ctx, func(ctx context.Context, payments *stripe.Service, _ satellite.DB) error {
return payments.InvoiceApplyToBeAggregatedProjectRecords(ctx, periodStart)
})
}
func cmdCreateCustomerInvoices(cmd *cobra.Command, args []string) (err error) { func cmdCreateCustomerInvoices(cmd *cobra.Command, args []string) (err error) {
ctx, _ := process.Ctx(cmd) ctx, _ := process.Ctx(cmd)
@ -847,7 +907,7 @@ func cmdGenerateCustomerInvoices(cmd *cobra.Command, args []string) (err error)
} }
return runBillingCmd(ctx, func(ctx context.Context, payments *stripe.Service, _ satellite.DB) error { return runBillingCmd(ctx, func(ctx context.Context, payments *stripe.Service, _ satellite.DB) error {
return payments.GenerateInvoices(ctx, periodStart) return payments.GenerateInvoices(ctx, periodStart, aggregate)
}) })
} }
@ -862,6 +922,18 @@ func cmdFinalizeCustomerInvoices(cmd *cobra.Command, args []string) (err error)
func cmdPayCustomerInvoices(cmd *cobra.Command, args []string) (err error) { func cmdPayCustomerInvoices(cmd *cobra.Command, args []string) (err error) {
ctx, _ := process.Ctx(cmd) ctx, _ := process.Ctx(cmd)
return runBillingCmd(ctx, func(ctx context.Context, payments *stripe.Service, _ satellite.DB) error {
err := payments.InvoiceApplyCustomerTokenBalance(ctx, args[0])
if err != nil {
return errs.New("error applying native token payments to invoice for customer: %v", err)
}
return payments.PayCustomerInvoices(ctx, args[0])
})
}
func cmdPayAllInvoices(cmd *cobra.Command, args []string) (err error) {
ctx, _ := process.Ctx(cmd)
periodStart, err := parseYearMonth(args[0]) periodStart, err := parseYearMonth(args[0])
if err != nil { if err != nil {
return err return err
@ -876,6 +948,20 @@ func cmdPayCustomerInvoices(cmd *cobra.Command, args []string) (err error) {
}) })
} }
func cmdFailPendingInvoiceTokenPayments(cmd *cobra.Command, args []string) (err error) {
ctx, _ := process.Ctx(cmd)
return runBillingCmd(ctx, func(ctx context.Context, payments *stripe.Service, _ satellite.DB) error {
return payments.FailPendingInvoiceTokenPayments(ctx, strings.Split(args[0], ","))
})
}
func cmdCompletePendingInvoiceTokenPayments(cmd *cobra.Command, args []string) (err error) {
ctx, _ := process.Ctx(cmd)
return runBillingCmd(ctx, func(ctx context.Context, payments *stripe.Service, _ satellite.DB) error {
return payments.CompletePendingInvoiceTokenPayments(ctx, strings.Split(args[0], ","))
})
}
func cmdStripeCustomer(cmd *cobra.Command, args []string) (err error) { func cmdStripeCustomer(cmd *cobra.Command, args []string) (err error) {
ctx, _ := process.Ctx(cmd) ctx, _ := process.Ctx(cmd)
@ -885,6 +971,9 @@ func cmdStripeCustomer(cmd *cobra.Command, args []string) (err error) {
func cmdConsistencyGECleanup(cmd *cobra.Command, args []string) error { func cmdConsistencyGECleanup(cmd *cobra.Command, args []string) error {
ctx, _ := process.Ctx(cmd) ctx, _ := process.Ctx(cmd)
if runCfg.GracefulExit.TimeBased {
return errs.New("this command is not supported with time-based graceful exit")
}
before, err := time.Parse("2006-01-02", consistencyGECleanupCfg.Before) before, err := time.Parse("2006-01-02", consistencyGECleanupCfg.Before)
if err != nil { if err != nil {
return errs.New("before flag value isn't of the expected format. %+v", err) return errs.New("before flag value isn't of the expected format. %+v", err)
@ -932,7 +1021,7 @@ func cmdRestoreTrash(cmd *cobra.Command, args []string) error {
successes := new(int64) successes := new(int64)
failures := new(int64) failures := new(int64)
undelete := func(node *overlay.SelectedNode) { undelete := func(node *nodeselection.SelectedNode) {
log.Info("starting restore trash", zap.String("Node ID", node.ID.String())) log.Info("starting restore trash", zap.String("Node ID", node.ID.String()))
ctx, cancel := context.WithTimeout(ctx, 10*time.Second) ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
@ -966,9 +1055,9 @@ func cmdRestoreTrash(cmd *cobra.Command, args []string) error {
log.Info("successful restore trash", zap.String("Node ID", node.ID.String())) log.Info("successful restore trash", zap.String("Node ID", node.ID.String()))
} }
var nodes []*overlay.SelectedNode var nodes []*nodeselection.SelectedNode
if len(args) == 0 { if len(args) == 0 {
err = db.OverlayCache().IterateAllContactedNodes(ctx, func(ctx context.Context, node *overlay.SelectedNode) error { err = db.OverlayCache().IterateAllContactedNodes(ctx, func(ctx context.Context, node *nodeselection.SelectedNode) error {
nodes = append(nodes, node) nodes = append(nodes, node)
return nil return nil
}) })
@ -985,7 +1074,7 @@ func cmdRestoreTrash(cmd *cobra.Command, args []string) error {
if err != nil { if err != nil {
return err return err
} }
nodes = append(nodes, &overlay.SelectedNode{ nodes = append(nodes, &nodeselection.SelectedNode{
ID: dossier.Id, ID: dossier.Id,
Address: dossier.Address, Address: dossier.Address,
LastNet: dossier.LastNet, LastNet: dossier.LastNet,

View File

@ -18,8 +18,6 @@ func cmdRangedLoopRun(cmd *cobra.Command, args []string) (err error) {
ctx, _ := process.Ctx(cmd) ctx, _ := process.Ctx(cmd)
log := zap.L() log := zap.L()
runCfg.Debug.Address = *process.DebugAddrFlag
db, err := satellitedb.Open(ctx, log.Named("db"), runCfg.Database, satellitedb.Options{ApplicationName: "satellite-rangedloop"}) db, err := satellitedb.Open(ctx, log.Named("db"), runCfg.Database, satellitedb.Options{ApplicationName: "satellite-rangedloop"})
if err != nil { if err != nil {
return errs.New("Error starting master database on satellite rangedloop: %+v", err) return errs.New("Error starting master database on satellite rangedloop: %+v", err)

View File

@ -16,7 +16,6 @@ import (
"time" "time"
"github.com/spf13/cobra" "github.com/spf13/cobra"
"github.com/vivint/infectious"
"github.com/zeebo/errs" "github.com/zeebo/errs"
"go.uber.org/zap" "go.uber.org/zap"
"golang.org/x/sync/errgroup" "golang.org/x/sync/errgroup"
@ -94,7 +93,12 @@ func cmdRepairSegment(cmd *cobra.Command, args []string) (err error) {
dialer := rpc.NewDefaultDialer(tlsOptions) dialer := rpc.NewDefaultDialer(tlsOptions)
overlay, err := overlay.NewService(log.Named("overlay"), db.OverlayCache(), db.NodeEvents(), config.Console.ExternalAddress, config.Console.SatelliteName, config.Overlay) placement, err := config.Placement.Parse()
if err != nil {
return err
}
overlayService, err := overlay.NewService(log.Named("overlay"), db.OverlayCache(), db.NodeEvents(), placement.CreateFilters, config.Console.ExternalAddress, config.Console.SatelliteName, config.Overlay)
if err != nil { if err != nil {
return err return err
} }
@ -102,8 +106,9 @@ func cmdRepairSegment(cmd *cobra.Command, args []string) (err error) {
orders, err := orders.NewService( orders, err := orders.NewService(
log.Named("orders"), log.Named("orders"),
signing.SignerFromFullIdentity(identity), signing.SignerFromFullIdentity(identity),
overlay, overlayService,
orders.NewNoopDB(), orders.NewNoopDB(),
placement.CreateFilters,
config.Orders, config.Orders,
) )
if err != nil { if err != nil {
@ -122,9 +127,10 @@ func cmdRepairSegment(cmd *cobra.Command, args []string) (err error) {
log.Named("segment-repair"), log.Named("segment-repair"),
metabaseDB, metabaseDB,
orders, orders,
overlay, overlayService,
nil, // TODO add noop version nil, // TODO add noop version
ecRepairer, ecRepairer,
placement.CreateFilters,
config.Checker.RepairOverrides, config.Checker.RepairOverrides,
config.Repairer, config.Repairer,
) )
@ -132,7 +138,7 @@ func cmdRepairSegment(cmd *cobra.Command, args []string) (err error) {
// TODO reorganize to avoid using peer. // TODO reorganize to avoid using peer.
peer := &satellite.Repairer{} peer := &satellite.Repairer{}
peer.Overlay = overlay peer.Overlay = overlayService
peer.Orders.Service = orders peer.Orders.Service = orders
peer.EcRepairer = ecRepairer peer.EcRepairer = ecRepairer
peer.SegmentRepairer = segmentRepairer peer.SegmentRepairer = segmentRepairer
@ -274,10 +280,8 @@ func reuploadSegment(ctx context.Context, log *zap.Logger, peer *satellite.Repai
return errs.New("not enough new nodes were found for repair: min %v got %v", redundancy.RepairThreshold(), len(newNodes)) return errs.New("not enough new nodes were found for repair: min %v got %v", redundancy.RepairThreshold(), len(newNodes))
} }
optimalThresholdMultiplier := float64(1) // is this value fine?
numHealthyInExcludedCountries := 0
putLimits, putPrivateKey, err := peer.Orders.Service.CreatePutRepairOrderLimits(ctx, segment, make([]*pb.AddressedOrderLimit, len(newNodes)), putLimits, putPrivateKey, err := peer.Orders.Service.CreatePutRepairOrderLimits(ctx, segment, make([]*pb.AddressedOrderLimit, len(newNodes)),
make(map[int32]struct{}), newNodes, optimalThresholdMultiplier, numHealthyInExcludedCountries) make(map[uint16]struct{}), newNodes)
if err != nil { if err != nil {
return errs.New("could not create PUT_REPAIR order limits: %w", err) return errs.New("could not create PUT_REPAIR order limits: %w", err)
} }
@ -376,7 +380,7 @@ func downloadSegment(ctx context.Context, log *zap.Logger, peer *satellite.Repai
len(pieceReaders), redundancy.RequiredCount()) len(pieceReaders), redundancy.RequiredCount())
} }
fec, err := infectious.NewFEC(redundancy.RequiredCount(), redundancy.TotalCount()) fec, err := eestream.NewFEC(redundancy.RequiredCount(), redundancy.TotalCount())
if err != nil { if err != nil {
return nil, failedDownloads, err return nil, failedDownloads, err
} }

View File

@ -21,8 +21,6 @@ func cmdRepairerRun(cmd *cobra.Command, args []string) (err error) {
ctx, _ := process.Ctx(cmd) ctx, _ := process.Ctx(cmd)
log := zap.L() log := zap.L()
runCfg.Debug.Address = *process.DebugAddrFlag
identity, err := runCfg.Identity.Load() identity, err := runCfg.Identity.Load()
if err != nil { if err != nil {
log.Error("Failed to load identity.", zap.Error(err)) log.Error("Failed to load identity.", zap.Error(err))

45
cmd/satellite/ui.go Normal file
View File

@ -0,0 +1,45 @@
// Copyright (C) 2023 Storj Labs, Inc.
// See LICENSE for copying information.
package main
import (
"github.com/spf13/cobra"
"github.com/zeebo/errs"
"go.uber.org/zap"
"storj.io/private/process"
"storj.io/storj/satellite"
)
func cmdUIRun(cmd *cobra.Command, args []string) (err error) {
ctx, _ := process.Ctx(cmd)
log := zap.L()
identity, err := runCfg.Identity.Load()
if err != nil {
log.Error("Failed to load identity.", zap.Error(err))
return errs.New("Failed to load identity: %+v", err)
}
satAddr := runCfg.Config.Contact.ExternalAddress
if satAddr == "" {
return errs.New("cannot run satellite ui if contact.external-address is not set")
}
apiAddress := runCfg.Config.Console.ExternalAddress
if apiAddress == "" {
apiAddress = runCfg.Config.Console.Address
}
peer, err := satellite.NewUI(log, identity, &runCfg.Config, process.AtomicLevel(cmd), satAddr, apiAddress)
if err != nil {
return err
}
if err := process.InitMetricsWithHostname(ctx, log, nil); err != nil {
log.Warn("Failed to initialize telemetry batcher on satellite api", zap.Error(err))
}
runError := peer.Run(ctx)
closeError := peer.Close()
return errs.Combine(runError, closeError)
}

View File

@ -0,0 +1,243 @@
// Copyright (C) 2023 Storj Labs, Inc.
// See LICENSE for copying information.
package main
import (
"context"
"github.com/spf13/cobra"
"github.com/zeebo/errs"
"go.uber.org/zap"
"storj.io/common/storj"
"storj.io/private/cfgstruct"
"storj.io/private/process"
"storj.io/storj/storagenode"
"storj.io/storj/storagenode/pieces"
"storj.io/storj/storagenode/satellites"
"storj.io/storj/storagenode/storagenodedb"
"storj.io/storj/storagenode/trust"
)
// runCfg defines configuration for run command.
type forgetSatelliteCfg struct {
storagenode.Config
SatelliteIDs []string `internal:"true"`
AllUntrusted bool `help:"Clean up all untrusted satellites" default:"false"`
Force bool `help:"Force removal of satellite data if not listed in satelliteDB cache or marked as untrusted" default:"false"`
}
func newForgetSatelliteCmd(f *Factory) *cobra.Command {
var cfg forgetSatelliteCfg
cmd := &cobra.Command{
Use: "forget-satellite [satellite_IDs...]",
Short: "Remove an untrusted satellite from the trust cache and clean up its data",
Long: "Forget a satellite.\n" +
"The command shows the list of the available untrusted satellites " +
"and removes the selected satellites from the trust cache and clean up the available data",
Example: `
# Specify satellite ID to forget
$ storagenode forget-satellite --identity-dir /path/to/identityDir --config-dir /path/to/configDir satellite_ID
# Specify multiple satellite IDs to forget
$ storagenode forget-satellite satellite_ID1 satellite_ID2 --identity-dir /path/to/identityDir --config-dir /path/to/configDir
# Clean up all untrusted satellites
# This checks for untrusted satellites in both the satelliteDB cache and the excluded satellites list
# specified in the config.yaml file
$ storagenode forget-satellite --all-untrusted --identity-dir /path/to/identityDir --config-dir /path/to/configDir
# For force removal of data for untrusted satellites that are not listed in satelliteDB cache or marked as untrusted
$ storagenode forget-satellite satellite_ID1 satellite_ID2 --force --identity-dir /path/to/identityDir --config-dir /path/to/configDir
`,
RunE: func(cmd *cobra.Command, args []string) error {
cfg.SatelliteIDs = args
if len(args) > 0 && cfg.AllUntrusted {
return errs.New("cannot specify both satellite IDs and --all-untrusted")
}
if len(args) == 0 && !cfg.AllUntrusted {
return errs.New("must specify either satellite ID(s) as arguments or --all-untrusted flag")
}
if cfg.AllUntrusted && cfg.Force {
return errs.New("cannot specify both --all-untrusted and --force")
}
ctx, _ := process.Ctx(cmd)
return cmdForgetSatellite(ctx, zap.L(), &cfg)
},
Annotations: map[string]string{"type": "helper"},
}
process.Bind(cmd, &cfg, f.Defaults, cfgstruct.ConfDir(f.ConfDir), cfgstruct.IdentityDir(f.IdentityDir))
return cmd
}
func cmdForgetSatellite(ctx context.Context, log *zap.Logger, cfg *forgetSatelliteCfg) (err error) {
// we don't really need the identity, but we load it as a sanity check
ident, err := cfg.Identity.Load()
if err != nil {
log.Fatal("Failed to load identity.", zap.Error(err))
} else {
log.Info("Identity loaded.", zap.Stringer("Node ID", ident.ID))
}
db, err := storagenodedb.OpenExisting(ctx, log.Named("db"), cfg.DatabaseConfig())
if err != nil {
return errs.New("Error starting master database on storagenode: %+v", err)
}
defer func() { err = errs.Combine(err, db.Close()) }()
satelliteDB := db.Satellites()
// get list of excluded satellites
excludedSatellites := make(map[storj.NodeID]bool)
for _, rule := range cfg.Storage2.Trust.Exclusions.Rules {
url, err := trust.ParseSatelliteURL(rule.String())
if err != nil {
log.Warn("Failed to parse satellite URL from exclusions list", zap.Error(err), zap.String("rule", rule.String()))
continue
}
excludedSatellites[url.ID] = false // false means the satellite has not been cleaned up yet.
}
if len(cfg.SatelliteIDs) > 0 {
for _, satelliteIDStr := range cfg.SatelliteIDs {
satelliteID, err := storj.NodeIDFromString(satelliteIDStr)
if err != nil {
return err
}
satellite := satellites.Satellite{
SatelliteID: satelliteID,
Status: satellites.Untrusted,
}
// check if satellite is excluded
cleanedUp, isExcluded := excludedSatellites[satelliteID]
if !isExcluded {
sat, err := satelliteDB.GetSatellite(ctx, satelliteID)
if err != nil {
return err
}
if !satellite.SatelliteID.IsZero() {
satellite = sat
}
if satellite.SatelliteID.IsZero() && !cfg.Force {
return errs.New("satellite %v not found. Specify --force to force data deletion", satelliteID)
}
log.Warn("Satellite not found in satelliteDB cache. Forcing removal of satellite data.", zap.Stringer("satelliteID", satelliteID))
}
if cleanedUp {
log.Warn("Satellite already cleaned up", zap.Stringer("satelliteID", satelliteID))
continue
}
err = cleanupSatellite(ctx, log, cfg, db, satellite)
if err != nil {
return err
}
}
} else {
sats, err := satelliteDB.GetSatellites(ctx)
if err != nil {
return err
}
hasUntrusted := false
for _, satellite := range sats {
if satellite.Status != satellites.Untrusted {
continue
}
hasUntrusted = true
err = cleanupSatellite(ctx, log, cfg, db, satellite)
if err != nil {
return err
}
excludedSatellites[satellite.SatelliteID] = true // true means the satellite has been cleaned up.
}
// clean up excluded satellites that might not be in the satelliteDB cache.
for satelliteID, cleanedUp := range excludedSatellites {
if !cleanedUp {
satellite := satellites.Satellite{
SatelliteID: satelliteID,
Status: satellites.Untrusted,
}
hasUntrusted = true
err = cleanupSatellite(ctx, log, cfg, db, satellite)
if err != nil {
return err
}
}
}
if !hasUntrusted {
log.Info("No untrusted satellites found. You can add satellites to the exclusions list in the config.yaml file.")
}
}
return nil
}
func cleanupSatellite(ctx context.Context, log *zap.Logger, cfg *forgetSatelliteCfg, db *storagenodedb.DB, satellite satellites.Satellite) error {
if satellite.Status != satellites.Untrusted && !cfg.Force {
log.Error("Satellite is not untrusted. Skipping", zap.Stringer("satelliteID", satellite.SatelliteID))
return nil
}
log.Info("Removing satellite from trust cache.", zap.Stringer("satelliteID", satellite.SatelliteID))
cache, err := trust.LoadCache(cfg.Storage2.Trust.CachePath)
if err != nil {
return err
}
deleted := cache.DeleteSatelliteEntry(satellite.SatelliteID)
if deleted {
if err := cache.Save(ctx); err != nil {
return err
}
log.Info("Satellite removed from trust cache.", zap.Stringer("satelliteID", satellite.SatelliteID))
}
log.Info("Cleaning up satellite data.", zap.Stringer("satelliteID", satellite.SatelliteID))
blobs := pieces.NewBlobsUsageCache(log.Named("blobscache"), db.Pieces())
if err := blobs.DeleteNamespace(ctx, satellite.SatelliteID.Bytes()); err != nil {
return err
}
log.Info("Cleaning up the trash.", zap.Stringer("satelliteID", satellite.SatelliteID))
err = blobs.DeleteTrashNamespace(ctx, satellite.SatelliteID.Bytes())
if err != nil {
return err
}
log.Info("Removing satellite info from reputation DB.", zap.Stringer("satelliteID", satellite.SatelliteID))
err = db.Reputation().Delete(ctx, satellite.SatelliteID)
if err != nil {
return err
}
// delete v0 pieces for the satellite, if any.
log.Info("Removing satellite v0 pieces if any.", zap.Stringer("satelliteID", satellite.SatelliteID))
err = db.V0PieceInfo().WalkSatelliteV0Pieces(ctx, db.Pieces(), satellite.SatelliteID, func(access pieces.StoredPieceAccess) error {
return db.Pieces().Delete(ctx, access.BlobRef())
})
if err != nil {
return err
}
log.Info("Removing satellite from satellites DB.", zap.Stringer("satelliteID", satellite.SatelliteID))
err = db.Satellites().DeleteSatellite(ctx, satellite.SatelliteID)
if err != nil {
return err
}
return nil
}

View File

@ -0,0 +1,254 @@
// Copyright (C) 2023 Storj Labs, Inc.
// See LICENSE for copying information.
package main
import (
"os"
"strings"
"testing"
"time"
"github.com/stretchr/testify/require"
"github.com/zeebo/errs"
"go.uber.org/zap/zaptest"
"storj.io/common/identity"
"storj.io/common/memory"
"storj.io/common/testcontext"
"storj.io/common/testrand"
"storj.io/storj/private/testplanet"
"storj.io/storj/storagenode/blobstore"
"storj.io/storj/storagenode/blobstore/filestore"
"storj.io/storj/storagenode/reputation"
"storj.io/storj/storagenode/satellites"
)
func Test_newForgetSatelliteCmd_Error(t *testing.T) {
tests := []struct {
name string
args string
wantErr string
}{
{
name: "no args",
args: "",
wantErr: "must specify either satellite ID(s) as arguments or --all-untrusted flag",
},
{
name: "Both satellite ID and --all-untrusted flag specified",
args: "--all-untrusted 1234567890123456789012345678901234567890123456789012345678901234",
wantErr: "cannot specify both satellite IDs and --all-untrusted",
},
{
name: "--all-untrusted and --force specified",
args: "--all-untrusted --force",
wantErr: "cannot specify both --all-untrusted and --force",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
cmd := newForgetSatelliteCmd(&Factory{})
cmd.SetArgs(strings.Fields(tt.args))
err := cmd.ExecuteContext(testcontext.New(t))
if tt.wantErr == "" {
require.NoError(t, err)
return
}
require.Equal(t, tt.wantErr, err.Error())
})
}
}
func Test_cmdForgetSatellite(t *testing.T) {
t.Skip("The tests and the behavior is currently flaky. See https://github.com/storj/storj/issues/6465")
testplanet.Run(t, testplanet.Config{
SatelliteCount: 2, StorageNodeCount: 1, UplinkCount: 0,
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
address := planet.StorageNodes[0].Server.PrivateAddr().String()
db := planet.StorageNodes[0].DB
log := zaptest.NewLogger(t)
store, err := filestore.NewAt(log, db.Config().Pieces, filestore.DefaultConfig)
require.NoError(t, err)
defer ctx.Check(store.Close)
satelliteID := planet.Satellites[0].ID()
blobSize := memory.KB
blobRef := blobstore.BlobRef{
Namespace: satelliteID.Bytes(),
Key: testrand.PieceID().Bytes(),
}
w, err := store.Create(ctx, blobRef, -1)
require.NoError(t, err)
_, err = w.Write(testrand.Bytes(blobSize))
require.NoError(t, err)
require.NoError(t, w.Commit(ctx))
// create a new satellite reputation
timestamp := time.Now().UTC()
reputationDB := db.Reputation()
stats := reputation.Stats{
SatelliteID: satelliteID,
Audit: reputation.Metric{
TotalCount: 6,
SuccessCount: 7,
Alpha: 8,
Beta: 9,
Score: 10,
UnknownAlpha: 11,
UnknownBeta: 12,
UnknownScore: 13,
},
OnlineScore: 14,
UpdatedAt: timestamp,
JoinedAt: timestamp,
}
err = reputationDB.Store(ctx, stats)
require.NoError(t, err)
// test that the reputation was stored correctly
rstats, err := reputationDB.Get(ctx, satelliteID)
require.NoError(t, err)
require.NotNil(t, rstats)
require.Equal(t, stats, *rstats)
// insert a new untrusted satellite in the database
err = db.Satellites().SetAddressAndStatus(ctx, satelliteID, address, satellites.Untrusted)
require.NoError(t, err)
// test that the satellite was inserted correctly
satellite, err := db.Satellites().GetSatellite(ctx, satelliteID)
require.NoError(t, err)
require.Equal(t, satellites.Untrusted, satellite.Status)
// set up the identity
ident := planet.StorageNodes[0].Identity
identConfig := identity.Config{
CertPath: ctx.File("identity", "identity.cert"),
KeyPath: ctx.File("identity", "identity.Key"),
}
err = identConfig.Save(ident)
require.NoError(t, err)
planet.StorageNodes[0].Config.Identity = identConfig
// run the forget satellite command with All flag
err = cmdForgetSatellite(ctx, log, &forgetSatelliteCfg{
AllUntrusted: true,
Config: planet.StorageNodes[0].Config,
})
require.NoError(t, err)
// !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
// TODO: this is for reproducing the bug,
// remove it once it's fixed.
// !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
time.Sleep(10 * time.Second)
// check that the blob was deleted
blobInfo, err := store.Stat(ctx, blobRef)
require.Error(t, err)
require.True(t, errs.Is(err, os.ErrNotExist))
require.Nil(t, blobInfo)
// check that the reputation was deleted
rstats, err = reputationDB.Get(ctx, satelliteID)
require.NoError(t, err)
require.Equal(t, &reputation.Stats{SatelliteID: satelliteID}, rstats)
// check that the satellite info was deleted from the database
satellite, err = db.Satellites().GetSatellite(ctx, satelliteID)
require.NoError(t, err)
require.True(t, satellite.SatelliteID.IsZero())
})
}
func Test_cmdForgetSatellite_Exclusions(t *testing.T) {
t.Skip("The tests and the behavior is currently flaky. See https://github.com/storj/storj/issues/6465")
testplanet.Run(t, testplanet.Config{
SatelliteCount: 2, StorageNodeCount: 1, UplinkCount: 0,
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
address := planet.StorageNodes[0].Server.PrivateAddr().String()
db := planet.StorageNodes[0].DB
log := zaptest.NewLogger(t)
store, err := filestore.NewAt(log, db.Config().Pieces, filestore.DefaultConfig)
require.NoError(t, err)
defer ctx.Check(store.Close)
satelliteID := planet.Satellites[0].ID()
blobSize := memory.KB
blobRef := blobstore.BlobRef{
Namespace: satelliteID.Bytes(),
Key: testrand.PieceID().Bytes(),
}
w, err := store.Create(ctx, blobRef, -1)
require.NoError(t, err)
_, err = w.Write(testrand.Bytes(blobSize))
require.NoError(t, err)
require.NoError(t, w.Commit(ctx))
// create a new satellite reputation
timestamp := time.Now().UTC()
reputationDB := db.Reputation()
stats := reputation.Stats{
SatelliteID: satelliteID,
Audit: reputation.Metric{
TotalCount: 6,
SuccessCount: 7,
Alpha: 8,
Beta: 9,
Score: 10,
UnknownAlpha: 11,
UnknownBeta: 12,
UnknownScore: 13,
},
OnlineScore: 14,
UpdatedAt: timestamp,
JoinedAt: timestamp,
}
err = reputationDB.Store(ctx, stats)
require.NoError(t, err)
// test that the reputation was stored correctly
rstats, err := reputationDB.Get(ctx, satelliteID)
require.NoError(t, err)
require.NotNil(t, rstats)
require.Equal(t, stats, *rstats)
// set up the identity
ident := planet.StorageNodes[0].Identity
identConfig := identity.Config{
CertPath: ctx.File("identity", "identity.cert"),
KeyPath: ctx.File("identity", "identity.Key"),
}
err = identConfig.Save(ident)
require.NoError(t, err)
planet.StorageNodes[0].Config.Identity = identConfig
// add the satellite to the exclusion list
err = planet.StorageNodes[0].Config.Storage2.Trust.Exclusions.Set(satelliteID.String() + "@" + address)
require.NoError(t, err)
// run the forget satellite command with All flag
err = cmdForgetSatellite(ctx, log, &forgetSatelliteCfg{
AllUntrusted: true,
Config: planet.StorageNodes[0].Config,
})
require.NoError(t, err)
// check that the blob was deleted
blobInfo, err := store.Stat(ctx, blobRef)
require.Error(t, err)
require.True(t, errs.Is(err, os.ErrNotExist))
require.Nil(t, blobInfo)
// check that the reputation was deleted
rstats, err = reputationDB.Get(ctx, satelliteID)
require.NoError(t, err)
require.Equal(t, &reputation.Stats{SatelliteID: satelliteID}, rstats)
// check that the satellite info was deleted from the database
satellite, err := db.Satellites().GetSatellite(ctx, satelliteID)
require.NoError(t, err)
require.True(t, satellite.SatelliteID.IsZero())
})
}

View File

@ -44,8 +44,6 @@ func cmdRun(cmd *cobra.Command, cfg *runCfg) (err error) {
ctx, _ := process.Ctx(cmd) ctx, _ := process.Ctx(cmd)
log := zap.L() log := zap.L()
cfg.Debug.Address = *process.DebugAddrFlag
mapDeprecatedConfigs(log, &cfg.StorageNodeFlags) mapDeprecatedConfigs(log, &cfg.StorageNodeFlags)
identity, err := cfg.Identity.Load() identity, err := cfg.Identity.Load()

View File

@ -59,6 +59,7 @@ func newRootCmd(setDefaults bool) (*cobra.Command, *Factory) {
newIssueAPIKeyCmd(factory), newIssueAPIKeyCmd(factory),
newGracefulExitInitCmd(factory), newGracefulExitInitCmd(factory),
newGracefulExitStatusCmd(factory), newGracefulExitStatusCmd(factory),
newForgetSatelliteCmd(factory),
// internal hidden commands // internal hidden commands
internalcmd.NewUsedSpaceFilewalkerCmd().Command, internalcmd.NewUsedSpaceFilewalkerCmd().Command,
internalcmd.NewGCFilewalkerCmd().Command, internalcmd.NewGCFilewalkerCmd().Command,

View File

@ -65,11 +65,15 @@ func (ce *consoleEndpoints) Token() string {
return ce.appendPath("/api/v0/auth/token") return ce.appendPath("/api/v0/auth/token")
} }
func (ce *consoleEndpoints) GraphQL() string { func (ce *consoleEndpoints) Projects() string {
return ce.appendPath("/api/v0/graphql") return ce.appendPath("/api/v0/projects")
} }
func (ce *consoleEndpoints) graphqlDo(request *http.Request, jsonResponse interface{}) error { func (ce *consoleEndpoints) APIKeys() string {
return ce.appendPath("/api/v0/api-keys")
}
func (ce *consoleEndpoints) httpDo(request *http.Request, jsonResponse interface{}) error {
resp, err := ce.client.Do(request) resp, err := ce.client.Do(request)
if err != nil { if err != nil {
return err return err
@ -81,24 +85,24 @@ func (ce *consoleEndpoints) graphqlDo(request *http.Request, jsonResponse interf
return err return err
} }
var response struct {
Data json.RawMessage
Errors []interface{}
}
if err = json.NewDecoder(bytes.NewReader(b)).Decode(&response); err != nil {
return err
}
if response.Errors != nil {
return errs.New("inner graphql error: %v", response.Errors)
}
if jsonResponse == nil { if jsonResponse == nil {
return errs.New("empty response: %q", b) return errs.New("empty response: %q", b)
} }
return json.NewDecoder(bytes.NewReader(response.Data)).Decode(jsonResponse) if resp.StatusCode >= 200 && resp.StatusCode < 300 {
return json.NewDecoder(bytes.NewReader(b)).Decode(jsonResponse)
}
var errResponse struct {
Error string `json:"error"`
}
err = json.NewDecoder(bytes.NewReader(b)).Decode(&errResponse)
if err != nil {
return err
}
return errs.New("request failed with status %d: %s", resp.StatusCode, errResponse.Error)
} }
func (ce *consoleEndpoints) createOrGetAPIKey(ctx context.Context) (string, error) { func (ce *consoleEndpoints) createOrGetAPIKey(ctx context.Context) (string, error) {
@ -464,49 +468,41 @@ func (ce *consoleEndpoints) getProject(ctx context.Context, token string) (strin
request, err := http.NewRequestWithContext( request, err := http.NewRequestWithContext(
ctx, ctx,
http.MethodGet, http.MethodGet,
ce.GraphQL(), ce.Projects(),
nil) nil)
if err != nil { if err != nil {
return "", errs.Wrap(err) return "", errs.Wrap(err)
} }
q := request.URL.Query()
q.Add("query", `query {myProjects{id}}`)
request.URL.RawQuery = q.Encode()
request.AddCookie(&http.Cookie{ request.AddCookie(&http.Cookie{
Name: ce.cookieName, Name: ce.cookieName,
Value: token, Value: token,
}) })
request.Header.Add("Content-Type", "application/graphql") request.Header.Add("Content-Type", "application/json")
var getProjects struct { var projects []struct {
MyProjects []struct { ID string `json:"id"`
ID string
}
} }
if err := ce.graphqlDo(request, &getProjects); err != nil { if err := ce.httpDo(request, &projects); err != nil {
return "", errs.Wrap(err) return "", errs.Wrap(err)
} }
if len(getProjects.MyProjects) == 0 { if len(projects) == 0 {
return "", errs.New("no projects") return "", errs.New("no projects")
} }
return getProjects.MyProjects[0].ID, nil return projects[0].ID, nil
} }
func (ce *consoleEndpoints) createProject(ctx context.Context, token string) (string, error) { func (ce *consoleEndpoints) createProject(ctx context.Context, token string) (string, error) {
rng := rand.NewSource(time.Now().UnixNano()) rng := rand.NewSource(time.Now().UnixNano())
createProjectQuery := fmt.Sprintf( body := fmt.Sprintf(`{"name":"TestProject-%d","description":""}`, rng.Int63())
`mutation {createProject(input:{name:"TestProject-%d",description:""}){id}}`,
rng.Int63())
request, err := http.NewRequestWithContext( request, err := http.NewRequestWithContext(
ctx, ctx,
http.MethodPost, http.MethodPost,
ce.GraphQL(), ce.Projects(),
bytes.NewReader([]byte(createProjectQuery))) bytes.NewReader([]byte(body)))
if err != nil { if err != nil {
return "", errs.Wrap(err) return "", errs.Wrap(err)
} }
@ -516,31 +512,27 @@ func (ce *consoleEndpoints) createProject(ctx context.Context, token string) (st
Value: token, Value: token,
}) })
request.Header.Add("Content-Type", "application/graphql") request.Header.Add("Content-Type", "application/json")
var createProject struct { var createdProject struct {
CreateProject struct { ID string `json:"id"`
ID string
}
} }
if err := ce.graphqlDo(request, &createProject); err != nil { if err := ce.httpDo(request, &createdProject); err != nil {
return "", errs.Wrap(err) return "", errs.Wrap(err)
} }
return createProject.CreateProject.ID, nil return createdProject.ID, nil
} }
func (ce *consoleEndpoints) createAPIKey(ctx context.Context, token, projectID string) (string, error) { func (ce *consoleEndpoints) createAPIKey(ctx context.Context, token, projectID string) (string, error) {
rng := rand.NewSource(time.Now().UnixNano()) rng := rand.NewSource(time.Now().UnixNano())
createAPIKeyQuery := fmt.Sprintf( apiKeyName := fmt.Sprintf("TestKey-%d", rng.Int63())
`mutation {createAPIKey(projectID:%q,name:"TestKey-%d"){key}}`,
projectID, rng.Int63())
request, err := http.NewRequestWithContext( request, err := http.NewRequestWithContext(
ctx, ctx,
http.MethodPost, http.MethodPost,
ce.GraphQL(), ce.APIKeys()+"/create/"+projectID,
bytes.NewReader([]byte(createAPIKeyQuery))) bytes.NewReader([]byte(apiKeyName)))
if err != nil { if err != nil {
return "", errs.Wrap(err) return "", errs.Wrap(err)
} }
@ -550,18 +542,16 @@ func (ce *consoleEndpoints) createAPIKey(ctx context.Context, token, projectID s
Value: token, Value: token,
}) })
request.Header.Add("Content-Type", "application/graphql") request.Header.Add("Content-Type", "application/json")
var createAPIKey struct { var createdKey struct {
CreateAPIKey struct { Key string `json:"key"`
Key string
}
} }
if err := ce.graphqlDo(request, &createAPIKey); err != nil { if err := ce.httpDo(request, &createdKey); err != nil {
return "", errs.Wrap(err) return "", errs.Wrap(err)
} }
return createAPIKey.CreateAPIKey.Key, nil return createdKey.Key, nil
} }
func generateActivationKey(userID uuid.UUID, email string, createdAt time.Time) (string, error) { func generateActivationKey(userID uuid.UUID, email string, createdAt time.Time) (string, error) {

View File

@ -39,6 +39,8 @@ const (
maxStoragenodeCount = 200 maxStoragenodeCount = 200
folderPermissions = 0744 folderPermissions = 0744
gatewayGracePeriod = 10 * time.Second
) )
var defaultAccess = "12edqtGZnqQo6QHwTB92EDqg9B1WrWn34r7ALu94wkqXL4eXjBNnVr6F5W7GhJjVqJCqxpFERmDR1dhZWyMt3Qq5zwrE9yygXeT6kBoS9AfiPuwB6kNjjxepg5UtPPtp4VLp9mP5eeyobKQRD5TsEsxTGhxamsrHvGGBPrZi8DeLtNYFMRTV6RyJVxpYX6MrPCw9HVoDQbFs7VcPeeRxRMQttSXL3y33BJhkqJ6ByFviEquaX5R2wjQT2Kx" var defaultAccess = "12edqtGZnqQo6QHwTB92EDqg9B1WrWn34r7ALu94wkqXL4eXjBNnVr6F5W7GhJjVqJCqxpFERmDR1dhZWyMt3Qq5zwrE9yygXeT6kBoS9AfiPuwB6kNjjxepg5UtPPtp4VLp9mP5eeyobKQRD5TsEsxTGhxamsrHvGGBPrZi8DeLtNYFMRTV6RyJVxpYX6MrPCw9HVoDQbFs7VcPeeRxRMQttSXL3y33BJhkqJ6ByFviEquaX5R2wjQT2Kx"
@ -536,11 +538,11 @@ func newNetwork(flags *Flags) (*Processes, error) {
return fmt.Errorf("failed to read config string: %w", err) return fmt.Errorf("failed to read config string: %w", err)
} }
// try with 100ms delays until we hit 3s // try with 100ms delays until we exceed the grace period
apiKey, start := "", time.Now() apiKey, start := "", time.Now()
for apiKey == "" { for apiKey == "" {
apiKey, err = newConsoleEndpoints(consoleAddress).createOrGetAPIKey(context.Background()) apiKey, err = newConsoleEndpoints(consoleAddress).createOrGetAPIKey(context.Background())
if err != nil && time.Since(start) > 3*time.Second { if err != nil && time.Since(start) > gatewayGracePeriod {
return fmt.Errorf("failed to create account: %w", err) return fmt.Errorf("failed to create account: %w", err)
} }
time.Sleep(100 * time.Millisecond) time.Sleep(100 * time.Millisecond)

View File

@ -255,7 +255,8 @@ func (process *Process) Exec(ctx context.Context, command string) (err error) {
if _, ok := process.Arguments[command]; !ok { if _, ok := process.Arguments[command]; !ok {
fmt.Fprintf(process.processes.Output, "%s running: %s\n", process.Name, command) fmt.Fprintf(process.processes.Output, "%s running: %s\n", process.Name, command)
return //TODO: This doesn't look right, but keeping the same behaviour as before.
return nil
} }
cmd := exec.CommandContext(ctx, executable, process.Arguments[command]...) cmd := exec.CommandContext(ctx, executable, process.Arguments[command]...)

View File

@ -4,10 +4,12 @@
package main package main
import ( import (
"context"
"encoding/hex" "encoding/hex"
"fmt" "fmt"
"os" "os"
"storj.io/common/identity"
"storj.io/common/storj" "storj.io/common/storj"
) )
@ -48,6 +50,17 @@ func main() {
} }
} }
if chain, err := os.ReadFile(os.Args[1]); err == nil {
if id, err := identity.PeerIdentityFromPEM(chain); err == nil {
output(id.ID)
return
}
if id, err := identity.DecodePeerIdentity(context.Background(), chain); err == nil {
output(id.ID)
return
}
}
fmt.Fprintf(os.Stderr, "unknown argument: %q", os.Args[1]) fmt.Fprintf(os.Stderr, "unknown argument: %q", os.Args[1])
usage() usage()
} }

View File

@ -0,0 +1,148 @@
// Copyright (C) 2023 Storj Labs, Inc.
// See LICENSE for copying information.
package main
import (
"context"
"encoding/json"
"fmt"
"strings"
"time"
"github.com/spf13/cobra"
"github.com/spf13/viper"
"github.com/zeebo/errs"
"go.uber.org/zap"
"storj.io/common/storj"
"storj.io/common/storj/location"
"storj.io/private/process"
"storj.io/storj/satellite/nodeselection"
"storj.io/storj/satellite/overlay"
)
var (
rootCmd = &cobra.Command{
Use: "placement-test <countrycode:...,lastipport:...,lastnet:...,tag:signer/key/value,tag:signer/key/value...>",
Short: "Test placement settings",
Long: `"This command helps testing placement configuration.
You can define a custom node with attributes, and all available placement configuration will be tested against the node.
Supported node attributes:
* countrycode
* lastipport
* lastnet
* tag (value should be in the form of signer/key/value)
EXAMPLES:
placement-test --placement '10:country("GB");12:country("DE")' countrycode=11
placement-test --placement /tmp/proposal.txt countrycode=US,tag=12Q8q2PofHPwycSwAVCpjNxxzWiDJhi8UV4ceZBo4hmNARpYcR7/soc2/true
Where /tmp/proposal.txt contains definitions, for example:
10:tag("12Q8q2PofHPwycSwAVCpjNxxzWiDJhi8UV4ceZBo4hmNARpYcR7","selected",notEmpty());
1:country("EU") && exclude(placement(10)) && annotation("location","eu-1");
2:country("EEA") && exclude(placement(10)) && annotation("location","eea-1");
3:country("US") && exclude(placement(10)) && annotation("location","us-1");
4:country("DE") && exclude(placement(10)) && annotation("location","de-1");
6:country("*","!BY", "!RU", "!NONE") && exclude(placement(10)) && annotation("location","custom-1")
`,
Args: cobra.ExactArgs(1),
RunE: func(cmd *cobra.Command, args []string) error {
ctx, _ := process.Ctx(cmd)
return testPlacement(ctx, args[0])
},
}
config Config
)
func testPlacement(ctx context.Context, fakeNode string) error {
node := &nodeselection.SelectedNode{}
for _, part := range strings.Split(fakeNode, ",") {
kv := strings.SplitN(part, "=", 2)
switch strings.ToLower(kv[0]) {
case "countrycode":
node.CountryCode = location.ToCountryCode(kv[1])
case "lastipport":
node.LastIPPort = kv[1]
case "lastnet":
node.LastNet = kv[1]
case "tag":
tkv := strings.SplitN(kv[1], "/", 3)
signer, err := storj.NodeIDFromString(tkv[0])
if err != nil {
return err
}
node.Tags = append(node.Tags, nodeselection.NodeTag{
Name: tkv[1],
Value: []byte(tkv[2]),
Signer: signer,
SignedAt: time.Now(),
NodeID: node.ID,
})
default:
panic("Unsupported field of SelectedNode: " + kv[0])
}
}
placement, err := config.Placement.Parse()
if err != nil {
return errs.Wrap(err)
}
fmt.Println("Node:")
jsonNode, err := json.MarshalIndent(node, " ", " ")
if err != nil {
return errs.Wrap(err)
}
fmt.Println(string(jsonNode))
for _, placementNum := range placement.SupportedPlacements() {
fmt.Printf("\n--------- Evaluating placement rule %d ---------\n", placementNum)
filter := placement.CreateFilters(placementNum)
fmt.Printf("Placement: %s\n", filter)
result := filter.Match(node)
fmt.Println("MATCH: ", result)
fmt.Println("Annotations: ")
if annotated, ok := filter.(nodeselection.NodeFilterWithAnnotation); ok {
fmt.Println(" location:", annotated.GetAnnotation("location"))
fmt.Println(" "+nodeselection.AutoExcludeSubnet+":", annotated.GetAnnotation(nodeselection.AutoExcludeSubnet))
} else {
fmt.Println(" no annotation presents")
}
}
return nil
}
// Config contains configuration of placement.
type Config struct {
Placement overlay.ConfigurablePlacementRule `help:"detailed placement rules in the form 'id:definition;id:definition;...' where id is a 16 bytes integer (use >10 for backward compatibility), definition is a combination of the following functions:country(2 letter country codes,...), tag(nodeId, key, bytes(value)) all(...,...)."`
}
func init() {
process.Bind(rootCmd, &config)
}
func main() {
process.ExecWithCustomOptions(rootCmd, process.ExecOptions{
LoadConfig: func(cmd *cobra.Command, vip *viper.Viper) error {
return nil
},
InitTracing: false,
LoggerFactory: func(logger *zap.Logger) *zap.Logger {
newLogger, level, err := process.NewLogger("placement-test")
if err != nil {
panic(err)
}
level.SetLevel(zap.WarnLevel)
return newLogger
},
})
}

View File

@ -142,10 +142,13 @@ type ReadCSVConfig struct {
} }
func verifySegments(cmd *cobra.Command, args []string) error { func verifySegments(cmd *cobra.Command, args []string) error {
ctx, _ := process.Ctx(cmd) ctx, _ := process.Ctx(cmd)
log := zap.L() log := zap.L()
return verifySegmentsInContext(ctx, log, cmd, satelliteCfg, rangeCfg)
}
func verifySegmentsInContext(ctx context.Context, log *zap.Logger, cmd *cobra.Command, satelliteCfg Satellite, rangeCfg RangeConfig) error {
// open default satellite database // open default satellite database
db, err := satellitedb.Open(ctx, log.Named("db"), satelliteCfg.Database, satellitedb.Options{ db, err := satellitedb.Open(ctx, log.Named("db"), satelliteCfg.Database, satellitedb.Options{
ApplicationName: "segment-verify", ApplicationName: "segment-verify",
@ -203,12 +206,12 @@ func verifySegments(cmd *cobra.Command, args []string) error {
dialer := rpc.NewDefaultDialer(tlsOptions) dialer := rpc.NewDefaultDialer(tlsOptions)
// setup dependencies for verification // setup dependencies for verification
overlay, err := overlay.NewService(log.Named("overlay"), db.OverlayCache(), db.NodeEvents(), "", "", satelliteCfg.Overlay) overlayService, err := overlay.NewService(log.Named("overlay"), db.OverlayCache(), db.NodeEvents(), overlay.NewPlacementDefinitions().CreateFilters, "", "", satelliteCfg.Overlay)
if err != nil { if err != nil {
return Error.Wrap(err) return Error.Wrap(err)
} }
ordersService, err := orders.NewService(log.Named("orders"), signing.SignerFromFullIdentity(identity), overlay, orders.NewNoopDB(), satelliteCfg.Orders) ordersService, err := orders.NewService(log.Named("orders"), signing.SignerFromFullIdentity(identity), overlayService, orders.NewNoopDB(), overlay.NewPlacementDefinitions().CreateFilters, satelliteCfg.Orders)
if err != nil { if err != nil {
return Error.Wrap(err) return Error.Wrap(err)
} }
@ -243,11 +246,10 @@ func verifySegments(cmd *cobra.Command, args []string) error {
// setup verifier // setup verifier
verifier := NewVerifier(log.Named("verifier"), dialer, ordersService, verifyConfig) verifier := NewVerifier(log.Named("verifier"), dialer, ordersService, verifyConfig)
service, err := NewService(log.Named("service"), metabaseDB, verifier, overlay, serviceConfig) service, err := NewService(log.Named("service"), metabaseDB, verifier, overlayService, serviceConfig)
if err != nil { if err != nil {
return Error.Wrap(err) return Error.Wrap(err)
} }
verifier.reportPiece = service.problemPieces.Write
defer func() { err = errs.Combine(err, service.Close()) }() defer func() { err = errs.Combine(err, service.Close()) }()
log.Debug("starting", zap.Any("config", service.config), zap.String("command", cmd.Name())) log.Debug("starting", zap.Any("config", service.config), zap.String("command", cmd.Name()))

View File

@ -0,0 +1,282 @@
// Copyright (C) 2023 Storj Labs, Inc.
// See LICENSE for copying information
package main
import (
"context"
"encoding/csv"
"errors"
"fmt"
"io"
"math/rand"
"os"
"strconv"
"strings"
"testing"
"time"
"github.com/jackc/pgx/v5/stdlib"
"github.com/spf13/cobra"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"go.uber.org/zap/zaptest"
"storj.io/common/memory"
"storj.io/common/storj"
"storj.io/common/testcontext"
"storj.io/common/testrand"
"storj.io/common/uuid"
"storj.io/private/dbutil/cockroachutil"
"storj.io/private/tagsql"
"storj.io/storj/private/testplanet"
"storj.io/storj/satellite/metabase"
"storj.io/storj/storagenode/pieces"
)
func TestCommandLineTool(t *testing.T) {
const (
nodeCount = 10
uplinkCount = 10
)
testplanet.Run(t, testplanet.Config{
SatelliteCount: 1, StorageNodeCount: nodeCount, UplinkCount: uplinkCount,
Reconfigure: testplanet.Reconfigure{
Satellite: testplanet.ReconfigureRS(nodeCount, nodeCount, nodeCount, nodeCount),
},
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
satellite := planet.Satellites[0]
// get the db connstrings that we can set in the global config (these are hilariously hard to get,
// but we really don't need to get them anywhere else in the codebase)
dbConnString := getConnStringFromDBConn(t, ctx, satellite.DB.Testing().RawDB())
metaDBConnString := getConnStringFromDBConn(t, ctx, satellite.Metabase.DB.UnderlyingTagSQL())
notFoundCSV := ctx.File("notfound.csv")
retryCSV := ctx.File("retry.csv")
problemPiecesCSV := ctx.File("problempieces.csv")
// set up global config that the main func will use
satelliteCfg := satelliteCfg
satelliteCfg.Config = satellite.Config
satelliteCfg.Database = dbConnString
satelliteCfg.Metainfo.DatabaseURL = metaDBConnString
satelliteCfg.Identity.KeyPath = ctx.File("identity-key")
satelliteCfg.Identity.CertPath = ctx.File("identity-cert")
require.NoError(t, satelliteCfg.Identity.Save(satellite.Identity))
rangeCfg := rangeCfg
rangeCfg.Verify = VerifierConfig{
PerPieceTimeout: time.Second,
OrderRetryThrottle: 500 * time.Millisecond,
RequestThrottle: 500 * time.Millisecond,
}
rangeCfg.Service = ServiceConfig{
NotFoundPath: notFoundCSV,
RetryPath: retryCSV,
ProblemPiecesPath: problemPiecesCSV,
Check: 0,
BatchSize: 10000,
Concurrency: 1000,
MaxOffline: 2,
OfflineStatusCacheTime: 10 * time.Second,
AsOfSystemInterval: -1 * time.Microsecond,
}
rangeCfg.Low = strings.Repeat("0", 32)
rangeCfg.High = strings.Repeat("f", 32)
// upload some data
data := testrand.Bytes(8 * memory.KiB)
for u, up := range planet.Uplinks {
for i := 0; i < nodeCount; i++ {
err := up.Upload(ctx, satellite, "bucket1", fmt.Sprintf("uplink%d/i%d", u, i), data)
require.NoError(t, err)
}
}
// take one node offline so there will be some pieces in the retry list
offlineNode := planet.StorageNodes[0]
require.NoError(t, planet.StopPeer(offlineNode))
// and delete 10% of pieces at random so there will be some pieces in the not-found list
const deleteFrac = 0.10
allDeletedPieces := make(map[storj.NodeID]map[storj.PieceID]struct{})
numDeletedPieces := 0
for nodeNum, node := range planet.StorageNodes {
if node.ID() == offlineNode.ID() {
continue
}
deletedPieces, err := deletePiecesRandomly(ctx, satellite.ID(), node, deleteFrac)
require.NoError(t, err, nodeNum)
allDeletedPieces[node.ID()] = deletedPieces
numDeletedPieces += len(deletedPieces)
}
// check that the number of segments we expect are present in the metainfo db
result, err := satellite.Metabase.DB.ListVerifySegments(ctx, metabase.ListVerifySegments{
CursorStreamID: uuid.UUID{},
CursorPosition: metabase.SegmentPosition{},
Limit: 10000,
})
require.NoError(t, err)
require.Len(t, result.Segments, uplinkCount*nodeCount)
// perform the verify!
log := zaptest.NewLogger(t)
err = verifySegmentsInContext(ctx, log, &cobra.Command{Use: "range"}, satelliteCfg, rangeCfg)
require.NoError(t, err)
// open the CSVs to check that we get the expected results
retryCSVHandle, err := os.Open(retryCSV)
require.NoError(t, err)
defer ctx.Check(retryCSVHandle.Close)
retryCSVReader := csv.NewReader(retryCSVHandle)
notFoundCSVHandle, err := os.Open(notFoundCSV)
require.NoError(t, err)
defer ctx.Check(notFoundCSVHandle.Close)
notFoundCSVReader := csv.NewReader(notFoundCSVHandle)
problemPiecesCSVHandle, err := os.Open(problemPiecesCSV)
require.NoError(t, err)
defer ctx.Check(problemPiecesCSVHandle.Close)
problemPiecesCSVReader := csv.NewReader(problemPiecesCSVHandle)
// in the retry CSV, we don't expect any rows, because there would need to be more than 5
// nodes offline to produce records here.
// TODO: make that 5 configurable so we can override it here and check results
header, err := retryCSVReader.Read()
require.NoError(t, err)
assert.Equal(t, []string{"stream id", "position", "found", "not found", "retry"}, header)
for {
record, err := retryCSVReader.Read()
if errors.Is(err, io.EOF) {
break
}
require.NoError(t, err)
assert.Fail(t, "unexpected record in retry.csv", "%v", record)
}
// we do expect plenty of rows in not-found.csv. we don't know exactly what pieces these
// pertain to, but we can add up all the reported not-found pieces and expect the total
// to match numDeletedPieces. In addition, for each segment, found+notfound+retry should
// equal nodeCount.
header, err = notFoundCSVReader.Read()
require.NoError(t, err)
assert.Equal(t, []string{"stream id", "position", "found", "not found", "retry"}, header)
identifiedNotFoundPieces := 0
for {
record, err := notFoundCSVReader.Read()
if errors.Is(err, io.EOF) {
break
}
require.NoError(t, err)
found, err := strconv.Atoi(record[2])
require.NoError(t, err)
notFound, err := strconv.Atoi(record[3])
require.NoError(t, err)
retry, err := strconv.Atoi(record[4])
require.NoError(t, err)
lineNum, _ := notFoundCSVReader.FieldPos(0)
assert.Equal(t, nodeCount, found+notFound+retry,
"line %d of not-found.csv contains record: %v where found+notFound+retry != %d", lineNum, record, nodeCount)
identifiedNotFoundPieces += notFound
}
assert.Equal(t, numDeletedPieces, identifiedNotFoundPieces)
// finally, in problem-pieces.csv, we can check results with more precision. we expect
// that all deleted pieces were identified, and that no pieces were identified as not found
// unless we deleted them specifically.
header, err = problemPiecesCSVReader.Read()
require.NoError(t, err)
assert.Equal(t, []string{"stream id", "position", "node id", "piece number", "outcome"}, header)
for {
record, err := problemPiecesCSVReader.Read()
if errors.Is(err, io.EOF) {
break
}
streamID, err := uuid.FromString(record[0])
require.NoError(t, err)
position, err := strconv.ParseUint(record[1], 10, 64)
require.NoError(t, err)
nodeID, err := storj.NodeIDFromString(record[2])
require.NoError(t, err)
pieceNum, err := strconv.ParseInt(record[3], 10, 16)
require.NoError(t, err)
outcome := record[4]
switch outcome {
case "NODE_OFFLINE":
// expect that this was the node we took offline
assert.Equal(t, offlineNode.ID(), nodeID,
"record %v said node %s was offline, but we didn't take it offline", record, nodeID)
case "NOT_FOUND":
segmentPosition := metabase.SegmentPositionFromEncoded(position)
segment, err := satellite.Metabase.DB.GetSegmentByPosition(ctx, metabase.GetSegmentByPosition{
StreamID: streamID,
Position: segmentPosition,
})
require.NoError(t, err)
pieceID := segment.RootPieceID.Derive(nodeID, int32(pieceNum))
deletedPiecesForNode, ok := allDeletedPieces[nodeID]
require.True(t, ok)
_, ok = deletedPiecesForNode[pieceID]
assert.True(t, ok, "we did not delete piece ID %s, but it was identified as not found", pieceID)
delete(deletedPiecesForNode, pieceID)
default:
assert.Fail(t, "unexpected outcome from problem-pieces.csv", "got %q, but expected \"NODE_OFFLINE\" or \"NOT_FOUND\"", outcome)
}
}
for node, deletedPieces := range allDeletedPieces {
assert.Empty(t, deletedPieces, "pieces were deleted from %v but were not reported in problem-pieces.csv", node)
}
})
}
func deletePiecesRandomly(ctx context.Context, satelliteID storj.NodeID, node *testplanet.StorageNode, rate float64) (deletedPieces map[storj.PieceID]struct{}, err error) {
deletedPieces = make(map[storj.PieceID]struct{})
err = node.Storage2.FileWalker.WalkSatellitePieces(ctx, satelliteID, func(access pieces.StoredPieceAccess) error {
if rand.Float64() < rate {
path, err := access.FullPath(ctx)
if err != nil {
return err
}
err = os.Remove(path)
if err != nil {
return err
}
deletedPieces[access.PieceID()] = struct{}{}
}
return nil
})
return deletedPieces, err
}
func getConnStringFromDBConn(t *testing.T, ctx *testcontext.Context, tagsqlDB tagsql.DB) (dbConnString string) {
type dbConnGetter interface {
StdlibConn() *stdlib.Conn
}
dbConn, err := tagsqlDB.Conn(ctx)
require.NoError(t, err)
defer ctx.Check(dbConn.Close)
err = dbConn.Raw(ctx, func(driverConn interface{}) error {
var stdlibConn *stdlib.Conn
switch conn := driverConn.(type) {
case dbConnGetter:
stdlibConn = conn.StdlibConn()
case *stdlib.Conn:
stdlibConn = conn
}
dbConnString = stdlibConn.Conn().Config().ConnString()
return nil
})
require.NoError(t, err)
if _, ok := tagsqlDB.Driver().(*cockroachutil.Driver); ok {
dbConnString = strings.ReplaceAll(dbConnString, "postgres://", "cockroach://")
}
return dbConnString
}

View File

@ -15,6 +15,7 @@ import (
"storj.io/common/uuid" "storj.io/common/uuid"
"storj.io/private/process" "storj.io/private/process"
"storj.io/storj/satellite/metabase" "storj.io/storj/satellite/metabase"
"storj.io/storj/satellite/nodeselection"
"storj.io/storj/satellite/overlay" "storj.io/storj/satellite/overlay"
"storj.io/storj/satellite/satellitedb" "storj.io/storj/satellite/satellitedb"
) )
@ -78,7 +79,7 @@ type NodeCheckConfig struct {
// NodeCheckOverlayDB contains dependencies from overlay that are needed for the processing. // NodeCheckOverlayDB contains dependencies from overlay that are needed for the processing.
type NodeCheckOverlayDB interface { type NodeCheckOverlayDB interface {
IterateAllContactedNodes(context.Context, func(context.Context, *overlay.SelectedNode) error) error IterateAllContactedNodes(context.Context, func(context.Context, *nodeselection.SelectedNode) error) error
IterateAllNodeDossiers(context.Context, func(context.Context, *overlay.NodeDossier) error) error IterateAllNodeDossiers(context.Context, func(context.Context, *overlay.NodeDossier) error) error
} }

View File

@ -5,6 +5,7 @@ package main
import ( import (
"context" "context"
"errors"
"sync" "sync"
"go.uber.org/zap" "go.uber.org/zap"
@ -82,24 +83,29 @@ func (service *Service) VerifyBatches(ctx context.Context, batches []*Batch) err
limiter := sync2.NewLimiter(service.config.Concurrency) limiter := sync2.NewLimiter(service.config.Concurrency)
for _, batch := range batches { for _, batch := range batches {
batch := batch batch := batch
log := service.log.With(zap.Int("num pieces", batch.Len()))
info, err := service.GetNodeInfo(ctx, batch.Alias) info, err := service.GetNodeInfo(ctx, batch.Alias)
if err != nil { if err != nil {
if ErrNoSuchNode.Has(err) { if ErrNoSuchNode.Has(err) {
service.log.Error("will not verify batch; consider pieces lost", log.Info("node has left the cluster; considering pieces lost",
zap.Int("alias", int(batch.Alias)), zap.Int("alias", int(batch.Alias)))
zap.Error(err)) for _, seg := range batch.Items {
seg.Status.MarkNotFound()
}
continue continue
} }
return Error.Wrap(err) return Error.Wrap(err)
} }
log = log.With(zap.Stringer("node ID", info.NodeURL.ID))
ignoreThrottle := service.priorityNodes.Contains(batch.Alias) ignoreThrottle := service.priorityNodes.Contains(batch.Alias)
limiter.Go(ctx, func() { limiter.Go(ctx, func() {
verifiedCount, err := service.verifier.Verify(ctx, batch.Alias, info.NodeURL, info.Version, batch.Items, ignoreThrottle) verifiedCount, err := service.verifier.Verify(ctx, batch.Alias, info.NodeURL, info.Version, batch.Items, ignoreThrottle)
if err != nil { if err != nil {
if ErrNodeOffline.Has(err) { switch {
case ErrNodeOffline.Has(err):
mu.Lock() mu.Lock()
if verifiedCount == 0 { if verifiedCount == 0 {
service.offlineNodes.Add(batch.Alias) service.offlineNodes.Add(batch.Alias)
@ -110,8 +116,14 @@ func (service *Service) VerifyBatches(ctx context.Context, batches []*Batch) err
} }
} }
mu.Unlock() mu.Unlock()
log.Info("node is offline; marking pieces as retryable")
return
case errors.Is(err, context.DeadlineExceeded):
log.Info("request to node timed out; marking pieces as retryable")
return
default:
log.Error("verifying a batch failed", zap.Error(err))
} }
service.log.Error("verifying a batch failed", zap.Error(err))
} else { } else {
mu.Lock() mu.Lock()
if service.offlineCount[batch.Alias] > 0 { if service.offlineCount[batch.Alias] > 0 {
@ -128,8 +140,12 @@ func (service *Service) VerifyBatches(ctx context.Context, batches []*Batch) err
// convertAliasToNodeURL converts a node alias to node url, using a cache if needed. // convertAliasToNodeURL converts a node alias to node url, using a cache if needed.
func (service *Service) convertAliasToNodeURL(ctx context.Context, alias metabase.NodeAlias) (_ storj.NodeURL, err error) { func (service *Service) convertAliasToNodeURL(ctx context.Context, alias metabase.NodeAlias) (_ storj.NodeURL, err error) {
service.mu.RLock()
nodeURL, ok := service.aliasToNodeURL[alias] nodeURL, ok := service.aliasToNodeURL[alias]
service.mu.RUnlock()
if !ok { if !ok {
service.mu.Lock()
defer service.mu.Unlock()
nodeID, ok := service.aliasMap.Node(alias) nodeID, ok := service.aliasMap.Node(alias)
if !ok { if !ok {
latest, err := service.metabase.LatestNodesAliasMap(ctx) latest, err := service.metabase.LatestNodesAliasMap(ctx)

View File

@ -10,10 +10,12 @@ import (
"io" "io"
"os" "os"
"strings" "strings"
"sync"
"sync/atomic" "sync/atomic"
"time" "time"
"github.com/spacemonkeygo/monkit/v3" "github.com/spacemonkeygo/monkit/v3"
"github.com/spf13/pflag"
"github.com/zeebo/errs" "github.com/zeebo/errs"
"go.uber.org/zap" "go.uber.org/zap"
@ -21,6 +23,7 @@ import (
"storj.io/common/uuid" "storj.io/common/uuid"
"storj.io/storj/satellite/audit" "storj.io/storj/satellite/audit"
"storj.io/storj/satellite/metabase" "storj.io/storj/satellite/metabase"
"storj.io/storj/satellite/nodeselection"
"storj.io/storj/satellite/overlay" "storj.io/storj/satellite/overlay"
) )
@ -46,7 +49,7 @@ type Verifier interface {
type Overlay interface { type Overlay interface {
// Get looks up the node by nodeID // Get looks up the node by nodeID
Get(ctx context.Context, nodeID storj.NodeID) (*overlay.NodeDossier, error) Get(ctx context.Context, nodeID storj.NodeID) (*overlay.NodeDossier, error)
SelectAllStorageNodesDownload(ctx context.Context, onlineWindow time.Duration, asOf overlay.AsOfSystemTimeConfig) ([]*overlay.SelectedNode, error) SelectAllStorageNodesDownload(ctx context.Context, onlineWindow time.Duration, asOf overlay.AsOfSystemTimeConfig) ([]*nodeselection.SelectedNode, error)
} }
// SegmentWriter allows writing segments to some output. // SegmentWriter allows writing segments to some output.
@ -70,6 +73,9 @@ type ServiceConfig struct {
OfflineStatusCacheTime time.Duration `help:"how long to cache a \"node offline\" status" default:"30m"` OfflineStatusCacheTime time.Duration `help:"how long to cache a \"node offline\" status" default:"30m"`
CreatedBefore DateFlag `help:"verify only segments created before specific date (date format 'YYYY-MM-DD')" default:""`
CreatedAfter DateFlag `help:"verify only segments created after specific date (date format 'YYYY-MM-DD')" default:"1970-01-01"`
AsOfSystemInterval time.Duration `help:"as of system interval" releaseDefault:"-5m" devDefault:"-1us" testDefault:"-1us"` AsOfSystemInterval time.Duration `help:"as of system interval" releaseDefault:"-5m" devDefault:"-1us" testDefault:"-1us"`
} }
@ -93,8 +99,9 @@ type Service struct {
verifier Verifier verifier Verifier
overlay Overlay overlay Overlay
aliasMap *metabase.NodeAliasMap mu sync.RWMutex
aliasToNodeURL map[metabase.NodeAlias]storj.NodeURL aliasToNodeURL map[metabase.NodeAlias]storj.NodeURL
aliasMap *metabase.NodeAliasMap
priorityNodes NodeAliasSet priorityNodes NodeAliasSet
ignoreNodes NodeAliasSet ignoreNodes NodeAliasSet
offlineNodes *nodeAliasExpiringSet offlineNodes *nodeAliasExpiringSet
@ -120,6 +127,10 @@ func NewService(log *zap.Logger, metabaseDB Metabase, verifier Verifier, overlay
return nil, errs.Combine(Error.Wrap(err), retry.Close(), notFound.Close()) return nil, errs.Combine(Error.Wrap(err), retry.Close(), notFound.Close())
} }
if nodeVerifier, ok := verifier.(*NodeVerifier); ok {
nodeVerifier.reportPiece = problemPieces.Write
}
return &Service{ return &Service{
log: log, log: log,
config: config, config: config,
@ -293,6 +304,9 @@ func (service *Service) ProcessRange(ctx context.Context, low, high uuid.UUID) (
CursorPosition: cursorPosition, CursorPosition: cursorPosition,
Limit: service.config.BatchSize, Limit: service.config.BatchSize,
CreatedAfter: service.config.CreatedAfter.time(),
CreatedBefore: service.config.CreatedBefore.time(),
AsOfSystemInterval: service.config.AsOfSystemInterval, AsOfSystemInterval: service.config.AsOfSystemInterval,
}) })
if err != nil { if err != nil {
@ -485,6 +499,9 @@ func (service *Service) ProcessSegmentsFromCSV(ctx context.Context, segmentSourc
} }
for n, verifySegment := range verifySegments.Segments { for n, verifySegment := range verifySegments.Segments {
segmentsData[n].VerifySegment = verifySegment segmentsData[n].VerifySegment = verifySegment
segmentsData[n].Status.Found = 0
segmentsData[n].Status.Retry = 0
segmentsData[n].Status.NotFound = 0
segments[n] = &segmentsData[n] segments[n] = &segmentsData[n]
} }
@ -617,3 +634,42 @@ func uuidBefore(v uuid.UUID) uuid.UUID {
} }
return v return v
} }
// DateFlag flag implementation for date, format YYYY-MM-DD.
type DateFlag struct {
time.Time
}
// String implements pflag.Value.
func (t *DateFlag) String() string {
return t.Format(time.DateOnly)
}
// Set implements pflag.Value.
func (t *DateFlag) Set(s string) error {
if s == "" {
t.Time = time.Now()
return nil
}
parsedTime, err := time.Parse(time.DateOnly, s)
if err != nil {
return err
}
t.Time = parsedTime
return nil
}
func (t *DateFlag) time() *time.Time {
if t.IsZero() {
return nil
}
return &t.Time
}
// Type implements pflag.Value.
func (t *DateFlag) Type() string {
return "time-flag"
}
var _ pflag.Value = &DateFlag{}

View File

@ -23,6 +23,7 @@ import (
segmentverify "storj.io/storj/cmd/tools/segment-verify" segmentverify "storj.io/storj/cmd/tools/segment-verify"
"storj.io/storj/private/testplanet" "storj.io/storj/private/testplanet"
"storj.io/storj/satellite/metabase" "storj.io/storj/satellite/metabase"
"storj.io/storj/satellite/nodeselection"
"storj.io/storj/satellite/overlay" "storj.io/storj/satellite/overlay"
) )
@ -344,10 +345,10 @@ func (db *metabaseMock) Get(ctx context.Context, nodeID storj.NodeID) (*overlay.
}, nil }, nil
} }
func (db *metabaseMock) SelectAllStorageNodesDownload(ctx context.Context, onlineWindow time.Duration, asOf overlay.AsOfSystemTimeConfig) ([]*overlay.SelectedNode, error) { func (db *metabaseMock) SelectAllStorageNodesDownload(ctx context.Context, onlineWindow time.Duration, asOf overlay.AsOfSystemTimeConfig) ([]*nodeselection.SelectedNode, error) {
var xs []*overlay.SelectedNode var xs []*nodeselection.SelectedNode
for nodeID := range db.nodeIDToAlias { for nodeID := range db.nodeIDToAlias {
xs = append(xs, &overlay.SelectedNode{ xs = append(xs, &nodeselection.SelectedNode{
ID: nodeID, ID: nodeID,
Address: &pb.NodeAddress{ Address: &pb.NodeAddress{
Address: fmt.Sprintf("nodeid:%v", nodeID), Address: fmt.Sprintf("nodeid:%v", nodeID),

View File

@ -4,7 +4,7 @@
package main_test package main_test
import ( import (
"strconv" "fmt"
"testing" "testing"
"time" "time"
@ -23,15 +23,19 @@ import (
) )
func TestVerifier(t *testing.T) { func TestVerifier(t *testing.T) {
const (
nodeCount = 10
uplinkCount = 10
)
testplanet.Run(t, testplanet.Config{ testplanet.Run(t, testplanet.Config{
SatelliteCount: 1, StorageNodeCount: 4, UplinkCount: 1, SatelliteCount: 1, StorageNodeCount: nodeCount, UplinkCount: uplinkCount,
Reconfigure: testplanet.Reconfigure{ Reconfigure: testplanet.Reconfigure{
Satellite: testplanet.ReconfigureRS(4, 4, 4, 4), Satellite: testplanet.ReconfigureRS(nodeCount, nodeCount, nodeCount, nodeCount),
}, },
}, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) { }, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
satellite := planet.Satellites[0] satellite := planet.Satellites[0]
snoCount := int32(len(planet.StorageNodes))
olderNodeVersion := "v1.68.1" // version without Exists endpoint olderNodeVersion := "v1.68.1" // version without Exists endpoint
newerNodeVersion := "v1.69.2" // minimum version with Exists endpoint newerNodeVersion := "v1.69.2" // minimum version with Exists endpoint
@ -46,7 +50,7 @@ func TestVerifier(t *testing.T) {
observedZapCore, observedLogs := observer.New(zap.DebugLevel) observedZapCore, observedLogs := observer.New(zap.DebugLevel)
observedLogger := zap.New(observedZapCore).Named("verifier") observedLogger := zap.New(observedZapCore).Named("verifier")
service := segmentverify.NewVerifier( verifier := segmentverify.NewVerifier(
observedLogger, observedLogger,
satellite.Dialer, satellite.Dialer,
satellite.Orders.Service, satellite.Orders.Service,
@ -54,9 +58,9 @@ func TestVerifier(t *testing.T) {
// upload some data // upload some data
data := testrand.Bytes(8 * memory.KiB) data := testrand.Bytes(8 * memory.KiB)
for _, up := range planet.Uplinks { for u, up := range planet.Uplinks {
for i := 0; i < 10; i++ { for i := 0; i < nodeCount; i++ {
err := up.Upload(ctx, satellite, "bucket1", strconv.Itoa(i), data) err := up.Upload(ctx, satellite, "bucket1", fmt.Sprintf("uplink%d/i%d", u, i), data)
require.NoError(t, err) require.NoError(t, err)
} }
} }
@ -67,50 +71,57 @@ func TestVerifier(t *testing.T) {
Limit: 10000, Limit: 10000,
}) })
require.NoError(t, err) require.NoError(t, err)
require.Len(t, result.Segments, uplinkCount*nodeCount)
validSegments := []*segmentverify.Segment{} validSegments := make([]*segmentverify.Segment, len(result.Segments))
for _, raw := range result.Segments { for i, raw := range result.Segments {
validSegments = append(validSegments, &segmentverify.Segment{ validSegments[i] = &segmentverify.Segment{VerifySegment: raw}
VerifySegment: raw,
Status: segmentverify.Status{Retry: snoCount},
})
} }
resetStatuses := func() {
for _, seg := range validSegments {
seg.Status = segmentverify.Status{Retry: nodeCount}
}
}
resetStatuses()
aliasMap, err := satellite.Metabase.DB.LatestNodesAliasMap(ctx) aliasMap, err := satellite.Metabase.DB.LatestNodesAliasMap(ctx)
require.NoError(t, err) require.NoError(t, err)
nodeWithExistsEndpoint := planet.StorageNodes[testrand.Intn(len(planet.StorageNodes)-1)] t.Run("verify all", func(t *testing.T) {
nodeWithExistsEndpoint := planet.StorageNodes[testrand.Intn(len(planet.StorageNodes)-1)]
var g errgroup.Group var g errgroup.Group
for _, node := range planet.StorageNodes { for _, node := range planet.StorageNodes {
node := node node := node
nodeVersion := olderNodeVersion nodeVersion := olderNodeVersion
if node == nodeWithExistsEndpoint { if node == nodeWithExistsEndpoint {
nodeVersion = newerNodeVersion nodeVersion = newerNodeVersion
}
alias, ok := aliasMap.Alias(node.ID())
require.True(t, ok)
g.Go(func() error {
_, err := verifier.Verify(ctx, alias, node.NodeURL(), nodeVersion, validSegments, true)
return err
})
} }
alias, ok := aliasMap.Alias(node.ID()) require.NoError(t, g.Wait())
require.True(t, ok) require.NotZero(t, len(observedLogs.All()))
g.Go(func() error {
_, err := service.Verify(ctx, alias, node.NodeURL(), nodeVersion, validSegments, true)
return err
})
}
require.NoError(t, g.Wait())
require.NotZero(t, len(observedLogs.All()))
// check that segments were verified with download method // check that segments were verified with download method
fallbackLogs := observedLogs.FilterMessage("fallback to download method").All() fallbackLogs := observedLogs.FilterMessage("fallback to download method").All()
require.Equal(t, 3, len(fallbackLogs)) require.Equal(t, nodeCount-1, len(fallbackLogs))
require.Equal(t, zap.DebugLevel, fallbackLogs[0].Level) require.Equal(t, zap.DebugLevel, fallbackLogs[0].Level)
// check that segments were verified with exists endpoint // check that segments were verified with exists endpoint
existsLogs := observedLogs.FilterMessage("verify segments using Exists method").All() existsLogs := observedLogs.FilterMessage("verify segments using Exists method").All()
require.Equal(t, 1, len(existsLogs)) require.Equal(t, 1, len(existsLogs))
require.Equal(t, zap.DebugLevel, existsLogs[0].Level) require.Equal(t, zap.DebugLevel, existsLogs[0].Level)
for _, seg := range validSegments { for segNum, seg := range validSegments {
require.Equal(t, segmentverify.Status{Found: snoCount, NotFound: 0, Retry: 0}, seg.Status) require.Equal(t, segmentverify.Status{Found: nodeCount, NotFound: 0, Retry: 0}, seg.Status, segNum)
} }
})
// segment not found // segment not found
alias0, ok := aliasMap.Alias(planet.StorageNodes[0].ID()) alias0, ok := aliasMap.Alias(planet.StorageNodes[0].ID())
@ -138,7 +149,7 @@ func TestVerifier(t *testing.T) {
var count int var count int
t.Run("segment not found using download method", func(t *testing.T) { t.Run("segment not found using download method", func(t *testing.T) {
// for older node version // for older node version
count, err = service.Verify(ctx, alias0, planet.StorageNodes[0].NodeURL(), olderNodeVersion, count, err = verifier.Verify(ctx, alias0, planet.StorageNodes[0].NodeURL(), olderNodeVersion,
[]*segmentverify.Segment{validSegment0, missingSegment, validSegment1}, true) []*segmentverify.Segment{validSegment0, missingSegment, validSegment1}, true)
require.NoError(t, err) require.NoError(t, err)
require.Equal(t, 3, count) require.Equal(t, 3, count)
@ -153,7 +164,7 @@ func TestVerifier(t *testing.T) {
validSegment1.Status = segmentverify.Status{Retry: 1} validSegment1.Status = segmentverify.Status{Retry: 1}
t.Run("segment not found using exists method", func(t *testing.T) { t.Run("segment not found using exists method", func(t *testing.T) {
count, err = service.Verify(ctx, alias0, planet.StorageNodes[0].NodeURL(), newerNodeVersion, count, err = verifier.Verify(ctx, alias0, planet.StorageNodes[0].NodeURL(), newerNodeVersion,
[]*segmentverify.Segment{validSegment0, missingSegment, validSegment1}, true) []*segmentverify.Segment{validSegment0, missingSegment, validSegment1}, true)
require.NoError(t, err) require.NoError(t, err)
require.Equal(t, 3, count) require.Equal(t, 3, count)
@ -162,31 +173,34 @@ func TestVerifier(t *testing.T) {
require.Equal(t, segmentverify.Status{Found: 1}, validSegment1.Status) require.Equal(t, segmentverify.Status{Found: 1}, validSegment1.Status)
}) })
resetStatuses()
t.Run("test throttling", func(t *testing.T) { t.Run("test throttling", func(t *testing.T) {
// Test throttling // Test throttling
verifyStart := time.Now() verifyStart := time.Now()
const throttleN = 5 const throttleN = 5
count, err = service.Verify(ctx, alias0, planet.StorageNodes[0].NodeURL(), olderNodeVersion, validSegments[:throttleN], false) count, err = verifier.Verify(ctx, alias0, planet.StorageNodes[0].NodeURL(), olderNodeVersion, validSegments[:throttleN], false)
require.NoError(t, err) require.NoError(t, err)
verifyDuration := time.Since(verifyStart) verifyDuration := time.Since(verifyStart)
require.Equal(t, throttleN, count) require.Equal(t, throttleN, count)
require.Greater(t, verifyDuration, config.RequestThrottle*(throttleN-1)) require.Greater(t, verifyDuration, config.RequestThrottle*(throttleN-1))
}) })
// TODO: test download timeout resetStatuses()
// TODO: test download timeout
t.Run("Node offline", func(t *testing.T) { t.Run("Node offline", func(t *testing.T) {
err = planet.StopNodeAndUpdate(ctx, planet.StorageNodes[0]) err = planet.StopNodeAndUpdate(ctx, planet.StorageNodes[0])
require.NoError(t, err) require.NoError(t, err)
// for older node version // for older node version
count, err = service.Verify(ctx, alias0, planet.StorageNodes[0].NodeURL(), olderNodeVersion, validSegments, true) count, err = verifier.Verify(ctx, alias0, planet.StorageNodes[0].NodeURL(), olderNodeVersion, validSegments, true)
require.Error(t, err) require.Error(t, err)
require.Equal(t, 0, count) require.Equal(t, 0, count)
require.True(t, segmentverify.ErrNodeOffline.Has(err)) require.True(t, segmentverify.ErrNodeOffline.Has(err))
// for node version with Exists endpoint // for node version with Exists endpoint
count, err = service.Verify(ctx, alias0, planet.StorageNodes[0].NodeURL(), newerNodeVersion, validSegments, true) count, err = verifier.Verify(ctx, alias0, planet.StorageNodes[0].NodeURL(), newerNodeVersion, validSegments, true)
require.Error(t, err) require.Error(t, err)
require.Equal(t, 0, count) require.Equal(t, 0, count)
require.True(t, segmentverify.ErrNodeOffline.Has(err)) require.True(t, segmentverify.ErrNodeOffline.Has(err))

View File

@ -0,0 +1,218 @@
// Copyright (C) 2023 Storj Labs, Inc.
// See LICENSE for copying information.
package main
import (
"context"
"encoding/base64"
"encoding/hex"
"fmt"
"path/filepath"
"strings"
"time"
"github.com/gogo/protobuf/proto"
"github.com/spf13/cobra"
"github.com/spf13/viper"
"github.com/zeebo/errs"
"go.uber.org/zap"
"storj.io/common/identity"
"storj.io/common/nodetag"
"storj.io/common/pb"
"storj.io/common/signing"
"storj.io/common/storj"
"storj.io/private/process"
)
var (
rootCmd = &cobra.Command{
Use: "tag-signer",
Short: "Sign key=value pairs with identity",
Long: "Node tags are arbitrary key value pairs signed by an authority. If the public key is configured on " +
"Satellite side, Satellite will check the signatures and save the tags, which can be used (for example)" +
" during node selection. Storagenodes can be configured to send encoded node tags to the Satellite. " +
"This utility helps creating/managing the values of this specific configuration value, which is encoded by default.",
}
signCmd = &cobra.Command{
Use: "sign <key=value> <key2=value> ...",
Short: "Create signed tagset",
Args: cobra.MinimumNArgs(1),
RunE: func(cmd *cobra.Command, args []string) error {
ctx, _ := process.Ctx(cmd)
encoded, err := signTags(ctx, config, args)
if err != nil {
return err
}
fmt.Println(encoded)
return nil
},
}
inspectCmd = &cobra.Command{
Use: "inspect <encoded string>",
Short: "Print out the details from an encoded node set",
Args: cobra.ExactArgs(1),
RunE: func(cmd *cobra.Command, args []string) error {
ctx, _ := process.Ctx(cmd)
return inspect(ctx, args[0])
},
}
config Config
)
// Config contains configuration required for signing.
type Config struct {
IdentityDir string `help:"location if the identity files" path:"true"`
NodeID string `help:"the ID of the node, which will used this tag "`
Confirm bool `help:"enable comma in tag values" default:"false"`
}
func init() {
rootCmd.AddCommand(signCmd)
rootCmd.AddCommand(inspectCmd)
process.Bind(signCmd, &config)
}
func signTags(ctx context.Context, cfg Config, tagPairs []string) (string, error) {
if cfg.IdentityDir == "" {
return "", errs.New("Please specify the identity, used as a signer with --identity-dir")
}
if cfg.NodeID == "" {
return "", errs.New("Please specify the --node-id")
}
identityConfig := identity.Config{
CertPath: filepath.Join(cfg.IdentityDir, "identity.cert"),
KeyPath: filepath.Join(cfg.IdentityDir, "identity.key"),
}
fullIdentity, err := identityConfig.Load()
if err != nil {
return "", err
}
signer := signing.SignerFromFullIdentity(fullIdentity)
nodeID, err := storj.NodeIDFromString(cfg.NodeID)
if err != nil {
return "", errs.New("Wrong NodeID format: %v", err)
}
tagSet := &pb.NodeTagSet{
NodeId: nodeID.Bytes(),
SignedAt: time.Now().Unix(),
}
tagSet.Tags, err = parseTagPairs(tagPairs, cfg.Confirm)
if err != nil {
return "", err
}
signedMessage, err := nodetag.Sign(ctx, tagSet, signer)
if err != nil {
return "", err
}
all := &pb.SignedNodeTagSets{
Tags: []*pb.SignedNodeTagSet{
signedMessage,
},
}
raw, err := proto.Marshal(all)
if err != nil {
return "", errs.Wrap(err)
}
return base64.StdEncoding.EncodeToString(raw), nil
}
func inspect(ctx context.Context, s string) error {
raw, err := base64.StdEncoding.DecodeString(s)
if err != nil {
return errs.New("Input is not in base64 format")
}
sets := &pb.SignedNodeTagSets{}
err = proto.Unmarshal(raw, sets)
if err != nil {
return errs.New("Input is not a protobuf encoded *pb.SignedNodeTagSets message")
}
for _, msg := range sets.Tags {
signerNodeID, err := storj.NodeIDFromBytes(msg.SignerNodeId)
if err != nil {
return err
}
fmt.Println("Signer: ", signerNodeID.String())
fmt.Println("Signature: ", hex.EncodeToString(msg.Signature))
tags := &pb.NodeTagSet{}
err = proto.Unmarshal(msg.SerializedTag, tags)
if err != nil {
return err
}
nodeID, err := storj.NodeIDFromBytes(tags.NodeId)
if err != nil {
return err
}
fmt.Println("SignedAt: ", time.Unix(tags.SignedAt, 0).Format(time.RFC3339))
fmt.Println("NodeID: ", nodeID.String())
fmt.Println("Tags:")
for _, tag := range tags.Tags {
fmt.Printf(" %s=%s\n", tag.Name, string(tag.Value))
}
fmt.Println()
}
return nil
}
func parseTagPairs(tagPairs []string, allowCommaValues bool) ([]*pb.Tag, error) {
tags := make([]*pb.Tag, 0, len(tagPairs))
for _, tag := range tagPairs {
tag = strings.TrimSpace(tag)
if len(tag) == 0 {
continue
}
if !allowCommaValues && strings.ContainsRune(tag, ',') {
return nil, errs.New("multiple tags should be separated by spaces instead of commas, or specify --confirm to enable commas in tag values")
}
parts := strings.SplitN(tag, "=", 2)
if len(parts) != 2 {
return nil, errs.New("tags should be in KEY=VALUE format, but it was %s", tag)
}
tags = append(tags, &pb.Tag{
Name: parts[0],
Value: []byte(parts[1]),
})
}
return tags, nil
}
func main() {
process.ExecWithCustomOptions(rootCmd, process.ExecOptions{
LoadConfig: func(cmd *cobra.Command, vip *viper.Viper) error {
return nil
},
InitTracing: false,
LoggerFactory: func(logger *zap.Logger) *zap.Logger {
newLogger, level, err := process.NewLogger("tag-signer")
if err != nil {
panic(err)
}
level.SetLevel(zap.WarnLevel)
return newLogger
},
})
}

View File

@ -0,0 +1,99 @@
// Copyright (C) 2023 Storj Labs, Inc.
// See LICENSE for copying information.
package main
import (
"testing"
"github.com/stretchr/testify/require"
"storj.io/common/pb"
)
func Test_parseTagPairs(t *testing.T) {
tests := []struct {
name string
args []string
confirm bool
expected []*pb.Tag
expectedError string
}{
{
name: "comma separated tag pairs without confirm flag",
args: []string{"key1=value1,key2=value2"},
expectedError: "multiple tags should be separated by spaces instead of commas, or specify --confirm to enable commas in tag values",
},
{
name: "comma separated tag pairs with confirm flag",
args: []string{"key1=value1,key2=value2"},
confirm: true,
expected: []*pb.Tag{
{
Name: "key1",
Value: []byte("value1,key2=value2"),
},
},
},
{
name: "single tag pair",
args: []string{"key1=value1"},
confirm: true,
expected: []*pb.Tag{
{
Name: "key1",
Value: []byte("value1"),
},
},
},
{
name: "multiple tag pairs",
args: []string{"key1=value1", "key2=value2"},
confirm: true,
expected: []*pb.Tag{
{
Name: "key1",
Value: []byte("value1"),
},
{
Name: "key2",
Value: []byte("value2"),
},
},
},
{
name: "multiple tag pairs with with comma values and confirm flag",
args: []string{"key1=value1", "key2=value2,value3"},
confirm: true,
expected: []*pb.Tag{
{
Name: "key1",
Value: []byte("value1"),
},
{
Name: "key2",
Value: []byte("value2,value3"),
},
},
},
{
name: "multiple tag pairs with with comma values without confirm flag",
args: []string{"key1=value1", "key2=value2,value3"},
expectedError: "multiple tags should be separated by spaces instead of commas, or specify --confirm to enable commas in tag values",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got, err := parseTagPairs(tt.args, tt.confirm)
if tt.expectedError != "" {
require.Error(t, err)
require.Contains(t, err.Error(), tt.expectedError)
return
}
require.NoError(t, err)
require.Equal(t, tt.expected, got)
})
}
}

View File

@ -29,6 +29,8 @@ type accessPermissions struct {
notBefore *time.Time notBefore *time.Time
notAfter *time.Time notAfter *time.Time
maxObjectTTL *time.Duration
} }
func (ap *accessPermissions) Setup(params clingy.Parameters, prefixFlags bool) { func (ap *accessPermissions) Setup(params clingy.Parameters, prefixFlags bool) {
@ -65,6 +67,12 @@ func (ap *accessPermissions) Setup(params clingy.Parameters, prefixFlags bool) {
"Disallow access after this time (e.g. '+2h', 'now', '2020-01-02T15:04:05Z0700', 'none')", "Disallow access after this time (e.g. '+2h', 'now', '2020-01-02T15:04:05Z0700', 'none')",
nil, clingy.Transform(parseHumanDateNotAfter), clingy.Type("relative_date"), clingy.Optional).(*time.Time) nil, clingy.Transform(parseHumanDateNotAfter), clingy.Type("relative_date"), clingy.Optional).(*time.Time)
params.Break()
ap.maxObjectTTL = params.Flag("max-object-ttl",
"The object is automatically deleted after this period. (e.g. '1h30m', '24h', '720h')",
nil, clingy.Transform(time.ParseDuration), clingy.Type("period"), clingy.Optional).(*time.Duration)
if !prefixFlags { if !prefixFlags {
ap.prefixes = params.Arg("prefix", "Key prefix access will be restricted to", ap.prefixes = params.Arg("prefix", "Key prefix access will be restricted to",
clingy.Transform(ulloc.Parse), clingy.Transform(ulloc.Parse),
@ -93,6 +101,7 @@ func (ap *accessPermissions) Apply(access *uplink.Access) (*uplink.Access, error
AllowUpload: ap.AllowUpload(), AllowUpload: ap.AllowUpload(),
NotBefore: ap.NotBefore(), NotBefore: ap.NotBefore(),
NotAfter: ap.NotAfter(), NotAfter: ap.NotAfter(),
MaxObjectTTL: ap.MaxObjectTTL(),
} }
// if we aren't actually restricting anything, then we don't need to Share. // if we aren't actually restricting anything, then we don't need to Share.
@ -120,9 +129,10 @@ func defaulted[T any](val *T, def T) T {
return def return def
} }
func (ap *accessPermissions) NotBefore() time.Time { return defaulted(ap.notBefore, time.Time{}) } func (ap *accessPermissions) NotBefore() time.Time { return defaulted(ap.notBefore, time.Time{}) }
func (ap *accessPermissions) NotAfter() time.Time { return defaulted(ap.notAfter, time.Time{}) } func (ap *accessPermissions) NotAfter() time.Time { return defaulted(ap.notAfter, time.Time{}) }
func (ap *accessPermissions) AllowDelete() bool { return !defaulted(ap.disallowDeletes, ap.readonly) } func (ap *accessPermissions) AllowDelete() bool { return !defaulted(ap.disallowDeletes, ap.readonly) }
func (ap *accessPermissions) AllowList() bool { return !defaulted(ap.disallowLists, ap.writeonly) } func (ap *accessPermissions) AllowList() bool { return !defaulted(ap.disallowLists, ap.writeonly) }
func (ap *accessPermissions) AllowDownload() bool { return !defaulted(ap.disallowReads, ap.writeonly) } func (ap *accessPermissions) AllowDownload() bool { return !defaulted(ap.disallowReads, ap.writeonly) }
func (ap *accessPermissions) AllowUpload() bool { return !defaulted(ap.disallowWrites, ap.readonly) } func (ap *accessPermissions) AllowUpload() bool { return !defaulted(ap.disallowWrites, ap.readonly) }
func (ap *accessPermissions) MaxObjectTTL() *time.Duration { return ap.maxObjectTTL }

View File

@ -8,7 +8,6 @@ import (
"fmt" "fmt"
"github.com/zeebo/clingy" "github.com/zeebo/clingy"
"github.com/zeebo/errs"
"storj.io/storj/cmd/uplink/ulext" "storj.io/storj/cmd/uplink/ulext"
) )
@ -33,7 +32,7 @@ func (c *cmdAccessUse) Execute(ctx context.Context) error {
return err return err
} }
if _, ok := accesses[c.access]; !ok { if _, ok := accesses[c.access]; !ok {
return errs.New("unknown access: %q", c.access) return fmt.Errorf("ERROR: access %q does not exist. Use 'uplink access list' to see existing accesses", c.access)
} }
if err := c.ex.SaveAccessInfo(c.access, accesses); err != nil { if err := c.ex.SaveAccessInfo(c.access, accesses); err != nil {
return err return err

View File

@ -15,7 +15,6 @@ import (
"sync" "sync"
"time" "time"
"github.com/VividCortex/ewma"
"github.com/vbauerster/mpb/v8" "github.com/vbauerster/mpb/v8"
"github.com/vbauerster/mpb/v8/decor" "github.com/vbauerster/mpb/v8/decor"
"github.com/zeebo/clingy" "github.com/zeebo/clingy"
@ -85,8 +84,7 @@ func (c *cmdCp) Setup(params clingy.Parameters) {
).(bool) ).(bool)
c.byteRange = params.Flag("range", "Downloads the specified range bytes of an object. For more information about the HTTP Range header, see https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35", "").(string) c.byteRange = params.Flag("range", "Downloads the specified range bytes of an object. For more information about the HTTP Range header, see https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35", "").(string)
parallelism := params.Flag("parallelism", "Controls how many parallel chunks to upload/download from a file", nil, c.parallelism = params.Flag("parallelism", "Controls how many parallel parts to upload/download from a file", 1,
clingy.Optional,
clingy.Short('p'), clingy.Short('p'),
clingy.Transform(strconv.Atoi), clingy.Transform(strconv.Atoi),
clingy.Transform(func(n int) (int, error) { clingy.Transform(func(n int) (int, error) {
@ -95,8 +93,8 @@ func (c *cmdCp) Setup(params clingy.Parameters) {
} }
return n, nil return n, nil
}), }),
).(*int) ).(int)
c.parallelismChunkSize = params.Flag("parallelism-chunk-size", "Set the size of the chunks for parallelism, 0 means automatic adjustment", memory.Size(0), c.parallelismChunkSize = params.Flag("parallelism-chunk-size", "Set the size of the parts for parallelism, 0 means automatic adjustment", memory.Size(0),
clingy.Transform(memory.ParseString), clingy.Transform(memory.ParseString),
clingy.Transform(func(n int64) (memory.Size, error) { clingy.Transform(func(n int64) (memory.Size, error) {
if n < 0 { if n < 0 {
@ -107,17 +105,16 @@ func (c *cmdCp) Setup(params clingy.Parameters) {
).(memory.Size) ).(memory.Size)
c.uploadConfig = testuplink.DefaultConcurrentSegmentUploadsConfig() c.uploadConfig = testuplink.DefaultConcurrentSegmentUploadsConfig()
maxConcurrent := params.Flag( c.uploadConfig.SchedulerOptions.MaximumConcurrent = params.Flag(
"maximum-concurrent-pieces", "maximum-concurrent-pieces",
"Maximum concurrent pieces to upload at once per transfer", "Maximum concurrent pieces to upload at once per part",
nil, c.uploadConfig.SchedulerOptions.MaximumConcurrent,
clingy.Optional,
clingy.Transform(strconv.Atoi), clingy.Transform(strconv.Atoi),
clingy.Advanced, clingy.Advanced,
).(*int) ).(int)
c.uploadConfig.SchedulerOptions.MaximumConcurrentHandles = params.Flag( c.uploadConfig.SchedulerOptions.MaximumConcurrentHandles = params.Flag(
"maximum-concurrent-segments", "maximum-concurrent-segments",
"Maximum concurrent segments to upload at once per transfer", "Maximum concurrent segments to upload at once per part",
c.uploadConfig.SchedulerOptions.MaximumConcurrentHandles, c.uploadConfig.SchedulerOptions.MaximumConcurrentHandles,
clingy.Transform(strconv.Atoi), clingy.Transform(strconv.Atoi),
clingy.Advanced, clingy.Advanced,
@ -133,28 +130,6 @@ func (c *cmdCp) Setup(params clingy.Parameters) {
clingy.Advanced, clingy.Advanced,
).(string) ).(string)
{ // handle backwards compatibility around parallelism and maximum concurrent pieces
addr := func(x int) *int { return &x }
switch {
// if neither are actively set, use defaults
case parallelism == nil && maxConcurrent == nil:
parallelism = addr(1)
maxConcurrent = addr(c.uploadConfig.SchedulerOptions.MaximumConcurrent)
// if parallelism is not set, use a value based on maxConcurrent
case parallelism == nil:
parallelism = addr((*maxConcurrent + 99) / 100)
// if maxConcurrent is not set, use a value based on parallelism
case maxConcurrent == nil:
maxConcurrent = addr(100 * *parallelism)
}
c.uploadConfig.SchedulerOptions.MaximumConcurrent = *maxConcurrent
c.parallelism = *parallelism
}
c.inmemoryEC = params.Flag("inmemory-erasure-coding", "Keep erasure-coded pieces in-memory instead of writing them on the disk during upload", false, c.inmemoryEC = params.Flag("inmemory-erasure-coding", "Keep erasure-coded pieces in-memory instead of writing them on the disk during upload", false,
clingy.Transform(strconv.ParseBool), clingy.Transform(strconv.ParseBool),
clingy.Boolean, clingy.Boolean,
@ -194,9 +169,10 @@ func (c *cmdCp) Execute(ctx context.Context) error {
fs, err := c.ex.OpenFilesystem(ctx, c.access, fs, err := c.ex.OpenFilesystem(ctx, c.access,
ulext.ConcurrentSegmentUploadsConfig(c.uploadConfig), ulext.ConcurrentSegmentUploadsConfig(c.uploadConfig),
ulext.ConnectionPoolOptions(rpcpool.Options{ ulext.ConnectionPoolOptions(rpcpool.Options{
// Add a bit more capacity for connections to the satellite // Allow at least as many connections as the maximum concurrent pieces per
Capacity: c.uploadConfig.SchedulerOptions.MaximumConcurrent + 5, // parallel part per transfer, plus a few extra for the satellite.
KeyCapacity: 5, Capacity: c.transfers*c.parallelism*c.uploadConfig.SchedulerOptions.MaximumConcurrent + 5,
KeyCapacity: 2,
IdleExpiration: 2 * time.Minute, IdleExpiration: 2 * time.Minute,
})) }))
if err != nil { if err != nil {
@ -419,17 +395,6 @@ func (c *cmdCp) copyFile(ctx context.Context, fs ulfs.Filesystem, source, dest u
} }
defer func() { _ = mwh.Abort(ctx) }() defer func() { _ = mwh.Abort(ctx) }()
// if we're uploading, do a single part of maximum size
if dest.Remote() {
return errs.Wrap(c.singleCopy(
ctx,
source, dest,
mrh, mwh,
offset, length,
bar,
))
}
partSize, err := c.calculatePartSize(mrh.Length(), c.parallelismChunkSize.Int64()) partSize, err := c.calculatePartSize(mrh.Length(), c.parallelismChunkSize.Int64())
if err != nil { if err != nil {
return err return err
@ -448,13 +413,15 @@ func (c *cmdCp) copyFile(ctx context.Context, fs ulfs.Filesystem, source, dest u
// calculatePartSize returns the needed part size in order to upload the file with size of 'length'. // calculatePartSize returns the needed part size in order to upload the file with size of 'length'.
// It hereby respects if the client requests/prefers a certain size and only increases if needed. // It hereby respects if the client requests/prefers a certain size and only increases if needed.
func (c *cmdCp) calculatePartSize(length, preferredSize int64) (requiredSize int64, err error) { func (c *cmdCp) calculatePartSize(length, preferredSize int64) (requiredSize int64, err error) {
segC := (length / maxPartCount / (memory.MiB * 64).Int64()) + 1 segC := (length / maxPartCount / memory.GiB.Int64()) + 1
requiredSize = segC * (memory.MiB * 64).Int64() requiredSize = segC * memory.GiB.Int64()
switch { switch {
case preferredSize == 0: case preferredSize == 0:
return requiredSize, nil return requiredSize, nil
case requiredSize <= preferredSize: case requiredSize <= preferredSize:
return preferredSize, nil return preferredSize, nil
case length < 0: // let the user pick their size if we don't have a length to know better
return preferredSize, nil
default: default:
return 0, errs.New(fmt.Sprintf("the specified chunk size %s is too small, requires %s or larger", return 0, errs.New(fmt.Sprintf("the specified chunk size %s is too small, requires %s or larger",
memory.FormatBytes(preferredSize), memory.FormatBytes(requiredSize))) memory.FormatBytes(preferredSize), memory.FormatBytes(requiredSize)))
@ -535,8 +502,8 @@ func (c *cmdCp) parallelCopy(
} }
var readBufs *ulfs.BytesPool var readBufs *ulfs.BytesPool
if p > 1 && chunkSize > 0 && (source.Std() || dest.Std()) { if p > 1 && chunkSize > 0 && source.Std() {
// Create the read buffer pool only for uploads from stdin and downloads to stdout with parallelism > 1. // Create the read buffer pool only for uploads from stdin with parallelism > 1.
readBufs = ulfs.NewBytesPool(int(chunkSize)) readBufs = ulfs.NewBytesPool(int(chunkSize))
} }
@ -557,6 +524,14 @@ func (c *cmdCp) parallelCopy(
break break
} }
if i == 0 && bar != nil {
info, err := src.Info(ctx)
if err == nil {
bar.SetTotal(info.ContentLength, false)
bar.EnableTriggerComplete()
}
}
wh, err := dst.NextPart(ctx, chunk) wh, err := dst.NextPart(ctx, chunk)
if err != nil { if err != nil {
_ = rh.Close() _ = rh.Close()
@ -578,12 +553,8 @@ func (c *cmdCp) parallelCopy(
var w io.Writer = wh var w io.Writer = wh
if bar != nil { if bar != nil {
bar.SetTotal(rh.Info().ContentLength, false)
bar.EnableTriggerComplete()
pw := bar.ProxyWriter(w) pw := bar.ProxyWriter(w)
defer func() { defer func() { _ = pw.Close() }()
_ = pw.Close()
}()
w = pw w = pw
} }
@ -619,65 +590,9 @@ func (c *cmdCp) parallelCopy(
return errs.Wrap(combineErrs(es)) return errs.Wrap(combineErrs(es))
} }
func (c *cmdCp) singleCopy(
ctx context.Context,
source, dest ulloc.Location,
src ulfs.MultiReadHandle,
dst ulfs.MultiWriteHandle,
offset, length int64,
bar *mpb.Bar) error {
if offset != 0 {
if err := src.SetOffset(offset); err != nil {
return err
}
}
ctx, cancel := context.WithCancel(ctx)
defer cancel()
rh, err := src.NextPart(ctx, length)
if err != nil {
return errs.Wrap(err)
}
defer func() { _ = rh.Close() }()
wh, err := dst.NextPart(ctx, length)
if err != nil {
return errs.Wrap(err)
}
defer func() { _ = wh.Abort() }()
var w io.Writer = wh
if bar != nil {
bar.SetTotal(rh.Info().ContentLength, false)
bar.EnableTriggerComplete()
pw := bar.ProxyWriter(w)
defer func() { _ = pw.Close() }()
w = pw
}
if _, err := sync2.Copy(ctx, w, rh); err != nil {
return errs.Wrap(err)
}
if err := wh.Commit(); err != nil {
return errs.Wrap(err)
}
if err := dst.Commit(ctx); err != nil {
return errs.Wrap(err)
}
return nil
}
func newProgressBar(progress *mpb.Progress, name string, which, total int) *mpb.Bar { func newProgressBar(progress *mpb.Progress, name string, which, total int) *mpb.Bar {
const counterFmt = " % .2f / % .2f" const counterFmt = " % .2f / % .2f"
const percentageFmt = "%.2f " const percentageFmt = "%.2f "
const speedFmt = "% .2f"
movingAverage := ewma.NewMovingAverage()
prepends := []decor.Decorator{decor.Name(name + " ")} prepends := []decor.Decorator{decor.Name(name + " ")}
if total > 1 { if total > 1 {
@ -687,7 +602,6 @@ func newProgressBar(progress *mpb.Progress, name string, which, total int) *mpb.
appends := []decor.Decorator{ appends := []decor.Decorator{
decor.NewPercentage(percentageFmt), decor.NewPercentage(percentageFmt),
decor.MovingAverageSpeed(decor.SizeB1024(1024), speedFmt, movingAverage),
} }
return progress.AddBar(0, return progress.AddBar(0,

View File

@ -99,46 +99,51 @@ func TestCpDownload(t *testing.T) {
func TestCpPartSize(t *testing.T) { func TestCpPartSize(t *testing.T) {
c := newCmdCp(nil) c := newCmdCp(nil)
// 1GiB file, should return 64MiB // 10 GiB file, should return 1 GiB
partSize, err := c.calculatePartSize(memory.GiB.Int64(), c.parallelismChunkSize.Int64()) partSize, err := c.calculatePartSize(10*memory.GiB.Int64(), c.parallelismChunkSize.Int64())
require.NoError(t, err) require.NoError(t, err)
require.EqualValues(t, memory.MiB*64, partSize) require.EqualValues(t, 1*memory.GiB, partSize)
// 640 GB file, should return 64MiB. // 10000 GB file, should return 1 GiB.
partSize, err = c.calculatePartSize(memory.GB.Int64()*640, c.parallelismChunkSize.Int64()) partSize, err = c.calculatePartSize(10000*memory.GB.Int64(), c.parallelismChunkSize.Int64())
require.NoError(t, err) require.NoError(t, err)
require.EqualValues(t, memory.MiB*64, partSize) require.EqualValues(t, 1*memory.GiB, partSize)
// 640GiB file, should return 128MiB. // 10000 GiB file, should return 2 GiB.
partSize, err = c.calculatePartSize(memory.GiB.Int64()*640, c.parallelismChunkSize.Int64()) partSize, err = c.calculatePartSize(10000*memory.GiB.Int64(), c.parallelismChunkSize.Int64())
require.NoError(t, err) require.NoError(t, err)
require.EqualValues(t, memory.MiB*128, partSize) require.EqualValues(t, 2*memory.GiB, partSize)
// 1TiB file, should return 128MiB. // 10 TiB file, should return 2 GiB.
partSize, err = c.calculatePartSize(memory.TiB.Int64(), c.parallelismChunkSize.Int64()) partSize, err = c.calculatePartSize(10*memory.TiB.Int64(), c.parallelismChunkSize.Int64())
require.NoError(t, err) require.NoError(t, err)
require.EqualValues(t, memory.MiB*128, partSize) require.EqualValues(t, 2*memory.GiB, partSize)
// 1.3TiB file, should return 192MiB. // 20001 GiB file, should return 3 GiB.
partSize, err = c.calculatePartSize(memory.GiB.Int64()*1300, c.parallelismChunkSize.Int64()) partSize, err = c.calculatePartSize(20001*memory.GiB.Int64(), c.parallelismChunkSize.Int64())
require.NoError(t, err) require.NoError(t, err)
require.EqualValues(t, memory.MiB*192, partSize) require.EqualValues(t, 3*memory.GiB, partSize)
// should return 1GiB as requested. // should return 1GiB as requested.
partSize, err = c.calculatePartSize(memory.GiB.Int64()*1300, memory.GiB.Int64()) partSize, err = c.calculatePartSize(memory.GiB.Int64()*1300, memory.GiB.Int64())
require.NoError(t, err) require.NoError(t, err)
require.EqualValues(t, memory.GiB, partSize) require.EqualValues(t, memory.GiB, partSize)
// should return 192 MiB and error, since preferred is too low. // should return 1 GiB and error, since preferred is too low.
partSize, err = c.calculatePartSize(memory.GiB.Int64()*1300, memory.MiB.Int64()) partSize, err = c.calculatePartSize(1300*memory.GiB.Int64(), memory.MiB.Int64())
require.Error(t, err) require.Error(t, err)
require.Equal(t, "the specified chunk size 1.0 MiB is too small, requires 192.0 MiB or larger", err.Error()) require.Equal(t, "the specified chunk size 1.0 MiB is too small, requires 1.0 GiB or larger", err.Error())
require.Zero(t, partSize) require.Zero(t, partSize)
// negative length should return 64MiB part size // negative length should return asked for amount
partSize, err = c.calculatePartSize(-1, c.parallelismChunkSize.Int64()) partSize, err = c.calculatePartSize(-1, 1*memory.GiB.Int64())
require.NoError(t, err) require.NoError(t, err)
require.EqualValues(t, memory.MiB*64, partSize) require.EqualValues(t, 1*memory.GiB, partSize)
// negative length should return specified amount
partSize, err = c.calculatePartSize(-1, 100)
require.NoError(t, err)
require.EqualValues(t, 100, partSize)
} }
func TestCpUpload(t *testing.T) { func TestCpUpload(t *testing.T) {

View File

@ -104,15 +104,16 @@ func (c *cmdShare) Execute(ctx context.Context) error {
fmt.Fprintf(clingy.Stdout(ctx), "Sharing access to satellite %s\n", access.SatelliteAddress()) fmt.Fprintf(clingy.Stdout(ctx), "Sharing access to satellite %s\n", access.SatelliteAddress())
fmt.Fprintf(clingy.Stdout(ctx), "=========== ACCESS RESTRICTIONS ==========================================================\n") fmt.Fprintf(clingy.Stdout(ctx), "=========== ACCESS RESTRICTIONS ==========================================================\n")
fmt.Fprintf(clingy.Stdout(ctx), "Download : %s\n", formatPermission(c.ap.AllowDownload())) fmt.Fprintf(clingy.Stdout(ctx), "Download : %s\n", formatPermission(c.ap.AllowDownload()))
fmt.Fprintf(clingy.Stdout(ctx), "Upload : %s\n", formatPermission(c.ap.AllowUpload())) fmt.Fprintf(clingy.Stdout(ctx), "Upload : %s\n", formatPermission(c.ap.AllowUpload()))
fmt.Fprintf(clingy.Stdout(ctx), "Lists : %s\n", formatPermission(c.ap.AllowList())) fmt.Fprintf(clingy.Stdout(ctx), "Lists : %s\n", formatPermission(c.ap.AllowList()))
fmt.Fprintf(clingy.Stdout(ctx), "Deletes : %s\n", formatPermission(c.ap.AllowDelete())) fmt.Fprintf(clingy.Stdout(ctx), "Deletes : %s\n", formatPermission(c.ap.AllowDelete()))
fmt.Fprintf(clingy.Stdout(ctx), "NotBefore : %s\n", formatTimeRestriction(c.ap.NotBefore())) fmt.Fprintf(clingy.Stdout(ctx), "NotBefore : %s\n", formatTimeRestriction(c.ap.NotBefore()))
fmt.Fprintf(clingy.Stdout(ctx), "NotAfter : %s\n", formatTimeRestriction(c.ap.NotAfter())) fmt.Fprintf(clingy.Stdout(ctx), "NotAfter : %s\n", formatTimeRestriction(c.ap.NotAfter()))
fmt.Fprintf(clingy.Stdout(ctx), "Paths : %s\n", formatPaths(c.ap.prefixes)) fmt.Fprintf(clingy.Stdout(ctx), "MaxObjectTTL : %s\n", formatDuration(c.ap.maxObjectTTL))
fmt.Fprintf(clingy.Stdout(ctx), "Paths : %s\n", formatPaths(c.ap.prefixes))
fmt.Fprintf(clingy.Stdout(ctx), "=========== SERIALIZED ACCESS WITH THE ABOVE RESTRICTIONS TO SHARE WITH OTHERS ===========\n") fmt.Fprintf(clingy.Stdout(ctx), "=========== SERIALIZED ACCESS WITH THE ABOVE RESTRICTIONS TO SHARE WITH OTHERS ===========\n")
fmt.Fprintf(clingy.Stdout(ctx), "Access : %s\n", newAccessData) fmt.Fprintf(clingy.Stdout(ctx), "Access : %s\n", newAccessData)
if c.register { if c.register {
credentials, err := RegisterAccess(ctx, access, c.authService, c.public, c.caCert) credentials, err := RegisterAccess(ctx, access, c.authService, c.public, c.caCert)
@ -182,6 +183,13 @@ func formatTimeRestriction(t time.Time) string {
return formatTime(true, t) return formatTime(true, t)
} }
func formatDuration(d *time.Duration) string {
if d == nil {
return "Not set"
}
return d.String()
}
func formatPaths(sharePrefixes []uplink.SharePrefix) string { func formatPaths(sharePrefixes []uplink.SharePrefix) string {
if len(sharePrefixes) == 0 { if len(sharePrefixes) == 0 {
return "WARNING! The entire project is shared!" return "WARNING! The entire project is shared!"

View File

@ -33,15 +33,16 @@ func TestShare(t *testing.T) {
state.Succeed(t, "share", "sj://some/prefix").RequireStdoutGlob(t, ` state.Succeed(t, "share", "sj://some/prefix").RequireStdoutGlob(t, `
Sharing access to satellite * Sharing access to satellite *
=========== ACCESS RESTRICTIONS ========================================================== =========== ACCESS RESTRICTIONS ==========================================================
Download : Allowed Download : Allowed
Upload : Disallowed Upload : Disallowed
Lists : Allowed Lists : Allowed
Deletes : Disallowed Deletes : Disallowed
NotBefore : No restriction NotBefore : No restriction
NotAfter : No restriction NotAfter : No restriction
Paths : sj://some/prefix MaxObjectTTL : Not set
Paths : sj://some/prefix
=========== SERIALIZED ACCESS WITH THE ABOVE RESTRICTIONS TO SHARE WITH OTHERS =========== =========== SERIALIZED ACCESS WITH THE ABOVE RESTRICTIONS TO SHARE WITH OTHERS ===========
Access : * Access : *
`) `)
}) })
@ -51,15 +52,16 @@ func TestShare(t *testing.T) {
state.Succeed(t, "share", "--readonly", "sj://some/prefix").RequireStdoutGlob(t, ` state.Succeed(t, "share", "--readonly", "sj://some/prefix").RequireStdoutGlob(t, `
Sharing access to satellite * Sharing access to satellite *
=========== ACCESS RESTRICTIONS ========================================================== =========== ACCESS RESTRICTIONS ==========================================================
Download : Allowed Download : Allowed
Upload : Disallowed Upload : Disallowed
Lists : Allowed Lists : Allowed
Deletes : Disallowed Deletes : Disallowed
NotBefore : No restriction NotBefore : No restriction
NotAfter : No restriction NotAfter : No restriction
Paths : sj://some/prefix MaxObjectTTL : Not set
Paths : sj://some/prefix
=========== SERIALIZED ACCESS WITH THE ABOVE RESTRICTIONS TO SHARE WITH OTHERS =========== =========== SERIALIZED ACCESS WITH THE ABOVE RESTRICTIONS TO SHARE WITH OTHERS ===========
Access : * Access : *
`) `)
}) })
@ -69,15 +71,16 @@ func TestShare(t *testing.T) {
state.Succeed(t, "share", "--disallow-lists", "sj://some/prefix").RequireStdoutGlob(t, ` state.Succeed(t, "share", "--disallow-lists", "sj://some/prefix").RequireStdoutGlob(t, `
Sharing access to satellite * Sharing access to satellite *
=========== ACCESS RESTRICTIONS ========================================================== =========== ACCESS RESTRICTIONS ==========================================================
Download : Allowed Download : Allowed
Upload : Disallowed Upload : Disallowed
Lists : Disallowed Lists : Disallowed
Deletes : Disallowed Deletes : Disallowed
NotBefore : No restriction NotBefore : No restriction
NotAfter : No restriction NotAfter : No restriction
Paths : sj://some/prefix MaxObjectTTL : Not set
Paths : sj://some/prefix
=========== SERIALIZED ACCESS WITH THE ABOVE RESTRICTIONS TO SHARE WITH OTHERS =========== =========== SERIALIZED ACCESS WITH THE ABOVE RESTRICTIONS TO SHARE WITH OTHERS ===========
Access : * Access : *
`) `)
}) })
@ -87,15 +90,16 @@ func TestShare(t *testing.T) {
state.Succeed(t, "share", "--disallow-reads", "sj://some/prefix").RequireStdoutGlob(t, ` state.Succeed(t, "share", "--disallow-reads", "sj://some/prefix").RequireStdoutGlob(t, `
Sharing access to satellite * Sharing access to satellite *
=========== ACCESS RESTRICTIONS ========================================================== =========== ACCESS RESTRICTIONS ==========================================================
Download : Disallowed Download : Disallowed
Upload : Disallowed Upload : Disallowed
Lists : Allowed Lists : Allowed
Deletes : Disallowed Deletes : Disallowed
NotBefore : No restriction NotBefore : No restriction
NotAfter : No restriction NotAfter : No restriction
Paths : sj://some/prefix MaxObjectTTL : Not set
Paths : sj://some/prefix
=========== SERIALIZED ACCESS WITH THE ABOVE RESTRICTIONS TO SHARE WITH OTHERS =========== =========== SERIALIZED ACCESS WITH THE ABOVE RESTRICTIONS TO SHARE WITH OTHERS ===========
Access : * Access : *
`) `)
}) })
@ -116,33 +120,54 @@ func TestShare(t *testing.T) {
state.Succeed(t, "share", "--public", "--not-after=none", "sj://some/prefix").RequireStdoutGlob(t, ` state.Succeed(t, "share", "--public", "--not-after=none", "sj://some/prefix").RequireStdoutGlob(t, `
Sharing access to satellite * Sharing access to satellite *
=========== ACCESS RESTRICTIONS ========================================================== =========== ACCESS RESTRICTIONS ==========================================================
Download : Allowed Download : Allowed
Upload : Disallowed Upload : Disallowed
Lists : Allowed Lists : Allowed
Deletes : Disallowed Deletes : Disallowed
NotBefore : No restriction NotBefore : No restriction
NotAfter : No restriction NotAfter : No restriction
Paths : sj://some/prefix MaxObjectTTL : Not set
Paths : sj://some/prefix
=========== SERIALIZED ACCESS WITH THE ABOVE RESTRICTIONS TO SHARE WITH OTHERS =========== =========== SERIALIZED ACCESS WITH THE ABOVE RESTRICTIONS TO SHARE WITH OTHERS ===========
Access : * Access : *
`) `)
}) })
t.Run("share access with --not-after time restriction parameter", func(t *testing.T) { t.Run("share access with --not-after", func(t *testing.T) {
state := ultest.Setup(commands) state := ultest.Setup(commands)
state.Succeed(t, "share", "--not-after", "2022-01-01T15:01:01-01:00", "sj://some/prefix").RequireStdoutGlob(t, ` state.Succeed(t, "share", "--not-after", "2022-01-01T15:01:01-01:00", "sj://some/prefix").RequireStdoutGlob(t, `
Sharing access to satellite * Sharing access to satellite *
=========== ACCESS RESTRICTIONS ========================================================== =========== ACCESS RESTRICTIONS ==========================================================
Download : Allowed Download : Allowed
Upload : Disallowed Upload : Disallowed
Lists : Allowed Lists : Allowed
Deletes : Disallowed Deletes : Disallowed
NotBefore : No restriction NotBefore : No restriction
NotAfter : 2022-01-01 16:01:01 NotAfter : 2022-01-01 16:01:01
Paths : sj://some/prefix MaxObjectTTL : Not set
Paths : sj://some/prefix
=========== SERIALIZED ACCESS WITH THE ABOVE RESTRICTIONS TO SHARE WITH OTHERS =========== =========== SERIALIZED ACCESS WITH THE ABOVE RESTRICTIONS TO SHARE WITH OTHERS ===========
Access : * Access : *
`)
})
t.Run("share access with --max-object-ttl", func(t *testing.T) {
state := ultest.Setup(commands)
state.Succeed(t, "share", "--max-object-ttl", "720h", "--readonly=false", "sj://some/prefix").RequireStdoutGlob(t, `
Sharing access to satellite *
=========== ACCESS RESTRICTIONS ==========================================================
Download : Allowed
Upload : Allowed
Lists : Allowed
Deletes : Allowed
NotBefore : No restriction
NotAfter : No restriction
MaxObjectTTL : 720h0m0s
Paths : sj://some/prefix
=========== SERIALIZED ACCESS WITH THE ABOVE RESTRICTIONS TO SHARE WITH OTHERS ===========
Access : *
`) `)
}) })
@ -184,15 +209,16 @@ func TestShare(t *testing.T) {
expected := ` expected := `
Sharing access to satellite * Sharing access to satellite *
=========== ACCESS RESTRICTIONS ========================================================== =========== ACCESS RESTRICTIONS ==========================================================
Download : Allowed Download : Allowed
Upload : Disallowed Upload : Disallowed
Lists : Allowed Lists : Allowed
Deletes : Disallowed Deletes : Disallowed
NotBefore : No restriction NotBefore : No restriction
NotAfter : No restriction NotAfter : No restriction
Paths : sj://some/prefix MaxObjectTTL : Not set
Paths : sj://some/prefix
=========== SERIALIZED ACCESS WITH THE ABOVE RESTRICTIONS TO SHARE WITH OTHERS =========== =========== SERIALIZED ACCESS WITH THE ABOVE RESTRICTIONS TO SHARE WITH OTHERS ===========
Access : * Access : *
========== GATEWAY CREDENTIALS =========================================================== ========== GATEWAY CREDENTIALS ===========================================================
Access Key ID: accesskeyid Access Key ID: accesskeyid
Secret Key : secretkey Secret Key : secretkey

View File

@ -21,6 +21,8 @@ import (
"github.com/jtolio/eventkit" "github.com/jtolio/eventkit"
"github.com/spacemonkeygo/monkit/v3" "github.com/spacemonkeygo/monkit/v3"
"github.com/spacemonkeygo/monkit/v3/collect"
"github.com/spacemonkeygo/monkit/v3/present"
"github.com/zeebo/clingy" "github.com/zeebo/clingy"
"github.com/zeebo/errs" "github.com/zeebo/errs"
"go.uber.org/zap" "go.uber.org/zap"
@ -28,6 +30,7 @@ import (
"storj.io/common/experiment" "storj.io/common/experiment"
"storj.io/common/rpc/rpctracing" "storj.io/common/rpc/rpctracing"
"storj.io/common/sync2/mpscqueue"
"storj.io/common/tracing" "storj.io/common/tracing"
jaeger "storj.io/monkit-jaeger" jaeger "storj.io/monkit-jaeger"
"storj.io/private/version" "storj.io/private/version"
@ -68,8 +71,9 @@ type external struct {
} }
debug struct { debug struct {
pprofFile string pprofFile string
traceFile string traceFile string
monkitTraceFile string
} }
events struct { events struct {
@ -124,7 +128,7 @@ func (ex *external) Setup(f clingy.Flags) {
).(string) ).(string)
ex.tracing.tags = f.Flag( ex.tracing.tags = f.Flag(
"trace-tags", "coma separated k=v pairs to be added to distributed traces", map[string]string{}, "trace-tags", "comma separated k=v pairs to be added to distributed traces", map[string]string{},
clingy.Advanced, clingy.Advanced,
clingy.Transform(func(val string) (map[string]string, error) { clingy.Transform(func(val string) (map[string]string, error) {
res := map[string]string{} res := map[string]string{}
@ -151,6 +155,11 @@ func (ex *external) Setup(f clingy.Flags) {
clingy.Advanced, clingy.Advanced,
).(string) ).(string)
ex.debug.monkitTraceFile = f.Flag(
"debug-monkit-trace", "File to collect Monkit trace data. Understands file extensions .json and .svg", "",
clingy.Advanced,
).(string)
ex.analytics = f.Flag( ex.analytics = f.Flag(
"analytics", "Whether to send usage information to Storj", nil, "analytics", "Whether to send usage information to Storj", nil,
clingy.Transform(strconv.ParseBool), clingy.Optional, clingy.Boolean, clingy.Transform(strconv.ParseBool), clingy.Optional, clingy.Boolean,
@ -371,8 +380,60 @@ func (ex *external) Wrap(ctx context.Context, cmd clingy.Command) (err error) {
eventkit.DefaultRegistry.Scope("init").Event("init") eventkit.DefaultRegistry.Scope("init").Event("init")
} }
defer mon.Task()(&ctx)(&err) var workErr error
return cmd.Execute(ctx) work := func(ctx context.Context) {
defer mon.Task()(&ctx)(&err)
workErr = cmd.Execute(ctx)
}
var formatter func(io.Writer, []*collect.FinishedSpan) error
switch {
default:
work(ctx)
return workErr
case strings.HasSuffix(strings.ToLower(ex.debug.monkitTraceFile), ".svg"):
formatter = present.SpansToSVG
case strings.HasSuffix(strings.ToLower(ex.debug.monkitTraceFile), ".json"):
formatter = present.SpansToJSON
}
spans := mpscqueue.New[collect.FinishedSpan]()
collector := func(s *monkit.Span, err error, panicked bool, finish time.Time) {
spans.Enqueue(collect.FinishedSpan{
Span: s,
Err: err,
Panicked: panicked,
Finish: finish,
})
}
defer collect.ObserveAllTraces(monkit.Default, spanCollectorFunc(collector))()
work(ctx)
fh, err := os.Create(ex.debug.monkitTraceFile)
if err != nil {
return errs.Combine(workErr, err)
}
var spanSlice []*collect.FinishedSpan
for {
next, ok := spans.Dequeue()
if !ok {
break
}
spanSlice = append(spanSlice, &next)
}
err = formatter(fh, spanSlice)
return errs.Combine(workErr, err, fh.Close())
}
type spanCollectorFunc func(*monkit.Span, error, bool, time.Time)
func (f spanCollectorFunc) Start(*monkit.Span) {}
func (f spanCollectorFunc) Finish(s *monkit.Span, err error, panicked bool, finish time.Time) {
f(s, err, panicked, finish)
} }
func tracked(ctx context.Context, cb func(context.Context)) (done func()) { func tracked(ctx context.Context, cb func(context.Context)) (done func()) {

View File

@ -43,6 +43,16 @@ func (ex *external) OpenProject(ctx context.Context, accessName string, options
UserAgent: uplinkCLIUserAgent, UserAgent: uplinkCLIUserAgent,
} }
userAgents, err := ex.Dynamic("client.user-agent")
if err != nil {
return nil, err
}
if len(userAgents) > 0 {
if ua := userAgents[len(userAgents)-1]; ua != "" {
config.UserAgent = ua
}
}
if opts.ConnectionPoolOptions != (rpcpool.Options{}) { if opts.ConnectionPoolOptions != (rpcpool.Options{}) {
if err := transport.SetConnectionPool(ctx, &config, rpcpool.New(opts.ConnectionPoolOptions)); err != nil { if err := transport.SetConnectionPool(ctx, &config, rpcpool.New(opts.ConnectionPoolOptions)); err != nil {
return nil, err return nil, err

View File

@ -0,0 +1,273 @@
# Node and operator certification
## Abstract
This is a proposal for a small feature and service that allows for nodes and
operators to have signed tags of certain kinds for use in project-specific or
Satellite-specific node selection.
## Background/context
We have a couple of ongoing needs:
* 1099 KYC
* Private storage node networks
* SOC2/HIPAA/etc node certification
* Voting and operator signaling
### 1099 KYC
The United States has a rule that if node operators earn more than $600/year,
we need to file a 1099 for each of them. Our current way of dealing with this
is manual and time consuming, and so it would be nice to automate it.
Ultimately, we should be able to automatically:
1) keep track of which nodes are run by operators under or over the $600
threshold.
2) keep track of if an automated KYC service has signed off that we have the
necessary information to file a 1099.
3) automatically suspend nodes that have earned more than $600 but have not
provided legally required information.
### Private storage node networks
We have seen growing interest from customers that want to bring their own
hard drives, or be extremely choosy about the nodes they are willing to work
with. The current way we are solving this is spinning up private Satellites
that are configured to only work with the nodes those customers provide, but
it would be better if we didn't have to start custom Satellites for this.
Instead, it would be nice to have a per-project configuration on an existing
Satellite that allowed that project to specify a specific subset of verified
or validated nodes, e.g., Project A should be able to say only nodes from
node providers B and C should be selected. Symmetrically, Nodes from providers
B and C may only want to accept data from certain projects, like Project A.
When nodes from providers B and C are added to the Satellite, they should be
able to provide a provider-specific signature, and requirements about
customer-specific requirements, if any.
### SOC2/HIPAA/etc node certification
This is actually just a slightly different shape of the private storage node
network problem, but instead of being provider-specific, it is property
specific.
Perhaps Project D has a compliance requirement. They can only store data
on nodes that meet specific requirements.
Node operators E and F are willing to conform and attest to these compliance
requirements, but don't know about project D. It would be nice if Node
operators E and F could navigate to a compliance portal and see a list of
potential compliance attestations available. For possible compliance
attestations, node operators could sign agreements for these, and then receive
a verified signature that shows their selected compliance options.
Then, Project D's node selection process would filter by nodes that had been
approved for the necessary compliance requirements.
### Voting and operator signaling
As Satellite operators ourselves, we are currently engaged in a discussion about
pricing changes with storage node operators. Future Satellite operators may find
themselves in similar situations. It would be nice if storage node operators
could indicate votes for values. This would potentially be more representative
of network sentiment than posts on a forum.
Note that this isn't a transparent voting scheme, where other voters can see
the votes made, so this may not be a great voting solution in general.
## Design and implementation
I believe there are two basic building blocks that solves all of the above
issues:
* Signed node tags (with potential values)
* A document signing service
### Signed node tags
The network representation:
```
message Tag {
// Note that there is a signal flat namespace of all names per
// signer node id. Signers should be careful to make sure that
// there are no name collisions. For self-signed content-hash
// based values, the name should have the prefix of the content
// hash.
string name = 1;
bytes value = 2; // optional, representation dependent on name.
}
message TagSet {
// must always be set. this is the node the signer is signing for.
bytes node_id = 1;
repeated Tag tags = 2;
// must always be set. this makes sure the signature is signing the
// timestamp inside.
int64 timestamp = 3;
}
message SignedTagSet {
// this is the seralized form of TagSet, serialized so that
// the signature process has something stable to work with.
bytes serialized_tag = 1;
// this is who signed (could be self signed, could be well known).
bytes signer_node_id = 3;
bytes signature = 4;
}
message SignedTagSets {
repeated SignedTagSet tags = 1;
}
```
Note that every tag is signing a name/value pair (value optional) against
a specific node id.
Note also that names are only unique within the namespace of a given signer.
The database representation on the Satellite. N.B.: nothing should be entered
into this database without validation:
```
model signed_tags (
field node_id blob
field name text
field value blob
field timestamp int64
field signer_node_id blob
)
```
The "signer_node_id" is worth more explanation. Every signer should have a
stable node id. Satellites and storage nodes already have one, but any other
service that validates node tags would also need one.
In particular, the document signing service (below) would have its own unique
node id for signing tags, whereas for voting-style tags or tags based on a
content-addressed identifier (e.g. a hash of a document), the nodes would
self-sign.
### Document signing service
We would start a small web service, where users can log in and sign and fill
out documents. This web service would then create a unique activation code
that storage node operators could run on their storage nodes for activation and
signing. They could run `storagenode activate <code>` and then the node would
reach out to the signing service and get a `SignedTag` related to that node
given the information the user provided. The node could then present these
to the satellite.
Ultimately, the document signing service will require a separate design doc,
but here are some considerations for it:
Activation codes must expire shortly. Even Netflix has two hours of validity
for their service code - for a significantly less critical use case. What would
be a usable validity time for our use case? 15 minutes? 1 hour? Should we make
it configurable?
We want to still keep usability in mind for a SNO who needs to activate 500
nodes.
It would be even better if the SNO could force invalidating the activation code
when they are done with it.
As activation codes expire, the SNO should be able to generate a new activation
code if they want to associate a new node to an already signed document.
It should be hard to brute-force activation codes. They shouldn't be simple
numbers (4-digit or 6-digit) but something as complex as UUID.
It's also possible that SNO uses some signature mechanism during signing service
authentication, and the same signature is used for activation. If the same
signature mechanism is used during activation then no token is necessary.
### Update node selection
Once the above two building blocks exist, many problems become much more easily
solvable.
We would want to extend node selection to be able to do queries,
given project-specific configuration, based on these signed_tag values.
Because node selection mostly happens in memory from cached node table data,
it should be easy to add some denormalized data for certain selected cases,
such as:
* Document hashes nodes have self signed.
* Approval states based on well known third party signer nodes (a KYC service).
Once these fields exist, then node selection can happen as before, filtering
for the appropriate value given project settings.
## How these building blocks work for the example use cases
### 1099 KYC
The document signing service would have a KYC (Know Your Customer) form. Once
filled out, the document signing service would make a `TagSet` that includes all
of the answers to the KYC questions, for the given node id, signed by the
document signing service's node id.
The node would hang on to this `SignedTagSet` and submit it along with others
in a `SignedTagSets` to Satellites occasionally (maybe once a month during
node CheckIn).
### Private storage node networks
Storage node provisioning would provide nodes with a signed `SignedTagSet`
from a provisioning service that had its own node id. Then a private Satellite
could be configured to require that all nodes present a `SignedTagSet` signed
by the configured provisioning service that has that node's id in it.
Notably - this functionality could also be solved by the older waitlist node
identity signing certificate process, but we are slowly removing what remains
of that feature over time.
This functionality could also be solved by setting the Satellite's minimum
allowable node id difficulty to the maximum possible difficulty, thus preventing
any automatic node registration, and manually inserting node ids into the
database. This is what we are currently doing for private network trials, but
if `SignedTagSet`s existed, that would be easier.
### SOC2/HIPAA/etc node certification
For any type of document that doesn't require any third party service
(such as government id validation, etc), the document and its fields can be
filled out and self signed by the node, along with a content hash of the
document in question.
The node would create a `TagSet`, where one field is the hash of the legal
document that was agreed upon, and the remaining fields (with names prefixed
by the document's content hash) would be form fields
that the node operator filled in and ascribed to the document. Then, the
`TagSet` would be signed by the node itself. The cryptographic nature of the
content hash inside the `TagSet` would validate what the node operator had
agreed to.
### Voting and operator signaling
Node operators could self sign additional `Tag`s inside of a miscellaneous
`TagSet`, including `Tag`s such as
```
"storage-node-vote-20230611-network-change": "yes"
```
Or similar.
## Open problems
* Revocation? - `TagSets` have a timestamp inside that must be filled out. In
The future, certain tags could have an expiry or updated values or similar.
## Other options
## Wrapup
## Related work

View File

@ -0,0 +1,163 @@
# Fix deletes (and server side copy!)
## Abstract
Hey, let's make deletes faster by relying on GC. If we do this, there are some
additional fun implications.
## Background/context
We are having a lot of trouble with deletes with customers. In the last month
we have received critical feedback from a couple of customers (ask if you want
to know) about how hard it is to delete a bucket. A customer wants to stop
paying us for a bucket they no longer want, maybe due to the high per-segment
fee or otherwise.
The main thing customers want is to be able to issue a delete and have us
manage the delete process in the background.
There are two kinds of deletes right now (besides setting a TTL on objects) - explicit deletes and garbage
collection. Explicit deletes are supposed to happen immediately and not result
in unpaid data for the storage node (though they don't right now), and garbage
is generated due to long tail cancelation or other reasons, but is unfortunately
a cost to storage node operators in that they are not paid for data that is
considered garbage. Garbage is cleaned up by a garbage collection process that
stores data for an additional week after being identified as garbage in the
trash for recovery purposes. We have long desired to have as many deletes be
explicit deletes as possible for the above reasons.
The way explict deletes work right now is that the Uplink sends the Satellite a
delete request. The Satellite, in an attempt to both provide backpressure and
reduce garbage, then issues delete requests to the storage nodes, while keeping
the Uplink waiting. The benefit of the Satellite doing this is that the
Satellite attempts to batch some of these delete requests.
Unfortunately, because backups are snapshots at points in time, and Satellites
might be recovered from backup, storage nodes are currently unable to fully
delete these explicitly deleted objects. The process for recovering a Satellite
from backup is to first recover its backed up metadata, and then to issue a
restore-from-trash to all storage nodes. So, as a result, any of the gains we've
tried to get from explicit deletes are illusory because explicitly deleted data
goes into the trash just like any other garbage.
It has been our intention to eventually restore the functionality of storage
nodes being able to explicitly delete data through some sort of proof-of-delete
system that storage nodes can present to amnesiatic Satellites, or to improve
the Satellite backup system to have a write ahead log so that backups don't
forget anything. But, this has remained a low priority for years, and the
costs of doing so might outweigh the benefits.
One additional consideration about explicit deletes is that it complicates
server-side copy. Server-side copy must keep track of reference counting or
reference lists so that explicit deletes are not errantly issued too soon.
Keeping track of reference counting or reference lists is a significant burden
of bookkeeping. It adds many additional corner cases in nearly every object
interaction path, and reduces the overall performance of copied objects by
increasing the amount of database requests for them.
Consider instead another option! We don't do any of this!
## Design and implementation
No explicit deletes. When an uplink deletes data, it deletes it from the
Satellite only.
The Satellite will clean the data up on the storage nodes through the standard
garbage collection process.
That's it!
In case you're wondering, here are stats about optimal bloom filter sizing:
```
pieces size (10% false positives)
100000 58.4 KiB
1000000 583.9 KiB
10000000 5.7 MiB
100000000 57.0 MiB
```
### BUT WAIT, THERE'S MORE
If we no longer have explicit deletes, we can dramatically simplify server-side
copy! Instead of having many other tables with backreferences and keeping track
of copied objects separately and differently from uncopied objects and ancestor
objects and so on, we don't need any of that.
Copied objects can simply be full copies of the metadata, and we don't need to
keep track of when the last copy of a specific stream disappears.
This would considerably improve Satellite performance, load, and overhead on
copied objects.
This would considerably reduce the complexity of the Satellite codebase and data
model, which itself would reduce the challenges developers face when interacting
with our object model.
## Other options
Stick with the current plan.
## Migration
Migration can happen in the following order:
* We will first need to stop doing explicit deletes everywhere, so that
we don't accidentally delete anything.
* Then we will need to remove the server side copy code and just make object
copies actually just copy the straight metadata without all the copied object
bookkeeping.
* Once there is no risk and there is no incoming queue, then we can have a job
that iterates through all existing copied objects and denormalizes them to
get rid of the copied object bookkeeping.
## Wrapup
We should just do this. It feels painful to give up on explicit deletes but
considering we have not had them actually working for years and everyone seems
happy and it hasn't been any priority to fix, we could bite the bullet, commit
to this, and dramatically improve lots of other things.
It also feels painful to give up on the existing server-side copy design, but
that is a sunk cost.
## Additional Notes
1. With this proposal Storagenodes will store for more time (Until GC cleans up the files). I think it should be acceptable:
* For objects stored for longer period time, it doesn't give big difference (1 year vs 1 year + 1 day...)
* For object uploaded / downloaded in short period of time: It doesn't make sense just to upload + delete. For upload + download + delete, it's a good business anyway, as the big money is in egress, not in the storage. As an SNO, I am fine with this.
2. GDPR includes 'right to be forgotten'. I think this proposal should be compatible (but IANAL): if metadata (including the encryption key) is not available any more, there isn't any way to read it.
3. There is one exception: let's say I started to download some data, but meantime the owner deleted it. Explicit delete may block the read (pieces are disappearing, remaining segments might be missing...)
While this proposal would enable to finish the downloads if I already have the orderlimits from the satellite (pieces will remain there until next GC).
Don't know if this difference matters or not.
One other point on objects that are stored for a short amount of time above - we can potentially introduce a minimum storage duration to help cover costs.
## Q&A
> 1. what with node tallies? without additional bookkeeping it may be hard to not pay SNO for copies, SNO will be payed for storing single piece multiple times because we are just collecting pieces from segments to calc nodes tally.
> 2. how we will handle repairs? will we leave it as is and copy and original will be repaired on its own?
> 3. do we plan to pay for one week of additional storage? data won't be in trash.
> 4. we need to remember that currently segment copy doesn't keep pieces. pieces are main size factor for segments table. We need to take into account that if we will have duplications table size will grow. not a blocker but worth to remember.
These are good questions!
Ultimately, I think these are maybe questions for the product team to figure out, but my gut reaction is:
* according to the stats, there are very few copied objects. copied objects form a fraction of a percent of all data
* so, what if we just take questions one and three together and call it a wash? we overpay nodes by paying individually for each copy, and then don't pay nodes for the additional time before GC moves the deleted object to the trash? if we go this route, it also seems fine to let repair do multiple individual repairs.
i think my opinion would change if copies became a nontrivial amount of our data of course, and this may need to be revisited.
## Related work

View File

@ -11,7 +11,7 @@ This testplan is going to cover the Access Grants Page. This page lists access g
| Test Scenario | Test Case | Description | Comments | | Test Scenario | Test Case | Description | Comments |
|---------------------------------|---------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------| |---------------------------------|---------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|
| Access Grant Management Page UI | Click on the Access Management Section of Storj DCS Sidebar | While the sidebar is present, if the user clicks on the access management section of the sidebar then the user should be redirected to the access grant management page | | | Access Grant Management Page UI | Click on the Access Management Section of Storj Sidebar | While the sidebar is present, if the user clicks on the access management section of the sidebar then the user should be redirected to the access grant management page | |
| | Confirm Access Grant Management Page | While the user is in their access grant management page, the user should be able to see the Access Management Header and a header named My Access Keys with a list of access keys if the user has created any, a button for a new access grant and a search bar to search for any access grants | | | | Confirm Access Grant Management Page | While the user is in their access grant management page, the user should be able to see the Access Management Header and a header named My Access Keys with a list of access keys if the user has created any, a button for a new access grant and a search bar to search for any access grants | |
| | Access Grant More Info Button | Under the access management header, there is a more info button that leads to an explanation of access grants, so if it is clicked user should be redirected to storj-labs access grants concepts page | | | | Access Grant More Info Button | Under the access management header, there is a more info button that leads to an explanation of access grants, so if it is clicked user should be redirected to storj-labs access grants concepts page | |
| | Click More Info Button on Access Grant with Limited Permissions | When a user clicks on the more info button for said access grant with limited permissions, it should show the stated permissions | | | | Click More Info Button on Access Grant with Limited Permissions | When a user clicks on the more info button for said access grant with limited permissions, it should show the stated permissions | |
@ -20,7 +20,7 @@ This testplan is going to cover the Access Grants Page. This page lists access g
| | Access Grants Shortcuts- Learn More Button | If user clicks on learn more button on the access grants shortcuts, then user should be redirected to Storj-labs page with more information about access grants | | | | Access Grants Shortcuts- Learn More Button | If user clicks on learn more button on the access grants shortcuts, then user should be redirected to Storj-labs page with more information about access grants | |
| | API Keys Shortcuts- Create API Keys Button | If user clicks on create API keys button on the API keys shortcut, then user should be presented with a modal allowing user to create API keys (at the end user should also be able to copy said API key and Satellite Address or save it in a text file) | | | | API Keys Shortcuts- Create API Keys Button | If user clicks on create API keys button on the API keys shortcut, then user should be presented with a modal allowing user to create API keys (at the end user should also be able to copy said API key and Satellite Address or save it in a text file) | |
| | API Keys Shortcuts- Learn More Button | If user clicks on learn more button on the API keys shortcut, then user should be redirected to Storj-labs page with more information about API keys | | | | API Keys Shortcuts- Learn More Button | If user clicks on learn more button on the API keys shortcut, then user should be redirected to Storj-labs page with more information about API keys | |
| | S3 Credentials Shortcuts- Create S3 Credentials Button | If user clicks on create S3 credentials button on the S3 credentials shortcuts, then user should be presented with a modal to create S3 credentials to switch backend of an app using S3 compatible object storage to Storj DCS (at the end user should also be able to copy said S3 credentials; secret key, access key and endpoint on clipboard or download as a text file) | | | | S3 Credentials Shortcuts- Create S3 Credentials Button | If user clicks on create S3 credentials button on the S3 credentials shortcuts, then user should be presented with a modal to create S3 credentials to switch backend of an app using S3 compatible object storage to Storj (at the end user should also be able to copy said S3 credentials; secret key, access key and endpoint on clipboard or download as a text file) | |
| | S3 Credentials Shortcuts- Learn More Button | If user clicks on learn more button on the S3 credentials shortcut, then user should be redirected to Storj-labs page with more information on S3 credentials | | | | S3 Credentials Shortcuts- Learn More Button | If user clicks on learn more button on the S3 credentials shortcut, then user should be redirected to Storj-labs page with more information on S3 credentials | |
| | First Visit Check for About Access Grants | If user visits access management page for the first time, the user should see an about access grant message explaining what access grants are (this message should also be dismissible) | | | | First Visit Check for About Access Grants | If user visits access management page for the first time, the user should see an about access grant message explaining what access grants are (this message should also be dismissible) | |
| | Check for About Access Grants after First Visit | If user visits access management page again after their first time ( and presses dismiss), then for every subsequent visit to this page the user should not be presented with this access grant message | | | | Check for About Access Grants after First Visit | If user visits access management page again after their first time ( and presses dismiss), then for every subsequent visit to this page the user should not be presented with this access grant message | |

View File

@ -0,0 +1,17 @@
# Graceful Exit Revamp
## Background
This testplan covers Graceful Exit Revamp
&nbsp;
| Test Scenario | Test Case | Description | Comments |
|---------------|-----------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|
| Graceful Exit | Happy path | Perform GE on the node, satellite not send any new pieces to this node. Pieces on this node marked as "retrievable but unhealthy". After one month (with an appropriately high online score), the node will be considered exited. | Covered |
| | GE on Disqualified Node | Make sure GE was not initiated for the disqualified node. | Covered |
| | Double exit | Perform GE on the node and after receiving success message do it once again. Make sure node can not do it twice | Covered |
| | Low online score | Perform GE on node with less then 50% of score. Node should fail to GE | Covered |
| | Two many nodes call GE at the same time | We should transfer all the pieces to available nodes anyway. Example: start with 8 nodes(RS settings 2,3,4,4) and call GE on 4 nodes at the same time | |
| | Audits | SN should receive audits even if it perform GE at the moment | Covered? |
| | GE on Suspended node | Make sure GE was not initiated for the suspended node (Unknown audit errors). | |
| | GE started before feature deployment | Node should stop transferring new pieces and should be treated by tne new rules. | |

View File

@ -0,0 +1,46 @@
# Object Versioning
## Background
This testplan covers Object Versioning
&nbsp;
| Test Scenario | Test Case | Description | Comments |
|---------------|-----------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------|
| Copy | To a bucket that has versioning enabled | Should add one version to it(make a new version and make in latest) | check the column "versioning_state" in "bucket_metainfo" |
| | To a bucket that has version disabled | Make sure GE was not initiated for the disqualified node. | |
| | Copy object | Should support copying a specific version, should copy the latest version of an object if not specified | |
| Move | To a bucket that has versioning enabled | Should add one version to it | check the column "versioning_state" in "bucket_metainfo" |
| | To a bucket that has version disabled | Perform GE on node with less then 50% of score. Node should fail to GE | |
| Delete | Delete one from many versions | Create 3 versions of the same file and delete the middle one indicating the version id | |
| | All versions | Unconditionally deletes all versions of an object | |
| | Delete bucket | Force delete bucket with files that has versioning. We should keep all versions of the files unless manually deleted | |
| Restore | Delete and restore | Delete version of the file and restore from that version | |
| | Restore | Create few versions of the file and restore from latest to older version | |
| Create | Create new bucket | Versioning should be inherited from project level | |
| Suspend | Suspend versioning | Suspend versioning on a bucket that had versioning enabled. 3 versions of a file exists. Try to upload the same file again. -> the newest file gets overriden. The older 2 versions stay intact | |
| Update | Update metadata | Metadata update should not create new version. Takes the version as input but does not use it. Only updates the metadata for the highest committed object version. | |
| List | all versions | Unconditionally returns all object versions. Listing all versions should include delete markers. Versions come out created last to first | |
| UI | UI | UI should always show the latest version of each object | |
| Buckets | Old | Old buckets created before the feature should be in "unsupported" state | |
| | Enable versioning after upload | Upload obj to a bucket with versioning disabled and then enable versioning. Check version of the object | |
| PutObject | Versioning enabled | When object with same name uploaded to a bucket we should create new unique version of the object | |
| | Versioning disabled | Latest version of the object is overwritten by the new object, new object has a version ID of null | |
| | Multipart | Multipart upload with versioning enabled | |
| | Expiration | Create object with expiration in versioned bucket, delete marker should be applied to it | |
## Third-party test suite
These test suites have good tests inside, so we should run all versioning
related tests in them
* https://github.com/ceph/s3-tests/blob/master/s3tests_boto3/functional/test_s3.py
* https://github.com/snowflakedb/snowflake-s3compat-api-test-suite
## Questions
* Can a customer set a maximum number of versions?
* Can a customer pin specific versions to make sure they can't be deleted
by malware?
* Can a project member with a restricted access grant modify the version
flag on a bucket? Which permissions does the access grant need?

View File

@ -0,0 +1,25 @@
# Mini Cowbell Testplan
&nbsp;
## Background
We want to deploy the entire Storj stack on environments that have kubernetes running on 5 NUCs.
&nbsp;
## Pre-condition
Configuration for satellites that only have 5 node and the recommended RS scheme is [2,3,4,4] where:
- 2 is the number of required pieces to reconstitute the segment.
- 3 is the repair threshold, i.e. if a segment remains with only 3 healthy pieces, it will be repaired.
- 4 is the success threshold, i.e. the number of pieces required for a successful upload or repair.
- 4 is the number of total erasure-coded pieces that will be generated.
| Test Scenario | Test Case | Description | Comments |
|---------------|--------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Upload | Upload with all nodes online | Every file is uploaded to 4 nodes with 2x expansion factor. So one node has no files. | Happy path scenario |
| | Upload with one node offline | If one of five nodes fails and goes offline, 80% of the stored data will lose one erasure-coded piece. The health status of these segments will be reduced from 4 pieces to 3 pieces and will mark these segments for repair. overlay.node.online-window: 4h0m0s -> for about 4 hours the node will still be selected for uploads) | Uploads will continue uninterrupted if the client uses the new refactored upload path. This improved upload logic will request the satellite for a new node if the satellite selects the offline node for the upload, unaware it is already offline. If the client uses the old upload logic, uploads may fail if the satellite selects the offline node (20% chance). When the satellite detects the offline node, all uploads will be successful. |
| Download | Download with one node offline | If one of five nodes fails and goes offline, 80% of the stored data will lose one erasure-coded piece. The health status of these segments will be reduced from 4 pieces to 3 pieces and will mark these segments for repair. overlay.node.online-window: 4h0m0s -> for about 4 hours the node will still be selected for downloads) | |
| Repair | Repair with 2 nodes disqualified | Disqualify 2 nodes so the repair download are still possible but there is no node available for an upload, shouldn't consume download bandwidth and error out early. Only spend download bandwidth when there is at least one node available for an upload | If two nodes go offline, there are remaining pieces in the worst case, which cannot be repaired and is a de facto data loss if the offline nodes are damaged. |
| Audit | | Audits can't identify corrupted pieces with just the minimum number of pieces. Reputation should not increase. Audits should be able to identify corrupted pieces with minumum + 1 pieces. Reputation should decrease. | |
| Upgrades | Nodes restart for upgrades | No more than a single node goes offline for maintenance. Otherwise, normal operation of the network cannot be ensured. | Occasionally, nodes may need to restart due to software updates. This brings the node offline for some period of time |

View File

@ -0,0 +1,58 @@
## Storj Private Cloud - Test Plan
## Test Scenarios
Some test ideas:
- Upload and download some data
- Server side copy and server side move
- Multipart uploads
- Versioning (replace and existing file)
- Audit identifies a bad node and Repair finds new good nodes for the pieces (integration test inclusing audit reservoier sampling, audit job, reverifier, repair checker, repair worker)
- Repair checker and repair worker performance with a million segments in the repair queue (repair queue needs to be ordered by null values first)
- ranged loop performance (do we get better performance from running 2 range loops vs a single range?)
- Upload, Download, List, Delete performance with a million segments in the DB.
- Garbage collection especially the bloom filter creation. Needs to be run from a backup DB and can't be run from the live DB.
- Storage nodes and customer accounting
- Account upload and download limits (redis cache)
- Customer signup with onboarding including creating an access grant
- Token payments
- Graceful exit
- Node selection with geofencing, suspended nodes, disqualified nodes, offline nodes, nodes running outdated versions, nodes out of disk space
Bonus section (technically out of scope but still interresting questions for other tickets)
- Should a private satellite require a stripe account for the billing section? How does the UI look like without a stripe account? How can the customer upgrade to a pro account without having to add a credit card.
- Does the satellite need to be able to send out emails? For signup we have a simulation mode but for other features like project member invite we can't skip the email currently. (Other features with similar issues: storage node notifications, account freeze, password reset)
- What is the plan for the initial vetting period? A brand new satellite with brand new nodes will not be able to upload any date because not enough vetted nodes. -> config change to upload to unvetted nodes. -> risk about uploading too much data to unvetted nodes by keeping this setting longer than nessesary)
&nbsp;
&nbsp;
## [Test Plan Table]
| Test Scenario | Test Case | Description | Comments |
|-----------------------------|------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------|
| Upload | Small file | Do the upload for 1 KiB, 5 KiB, 1 MiB, 64 MiB files. | |
| | Big file | Do the upload 1024Mb files | |
| | Multipart upload | Upload big file to check the multipart upload | |
| Download | Inline segment | User should download inline segment without any errors | |
| | Remote segment | User should download remote segment without any errors | |
| | Copy 10000 Or More Segments | If a user uploads an object with 10000 segments or more and server side copies it from the source object to the destination object, it should be possible | |
| | Copy inline segment | User should copy inline segment without any errors | |
| | Copy remote segment | User should copy remote segment without any errors | |
| Move | Move object | Move object from one bucket to another bucket | |
| Versioning | Replace and existing file | User should be able to update existing file | |
| DB- Table Segment | Expiration Date | If a user uses Server-side copy, then the source object and the destination object must have the same expiration date | Might be redundant test because of segment table removing |
| DB - Table `segment_copies` | Ancestor_stream_id negative | If a segment with `stream_id = S` hasn't been copied, then the `segment_copies` table has no row having `ancestor_stream_id = S` | Might be redundant test because of segment table removing |
| | Ancestor_stream_id positive | If a segment with `stream_id = S` has been copied, then the `segment_copies` table has at least one row having `ancestor_stream_id = S` | Might be redundant test because of segment table removing |
| Repair | Data repair | Upload some data then kill some nodes and disqualify 1 node(should be enough storage nodes to upload repaired segments). Repaired segment should not contain any piece in the killed and DQ nodes. Downloads the data from new nodes and check that it's the same than the uploaded one. | This test should be in the code |
| Token payments | Multiple Transactions | If a user has a pending transaction and then performs another transaction with a higher nonce using the same address, the new transaction has to wait until the previous transaction with the lower nonce is confirmed (standard behavior of geth, nothing to test for us) | |
| | Invoice Generation | When an invoice is generated and "paid", coupons should be used first, followed by storj balance and then lastly credit card | |
| Performance | Repair queue index has to be null value first. | https://storj.slack.com/archives/C01427KSZ1P/p1589815803066100 | |
| Garbage Collection | Garbage Collection | Needs to be run from a backup DB and can't be run from the live DB | |
| Accounting | Customer | Generate the full invoice cycle | |
| | Storage node | Generate the invoice | |
| Account limits | Upload | Verify that limits are working | |
| | Download | Verify that limits are working | |
| Signup | Customer signup | Customer signup with onboarding including creating an access grant | |

61
go.mod
View File

@ -1,9 +1,8 @@
module storj.io/storj module storj.io/storj
go 1.18 go 1.19
require ( require (
github.com/VividCortex/ewma v1.2.0
github.com/alessio/shellescape v1.2.2 github.com/alessio/shellescape v1.2.2
github.com/alicebob/miniredis/v2 v2.13.3 github.com/alicebob/miniredis/v2 v2.13.3
github.com/blang/semver v3.5.1+incompatible github.com/blang/semver v3.5.1+incompatible
@ -22,49 +21,50 @@ require (
github.com/jackc/pgx/v5 v5.3.1 github.com/jackc/pgx/v5 v5.3.1
github.com/jtolds/monkit-hw/v2 v2.0.0-20191108235325-141a0da276b3 github.com/jtolds/monkit-hw/v2 v2.0.0-20191108235325-141a0da276b3
github.com/jtolio/eventkit v0.0.0-20230607152326-4668f79ff72d github.com/jtolio/eventkit v0.0.0-20230607152326-4668f79ff72d
github.com/jtolio/mito v0.0.0-20230523171229-d78ef06bb77b
github.com/jtolio/noiseconn v0.0.0-20230301220541-88105e6c8ac6 github.com/jtolio/noiseconn v0.0.0-20230301220541-88105e6c8ac6
github.com/loov/hrtime v1.0.3 github.com/loov/hrtime v1.0.3
github.com/mattn/go-sqlite3 v1.14.12 github.com/mattn/go-sqlite3 v1.14.12
github.com/nsf/jsondiff v0.0.0-20200515183724-f29ed568f4ce github.com/nsf/jsondiff v0.0.0-20200515183724-f29ed568f4ce
github.com/nsf/termbox-go v0.0.0-20200418040025-38ba6e5628f1 github.com/nsf/termbox-go v0.0.0-20200418040025-38ba6e5628f1
github.com/oschwald/maxminddb-golang v1.8.0 github.com/oschwald/maxminddb-golang v1.12.0
github.com/pquerna/otp v1.3.0 github.com/pquerna/otp v1.3.0
github.com/redis/go-redis/v9 v9.0.3 github.com/redis/go-redis/v9 v9.0.3
github.com/shopspring/decimal v1.2.0 github.com/shopspring/decimal v1.2.0
github.com/spacemonkeygo/monkit/v3 v3.0.20-0.20230419135619-fb89f20752cb github.com/spacemonkeygo/monkit/v3 v3.0.22
github.com/spacemonkeygo/tlshowdy v0.0.0-20160207005338-8fa2cec1d7cd github.com/spacemonkeygo/tlshowdy v0.0.0-20160207005338-8fa2cec1d7cd
github.com/spf13/cobra v1.1.3 github.com/spf13/cobra v1.1.3
github.com/spf13/pflag v1.0.5 github.com/spf13/pflag v1.0.5
github.com/spf13/viper v1.7.1 github.com/spf13/viper v1.7.1
github.com/stretchr/testify v1.8.2 github.com/stretchr/testify v1.8.4
github.com/stripe/stripe-go/v72 v72.90.0 github.com/stripe/stripe-go/v75 v75.8.0
github.com/vbauerster/mpb/v8 v8.4.0 github.com/vbauerster/mpb/v8 v8.4.0
github.com/vivint/infectious v0.0.0-20200605153912-25a574ae18a3
github.com/zeebo/assert v1.3.1 github.com/zeebo/assert v1.3.1
github.com/zeebo/blake3 v0.2.3 github.com/zeebo/blake3 v0.2.3
github.com/zeebo/clingy v0.0.0-20230602044025-906be850f10d github.com/zeebo/clingy v0.0.0-20230602044025-906be850f10d
github.com/zeebo/errs v1.3.0 github.com/zeebo/errs v1.3.0
github.com/zeebo/errs/v2 v2.0.3 github.com/zeebo/errs/v2 v2.0.3
github.com/zeebo/ini v0.0.0-20210514163846-cc8fbd8d9599 github.com/zeebo/ini v0.0.0-20210514163846-cc8fbd8d9599
github.com/zeebo/structs v1.0.3-0.20230601144555-f2db46069602
github.com/zyedidia/generic v1.2.1 github.com/zyedidia/generic v1.2.1
go.etcd.io/bbolt v1.3.5 go.etcd.io/bbolt v1.3.5
go.uber.org/zap v1.16.0 go.uber.org/zap v1.16.0
golang.org/x/crypto v0.7.0 golang.org/x/crypto v0.12.0
golang.org/x/exp v0.0.0-20221205204356-47842c84f3db golang.org/x/exp v0.0.0-20221205204356-47842c84f3db
golang.org/x/net v0.9.0 golang.org/x/net v0.10.0
golang.org/x/oauth2 v0.7.0 golang.org/x/oauth2 v0.7.0
golang.org/x/sync v0.1.0 golang.org/x/sync v0.3.0
golang.org/x/sys v0.7.0 golang.org/x/sys v0.13.0
golang.org/x/term v0.7.0 golang.org/x/term v0.11.0
golang.org/x/text v0.9.0 golang.org/x/text v0.12.0
golang.org/x/time v0.0.0-20200630173020-3af7569d3a1e golang.org/x/time v0.0.0-20200630173020-3af7569d3a1e
gopkg.in/segmentio/analytics-go.v3 v3.1.0 gopkg.in/segmentio/analytics-go.v3 v3.1.0
gopkg.in/yaml.v3 v3.0.1 gopkg.in/yaml.v3 v3.0.1
storj.io/common v0.0.0-20230602145716-d6ea82d58b3d storj.io/common v0.0.0-20231130134106-1fa84867e323
storj.io/drpc v0.0.33 storj.io/drpc v0.0.33
storj.io/monkit-jaeger v0.0.0-20220915074555-d100d7589f41 storj.io/monkit-jaeger v0.0.0-20230707083646-f15e6e8b7e8c
storj.io/private v0.0.0-20230627140631-807a2f00d0e1 storj.io/private v0.0.0-20231127092015-c439a594bc1d
storj.io/uplink v1.10.1-0.20230626081029-035890d408c2 storj.io/uplink v1.12.3-0.20231130143633-4a092fa01b98
) )
require ( require (
@ -72,6 +72,7 @@ require (
cloud.google.com/go/compute v1.19.0 // indirect cloud.google.com/go/compute v1.19.0 // indirect
cloud.google.com/go/compute/metadata v0.2.3 // indirect cloud.google.com/go/compute/metadata v0.2.3 // indirect
cloud.google.com/go/profiler v0.3.1 // indirect cloud.google.com/go/profiler v0.3.1 // indirect
github.com/VividCortex/ewma v1.2.0 // indirect
github.com/acarl005/stripansi v0.0.0-20180116102854-5a71ef0e047d // indirect github.com/acarl005/stripansi v0.0.0-20180116102854-5a71ef0e047d // indirect
github.com/alicebob/gopher-json v0.0.0-20200520072559-a9ecdc9d1d3a // indirect github.com/alicebob/gopher-json v0.0.0-20200520072559-a9ecdc9d1d3a // indirect
github.com/apache/thrift v0.12.0 // indirect github.com/apache/thrift v0.12.0 // indirect
@ -83,7 +84,7 @@ require (
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
github.com/flynn/noise v1.0.0 // indirect github.com/flynn/noise v1.0.0 // indirect
github.com/fsnotify/fsnotify v1.5.4 // indirect github.com/fsnotify/fsnotify v1.5.4 // indirect
github.com/go-task/slim-sprig v0.0.0-20210107165309-348f09dbbbc0 // indirect github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572 // indirect
github.com/golang-jwt/jwt v3.2.1+incompatible // indirect github.com/golang-jwt/jwt v3.2.1+incompatible // indirect
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
github.com/golang/protobuf v1.5.3 // indirect github.com/golang/protobuf v1.5.3 // indirect
@ -100,22 +101,25 @@ require (
github.com/jackc/pgproto3/v2 v2.3.2 // indirect github.com/jackc/pgproto3/v2 v2.3.2 // indirect
github.com/jackc/pgservicefile v0.0.0-20221227161230-091c0ba34f0a // indirect github.com/jackc/pgservicefile v0.0.0-20221227161230-091c0ba34f0a // indirect
github.com/jtolds/tracetagger/v2 v2.0.0-rc5 // indirect github.com/jtolds/tracetagger/v2 v2.0.0-rc5 // indirect
github.com/jtolio/crawlspace v0.0.0-20231116162947-3ec5cc6b36c5 // indirect
github.com/jtolio/crawlspace/tools v0.0.0-20231115161146-57d90b78ce62 // indirect
github.com/klauspost/compress v1.15.10 // indirect github.com/klauspost/compress v1.15.10 // indirect
github.com/klauspost/cpuid/v2 v2.0.12 // indirect github.com/klauspost/cpuid/v2 v2.0.12 // indirect
github.com/kr/pretty v0.3.1 // indirect
github.com/kr/text v0.2.0 // indirect
github.com/magiconair/properties v1.8.5 // indirect github.com/magiconair/properties v1.8.5 // indirect
github.com/mattn/go-colorable v0.1.7 // indirect github.com/mattn/go-colorable v0.1.7 // indirect
github.com/mattn/go-isatty v0.0.12 // indirect github.com/mattn/go-isatty v0.0.12 // indirect
github.com/mattn/go-runewidth v0.0.14 // indirect github.com/mattn/go-runewidth v0.0.14 // indirect
github.com/mitchellh/mapstructure v1.4.1 // indirect github.com/mitchellh/mapstructure v1.4.1 // indirect
github.com/onsi/ginkgo/v2 v2.2.0 // indirect github.com/onsi/ginkgo/v2 v2.9.5 // indirect
github.com/pelletier/go-toml v1.9.0 // indirect github.com/pelletier/go-toml v1.9.0 // indirect
github.com/pkg/errors v0.9.1 // indirect github.com/pkg/errors v0.9.1 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/quic-go/qtls-go1-18 v0.2.0 // indirect github.com/quic-go/qtls-go1-20 v0.4.1 // indirect
github.com/quic-go/qtls-go1-19 v0.2.0 // indirect github.com/quic-go/quic-go v0.40.0 // indirect
github.com/quic-go/qtls-go1-20 v0.1.0 // indirect
github.com/quic-go/quic-go v0.32.0 // indirect
github.com/rivo/uniseg v0.4.4 // indirect github.com/rivo/uniseg v0.4.4 // indirect
github.com/rogpeppe/go-internal v1.10.0 // indirect
github.com/segmentio/backo-go v0.0.0-20200129164019-23eae7c10bd3 // indirect github.com/segmentio/backo-go v0.0.0-20200129164019-23eae7c10bd3 // indirect
github.com/spacemonkeygo/spacelog v0.0.0-20180420211403-2296661a0572 // indirect github.com/spacemonkeygo/spacelog v0.0.0-20180420211403-2296661a0572 // indirect
github.com/spf13/afero v1.6.0 // indirect github.com/spf13/afero v1.6.0 // indirect
@ -126,14 +130,16 @@ require (
github.com/yuin/gopher-lua v0.0.0-20191220021717-ab39c6098bdb // indirect github.com/yuin/gopher-lua v0.0.0-20191220021717-ab39c6098bdb // indirect
github.com/zeebo/admission/v3 v3.0.3 // indirect github.com/zeebo/admission/v3 v3.0.3 // indirect
github.com/zeebo/float16 v0.1.0 // indirect github.com/zeebo/float16 v0.1.0 // indirect
github.com/zeebo/goof v0.0.0-20230830143729-8a73f2ee257d // indirect
github.com/zeebo/incenc v0.0.0-20180505221441-0d92902eec54 // indirect github.com/zeebo/incenc v0.0.0-20180505221441-0d92902eec54 // indirect
github.com/zeebo/mwc v0.0.4 // indirect github.com/zeebo/mwc v0.0.4 // indirect
github.com/zeebo/structs v1.0.3-0.20230601144555-f2db46069602 // indirect github.com/zeebo/sudo v1.0.2 // indirect
go.opencensus.io v0.24.0 // indirect go.opencensus.io v0.24.0 // indirect
go.uber.org/atomic v1.7.0 // indirect go.uber.org/atomic v1.7.0 // indirect
go.uber.org/mock v0.3.0 // indirect
go.uber.org/multierr v1.6.0 // indirect go.uber.org/multierr v1.6.0 // indirect
golang.org/x/mod v0.8.0 // indirect golang.org/x/mod v0.11.0 // indirect
golang.org/x/tools v0.6.0 // indirect golang.org/x/tools v0.9.1 // indirect
google.golang.org/api v0.118.0 // indirect google.golang.org/api v0.118.0 // indirect
google.golang.org/appengine v1.6.7 // indirect google.golang.org/appengine v1.6.7 // indirect
google.golang.org/genproto v0.0.0-20230410155749-daa745c078e1 // indirect google.golang.org/genproto v0.0.0-20230410155749-daa745c078e1 // indirect
@ -141,5 +147,6 @@ require (
google.golang.org/protobuf v1.30.0 // indirect google.golang.org/protobuf v1.30.0 // indirect
gopkg.in/ini.v1 v1.62.0 // indirect gopkg.in/ini.v1 v1.62.0 // indirect
gopkg.in/yaml.v2 v2.4.0 // indirect gopkg.in/yaml.v2 v2.4.0 // indirect
storj.io/picobuf v0.0.1 // indirect storj.io/infectious v0.0.2 // indirect
storj.io/picobuf v0.0.2-0.20230906122608-c4ba17033c6c // indirect
) )

114
go.sum
View File

@ -103,6 +103,7 @@ github.com/coreos/go-systemd v0.0.0-20190719114852-fd7a80b32e1f/go.mod h1:F5haX7
github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f/go.mod h1:E3G3o1h8I7cfcXa63jLwjI0eiQQMgzzUDFVpN/nH/eA= github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f/go.mod h1:E3G3o1h8I7cfcXa63jLwjI0eiQQMgzzUDFVpN/nH/eA=
github.com/cpuguy83/go-md2man/v2 v2.0.0/go.mod h1:maD7wRr/U5Z6m/iR4s+kqSMx2CaBsrgA7czyZG/E6dU= github.com/cpuguy83/go-md2man/v2 v2.0.0/go.mod h1:maD7wRr/U5Z6m/iR4s+kqSMx2CaBsrgA7czyZG/E6dU=
github.com/creack/pty v1.1.7/go.mod h1:lj5s0c3V2DBrqTV7llrYr5NG6My20zk30Fl46Y7DoTY= github.com/creack/pty v1.1.7/go.mod h1:lj5s0c3V2DBrqTV7llrYr5NG6My20zk30Fl46Y7DoTY=
github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
@ -143,12 +144,14 @@ github.com/go-kit/log v0.1.0/go.mod h1:zbhenjAZHb184qTLMA9ZjW7ThYL0H2mk7Q6pNt4vb
github.com/go-logfmt/logfmt v0.3.0/go.mod h1:Qt1PoO58o5twSAckw1HlFXLmHsOX5/0LbT9GBnD5lWE= github.com/go-logfmt/logfmt v0.3.0/go.mod h1:Qt1PoO58o5twSAckw1HlFXLmHsOX5/0LbT9GBnD5lWE=
github.com/go-logfmt/logfmt v0.4.0/go.mod h1:3RMwSq7FuexP4Kalkev3ejPJsZTpXXBr9+V4qmtdjCk= github.com/go-logfmt/logfmt v0.4.0/go.mod h1:3RMwSq7FuexP4Kalkev3ejPJsZTpXXBr9+V4qmtdjCk=
github.com/go-logfmt/logfmt v0.5.0/go.mod h1:wCYkCAKZfumFQihp8CzCvQ3paCTfi41vtzG1KdI/P7A= github.com/go-logfmt/logfmt v0.5.0/go.mod h1:wCYkCAKZfumFQihp8CzCvQ3paCTfi41vtzG1KdI/P7A=
github.com/go-logr/logr v1.2.4 h1:g01GSCwiDw2xSZfjJ2/T9M+S6pFdcNtFYsp+Y43HYDQ=
github.com/go-oauth2/oauth2/v4 v4.4.2 h1:tWQlR5I4/qhWiyOME67BAFmo622yi+2mm7DMm8DpMdg= github.com/go-oauth2/oauth2/v4 v4.4.2 h1:tWQlR5I4/qhWiyOME67BAFmo622yi+2mm7DMm8DpMdg=
github.com/go-oauth2/oauth2/v4 v4.4.2/go.mod h1:K4DemYzNwwYnIDOPdHtX/7SlO0AHdtlphsTgE7lA3PA= github.com/go-oauth2/oauth2/v4 v4.4.2/go.mod h1:K4DemYzNwwYnIDOPdHtX/7SlO0AHdtlphsTgE7lA3PA=
github.com/go-session/session v3.1.2+incompatible/go.mod h1:8B3iivBQjrz/JtC68Np2T1yBBLxTan3mn/3OM0CyRt0= github.com/go-session/session v3.1.2+incompatible/go.mod h1:8B3iivBQjrz/JtC68Np2T1yBBLxTan3mn/3OM0CyRt0=
github.com/go-stack/stack v1.8.0/go.mod h1:v0f6uXyyMGvRgIKkXu+yp6POWl0qKG85gN/melR3HDY= github.com/go-stack/stack v1.8.0/go.mod h1:v0f6uXyyMGvRgIKkXu+yp6POWl0qKG85gN/melR3HDY=
github.com/go-task/slim-sprig v0.0.0-20210107165309-348f09dbbbc0 h1:p104kn46Q8WdvHunIJ9dAyjPVtrBPhSr3KT2yUst43I=
github.com/go-task/slim-sprig v0.0.0-20210107165309-348f09dbbbc0/go.mod h1:fyg7847qk6SyHyPtNmDHnmrv/HOrqktSC+C9fM+CJOE= github.com/go-task/slim-sprig v0.0.0-20210107165309-348f09dbbbc0/go.mod h1:fyg7847qk6SyHyPtNmDHnmrv/HOrqktSC+C9fM+CJOE=
github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572 h1:tfuBGBXKqDEevZMzYi5KSi8KkcZtzBcTgAUUtapy0OI=
github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572/go.mod h1:9Pwr4B2jHnOSGXyyzV8ROjYa2ojvAY6HCGYYfMoC3Ls=
github.com/gofrs/uuid v4.0.0+incompatible/go.mod h1:b2aQJv3Z4Fp6yNu3cdSllBxTCLRxnplIgP/c0N/04lM= github.com/gofrs/uuid v4.0.0+incompatible/go.mod h1:b2aQJv3Z4Fp6yNu3cdSllBxTCLRxnplIgP/c0N/04lM=
github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ= github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ=
github.com/gogo/protobuf v1.2.1/go.mod h1:hp+jE20tsWTFYpLwKvXlhS1hjn+gTNwPg2I6zVXpSg4= github.com/gogo/protobuf v1.2.1/go.mod h1:hp+jE20tsWTFYpLwKvXlhS1hjn+gTNwPg2I6zVXpSg4=
@ -322,8 +325,14 @@ github.com/jtolds/monkit-hw/v2 v2.0.0-20191108235325-141a0da276b3 h1:dITCBge70U9
github.com/jtolds/monkit-hw/v2 v2.0.0-20191108235325-141a0da276b3/go.mod h1:eo5po8nCwRcvZIIR8eGi7PKthzXuunpXzUmXzxCBfBc= github.com/jtolds/monkit-hw/v2 v2.0.0-20191108235325-141a0da276b3/go.mod h1:eo5po8nCwRcvZIIR8eGi7PKthzXuunpXzUmXzxCBfBc=
github.com/jtolds/tracetagger/v2 v2.0.0-rc5 h1:SriMFVtftPsQmG+0xaABotz9HnoKoo1QM/oggqfpGh8= github.com/jtolds/tracetagger/v2 v2.0.0-rc5 h1:SriMFVtftPsQmG+0xaABotz9HnoKoo1QM/oggqfpGh8=
github.com/jtolds/tracetagger/v2 v2.0.0-rc5/go.mod h1:61Fh+XhbBONy+RsqkA+xTtmaFbEVL040m9FAF/hTrjQ= github.com/jtolds/tracetagger/v2 v2.0.0-rc5/go.mod h1:61Fh+XhbBONy+RsqkA+xTtmaFbEVL040m9FAF/hTrjQ=
github.com/jtolio/crawlspace v0.0.0-20231116162947-3ec5cc6b36c5 h1:RSt5K+VT7bPr6A9DW/8Kav6V6aYB+8Vqn6ygqp6S0UM=
github.com/jtolio/crawlspace v0.0.0-20231116162947-3ec5cc6b36c5/go.mod h1:ruaBEBN4k5AmKzmI6K2LsfLno2t5tPgvSUB2dyiHHqo=
github.com/jtolio/crawlspace/tools v0.0.0-20231115161146-57d90b78ce62 h1:51cqrrnWE0zKhZFepIgnY7JSHgN5uGMX1aVFHjtc1ek=
github.com/jtolio/crawlspace/tools v0.0.0-20231115161146-57d90b78ce62/go.mod h1:Fa/Qz4+Sh0xCARqEKUdF7RCGMZcF3ilqBIfS2eVfA/Y=
github.com/jtolio/eventkit v0.0.0-20230607152326-4668f79ff72d h1:MAGZUXA8MLSA5oJT1Gua3nLSyTYF2uvBgM4Sfs5+jts= github.com/jtolio/eventkit v0.0.0-20230607152326-4668f79ff72d h1:MAGZUXA8MLSA5oJT1Gua3nLSyTYF2uvBgM4Sfs5+jts=
github.com/jtolio/eventkit v0.0.0-20230607152326-4668f79ff72d/go.mod h1:PXFUrknJu7TkBNyL8t7XWDPtDFFLFrNQQAdsXv9YfJE= github.com/jtolio/eventkit v0.0.0-20230607152326-4668f79ff72d/go.mod h1:PXFUrknJu7TkBNyL8t7XWDPtDFFLFrNQQAdsXv9YfJE=
github.com/jtolio/mito v0.0.0-20230523171229-d78ef06bb77b h1:HKvXTXZTeUHXRibg2ilZlkGSQP6A3cs0zXrBd4xMi6M=
github.com/jtolio/mito v0.0.0-20230523171229-d78ef06bb77b/go.mod h1:Mrym6OnPMkBKvN8/uXSkyhFSh6ndKKYE+Q4kxCfQ4V0=
github.com/jtolio/noiseconn v0.0.0-20230301220541-88105e6c8ac6 h1:iVMQyk78uOpX/UKjEbzyBdptXgEz6jwGwo7kM9IQ+3U= github.com/jtolio/noiseconn v0.0.0-20230301220541-88105e6c8ac6 h1:iVMQyk78uOpX/UKjEbzyBdptXgEz6jwGwo7kM9IQ+3U=
github.com/jtolio/noiseconn v0.0.0-20230301220541-88105e6c8ac6/go.mod h1:MEkhEPFwP3yudWO0lj6vfYpLIB+3eIcuIW+e0AZzUQk= github.com/jtolio/noiseconn v0.0.0-20230301220541-88105e6c8ac6/go.mod h1:MEkhEPFwP3yudWO0lj6vfYpLIB+3eIcuIW+e0AZzUQk=
github.com/julienschmidt/httprouter v1.2.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7VTCxuUUipMqKk8s4w= github.com/julienschmidt/httprouter v1.2.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7VTCxuUUipMqKk8s4w=
@ -344,11 +353,13 @@ github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515/go.mod h1:+0opPa2QZZtGFB
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo= github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
github.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI= github.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI=
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE= github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
github.com/kr/pty v1.1.3/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= github.com/kr/pty v1.1.3/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
github.com/kr/pty v1.1.8/go.mod h1:O1sed60cT9XZ5uDucP5qwvh+TE3NnUj51EiZO/lmSfw= github.com/kr/pty v1.1.8/go.mod h1:O1sed60cT9XZ5uDucP5qwvh+TE3NnUj51EiZO/lmSfw=
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/lib/pq v1.0.0/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo= github.com/lib/pq v1.0.0/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo=
github.com/lib/pq v1.1.0/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo= github.com/lib/pq v1.1.0/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo=
github.com/lib/pq v1.2.0/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo= github.com/lib/pq v1.2.0/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo=
@ -420,19 +431,20 @@ github.com/onsi/ginkgo v1.16.2/go.mod h1:CObGmKUOKaSC0RjmoAK7tKyn4Azo5P2IWuoMnvw
github.com/onsi/ginkgo v1.16.4/go.mod h1:dX+/inL/fNMqNlz0e9LfyB9TswhZpCVdJM/Z6Vvnwo0= github.com/onsi/ginkgo v1.16.4/go.mod h1:dX+/inL/fNMqNlz0e9LfyB9TswhZpCVdJM/Z6Vvnwo0=
github.com/onsi/ginkgo v1.16.5 h1:8xi0RTUf59SOSfEtZMvwTvXYMzG4gV23XVHOZiXNtnE= github.com/onsi/ginkgo v1.16.5 h1:8xi0RTUf59SOSfEtZMvwTvXYMzG4gV23XVHOZiXNtnE=
github.com/onsi/ginkgo v1.16.5/go.mod h1:+E8gABHa3K6zRBolWtd+ROzc/U5bkGt0FwiG042wbpU= github.com/onsi/ginkgo v1.16.5/go.mod h1:+E8gABHa3K6zRBolWtd+ROzc/U5bkGt0FwiG042wbpU=
github.com/onsi/ginkgo/v2 v2.2.0 h1:3ZNA3L1c5FYDFTTxbFeVGGD8jYvjYauHD30YgLxVsNI= github.com/onsi/ginkgo/v2 v2.9.5 h1:+6Hr4uxzP4XIUyAkg61dWBw8lb/gc4/X5luuxN/EC+Q=
github.com/onsi/ginkgo/v2 v2.2.0/go.mod h1:MEH45j8TBi6u9BMogfbp0stKC5cdGjumZj5Y7AG4VIk= github.com/onsi/ginkgo/v2 v2.9.5/go.mod h1:tvAoo1QUJwNEU2ITftXTpR7R1RbCzoZUOs3RonqW57k=
github.com/onsi/gomega v1.7.1/go.mod h1:XdKZgCCFLUoM/7CFJVPcG8C1xQ1AJ0vpAezJrB7JYyY= github.com/onsi/gomega v1.7.1/go.mod h1:XdKZgCCFLUoM/7CFJVPcG8C1xQ1AJ0vpAezJrB7JYyY=
github.com/onsi/gomega v1.10.1/go.mod h1:iN09h71vgCQne3DLsj+A5owkum+a2tYe+TOCB1ybHNo= github.com/onsi/gomega v1.10.1/go.mod h1:iN09h71vgCQne3DLsj+A5owkum+a2tYe+TOCB1ybHNo=
github.com/onsi/gomega v1.13.0/go.mod h1:lRk9szgn8TxENtWd0Tp4c3wjlRfMTMH27I+3Je41yGY= github.com/onsi/gomega v1.13.0/go.mod h1:lRk9szgn8TxENtWd0Tp4c3wjlRfMTMH27I+3Je41yGY=
github.com/onsi/gomega v1.20.1 h1:PA/3qinGoukvymdIDV8pii6tiZgC8kbmJO6Z5+b002Q= github.com/onsi/gomega v1.27.6 h1:ENqfyGeS5AX/rlXDd/ETokDz93u0YufY1Pgxuy/PvWE=
github.com/openzipkin/zipkin-go v0.1.1/go.mod h1:NtoC/o8u3JlF1lSlyPNswIbeQH9bJTmOf0Erfk+hxe8= github.com/openzipkin/zipkin-go v0.1.1/go.mod h1:NtoC/o8u3JlF1lSlyPNswIbeQH9bJTmOf0Erfk+hxe8=
github.com/oschwald/maxminddb-golang v1.8.0 h1:Uh/DSnGoxsyp/KYbY1AuP0tYEwfs0sCph9p/UMXK/Hk= github.com/oschwald/maxminddb-golang v1.12.0 h1:9FnTOD0YOhP7DGxGsq4glzpGy5+w7pq50AS6wALUMYs=
github.com/oschwald/maxminddb-golang v1.8.0/go.mod h1:RXZtst0N6+FY/3qCNmZMBApR19cdQj43/NM9VkrNAis= github.com/oschwald/maxminddb-golang v1.12.0/go.mod h1:q0Nob5lTCqyQ8WT6FYgS1L7PXKVVbgiymefNwIjPzgY=
github.com/pascaldekloe/goe v0.0.0-20180627143212-57f6aae5913c/go.mod h1:lzWF7FIEvWOWxwDKqyGYQf6ZUaNfKdP144TG7ZOy1lc= github.com/pascaldekloe/goe v0.0.0-20180627143212-57f6aae5913c/go.mod h1:lzWF7FIEvWOWxwDKqyGYQf6ZUaNfKdP144TG7ZOy1lc=
github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic= github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic=
github.com/pelletier/go-toml v1.9.0 h1:NOd0BRdOKpPf0SxkL3HxSQOG7rNh+4kl6PHcBPFs7Q0= github.com/pelletier/go-toml v1.9.0 h1:NOd0BRdOKpPf0SxkL3HxSQOG7rNh+4kl6PHcBPFs7Q0=
github.com/pelletier/go-toml v1.9.0/go.mod h1:u1nR/EPcESfeI/szUZKdtJ0xRNbUoANCkoOuaOx1Y+c= github.com/pelletier/go-toml v1.9.0/go.mod h1:u1nR/EPcESfeI/szUZKdtJ0xRNbUoANCkoOuaOx1Y+c=
github.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e/go.mod h1:pJLUxLENpZxwdsKMEsNbx1VGcRFpLqf3715MtcvvzbA=
github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
@ -456,14 +468,10 @@ github.com/prometheus/procfs v0.0.0-20180725123919-05ee40e3a273/go.mod h1:c3At6R
github.com/prometheus/procfs v0.0.0-20181005140218-185b4288413d/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk= github.com/prometheus/procfs v0.0.0-20181005140218-185b4288413d/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
github.com/prometheus/procfs v0.0.0-20190507164030-5867b95ac084/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsTZCD3I8kEA= github.com/prometheus/procfs v0.0.0-20190507164030-5867b95ac084/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsTZCD3I8kEA=
github.com/prometheus/tsdb v0.7.1/go.mod h1:qhTCs0VvXwvX/y3TZrWD7rabWM+ijKTux40TwIPHuXU= github.com/prometheus/tsdb v0.7.1/go.mod h1:qhTCs0VvXwvX/y3TZrWD7rabWM+ijKTux40TwIPHuXU=
github.com/quic-go/qtls-go1-18 v0.2.0 h1:5ViXqBZ90wpUcZS0ge79rf029yx0dYB0McyPJwqqj7U= github.com/quic-go/qtls-go1-20 v0.4.1 h1:D33340mCNDAIKBqXuAvexTNMUByrYmFYVfKfDN5nfFs=
github.com/quic-go/qtls-go1-18 v0.2.0/go.mod h1:moGulGHK7o6O8lSPSZNoOwcLvJKJ85vVNc7oJFD65bc= github.com/quic-go/qtls-go1-20 v0.4.1/go.mod h1:X9Nh97ZL80Z+bX/gUXMbipO6OxdiDi58b/fMC9mAL+k=
github.com/quic-go/qtls-go1-19 v0.2.0 h1:Cvn2WdhyViFUHoOqK52i51k4nDX8EwIh5VJiVM4nttk= github.com/quic-go/quic-go v0.40.0 h1:GYd1iznlKm7dpHD7pOVpUvItgMPo/jrMgDWZhMCecqw=
github.com/quic-go/qtls-go1-19 v0.2.0/go.mod h1:ySOI96ew8lnoKPtSqx2BlI5wCpUVPT05RMAlajtnyOI= github.com/quic-go/quic-go v0.40.0/go.mod h1:PeN7kuVJ4xZbxSv/4OX6S1USOX8MJvydwpTx31vx60c=
github.com/quic-go/qtls-go1-20 v0.1.0 h1:d1PK3ErFy9t7zxKsG3NXBJXZjp/kMLoIb3y/kV54oAI=
github.com/quic-go/qtls-go1-20 v0.1.0/go.mod h1:JKtK6mjbAVcUTN/9jZpvLbGxvdWIKS8uT7EiStoU1SM=
github.com/quic-go/quic-go v0.32.0 h1:lY02md31s1JgPiiyfqJijpu/UX/Iun304FI3yUqX7tA=
github.com/quic-go/quic-go v0.32.0/go.mod h1:/fCsKANhQIeD5l76c2JFU+07gVE3KaA0FP+0zMWwfwo=
github.com/redis/go-redis/v9 v9.0.3 h1:+7mmR26M0IvyLxGZUHxu4GiBkJkVDid0Un+j4ScYu4k= github.com/redis/go-redis/v9 v9.0.3 h1:+7mmR26M0IvyLxGZUHxu4GiBkJkVDid0Un+j4ScYu4k=
github.com/redis/go-redis/v9 v9.0.3/go.mod h1:WqMKv5vnQbRuZstUwxQI195wHy+t4PuXDOjzMvcuQHk= github.com/redis/go-redis/v9 v9.0.3/go.mod h1:WqMKv5vnQbRuZstUwxQI195wHy+t4PuXDOjzMvcuQHk=
github.com/rivo/uniseg v0.2.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc= github.com/rivo/uniseg v0.2.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc=
@ -472,7 +480,9 @@ github.com/rivo/uniseg v0.4.4/go.mod h1:FN3SvrM+Zdj16jyLfmOkMNblXMcoc8DfTHruCPUc
github.com/rogpeppe/fastuuid v0.0.0-20150106093220-6724a57986af/go.mod h1:XWv6SoW27p1b0cqNHllgS5HIMJraePCO15w5zCzIWYg= github.com/rogpeppe/fastuuid v0.0.0-20150106093220-6724a57986af/go.mod h1:XWv6SoW27p1b0cqNHllgS5HIMJraePCO15w5zCzIWYg=
github.com/rogpeppe/fastuuid v1.2.0/go.mod h1:jVj6XXZzXRy/MSR5jhDC/2q6DgLz+nrA6LYCDYWNEvQ= github.com/rogpeppe/fastuuid v1.2.0/go.mod h1:jVj6XXZzXRy/MSR5jhDC/2q6DgLz+nrA6LYCDYWNEvQ=
github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4= github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs=
github.com/rogpeppe/go-internal v1.10.0 h1:TMyTOH3F/DB16zRVcYyreMH6GnZZrwQVAoYjRBZyWFQ= github.com/rogpeppe/go-internal v1.10.0 h1:TMyTOH3F/DB16zRVcYyreMH6GnZZrwQVAoYjRBZyWFQ=
github.com/rogpeppe/go-internal v1.10.0/go.mod h1:UQnix2H7Ngw/k4C5ijL5+65zddjncjaFoBhdsK/akog=
github.com/rs/xid v1.2.1/go.mod h1:+uKXf+4Djp6Md1KODXJxgGQPKngRmWyn10oCKFzNHOQ= github.com/rs/xid v1.2.1/go.mod h1:+uKXf+4Djp6Md1KODXJxgGQPKngRmWyn10oCKFzNHOQ=
github.com/rs/zerolog v1.13.0/go.mod h1:YbFCdg8HfsridGWAh22vktObvhZbQsZXe4/zB0OKkWU= github.com/rs/zerolog v1.13.0/go.mod h1:YbFCdg8HfsridGWAh22vktObvhZbQsZXe4/zB0OKkWU=
github.com/rs/zerolog v1.15.0/go.mod h1:xYTKnLHcpfU2225ny5qZjxnj9NvkumZYjJHlAThCjNc= github.com/rs/zerolog v1.15.0/go.mod h1:xYTKnLHcpfU2225ny5qZjxnj9NvkumZYjJHlAThCjNc=
@ -526,8 +536,8 @@ github.com/sourcegraph/syntaxhighlight v0.0.0-20170531221838-bd320f5d308e/go.mod
github.com/spacemonkeygo/monkit/v3 v3.0.0-20191108235033-eacca33b3037/go.mod h1:JcK1pCbReQsOsMKF/POFSZCq7drXFybgGmbc27tuwes= github.com/spacemonkeygo/monkit/v3 v3.0.0-20191108235033-eacca33b3037/go.mod h1:JcK1pCbReQsOsMKF/POFSZCq7drXFybgGmbc27tuwes=
github.com/spacemonkeygo/monkit/v3 v3.0.4/go.mod h1:JcK1pCbReQsOsMKF/POFSZCq7drXFybgGmbc27tuwes= github.com/spacemonkeygo/monkit/v3 v3.0.4/go.mod h1:JcK1pCbReQsOsMKF/POFSZCq7drXFybgGmbc27tuwes=
github.com/spacemonkeygo/monkit/v3 v3.0.18/go.mod h1:kj1ViJhlyADa7DiA4xVnTuPA46lFKbM7mxQTrXCuJP4= github.com/spacemonkeygo/monkit/v3 v3.0.18/go.mod h1:kj1ViJhlyADa7DiA4xVnTuPA46lFKbM7mxQTrXCuJP4=
github.com/spacemonkeygo/monkit/v3 v3.0.20-0.20230419135619-fb89f20752cb h1:kWLHxcYDcloMFEJMngxuKh8wcLl9RjjeAN2a9AtTtCg= github.com/spacemonkeygo/monkit/v3 v3.0.22 h1:4/g8IVItBDKLdVnqrdHZrCVPpIrwDBzl1jrV0IHQHDU=
github.com/spacemonkeygo/monkit/v3 v3.0.20-0.20230419135619-fb89f20752cb/go.mod h1:kj1ViJhlyADa7DiA4xVnTuPA46lFKbM7mxQTrXCuJP4= github.com/spacemonkeygo/monkit/v3 v3.0.22/go.mod h1:XkZYGzknZwkD0AKUnZaSXhRiVTLCkq7CWVa3IsE72gA=
github.com/spacemonkeygo/monotime v0.0.0-20180824235756-e3f48a95f98a/go.mod h1:ul4bvvnCOPZgq8w0nTkSmWVg/hauVpFS97Am1YM1XXo= github.com/spacemonkeygo/monotime v0.0.0-20180824235756-e3f48a95f98a/go.mod h1:ul4bvvnCOPZgq8w0nTkSmWVg/hauVpFS97Am1YM1XXo=
github.com/spacemonkeygo/spacelog v0.0.0-20180420211403-2296661a0572 h1:RC6RW7j+1+HkWaX/Yh71Ee5ZHaHYt7ZP4sQgUrm6cDU= github.com/spacemonkeygo/spacelog v0.0.0-20180420211403-2296661a0572 h1:RC6RW7j+1+HkWaX/Yh71Ee5ZHaHYt7ZP4sQgUrm6cDU=
github.com/spacemonkeygo/spacelog v0.0.0-20180420211403-2296661a0572/go.mod h1:w0SWMsp6j9O/dk4/ZpIhL+3CkG8ofA2vuv7k+ltqUMc= github.com/spacemonkeygo/spacelog v0.0.0-20180420211403-2296661a0572/go.mod h1:w0SWMsp6j9O/dk4/ZpIhL+3CkG8ofA2vuv7k+ltqUMc=
@ -565,10 +575,10 @@ github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU= github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
github.com/stretchr/testify v1.8.2 h1:+h33VjcLVPDHtOdpUCuF+7gSuG3yGIftsP1YvFihtJ8= github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk=
github.com/stretchr/testify v1.8.2/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
github.com/stripe/stripe-go/v72 v72.90.0 h1:fvJ/aL1rHHWRj5buuayb/2ufJued1UR1HEVavsoZoFs= github.com/stripe/stripe-go/v75 v75.8.0 h1:kXdHvihp03v64L0C+xXGjolsdzdOmCqwKLnK2wA6bio=
github.com/stripe/stripe-go/v72 v72.90.0/go.mod h1:QwqJQtduHubZht9mek5sds9CtQcKFdsykV9ZepRWwo0= github.com/stripe/stripe-go/v75 v75.8.0/go.mod h1:wT44gah+eCY8Z0aSpY/vQlYYbicU9uUAbAqdaUxxDqE=
github.com/subosito/gotenv v1.2.0 h1:Slr1R9HxAlEKefgq5jn9U+DnETlIUa6HfgEzj0g5d7s= github.com/subosito/gotenv v1.2.0 h1:Slr1R9HxAlEKefgq5jn9U+DnETlIUa6HfgEzj0g5d7s=
github.com/subosito/gotenv v1.2.0/go.mod h1:N0PQaV/YGNqwC0u51sEeR/aUtSLEXKX9iv69rRypqCw= github.com/subosito/gotenv v1.2.0/go.mod h1:N0PQaV/YGNqwC0u51sEeR/aUtSLEXKX9iv69rRypqCw=
github.com/tarm/serial v0.0.0-20180830185346-98f6abe2eb07/go.mod h1:kDXzergiv9cbyO7IOYJZWg1U88JhDg3PB6klq9Hg2pA= github.com/tarm/serial v0.0.0-20180830185346-98f6abe2eb07/go.mod h1:kDXzergiv9cbyO7IOYJZWg1U88JhDg3PB6klq9Hg2pA=
@ -600,8 +610,6 @@ github.com/vbauerster/mpb/v8 v8.4.0 h1:Jq2iNA7T6SydpMVOwaT+2OBWlXS9Th8KEvBqeu5ee
github.com/vbauerster/mpb/v8 v8.4.0/go.mod h1:vjp3hSTuCtR+x98/+2vW3eZ8XzxvGoP8CPseHMhiPyc= github.com/vbauerster/mpb/v8 v8.4.0/go.mod h1:vjp3hSTuCtR+x98/+2vW3eZ8XzxvGoP8CPseHMhiPyc=
github.com/viant/assertly v0.4.8/go.mod h1:aGifi++jvCrUaklKEKT0BU95igDNaqkvz+49uaYMPRU= github.com/viant/assertly v0.4.8/go.mod h1:aGifi++jvCrUaklKEKT0BU95igDNaqkvz+49uaYMPRU=
github.com/viant/toolbox v0.24.0/go.mod h1:OxMCG57V0PXuIP2HNQrtJf2CjqdmbrOx5EkMILuUhzM= github.com/viant/toolbox v0.24.0/go.mod h1:OxMCG57V0PXuIP2HNQrtJf2CjqdmbrOx5EkMILuUhzM=
github.com/vivint/infectious v0.0.0-20200605153912-25a574ae18a3 h1:zMsHhfK9+Wdl1F7sIKLyx3wrOFofpb3rWFbA4HgcK5k=
github.com/vivint/infectious v0.0.0-20200605153912-25a574ae18a3/go.mod h1:R0Gbuw7ElaGSLOZUSwBm/GgVwMd30jWxBDdAyMOeTuc=
github.com/xeipuuv/gojsonpointer v0.0.0-20180127040702-4e3ac2762d5f h1:J9EGpcZtP0E/raorCMxlFGSTBrsSlaDGf3jU/qvAE2c= github.com/xeipuuv/gojsonpointer v0.0.0-20180127040702-4e3ac2762d5f h1:J9EGpcZtP0E/raorCMxlFGSTBrsSlaDGf3jU/qvAE2c=
github.com/xeipuuv/gojsonpointer v0.0.0-20180127040702-4e3ac2762d5f/go.mod h1:N2zxlSyiKSe5eX1tZViRH5QA0qijqEDrYZiPEAiq3wU= github.com/xeipuuv/gojsonpointer v0.0.0-20180127040702-4e3ac2762d5f/go.mod h1:N2zxlSyiKSe5eX1tZViRH5QA0qijqEDrYZiPEAiq3wU=
github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415 h1:EzJWgHovont7NscjpAxXsDA8S8BMYve8Y5+7cuRE7R0= github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415 h1:EzJWgHovont7NscjpAxXsDA8S8BMYve8Y5+7cuRE7R0=
@ -644,6 +652,8 @@ github.com/zeebo/errs/v2 v2.0.3 h1:WwqAmopgot4ZC+CgIveP+H91Nf78NDEGWjtAXen45Hw=
github.com/zeebo/errs/v2 v2.0.3/go.mod h1:OKmvVZt4UqpyJrYFykDKm168ZquJ55pbbIVUICNmLN0= github.com/zeebo/errs/v2 v2.0.3/go.mod h1:OKmvVZt4UqpyJrYFykDKm168ZquJ55pbbIVUICNmLN0=
github.com/zeebo/float16 v0.1.0 h1:kRqxv5og6z1emEyz5FpW0/BVHe5VfxEAw6b1ljCZlUc= github.com/zeebo/float16 v0.1.0 h1:kRqxv5og6z1emEyz5FpW0/BVHe5VfxEAw6b1ljCZlUc=
github.com/zeebo/float16 v0.1.0/go.mod h1:fssGvvXu+XS8MH57cKmyrLB/cqioYeYX/2mXCN3a5wo= github.com/zeebo/float16 v0.1.0/go.mod h1:fssGvvXu+XS8MH57cKmyrLB/cqioYeYX/2mXCN3a5wo=
github.com/zeebo/goof v0.0.0-20230830143729-8a73f2ee257d h1:BcGKO/7ni6YuQHLTEy5I9ujNb7Z3Xw5edcQRpZnCwSg=
github.com/zeebo/goof v0.0.0-20230830143729-8a73f2ee257d/go.mod h1:nbQ8jtLiWGVGehuiqVKJp/Oc9FnzA56AZ0tG/srGTGY=
github.com/zeebo/incenc v0.0.0-20180505221441-0d92902eec54 h1:+cwNE5KJ3pika4HuzmDHkDlK5myo0G9Sv+eO7WWxnUQ= github.com/zeebo/incenc v0.0.0-20180505221441-0d92902eec54 h1:+cwNE5KJ3pika4HuzmDHkDlK5myo0G9Sv+eO7WWxnUQ=
github.com/zeebo/incenc v0.0.0-20180505221441-0d92902eec54/go.mod h1:EI8LcOBDlSL3POyqwC1eJhOYlMBMidES+613EtmmT5w= github.com/zeebo/incenc v0.0.0-20180505221441-0d92902eec54/go.mod h1:EI8LcOBDlSL3POyqwC1eJhOYlMBMidES+613EtmmT5w=
github.com/zeebo/ini v0.0.0-20210514163846-cc8fbd8d9599 h1:aYOFLPl7mY7PFFuLuYoBqlP46yJ7rZONGlXMS4/6QpA= github.com/zeebo/ini v0.0.0-20210514163846-cc8fbd8d9599 h1:aYOFLPl7mY7PFFuLuYoBqlP46yJ7rZONGlXMS4/6QpA=
@ -654,6 +664,8 @@ github.com/zeebo/pcg v1.0.1 h1:lyqfGeWiv4ahac6ttHs+I5hwtH/+1mrhlCtVNQM2kHo=
github.com/zeebo/pcg v1.0.1/go.mod h1:09F0S9iiKrwn9rlI5yjLkmrug154/YRW6KnnXVDM/l4= github.com/zeebo/pcg v1.0.1/go.mod h1:09F0S9iiKrwn9rlI5yjLkmrug154/YRW6KnnXVDM/l4=
github.com/zeebo/structs v1.0.3-0.20230601144555-f2db46069602 h1:nMxsvi3pTJapmPpdShLdCO8sbCqd8XkjKYMssSJrfiM= github.com/zeebo/structs v1.0.3-0.20230601144555-f2db46069602 h1:nMxsvi3pTJapmPpdShLdCO8sbCqd8XkjKYMssSJrfiM=
github.com/zeebo/structs v1.0.3-0.20230601144555-f2db46069602/go.mod h1:hthZGQud7FXSu0Rd7Q6LRMmJ2pvvBvCkZ/LAmpkn5u4= github.com/zeebo/structs v1.0.3-0.20230601144555-f2db46069602/go.mod h1:hthZGQud7FXSu0Rd7Q6LRMmJ2pvvBvCkZ/LAmpkn5u4=
github.com/zeebo/sudo v1.0.2 h1:6RpQNYeWtd7ycPwYSRgceNdbjodamyyuapNB8mQ1V0M=
github.com/zeebo/sudo v1.0.2/go.mod h1:bO8DB2LXZchv4WMBzo1sCYp24BxAtwa0Lp0XTXU3cU4=
github.com/zenazn/goji v0.9.0/go.mod h1:7S9M489iMyHBNxwZnk9/EHS098H4/F6TATF2mIxtB1Q= github.com/zenazn/goji v0.9.0/go.mod h1:7S9M489iMyHBNxwZnk9/EHS098H4/F6TATF2mIxtB1Q=
github.com/zyedidia/generic v1.2.1 h1:Zv5KS/N2m0XZZiuLS82qheRG4X1o5gsWreGb0hR7XDc= github.com/zyedidia/generic v1.2.1 h1:Zv5KS/N2m0XZZiuLS82qheRG4X1o5gsWreGb0hR7XDc=
github.com/zyedidia/generic v1.2.1/go.mod h1:ly2RBz4mnz1yeuVbQA/VFwGjK3mnHGRj1JuoG336Bis= github.com/zyedidia/generic v1.2.1/go.mod h1:ly2RBz4mnz1yeuVbQA/VFwGjK3mnHGRj1JuoG336Bis=
@ -672,6 +684,8 @@ go.uber.org/atomic v1.5.0/go.mod h1:sABNBOSYdrvTF6hTgEIbc7YasKWGhgEQZyfxyTvoXHQ=
go.uber.org/atomic v1.6.0/go.mod h1:sABNBOSYdrvTF6hTgEIbc7YasKWGhgEQZyfxyTvoXHQ= go.uber.org/atomic v1.6.0/go.mod h1:sABNBOSYdrvTF6hTgEIbc7YasKWGhgEQZyfxyTvoXHQ=
go.uber.org/atomic v1.7.0 h1:ADUqmZGgLDDfbSL9ZmPxKTybcoEYHgpYfELNoN+7hsw= go.uber.org/atomic v1.7.0 h1:ADUqmZGgLDDfbSL9ZmPxKTybcoEYHgpYfELNoN+7hsw=
go.uber.org/atomic v1.7.0/go.mod h1:fEN4uk6kAWBTFdckzkM89CLk9XfWZrxpCo0nPH17wJc= go.uber.org/atomic v1.7.0/go.mod h1:fEN4uk6kAWBTFdckzkM89CLk9XfWZrxpCo0nPH17wJc=
go.uber.org/mock v0.3.0 h1:3mUxI1No2/60yUYax92Pt8eNOEecx2D3lcXZh2NEZJo=
go.uber.org/mock v0.3.0/go.mod h1:a6FSlNadKUHUa9IP5Vyt1zh4fC7uAwxMutEAscFbkZc=
go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/0= go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/0=
go.uber.org/multierr v1.3.0/go.mod h1:VgVr7evmIr6uPjLBxg28wmKNXyqE9akIJ5XnfpiKl+4= go.uber.org/multierr v1.3.0/go.mod h1:VgVr7evmIr6uPjLBxg28wmKNXyqE9akIJ5XnfpiKl+4=
go.uber.org/multierr v1.5.0/go.mod h1:FeouvMocqHpRaaGuG9EjoKcStLC43Zu/fmqdUMPcKYU= go.uber.org/multierr v1.5.0/go.mod h1:FeouvMocqHpRaaGuG9EjoKcStLC43Zu/fmqdUMPcKYU=
@ -705,8 +719,8 @@ golang.org/x/crypto v0.0.0-20210711020723-a769d52b0f97/go.mod h1:GvvjBRRGRdwPK5y
golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/crypto v0.0.0-20220314234659-1baeb1ce4c0b/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4= golang.org/x/crypto v0.0.0-20220314234659-1baeb1ce4c0b/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
golang.org/x/crypto v0.0.0-20220525230936-793ad666bf5e/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4= golang.org/x/crypto v0.0.0-20220525230936-793ad666bf5e/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
golang.org/x/crypto v0.7.0 h1:AvwMYaRytfdeVt3u6mLaxYtErKYjxA2OXjJ1HHq6t3A= golang.org/x/crypto v0.12.0 h1:tFM/ta59kqch6LlvYnPa0yx5a83cL2nHflFhYKvv9Yk=
golang.org/x/crypto v0.7.0/go.mod h1:pYwdfH91IfpZVANVyUOhSIPZaFoJGxTFbZhFTx+dXZU= golang.org/x/crypto v0.12.0/go.mod h1:NF0Gs7EO5K4qLn+Ylc+fih8BSTeIjAP05siRnAh98yw=
golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8= golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8=
@ -736,8 +750,8 @@ golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.4.2/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.4.2/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.6.0-dev.0.20220106191415-9b9b3d81d5e3/go.mod h1:3p9vT2HGsQu2K1YbXdKPJLVgG5VJdoTa1poYQBtP1AY= golang.org/x/mod v0.6.0-dev.0.20220106191415-9b9b3d81d5e3/go.mod h1:3p9vT2HGsQu2K1YbXdKPJLVgG5VJdoTa1poYQBtP1AY=
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4= golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
golang.org/x/mod v0.8.0 h1:LUYupSeNrTNCGzR/hVBk2NHZO4hXcVaW1k4Qx7rjPx8= golang.org/x/mod v0.11.0 h1:bUO06HqtnRcc/7l71XBe4WcqTZ+3AH1J59zWDDwLKgU=
golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs= golang.org/x/mod v0.11.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
@ -769,12 +783,13 @@ golang.org/x/net v0.0.0-20201110031124-69a78807bb2b/go.mod h1:sp8m0HH+o8qH0wwXwY
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg= golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4/go.mod h1:p54w0d4576C0XHj96bSt6lcn1PtDYWL6XObtHCRCNQM= golang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4/go.mod h1:p54w0d4576C0XHj96bSt6lcn1PtDYWL6XObtHCRCNQM=
golang.org/x/net v0.0.0-20210428140749-89ef3d95e781/go.mod h1:OJAsFXCWl8Ukc7SiCT/9KSuxbyM7479/AVlXFRxuMCk= golang.org/x/net v0.0.0-20210428140749-89ef3d95e781/go.mod h1:OJAsFXCWl8Ukc7SiCT/9KSuxbyM7479/AVlXFRxuMCk=
golang.org/x/net v0.0.0-20210520170846-37e1c6afe023/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20211015210444-4f30a5c0130f/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= golang.org/x/net v0.0.0-20211015210444-4f30a5c0130f/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20220526153639-5463443f8c37/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c= golang.org/x/net v0.0.0-20220526153639-5463443f8c37/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c= golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
golang.org/x/net v0.9.0 h1:aWJ/m6xSmxWBx+V0XRHTlrYrPG56jKsLdTFmsSsCzOM= golang.org/x/net v0.10.0 h1:X2//UzNDwYmtCLn7To6G58Wr6f5ahEAQgKNzv9Y951M=
golang.org/x/net v0.9.0/go.mod h1:d48xBJpPfHeWQsugry2m+kC02ZBRGRgulfHnEXEuWns= golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20181017192945-9dcd33a902f4/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= golang.org/x/oauth2 v0.0.0-20181017192945-9dcd33a902f4/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20181203162652-d668ce993890/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= golang.org/x/oauth2 v0.0.0-20181203162652-d668ce993890/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
@ -794,8 +809,8 @@ golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJ
golang.org/x/sync v0.0.0-20210220032951-036812b2e83c/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20210220032951-036812b2e83c/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20220819030929-7fc1605a5dde/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20220819030929-7fc1605a5dde/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.1.0 h1:wsuoTGHzEhffawBOhz5CYhcrV4IdKZbEyZjBMuTp12o= golang.org/x/sync v0.3.0 h1:ftCYgMx6zT/asHUrPw8BLLscYtGznsLAnjq5RH9P66E=
golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.3.0/go.mod h1:FU7BRWz2tNW+3quACPkgCx/L+uEAv1htQ0V83Z9Rj+Y=
golang.org/x/sys v0.0.0-20180823144017-11551d06cbcc/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20180823144017-11551d06cbcc/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
@ -821,7 +836,6 @@ golang.org/x/sys v0.0.0-20190904154756-749cb33beabd/go.mod h1:h1NjWce9XRLGQEsW7w
golang.org/x/sys v0.0.0-20191005200804-aed5e4c7ecf9/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20191005200804-aed5e4c7ecf9/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191026070338-33540a1f6037/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20191026070338-33540a1f6037/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191120155948-bd437916bb0e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20191120155948-bd437916bb0e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191224085550-c709ea063b76/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200116001909-b77594299b42/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200116001909-b77594299b42/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200202164722-d101bd2416d5/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200202164722-d101bd2416d5/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
@ -841,13 +855,13 @@ golang.org/x/sys v0.0.0-20211019181941-9d821ace8654/go.mod h1:oPkhp1MJrh7nUepCBc
golang.org/x/sys v0.0.0-20220412211240-33da011f77ad/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220412211240-33da011f77ad/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.7.0 h1:3jlCCIQZPdOYu1h8BkNvLz8Kgwtae2cagcG/VamtZRU= golang.org/x/sys v0.13.0 h1:Af8nKPmuFypiUBjVoU9V20FiaFXOcuZI21p0ycVYYGE=
golang.org/x/sys v0.7.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.13.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/term v0.0.0-20201117132131-f5c789dd3221/go.mod h1:Nr5EML6q2oocZ2LXRh80K7BxOlk5/8JxuGnuhpl+muw= golang.org/x/term v0.0.0-20201117132131-f5c789dd3221/go.mod h1:Nr5EML6q2oocZ2LXRh80K7BxOlk5/8JxuGnuhpl+muw=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/term v0.7.0 h1:BEvjmm5fURWqcfbSKTdpkDXYBrUS1c0m8agp14W48vQ= golang.org/x/term v0.11.0 h1:F9tnn/DA/Im8nCwm+fX+1/eBwi4qFjRT++MhtVC4ZX0=
golang.org/x/term v0.7.0/go.mod h1:P32HKFT3hSsZrRxla30E9HqToFYAQPCMs/zFMBUFqPY= golang.org/x/term v0.11.0/go.mod h1:zC9APTIj3jG3FdV/Ons+XE1riIZXG4aZ4GTHiPZJPIU=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk= golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
@ -856,8 +870,8 @@ golang.org/x/text v0.3.4/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
golang.org/x/text v0.3.8/go.mod h1:E6s5w1FMmriuDzIBO73fBruAKo1PCIq6d2Q6DHfQ8WQ= golang.org/x/text v0.3.8/go.mod h1:E6s5w1FMmriuDzIBO73fBruAKo1PCIq6d2Q6DHfQ8WQ=
golang.org/x/text v0.9.0 h1:2sjJmO8cDvYveuX97RDLsxlyUxLl+GHoLxBiRdHllBE= golang.org/x/text v0.12.0 h1:k+n5B8goJNdU7hSvEtMUz3d1Q6D/XW4COJSJR6fN0mc=
golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8= golang.org/x/text v0.12.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE=
golang.org/x/time v0.0.0-20180412165947-fbb02b2291d2/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20180412165947-fbb02b2291d2/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
@ -896,8 +910,8 @@ golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4f
golang.org/x/tools v0.1.1/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk= golang.org/x/tools v0.1.1/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk=
golang.org/x/tools v0.1.10/go.mod h1:Uh6Zz+xoGYZom868N8YTex3t7RhtHDBrE8Gzo9bV56E= golang.org/x/tools v0.1.10/go.mod h1:Uh6Zz+xoGYZom868N8YTex3t7RhtHDBrE8Gzo9bV56E=
golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc= golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
golang.org/x/tools v0.6.0 h1:BOw41kyTf3PuCW1pVQf8+Cyg8pMlkYB1oo9iJ6D/lKM= golang.org/x/tools v0.9.1 h1:8WMNJAz3zrtPmnYC7ISf5dEn3MT0gY7jBJfw27yrrLo=
golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU= golang.org/x/tools v0.9.1/go.mod h1:owI94Op576fPu3cIGQeHs3joujW/2Oc6MtlxbF5dfNc=
golang.org/x/xerrors v0.0.0-20190410155217-1f06c39b4373/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20190410155217-1f06c39b4373/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20190513163551-3ee3066db522/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20190513163551-3ee3066db522/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
@ -1013,16 +1027,18 @@ rsc.io/binaryregexp v0.2.0/go.mod h1:qTv7/COck+e2FymRvadv62gMdZztPaShugOCi3I+8D8
sourcegraph.com/sourcegraph/go-diff v0.5.0/go.mod h1:kuch7UrkMzY0X+p9CRK03kfuPQ2zzQcaEFbx8wA8rck= sourcegraph.com/sourcegraph/go-diff v0.5.0/go.mod h1:kuch7UrkMzY0X+p9CRK03kfuPQ2zzQcaEFbx8wA8rck=
sourcegraph.com/sqs/pbtypes v0.0.0-20180604144634-d3ebe8f20ae4/go.mod h1:ketZ/q3QxT9HOBeFhu6RdvsftgpsbFHBF5Cas6cDKZ0= sourcegraph.com/sqs/pbtypes v0.0.0-20180604144634-d3ebe8f20ae4/go.mod h1:ketZ/q3QxT9HOBeFhu6RdvsftgpsbFHBF5Cas6cDKZ0=
storj.io/common v0.0.0-20220719163320-cd2ef8e1b9b0/go.mod h1:mCYV6Ud5+cdbuaxdPD5Zht/HYaIn0sffnnws9ErkrMQ= storj.io/common v0.0.0-20220719163320-cd2ef8e1b9b0/go.mod h1:mCYV6Ud5+cdbuaxdPD5Zht/HYaIn0sffnnws9ErkrMQ=
storj.io/common v0.0.0-20230602145716-d6ea82d58b3d h1:AXdJxmg4Jqdz1nmogSrImKOHAU+bn8JCy8lHYnTwP0Y= storj.io/common v0.0.0-20231130134106-1fa84867e323 h1:0+vWHYPJyjZABb8Qyj1H2tCqpvyXMrN0GwTWu7vZ9nA=
storj.io/common v0.0.0-20230602145716-d6ea82d58b3d/go.mod h1:zu2L8WdpvfIBrCbBTgPsz4qhHSArYSiDgRcV1RLlIF8= storj.io/common v0.0.0-20231130134106-1fa84867e323/go.mod h1:qjHfzW5RlGg5z04CwIEjJd1eQ3HCGhUNtxZ6K/W7yqM=
storj.io/drpc v0.0.32/go.mod h1:6rcOyR/QQkSTX/9L5ZGtlZaE2PtXTTZl8d+ulSeeYEg= storj.io/drpc v0.0.32/go.mod h1:6rcOyR/QQkSTX/9L5ZGtlZaE2PtXTTZl8d+ulSeeYEg=
storj.io/drpc v0.0.33 h1:yCGZ26r66ZdMP0IcTYsj7WDAUIIjzXk6DJhbhvt9FHI= storj.io/drpc v0.0.33 h1:yCGZ26r66ZdMP0IcTYsj7WDAUIIjzXk6DJhbhvt9FHI=
storj.io/drpc v0.0.33/go.mod h1:vR804UNzhBa49NOJ6HeLjd2H3MakC1j5Gv8bsOQT6N4= storj.io/drpc v0.0.33/go.mod h1:vR804UNzhBa49NOJ6HeLjd2H3MakC1j5Gv8bsOQT6N4=
storj.io/monkit-jaeger v0.0.0-20220915074555-d100d7589f41 h1:SVuEocEhZfFc13J1AmlVLitdGXTVrvmbzN4Z9C9Ms40= storj.io/infectious v0.0.2 h1:rGIdDC/6gNYAStsxsZU79D/MqFjNyJc1tsyyj9sTl7Q=
storj.io/monkit-jaeger v0.0.0-20220915074555-d100d7589f41/go.mod h1:iK+dmHZZXQlW7ahKdNSOo+raMk5BDL2wbD62FIeXLWs= storj.io/infectious v0.0.2/go.mod h1:QEjKKww28Sjl1x8iDsjBpOM4r1Yp8RsowNcItsZJ1Vs=
storj.io/picobuf v0.0.1 h1:ekEvxSQCbEjTVIi/qxj2za13SJyfRE37yE30IBkZeT0= storj.io/monkit-jaeger v0.0.0-20230707083646-f15e6e8b7e8c h1:92Hl7mBzjfMNNkkO3uVp62ZC8yZuBNcz20EVcKNzpkQ=
storj.io/picobuf v0.0.1/go.mod h1:7ZTAMs6VesgTHbbhFU79oQ9hDaJ+MD4uoFQZ1P4SEz0= storj.io/monkit-jaeger v0.0.0-20230707083646-f15e6e8b7e8c/go.mod h1:iK+dmHZZXQlW7ahKdNSOo+raMk5BDL2wbD62FIeXLWs=
storj.io/private v0.0.0-20230627140631-807a2f00d0e1 h1:O2+Xjq8H4TKad2cnhvjitK3BtwkGtJ2TfRCHOIN8e7w= storj.io/picobuf v0.0.2-0.20230906122608-c4ba17033c6c h1:or/DtG5uaZpzimL61ahlgAA+MTYn/U3txz4fe+XBFUg=
storj.io/private v0.0.0-20230627140631-807a2f00d0e1/go.mod h1:mfdHEaAcTARpd4/Hc6N5uxwB1ZG3jtPdVlle57xzQxQ= storj.io/picobuf v0.0.2-0.20230906122608-c4ba17033c6c/go.mod h1:JCuc3C0gzCJHQ4J6SOx/Yjg+QTpX0D+Fvs5H46FETCk=
storj.io/uplink v1.10.1-0.20230626081029-035890d408c2 h1:XnJR9egrqvAqx5oCRu2b13ubK0iu0qTX12EAa6lAPhg= storj.io/private v0.0.0-20231127092015-c439a594bc1d h1:snE4Ec2k4bLNRsNq5YcKH6njS56zF30SR8u4Fgeksy4=
storj.io/uplink v1.10.1-0.20230626081029-035890d408c2/go.mod h1:cDlpDWGJykXfYE7NtO1EeArGFy12K5Xj8pV8ufpUCKE= storj.io/private v0.0.0-20231127092015-c439a594bc1d/go.mod h1:vLbKaAmrBdkrFd8ZvTgNUJ+kLKl25Y4kkwii7K2gWMI=
storj.io/uplink v1.12.3-0.20231130143633-4a092fa01b98 h1:EZ8MPk01yvDqwP8x2oI5Q3zkE4ef6K+GXpI6kJjBfxY=
storj.io/uplink v1.12.3-0.20231130143633-4a092fa01b98/go.mod h1:w+dXLZ8X3vtK3xis9jsMiBS0bzw4kU5foo5GOsIW7QM=

View File

@ -128,7 +128,6 @@ storj.io/storj/satellite/repair/repairer."repair_too_many_nodes_failed" Meter
storj.io/storj/satellite/repair/repairer."repair_unnecessary" Meter storj.io/storj/satellite/repair/repairer."repair_unnecessary" Meter
storj.io/storj/satellite/repair/repairer."repairer_segments_below_min_req" Counter storj.io/storj/satellite/repair/repairer."repairer_segments_below_min_req" Counter
storj.io/storj/satellite/repair/repairer."segment_deleted_before_repair" Meter storj.io/storj/satellite/repair/repairer."segment_deleted_before_repair" Meter
storj.io/storj/satellite/repair/repairer."segment_repair_count" IntVal
storj.io/storj/satellite/repair/repairer."segment_time_until_repair" IntVal storj.io/storj/satellite/repair/repairer."segment_time_until_repair" IntVal
storj.io/storj/satellite/repair/repairer."time_for_repair" FloatVal storj.io/storj/satellite/repair/repairer."time_for_repair" FloatVal
storj.io/storj/satellite/repair/repairer."time_since_checker_queue" FloatVal storj.io/storj/satellite/repair/repairer."time_since_checker_queue" FloatVal

View File

@ -202,10 +202,6 @@ func (obj *DB) Open(ctx context.Context) (*Tx, error) {
}, nil }, nil
} }
func (obj *DB) NewRx() *Rx {
return &Rx{db: obj}
}
func DeleteAll(ctx context.Context, db *DB) (int64, error) { func DeleteAll(ctx context.Context, db *DB) (int64, error) {
tx, err := db.Open(ctx) tx, err := db.Open(ctx)
if err != nil { if err != nil {
@ -1365,132 +1361,6 @@ func (obj *sqlite3Impl) deleteAll(ctx context.Context) (count int64, err error)
} }
type Rx struct {
db *DB
tx *Tx
}
func (rx *Rx) UnsafeTx(ctx context.Context) (unsafe_tx tagsql.Tx, err error) {
tx, err := rx.getTx(ctx)
if err != nil {
return nil, err
}
return tx.Tx, nil
}
func (rx *Rx) getTx(ctx context.Context) (tx *Tx, err error) {
if rx.tx == nil {
if rx.tx, err = rx.db.Open(ctx); err != nil {
return nil, err
}
}
return rx.tx, nil
}
func (rx *Rx) Rebind(s string) string {
return rx.db.Rebind(s)
}
func (rx *Rx) Commit() (err error) {
if rx.tx != nil {
err = rx.tx.Commit()
rx.tx = nil
}
return err
}
func (rx *Rx) Rollback() (err error) {
if rx.tx != nil {
err = rx.tx.Rollback()
rx.tx = nil
}
return err
}
func (rx *Rx) All_Node(ctx context.Context) (
rows []*Node, err error) {
var tx *Tx
if tx, err = rx.getTx(ctx); err != nil {
return
}
return tx.All_Node(ctx)
}
func (rx *Rx) Count_Node(ctx context.Context) (
count int64, err error) {
var tx *Tx
if tx, err = rx.getTx(ctx); err != nil {
return
}
return tx.Count_Node(ctx)
}
func (rx *Rx) Create_Node(ctx context.Context,
node_id Node_Id_Field,
node_name Node_Name_Field,
node_public_address Node_PublicAddress_Field,
node_api_secret Node_ApiSecret_Field) (
node *Node, err error) {
var tx *Tx
if tx, err = rx.getTx(ctx); err != nil {
return
}
return tx.Create_Node(ctx, node_id, node_name, node_public_address, node_api_secret)
}
func (rx *Rx) Delete_Node_By_Id(ctx context.Context,
node_id Node_Id_Field) (
deleted bool, err error) {
var tx *Tx
if tx, err = rx.getTx(ctx); err != nil {
return
}
return tx.Delete_Node_By_Id(ctx, node_id)
}
func (rx *Rx) Get_Node_By_Id(ctx context.Context,
node_id Node_Id_Field) (
node *Node, err error) {
var tx *Tx
if tx, err = rx.getTx(ctx); err != nil {
return
}
return tx.Get_Node_By_Id(ctx, node_id)
}
func (rx *Rx) Limited_Node(ctx context.Context,
limit int, offset int64) (
rows []*Node, err error) {
var tx *Tx
if tx, err = rx.getTx(ctx); err != nil {
return
}
return tx.Limited_Node(ctx, limit, offset)
}
func (rx *Rx) UpdateNoReturn_Node_By_Id(ctx context.Context,
node_id Node_Id_Field,
update Node_Update_Fields) (
err error) {
var tx *Tx
if tx, err = rx.getTx(ctx); err != nil {
return
}
return tx.UpdateNoReturn_Node_By_Id(ctx, node_id, update)
}
func (rx *Rx) Update_Node_By_Id(ctx context.Context,
node_id Node_Id_Field,
update Node_Update_Fields) (
node *Node, err error) {
var tx *Tx
if tx, err = rx.getTx(ctx); err != nil {
return
}
return tx.Update_Node_By_Id(ctx, node_id, update)
}
type Methods interface { type Methods interface {
All_Node(ctx context.Context) ( All_Node(ctx context.Context) (
rows []*Node, err error) rows []*Node, err error)

View File

@ -69,7 +69,9 @@ type DiskSpace struct {
Allocated int64 `json:"allocated"` Allocated int64 `json:"allocated"`
Used int64 `json:"usedPieces"` Used int64 `json:"usedPieces"`
Trash int64 `json:"usedTrash"` Trash int64 `json:"usedTrash"`
Free int64 `json:"free"` // Free is the actual amount of free space on the whole disk, not just allocated disk space, in bytes.
Free int64 `json:"free"`
// Available is the amount of free space on the allocated disk space, in bytes.
Available int64 `json:"available"` Available int64 `json:"available"`
Overused int64 `json:"overused"` Overused int64 `json:"overused"`
} }

View File

@ -5,23 +5,76 @@ package apigen
import ( import (
"fmt" "fmt"
"path"
"reflect" "reflect"
"regexp"
"sort"
"strings" "strings"
"unicode"
"unicode/utf8"
"storj.io/storj/private/api" "storj.io/storj/private/api"
) )
// groupNameAndPrefixRegExp guarantees that Group name and prefix are empty or have are only formed
// by ASCII letters or digits and not starting with a digit.
var groupNameAndPrefixRegExp = regexp.MustCompile(`^([A-Za-z][0-9A-Za-z]*)?$`)
// API represents specific API's configuration. // API represents specific API's configuration.
type API struct { type API struct {
Version string // Version is the corresponding version of the API.
Description string // It's concatenated to the BasePath, so assuming the base path is "/api" and the version is "v1"
PackageName string // the API paths will begin with `/api/v1`.
// When empty, the version doesn't appear in the API paths. If it starts or ends with one or more
// "/", they are stripped from the API endpoint paths.
Version string
Description string
// The package name to use for the Go generated code.
// If omitted, the last segment of the PackagePath will be used as the package name.
PackageName string
// The path of the package that will use the generated Go code.
// This is used to prevent the code from importing its own package.
PackagePath string
// BasePath is the base path for the API endpoints. E.g. "/api".
// It doesn't require to begin with "/". When empty, "/" is used.
BasePath string
Auth api.Auth Auth api.Auth
EndpointGroups []*EndpointGroup EndpointGroups []*EndpointGroup
} }
// Group adds new endpoints group to API. // Group adds new endpoints group to API.
// name must be `^([A-Z0-9]\w*)?$“
// prefix must be `^\w*$`.
func (a *API) Group(name, prefix string) *EndpointGroup { func (a *API) Group(name, prefix string) *EndpointGroup {
if !groupNameAndPrefixRegExp.MatchString(name) {
panic(
fmt.Sprintf(
"invalid name for API Endpoint Group. name must fulfill the regular expression %q, got %q",
groupNameAndPrefixRegExp,
name,
),
)
}
if !groupNameAndPrefixRegExp.MatchString(prefix) {
panic(
fmt.Sprintf(
"invalid prefix for API Endpoint Group %q. prefix must fulfill the regular expression %q, got %q",
name,
groupNameAndPrefixRegExp,
prefix,
),
)
}
for _, g := range a.EndpointGroups {
if strings.EqualFold(g.Name, name) {
panic(fmt.Sprintf("name has to be case-insensitive unique across all the groups. name=%q", name))
}
if strings.EqualFold(g.Prefix, prefix) {
panic(fmt.Sprintf("prefix has to be case-insensitive unique across all the groups. prefix=%q", prefix))
}
}
group := &EndpointGroup{ group := &EndpointGroup{
Name: name, Name: name,
Prefix: prefix, Prefix: prefix,
@ -32,6 +85,14 @@ func (a *API) Group(name, prefix string) *EndpointGroup {
return group return group
} }
func (a *API) endpointBasePath() string {
if strings.HasPrefix(a.BasePath, "/") {
return path.Join(a.BasePath, a.Version)
}
return "/" + path.Join(a.BasePath, a.Version)
}
// StringBuilder is an extension of strings.Builder that allows for writing formatted lines. // StringBuilder is an extension of strings.Builder that allows for writing formatted lines.
type StringBuilder struct{ strings.Builder } type StringBuilder struct{ strings.Builder }
@ -51,17 +112,6 @@ func getElementaryType(t reflect.Type) reflect.Type {
} }
} }
// filter returns a new slice of reflect.Type values that satisfy the given keep function.
func filter(types []reflect.Type, keep func(reflect.Type) bool) []reflect.Type {
filtered := make([]reflect.Type, 0, len(types))
for _, t := range types {
if keep(t) {
filtered = append(filtered, t)
}
}
return filtered
}
// isNillableType returns whether instances of the given type can be nil. // isNillableType returns whether instances of the given type can be nil.
func isNillableType(t reflect.Type) bool { func isNillableType(t reflect.Type) bool {
switch t.Kind() { switch t.Kind() {
@ -70,3 +120,100 @@ func isNillableType(t reflect.Type) bool {
} }
return false return false
} }
// isJSONOmittableType returns whether the "omitempty" JSON tag option works with struct fields of this type.
func isJSONOmittableType(t reflect.Type) bool {
switch t.Kind() {
case reflect.Array, reflect.Map, reflect.Slice, reflect.String,
reflect.Bool,
reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64,
reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Uintptr,
reflect.Float32, reflect.Float64,
reflect.Interface, reflect.Pointer:
return true
}
return false
}
func capitalize(s string) string {
r, size := utf8.DecodeRuneInString(s)
if size <= 0 {
return s
}
return string(unicode.ToTitle(r)) + s[size:]
}
func uncapitalize(s string) string {
r, size := utf8.DecodeRuneInString(s)
if size <= 0 {
return s
}
return string(unicode.ToLower(r)) + s[size:]
}
type typeAndName struct {
Type reflect.Type
Name string
}
func mapToSlice(typesAndNames map[reflect.Type]string) []typeAndName {
list := make([]typeAndName, 0, len(typesAndNames))
for t, n := range typesAndNames {
list = append(list, typeAndName{Type: t, Name: n})
}
sort.SliceStable(list, func(i, j int) bool {
return list[i].Name < list[j].Name
})
return list
}
// filter returns a new slice of typeAndName values that satisfy the given keep function.
func filter(types []typeAndName, keep func(typeAndName) bool) []typeAndName {
filtered := make([]typeAndName, 0, len(types))
for _, t := range types {
if keep(t) {
filtered = append(filtered, t)
}
}
return filtered
}
type jsonTagInfo struct {
FieldName string
OmitEmpty bool
Skip bool
}
func parseJSONTag(structType reflect.Type, field reflect.StructField) jsonTagInfo {
tag, ok := field.Tag.Lookup("json")
if !ok {
panic(fmt.Sprintf("(%s).%s missing json tag", structType.String(), field.Name))
}
options := strings.Split(tag, ",")
for i, opt := range options {
options[i] = strings.TrimSpace(opt)
}
fieldName := options[0]
if fieldName == "" {
panic(fmt.Sprintf("(%s).%s missing json field name", structType.String(), field.Name))
}
if fieldName == "-" && len(options) == 1 {
return jsonTagInfo{Skip: true}
}
info := jsonTagInfo{FieldName: fieldName}
for _, opt := range options[1:] {
if opt == "omitempty" {
info.OmitEmpty = isJSONOmittableType(field.Type)
break
}
}
return info
}

View File

@ -0,0 +1,118 @@
// Copyright (C) 2023 Storj Labs, Inc.
// See LICENSE for copying information.
package apigen
import (
"fmt"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestAPI_endpointBasePath(t *testing.T) {
cases := []struct {
version string
basePath string
expected string
}{
{version: "", basePath: "", expected: "/"},
{version: "v1", basePath: "", expected: "/v1"},
{version: "v0", basePath: "/", expected: "/v0"},
{version: "", basePath: "api", expected: "/api"},
{version: "v2", basePath: "api", expected: "/api/v2"},
{version: "v2", basePath: "/api", expected: "/api/v2"},
{version: "v2", basePath: "api/", expected: "/api/v2"},
{version: "v2", basePath: "/api/", expected: "/api/v2"},
{version: "/v3", basePath: "api", expected: "/api/v3"},
{version: "/v3/", basePath: "api", expected: "/api/v3"},
{version: "v3/", basePath: "api", expected: "/api/v3"},
{version: "//v3/", basePath: "api", expected: "/api/v3"},
{version: "v3///", basePath: "api", expected: "/api/v3"},
{version: "/v3///", basePath: "/api/test/", expected: "/api/test/v3"},
{version: "/v4.2", basePath: "api/test", expected: "/api/test/v4.2"},
{version: "/v4/2", basePath: "/api/test", expected: "/api/test/v4/2"},
}
for _, c := range cases {
t.Run(fmt.Sprintf("version:%s basePath: %s", c.version, c.basePath), func(t *testing.T) {
a := API{
Version: c.version,
BasePath: c.basePath,
}
assert.Equal(t, c.expected, a.endpointBasePath())
})
}
}
func TestAPI_Group(t *testing.T) {
t.Run("valid name and prefix", func(t *testing.T) {
api := API{}
require.NotPanics(t, func() {
api.Group("testName", "tName")
})
require.NotPanics(t, func() {
api.Group("TestName1", "TName1")
})
})
t.Run("invalid name", func(t *testing.T) {
api := API{}
require.Panics(t, func() {
api.Group("1testName", "tName")
})
require.Panics(t, func() {
api.Group("test-name", "tName")
})
})
t.Run("invalid prefix", func(t *testing.T) {
api := API{}
require.Panics(t, func() {
api.Group("testName", "5tName")
})
require.Panics(t, func() {
api.Group("testname", "t_name")
})
})
t.Run("group with repeated name", func(t *testing.T) {
api := API{}
require.NotPanics(t, func() {
api.Group("testName", "tName")
})
require.Panics(t, func() {
api.Group("TESTNAME", "tName2")
})
require.Panics(t, func() {
api.Group("testname", "tName3")
})
})
t.Run("group with repeated prefix", func(t *testing.T) {
api := API{}
require.NotPanics(t, func() {
api.Group("testName", "tName")
})
require.Panics(t, func() {
api.Group("testName2", "tname")
})
require.Panics(t, func() {
api.Group("testname3", "tnamE")
})
})
}

View File

@ -7,6 +7,7 @@ import (
"fmt" "fmt"
"os" "os"
"reflect" "reflect"
"regexp"
"strings" "strings"
"time" "time"
@ -34,14 +35,39 @@ func (api *API) generateDocumentation() string {
wf := func(format string, args ...any) { _, _ = fmt.Fprintf(&doc, format, args...) } wf := func(format string, args ...any) { _, _ = fmt.Fprintf(&doc, format, args...) }
wf("# API Docs\n\n") wf("# API Docs\n\n")
wf("**Description:** %s\n\n", api.Description) if api.Description != "" {
wf("**Version:** `%s`\n\n", api.Version) wf("**Description:** %s\n\n", api.Description)
}
if api.Version != "" {
wf("**Version:** `%s`\n\n", api.Version)
}
wf("<h2 id='list-of-endpoints'>List of Endpoints</h2>\n\n")
getEndpointLink := func(group, endpoint string) string {
fullName := group + "-" + endpoint
fullName = strings.ReplaceAll(fullName, " ", "-")
nonAlphanumericRegex := regexp.MustCompile(`[^a-zA-Z0-9-]+`)
fullName = nonAlphanumericRegex.ReplaceAllString(fullName, "")
return strings.ToLower(fullName)
}
for _, group := range api.EndpointGroups {
wf("* %s\n", group.Name)
for _, endpoint := range group.endpoints {
wf(" * [%s](#%s)\n", endpoint.Name, getEndpointLink(group.Name, endpoint.Name))
}
}
wf("\n")
for _, group := range api.EndpointGroups { for _, group := range api.EndpointGroups {
for _, endpoint := range group.endpoints { for _, endpoint := range group.endpoints {
wf("## %s\n\n", endpoint.Name) wf(
"<h3 id='%s'>%s (<a href='#list-of-endpoints'>go to full list</a>)</h3>\n\n",
getEndpointLink(group.Name, endpoint.Name),
endpoint.Name,
)
wf("%s\n\n", endpoint.Description) wf("%s\n\n", endpoint.Description)
wf("`%s /%s%s`\n\n", endpoint.Method, group.Prefix, endpoint.Path) wf("`%s %s/%s%s`\n\n", endpoint.Method, api.endpointBasePath(), group.Prefix, endpoint.Path)
if len(endpoint.QueryParams) > 0 { if len(endpoint.QueryParams) > 0 {
wf("**Query Params:**\n\n") wf("**Query Params:**\n\n")
@ -66,13 +92,13 @@ func (api *API) generateDocumentation() string {
requestType := reflect.TypeOf(endpoint.Request) requestType := reflect.TypeOf(endpoint.Request)
if requestType != nil { if requestType != nil {
wf("**Request body:**\n\n") wf("**Request body:**\n\n")
wf("```json\n%s\n```\n\n", getTypeNameRecursively(requestType, 0)) wf("```typescript\n%s\n```\n\n", getTypeNameRecursively(requestType, 0))
} }
responseType := reflect.TypeOf(endpoint.Response) responseType := reflect.TypeOf(endpoint.Response)
if responseType != nil { if responseType != nil {
wf("**Response body:**\n\n") wf("**Response body:**\n\n")
wf("```json\n%s\n```\n\n", getTypeNameRecursively(responseType, 0)) wf("```typescript\n%s\n```\n\n", getTypeNameRecursively(responseType, 0))
} }
} }
} }
@ -123,7 +149,6 @@ func getTypeNameRecursively(t reflect.Type, level int) string {
elemType := t.Elem() elemType := t.Elem()
if elemType.Kind() == reflect.Uint8 { // treat []byte as string in docs if elemType.Kind() == reflect.Uint8 { // treat []byte as string in docs
return prefix + "string" return prefix + "string"
} }
return fmt.Sprintf("%s[\n%s\n%s]\n", prefix, getTypeNameRecursively(elemType, level+1), prefix) return fmt.Sprintf("%s[\n%s\n%s]\n", prefix, getTypeNameRecursively(elemType, level+1), prefix)
case reflect.Struct: case reflect.Struct:
@ -132,7 +157,7 @@ func getTypeNameRecursively(t reflect.Type, level int) string {
if typeName != "unknown" { if typeName != "unknown" {
toReturn := typeName toReturn := typeName
if len(elaboration) > 0 { if len(elaboration) > 0 {
toReturn += " (" + elaboration + ")" toReturn += " // " + elaboration
} }
return toReturn return toReturn
} }
@ -140,9 +165,9 @@ func getTypeNameRecursively(t reflect.Type, level int) string {
var fields []string var fields []string
for i := 0; i < t.NumField(); i++ { for i := 0; i < t.NumField(); i++ {
field := t.Field(i) field := t.Field(i)
jsonTag := field.Tag.Get("json") jsonInfo := parseJSONTag(t, field)
if jsonTag != "" && jsonTag != "-" { if !jsonInfo.Skip {
fields = append(fields, prefix+"\t"+jsonTag+": "+getTypeNameRecursively(field.Type, level+1)) fields = append(fields, prefix+"\t"+jsonInfo.FieldName+": "+getTypeNameRecursively(field.Type, level+1))
} }
} }
return fmt.Sprintf("%s{\n%s\n%s}\n", prefix, strings.Join(fields, "\n"), prefix) return fmt.Sprintf("%s{\n%s\n%s}\n", prefix, strings.Join(fields, "\n"), prefix)
@ -150,7 +175,7 @@ func getTypeNameRecursively(t reflect.Type, level int) string {
typeName, elaboration := getDocType(t) typeName, elaboration := getDocType(t)
toReturn := typeName toReturn := typeName
if len(elaboration) > 0 { if len(elaboration) > 0 {
toReturn += " (" + elaboration + ")" toReturn += " // " + elaboration
} }
return toReturn return toReturn
} }

View File

@ -4,75 +4,256 @@
package apigen package apigen
import ( import (
"fmt"
"net/http" "net/http"
"reflect" "reflect"
"regexp"
"strings"
"time"
"unicode"
"github.com/zeebo/errs"
"storj.io/common/uuid"
)
var (
errsEndpoint = errs.Class("Endpoint")
goNameRegExp = regexp.MustCompile(`^[A-Z]\w*$`)
typeScriptNameRegExp = regexp.MustCompile(`^[a-z][a-zA-Z0-9_$]*$`)
) )
// Endpoint represents endpoint's configuration. // Endpoint represents endpoint's configuration.
//
// Passing an anonymous type to the fields that define the request or response will make the API
// generator to panic. Anonymous types aren't allowed such as named structs that have fields with
// direct or indirect of anonymous types, slices or arrays whose direct or indirect elements are of
// anonymous types.
type Endpoint struct { type Endpoint struct {
Name string // Name is a free text used to name the endpoint for documentation purpose.
Description string // It cannot be empty.
MethodName string Name string
RequestName string // Description is a free text to describe the endpoint for documentation purpose.
NoCookieAuth bool Description string
NoAPIAuth bool // GoName is an identifier used by the Go generator to generate specific server side code for this
Request interface{} // endpoint.
Response interface{} //
QueryParams []Param // It must start with an uppercase letter and fulfill the Go language specification for method
PathParams []Param // names (https://go.dev/ref/spec#MethodName).
// It cannot be empty.
GoName string
// TypeScriptName is an identifier used by the TypeScript generator to generate specific client
// code for this endpoint
//
// It must start with a lowercase letter and can only contains letters, digits, _, and $.
// It cannot be empty.
TypeScriptName string
// Request is the type that defines the format of the request body.
Request interface{}
// Response is the type that defines the format of the response body.
Response interface{}
// QueryParams is the list of query parameters that the endpoint accepts.
QueryParams []Param
// PathParams is the list of path parameters that appear in the path associated with this
// endpoint.
PathParams []Param
// ResponseMock is the data to use as a response for the generated mocks.
// It must be of the same type than Response.
// If a mock generator is called it must not be nil unless Response is nil.
ResponseMock interface{}
// Settings is the data to pass to the middleware handlers to adapt the generated
// code to this endpoints.
//
// Not all the middlware handlers need extra data. Some of them use this data to disable it in
// some endpoints.
Settings map[any]any
} }
// CookieAuth returns endpoint's cookie auth status. // Validate validates the endpoint fields values are correct according to the documented constraints.
func (e *Endpoint) CookieAuth() bool { func (e *Endpoint) Validate() error {
return !e.NoCookieAuth newErr := func(m string, a ...any) error {
e := fmt.Sprintf(". Endpoint: %s", e.Name)
m += e
return errsEndpoint.New(m, a...)
}
if e.Name == "" {
return newErr("Name cannot be empty")
}
if e.Description == "" {
return newErr("Description cannot be empty")
}
if !goNameRegExp.MatchString(e.GoName) {
return newErr("GoName doesn't match the regular expression %q", goNameRegExp)
}
if !typeScriptNameRegExp.MatchString(e.TypeScriptName) {
return newErr("TypeScriptName doesn't match the regular expression %q", typeScriptNameRegExp)
}
if e.Request != nil {
switch t := reflect.TypeOf(e.Request); t.Kind() {
case reflect.Invalid,
reflect.Complex64,
reflect.Complex128,
reflect.Chan,
reflect.Func,
reflect.Interface,
reflect.Map,
reflect.Pointer,
reflect.UnsafePointer:
return newErr("Request cannot be of a type %q", t.Kind())
case reflect.Array, reflect.Slice:
if t.Elem().Name() == "" {
return newErr("Request cannot be of %q of anonymous struct elements", t.Kind())
}
case reflect.Struct:
if t.Name() == "" {
return newErr("Request cannot be of an anonymous struct")
}
}
}
if e.Response != nil {
switch t := reflect.TypeOf(e.Response); t.Kind() {
case reflect.Invalid,
reflect.Complex64,
reflect.Complex128,
reflect.Chan,
reflect.Func,
reflect.Interface,
reflect.Map,
reflect.Pointer,
reflect.UnsafePointer:
return newErr("Response cannot be of a type %q", t.Kind())
case reflect.Array, reflect.Slice:
if t.Elem().Name() == "" {
return newErr("Response cannot be of %q of anonymous struct elements", t.Kind())
}
case reflect.Struct:
if t.Name() == "" {
return newErr("Response cannot be of an anonymous struct")
}
}
if e.ResponseMock != nil {
if m, r := reflect.TypeOf(e.ResponseMock), reflect.TypeOf(e.Response); m != r {
return newErr(
"ResponseMock isn't of the same type than Response. Have=%q Want=%q", m, r,
)
}
}
}
return nil
} }
// APIAuth returns endpoint's API auth status. // FullEndpoint represents endpoint with path and method.
func (e *Endpoint) APIAuth() bool { type FullEndpoint struct {
return !e.NoAPIAuth
}
// fullEndpoint represents endpoint with path and method.
type fullEndpoint struct {
Endpoint Endpoint
Path string Path string
Method string Method string
} }
// EndpointGroup represents endpoints group. // EndpointGroup represents endpoints group.
// You should always create a group using API.Group because it validates the field values to
// guarantee correct code generation.
type EndpointGroup struct { type EndpointGroup struct {
Name string // Name is the group name.
Prefix string //
endpoints []*fullEndpoint // Go generator uses it as part of type, functions, interfaces names, and in code comments.
// The casing is adjusted according where it's used.
//
// TypeScript generator uses it as part of types names for the API functionality of this group.
// The casing is adjusted according where it's used.
//
// Document generator uses as it is.
Name string
// Prefix is a prefix used for
//
// Go generator uses it as part of variables names, error messages, and the URL base path for the group.
// The casing is adjusted according where it's used, but for the URL base path, lowercase is used.
//
// TypeScript generator uses it for composing the URL base path (lowercase).
//
// Document generator uses as it is.
Prefix string
// Middleware is a list of additional processing of requests that apply to all the endpoints of this group.
Middleware []Middleware
// endpoints is the list of endpoints added to this group through the "HTTP method" methods (e.g.
// Get, Patch, etc.).
endpoints []*FullEndpoint
} }
// Get adds new GET endpoint to endpoints group. // Get adds new GET endpoint to endpoints group.
// It panics if path doesn't begin with '/'.
func (eg *EndpointGroup) Get(path string, endpoint *Endpoint) { func (eg *EndpointGroup) Get(path string, endpoint *Endpoint) {
eg.addEndpoint(path, http.MethodGet, endpoint) eg.addEndpoint(path, http.MethodGet, endpoint)
} }
// Patch adds new PATCH endpoint to endpoints group. // Patch adds new PATCH endpoint to endpoints group.
// It panics if path doesn't begin with '/'.
func (eg *EndpointGroup) Patch(path string, endpoint *Endpoint) { func (eg *EndpointGroup) Patch(path string, endpoint *Endpoint) {
eg.addEndpoint(path, http.MethodPatch, endpoint) eg.addEndpoint(path, http.MethodPatch, endpoint)
} }
// Post adds new POST endpoint to endpoints group. // Post adds new POST endpoint to endpoints group.
// It panics if path doesn't begin with '/'.
func (eg *EndpointGroup) Post(path string, endpoint *Endpoint) { func (eg *EndpointGroup) Post(path string, endpoint *Endpoint) {
eg.addEndpoint(path, http.MethodPost, endpoint) eg.addEndpoint(path, http.MethodPost, endpoint)
} }
// Delete adds new DELETE endpoint to endpoints group. // Delete adds new DELETE endpoint to endpoints group.
// It panics if path doesn't begin with '/'.
func (eg *EndpointGroup) Delete(path string, endpoint *Endpoint) { func (eg *EndpointGroup) Delete(path string, endpoint *Endpoint) {
eg.addEndpoint(path, http.MethodDelete, endpoint) eg.addEndpoint(path, http.MethodDelete, endpoint)
} }
// addEndpoint adds new endpoint to endpoints list. // addEndpoint adds new endpoint to endpoints list.
// It panics if:
// - path doesn't begin with '/'.
// - endpoint.Validate() returns an error.
// - An Endpoint with the same path and method already exists.
func (eg *EndpointGroup) addEndpoint(path, method string, endpoint *Endpoint) { func (eg *EndpointGroup) addEndpoint(path, method string, endpoint *Endpoint) {
ep := &fullEndpoint{*endpoint, path, method} if !strings.HasPrefix(path, "/") {
for i, e := range eg.endpoints { panic(
fmt.Sprintf(
"invalid path for method %q of EndpointGroup %q. path must start with slash, got %q",
method,
eg.Name,
path,
),
)
}
if err := endpoint.Validate(); err != nil {
panic(err)
}
ep := &FullEndpoint{*endpoint, path, method}
for _, e := range eg.endpoints {
if e.Path == path && e.Method == method { if e.Path == path && e.Method == method {
eg.endpoints[i] = ep panic(fmt.Sprintf("there is already an endpoint defined with path %q and method %q", path, method))
return }
if e.GoName == ep.GoName {
panic(
fmt.Sprintf("GoName %q is already used by the endpoint with path %q and method %q", e.GoName, e.Path, e.Method),
)
}
if e.TypeScriptName == ep.TypeScriptName {
panic(
fmt.Sprintf(
"TypeScriptName %q is already used by the endpoint with path %q and method %q",
e.TypeScriptName,
e.Path,
e.Method,
),
)
} }
} }
eg.endpoints = append(eg.endpoints, ep) eg.endpoints = append(eg.endpoints, ep)
@ -84,10 +265,176 @@ type Param struct {
Type reflect.Type Type reflect.Type
} }
// NewParam constructor which creates new Param entity by given name and type. // NewParam constructor which creates new Param entity by given name and type through instance.
//
// instance can only be a unsigned integer (of any size), string, uuid.UUID or time.Time, otherwise
// it panics.
func NewParam(name string, instance interface{}) Param { func NewParam(name string, instance interface{}) Param {
switch t := reflect.TypeOf(instance); t {
case reflect.TypeOf(uuid.UUID{}), reflect.TypeOf(time.Time{}):
default:
switch k := t.Kind(); k {
case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.String:
default:
panic(
fmt.Sprintf(
`Unsupported parameter, only types: %q, %q, string, and "unsigned numbers" are supported . Found type=%q, Kind=%q`,
reflect.TypeOf(uuid.UUID{}),
reflect.TypeOf(time.Time{}),
t,
k,
),
)
}
}
return Param{ return Param{
Name: name, Name: name,
Type: reflect.TypeOf(instance), Type: reflect.TypeOf(instance),
} }
} }
// Middleware allows to generate custom code that's executed at the beginning of the handler.
//
// The implementation must declare their dependencies through unexported struct fields which doesn't
// begin with underscore (_), except fields whose name is just underscore (the blank identifier).
// The API generator will add the import those dependencies and allow to pass them through the
// constructor parameters of the group handler implementation, except the fields named with the
// blank identifier that should be only used to import packages that the generated code needs.
//
// The limitation of using fields with the blank identifier as its names is that those packages
// must at least to export a type, hence, it isn't possible to import packages that only export
// constants or variables.
//
// Middleware implementation with the same struct field name and type will be handled as one
// parameter, so the dependency will be shared between them. If they have the same struct field
// name, but a different type, the API generator will panic.
// NOTE types are compared as [package].[type name], hence, package name collision are not handled
// and it will produce code that doesn't compile.
type Middleware interface {
// Generate generates the code that the API generator adds to a handler endpoint before calling
// the service.
//
// All the dependencies defined as struct fields of the implementation of this interface are
// available as fields of the struct handler. The generated code is executed inside of the methods
// of the struct handler, hence it has access to all its fields. The handler instance is available
// through the variable name h. For example:
//
// type middlewareImpl struct {
// log *zap.Logger // Import path: "go.uber.org/zap"
// auth api.Auth // Import path: "storj.io/storj/private/api"
// }
//
// The generated code can access to log and auth through h.log and h.auth.
//
// Each handler method where the code is executed has access to the following variables names:
// ctx of type context.Context, w of type http.ResponseWriter, and r of type *http.Request.
// Make sure to not declare variable with those names in the generated code unless that's wrapped
// in a scope.
Generate(api *API, group *EndpointGroup, ep *FullEndpoint) string
}
func middlewareImports(m any) []string {
imports := []string{}
middlewareWalkFields(m, func(f reflect.StructField) {
if p := f.Type.PkgPath(); p != "" {
imports = append(imports, p)
}
})
return imports
}
// middlewareFields returns the list of fields of a middleware implementation. It panics if m isn't
// a struct type, it has embedded fields, or it has unexported fields.
func middlewareFields(api *API, m any) []middlewareField {
fields := []middlewareField{}
middlewareWalkFields(m, func(f reflect.StructField) {
if f.Name == "_" {
return
}
psymbol := ""
t := f.Type
if t.Kind() == reflect.Pointer {
psymbol = "*"
t = f.Type.Elem()
}
typeref := psymbol + t.Name()
if p := t.PkgPath(); p != "" && p != api.PackagePath {
pn, _ := importPath(p).PkgName()
typeref = fmt.Sprintf("%s%s.%s", psymbol, pn, t.Name())
}
fields = append(fields, middlewareField{Name: f.Name, Type: typeref})
})
return fields
}
func middlewareWalkFields(m any, walk func(f reflect.StructField)) {
t := reflect.TypeOf(m)
if t.Kind() != reflect.Struct {
panic(fmt.Sprintf("middleware %q isn't a struct type", t.Name()))
}
for i := 0; i < t.NumField(); i++ {
f := t.FieldByIndex([]int{i})
if f.Anonymous {
panic(fmt.Sprintf("middleware %q has a embedded field %q", t.Name(), f.Name))
}
if f.Name != "_" {
// Disallow fields that begin with underscore.
if !unicode.IsLetter([]rune(f.Name)[0]) {
panic(
fmt.Sprintf(
"middleware %q has a field name beginning with no letter %q. Change it to begin with lower case letter",
t.Name(),
f.Name,
),
)
}
if unicode.IsUpper([]rune(f.Name)[0]) {
panic(
fmt.Sprintf(
"middleware %q has a field name beginning with upper case %q. Change it to begin with lower case",
t.Name(),
f.Name,
),
)
}
}
walk(f)
}
}
// middlewareField has the name of the field and type for adding to handler structs that the
// API generator generates during the generation phase.
type middlewareField struct {
// Name is the name of the field. It must fulfill Go identifiers specification
// https://go.dev/ref/spec#Identifiers
Name string
// Type is the type's name of the field.
Type string
}
// LoadSetting returns from endpoint.Settings the value assigned to key or
// returns defaultValue if the key doesn't exist.
//
// It panics if key doesn't have a value of the type T.
func LoadSetting[T any](key any, endpoint *FullEndpoint, defaultValue T) T {
v, ok := endpoint.Settings[key]
if !ok {
return defaultValue
}
vt, vtok := v.(T)
if !vtok {
panic(fmt.Sprintf("expected %T got %T", vt, v))
}
return vt
}

View File

@ -0,0 +1,290 @@
// Copyright (C) 2022 Storj Labs, Inc.
// See LICENSE for copying information.
package apigen
import (
"fmt"
"math/rand"
"net/http"
"reflect"
"strconv"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestEndpoint_Validate(t *testing.T) {
validEndpoint := Endpoint{
Name: "Test Endpoint",
Description: "This is an Endpoint purely for testing purposes",
GoName: "GenTest",
TypeScriptName: "genTest",
}
tcases := []struct {
testName string
endpointFn func() *Endpoint
errMsg string
}{
{
testName: "valid endpoint",
endpointFn: func() *Endpoint {
return &validEndpoint
},
},
{
testName: "empty name",
endpointFn: func() *Endpoint {
e := validEndpoint
e.Name = ""
return &e
},
errMsg: "Name cannot be empty",
},
{
testName: "empty description",
endpointFn: func() *Endpoint {
e := validEndpoint
e.Description = ""
return &e
},
errMsg: "Description cannot be empty",
},
{
testName: "empty Go name",
endpointFn: func() *Endpoint {
e := validEndpoint
e.GoName = ""
return &e
},
errMsg: "GoName doesn't match the regular expression",
},
{
testName: "no capitalized Go name ",
endpointFn: func() *Endpoint {
e := validEndpoint
e.GoName = "genTest"
return &e
},
errMsg: "GoName doesn't match the regular expression",
},
{
testName: "symbol in Go name",
endpointFn: func() *Endpoint {
e := validEndpoint
e.GoName = "GenTe$t"
return &e
},
errMsg: "GoName doesn't match the regular expression",
},
{
testName: "empty TypeScript name",
endpointFn: func() *Endpoint {
e := validEndpoint
e.TypeScriptName = ""
return &e
},
errMsg: "TypeScriptName doesn't match the regular expression",
},
{
testName: "capitalized TypeScript name ",
endpointFn: func() *Endpoint {
e := validEndpoint
e.TypeScriptName = "GenTest"
return &e
},
errMsg: "TypeScriptName doesn't match the regular expression",
},
{
testName: "dash in TypeScript name",
endpointFn: func() *Endpoint {
e := validEndpoint
e.TypeScriptName = "genTest-2"
return &e
},
errMsg: "TypeScriptName doesn't match the regular expression",
},
{
testName: "invalid Request type",
endpointFn: func() *Endpoint {
request := &struct {
Name string `json:"name"`
}{}
e := validEndpoint
e.Request = request
return &e
},
errMsg: fmt.Sprintf("Request cannot be of a type %q", reflect.Pointer),
},
{
testName: "invalid Response type",
endpointFn: func() *Endpoint {
e := validEndpoint
e.Response = map[string]string{}
return &e
},
errMsg: fmt.Sprintf("Response cannot be of a type %q", reflect.Map),
},
{
testName: "different ResponseMock type",
endpointFn: func() *Endpoint {
e := validEndpoint
e.Response = int(0)
e.ResponseMock = int8(0)
return &e
},
errMsg: fmt.Sprintf(
"ResponseMock isn't of the same type than Response. Have=%q Want=%q",
reflect.TypeOf(int8(0)),
reflect.TypeOf(int(0)),
),
},
}
for _, tc := range tcases {
t.Run(tc.testName, func(t *testing.T) {
ep := tc.endpointFn()
err := ep.Validate()
if tc.errMsg == "" {
require.NoError(t, err)
return
}
require.Error(t, err)
require.ErrorContains(t, err, tc.errMsg)
})
}
}
func TestEndpointGroup(t *testing.T) {
t.Run("add endpoints", func(t *testing.T) {
endpointFn := func(postfix string) *Endpoint {
return &Endpoint{
Name: "Test Endpoint",
Description: "This is an Endpoint purely for testing purposes",
GoName: "GenTest" + postfix,
TypeScriptName: "genTest" + postfix,
}
}
path := "/" + strconv.Itoa(rand.Int())
eg := EndpointGroup{}
assert.NotPanics(t, func() { eg.Get(path, endpointFn(http.MethodGet)) }, "Get")
assert.NotPanics(t, func() { eg.Patch(path, endpointFn(http.MethodPatch)) }, "Patch")
assert.NotPanics(t, func() { eg.Post(path, endpointFn(http.MethodPost)) }, "Post")
assert.NotPanics(t, func() { eg.Delete(path, endpointFn(http.MethodDelete)) }, "Delete")
require.Len(t, eg.endpoints, 4, "Group endpoints count")
for i, m := range []string{http.MethodGet, http.MethodPatch, http.MethodPost, http.MethodDelete} {
ep := eg.endpoints[i]
assert.Equal(t, m, ep.Method)
assert.Equal(t, path, ep.Path)
assert.EqualValues(t, endpointFn(m), &ep.Endpoint)
}
})
t.Run("path does not begin with slash", func(t *testing.T) {
endpointFn := func(postfix string) *Endpoint {
return &Endpoint{
Name: "Test Endpoint",
Description: "This is an Endpoint purely for testing purposes",
GoName: "GenTest" + postfix,
TypeScriptName: "genTest" + postfix,
}
}
path := strconv.Itoa(rand.Int())
eg := EndpointGroup{}
assert.Panics(t, func() { eg.Get(path, endpointFn(http.MethodGet)) }, "Get")
assert.Panics(t, func() { eg.Patch(path, endpointFn(http.MethodPatch)) }, "Patch")
assert.Panics(t, func() { eg.Post(path, endpointFn(http.MethodPost)) }, "Post")
assert.Panics(t, func() { eg.Delete(path, endpointFn(http.MethodDelete)) }, "Delete")
})
t.Run("invalid endpoint", func(t *testing.T) {
endpointFn := func(postfix string) *Endpoint {
return &Endpoint{
Name: "",
Description: "This is an Endpoint purely for testing purposes",
GoName: "GenTest" + postfix,
TypeScriptName: "genTest" + postfix,
}
}
path := "/" + strconv.Itoa(rand.Int())
eg := EndpointGroup{}
assert.Panics(t, func() { eg.Get(path, endpointFn(http.MethodGet)) }, "Get")
assert.Panics(t, func() { eg.Patch(path, endpointFn(http.MethodPatch)) }, "Patch")
assert.Panics(t, func() { eg.Post(path, endpointFn(http.MethodPost)) }, "Post")
assert.Panics(t, func() { eg.Delete(path, endpointFn(http.MethodDelete)) }, "Delete")
})
t.Run("endpoint duplicate path method", func(t *testing.T) {
endpointFn := func(postfix string) *Endpoint {
return &Endpoint{
Name: "Test Endpoint",
Description: "This is an Endpoint purely for testing purposes",
GoName: "GenTest" + postfix,
TypeScriptName: "genTest" + postfix,
}
}
path := "/" + strconv.Itoa(rand.Int())
eg := EndpointGroup{}
assert.NotPanics(t, func() { eg.Get(path, endpointFn(http.MethodGet)) }, "Get")
assert.NotPanics(t, func() { eg.Patch(path, endpointFn(http.MethodPatch)) }, "Patch")
assert.NotPanics(t, func() { eg.Post(path, endpointFn(http.MethodPost)) }, "Post")
assert.NotPanics(t, func() { eg.Delete(path, endpointFn(http.MethodDelete)) }, "Delete")
assert.Panics(t, func() { eg.Get(path, endpointFn(http.MethodGet)) }, "Get")
assert.Panics(t, func() { eg.Patch(path, endpointFn(http.MethodPatch)) }, "Patch")
assert.Panics(t, func() { eg.Post(path, endpointFn(http.MethodPost)) }, "Post")
assert.Panics(t, func() { eg.Delete(path, endpointFn(http.MethodDelete)) }, "Delete")
})
t.Run("endpoint duplicate GoName", func(t *testing.T) {
endpointFn := func(postfix string) *Endpoint {
return &Endpoint{
Name: "Test Endpoint",
Description: "This is an Endpoint purely for testing purposes",
GoName: "GenTest",
TypeScriptName: "genTest" + postfix,
}
}
path := "/" + strconv.Itoa(rand.Int())
eg := EndpointGroup{}
assert.NotPanics(t, func() { eg.Get(path, endpointFn(http.MethodGet)) }, "Get")
assert.Panics(t, func() { eg.Patch(path, endpointFn(http.MethodPatch)) }, "Patch")
assert.Panics(t, func() { eg.Post(path, endpointFn(http.MethodPost)) }, "Post")
assert.Panics(t, func() { eg.Delete(path, endpointFn(http.MethodDelete)) }, "Delete")
})
t.Run("endpoint duplicate TypeScriptName", func(t *testing.T) {
endpointFn := func(postfix string) *Endpoint {
return &Endpoint{
Name: "Test Endpoint",
Description: "This is an Endpoint purely for testing purposes",
GoName: "GenTest" + postfix,
TypeScriptName: "genTest",
}
}
path := "/" + strconv.Itoa(rand.Int())
eg := EndpointGroup{}
assert.NotPanics(t, func() { eg.Patch(path, endpointFn(http.MethodPatch)) }, "Patch")
assert.Panics(t, func() { eg.Get(path, endpointFn(http.MethodGet)) }, "Get")
assert.Panics(t, func() { eg.Post(path, endpointFn(http.MethodPost)) }, "Post")
assert.Panics(t, func() { eg.Delete(path, endpointFn(http.MethodDelete)) }, "Delete")
})
}

View File

@ -16,44 +16,196 @@ import (
"storj.io/common/uuid" "storj.io/common/uuid"
"storj.io/storj/private/api" "storj.io/storj/private/api"
"storj.io/storj/private/apigen/example/myapi"
) )
const dateLayout = "2006-01-02T15:04:05.999Z" const dateLayout = "2006-01-02T15:04:05.999Z"
var ErrTestapiAPI = errs.Class("example testapi api") var ErrDocsAPI = errs.Class("example docs api")
var ErrUsersAPI = errs.Class("example users api")
type TestAPIService interface { type DocumentsService interface {
GenTestAPI(ctx context.Context, path string, id uuid.UUID, date time.Time, request struct{ Content string }) (*struct { Get(ctx context.Context) ([]myapi.Document, api.HTTPError)
ID uuid.UUID GetOne(ctx context.Context, path string) (*myapi.Document, api.HTTPError)
Date time.Time GetTag(ctx context.Context, path, tagName string) (*[2]string, api.HTTPError)
PathParam string GetVersions(ctx context.Context, path string) ([]myapi.Version, api.HTTPError)
Body string UpdateContent(ctx context.Context, path string, id uuid.UUID, date time.Time, request myapi.NewDocument) (*myapi.Document, api.HTTPError)
}, api.HTTPError)
} }
// TestAPIHandler is an api handler that exposes all testapi related functionality. type UsersService interface {
type TestAPIHandler struct { Get(ctx context.Context) ([]myapi.User, api.HTTPError)
Create(ctx context.Context, request []myapi.User) api.HTTPError
}
// DocumentsHandler is an api handler that implements all Documents API endpoints functionality.
type DocumentsHandler struct {
log *zap.Logger log *zap.Logger
mon *monkit.Scope mon *monkit.Scope
service TestAPIService service DocumentsService
auth api.Auth auth api.Auth
} }
func NewTestAPI(log *zap.Logger, mon *monkit.Scope, service TestAPIService, router *mux.Router, auth api.Auth) *TestAPIHandler { // UsersHandler is an api handler that implements all Users API endpoints functionality.
handler := &TestAPIHandler{ type UsersHandler struct {
log *zap.Logger
mon *monkit.Scope
service UsersService
}
func NewDocuments(log *zap.Logger, mon *monkit.Scope, service DocumentsService, router *mux.Router, auth api.Auth) *DocumentsHandler {
handler := &DocumentsHandler{
log: log, log: log,
mon: mon, mon: mon,
service: service, service: service,
auth: auth, auth: auth,
} }
testapiRouter := router.PathPrefix("/api/v0/testapi").Subrouter() docsRouter := router.PathPrefix("/api/v0/docs").Subrouter()
testapiRouter.HandleFunc("/{path}", handler.handleGenTestAPI).Methods("POST") docsRouter.HandleFunc("/", handler.handleGet).Methods("GET")
docsRouter.HandleFunc("/{path}", handler.handleGetOne).Methods("GET")
docsRouter.HandleFunc("/{path}/tag/{tagName}", handler.handleGetTag).Methods("GET")
docsRouter.HandleFunc("/{path}/versions", handler.handleGetVersions).Methods("GET")
docsRouter.HandleFunc("/{path}", handler.handleUpdateContent).Methods("POST")
return handler return handler
} }
func (h *TestAPIHandler) handleGenTestAPI(w http.ResponseWriter, r *http.Request) { func NewUsers(log *zap.Logger, mon *monkit.Scope, service UsersService, router *mux.Router) *UsersHandler {
handler := &UsersHandler{
log: log,
mon: mon,
service: service,
}
usersRouter := router.PathPrefix("/api/v0/users").Subrouter()
usersRouter.HandleFunc("/", handler.handleGet).Methods("GET")
usersRouter.HandleFunc("/", handler.handleCreate).Methods("POST")
return handler
}
func (h *DocumentsHandler) handleGet(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
var err error
defer h.mon.Task()(&ctx)(&err)
w.Header().Set("Content-Type", "application/json")
retVal, httpErr := h.service.Get(ctx)
if httpErr.Err != nil {
api.ServeError(h.log, w, httpErr.Status, httpErr.Err)
return
}
err = json.NewEncoder(w).Encode(retVal)
if err != nil {
h.log.Debug("failed to write json Get response", zap.Error(ErrDocsAPI.Wrap(err)))
}
}
func (h *DocumentsHandler) handleGetOne(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
var err error
defer h.mon.Task()(&ctx)(&err)
w.Header().Set("Content-Type", "application/json")
path, ok := mux.Vars(r)["path"]
if !ok {
api.ServeError(h.log, w, http.StatusBadRequest, errs.New("missing path route param"))
return
}
ctx, err = h.auth.IsAuthenticated(ctx, r, true, true)
if err != nil {
h.auth.RemoveAuthCookie(w)
api.ServeError(h.log, w, http.StatusUnauthorized, err)
return
}
retVal, httpErr := h.service.GetOne(ctx, path)
if httpErr.Err != nil {
api.ServeError(h.log, w, httpErr.Status, httpErr.Err)
return
}
err = json.NewEncoder(w).Encode(retVal)
if err != nil {
h.log.Debug("failed to write json GetOne response", zap.Error(ErrDocsAPI.Wrap(err)))
}
}
func (h *DocumentsHandler) handleGetTag(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
var err error
defer h.mon.Task()(&ctx)(&err)
w.Header().Set("Content-Type", "application/json")
path, ok := mux.Vars(r)["path"]
if !ok {
api.ServeError(h.log, w, http.StatusBadRequest, errs.New("missing path route param"))
return
}
tagName, ok := mux.Vars(r)["tagName"]
if !ok {
api.ServeError(h.log, w, http.StatusBadRequest, errs.New("missing tagName route param"))
return
}
ctx, err = h.auth.IsAuthenticated(ctx, r, true, true)
if err != nil {
h.auth.RemoveAuthCookie(w)
api.ServeError(h.log, w, http.StatusUnauthorized, err)
return
}
retVal, httpErr := h.service.GetTag(ctx, path, tagName)
if httpErr.Err != nil {
api.ServeError(h.log, w, httpErr.Status, httpErr.Err)
return
}
err = json.NewEncoder(w).Encode(retVal)
if err != nil {
h.log.Debug("failed to write json GetTag response", zap.Error(ErrDocsAPI.Wrap(err)))
}
}
func (h *DocumentsHandler) handleGetVersions(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
var err error
defer h.mon.Task()(&ctx)(&err)
w.Header().Set("Content-Type", "application/json")
path, ok := mux.Vars(r)["path"]
if !ok {
api.ServeError(h.log, w, http.StatusBadRequest, errs.New("missing path route param"))
return
}
ctx, err = h.auth.IsAuthenticated(ctx, r, true, true)
if err != nil {
h.auth.RemoveAuthCookie(w)
api.ServeError(h.log, w, http.StatusUnauthorized, err)
return
}
retVal, httpErr := h.service.GetVersions(ctx, path)
if httpErr.Err != nil {
api.ServeError(h.log, w, httpErr.Status, httpErr.Err)
return
}
err = json.NewEncoder(w).Encode(retVal)
if err != nil {
h.log.Debug("failed to write json GetVersions response", zap.Error(ErrDocsAPI.Wrap(err)))
}
}
func (h *DocumentsHandler) handleUpdateContent(w http.ResponseWriter, r *http.Request) {
ctx := r.Context() ctx := r.Context()
var err error var err error
defer h.mon.Task()(&ctx)(&err) defer h.mon.Task()(&ctx)(&err)
@ -90,7 +242,7 @@ func (h *TestAPIHandler) handleGenTestAPI(w http.ResponseWriter, r *http.Request
return return
} }
payload := struct{ Content string }{} payload := myapi.NewDocument{}
if err = json.NewDecoder(r.Body).Decode(&payload); err != nil { if err = json.NewDecoder(r.Body).Decode(&payload); err != nil {
api.ServeError(h.log, w, http.StatusBadRequest, err) api.ServeError(h.log, w, http.StatusBadRequest, err)
return return
@ -103,7 +255,7 @@ func (h *TestAPIHandler) handleGenTestAPI(w http.ResponseWriter, r *http.Request
return return
} }
retVal, httpErr := h.service.GenTestAPI(ctx, path, id, date, payload) retVal, httpErr := h.service.UpdateContent(ctx, path, id, date, payload)
if httpErr.Err != nil { if httpErr.Err != nil {
api.ServeError(h.log, w, httpErr.Status, httpErr.Err) api.ServeError(h.log, w, httpErr.Status, httpErr.Err)
return return
@ -111,6 +263,44 @@ func (h *TestAPIHandler) handleGenTestAPI(w http.ResponseWriter, r *http.Request
err = json.NewEncoder(w).Encode(retVal) err = json.NewEncoder(w).Encode(retVal)
if err != nil { if err != nil {
h.log.Debug("failed to write json GenTestAPI response", zap.Error(ErrTestapiAPI.Wrap(err))) h.log.Debug("failed to write json UpdateContent response", zap.Error(ErrDocsAPI.Wrap(err)))
}
}
func (h *UsersHandler) handleGet(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
var err error
defer h.mon.Task()(&ctx)(&err)
w.Header().Set("Content-Type", "application/json")
retVal, httpErr := h.service.Get(ctx)
if httpErr.Err != nil {
api.ServeError(h.log, w, httpErr.Status, httpErr.Err)
return
}
err = json.NewEncoder(w).Encode(retVal)
if err != nil {
h.log.Debug("failed to write json Get response", zap.Error(ErrUsersAPI.Wrap(err)))
}
}
func (h *UsersHandler) handleCreate(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
var err error
defer h.mon.Task()(&ctx)(&err)
w.Header().Set("Content-Type", "application/json")
payload := []myapi.User{}
if err = json.NewDecoder(r.Body).Decode(&payload); err != nil {
api.ServeError(h.log, w, http.StatusBadRequest, err)
return
}
httpErr := h.service.Create(ctx, payload)
if httpErr.Err != nil {
api.ServeError(h.log, w, httpErr.Status, httpErr.Err)
} }
} }

View File

@ -0,0 +1,224 @@
# API Docs
**Version:** `v0`
<h2 id='list-of-endpoints'>List of Endpoints</h2>
* Documents
* [Get Documents](#documents-get-documents)
* [Get One](#documents-get-one)
* [Get a tag](#documents-get-a-tag)
* [Get Version](#documents-get-version)
* [Update Content](#documents-update-content)
* Users
* [Get Users](#users-get-users)
* [Create User](#users-create-user)
<h3 id='documents-get-documents'>Get Documents (<a href='#list-of-endpoints'>go to full list</a>)</h3>
Get the paths to all the documents under the specified paths
`GET /api/v0/docs/`
**Response body:**
```typescript
[
{
id: string // UUID formatted as `00000000-0000-0000-0000-000000000000`
date: string // Date timestamp formatted as `2006-01-02T15:00:00Z`
pathParam: string
body: string
version: {
date: string // Date timestamp formatted as `2006-01-02T15:00:00Z`
number: number
}
metadata: {
owner: string
tags: [
unknown
]
}
}
]
```
<h3 id='documents-get-one'>Get One (<a href='#list-of-endpoints'>go to full list</a>)</h3>
Get the document in the specified path
`GET /api/v0/docs/{path}`
**Path Params:**
| name | type | elaboration |
|---|---|---|
| `path` | `string` | |
**Response body:**
```typescript
{
id: string // UUID formatted as `00000000-0000-0000-0000-000000000000`
date: string // Date timestamp formatted as `2006-01-02T15:00:00Z`
pathParam: string
body: string
version: {
date: string // Date timestamp formatted as `2006-01-02T15:00:00Z`
number: number
}
metadata: {
owner: string
tags: [
unknown
]
}
}
```
<h3 id='documents-get-a-tag'>Get a tag (<a href='#list-of-endpoints'>go to full list</a>)</h3>
Get the tag of the document in the specified path and tag label
`GET /api/v0/docs/{path}/tag/{tagName}`
**Path Params:**
| name | type | elaboration |
|---|---|---|
| `path` | `string` | |
| `tagName` | `string` | |
**Response body:**
```typescript
unknown
```
<h3 id='documents-get-version'>Get Version (<a href='#list-of-endpoints'>go to full list</a>)</h3>
Get all the version of the document in the specified path
`GET /api/v0/docs/{path}/versions`
**Path Params:**
| name | type | elaboration |
|---|---|---|
| `path` | `string` | |
**Response body:**
```typescript
[
{
date: string // Date timestamp formatted as `2006-01-02T15:00:00Z`
number: number
}
]
```
<h3 id='documents-update-content'>Update Content (<a href='#list-of-endpoints'>go to full list</a>)</h3>
Update the content of the document with the specified path and ID if the last update is before the indicated date
`POST /api/v0/docs/{path}`
**Query Params:**
| name | type | elaboration |
|---|---|---|
| `id` | `string` | UUID formatted as `00000000-0000-0000-0000-000000000000` |
| `date` | `string` | Date timestamp formatted as `2006-01-02T15:00:00Z` |
**Path Params:**
| name | type | elaboration |
|---|---|---|
| `path` | `string` | |
**Request body:**
```typescript
{
content: string
}
```
**Response body:**
```typescript
{
id: string // UUID formatted as `00000000-0000-0000-0000-000000000000`
date: string // Date timestamp formatted as `2006-01-02T15:00:00Z`
pathParam: string
body: string
version: {
date: string // Date timestamp formatted as `2006-01-02T15:00:00Z`
number: number
}
metadata: {
owner: string
tags: [
unknown
]
}
}
```
<h3 id='users-get-users'>Get Users (<a href='#list-of-endpoints'>go to full list</a>)</h3>
Get the list of registered users
`GET /api/v0/users/`
**Response body:**
```typescript
[
{
name: string
surname: string
email: string
}
]
```
<h3 id='users-create-user'>Create User (<a href='#list-of-endpoints'>go to full list</a>)</h3>
Create a user
`POST /api/v0/users/`
**Request body:**
```typescript
[
{
name: string
surname: string
email: string
}
]
```

View File

@ -0,0 +1,137 @@
// AUTOGENERATED BY private/apigen
// DO NOT EDIT.
import { Time, UUID } from '@/types/common';
export class Document {
id: UUID;
date: Time;
pathParam: string;
body: string;
version: Version;
metadata: Metadata;
}
export class Metadata {
owner?: string;
tags: string[][] | null;
}
export class NewDocument {
content: string;
}
export class User {
name: string;
surname: string;
email: string;
}
export class Version {
date: Time;
number: number;
}
class APIError extends Error {
constructor(
public readonly msg: string,
public readonly responseStatusCode?: number,
) {
super(msg);
}
}
export class DocumentsHttpApiV0 {
public readonly respStatusCode: number;
// When respStatuscode is passed, the client throws an APIError on each method call
// with respStatusCode as HTTP status code.
// respStatuscode must be equal or greater than 400
constructor(respStatusCode?: number) {
if (typeof respStatusCode === 'undefined') {
this.respStatusCode = 0;
return;
}
if (respStatusCode < 400) {
throw new Error('invalid response status code for API Error, it must be greater or equal than 400');
}
this.respStatusCode = respStatusCode;
}
public async get(): Promise<Document[]> {
if (this.respStatusCode !== 0) {
throw new APIError('mock error message: ' + this.respStatusCode, this.respStatusCode);
}
return JSON.parse('[{"id":"00000000-0000-0000-0000-000000000000","date":"0001-01-01T00:00:00Z","pathParam":"/workspace/notes.md","body":"","version":{"date":"0001-01-01T00:00:00Z","number":0},"metadata":{"owner":"Storj","tags":[["category","general"]]}}]') as Document[];
}
public async getOne(path: string): Promise<Document> {
if (this.respStatusCode !== 0) {
throw new APIError('mock error message: ' + this.respStatusCode, this.respStatusCode);
}
return JSON.parse('{"id":"00000000-0000-0000-0000-000000000000","date":"2001-02-02T04:05:06.000000007Z","pathParam":"ID","body":"## Notes","version":{"date":"2001-02-03T03:35:06.000000007Z","number":1},"metadata":{"tags":null}}') as Document;
}
public async getTag(path: string, tagName: string): Promise<string[]> {
if (this.respStatusCode !== 0) {
throw new APIError('mock error message: ' + this.respStatusCode, this.respStatusCode);
}
return JSON.parse('["category","notes"]') as string[];
}
public async getVersions(path: string): Promise<Version[]> {
if (this.respStatusCode !== 0) {
throw new APIError('mock error message: ' + this.respStatusCode, this.respStatusCode);
}
return JSON.parse('[{"date":"2001-01-19T04:05:06.000000007Z","number":1},{"date":"2001-02-02T23:05:06.000000007Z","number":2}]') as Version[];
}
public async updateContent(request: NewDocument, path: string, id: UUID, date: Time): Promise<Document> {
if (this.respStatusCode !== 0) {
throw new APIError('mock error message: ' + this.respStatusCode, this.respStatusCode);
}
return JSON.parse('{"id":"00000000-0000-0000-0000-000000000000","date":"2001-02-03T04:05:06.000000007Z","pathParam":"ID","body":"## Notes\n### General","version":{"date":"0001-01-01T00:00:00Z","number":0},"metadata":{"tags":null}}') as Document;
}
}
export class UsersHttpApiV0 {
public readonly respStatusCode: number;
// When respStatuscode is passed, the client throws an APIError on each method call
// with respStatusCode as HTTP status code.
// respStatuscode must be equal or greater than 400
constructor(respStatusCode?: number) {
if (typeof respStatusCode === 'undefined') {
this.respStatusCode = 0;
return;
}
if (respStatusCode < 400) {
throw new Error('invalid response status code for API Error, it must be greater or equal than 400');
}
this.respStatusCode = respStatusCode;
}
public async get(): Promise<User[]> {
if (this.respStatusCode !== 0) {
throw new APIError('mock error message: ' + this.respStatusCode, this.respStatusCode);
}
return JSON.parse('[{"name":"Storj","surname":"Labs","email":"storj@storj.test"},{"name":"Test1","surname":"Testing","email":"test1@example.test"},{"name":"Test2","surname":"Testing","email":"test2@example.test"}]') as User[];
}
public async create(request: User[]): Promise<void> {
if (this.respStatusCode !== 0) {
throw new APIError('mock error message: ' + this.respStatusCode, this.respStatusCode);
}
return;
}
}

View File

@ -0,0 +1,126 @@
// AUTOGENERATED BY private/apigen
// DO NOT EDIT.
import { HttpClient } from '@/utils/httpClient';
import { Time, UUID } from '@/types/common';
export class Document {
id: UUID;
date: Time;
pathParam: string;
body: string;
version: Version;
metadata: Metadata;
}
export class Metadata {
owner?: string;
tags: string[][] | null;
}
export class NewDocument {
content: string;
}
export class User {
name: string;
surname: string;
email: string;
}
export class Version {
date: Time;
number: number;
}
class APIError extends Error {
constructor(
public readonly msg: string,
public readonly responseStatusCode?: number,
) {
super(msg);
}
}
export class DocumentsHttpApiV0 {
private readonly http: HttpClient = new HttpClient();
private readonly ROOT_PATH: string = '/api/v0/docs';
public async get(): Promise<Document[]> {
const fullPath = `${this.ROOT_PATH}/`;
const response = await this.http.get(fullPath);
if (response.ok) {
return response.json().then((body) => body as Document[]);
}
const err = await response.json();
throw new APIError(err.error, response.status);
}
public async getOne(path: string): Promise<Document> {
const fullPath = `${this.ROOT_PATH}/${path}`;
const response = await this.http.get(fullPath);
if (response.ok) {
return response.json().then((body) => body as Document);
}
const err = await response.json();
throw new APIError(err.error, response.status);
}
public async getTag(path: string, tagName: string): Promise<string[]> {
const fullPath = `${this.ROOT_PATH}/${path}/${tagName}`;
const response = await this.http.get(fullPath);
if (response.ok) {
return response.json().then((body) => body as string[]);
}
const err = await response.json();
throw new APIError(err.error, response.status);
}
public async getVersions(path: string): Promise<Version[]> {
const fullPath = `${this.ROOT_PATH}/${path}`;
const response = await this.http.get(fullPath);
if (response.ok) {
return response.json().then((body) => body as Version[]);
}
const err = await response.json();
throw new APIError(err.error, response.status);
}
public async updateContent(request: NewDocument, path: string, id: UUID, date: Time): Promise<Document> {
const u = new URL(`${this.ROOT_PATH}/${path}`, window.location.href);
u.searchParams.set('id', id);
u.searchParams.set('date', date);
const fullPath = u.toString();
const response = await this.http.post(fullPath, JSON.stringify(request));
if (response.ok) {
return response.json().then((body) => body as Document);
}
const err = await response.json();
throw new APIError(err.error, response.status);
}
}
export class UsersHttpApiV0 {
private readonly http: HttpClient = new HttpClient();
private readonly ROOT_PATH: string = '/api/v0/users';
public async get(): Promise<User[]> {
const fullPath = `${this.ROOT_PATH}/`;
const response = await this.http.get(fullPath);
if (response.ok) {
return response.json().then((body) => body as User[]);
}
const err = await response.json();
throw new APIError(err.error, response.status);
}
public async create(request: User[]): Promise<void> {
const fullPath = `${this.ROOT_PATH}/`;
const response = await this.http.post(fullPath, JSON.stringify(request));
if (response.ok) {
return;
}
const err = await response.json();
throw new APIError(err.error, response.status);
}
}

View File

@ -7,26 +7,109 @@
package main package main
import ( import (
"fmt"
"net/http"
"time" "time"
"go.uber.org/zap"
"storj.io/common/uuid" "storj.io/common/uuid"
"storj.io/storj/private/api"
"storj.io/storj/private/apigen" "storj.io/storj/private/apigen"
"storj.io/storj/private/apigen/example/myapi"
) )
func main() { func main() {
a := &apigen.API{PackageName: "example"} a := &apigen.API{
PackagePath: "storj.io/storj/private/apigen/example",
Version: "v0",
BasePath: "/api",
}
g := a.Group("TestAPI", "testapi") g := a.Group("Documents", "docs")
g.Middleware = append(g.Middleware,
authMiddleware{},
)
now := time.Date(2001, 02, 03, 04, 05, 06, 07, time.UTC)
g.Get("/", &apigen.Endpoint{
Name: "Get Documents",
Description: "Get the paths to all the documents under the specified paths",
GoName: "Get",
TypeScriptName: "get",
Response: []myapi.Document{},
ResponseMock: []myapi.Document{{
ID: uuid.UUID{},
PathParam: "/workspace/notes.md",
Metadata: myapi.Metadata{
Owner: "Storj",
Tags: [][2]string{{"category", "general"}},
},
}},
Settings: map[any]any{
NoAPIKey: true,
NoCookie: true,
},
})
g.Get("/{path}", &apigen.Endpoint{
Name: "Get One",
Description: "Get the document in the specified path",
GoName: "GetOne",
TypeScriptName: "getOne",
Response: myapi.Document{},
PathParams: []apigen.Param{
apigen.NewParam("path", ""),
},
ResponseMock: myapi.Document{
ID: uuid.UUID{},
Date: now.Add(-24 * time.Hour),
PathParam: "ID",
Body: "## Notes",
Version: myapi.Version{
Date: now.Add(-30 * time.Minute),
Number: 1,
},
},
})
g.Get("/{path}/tag/{tagName}", &apigen.Endpoint{
Name: "Get a tag",
Description: "Get the tag of the document in the specified path and tag label ",
GoName: "GetTag",
TypeScriptName: "getTag",
Response: [2]string{},
PathParams: []apigen.Param{
apigen.NewParam("path", ""),
apigen.NewParam("tagName", ""),
},
ResponseMock: [2]string{"category", "notes"},
})
g.Get("/{path}/versions", &apigen.Endpoint{
Name: "Get Version",
Description: "Get all the version of the document in the specified path",
GoName: "GetVersions",
TypeScriptName: "getVersions",
Response: []myapi.Version{},
PathParams: []apigen.Param{
apigen.NewParam("path", ""),
},
ResponseMock: []myapi.Version{
{Date: now.Add(-360 * time.Hour), Number: 1},
{Date: now.Add(-5 * time.Hour), Number: 2},
},
})
g.Post("/{path}", &apigen.Endpoint{ g.Post("/{path}", &apigen.Endpoint{
MethodName: "GenTestAPI", Name: "Update Content",
Response: struct { Description: "Update the content of the document with the specified path and ID if the last update is before the indicated date",
ID uuid.UUID GoName: "UpdateContent",
Date time.Time TypeScriptName: "updateContent",
PathParam string Response: myapi.Document{},
Body string Request: myapi.NewDocument{},
}{},
Request: struct{ Content string }{},
QueryParams: []apigen.Param{ QueryParams: []apigen.Param{
apigen.NewParam("id", uuid.UUID{}), apigen.NewParam("id", uuid.UUID{}),
apigen.NewParam("date", time.Time{}), apigen.NewParam("date", time.Time{}),
@ -34,7 +117,79 @@ func main() {
PathParams: []apigen.Param{ PathParams: []apigen.Param{
apigen.NewParam("path", ""), apigen.NewParam("path", ""),
}, },
ResponseMock: myapi.Document{
ID: uuid.UUID{},
Date: now,
PathParam: "ID",
Body: "## Notes\n### General",
},
})
g = a.Group("Users", "users")
g.Get("/", &apigen.Endpoint{
Name: "Get Users",
Description: "Get the list of registered users",
GoName: "Get",
TypeScriptName: "get",
Response: []myapi.User{},
ResponseMock: []myapi.User{
{Name: "Storj", Surname: "Labs", Email: "storj@storj.test"},
{Name: "Test1", Surname: "Testing", Email: "test1@example.test"},
{Name: "Test2", Surname: "Testing", Email: "test2@example.test"},
},
})
g.Post("/", &apigen.Endpoint{
Name: "Create User",
Description: "Create a user",
GoName: "Create",
TypeScriptName: "create",
Request: []myapi.User{},
}) })
a.MustWriteGo("api.gen.go") a.MustWriteGo("api.gen.go")
a.MustWriteTS("client-api.gen.ts")
a.MustWriteTSMock("client-api-mock.gen.ts")
a.MustWriteDocs("apidocs.gen.md")
} }
// authMiddleware customize endpoints to authenticate requests by API Key or Cookie.
type authMiddleware struct {
log *zap.Logger
auth api.Auth
_ http.ResponseWriter // Import the http package to use its HTTP status constants
}
// Generate satisfies the apigen.Middleware.
func (a authMiddleware) Generate(api *apigen.API, group *apigen.EndpointGroup, ep *apigen.FullEndpoint) string {
noapikey := apigen.LoadSetting(NoAPIKey, ep, false)
nocookie := apigen.LoadSetting(NoCookie, ep, false)
if noapikey && nocookie {
return ""
}
return fmt.Sprintf(`ctx, err = h.auth.IsAuthenticated(ctx, r, %t, %t)
if err != nil {
h.auth.RemoveAuthCookie(w)
api.ServeError(h.log, w, http.StatusUnauthorized, err)
return
}`, !nocookie, !noapikey)
}
var _ apigen.Middleware = authMiddleware{}
type (
tagNoAPIKey struct{}
tagNoCookie struct{}
)
var (
// NoAPIKey is the key for endpoint settings to indicate that it doesn't use API Key
// authentication mechanism.
NoAPIKey tagNoAPIKey
// NoCookie is the key for endpoint settings to indicate that it doesn't use cookie authentication
// mechanism.
NoCookie tagNoCookie
)

View File

@ -0,0 +1,44 @@
// Copyright (C) 2023 Storj Labs, Inc.
// See LICENSE for copying information.
package myapi
import (
"time"
"storj.io/common/uuid"
)
// Document is a retrieved document.
type Document struct {
ID uuid.UUID `json:"id"`
Date time.Time `json:"date"`
PathParam string `json:"pathParam"`
Body string `json:"body"`
Version Version `json:"version"`
Metadata Metadata `json:"metadata"`
}
// Version is document version.
type Version struct {
Date time.Time `json:"date"`
Number uint `json:"number"`
}
// Metadata is metadata associated to a document.
type Metadata struct {
Owner string `json:"owner,omitempty"`
Tags [][2]string `json:"tags"`
}
// NewDocument contains the content the data to create a new document.
type NewDocument struct {
Content string `json:"content"`
}
// User contains information of a user.
type User struct {
Name string `json:"name"`
Surname string `json:"surname"`
Email string `json:"email"`
}

View File

@ -4,16 +4,16 @@
package apigen package apigen
import ( import (
"fmt"
"go/format" "go/format"
"os" "os"
"path/filepath"
"reflect" "reflect"
"sort" "slices"
"strings" "strings"
"time" "time"
"github.com/zeebo/errs" "github.com/zeebo/errs"
"golang.org/x/text/cases"
"golang.org/x/text/language"
"storj.io/common/uuid" "storj.io/common/uuid"
) )
@ -22,10 +22,11 @@ import (
const DateFormat = "2006-01-02T15:04:05.999Z" const DateFormat = "2006-01-02T15:04:05.999Z"
// MustWriteGo writes generated Go code into a file. // MustWriteGo writes generated Go code into a file.
// If an error occurs, it panics.
func (a *API) MustWriteGo(path string) { func (a *API) MustWriteGo(path string) {
generated, err := a.generateGo() generated, err := a.generateGo()
if err != nil { if err != nil {
panic(errs.Wrap(err)) panic(err)
} }
err = os.WriteFile(path, generated, 0644) err = os.WriteFile(path, generated, 0644)
@ -39,32 +40,38 @@ func (a *API) generateGo() ([]byte, error) {
result := &StringBuilder{} result := &StringBuilder{}
pf := result.Writelnf pf := result.Writelnf
getPackageName := func(path string) string { if a.PackagePath == "" {
pathPackages := strings.Split(path, "/") return nil, errs.New("Package path must be defined")
return pathPackages[len(pathPackages)-1] }
packageName := a.PackageName
if packageName == "" {
parts := strings.Split(a.PackagePath, "/")
packageName = parts[len(parts)-1]
} }
imports := struct { imports := struct {
All map[string]bool All map[importPath]bool
Standard []string Standard []importPath
External []string External []importPath
Internal []string Internal []importPath
}{ }{
All: make(map[string]bool), All: make(map[importPath]bool),
} }
i := func(paths ...string) { i := func(paths ...string) {
for _, path := range paths { for _, path := range paths {
if path == "" || getPackageName(path) == a.PackageName { if path == "" || path == a.PackagePath {
continue continue
} }
if _, ok := imports.All[path]; ok { ipath := importPath(path)
if _, ok := imports.All[ipath]; ok {
continue continue
} }
imports.All[path] = true imports.All[ipath] = true
var slice *[]string var slice *[]importPath
switch { switch {
case !strings.Contains(path, "."): case !strings.Contains(path, "."):
slice = &imports.Standard slice = &imports.Standard
@ -73,7 +80,7 @@ func (a *API) generateGo() ([]byte, error) {
default: default:
slice = &imports.External slice = &imports.External
} }
*slice = append(*slice, path) *slice = append(*slice, ipath)
} }
} }
@ -100,15 +107,25 @@ func (a *API) generateGo() ([]byte, error) {
for _, group := range a.EndpointGroups { for _, group := range a.EndpointGroups {
i("github.com/zeebo/errs") i("github.com/zeebo/errs")
pf("var Err%sAPI = errs.Class(\"%s %s api\")", cases.Title(language.Und).String(group.Prefix), a.PackageName, group.Prefix) pf(
"var Err%sAPI = errs.Class(\"%s %s api\")",
capitalize(group.Prefix),
packageName,
strings.ToLower(group.Prefix),
)
for _, m := range group.Middleware {
i(middlewareImports(m)...)
}
} }
pf("") pf("")
params := make(map[*fullEndpoint][]Param) params := make(map[*FullEndpoint][]Param)
for _, group := range a.EndpointGroups { for _, group := range a.EndpointGroups {
pf("type %sService interface {", group.Name) // Define the service interface
pf("type %sService interface {", capitalize(group.Name))
for _, e := range group.endpoints { for _, e := range group.endpoints {
params[e] = append(e.PathParams, e.QueryParams...) params[e] = append(e.PathParams, e.QueryParams...)
@ -131,9 +148,9 @@ func (a *API) generateGo() ([]byte, error) {
if !isNillableType(responseType) { if !isNillableType(responseType) {
returnParam = "*" + returnParam returnParam = "*" + returnParam
} }
pf("%s(ctx context.Context, "+paramStr+") (%s, api.HTTPError)", e.MethodName, returnParam) pf("%s(ctx context.Context, "+paramStr+") (%s, api.HTTPError)", e.GoName, returnParam)
} else { } else {
pf("%s(ctx context.Context, "+paramStr+") (api.HTTPError)", e.MethodName) pf("%s(ctx context.Context, "+paramStr+") (api.HTTPError)", e.GoName)
} }
} }
pf("}") pf("}")
@ -141,36 +158,104 @@ func (a *API) generateGo() ([]byte, error) {
} }
for _, group := range a.EndpointGroups { for _, group := range a.EndpointGroups {
cname := capitalize(group.Name)
i("go.uber.org/zap", "github.com/spacemonkeygo/monkit/v3") i("go.uber.org/zap", "github.com/spacemonkeygo/monkit/v3")
pf("// %sHandler is an api handler that exposes all %s related functionality.", group.Name, group.Prefix) pf(
pf("type %sHandler struct {", group.Name) "// %sHandler is an api handler that implements all %s API endpoints functionality.",
cname,
group.Name,
)
pf("type %sHandler struct {", cname)
pf("log *zap.Logger") pf("log *zap.Logger")
pf("mon *monkit.Scope") pf("mon *monkit.Scope")
pf("service %sService", group.Name) pf("service %sService", cname)
pf("auth api.Auth")
autodefinedFields := map[string]string{"log": "*zap.Logger", "mon": "*monkit.Scope", "service": cname + "Service"}
for _, m := range group.Middleware {
for _, f := range middlewareFields(a, m) {
if t, ok := autodefinedFields[f.Name]; ok {
if t != f.Type {
panic(
fmt.Sprintf(
"middleware %q has a field with name %q and type %q which clashes with another defined field with the same name but with type %q",
reflect.TypeOf(m).Name(),
f.Name,
f.Type,
t,
),
)
}
continue
}
autodefinedFields[f.Name] = f.Type
pf("%s %s", f.Name, f.Type)
}
}
pf("}") pf("}")
pf("") pf("")
} }
for _, group := range a.EndpointGroups { for _, group := range a.EndpointGroups {
cname := capitalize(group.Name)
i("github.com/gorilla/mux") i("github.com/gorilla/mux")
pf(
"func New%s(log *zap.Logger, mon *monkit.Scope, service %sService, router *mux.Router, auth api.Auth) *%sHandler {", autodedefined := map[string]struct{}{"log": {}, "mon": {}, "service": {}}
group.Name, middlewareArgs := make([]string, 0, len(group.Middleware))
group.Name, middlewareFieldsList := make([]string, 0, len(group.Middleware))
group.Name, for _, m := range group.Middleware {
) for _, f := range middlewareFields(a, m) {
pf("handler := &%sHandler{", group.Name) if _, ok := autodedefined[f.Name]; !ok {
middlewareArgs = append(middlewareArgs, fmt.Sprintf("%s %s", f.Name, f.Type))
middlewareFieldsList = append(middlewareFieldsList, fmt.Sprintf("%[1]s: %[1]s", f.Name))
}
}
}
if len(middlewareArgs) > 0 {
pf(
"func New%s(log *zap.Logger, mon *monkit.Scope, service %sService, router *mux.Router, %s) *%sHandler {",
cname,
cname,
strings.Join(middlewareArgs, ", "),
cname,
)
} else {
pf(
"func New%s(log *zap.Logger, mon *monkit.Scope, service %sService, router *mux.Router) *%sHandler {",
cname,
cname,
cname,
)
}
pf("handler := &%sHandler{", cname)
pf("log: log,") pf("log: log,")
pf("mon: mon,") pf("mon: mon,")
pf("service: service,") pf("service: service,")
pf("auth: auth,")
if len(middlewareFieldsList) > 0 {
pf(strings.Join(middlewareFieldsList, ",") + ",")
}
pf("}") pf("}")
pf("") pf("")
pf("%sRouter := router.PathPrefix(\"/api/v0/%s\").Subrouter()", group.Prefix, group.Prefix) pf(
"%sRouter := router.PathPrefix(\"%s/%s\").Subrouter()",
uncapitalize(group.Prefix),
a.endpointBasePath(),
strings.ToLower(group.Prefix),
)
for _, endpoint := range group.endpoints { for _, endpoint := range group.endpoints {
handlerName := "handle" + endpoint.MethodName handlerName := "handle" + endpoint.GoName
pf("%sRouter.HandleFunc(\"%s\", handler.%s).Methods(\"%s\")", group.Prefix, endpoint.Path, handlerName, endpoint.Method) pf(
"%sRouter.HandleFunc(\"%s\", handler.%s).Methods(\"%s\")",
uncapitalize(group.Prefix),
endpoint.Path,
handlerName,
endpoint.Method,
)
} }
pf("") pf("")
pf("return handler") pf("return handler")
@ -182,13 +267,12 @@ func (a *API) generateGo() ([]byte, error) {
for _, endpoint := range group.endpoints { for _, endpoint := range group.endpoints {
i("net/http") i("net/http")
pf("") pf("")
handlerName := "handle" + endpoint.MethodName handlerName := "handle" + endpoint.GoName
pf("func (h *%sHandler) %s(w http.ResponseWriter, r *http.Request) {", group.Name, handlerName) pf("func (h *%sHandler) %s(w http.ResponseWriter, r *http.Request) {", capitalize(group.Name), handlerName)
pf("ctx := r.Context()") pf("ctx := r.Context()")
pf("var err error") pf("var err error")
pf("defer h.mon.Task()(&ctx)(&err)") pf("defer h.mon.Task()(&ctx)(&err)")
pf("") pf("")
pf("w.Header().Set(\"Content-Type\", \"application/json\")") pf("w.Header().Set(\"Content-Type\", \"application/json\")")
pf("") pf("")
@ -200,17 +284,10 @@ func (a *API) generateGo() ([]byte, error) {
handleBody(pf, endpoint.Request) handleBody(pf, endpoint.Request)
} }
if !endpoint.NoCookieAuth || !endpoint.NoAPIAuth { for _, m := range group.Middleware {
pf("ctx, err = h.auth.IsAuthenticated(ctx, r, %v, %v)", !endpoint.NoCookieAuth, !endpoint.NoAPIAuth) pf(m.Generate(a, group, endpoint))
pf("if err != nil {")
if !endpoint.NoCookieAuth {
pf("h.auth.RemoveAuthCookie(w)")
}
pf("api.ServeError(h.log, w, http.StatusUnauthorized, err)")
pf("return")
pf("}")
pf("")
} }
pf("")
var methodFormat string var methodFormat string
if endpoint.Response != nil { if endpoint.Response != nil {
@ -227,7 +304,7 @@ func (a *API) generateGo() ([]byte, error) {
} }
methodFormat += ")" methodFormat += ")"
pf(methodFormat, endpoint.MethodName) pf(methodFormat, endpoint.GoName)
pf("if httpErr.Err != nil {") pf("if httpErr.Err != nil {")
pf("api.ServeError(h.log, w, httpErr.Status, httpErr.Err)") pf("api.ServeError(h.log, w, httpErr.Status, httpErr.Err)")
if endpoint.Response == nil { if endpoint.Response == nil {
@ -242,7 +319,11 @@ func (a *API) generateGo() ([]byte, error) {
pf("") pf("")
pf("err = json.NewEncoder(w).Encode(retVal)") pf("err = json.NewEncoder(w).Encode(retVal)")
pf("if err != nil {") pf("if err != nil {")
pf("h.log.Debug(\"failed to write json %s response\", zap.Error(Err%sAPI.Wrap(err)))", endpoint.MethodName, cases.Title(language.Und).String(group.Prefix)) pf(
"h.log.Debug(\"failed to write json %s response\", zap.Error(Err%sAPI.Wrap(err)))",
endpoint.GoName,
capitalize(group.Prefix),
)
pf("}") pf("}")
pf("}") pf("}")
} }
@ -256,16 +337,21 @@ func (a *API) generateGo() ([]byte, error) {
pf("// DO NOT EDIT.") pf("// DO NOT EDIT.")
pf("") pf("")
pf("package %s", a.PackageName) pf("package %s", packageName)
pf("") pf("")
pf("import (") pf("import (")
slices := [][]string{imports.Standard, imports.External, imports.Internal} all := [][]importPath{imports.Standard, imports.External, imports.Internal}
for sn, slice := range slices { for sn, slice := range all {
sort.Strings(slice) slices.Sort(slice)
for pn, path := range slice { for pn, path := range slice {
pf(`"%s"`, path) if r, ok := path.PkgName(); ok {
if pn == len(slice)-1 && sn < len(slices)-1 { pf(`%s "%s"`, r, path)
} else {
pf(`"%s"`, path)
}
if pn == len(slice)-1 && sn < len(all)-1 {
pf("") pf("")
} }
} }
@ -282,7 +368,7 @@ func (a *API) generateGo() ([]byte, error) {
output, err := format.Source([]byte(result.String())) output, err := format.Source([]byte(result.String()))
if err != nil { if err != nil {
return nil, err return nil, errs.Wrap(err)
} }
return output, nil return output, nil
@ -292,8 +378,17 @@ func (a *API) generateGo() ([]byte, error) {
// If type is from the same package then we use only type's name. // If type is from the same package then we use only type's name.
// If type is from external package then we use type along with its appropriate package name. // If type is from external package then we use type along with its appropriate package name.
func (a *API) handleTypesPackage(t reflect.Type) string { func (a *API) handleTypesPackage(t reflect.Type) string {
if strings.HasPrefix(t.String(), a.PackageName) { switch t.Kind() {
return t.Elem().Name() case reflect.Array:
return fmt.Sprintf("[%d]%s", t.Len(), a.handleTypesPackage(t.Elem()))
case reflect.Slice:
return "[]" + a.handleTypesPackage(t.Elem())
case reflect.Pointer:
return "*" + a.handleTypesPackage(t.Elem())
}
if t.PkgPath() == a.PackagePath {
return t.Name()
} }
return t.String() return t.String()
@ -381,3 +476,20 @@ func handleBody(pf func(format string, a ...interface{}), body interface{}) {
pf("}") pf("}")
pf("") pf("")
} }
type importPath string
// PkgName returns the name of the package based of the last part of the import
// path and false if the name isn't a rename, otherwise it returns true.
//
// The package name is renamed when the last part of the path contains hyphen
// (-) or dot (.) and the rename is this part with the hyphens and dots
// stripped.
func (i importPath) PkgName() (rename string, ok bool) {
b := filepath.Base(string(i))
if strings.Contains(b, "-") || strings.Contains(b, ".") {
return strings.ReplaceAll(strings.ReplaceAll(b, "-", ""), ".", ""), true
}
return b, false
}

View File

@ -25,17 +25,12 @@ import (
"storj.io/storj/private/api" "storj.io/storj/private/api"
"storj.io/storj/private/apigen" "storj.io/storj/private/apigen"
"storj.io/storj/private/apigen/example" "storj.io/storj/private/apigen/example"
"storj.io/storj/private/apigen/example/myapi"
) )
type ( type (
auth struct{} auth struct{}
service struct{} service struct{}
response = struct {
ID uuid.UUID
Date time.Time
PathParam string
Body string
}
) )
func (a auth) IsAuthenticated(ctx context.Context, r *http.Request, isCookieAuth, isKeyAuth bool) (context.Context, error) { func (a auth) IsAuthenticated(ctx context.Context, r *http.Request, isCookieAuth, isKeyAuth bool) (context.Context, error) {
@ -44,8 +39,42 @@ func (a auth) IsAuthenticated(ctx context.Context, r *http.Request, isCookieAuth
func (a auth) RemoveAuthCookie(w http.ResponseWriter) {} func (a auth) RemoveAuthCookie(w http.ResponseWriter) {}
func (s service) GenTestAPI(ctx context.Context, pathParam string, id uuid.UUID, date time.Time, body struct{ Content string }) (*response, api.HTTPError) { func (s service) Get(
return &response{ ctx context.Context,
) ([]myapi.Document, api.HTTPError) {
return []myapi.Document{}, api.HTTPError{}
}
func (s service) GetOne(
ctx context.Context,
pathParam string,
) (*myapi.Document, api.HTTPError) {
return &myapi.Document{}, api.HTTPError{}
}
func (s service) GetTag(
ctx context.Context,
pathParam string,
tagName string,
) (*[2]string, api.HTTPError) {
return &[2]string{}, api.HTTPError{}
}
func (s service) GetVersions(
ctx context.Context,
pathParam string,
) ([]myapi.Version, api.HTTPError) {
return []myapi.Version{}, api.HTTPError{}
}
func (s service) UpdateContent(
ctx context.Context,
pathParam string,
id uuid.UUID,
date time.Time,
body myapi.NewDocument,
) (*myapi.Document, api.HTTPError) {
return &myapi.Document{
ID: id, ID: id,
Date: date, Date: date,
PathParam: pathParam, PathParam: pathParam,
@ -53,7 +82,9 @@ func (s service) GenTestAPI(ctx context.Context, pathParam string, id uuid.UUID,
}, api.HTTPError{} }, api.HTTPError{}
} }
func send(ctx context.Context, method string, url string, body interface{}) ([]byte, error) { func send(ctx context.Context, t *testing.T, method string, url string, body interface{}) ([]byte, error) {
t.Helper()
var bodyReader io.Reader = http.NoBody var bodyReader io.Reader = http.NoBody
if body != nil { if body != nil {
bodyJSON, err := json.Marshal(body) bodyJSON, err := json.Marshal(body)
@ -73,6 +104,10 @@ func send(ctx context.Context, method string, url string, body interface{}) ([]b
return nil, err return nil, err
} }
if c := resp.StatusCode; c != http.StatusOK {
t.Fatalf("unexpected status code. Want=%d, Got=%d", http.StatusOK, c)
}
respBody, err := io.ReadAll(resp.Body) respBody, err := io.ReadAll(resp.Body)
if err != nil { if err != nil {
return nil, err return nil, err
@ -90,7 +125,7 @@ func TestAPIServer(t *testing.T) {
defer ctx.Cleanup() defer ctx.Cleanup()
router := mux.NewRouter() router := mux.NewRouter()
example.NewTestAPI(zaptest.NewLogger(t), monkit.Package(), service{}, router, auth{}) example.NewDocuments(zaptest.NewLogger(t), monkit.Package(), service{}, router, auth{})
server := httptest.NewServer(router) server := httptest.NewServer(router)
defer server.Close() defer server.Close()
@ -98,15 +133,15 @@ func TestAPIServer(t *testing.T) {
id, err := uuid.New() id, err := uuid.New()
require.NoError(t, err) require.NoError(t, err)
expected := response{ expected := myapi.Document{
ID: id, ID: id,
Date: time.Now(), Date: time.Now(),
PathParam: "foo", PathParam: "foo",
Body: "bar", Body: "bar",
} }
resp, err := send(ctx, http.MethodPost, resp, err := send(ctx, t, http.MethodPost,
fmt.Sprintf("%s/api/v0/testapi/%s?id=%s&date=%s", fmt.Sprintf("%s/api/v0/docs/%s?id=%s&date=%s",
server.URL, server.URL,
expected.PathParam, expected.PathParam,
url.QueryEscape(expected.ID.String()), url.QueryEscape(expected.ID.String()),
@ -115,13 +150,16 @@ func TestAPIServer(t *testing.T) {
) )
require.NoError(t, err) require.NoError(t, err)
var actual map[string]string fmt.Println(string(resp))
var actual map[string]any
require.NoError(t, json.Unmarshal(resp, &actual)) require.NoError(t, json.Unmarshal(resp, &actual))
for _, key := range []string{"ID", "Date", "PathParam", "Body"} { for _, key := range []string{"id", "date", "pathParam", "body"} {
require.Contains(t, actual, key) require.Contains(t, actual, key)
} }
require.Equal(t, expected.ID.String(), actual["ID"]) require.Equal(t, expected.ID.String(), actual["id"].(string))
require.Equal(t, expected.Date.Format(apigen.DateFormat), actual["Date"]) require.Equal(t, expected.Date.Format(apigen.DateFormat), actual["date"].(string))
require.Equal(t, expected.Body, actual["Body"]) require.Equal(t, expected.PathParam, actual["pathParam"].(string))
require.Equal(t, expected.Body, actual["body"].(string))
} }

View File

@ -12,7 +12,10 @@ import (
"github.com/zeebo/errs" "github.com/zeebo/errs"
) )
// MustWriteTS writes generated TypeScript code into a file. // MustWriteTS writes generated TypeScript code into a file indicated by path.
// The generated code is an API client to run in the browser.
//
// If an error occurs, it panics.
func (a *API) MustWriteTS(path string) { func (a *API) MustWriteTS(path string) {
f := newTSGenFile(path, a) f := newTSGenFile(path, a)
@ -57,8 +60,18 @@ func (f *tsGenFile) generateTS() {
f.registerTypes() f.registerTypes()
f.result += f.types.GenerateTypescriptDefinitions() f.result += f.types.GenerateTypescriptDefinitions()
f.result += `
class APIError extends Error {
constructor(
public readonly msg: string,
public readonly responseStatusCode?: number,
) {
super(msg);
}
}
`
for _, group := range f.api.EndpointGroups { for _, group := range f.api.EndpointGroups {
// Not sure if this is a good name
f.createAPIClient(group) f.createAPIClient(group)
} }
} }
@ -83,45 +96,50 @@ func (f *tsGenFile) registerTypes() {
} }
func (f *tsGenFile) createAPIClient(group *EndpointGroup) { func (f *tsGenFile) createAPIClient(group *EndpointGroup) {
f.pf("\nexport class %sHttpApi%s {", group.Prefix, strings.ToUpper(f.api.Version)) f.pf("\nexport class %sHttpApi%s {", capitalize(group.Name), strings.ToUpper(f.api.Version))
f.pf("\tprivate readonly http: HttpClient = new HttpClient();") f.pf("\tprivate readonly http: HttpClient = new HttpClient();")
f.pf("\tprivate readonly ROOT_PATH: string = '/api/%s/%s';", f.api.Version, group.Prefix) f.pf("\tprivate readonly ROOT_PATH: string = '%s/%s';", f.api.endpointBasePath(), strings.ToLower(group.Prefix))
for _, method := range group.endpoints { for _, method := range group.endpoints {
f.pf("") f.pf("")
funcArgs, path := f.getArgsAndPath(method) funcArgs, path := f.getArgsAndPath(method, group)
returnStmt := "return" returnStmt := "return"
returnType := "void" returnType := "void"
if method.Response != nil { if method.Response != nil {
returnType = TypescriptTypeName(getElementaryType(reflect.TypeOf(method.Response))) returnType = TypescriptTypeName(reflect.TypeOf(method.Response))
if v := reflect.ValueOf(method.Response); v.Kind() == reflect.Array || v.Kind() == reflect.Slice {
returnType = fmt.Sprintf("Array<%s>", returnType)
}
returnStmt += fmt.Sprintf(" response.json().then((body) => body as %s)", returnType) returnStmt += fmt.Sprintf(" response.json().then((body) => body as %s)", returnType)
} }
returnStmt += ";" returnStmt += ";"
f.pf("\tpublic async %s(%s): Promise<%s> {", method.RequestName, funcArgs, returnType) f.pf("\tpublic async %s(%s): Promise<%s> {", method.TypeScriptName, funcArgs, returnType)
f.pf("\t\tconst path = `%s`;", path) if len(method.QueryParams) > 0 {
f.pf("\t\tconst u = new URL(`%s`, window.location.href);", path)
for _, p := range method.QueryParams {
f.pf("\t\tu.searchParams.set('%s', %s);", p.Name, p.Name)
}
f.pf("\t\tconst fullPath = u.toString();")
} else {
f.pf("\t\tconst fullPath = `%s`;", path)
}
if method.Request != nil { if method.Request != nil {
f.pf("\t\tconst response = await this.http.%s(path, JSON.stringify(request));", strings.ToLower(method.Method)) f.pf("\t\tconst response = await this.http.%s(fullPath, JSON.stringify(request));", strings.ToLower(method.Method))
} else { } else {
f.pf("\t\tconst response = await this.http.%s(path);", strings.ToLower(method.Method)) f.pf("\t\tconst response = await this.http.%s(fullPath);", strings.ToLower(method.Method))
} }
f.pf("\t\tif (response.ok) {") f.pf("\t\tif (response.ok) {")
f.pf("\t\t\t%s", returnStmt) f.pf("\t\t\t%s", returnStmt)
f.pf("\t\t}") f.pf("\t\t}")
f.pf("\t\tconst err = await response.json();") f.pf("\t\tconst err = await response.json();")
f.pf("\t\tthrow new Error(err.error);") f.pf("\t\tthrow new APIError(err.error, response.status);")
f.pf("\t}") f.pf("\t}")
} }
f.pf("}") f.pf("}")
} }
func (f *tsGenFile) getArgsAndPath(method *fullEndpoint) (funcArgs, path string) { func (f *tsGenFile) getArgsAndPath(method *FullEndpoint, group *EndpointGroup) (funcArgs, path string) {
// remove path parameter placeholders // remove path parameter placeholders
path = method.Path path = method.Path
i := strings.Index(path, "{") i := strings.Index(path, "{")
@ -131,8 +149,7 @@ func (f *tsGenFile) getArgsAndPath(method *fullEndpoint) (funcArgs, path string)
path = "${this.ROOT_PATH}" + path path = "${this.ROOT_PATH}" + path
if method.Request != nil { if method.Request != nil {
t := getElementaryType(reflect.TypeOf(method.Request)) funcArgs += fmt.Sprintf("request: %s, ", TypescriptTypeName(reflect.TypeOf(method.Request)))
funcArgs += fmt.Sprintf("request: %s, ", TypescriptTypeName(t))
} }
for _, p := range method.PathParams { for _, p := range method.PathParams {
@ -140,15 +157,8 @@ func (f *tsGenFile) getArgsAndPath(method *fullEndpoint) (funcArgs, path string)
path += fmt.Sprintf("/${%s}", p.Name) path += fmt.Sprintf("/${%s}", p.Name)
} }
for i, p := range method.QueryParams { for _, p := range method.QueryParams {
if i == 0 {
path += "?"
} else {
path += "&"
}
funcArgs += fmt.Sprintf("%s: %s, ", p.Name, TypescriptTypeName(p.Type)) funcArgs += fmt.Sprintf("%s: %s, ", p.Name, TypescriptTypeName(p.Type))
path += fmt.Sprintf("%s=${%s}", p.Name, p.Name)
} }
path = strings.ReplaceAll(path, "//", "/") path = strings.ReplaceAll(path, "//", "/")

129
private/apigen/tsgenmock.go Normal file
View File

@ -0,0 +1,129 @@
// Copyright (C) 2023 Storj Labs, Inc.
// See LICENSE for copying information.
package apigen
import (
"encoding/json"
"fmt"
"reflect"
"strings"
"github.com/zeebo/errs"
)
// MustWriteTSMock writes generated TypeScript code into a file indicated by path.
// The generated code is an API client mock to run in the browser.
//
// If an error occurs, it panics.
func (a *API) MustWriteTSMock(path string) {
f := newTSGenMockFile(path, a)
f.generateTS()
err := f.write()
if err != nil {
panic(errs.Wrap(err))
}
}
type tsGenMockFile struct {
*tsGenFile
}
func newTSGenMockFile(filepath string, api *API) *tsGenMockFile {
return &tsGenMockFile{
tsGenFile: newTSGenFile(filepath, api),
}
}
func (f *tsGenMockFile) generateTS() {
f.pf("// AUTOGENERATED BY private/apigen")
f.pf("// DO NOT EDIT.")
f.registerTypes()
f.result += f.types.GenerateTypescriptDefinitions()
f.result += `
class APIError extends Error {
constructor(
public readonly msg: string,
public readonly responseStatusCode?: number,
) {
super(msg);
}
}
`
for _, group := range f.api.EndpointGroups {
f.createAPIClient(group)
}
}
func (f *tsGenMockFile) createAPIClient(group *EndpointGroup) {
f.pf("\nexport class %sHttpApi%s {", capitalize(group.Name), strings.ToUpper(f.api.Version))
// Properties.
f.pf("\tpublic readonly respStatusCode: number;")
f.pf("")
// Constructor
f.pf("\t// When respStatuscode is passed, the client throws an APIError on each method call")
f.pf("\t// with respStatusCode as HTTP status code.")
f.pf("\t// respStatuscode must be equal or greater than 400")
f.pf("\tconstructor(respStatusCode?: number) {")
f.pf("\t\tif (typeof respStatusCode === 'undefined') {")
f.pf("\t\t\tthis.respStatusCode = 0;")
f.pf("\t\t\treturn;")
f.pf("\t\t}")
f.pf("")
f.pf("\t\tif (respStatusCode < 400) {")
f.pf("\t\t\tthrow new Error('invalid response status code for API Error, it must be greater or equal than 400');")
f.pf("\t\t}")
f.pf("")
f.pf("\t\tthis.respStatusCode = respStatusCode;")
f.pf("\t}")
// Methods to call API endpoints.
for _, method := range group.endpoints {
f.pf("")
funcArgs, _ := f.getArgsAndPath(method, group)
returnType := "void"
if method.Response != nil {
if method.ResponseMock == nil {
panic(
fmt.Sprintf(
"ResponseMock is nil and Response isn't nil. Endpoint.Method=%q, Endpoint.Path=%q",
method.Method, method.Path,
))
}
returnType = TypescriptTypeName(reflect.TypeOf(method.Response))
}
f.pf("\tpublic async %s(%s): Promise<%s> {", method.TypeScriptName, funcArgs, returnType)
f.pf("\t\tif (this.respStatusCode !== 0) {")
f.pf("\t\t\tthrow new APIError('mock error message: ' + this.respStatusCode, this.respStatusCode);")
f.pf("\t\t}")
f.pf("")
if method.ResponseMock != nil {
res, err := json.Marshal(method.ResponseMock)
if err != nil {
panic(
fmt.Sprintf(
"error when marshaling ResponseMock: %+v. Endpoint.Method=%q, Endpoint.Path=%q",
err, method.Method, method.Path,
))
}
f.pf("\t\treturn JSON.parse('%s') as %s;", string(res), returnType)
} else {
f.pf("\t\treturn;")
}
f.pf("\t}")
}
f.pf("}")
}

View File

@ -36,41 +36,60 @@ type Types struct {
// Register registers a type for generation. // Register registers a type for generation.
func (types *Types) Register(t reflect.Type) { func (types *Types) Register(t reflect.Type) {
if t.Name() == "" {
switch t.Kind() {
case reflect.Array, reflect.Slice, reflect.Ptr:
if t.Elem().Name() == "" {
panic(
fmt.Sprintf("register an %q of elements of an anonymous type is not supported", t.Name()),
)
}
default:
panic("register an anonymous type is not supported. All the types must have a name")
}
}
types.top[t] = struct{}{} types.top[t] = struct{}{}
} }
// All returns a slice containing every top-level type and their dependencies. // All returns a map containing every top-level and their dependency types with their associated name.
func (types *Types) All() []reflect.Type { func (types *Types) All() map[reflect.Type]string {
seen := map[reflect.Type]struct{}{} all := map[reflect.Type]string{}
all := []reflect.Type{}
var walk func(t reflect.Type) var walk func(t reflect.Type)
walk = func(t reflect.Type) { walk = func(t reflect.Type) {
if _, ok := seen[t]; ok { if _, ok := all[t]; ok {
return
}
seen[t] = struct{}{}
all = append(all, t)
if _, ok := commonClasses[t]; ok {
return return
} }
switch t.Kind() { if n, ok := commonClasses[t]; ok {
case reflect.Array, reflect.Ptr, reflect.Slice: all[t] = n
return
}
switch k := t.Kind(); k {
case reflect.Ptr:
walk(t.Elem())
case reflect.Array, reflect.Slice:
walk(t.Elem()) walk(t.Elem())
case reflect.Struct: case reflect.Struct:
for i := 0; i < t.NumField(); i++ { if t.Name() == "" {
walk(t.Field(i).Type) panic(fmt.Sprintf("BUG: found an anonymous 'struct'. Found type=%q", t))
} }
case reflect.Bool:
case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64: all[t] = t.Name()
case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64:
case reflect.Float32, reflect.Float64: for i := 0; i < t.NumField(); i++ {
case reflect.String: field := t.Field(i)
break walk(field.Type)
}
case reflect.Bool,
reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64,
reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64,
reflect.Float32, reflect.Float64,
reflect.String:
all[t] = t.Name()
default: default:
panic(fmt.Sprintf("type '%s' is not supported", t.Kind().String())) panic(fmt.Sprintf("type %q is not supported", t.Kind().String()))
} }
} }
@ -78,10 +97,6 @@ func (types *Types) All() []reflect.Type {
walk(t) walk(t)
} }
sort.Slice(all, func(i, j int) bool {
return strings.Compare(all[i].Name(), all[j].Name()) < 0
})
return all return all
} }
@ -90,40 +105,44 @@ func (types *Types) GenerateTypescriptDefinitions() string {
var out StringBuilder var out StringBuilder
pf := out.Writelnf pf := out.Writelnf
pf(types.getTypescriptImports()) {
i := types.getTypescriptImports()
if i != "" {
pf(i)
}
}
all := filter(types.All(), func(t reflect.Type) bool { allTypes := types.All()
if _, ok := commonClasses[t]; ok { namedTypes := mapToSlice(allTypes)
allStructs := filter(namedTypes, func(tn typeAndName) bool {
if _, ok := commonClasses[tn.Type]; ok {
return false return false
} }
return t.Kind() == reflect.Struct
return tn.Type.Kind() == reflect.Struct
}) })
for _, t := range all { for _, t := range allStructs {
func() { func() {
pf("\nexport class %s {", t.Name()) name := capitalize(t.Name)
pf("\nexport class %s {", name)
defer pf("}") defer pf("}")
for i := 0; i < t.NumField(); i++ { for i := 0; i < t.Type.NumField(); i++ {
field := t.Field(i) field := t.Type.Field(i)
attributes := strings.Fields(field.Tag.Get("json")) jsonInfo := parseJSONTag(t.Type, field)
if len(attributes) == 0 || attributes[0] == "" { if jsonInfo.Skip {
pathParts := strings.Split(t.PkgPath(), "/")
pkg := pathParts[len(pathParts)-1]
panic(fmt.Sprintf("(%s.%s).%s missing json declaration", pkg, t.Name(), field.Name))
}
jsonField := attributes[0]
if jsonField == "-" {
continue continue
} }
isOptional := "" var isOptional, isNullable string
if isNillableType(t) { if jsonInfo.OmitEmpty {
isOptional = "?" isOptional = "?"
} else if isNillableType(field.Type) {
isNullable = " | null"
} }
pf("\t%s%s: %s;", jsonField, isOptional, TypescriptTypeName(field.Type)) pf("\t%s%s: %s%s;", jsonInfo.FieldName, isOptional, TypescriptTypeName(field.Type), isNullable)
} }
}() }()
} }
@ -135,8 +154,7 @@ func (types *Types) GenerateTypescriptDefinitions() string {
func (types *Types) getTypescriptImports() string { func (types *Types) getTypescriptImports() string {
classes := []string{} classes := []string{}
all := types.All() for t := range types.All() {
for _, t := range all {
if tsClass, ok := commonClasses[t]; ok { if tsClass, ok := commonClasses[t]; ok {
classes = append(classes, tsClass) classes = append(classes, tsClass)
} }
@ -154,6 +172,7 @@ func (types *Types) getTypescriptImports() string {
} }
// TypescriptTypeName gets the corresponding TypeScript type for a provided reflect.Type. // TypescriptTypeName gets the corresponding TypeScript type for a provided reflect.Type.
// If the type is an anonymous struct, it returns an empty string.
func TypescriptTypeName(t reflect.Type) string { func TypescriptTypeName(t reflect.Type) string {
if override, ok := commonClasses[t]; ok { if override, ok := commonClasses[t]; ok {
return override return override
@ -162,15 +181,18 @@ func TypescriptTypeName(t reflect.Type) string {
switch t.Kind() { switch t.Kind() {
case reflect.Ptr: case reflect.Ptr:
return TypescriptTypeName(t.Elem()) return TypescriptTypeName(t.Elem())
case reflect.Slice: case reflect.Array, reflect.Slice:
if t.Name() != "" {
return capitalize(t.Name())
}
// []byte ([]uint8) is marshaled as a base64 string // []byte ([]uint8) is marshaled as a base64 string
elem := t.Elem() elem := t.Elem()
if elem.Kind() == reflect.Uint8 { if elem.Kind() == reflect.Uint8 {
return "string" return "string"
} }
fallthrough
case reflect.Array: return TypescriptTypeName(elem) + "[]"
return TypescriptTypeName(t.Elem()) + "[]"
case reflect.String: case reflect.String:
return "string" return "string"
case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64: case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
@ -182,8 +204,11 @@ func TypescriptTypeName(t reflect.Type) string {
case reflect.Bool: case reflect.Bool:
return "boolean" return "boolean"
case reflect.Struct: case reflect.Struct:
return t.Name() if t.Name() == "" {
panic(fmt.Sprintf(`anonymous struct aren't accepted because their type doesn't have a name. Type="%+v"`, t))
}
return capitalize(t.Name())
default: default:
panic("unhandled type: " + t.Name()) panic(fmt.Sprintf(`unhandled type. Type="%+v"`, t))
} }
} }

View File

@ -0,0 +1,81 @@
// Copyright (C) 2023 Storj Labs, Inc.
// See LICENSE for copying information.
package apigen
import (
"reflect"
"testing"
"github.com/stretchr/testify/require"
)
type testTypesValoration struct {
Points uint
}
func TestTypes(t *testing.T) {
t.Run("Register panics with some anonymous types", func(t *testing.T) {
types := NewTypes()
require.Panics(t, func() {
types.Register(reflect.TypeOf([2]struct{}{}))
}, "array")
require.Panics(t, func() {
types.Register(reflect.TypeOf([]struct{}{}))
}, "slice")
require.Panics(t, func() {
types.Register(reflect.TypeOf(struct{}{}))
}, "struct")
})
t.Run("All returns nested types", func(t *testing.T) {
typesList := []reflect.Type{
reflect.TypeOf(true),
reflect.TypeOf(int64(10)),
reflect.TypeOf(uint8(9)),
reflect.TypeOf(float64(99.9)),
reflect.TypeOf("this is a test"),
reflect.TypeOf(testTypesValoration{}),
}
types := NewTypes()
for _, li := range typesList {
types.Register(li)
}
allTypes := types.All()
require.Len(t, allTypes, 7, "total number of types")
require.Subset(t, allTypes, typesList, "all types contains at least the registered ones")
})
t.Run("Anonymous types panics", func(t *testing.T) {
type Address struct {
Address string
PO string
}
type Job struct {
Company string
Position string
StartingYear uint
ContractClauses []struct { // This is what it makes Types.All to panic
ClauseID uint
CauseDesc string
}
}
type Citizen struct {
Name string
Addresses []Address
Job Job
}
types := NewTypes()
types.Register(reflect.TypeOf(Citizen{}))
require.Panics(t, func() {
types.All()
})
})
}

View File

@ -27,7 +27,9 @@ message DiskSpaceResponse {
int64 allocated = 1; int64 allocated = 1;
int64 used_pieces = 2; int64 used_pieces = 2;
int64 used_trash = 3; int64 used_trash = 3;
// Free is the actual amount of free space on the whole disk, not just allocated disk space, in bytes.
int64 free = 4; int64 free = 4;
// Available is the amount of free space on the allocated disk space, in bytes.
int64 available = 5; int64 available = 5;
int64 overused = 6; int64 overused = 6;
} }

View File

@ -55,18 +55,20 @@ func (sender *SMTPSender) communicate(ctx context.Context, client *smtp.Client,
// before creating SMTPSender // before creating SMTPSender
host, _, _ := net.SplitHostPort(sender.ServerAddress) host, _, _ := net.SplitHostPort(sender.ServerAddress)
// send smtp hello or ehlo msg and establish connection over tls if sender.Auth != nil {
err := client.StartTLS(&tls.Config{ServerName: host}) // send smtp hello or ehlo msg and establish connection over tls
if err != nil { err := client.StartTLS(&tls.Config{ServerName: host})
return err if err != nil {
return err
}
err = client.Auth(sender.Auth)
if err != nil {
return err
}
} }
err = client.Auth(sender.Auth) err := client.Mail(sender.From.Address)
if err != nil {
return err
}
err = client.Mail(sender.From.Address)
if err != nil { if err != nil {
return err return err
} }

View File

@ -0,0 +1,51 @@
// Copyright (C) 2023 Storj Labs, Inc.
// See LICENSE for copying information.
package server
import (
"os/exec"
"strconv"
"strings"
"sync"
"syscall"
"go.uber.org/zap"
)
const tcpFastOpen = 1025
func setTCPFastOpen(fd uintptr, _queue int) error {
return syscall.SetsockoptInt(int(fd), syscall.IPPROTO_TCP, tcpFastOpen, 1)
}
var tryInitFastOpenOnce sync.Once
var initFastOpenPossiblyEnabled bool
// tryInitFastOpen returns true if fastopen support is possibly enabled.
func tryInitFastOpen(log *zap.Logger) bool {
tryInitFastOpenOnce.Do(func() {
initFastOpenPossiblyEnabled = true
output, err := exec.Command("sysctl", "-n", "net.inet.tcp.fastopen.server_enable").Output()
if err != nil {
log.Sugar().Infof("kernel support for tcp fast open unknown")
initFastOpenPossiblyEnabled = true
return
}
enabled, err := strconv.ParseBool(strings.TrimSpace(string(output)))
if err != nil {
log.Sugar().Infof("kernel support for tcp fast open unparsable")
initFastOpenPossiblyEnabled = true
return
}
if enabled {
log.Sugar().Infof("kernel support for server-side tcp fast open enabled.")
} else {
log.Sugar().Infof("kernel support for server-side tcp fast open not enabled.")
log.Sugar().Infof("enable with: sysctl net.inet.tcp.fastopen.server_enable=1")
log.Sugar().Infof("enable on-boot by setting net.inet.tcp.fastopen.server_enable=1 in /etc/sysctl.conf")
}
initFastOpenPossiblyEnabled = enabled
})
return initFastOpenPossiblyEnabled
}

View File

@ -1,8 +1,8 @@
// Copyright (C) 2023 Storj Labs, Inc. // Copyright (C) 2023 Storj Labs, Inc.
// See LICENSE for copying information. // See LICENSE for copying information.
//go:build !linux && !windows //go:build !linux && !windows && !freebsd
// +build !linux,!windows // +build !linux,!windows,!freebsd
package server package server

View File

@ -4,22 +4,44 @@
package server package server
import ( import (
"context"
"net"
"sync"
"syscall" "syscall"
"go.uber.org/zap" "go.uber.org/zap"
) )
const tcpFastOpenServer = 15 const tcpFastOpen = 15 // Corresponds to TCP_FASTOPEN from MS SDK
func setTCPFastOpen(fd uintptr, queue int) error { func setTCPFastOpen(fd uintptr, queue int) error {
return syscall.SetsockoptInt(syscall.Handle(fd), syscall.IPPROTO_TCP, tcpFastOpenServer, 1) return syscall.SetsockoptInt(syscall.Handle(fd), syscall.IPPROTO_TCP, tcpFastOpen, 1)
} }
var tryInitFastOpenOnce sync.Once
var initFastOpenPossiblyEnabled bool
// tryInitFastOpen returns true if fastopen support is possibly enabled. // tryInitFastOpen returns true if fastopen support is possibly enabled.
func tryInitFastOpen(*zap.Logger) bool { func tryInitFastOpen(*zap.Logger) bool {
// should we log or check something along the lines of tryInitFastOpenOnce.Do(func() {
// netsh int tcp set global fastopen=enabled // TCP-FASTOPEN is supported as of Windows 10 build 1607, but is
// netsh int tcp set global fastopenfallback=disabled // enabled per socket. If the socket option isn't supported then the
// ? // call to opt-in will fail. So as long as we can set up a listening
return false // socket with the right socket option set, we should be good.
if listener, err := (&net.ListenConfig{
Control: func(network, addr string, c syscall.RawConn) error {
var sockOptErr error
if controlErr := c.Control(func(fd uintptr) {
sockOptErr = setTCPFastOpen(fd, 0) // queue is unused
}); controlErr != nil {
return controlErr
}
return sockOptErr
},
}).Listen(context.Background(), "tcp", "127.0.0.1:0"); err == nil {
listener.Close()
initFastOpenPossiblyEnabled = true
}
})
return initFastOpenPossiblyEnabled
} }

View File

@ -91,7 +91,7 @@ func (planet *Planet) newMultinode(ctx context.Context, prefix string, index int
config := multinode.Config{ config := multinode.Config{
Debug: debug.Config{ Debug: debug.Config{
Address: "", Addr: "",
}, },
Console: server.Config{ Console: server.Config{
Address: "127.0.0.1:0", Address: "127.0.0.1:0",

View File

@ -66,10 +66,10 @@ type Satellite struct {
Core *satellite.Core Core *satellite.Core
API *satellite.API API *satellite.API
UI *satellite.UI
Repairer *satellite.Repairer Repairer *satellite.Repairer
Auditor *satellite.Auditor Auditor *satellite.Auditor
Admin *satellite.Admin Admin *satellite.Admin
GC *satellite.GarbageCollection
GCBF *satellite.GarbageCollectionBF GCBF *satellite.GarbageCollectionBF
RangedLoop *satellite.RangedLoop RangedLoop *satellite.RangedLoop
@ -173,12 +173,17 @@ type Satellite struct {
Service *mailservice.Service Service *mailservice.Service
} }
Console struct { ConsoleBackend struct {
Listener net.Listener Listener net.Listener
Service *console.Service Service *console.Service
Endpoint *consoleweb.Server Endpoint *consoleweb.Server
} }
ConsoleFrontend struct {
Listener net.Listener
Endpoint *consoleweb.Server
}
NodeStats struct { NodeStats struct {
Endpoint *nodestats.Endpoint Endpoint *nodestats.Endpoint
} }
@ -256,7 +261,7 @@ func (system *Satellite) AddProject(ctx context.Context, ownerID uuid.UUID, name
if err != nil { if err != nil {
return nil, errs.Wrap(err) return nil, errs.Wrap(err)
} }
project, err := system.API.Console.Service.CreateProject(ctx, console.ProjectInfo{ project, err := system.API.Console.Service.CreateProject(ctx, console.UpsertProjectInfo{
Name: name, Name: name,
}) })
if err != nil { if err != nil {
@ -285,7 +290,6 @@ func (system *Satellite) Close() error {
system.Repairer.Close(), system.Repairer.Close(),
system.Auditor.Close(), system.Auditor.Close(),
system.Admin.Close(), system.Admin.Close(),
system.GC.Close(),
system.GCBF.Close(), system.GCBF.Close(),
) )
} }
@ -300,6 +304,11 @@ func (system *Satellite) Run(ctx context.Context) (err error) {
group.Go(func() error { group.Go(func() error {
return errs2.IgnoreCanceled(system.API.Run(ctx)) return errs2.IgnoreCanceled(system.API.Run(ctx))
}) })
if system.UI != nil {
group.Go(func() error {
return errs2.IgnoreCanceled(system.UI.Run(ctx))
})
}
group.Go(func() error { group.Go(func() error {
return errs2.IgnoreCanceled(system.Repairer.Run(ctx)) return errs2.IgnoreCanceled(system.Repairer.Run(ctx))
}) })
@ -309,9 +318,6 @@ func (system *Satellite) Run(ctx context.Context) (err error) {
group.Go(func() error { group.Go(func() error {
return errs2.IgnoreCanceled(system.Admin.Run(ctx)) return errs2.IgnoreCanceled(system.Admin.Run(ctx))
}) })
group.Go(func() error {
return errs2.IgnoreCanceled(system.GC.Run(ctx))
})
group.Go(func() error { group.Go(func() error {
return errs2.IgnoreCanceled(system.GCBF.Run(ctx)) return errs2.IgnoreCanceled(system.GCBF.Run(ctx))
}) })
@ -405,6 +411,7 @@ func (planet *Planet) newSatellite(ctx context.Context, prefix string, index int
// cfgstruct devDefaults. we need to make sure it's safe to remove // cfgstruct devDefaults. we need to make sure it's safe to remove
// these lines and then remove them. // these lines and then remove them.
config.Debug.Control = false config.Debug.Control = false
config.Debug.Addr = ""
config.Reputation.AuditHistory.OfflineDQEnabled = false config.Reputation.AuditHistory.OfflineDQEnabled = false
config.Server.Config.Extensions.Revocation = false config.Server.Config.Extensions.Revocation = false
config.Orders.OrdersSemaphoreSize = 0 config.Orders.OrdersSemaphoreSize = 0
@ -458,6 +465,10 @@ func (planet *Planet) newSatellite(ctx context.Context, prefix string, index int
config.Console.StaticDir = filepath.Join(developmentRoot, "web/satellite") config.Console.StaticDir = filepath.Join(developmentRoot, "web/satellite")
config.Payments.Storjscan.DisableLoop = true config.Payments.Storjscan.DisableLoop = true
if os.Getenv("STORJ_TEST_DISABLEQUIC") != "" {
config.Server.DisableQUIC = true
}
if planet.config.Reconfigure.Satellite != nil { if planet.config.Reconfigure.Satellite != nil {
planet.config.Reconfigure.Satellite(log, index, &config) planet.config.Reconfigure.Satellite(log, index, &config)
} }
@ -524,6 +535,15 @@ func (planet *Planet) newSatellite(ctx context.Context, prefix string, index int
return nil, errs.Wrap(err) return nil, errs.Wrap(err)
} }
// only run if front-end endpoints on console back-end server are disabled.
var ui *satellite.UI
if !config.Console.FrontendEnable {
ui, err = planet.newUI(ctx, index, identity, config, api.ExternalAddress, api.Console.Listener.Addr().String())
if err != nil {
return nil, errs.Wrap(err)
}
}
adminPeer, err := planet.newAdmin(ctx, index, identity, db, metabaseDB, config, versionInfo) adminPeer, err := planet.newAdmin(ctx, index, identity, db, metabaseDB, config, versionInfo)
if err != nil { if err != nil {
return nil, errs.Wrap(err) return nil, errs.Wrap(err)
@ -539,11 +559,6 @@ func (planet *Planet) newSatellite(ctx context.Context, prefix string, index int
return nil, errs.Wrap(err) return nil, errs.Wrap(err)
} }
gcPeer, err := planet.newGarbageCollection(ctx, index, identity, db, metabaseDB, config, versionInfo)
if err != nil {
return nil, errs.Wrap(err)
}
gcBFPeer, err := planet.newGarbageCollectionBF(ctx, index, db, metabaseDB, config, versionInfo) gcBFPeer, err := planet.newGarbageCollectionBF(ctx, index, db, metabaseDB, config, versionInfo)
if err != nil { if err != nil {
return nil, errs.Wrap(err) return nil, errs.Wrap(err)
@ -558,23 +573,23 @@ func (planet *Planet) newSatellite(ctx context.Context, prefix string, index int
peer.Mail.EmailReminders.TestSetLinkAddress("http://" + api.Console.Listener.Addr().String() + "/") peer.Mail.EmailReminders.TestSetLinkAddress("http://" + api.Console.Listener.Addr().String() + "/")
} }
return createNewSystem(prefix, log, config, peer, api, repairerPeer, auditorPeer, adminPeer, gcPeer, gcBFPeer, rangedLoopPeer), nil return createNewSystem(prefix, log, config, peer, api, ui, repairerPeer, auditorPeer, adminPeer, gcBFPeer, rangedLoopPeer), nil
} }
// createNewSystem makes a new Satellite System and exposes the same interface from // createNewSystem makes a new Satellite System and exposes the same interface from
// before we split out the API. In the short term this will help keep all the tests passing // before we split out the API. In the short term this will help keep all the tests passing
// without much modification needed. However long term, we probably want to rework this // without much modification needed. However long term, we probably want to rework this
// so it represents how the satellite will run when it is made up of many processes. // so it represents how the satellite will run when it is made up of many processes.
func createNewSystem(name string, log *zap.Logger, config satellite.Config, peer *satellite.Core, api *satellite.API, repairerPeer *satellite.Repairer, auditorPeer *satellite.Auditor, adminPeer *satellite.Admin, gcPeer *satellite.GarbageCollection, gcBFPeer *satellite.GarbageCollectionBF, rangedLoopPeer *satellite.RangedLoop) *Satellite { func createNewSystem(name string, log *zap.Logger, config satellite.Config, peer *satellite.Core, api *satellite.API, ui *satellite.UI, repairerPeer *satellite.Repairer, auditorPeer *satellite.Auditor, adminPeer *satellite.Admin, gcBFPeer *satellite.GarbageCollectionBF, rangedLoopPeer *satellite.RangedLoop) *Satellite {
system := &Satellite{ system := &Satellite{
Name: name, Name: name,
Config: config, Config: config,
Core: peer, Core: peer,
API: api, API: api,
UI: ui,
Repairer: repairerPeer, Repairer: repairerPeer,
Auditor: auditorPeer, Auditor: auditorPeer,
Admin: adminPeer, Admin: adminPeer,
GC: gcPeer,
GCBF: gcBFPeer, GCBF: gcBFPeer,
RangedLoop: rangedLoopPeer, RangedLoop: rangedLoopPeer,
} }
@ -622,7 +637,7 @@ func createNewSystem(name string, log *zap.Logger, config satellite.Config, peer
system.Audit.Reporter = auditorPeer.Audit.Reporter system.Audit.Reporter = auditorPeer.Audit.Reporter
system.Audit.ContainmentSyncChore = peer.Audit.ContainmentSyncChore system.Audit.ContainmentSyncChore = peer.Audit.ContainmentSyncChore
system.GarbageCollection.Sender = gcPeer.GarbageCollection.Sender system.GarbageCollection.Sender = peer.GarbageCollection.Sender
system.ExpiredDeletion.Chore = peer.ExpiredDeletion.Chore system.ExpiredDeletion.Chore = peer.ExpiredDeletion.Chore
system.ZombieDeletion.Chore = peer.ZombieDeletion.Chore system.ZombieDeletion.Chore = peer.ZombieDeletion.Chore
@ -666,13 +681,28 @@ func (planet *Planet) newAPI(ctx context.Context, index int, identity *identity.
return satellite.NewAPI(log, identity, db, metabaseDB, revocationDB, liveAccounting, rollupsWriteCache, &config, versionInfo, nil) return satellite.NewAPI(log, identity, db, metabaseDB, revocationDB, liveAccounting, rollupsWriteCache, &config, versionInfo, nil)
} }
func (planet *Planet) newUI(ctx context.Context, index int, identity *identity.FullIdentity, config satellite.Config, satelliteAddr, consoleAPIAddr string) (_ *satellite.UI, err error) {
defer mon.Task()(&ctx)(&err)
prefix := "satellite-ui" + strconv.Itoa(index)
log := planet.log.Named(prefix)
return satellite.NewUI(log, identity, &config, nil, satelliteAddr, consoleAPIAddr)
}
func (planet *Planet) newAdmin(ctx context.Context, index int, identity *identity.FullIdentity, db satellite.DB, metabaseDB *metabase.DB, config satellite.Config, versionInfo version.Info) (_ *satellite.Admin, err error) { func (planet *Planet) newAdmin(ctx context.Context, index int, identity *identity.FullIdentity, db satellite.DB, metabaseDB *metabase.DB, config satellite.Config, versionInfo version.Info) (_ *satellite.Admin, err error) {
defer mon.Task()(&ctx)(&err) defer mon.Task()(&ctx)(&err)
prefix := "satellite-admin" + strconv.Itoa(index) prefix := "satellite-admin" + strconv.Itoa(index)
log := planet.log.Named(prefix) log := planet.log.Named(prefix)
return satellite.NewAdmin(log, identity, db, metabaseDB, versionInfo, &config, nil) liveAccounting, err := live.OpenCache(ctx, log.Named("live-accounting"), config.LiveAccounting)
if err != nil {
return nil, errs.Wrap(err)
}
planet.databases = append(planet.databases, liveAccounting)
return satellite.NewAdmin(log, identity, db, metabaseDB, liveAccounting, versionInfo, &config, nil)
} }
func (planet *Planet) newRepairer(ctx context.Context, index int, identity *identity.FullIdentity, db satellite.DB, metabaseDB *metabase.DB, config satellite.Config, versionInfo version.Info) (_ *satellite.Repairer, err error) { func (planet *Planet) newRepairer(ctx context.Context, index int, identity *identity.FullIdentity, db satellite.DB, metabaseDB *metabase.DB, config satellite.Config, versionInfo version.Info) (_ *satellite.Repairer, err error) {
@ -713,20 +743,6 @@ func (cache rollupsWriteCacheCloser) Close() error {
return cache.RollupsWriteCache.CloseAndFlush(context.TODO()) return cache.RollupsWriteCache.CloseAndFlush(context.TODO())
} }
func (planet *Planet) newGarbageCollection(ctx context.Context, index int, identity *identity.FullIdentity, db satellite.DB, metabaseDB *metabase.DB, config satellite.Config, versionInfo version.Info) (_ *satellite.GarbageCollection, err error) {
defer mon.Task()(&ctx)(&err)
prefix := "satellite-gc" + strconv.Itoa(index)
log := planet.log.Named(prefix)
revocationDB, err := revocation.OpenDBFromCfg(ctx, config.Server.Config)
if err != nil {
return nil, errs.Wrap(err)
}
planet.databases = append(planet.databases, revocationDB)
return satellite.NewGarbageCollection(log, identity, db, metabaseDB, revocationDB, versionInfo, &config, nil)
}
func (planet *Planet) newGarbageCollectionBF(ctx context.Context, index int, db satellite.DB, metabaseDB *metabase.DB, config satellite.Config, versionInfo version.Info) (_ *satellite.GarbageCollectionBF, err error) { func (planet *Planet) newGarbageCollectionBF(ctx context.Context, index int, db satellite.DB, metabaseDB *metabase.DB, config satellite.Config, versionInfo version.Info) (_ *satellite.GarbageCollectionBF, err error) {
defer mon.Task()(&ctx)(&err) defer mon.Task()(&ctx)(&err)
@ -746,7 +762,6 @@ func (planet *Planet) newRangedLoop(ctx context.Context, index int, db satellite
prefix := "satellite-ranged-loop" + strconv.Itoa(index) prefix := "satellite-ranged-loop" + strconv.Itoa(index)
log := planet.log.Named(prefix) log := planet.log.Named(prefix)
return satellite.NewRangedLoop(log, db, metabaseDB, &config, nil) return satellite.NewRangedLoop(log, db, metabaseDB, &config, nil)
} }

View File

@ -21,6 +21,7 @@ import (
"storj.io/common/peertls/tlsopts" "storj.io/common/peertls/tlsopts"
"storj.io/common/storj" "storj.io/common/storj"
"storj.io/private/debug" "storj.io/private/debug"
"storj.io/storj/cmd/storagenode/internalcmd"
"storj.io/storj/private/revocation" "storj.io/storj/private/revocation"
"storj.io/storj/private/server" "storj.io/storj/private/server"
"storj.io/storj/storagenode" "storj.io/storj/storagenode"
@ -133,7 +134,7 @@ func (planet *Planet) newStorageNode(ctx context.Context, prefix string, index,
}, },
}, },
Debug: debug.Config{ Debug: debug.Config{
Address: "", Addr: "",
}, },
Preflight: preflight.Config{ Preflight: preflight.Config{
LocalTimeCheck: false, LocalTimeCheck: false,
@ -215,6 +216,14 @@ func (planet *Planet) newStorageNode(ctx context.Context, prefix string, index,
MinDownloadTimeout: 2 * time.Minute, MinDownloadTimeout: 2 * time.Minute,
}, },
} }
if os.Getenv("STORJ_TEST_DISABLEQUIC") != "" {
config.Server.DisableQUIC = true
}
// enable the lazy filewalker
config.Pieces.EnableLazyFilewalker = true
if planet.config.Reconfigure.StorageNode != nil { if planet.config.Reconfigure.StorageNode != nil {
planet.config.Reconfigure.StorageNode(index, &config) planet.config.Reconfigure.StorageNode(index, &config)
} }
@ -275,6 +284,21 @@ func (planet *Planet) newStorageNode(ctx context.Context, prefix string, index,
return nil, errs.New("error while trying to issue new api key: %v", err) return nil, errs.New("error while trying to issue new api key: %v", err)
} }
{
// set up the used space lazyfilewalker filewalker
cmd := internalcmd.NewUsedSpaceFilewalkerCmd()
cmd.Logger = log.Named("used-space-filewalker")
cmd.Ctx = ctx
peer.Storage2.LazyFileWalker.TestingSetUsedSpaceCmd(cmd)
}
{
// set up the GC lazyfilewalker filewalker
cmd := internalcmd.NewGCFilewalkerCmd()
cmd.Logger = log.Named("gc-filewalker")
cmd.Ctx = ctx
peer.Storage2.LazyFileWalker.TestingSetGCCmd(cmd)
}
return &StorageNode{ return &StorageNode{
Name: prefix, Name: prefix,
Config: config, Config: config,

View File

@ -27,6 +27,7 @@ import (
"storj.io/storj/private/revocation" "storj.io/storj/private/revocation"
"storj.io/storj/private/server" "storj.io/storj/private/server"
"storj.io/storj/private/testplanet" "storj.io/storj/private/testplanet"
"storj.io/storj/satellite/nodeselection"
"storj.io/uplink" "storj.io/uplink"
"storj.io/uplink/private/metaclient" "storj.io/uplink/private/metaclient"
) )
@ -105,9 +106,15 @@ func TestDownloadWithSomeNodesOffline(t *testing.T) {
} }
// confirm that we marked the correct number of storage nodes as offline // confirm that we marked the correct number of storage nodes as offline
nodes, err := satellite.Overlay.Service.Reliable(ctx) allNodes, err := satellite.Overlay.Service.GetParticipatingNodes(ctx)
require.NoError(t, err) require.NoError(t, err)
require.Len(t, nodes, len(planet.StorageNodes)-toKill) online := make([]nodeselection.SelectedNode, 0, len(allNodes))
for _, node := range allNodes {
if node.Online {
online = append(online, node)
}
}
require.Len(t, online, len(planet.StorageNodes)-toKill)
// we should be able to download data without any of the original nodes // we should be able to download data without any of the original nodes
newData, err := ul.Download(ctx, satellite, "testbucket", "test/path") newData, err := ul.Download(ctx, satellite, "testbucket", "test/path")

View File

@ -13,7 +13,6 @@ import (
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"github.com/vivint/infectious"
"storj.io/common/memory" "storj.io/common/memory"
"storj.io/common/pb" "storj.io/common/pb"
@ -41,7 +40,7 @@ func TestECClient(t *testing.T) {
k := storageNodes / 2 k := storageNodes / 2
n := storageNodes n := storageNodes
fc, err := infectious.NewFEC(k, n) fc, err := eestream.NewFEC(k, n)
require.NoError(t, err) require.NoError(t, err)
es := eestream.NewRSScheme(fc, dataSize.Int()/n) es := eestream.NewRSScheme(fc, dataSize.Int()/n)

View File

@ -101,7 +101,14 @@ func newTestPeer(t *testing.T, ctx *testcontext.Context) *versioncontrol.Peer {
}, },
Binary: testVersions, Binary: testVersions,
} }
peer, err := versioncontrol.New(zaptest.NewLogger(t), serverConfig)
return newTestPeerWithConfig(t, ctx, serverConfig)
}
func newTestPeerWithConfig(t *testing.T, ctx *testcontext.Context, config *versioncontrol.Config) *versioncontrol.Peer {
t.Helper()
peer, err := versioncontrol.New(zaptest.NewLogger(t), config)
require.NoError(t, err) require.NoError(t, err)
ctx.Go(func() error { ctx.Go(func() error {

View File

@ -98,11 +98,13 @@ func (service *Service) checkVersion(ctx context.Context) (_ version.SemVer, all
service.checked.Release() service.checked.Release()
}() }()
allowedVersions, err := service.client.All(ctx) process, err := service.client.Process(ctx, service.service)
if err != nil { if err != nil {
service.log.Error("failed to get process version info", zap.Error(err))
return service.acceptedVersion, true return service.acceptedVersion, true
} }
suggestedVersion, err := allowedVersions.Processes.Storagenode.Suggested.SemVer()
suggestedVersion, err := process.Suggested.SemVer()
if err != nil { if err != nil {
return service.acceptedVersion, true return service.acceptedVersion, true
} }
@ -121,28 +123,40 @@ func (service *Service) checkVersion(ctx context.Context) (_ version.SemVer, all
return suggestedVersion, true return suggestedVersion, true
} }
minimumOld, err := service.client.OldMinimum(ctx, service.service) minimum, err = process.Minimum.SemVer()
if err != nil { if err != nil {
// Log about the error, but dont crash the Service and allow further operation
service.log.Error("Failed to do periodic version check.", zap.Error(err))
return suggestedVersion, true return suggestedVersion, true
} }
minimum, err = version.NewSemVer(minimumOld.String()) if minimum.IsZero() {
if err != nil { // if the minimum version is not set, we check if the old minimum version is set
service.log.Error("Failed to convert old sem version to sem version.") // TODO: I'm not sure if we should remove this check and stop supporting the old format,
return suggestedVersion, true // but it seems like it's no longer needed, assuming there are no known community
// satellites (or SNOs personally) running an old version control server, which (I think)
// is very obviously 100% true currently.
minimumOld, err := service.client.OldMinimum(ctx, service.service)
if err != nil {
return suggestedVersion, true
}
minOld, err := version.NewSemVer(minimumOld.String())
if err != nil {
service.log.Error("failed to convert old sem version to new sem version", zap.Error(err))
return suggestedVersion, true
}
minimum = minOld
} }
service.log.Debug("Allowed minimum version from control server.", zap.Stringer("Minimum Version", minimum.Version)) service.log.Debug("Allowed minimum version from control server.", zap.Stringer("Minimum Version", minimum.Version))
if isAcceptedVersion(service.Info.Version, minimumOld) { if service.Info.Version.Compare(minimum) >= 0 {
service.log.Debug("Running on allowed version.", zap.Stringer("Version", service.Info.Version.Version)) service.log.Debug("Running on allowed version.", zap.Stringer("Version", service.Info.Version.Version))
return suggestedVersion, true return suggestedVersion, true
} }
service.log.Warn("version not allowed/outdated", service.log.Warn("version not allowed/outdated",
zap.Stringer("current version", service.Info.Version.Version), zap.Stringer("current version", service.Info.Version.Version),
zap.Stringer("minimum allowed version", minimumOld), zap.String("minimum allowed version", minimum.String()),
) )
return suggestedVersion, false return suggestedVersion, false
} }
@ -168,8 +182,3 @@ func (service *Service) SetAcceptedVersion(version version.SemVer) {
func (service *Service) Checked() bool { func (service *Service) Checked() bool {
return service.checked.Released() return service.checked.Released()
} }
// isAcceptedVersion compares and checks if the passed version is greater/equal than the minimum required version.
func isAcceptedVersion(test version.SemVer, target version.OldSemVer) bool {
return test.Major > uint64(target.Major) || (test.Major == uint64(target.Major) && (test.Minor > uint64(target.Minor) || (test.Minor == uint64(target.Minor) && test.Patch >= uint64(target.Patch))))
}

View File

@ -0,0 +1,111 @@
// Copyright (C) 2023 Storj Labs, Inc.
// See LICENSE for copying information.
package checker_test
import (
"testing"
"github.com/stretchr/testify/require"
"go.uber.org/zap/zaptest"
"storj.io/common/testcontext"
"storj.io/private/version"
"storj.io/storj/private/version/checker"
"storj.io/storj/versioncontrol"
)
func TestVersion(t *testing.T) {
ctx := testcontext.New(t)
defer ctx.Cleanup()
minimum := "v1.89.5"
suggested := "v1.90.2"
testVersions := newTestVersions(t)
testVersions.Storagenode.Minimum.Version = minimum
testVersions.Storagenode.Suggested.Version = suggested
serverConfig := &versioncontrol.Config{
Address: "127.0.0.1:0",
Versions: versioncontrol.OldVersionConfig{
Satellite: "v0.0.1",
Storagenode: "v0.0.1",
Uplink: "v0.0.1",
Gateway: "v0.0.1",
Identity: "v0.0.1",
},
Binary: testVersions,
}
peer := newTestPeerWithConfig(t, ctx, serverConfig)
defer ctx.Check(peer.Close)
clientConfig := checker.ClientConfig{
ServerAddress: "http://" + peer.Addr(),
RequestTimeout: 0,
}
config := checker.Config{
ClientConfig: clientConfig,
}
t.Run("CheckVersion", func(t *testing.T) {
type args struct {
name string
version string
errorMsg string
isAcceptedVersion bool
}
tests := []args{
{
name: "runs outdated version",
version: "1.80.0",
errorMsg: "outdated software version (v1.80.0), please update",
isAcceptedVersion: false,
},
{
name: "runs minimum version",
version: minimum,
isAcceptedVersion: true,
},
{
name: "runs suggested version",
version: suggested,
isAcceptedVersion: true,
},
{
name: "runs version newer than minimum",
version: "v1.90.2",
isAcceptedVersion: true,
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
ver, err := version.NewSemVer(test.version)
require.NoError(t, err)
versionInfo := version.Info{
Version: ver,
Release: true,
}
service := checker.NewService(zaptest.NewLogger(t), config, versionInfo, "storagenode")
latest, err := service.CheckVersion(ctx)
if test.errorMsg != "" {
require.Error(t, err)
require.Contains(t, err.Error(), test.errorMsg)
} else {
require.NoError(t, err)
}
require.Equal(t, suggested, latest.String())
minVersion, isAllowed := service.IsAllowed(ctx)
require.Equal(t, isAllowed, test.isAcceptedVersion)
require.Equal(t, minimum, minVersion.String())
})
}
})
}

View File

@ -6,16 +6,16 @@ package version
import _ "unsafe" // needed for go:linkname import _ "unsafe" // needed for go:linkname
//go:linkname buildTimestamp storj.io/private/version.buildTimestamp //go:linkname buildTimestamp storj.io/private/version.buildTimestamp
var buildTimestamp string var buildTimestamp string = "1702047568"
//go:linkname buildCommitHash storj.io/private/version.buildCommitHash //go:linkname buildCommitHash storj.io/private/version.buildCommitHash
var buildCommitHash string var buildCommitHash string = "5767191bfc1a5eca25502780d90f8bbf52e7af40"
//go:linkname buildVersion storj.io/private/version.buildVersion //go:linkname buildVersion storj.io/private/version.buildVersion
var buildVersion string var buildVersion string = "v1.94.1"
//go:linkname buildRelease storj.io/private/version.buildRelease //go:linkname buildRelease storj.io/private/version.buildRelease
var buildRelease string var buildRelease string = "true"
// ensure that linter understands that the variables are being used. // ensure that linter understands that the variables are being used.
func init() { use(buildTimestamp, buildCommitHash, buildVersion, buildRelease) } func init() { use(buildTimestamp, buildCommitHash, buildVersion, buildRelease) }

View File

@ -4,24 +4,32 @@
package web package web
import ( import (
"context"
"encoding/json" "encoding/json"
"net/http" "net/http"
"go.uber.org/zap" "go.uber.org/zap"
"storj.io/common/http/requestid"
) )
// ServeJSONError writes a JSON error to the response output stream. // ServeJSONError writes a JSON error to the response output stream.
func ServeJSONError(log *zap.Logger, w http.ResponseWriter, status int, err error) { func ServeJSONError(ctx context.Context, log *zap.Logger, w http.ResponseWriter, status int, err error) {
ServeCustomJSONError(log, w, status, err, err.Error()) ServeCustomJSONError(ctx, log, w, status, err, err.Error())
} }
// ServeCustomJSONError writes a JSON error with a custom message to the response output stream. // ServeCustomJSONError writes a JSON error with a custom message to the response output stream.
func ServeCustomJSONError(log *zap.Logger, w http.ResponseWriter, status int, err error, msg string) { func ServeCustomJSONError(ctx context.Context, log *zap.Logger, w http.ResponseWriter, status int, err error, msg string) {
fields := []zap.Field{ fields := []zap.Field{
zap.Int("code", status), zap.Int("code", status),
zap.String("message", msg), zap.String("message", msg),
zap.Error(err), zap.Error(err),
} }
if requestID := requestid.FromContext(ctx); requestID != "" {
fields = append(fields, zap.String("requestID", requestID))
}
switch status { switch status {
case http.StatusNoContent: case http.StatusNoContent:
return return

View File

@ -87,12 +87,12 @@ func (rl *RateLimiter) Limit(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
key, err := rl.keyFunc(r) key, err := rl.keyFunc(r)
if err != nil { if err != nil {
ServeCustomJSONError(rl.log, w, http.StatusInternalServerError, err, internalServerErrMsg) ServeCustomJSONError(r.Context(), rl.log, w, http.StatusInternalServerError, err, internalServerErrMsg)
return return
} }
limit := rl.getUserLimit(key) limit := rl.getUserLimit(key)
if !limit.Allow() { if !limit.Allow() {
ServeJSONError(rl.log, w, http.StatusTooManyRequests, errs.New(rateLimitErrMsg)) ServeJSONError(r.Context(), rl.log, w, http.StatusTooManyRequests, errs.New(rateLimitErrMsg))
return return
} }
next.ServeHTTP(w, r) next.ServeHTTP(w, r)

View File

@ -5,6 +5,7 @@ package accounting
import ( import (
"context" "context"
"fmt"
"time" "time"
"storj.io/common/memory" "storj.io/common/memory"
@ -111,16 +112,16 @@ type ProjectUsageByDay struct {
// BucketUsage consist of total bucket usage for period. // BucketUsage consist of total bucket usage for period.
type BucketUsage struct { type BucketUsage struct {
ProjectID uuid.UUID ProjectID uuid.UUID `json:"projectID"`
BucketName string BucketName string `json:"bucketName"`
Storage float64 Storage float64 `json:"storage"`
Egress float64 Egress float64 `json:"egress"`
ObjectCount int64 ObjectCount int64 `json:"objectCount"`
SegmentCount int64 SegmentCount int64 `json:"segmentCount"`
Since time.Time Since time.Time `json:"since"`
Before time.Time Before time.Time `json:"before"`
} }
// BucketUsageCursor holds info for bucket usage // BucketUsageCursor holds info for bucket usage
@ -133,15 +134,15 @@ type BucketUsageCursor struct {
// BucketUsagePage represents bucket usage page result. // BucketUsagePage represents bucket usage page result.
type BucketUsagePage struct { type BucketUsagePage struct {
BucketUsages []BucketUsage BucketUsages []BucketUsage `json:"bucketUsages"`
Search string Search string `json:"search"`
Limit uint Limit uint `json:"limit"`
Offset uint64 Offset uint64 `json:"offset"`
PageCount uint PageCount uint `json:"pageCount"`
CurrentPage uint CurrentPage uint `json:"currentPage"`
TotalCount uint64 TotalCount uint64 `json:"totalCount"`
} }
// BucketUsageRollup is total bucket usage info // BucketUsageRollup is total bucket usage info
@ -164,6 +165,36 @@ type BucketUsageRollup struct {
Before time.Time `json:"before"` Before time.Time `json:"before"`
} }
// ProjectReportItem is total bucket usage info with project details for certain period.
type ProjectReportItem struct {
ProjectID uuid.UUID
ProjectName string
BucketName string
Storage float64
Egress float64
SegmentCount float64
ObjectCount float64
Since time.Time `json:"since"`
Before time.Time `json:"before"`
}
// ToStringSlice converts report item values to a slice of strings.
func (b *ProjectReportItem) ToStringSlice() []string {
return []string{
b.ProjectName,
b.ProjectID.String(),
b.BucketName,
fmt.Sprintf("%f", b.Storage),
fmt.Sprintf("%f", b.Egress),
fmt.Sprintf("%f", b.ObjectCount),
fmt.Sprintf("%f", b.SegmentCount),
b.Since.String(),
b.Before.String(),
}
}
// Usage contains project's usage split on segments and storage. // Usage contains project's usage split on segments and storage.
type Usage struct { type Usage struct {
Storage int64 Storage int64
@ -219,6 +250,8 @@ type ProjectAccounting interface {
GetProjectSettledBandwidthTotal(ctx context.Context, projectID uuid.UUID, from time.Time) (_ int64, err error) GetProjectSettledBandwidthTotal(ctx context.Context, projectID uuid.UUID, from time.Time) (_ int64, err error)
// GetProjectBandwidth returns project allocated bandwidth for the specified year, month and day. // GetProjectBandwidth returns project allocated bandwidth for the specified year, month and day.
GetProjectBandwidth(ctx context.Context, projectID uuid.UUID, year int, month time.Month, day int, asOfSystemInterval time.Duration) (int64, error) GetProjectBandwidth(ctx context.Context, projectID uuid.UUID, year int, month time.Month, day int, asOfSystemInterval time.Duration) (int64, error)
// GetProjectSettledBandwidth returns the used settled bandwidth for the specified year and month.
GetProjectSettledBandwidth(ctx context.Context, projectID uuid.UUID, year int, month time.Month, asOfSystemInterval time.Duration) (int64, error)
// GetProjectDailyBandwidth returns bandwidth (allocated and settled) for the specified day. // GetProjectDailyBandwidth returns bandwidth (allocated and settled) for the specified day.
GetProjectDailyBandwidth(ctx context.Context, projectID uuid.UUID, year int, month time.Month, day int) (int64, int64, int64, error) GetProjectDailyBandwidth(ctx context.Context, projectID uuid.UUID, year int, month time.Month, day int) (int64, int64, int64, error)
// DeleteProjectBandwidthBefore deletes project bandwidth rollups before the given time // DeleteProjectBandwidthBefore deletes project bandwidth rollups before the given time

View File

@ -26,6 +26,7 @@ type Config struct {
StorageBackend string `help:"what to use for storing real-time accounting data"` StorageBackend string `help:"what to use for storing real-time accounting data"`
BandwidthCacheTTL time.Duration `default:"5m" help:"bandwidth cache key time to live"` BandwidthCacheTTL time.Duration `default:"5m" help:"bandwidth cache key time to live"`
AsOfSystemInterval time.Duration `default:"-10s" help:"as of system interval"` AsOfSystemInterval time.Duration `default:"-10s" help:"as of system interval"`
BatchSize int `default:"5000" help:"how much projects usage should be requested from redis cache at once"`
} }
// OpenCache creates a new accounting.Cache instance using the type specified backend in // OpenCache creates a new accounting.Cache instance using the type specified backend in
@ -49,7 +50,7 @@ func OpenCache(ctx context.Context, log *zap.Logger, config Config) (accounting.
backendType = parts[0] backendType = parts[0]
switch backendType { switch backendType {
case "redis": case "redis":
return openRedisLiveAccounting(ctx, config.StorageBackend) return openRedisLiveAccounting(ctx, config.StorageBackend, config.BatchSize)
default: default:
return nil, Error.New("unrecognized live accounting backend specifier %q. Currently only redis is supported", backendType) return nil, Error.New("unrecognized live accounting backend specifier %q. Currently only redis is supported", backendType)
} }

View File

@ -6,6 +6,7 @@ package live_test
import ( import (
"context" "context"
"math/rand" "math/rand"
"strconv"
"testing" "testing"
"time" "time"
@ -136,19 +137,28 @@ func TestGetAllProjectTotals(t *testing.T) {
require.NoError(t, err) require.NoError(t, err)
} }
usage, err := cache.GetAllProjectTotals(ctx) for _, batchSize := range []int{1, 2, 3, 10, 13, 10000} {
require.NoError(t, err) t.Run("batch-size-"+strconv.Itoa(batchSize), func(t *testing.T) {
require.Len(t, usage, len(projectIDs)) config.BatchSize = batchSize
testCache, err := live.OpenCache(ctx, zaptest.NewLogger(t).Named("live-accounting"), config)
require.NoError(t, err)
defer ctx.Check(testCache.Close)
// make sure each project ID and total was received usage, err := testCache.GetAllProjectTotals(ctx)
for _, projID := range projectIDs { require.NoError(t, err)
totalStorage, err := cache.GetProjectStorageUsage(ctx, projID) require.Len(t, usage, len(projectIDs))
require.NoError(t, err)
assert.Equal(t, totalStorage, usage[projID].Storage)
totalSegments, err := cache.GetProjectSegmentUsage(ctx, projID) // make sure each project ID and total was received
require.NoError(t, err) for _, projID := range projectIDs {
assert.Equal(t, totalSegments, usage[projID].Segments) totalStorage, err := testCache.GetProjectStorageUsage(ctx, projID)
require.NoError(t, err)
assert.Equal(t, totalStorage, usage[projID].Storage)
totalSegments, err := testCache.GetProjectSegmentUsage(ctx, projID)
require.NoError(t, err)
assert.Equal(t, totalSegments, usage[projID].Segments)
}
})
} }
}) })
} }

View File

@ -11,6 +11,7 @@ import (
"time" "time"
"github.com/redis/go-redis/v9" "github.com/redis/go-redis/v9"
"github.com/zeebo/errs/v2"
"storj.io/common/uuid" "storj.io/common/uuid"
"storj.io/storj/satellite/accounting" "storj.io/storj/satellite/accounting"
@ -18,6 +19,8 @@ import (
type redisLiveAccounting struct { type redisLiveAccounting struct {
client *redis.Client client *redis.Client
batchSize int
} }
// openRedisLiveAccounting returns a redisLiveAccounting cache instance. // openRedisLiveAccounting returns a redisLiveAccounting cache instance.
@ -29,14 +32,15 @@ type redisLiveAccounting struct {
// it fails then it returns an instance and accounting.ErrSystemOrNetError // it fails then it returns an instance and accounting.ErrSystemOrNetError
// because it means that Redis may not be operative at this precise moment but // because it means that Redis may not be operative at this precise moment but
// it may be in future method calls as it handles automatically reconnects. // it may be in future method calls as it handles automatically reconnects.
func openRedisLiveAccounting(ctx context.Context, address string) (*redisLiveAccounting, error) { func openRedisLiveAccounting(ctx context.Context, address string, batchSize int) (*redisLiveAccounting, error) {
opts, err := redis.ParseURL(address) opts, err := redis.ParseURL(address)
if err != nil { if err != nil {
return nil, accounting.ErrInvalidArgument.Wrap(err) return nil, accounting.ErrInvalidArgument.Wrap(err)
} }
cache := &redisLiveAccounting{ cache := &redisLiveAccounting{
client: redis.NewClient(opts), client: redis.NewClient(opts),
batchSize: batchSize,
} }
// ping here to verify we are able to connect to Redis with the initialized client. // ping here to verify we are able to connect to Redis with the initialized client.
@ -52,7 +56,7 @@ func openRedisLiveAccounting(ctx context.Context, address string) (*redisLiveAcc
func (cache *redisLiveAccounting) GetProjectStorageUsage(ctx context.Context, projectID uuid.UUID) (totalUsed int64, err error) { func (cache *redisLiveAccounting) GetProjectStorageUsage(ctx context.Context, projectID uuid.UUID) (totalUsed int64, err error) {
defer mon.Task()(&ctx, projectID)(&err) defer mon.Task()(&ctx, projectID)(&err)
return cache.getInt64(ctx, string(projectID[:])) return cache.getInt64(ctx, createStorageProjectIDKey(projectID))
} }
// GetProjectBandwidthUsage returns the current bandwidth usage // GetProjectBandwidthUsage returns the current bandwidth usage
@ -175,7 +179,7 @@ func (cache *redisLiveAccounting) AddProjectSegmentUsageUpToLimit(ctx context.Co
func (cache *redisLiveAccounting) AddProjectStorageUsage(ctx context.Context, projectID uuid.UUID, spaceUsed int64) (err error) { func (cache *redisLiveAccounting) AddProjectStorageUsage(ctx context.Context, projectID uuid.UUID, spaceUsed int64) (err error) {
defer mon.Task()(&ctx, projectID, spaceUsed)(&err) defer mon.Task()(&ctx, projectID, spaceUsed)(&err)
_, err = cache.client.IncrBy(ctx, string(projectID[:]), spaceUsed).Result() _, err = cache.client.IncrBy(ctx, createStorageProjectIDKey(projectID), spaceUsed).Result()
if err != nil { if err != nil {
return accounting.ErrSystemOrNetError.New("Redis incrby failed: %w", err) return accounting.ErrSystemOrNetError.New("Redis incrby failed: %w", err)
} }
@ -216,6 +220,7 @@ func (cache *redisLiveAccounting) GetAllProjectTotals(ctx context.Context) (_ ma
defer mon.Task()(&ctx)(&err) defer mon.Task()(&ctx)(&err)
projects := make(map[uuid.UUID]accounting.Usage) projects := make(map[uuid.UUID]accounting.Usage)
it := cache.client.Scan(ctx, 0, "*", 0).Iterator() it := cache.client.Scan(ctx, 0, "*", 0).Iterator()
for it.Next(ctx) { for it.Next(ctx) {
key := it.Val() key := it.Val()
@ -231,58 +236,112 @@ func (cache *redisLiveAccounting) GetAllProjectTotals(ctx context.Context) (_ ma
return nil, accounting.ErrUnexpectedValue.New("cannot parse the key as UUID; key=%q", key) return nil, accounting.ErrUnexpectedValue.New("cannot parse the key as UUID; key=%q", key)
} }
usage := accounting.Usage{} projects[projectID] = accounting.Usage{}
if seenUsage, seen := projects[projectID]; seen {
if seenUsage.Segments != 0 {
continue
}
usage = seenUsage
}
segmentUsage, err := cache.GetProjectSegmentUsage(ctx, projectID)
if err != nil {
if accounting.ErrKeyNotFound.Has(err) {
continue
}
return nil, err
}
usage.Segments = segmentUsage
projects[projectID] = usage
} else { } else {
projectID, err := uuid.FromBytes([]byte(key)) projectID, err := uuid.FromBytes([]byte(key))
if err != nil { if err != nil {
return nil, accounting.ErrUnexpectedValue.New("cannot parse the key as UUID; key=%q", key) return nil, accounting.ErrUnexpectedValue.New("cannot parse the key as UUID; key=%q", key)
} }
usage := accounting.Usage{} projects[projectID] = accounting.Usage{}
if seenUsage, seen := projects[projectID]; seen {
if seenUsage.Storage != 0 {
continue
}
usage = seenUsage
}
storageUsage, err := cache.getInt64(ctx, key)
if err != nil {
if accounting.ErrKeyNotFound.Has(err) {
continue
}
return nil, err
}
usage.Storage = storageUsage
projects[projectID] = usage
} }
} }
return cache.fillUsage(ctx, projects)
}
func (cache *redisLiveAccounting) fillUsage(ctx context.Context, projects map[uuid.UUID]accounting.Usage) (_ map[uuid.UUID]accounting.Usage, err error) {
defer mon.Task()(&ctx)(&err)
if len(projects) == 0 {
return nil, nil
}
projectIDs := make([]uuid.UUID, 0, cache.batchSize)
segmentKeys := make([]string, 0, cache.batchSize)
storageKeys := make([]string, 0, cache.batchSize)
fetchProjectsUsage := func() error {
if len(projectIDs) == 0 {
return nil
}
segmentResult, err := cache.client.MGet(ctx, segmentKeys...).Result()
if err != nil {
return accounting.ErrGetProjectLimitCache.Wrap(err)
}
storageResult, err := cache.client.MGet(ctx, storageKeys...).Result()
if err != nil {
return accounting.ErrGetProjectLimitCache.Wrap(err)
}
// Note, because we are using a cache, it might be empty and not contain the
// information we are looking for -- or they might be still empty for some reason.
for i, projectID := range projectIDs {
segmentsUsage, err := parseAnyAsInt64(segmentResult[i])
if err != nil {
return errs.Wrap(err)
}
storageUsage, err := parseAnyAsInt64(storageResult[i])
if err != nil {
return errs.Wrap(err)
}
projects[projectID] = accounting.Usage{
Segments: segmentsUsage,
Storage: storageUsage,
}
}
return nil
}
for projectID := range projects {
projectIDs = append(projectIDs, projectID)
segmentKeys = append(segmentKeys, createSegmentProjectIDKey(projectID))
storageKeys = append(storageKeys, createStorageProjectIDKey(projectID))
if len(projectIDs) >= cache.batchSize {
err := fetchProjectsUsage()
if err != nil {
return nil, err
}
projectIDs = projectIDs[:0]
segmentKeys = segmentKeys[:0]
storageKeys = storageKeys[:0]
}
}
err = fetchProjectsUsage()
if err != nil {
return nil, err
}
return projects, nil return projects, nil
} }
func parseAnyAsInt64(v any) (int64, error) {
if v == nil {
return 0, nil
}
s, ok := v.(string)
if !ok {
return 0, accounting.ErrUnexpectedValue.New("cannot parse the value as int64; val=%q", v)
}
i, err := strconv.ParseInt(s, 10, 64)
if err != nil {
return 0, accounting.ErrUnexpectedValue.New("cannot parse the value as int64; val=%q", v)
}
return i, nil
}
// Close the DB connection. // Close the DB connection.
func (cache *redisLiveAccounting) Close() error { func (cache *redisLiveAccounting) Close() error {
err := cache.client.Close() err := cache.client.Close()
@ -325,3 +384,8 @@ func createBandwidthProjectIDKey(projectID uuid.UUID, now time.Time) string {
func createSegmentProjectIDKey(projectID uuid.UUID) string { func createSegmentProjectIDKey(projectID uuid.UUID) string {
return string(projectID[:]) + ":segment" return string(projectID[:]) + ":segment"
} }
// createStorageProjectIDKey creates the storage project key.
func createStorageProjectIDKey(projectID uuid.UUID) string {
return string(projectID[:])
}

View File

@ -40,7 +40,8 @@ type ProjectLimitConfig struct {
// ProjectLimitCache stores the values for both storage usage limit and bandwidth limit for // ProjectLimitCache stores the values for both storage usage limit and bandwidth limit for
// each project ID if they differ from the default limits. // each project ID if they differ from the default limits.
type ProjectLimitCache struct { type ProjectLimitCache struct {
projectLimitDB ProjectLimitDB projectLimitDB ProjectLimitDB
defaultMaxUsage memory.Size defaultMaxUsage memory.Size
defaultMaxBandwidth memory.Size defaultMaxBandwidth memory.Size
defaultMaxSegments int64 defaultMaxSegments int64
@ -121,10 +122,6 @@ func (c *ProjectLimitCache) getProjectLimits(ctx context.Context, projectID uuid
defaultSegments := c.defaultMaxSegments defaultSegments := c.defaultMaxSegments
projectLimits.Segments = &defaultSegments projectLimits.Segments = &defaultSegments
} }
if projectLimits.Segments == nil {
defaultSegments := c.defaultMaxSegments
projectLimits.Segments = &defaultSegments
}
return projectLimits, nil return projectLimits, nil
} }

Some files were not shown because too many files have changed in this diff Show More