Commit Graph

8861 Commits

Author SHA1 Message Date
Moby von Briesen
03a8e7c81a satellite/analytics: improve error logging
This inserts additional information provided in error messages from
Hubspot into the Go error that is eventually logged out.
Before, we would see the a generic "sending event failed" log.
With this change, we will see more detailed information in the log, such
as a list of required fields that were not submitted.

Change-Id: I24da0646bca62f459377abe6281403020fb54c49
2023-11-29 09:49:25 +00:00
Jeremy Wharton
4822b18472 satellite/admin/back-office/ui: implement view user functionality
This change adds a search field to the new admin UI through which user
email addresses may be submitted. If the email belongs to a verified
user, the client will be redirected to the Account Details page which
is populated with the user's information.

Resolves #6469
Resolves #6475

Change-Id: Icbf3cb3f8374f2764e73a523f111c5ecf3d06569
2023-11-28 16:31:37 +00:00
Egon Elbre
7186525d5c satellite/metabase: test zombie deletion with versioning
Change-Id: I495745f84ea0a7c36763d2889170c4068347dcb7
2023-11-28 12:12:26 +00:00
Ivan Fraixedes
6885ca27fb private/apigen: Inject middleware code after param parsing
Inject the middleware code after the query and path parameters parsing
because in the case that a middleware needs to execute a more expensive
logic, it isn't executed if there are malformed or missing parameters.

Change-Id: I883ade8ee42d25accb153c78d9f583d883730d6a
2023-11-27 23:02:44 +00:00
Vitalii
5e3cab29a2 satellite/{cmd, payments}: add optional command flag to toggle invoice items aggregation
Added new optional '--aggregate' flag for billing.generate-invoices and billing.prepare-invoice-records commands to toggle invoice items aggregation.
Added new explicit command billing.create-aggregated-project-invoice-items which should be used after preparing aggregated invoice records (in cases when invoice generation happens step-by-step).

Change-Id: I04fc0110be5263edb959306d5314a4a1a8eec3ba
2023-11-27 22:11:15 +00:00
Jeremy Wharton
d2819522c6 satellite/admin/back-office: add endpoint to get users by email
This change adds an endpoint to the back office API that returns user
info based on email address.

References #6503

Change-Id: Ib48d30b0b6c6862887b3f8114f50538b3deca57b
2023-11-27 21:05:12 +00:00
Michal Niewrzal
220920edb9 satellite/metainfo: return NotFound status for (Get|Set)BucketVersioning
We need to return NotFound rpc status when bucket doesn't exist.

Change-Id: I3abdd588e2af288618b05513884bc0c545bcbacf
2023-11-27 19:45:45 +00:00
Ivan Fraixedes
c2788ab6ae satellite/admin/back-office: Add README with API guidelines
Add a README document with some general API guidelines that we wrote in
a Slack canvas to discuss them.

Change-Id: Iec933edeb7622b78a98155512b25267d12879837
2023-11-27 18:42:14 +00:00
Ivan Fraixedes
269dd5602e private/apigen: Don't print empty description & version in docs
If the API description or version is empty, don't print out in the docs
a placeholder for them.

In commit bcde51138d a space between the
placeholder and description was removed because it was printing a
trailing space if the description was empty, but that introduced another
bug, which is not adding the space when the description is empty.

This commit fixes the bug, no printing the placeholder if the
description is empty, otherwise print it with the corresponding blank
space in between the placeholder and the description.

Change-Id: I19c92c3dd7dfb38b8b2a43821bafc726ffad59d7
2023-11-27 16:53:33 +00:00
Egon Elbre
4620d6d4cc satellite/accounting/tally: simplify code
Change-Id: I50bdecad5d85e07fdc0dfbd11bb03f8f6bbba56e
2023-11-27 17:01:47 +02:00
Egon Elbre
89dad05c65 satellite/metabase: don't create delete markers for pending
Fix a case where it was possible to create a delete marker when only
pending object was present.

The solution is not pretty, but we have a TODO note to cleanup the
precommit code, so let's fix the bug first.

Change-Id: I0ab66d745443c9dccbf29ef32389dd912b2d9caf
2023-11-23 11:13:21 +00:00
Ivan Fraixedes
adcd810e37 private/apigen: Allow to customize handlers logic
The API generator doesn't have a way to customize each Go handler
endpoint unless that the Go generator is modified.

This commit adds a way to customize each endpoint injecting instances of
types that implement an interface (Middleware) that return the code to inject.

To show how it works, the commit get rid of the 2 fields that we used to
customize the authentication request with the logic that the
satellite/console/consoleweb/consoleapi needs and replace the hardcoded
customization using this new way to customize handlers.

This new way should allow to hook the satellite/admin/back-office
authorization into the handlers using a Middleware implementation.

Change-Id: I894aa0026b30fa2f4a5604a6c34c22e0ed582e2b
2023-11-23 06:57:40 +00:00
Cameron
479fbb628c satellite/satellitedb: fix bucket sum for daily usage
The code to get daily storage usage is a little confusing. There is a
portion of the query which looks like it is trying to sum the storage
for buckets by each day, but it doesn't work. There is also a section
which sums the buckets by reading the returned rows, but it assumes that
the rows are ordered by the date, which they are not. This results in
the potential for multiple entries for the same day being created: one
for each bucket. The UI doesn't know about this, so the entries for the
same day overwrite each other and result in the appearance of less
stored data on the graph.

To fix it, don't group by bucket and allow the query to sum the buckets
together.

Change-Id: Id3ec25238e6e6346b93b10f834bdc9c70023f1e1
2023-11-22 16:25:46 -05:00
Igor
fb894b3720
scripts/draft-release.sh: improve changelog script (#6508)
- Improved commit message parsing for better format handling
- Added argument validation and user prompts for interactive mode
- Implemented enhanced error handling and logging
2023-11-22 19:26:03 +01:00
andriikotko
3c2596e3e2
mainteiners.md: update release process description (#6430)
Co-authored-by: Antonio Franco <antonio@storj.io>
2023-11-22 17:01:04 +01:00
Vitalii
6b1c62d7b2 satellite/{payments, db}: aggregate invoice items if many projects
Implemented invoice items aggregation if projects count is more than 83 for a single invoice.

Change-Id: I6bce81e537eaaddd9297a85718b594047436964a
2023-11-22 11:47:48 +00:00
Wilfred Asomani
f09f352628 satellite/{console,emails}: add activation code email
This change adds an email template for sending activation codes.

Issue: #6428

Change-Id: Ief56a2e32ac0a335561572b9573215da24d9146b
2023-11-22 10:01:27 +00:00
Wilfred Asomani
15b90661f4 satellite/{console,web}: add activation code config flag
This change adds a config flag for whether signup activation code
should be used.

Issue: #6428

Change-Id: I2b2b1e76f6013eab6438aefba848af0a4bd62544
2023-11-22 09:58:35 +00:00
Wilfred Asomani
26574fb2bd satellite/db: add account activation columns
This change adds a new column to the user table, which will hold a
random code for account activation. And another to hold the signup
request ID as another layer of verification.

Issue: #6428

Change-Id: Icd46cb5d8fc76102264d599aca27686cd8b2e84e
2023-11-22 09:57:52 +00:00
Michal Niewrzal
573ce712f2 satellite/bloomfilter: don't create BF for disqualified nodes
Currently we have large set of nodes that are already disqualified and
we are not sending bloom filters to them. The issue is that we are still
generating filters for them while garbage collection process. Even if
we have only segment with one piece which was stored on this node. This
consumes additional memory and processing powers.

This change is changing logic behind `AllPieceCounts` (renamed to
ActiveNodesPieceCounts) to return piece count for all nodes except disqualified one (even with piece count = 0). With this change we can
modify GC observer to skip nodes that where not returned by
ActiveNodesPieceCounts.

Change-Id: Ic75159135abe535084d8aeee560bb801a4a03e17
2023-11-22 07:07:51 +00:00
Egon Elbre
07cb8dc677 satelite/metabase: tiny todo cleanups
Change-Id: I655176e50202b0a363e4dce6299c02545b491244
2023-11-21 19:38:54 +00:00
Egon Elbre
11b083c611 satellite/metabase: test CommitObject.DisallowDelete
Change-Id: I4f408dd23ba2875a0aa46fb4dbd494c3c7dbc0af
2023-11-21 18:47:05 +00:00
Jeremy Wharton
587fa8fdff private/apigen: use correct TS type for nillable fields
This change gives the proper type to TS class fields generated from
nillable Go struct fields. Previously, Go struct fields having a nil
representation ([]Type, *Type, etc.) were translated into TypeScript as
"Type | undefined". This isn't correct because these fields, when nil,
are given the value "null" when marshalled into JSON. This change fixes
this issue by giving these fields the type "Type | null".

Change-Id: I5a1a83eb3810a3cba10895bb2f0f75ca5fd7d1b5
2023-11-21 17:58:07 +00:00
Márton Elek
0f4f1ddde8 satellite/durability: use single classifier per observer instance
the new bus_factor calculation doesn't make sense with different classes, as we have overlaps.

For example: it can detect a risk if we loose one country and one different subnet (with possible overlap).

It's better to calculate the stat and bus_factor per class (net, country, ...).

It also makes it easier to measure execution time per class.

Change-Id: I7d4d5f7cb811cd50c5831077b43e001908aab96b
2023-11-21 17:08:34 +00:00
Márton Elek
7c25e3733a build: move Run Versions test to a separated job
Run Versions test is intermittent and slow.

We need to rewrite it with storj-up, preferable with a container where
 old versions are cached, instead of rebuilding them everytime
 (which is -- IMHO -- the root cause of the intermittency).

As we discussed on the chat, we can separate the runs, until we prioritize this work.

Change-Id: I7fdf6fdb20625fd76d6334be0d0afe72af1b734a
2023-11-21 15:21:19 +00:00
Michal Niewrzal
51ba901737 satellite/metainfo: (Get|Download)Object returns MethodNotAllowed on delete marker
For S3 compatibility we should not allow returning pure delete
marker object. If metabase returns delete marker metainfo will
return MethodNotAllowed rpc status.

https://github.com/storj/storj/issues/6522

Change-Id: I89804b2bd22da0e5beec8f106e74b74733e19a52
2023-11-21 13:27:54 +00:00
Márton Elek
f2eca99bde satellite/repair: stat method to the repair queue to return with statistics
Change-Id: I2e07b116df9b282978a794423bd38803e2778755
2023-11-21 11:37:49 +00:00
JT Olio
2d8cca49b2 cmd/uplink: support writing monkit traces to file
Change-Id: I759d1687cf890e21cb260a5377c7153bfd0eb486
2023-11-20 22:07:00 +00:00
Clement Sam
b8b5afba73 satellite/satellitedb: remove dbx references to nodes.type column
Updates https://github.com/storj/storj/issues/5426

Change-Id: I36f12f354d9f2d575e4bd99b0520892f846bdc19
2023-11-20 18:08:37 +00:00
Márton Elek
0fdacfed8f satellite/durability: observer must reset between executions
Change-Id: I8f5b951beba513b219c4bb5680658f5e8b54538d
2023-11-20 16:56:43 +00:00
Jeremy Wharton
866780bcde web/satellite: fix papa parse worker plugin issue in watch mode
This change fixes an issue that caused the Papa Parse worker plugin to
break during Vite's watch mode.

Change-Id: I84d03d38e2d0d6fe4b7536706638640d3ed259a6
2023-11-20 16:07:51 +00:00
Vitalii
6d22ec21d0 web/satellite/vuetify-poc: buckets limit card for project dashboard
Show buckets limit card on project dashboard instead of coupon card if user is in paid tier or is not the owner of currently selected project.

Issue:
https://github.com/storj/storj/issues/6492

Change-Id: I033bbcced1b0fb77436607847cd9f10830330009
2023-11-20 14:26:51 +00:00
Clement Sam
f5164c78cd Revert "satellite/satellitedb: drop type column on nodes table"
This reverts commit 2b4f347c33.

Reason for revert: not a zero-downtime migration. We need to deploy the dbx changes before completely dropping the column.

Change-Id: Ic9215650242b7848d54f8a5f863a13bc18b60149
2023-11-20 13:36:44 +00:00
Wilfred Asomani
33fb21c8e0 satellite/admin: add endpoints to legal freeze/unfreeze users
This change adds two new admin endpoints to freeze users for legal
review and to remove them from that state

Issue: storj/storj-private#492

Change-Id: I6c8e3ffcb80375e81e78bc6ecc785c1047328cf7
2023-11-20 11:23:23 +00:00
Clement Sam
2b4f347c33 satellite/satellitedb: drop type column on nodes table
Updates https://github.com/storj/storj/issues/5426

Change-Id: If554f30124f234d1c50b2e8059e3d1dfec389ebb
2023-11-20 10:09:17 +00:00
Jeremy Wharton
0c591fa25a private/apigen: handle omitempty JSON option in structs
This change makes the TypeScript API code generator properly handle
struct fields with the "omitempty" option in the JSON struct tag.

Change-Id: I9b22ce33a8b8c39c115ec827a8e5b7e85d856f83
2023-11-18 07:09:42 -06:00
dlamarmorgan
cbc82690d7 satellite: implement metainfo get/set versioning endpoint
Change-Id: Ic43ea8419e7e7092dd191c8ed9f6e4eb4bdff20b
2023-11-18 01:26:34 +00:00
Egon Elbre
13d02d9d11 satellite/metabase: disallow moving delete markers
Change-Id: I5427ce2cf6783f83e77fcd8ca2fe7b98f78a945b
2023-11-17 21:56:33 +00:00
Michal Niewrzal
26a04a5929 satellite/metainfo: use bucket versioning state for listing
Currently we need to use different methods for listing objects
depends on versioning state.

Change-Id: I5747a699ba2dafcfc384216e4be0b662c8ae95fa
2023-11-17 21:07:58 +00:00
nerdatwork
bae3f7f73d
web/multinode: Show Storj tokens on Etherscan (#6483)
Currently the "View on Etherscan" link doesn't show Storj tokens on Etherscan like it should on MND (Multi Node Dashboard).
2023-11-17 17:57:53 +01:00
Ivan Fraixedes
bcde51138d private/apigen: Remove trailing space in doc generator
Remove a trailing space that the doc generator was adding.

Change-Id: I5c24478f79726afbaa5c6772d569b35cafe7c288
2023-11-17 11:44:50 +00:00
Ivan Fraixedes
ed67ce33bb
satellite/console: Clarify return params of interface method
Name the return parameters in a User interface method to clarify what
they are.

Change-Id: I6a5c97659c8167df89b38016fcccbc8eb70a09c5
2023-11-17 11:45:52 +01:00
Márton Elek
257bdbac32 satellitedb/repair: update placement when InsertBatch updates records
Change-Id: If974ff5d57abbe5bd16ce4cb6643d8a12314fe12
2023-11-16 18:55:18 +00:00
Vitalii
e474b9728c web/satellite: fix redundant redirect on first login
Not partnered users were redirected to onboarding pricing plan step and back to all projects dashboard in case when they have not finished onboarding. That caused tons of redundant API requests to be done.

Change-Id: I454a8cc29730943b84a351416b78c4281c47f6ca
2023-11-16 18:05:46 +00:00
Jeremy Wharton
1ea81c8887 satellite/admin/back-office: add endpoint to get placement info
This change adds an endpoint to the back office API that returns
placement IDs and their locations.

References #6503

Change-Id: I20ee1c82dcb647d6d264317beceeb5e70f7a8e87
2023-11-16 13:15:32 +00:00
Ivan Fraixedes
e67691d51c private/apigen: Remove support for anonymous types
I made several commits to add support to the API generator for
anonymous types because the example was using them.

Later, I was told that we don't need to support them, so the example
should have never used anonymous types.

Supporting anonymous types has added complexity to the API generator and
we are finding bugs in corner cases whose fixes lead to adding more
complexity and some fixes doesn't seem to have a neat solution without
doing a big refactoring.

We decided to reduce the side effects that supporting anonymous type has
brought just removing it.

This commit remove the support for anonymous types and reflect that in
the example.

Change-Id: I2f8c87a0db0e229971ab1bef46cca16fee924191
2023-11-16 12:29:21 +00:00
Ivan Fraixedes
418673f7a2 satellite/admin/back-office: Implement authorization
Implement the authorization that will hook into each endpoint handler
through a wrapping handler for defining the permissions that each
endpoint requires.

Change-Id: I9c8f12b58f48e849e7ea35f372dddce5c9cfc5b5
2023-11-16 11:37:55 +00:00
Igor
720b75ad73
docs/testplan: add object versioning testplan (#6444)
* docs/testplan: add object versioning testplan
2023-11-16 11:46:25 +01:00
Ivan Fraixedes
359c09b57f satellite/admin/back-office: Specify router path prefix
For convenience of not having to modify the API generator to contemplate
the path prefix that we are adding to the back office server, we define
the path prefix in a constant than the admin server and the definition
of the API uses to adapt the router and the generated code.

Change-Id: Ic557b0e6e88e930e03647835759bb34e06e8bb48
2023-11-15 20:28:52 -06:00
Jeremy Wharton
032faefa4b private/apigen: fix URL construction in generated TypeScript code
This change fixes an incorrect invocation of the URL object constructor
in generated TypeScript HTTP clients.

Change-Id: I9011bc535f2096374d20b74b401d4cc38a0451fb
2023-11-16 01:12:58 +00:00