Commit Graph

636 Commits

Author SHA1 Message Date
Ivan Fraixedes
fb31761bad satellite/admin/back-office: Add auth middleware
Create an API generator middleware for being able to hook the new
satellite admin authorization in the endpoints.

The commit fixes a bug found in the API generator that caused that
fields of types of the same package of the generated code where wrongly
added. Concretely:

- The package matching was missing in the function middlewareFields,
  hence it was generating code that referenced types with the package
  name.
- middlewareFields function was not adding the pointer symbol (*) when
  the type was from the same package where the generated code is
  written.

There is also an accidental enhancement in the API generator because I
thought that the bug commented above corresponded to it, rather than
removing it, I though that was worthwhile to keep it because it was
already implemented. This enhancement allows to use fields in the
middleware with packages whose last path part contains `-` or `.`, using
a package rename in the import statement.

Change-Id: Ie98b303226a8e8845e494f25054867f95a283aa0
2023-12-01 00:29:49 +00:00
Márton Elek
1ba314428f go.mod: make bump-dependencies (uplink, common, ...)
It requires more work at this time, as https://review.dev.storj.io/c/storj/private/+/11066 modified the way how we configure the debug port.

Change-Id: I4a7cc999e13fd3514064a515b21885bb4d39ff16
2023-11-29 16:55:41 +00:00
Ivan Fraixedes
6885ca27fb private/apigen: Inject middleware code after param parsing
Inject the middleware code after the query and path parameters parsing
because in the case that a middleware needs to execute a more expensive
logic, it isn't executed if there are malformed or missing parameters.

Change-Id: I883ade8ee42d25accb153c78d9f583d883730d6a
2023-11-27 23:02:44 +00:00
Jeremy Wharton
d2819522c6 satellite/admin/back-office: add endpoint to get users by email
This change adds an endpoint to the back office API that returns user
info based on email address.

References #6503

Change-Id: Ib48d30b0b6c6862887b3f8114f50538b3deca57b
2023-11-27 21:05:12 +00:00
Ivan Fraixedes
269dd5602e private/apigen: Don't print empty description & version in docs
If the API description or version is empty, don't print out in the docs
a placeholder for them.

In commit bcde51138d a space between the
placeholder and description was removed because it was printing a
trailing space if the description was empty, but that introduced another
bug, which is not adding the space when the description is empty.

This commit fixes the bug, no printing the placeholder if the
description is empty, otherwise print it with the corresponding blank
space in between the placeholder and the description.

Change-Id: I19c92c3dd7dfb38b8b2a43821bafc726ffad59d7
2023-11-27 16:53:33 +00:00
Ivan Fraixedes
adcd810e37 private/apigen: Allow to customize handlers logic
The API generator doesn't have a way to customize each Go handler
endpoint unless that the Go generator is modified.

This commit adds a way to customize each endpoint injecting instances of
types that implement an interface (Middleware) that return the code to inject.

To show how it works, the commit get rid of the 2 fields that we used to
customize the authentication request with the logic that the
satellite/console/consoleweb/consoleapi needs and replace the hardcoded
customization using this new way to customize handlers.

This new way should allow to hook the satellite/admin/back-office
authorization into the handlers using a Middleware implementation.

Change-Id: I894aa0026b30fa2f4a5604a6c34c22e0ed582e2b
2023-11-23 06:57:40 +00:00
Jeremy Wharton
587fa8fdff private/apigen: use correct TS type for nillable fields
This change gives the proper type to TS class fields generated from
nillable Go struct fields. Previously, Go struct fields having a nil
representation ([]Type, *Type, etc.) were translated into TypeScript as
"Type | undefined". This isn't correct because these fields, when nil,
are given the value "null" when marshalled into JSON. This change fixes
this issue by giving these fields the type "Type | null".

Change-Id: I5a1a83eb3810a3cba10895bb2f0f75ca5fd7d1b5
2023-11-21 17:58:07 +00:00
Jeremy Wharton
0c591fa25a private/apigen: handle omitempty JSON option in structs
This change makes the TypeScript API code generator properly handle
struct fields with the "omitempty" option in the JSON struct tag.

Change-Id: I9b22ce33a8b8c39c115ec827a8e5b7e85d856f83
2023-11-18 07:09:42 -06:00
Ivan Fraixedes
bcde51138d private/apigen: Remove trailing space in doc generator
Remove a trailing space that the doc generator was adding.

Change-Id: I5c24478f79726afbaa5c6772d569b35cafe7c288
2023-11-17 11:44:50 +00:00
Ivan Fraixedes
e67691d51c private/apigen: Remove support for anonymous types
I made several commits to add support to the API generator for
anonymous types because the example was using them.

Later, I was told that we don't need to support them, so the example
should have never used anonymous types.

Supporting anonymous types has added complexity to the API generator and
we are finding bugs in corner cases whose fixes lead to adding more
complexity and some fixes doesn't seem to have a neat solution without
doing a big refactoring.

We decided to reduce the side effects that supporting anonymous type has
brought just removing it.

This commit remove the support for anonymous types and reflect that in
the example.

Change-Id: I2f8c87a0db0e229971ab1bef46cca16fee924191
2023-11-16 12:29:21 +00:00
Jeremy Wharton
032faefa4b private/apigen: fix URL construction in generated TypeScript code
This change fixes an incorrect invocation of the URL object constructor
in generated TypeScript HTTP clients.

Change-Id: I9011bc535f2096374d20b74b401d4cc38a0451fb
2023-11-16 01:12:58 +00:00
Jeremy Wharton
a52934ef4d private/apigen: prevent self imports
This change prevents Go code produced by the API generator from
importing its own package. Previously, we tried to prevent self imports
by skipping import paths whose last segment matched the generated Go
code's package name. However, aliased imports circumvented this.
We now require API definitions to define the Go package path so that we
can compare this with the import path directly.

Change-Id: I7ae7ec5e1a342d2f76cd28ff72d4cd7285c2820a
2023-11-15 21:41:20 +00:00
Jeremy Wharton
f1ce0ab6d0 private/apigen/example: use hardcoded time
This change makes the example API definition use a hardcoded timestamp
rather than the current time so that files aren't unnecessarily
changed each time the API code is generated.

Change-Id: I72e4ec98d29345b9f16ca2ca38a91e593f83ea87
2023-11-15 16:52:18 +00:00
Clement Sam
e27381f3af private/version: use minimum key in new sem version
Much evident on the storagenode dashboard, the minimum
version shown is a very old version and that is taken
from the deprecated part of version info from the
version control server, and we no longer update the
deprecated part on the server.

This change forces it to use the new sem version, and
checks for the old version if the server is probably
running an old version of the version control system.

Also fixes a bug where the suggested version returned
for all processes is taken from the storagenode part.

Issue: https://github.com/storj/storj/issues/5673

Change-Id: I57b7358c2826a6e25f441dfa9579a1aef50a7e89
2023-11-08 21:08:01 +00:00
Ivan Fraixedes
48877c05cc
private/apigen: Fix TS generator to align with linter
I introduced some subtle linter issues when I added the APIError class
and added the TypeScript mock generator.

This commit addresses them, so the linter doesn't yell about the
TypeScript generated sources.

Change-Id: Icc7dfa4169a228b1a5144d4a292f4350ee5ef9f0
2023-11-03 16:41:54 +01:00
Ivan Fraixedes
74757ffc1d private/apigen/example: Fix client TS generated file
We introduced some changes in 882c9d64e4
and one reviewer spot an issue which was fixed, but I forgot to
regenerate the files and the generated Typescript client got stale.

This commit regenerate the example code, so the generated code is align
with the current generator version.

Change-Id: Idc023adc9c6e230578f9e77e492126355b006203
2023-11-02 14:24:30 +00:00
Egon Elbre
0a4a1098d8 go.mod: bump common (and maybe fix quic)
Change-Id: Ib37213ab6d26d2b6adc727e913e3df3e12e9ee9d
2023-11-01 14:42:15 +02:00
Egon Elbre
38ad5a1318 private/testplanet: add STORJ_TEST_DISABLEQUIC environment flag
Currently Windows seems to dislike creating udp servers in a very
fast fashion. Add an environment flag that allows to disable quic,
which is the main culprit.

Individual tests can still override that setting.

Change-Id: I3b30c4aa7fcb148b2894335394fdfae6eaa372bb
2023-11-01 11:39:03 +00:00
Egon Elbre
46f7e5b7fa all: don't depend on infectious library directly
Change-Id: I0bcc719cc25e1d7ad36a16f65d40220eccba6556
2023-10-31 16:00:47 +02:00
Ivan Fraixedes
9338f3f088
private/apigen: Add a TypeScript client mock generator
Implement a TypeScript client mock generator to generate mocks with
static data specified in the API definition.

Change-Id: I11419f4bbf72576fcd829f9d4acd8471034ca571
2023-10-25 14:43:01 +02:00
Ivan Fraixedes
882c9d64e4
private/apigen: Create class for API errors (TypeScript)
Create a class that inherits the Javascript Error class to use it when
throwing errors on the TypeScript client when the ok property of the
Fetch API Response object is false, which means that the server
responded with a status code less than 200 or greater than 299.

The APIError class contains apart of the message, the response status
code which may be useful for the caller to decide what to do, for
example, it may decide to retry the call on certain response status
errors.

Change-Id: Ic48466b40cbf134a27d5c92a4af9f93232e84fca
2023-10-25 14:43:00 +02:00
Ivan Fraixedes
3e71ea555d private/apigen: Remove stale TODOs & improve doc
Remove some stale TODO comments and make a method documentation more
specific.

Change-Id: I21b9df0f22a09698014518b5a675eb115d1ba4a4
2023-10-19 12:42:58 +00:00
Jeremy Wharton
6d03b92ea6 private/apigen: fix TS class generation from collection of named struct
This change fixes an issue where the TypeScript API code generator
would silently fail to generate class definitions for named structs
used as the element type of array or slice struct fields.

This issue caused invalid TypeScript code to be generated from the
frontend config struct because it contained a slice of partnered
satellite structs. The TypeScript class definition of the frontend
config referenced the nonexistent partnered satellite class.
The satellite frontend config uses, so this issue caused the

Change-Id: Idfcb6ec1bbc603a43033ee4afb5b421b7454855c
2023-10-17 22:45:07 -05:00
Ivan Fraixedes
386a978310 private/apigen: Validate param types
The Go generator only supports certain types as query and path
parameters and it panics on any an unsupported type.

The Document and TypeScript generator don't have any validation for
them. TypeScript generator generates code that compiles, however, it
won't work properly with all the types not supported by the Go
generator.

Because it doesn't make sense that some types may work on the TypeScript
generator, while the Go generator doesn't, doing the validation in the
Param constructor is better because it reports the issue without having
to run the Go generator and it gives a more understanding panic message.

TypeScript generator generates code that works properly with all the
types supported by the Go generator, hence, there isn't any change int
he TypeScript generator in this commit.

Change-Id: I03085283942a98341726a1560f511a46540df9f5
2023-10-10 12:37:20 +00:00
Ivan Fraixedes
99c4359062 private/apigen: Avoid clashes of types names
The TypeScript generator create types of anonymous structs. When those
anonymous structs are in endpoints with the same name, but in different
endpoint groups, the generator panic because it create different types
with the same names.

This commit fixes that problem through using the endpoint group prefix
to create the types with different names.

Change-Id: Ibe87532609ce824b80951326f9777ed5b0cc2f7a
2023-10-05 07:30:09 +00:00
Ivan Fraixedes
3193ff9155 private/apigen: Don't force casing for API group name/prefix
The API generators rely on the Name and Prefix fields of the
EndpointGroup type to generate code.

Conventional naming code requires using upper or lower case for types,
functions, etc, however requiring the user to set this fields with the
correct casing seems cumbersome for them because they can be adjusted
depending where those values are used on the generated code.

This commit lifts the restriction for the user and adjust the casing of
them according to where they are used.

Change-Id: I700a879d13b4789b4d6ba0519b4d7508061eac73
2023-10-05 06:48:50 +00:00
Ethan Adams
6961d2e19d satellite/email: Add support for unauthenticated, cleartext SMTP connections
Change-Id: I11b4852122764c1ede188ca40d5edb14f2c4ee72
2023-10-04 16:55:55 +00:00
Ivan Fraixedes
47f344927d
private/apigen: Fix TS generator to apply naming conventions
Fix the TypeScript generator to generate code using the common
TypeScript conventions.

Closes https://github.com/storj/storj/issues/6360

Change-Id: I244896feea389670eca0df95d3ac58e25d600e14
2023-10-04 13:47:33 +02:00
Ivan Fraixedes
71c9547de5
private/apigen: Don't allow certain types for Request/Response
We cannot map certain types to TypeScript, hence we verify them in the
Endpoint.Validate method.

Even some types could be somehow mapped, we don't want to add more
complexity or allow types that don't make sense to be for a request or
response.

Change-Id: I51ecee286e637b1160e967d77f9ce6c7403ddfdc
2023-10-04 13:46:54 +02:00
Ivan Fraixedes
956109a097
private/apigen: Rename Endpoint fields
Rename the Endpoint fields MethodName and RequestName because they were
confusing for what they are used.

This commit also adds some validations for these fields values and other
validations for Endpoint and EndpointGroup to avoid generating invalid
code.

It also include some tests for these new validations.

Closes https://github.com/storj/storj/issues/6333

Change-Id: Iaabfc33935517889e3729c8b37be51a55eea366c
2023-10-04 13:46:54 +02:00
Ivan Fraixedes
ac86eb397a
private/apigen: Fix bug TypeScript optional class members
The TypesScript generator generates classes and checks for Go struct
field if the struct type is of a nullable type, however, it should check
if the field types is nullable because it doesn't make sense to check
that struct field for each field nor to assign to the field if it's
nullable or not depending on it.

Change-Id: Ia22a609a17752f520233c006cba17685fe142b32
2023-10-04 13:46:08 +02:00
Ivan Fraixedes
9d7ef17a26
private/apigen: Fix code generation for slices & arrays
Fix the API generator to generate valid TypeScript code when using
slices an arrays of any type (base types, struct types, anonymous struct
types, etc.).

Closes https://github.com/storj/storj/issues/6323

Change-Id: I580ae5305c58f65c2e4f4a35d14ca4ee509a9250
2023-10-02 17:05:35 +02:00
Ivan Fraixedes
48d7be7eab
private/apigen: Panic types defined in main package
The API generator was generating invalid code when types were defined in
a main package because the generated Go code was defining in import from
it.

This commit update the Go generator to panic with a explicit error
message if that situation happens.

The commit also add a new endpoint to the example with a named types
(i.e. no anonymous) to show that the Generator works fine with them.

Change-Id: Ieddd89c67048de50516f7ac7787d602660dc4a54
2023-09-26 14:04:33 +02:00
Ivan Fraixedes
7ab7ac49c8
private/apigen: Remove TypeScript errors from generated code
Remove the "duplicate identifier: 'path'" TypeScript errors from the
generated code.

Change-Id: I6b411a5cee720e2a16353034627954616c480f8a
2023-09-26 14:04:33 +02:00
Ivan Fraixedes
00484429d6
private/apigen: Generate valid TypeScript code with anonymous types
The API generator didn't generate valid TypeScript code when using
Go anonymous types.

This commit fixes that issue creating names for anonymous types.

Change-Id: Ice0748d8650686e3d3979523b8f218dc20eade5a
2023-09-26 14:04:33 +02:00
Ivan Fraixedes
a9901cc7d0
private/apigen: Add validations to improve usage
Add a few validations to panic with a nicer message or abort rather than
generating invalid code.

Also improve the panic message wrapping a standard error with errs2 at
the time that it's returned.

Change-Id: I1393933eb5f0bc3f86646bf4d0acfc64626efbe0
2023-09-26 14:04:33 +02:00
Ivan Fraixedes
822a13570e
private/apigen: Document Endpoint fields
Add document comments to tall the fields of the Endpoint struct.

Change-Id: Ibcd534f26474fc73fb378a9f30ce7d9401217557
2023-09-26 14:04:32 +02:00
Ivan Fraixedes
9a06de9058 private/apigen: Add test for Types.All
Add a test for Types.All method because we may need to adjust the logic
in future commits and we want to detect what has changed for good or
bad.

Change-Id: I1db4bf67db3c87513cb9aeb7b942c6c132fc4dd1
2023-09-25 21:15:06 +00:00
Ivan Fraixedes
926076bffd private/apigen/example: Add description to endpoint
Add the description field to the endpoint to show where it ends up when
generating code.

Change-Id: I415f088dbf795656ed1ef042ed41ebf39a517692
2023-09-19 18:31:20 +00:00
Andrew Harding
fd679c329c private/server: FreeBSD TCP fastopen support detection
Detects whether or not TCP fastopen is supported by running sysctl to
grab the current value of net.inet.tcp.fastopen.server_enable.

If not enabled, directs the operator to use sysctl to set it temporarily
and /etc/sysctl.conf to set it on-boot.

Setting the socket option always works whether it is enabled or not.

Verified that fastopen works on FreeBSD 13. Did not attempt on earlier
versions but support has been there since FreeBSD 10.

Change-Id: I2e0c457558a6fa7b7a1b18bc3c6684aff50b81a2
2023-09-18 21:17:42 +00:00
Andrew Harding
e4d6829971 private/server: Windows TCP fast-open support detection
Implements tryEnableFastOpen by creating a localhost socket and enabling
the TCP_FASTOPEN socket option. On builds where TCP_FASTOPEN isn't
available, setting the socket option fails and tryEnableFastOpen returns
false.

This was verified on a supporting build (latest Windows 10) and an
unsupporting build (Windows 8.1, couldn't find an ISO for an older
Windows 10 build).

Change-Id: I497117dc2f04acdd2b0cc836e20d12d69076b939
2023-09-18 14:15:30 +00:00
Márton Elek
e2006d821c satellite/overlay: change Reliable and KnownReliable
as GetParticipatingNodes and GetNodes, respectively.

We now want these functions to include offline and suspended nodes as
well, so that we can force immediate repair when pieces are out of
placement or in excluded countries. With that change, the old names no
longer made sense.

Change-Id: Icbcbad43dbde0ca8cbc80a4d17a896bb89b078b7
2023-09-02 23:34:50 +00:00
Moby von Briesen
b4c95a3b28 private/apigen/tsgen: Set query params better
The old way did not properly handle escaping, e.g. if the value of a
query param contained `&` or `=` inside it. By using
url.searchParams.set, we can safely add these types of arguments to the
path.

Change-Id: I62d3883b14f9d5a517e4a3d58f019014b46fd1b4
2023-08-31 20:09:29 -04:00
Egon Elbre
273ebd61d7 private/apigen/example: make it nicer
Change-Id: I1bd779090a902ed2b99b3993dc7cf61fc250f10f
2023-08-28 15:13:46 +00:00
Ivan Fraixedes
2d8f396eeb private/apigen: Make API base path configurable
Previously the base path for the API was hardcoded to `/api` and the
specified version.

This was not obvious that the generated code was setting that base path
and it was not flexible for serving the API under a different path than
`/api`.

We will likely need to set a different base path if we pretend to serve
the new back office API that we are going to implement alongside the
current admin API until the new back office is fully implemented and
verified that works properly.

This commit also fix add the base path of the endpoints to the
documentation because it was even more confusing for somebody that wants
to use the API having to find out them through looking to the generated
code.

Change-Id: I6efab6b6f3d295129d6f42f7fbba8c2dc19725f4
2023-08-28 14:35:01 +00:00
Ivan Fraixedes
d34c1b6825 private/apigen: Generate docs & typescript for example
Generate the documentation and the typescript code for the example of
that we have to generate an API through the API generator.

The JSON Go struct field tags are needed because the typescript
generator require them, otherwise it panics.

The test had to be adjusted according to match the tags so Go consider
the structs the same type, otherwise, it doesn't compile.

Change-Id: I3e4ebff9174885c50ca2058b86b7ec60e025945c
2023-08-25 12:30:21 +00:00
Moby von Briesen
d76f059c55 private/apigen: Update generated doc links
Make the link more human-friendly - to contain the text of the group and
endpoint names.

Also link back to list of endpoints from each endpoint.

Change-Id: Ia3e2ebe20b58b5f60ecefe9d35fb8fd90dd4b4d7
2023-08-25 11:51:57 +00:00
Moby von Briesen
7fbabbcf16 private/apigen: Add list of endpoints with links
This change makes it easier for someone reading the documentation to see
a full list of supported endpoints, and have direct links to the
details.

Change-Id: I46e2f809cfa2760845898eaa3d99db9066d435ef
2023-08-24 02:45:46 +00:00
Ivan Fraixedes
b0d072270b private/apigen: Add comment about panics
Add a comment in 2 methods about they panic as it's specified for the
`MustWriteDocs`.

Change-Id: I15d5fa420af37318d828bc633bed62988f8b207b
2023-08-23 18:50:37 +00:00
Egon Elbre
b5e1e5a9e2 private/apigen: use typescript in docs markdown
Fixes https://github.com/storj/storj/issues/6187

Change-Id: I34016513f29a9538343f0909951ab1a4605bb585
2023-08-17 15:33:21 +00:00