Commit Graph

629 Commits

Author SHA1 Message Date
Jeremy Wharton
0c591fa25a private/apigen: handle omitempty JSON option in structs
This change makes the TypeScript API code generator properly handle
struct fields with the "omitempty" option in the JSON struct tag.

Change-Id: I9b22ce33a8b8c39c115ec827a8e5b7e85d856f83
2023-11-18 07:09:42 -06:00
Ivan Fraixedes
bcde51138d private/apigen: Remove trailing space in doc generator
Remove a trailing space that the doc generator was adding.

Change-Id: I5c24478f79726afbaa5c6772d569b35cafe7c288
2023-11-17 11:44:50 +00:00
Ivan Fraixedes
e67691d51c private/apigen: Remove support for anonymous types
I made several commits to add support to the API generator for
anonymous types because the example was using them.

Later, I was told that we don't need to support them, so the example
should have never used anonymous types.

Supporting anonymous types has added complexity to the API generator and
we are finding bugs in corner cases whose fixes lead to adding more
complexity and some fixes doesn't seem to have a neat solution without
doing a big refactoring.

We decided to reduce the side effects that supporting anonymous type has
brought just removing it.

This commit remove the support for anonymous types and reflect that in
the example.

Change-Id: I2f8c87a0db0e229971ab1bef46cca16fee924191
2023-11-16 12:29:21 +00:00
Jeremy Wharton
032faefa4b private/apigen: fix URL construction in generated TypeScript code
This change fixes an incorrect invocation of the URL object constructor
in generated TypeScript HTTP clients.

Change-Id: I9011bc535f2096374d20b74b401d4cc38a0451fb
2023-11-16 01:12:58 +00:00
Jeremy Wharton
a52934ef4d private/apigen: prevent self imports
This change prevents Go code produced by the API generator from
importing its own package. Previously, we tried to prevent self imports
by skipping import paths whose last segment matched the generated Go
code's package name. However, aliased imports circumvented this.
We now require API definitions to define the Go package path so that we
can compare this with the import path directly.

Change-Id: I7ae7ec5e1a342d2f76cd28ff72d4cd7285c2820a
2023-11-15 21:41:20 +00:00
Jeremy Wharton
f1ce0ab6d0 private/apigen/example: use hardcoded time
This change makes the example API definition use a hardcoded timestamp
rather than the current time so that files aren't unnecessarily
changed each time the API code is generated.

Change-Id: I72e4ec98d29345b9f16ca2ca38a91e593f83ea87
2023-11-15 16:52:18 +00:00
Clement Sam
e27381f3af private/version: use minimum key in new sem version
Much evident on the storagenode dashboard, the minimum
version shown is a very old version and that is taken
from the deprecated part of version info from the
version control server, and we no longer update the
deprecated part on the server.

This change forces it to use the new sem version, and
checks for the old version if the server is probably
running an old version of the version control system.

Also fixes a bug where the suggested version returned
for all processes is taken from the storagenode part.

Issue: https://github.com/storj/storj/issues/5673

Change-Id: I57b7358c2826a6e25f441dfa9579a1aef50a7e89
2023-11-08 21:08:01 +00:00
Ivan Fraixedes
48877c05cc
private/apigen: Fix TS generator to align with linter
I introduced some subtle linter issues when I added the APIError class
and added the TypeScript mock generator.

This commit addresses them, so the linter doesn't yell about the
TypeScript generated sources.

Change-Id: Icc7dfa4169a228b1a5144d4a292f4350ee5ef9f0
2023-11-03 16:41:54 +01:00
Ivan Fraixedes
74757ffc1d private/apigen/example: Fix client TS generated file
We introduced some changes in 882c9d64e4
and one reviewer spot an issue which was fixed, but I forgot to
regenerate the files and the generated Typescript client got stale.

This commit regenerate the example code, so the generated code is align
with the current generator version.

Change-Id: Idc023adc9c6e230578f9e77e492126355b006203
2023-11-02 14:24:30 +00:00
Egon Elbre
0a4a1098d8 go.mod: bump common (and maybe fix quic)
Change-Id: Ib37213ab6d26d2b6adc727e913e3df3e12e9ee9d
2023-11-01 14:42:15 +02:00
Egon Elbre
38ad5a1318 private/testplanet: add STORJ_TEST_DISABLEQUIC environment flag
Currently Windows seems to dislike creating udp servers in a very
fast fashion. Add an environment flag that allows to disable quic,
which is the main culprit.

Individual tests can still override that setting.

Change-Id: I3b30c4aa7fcb148b2894335394fdfae6eaa372bb
2023-11-01 11:39:03 +00:00
Egon Elbre
46f7e5b7fa all: don't depend on infectious library directly
Change-Id: I0bcc719cc25e1d7ad36a16f65d40220eccba6556
2023-10-31 16:00:47 +02:00
Ivan Fraixedes
9338f3f088
private/apigen: Add a TypeScript client mock generator
Implement a TypeScript client mock generator to generate mocks with
static data specified in the API definition.

Change-Id: I11419f4bbf72576fcd829f9d4acd8471034ca571
2023-10-25 14:43:01 +02:00
Ivan Fraixedes
882c9d64e4
private/apigen: Create class for API errors (TypeScript)
Create a class that inherits the Javascript Error class to use it when
throwing errors on the TypeScript client when the ok property of the
Fetch API Response object is false, which means that the server
responded with a status code less than 200 or greater than 299.

The APIError class contains apart of the message, the response status
code which may be useful for the caller to decide what to do, for
example, it may decide to retry the call on certain response status
errors.

Change-Id: Ic48466b40cbf134a27d5c92a4af9f93232e84fca
2023-10-25 14:43:00 +02:00
Ivan Fraixedes
3e71ea555d private/apigen: Remove stale TODOs & improve doc
Remove some stale TODO comments and make a method documentation more
specific.

Change-Id: I21b9df0f22a09698014518b5a675eb115d1ba4a4
2023-10-19 12:42:58 +00:00
Jeremy Wharton
6d03b92ea6 private/apigen: fix TS class generation from collection of named struct
This change fixes an issue where the TypeScript API code generator
would silently fail to generate class definitions for named structs
used as the element type of array or slice struct fields.

This issue caused invalid TypeScript code to be generated from the
frontend config struct because it contained a slice of partnered
satellite structs. The TypeScript class definition of the frontend
config referenced the nonexistent partnered satellite class.
The satellite frontend config uses, so this issue caused the

Change-Id: Idfcb6ec1bbc603a43033ee4afb5b421b7454855c
2023-10-17 22:45:07 -05:00
Ivan Fraixedes
386a978310 private/apigen: Validate param types
The Go generator only supports certain types as query and path
parameters and it panics on any an unsupported type.

The Document and TypeScript generator don't have any validation for
them. TypeScript generator generates code that compiles, however, it
won't work properly with all the types not supported by the Go
generator.

Because it doesn't make sense that some types may work on the TypeScript
generator, while the Go generator doesn't, doing the validation in the
Param constructor is better because it reports the issue without having
to run the Go generator and it gives a more understanding panic message.

TypeScript generator generates code that works properly with all the
types supported by the Go generator, hence, there isn't any change int
he TypeScript generator in this commit.

Change-Id: I03085283942a98341726a1560f511a46540df9f5
2023-10-10 12:37:20 +00:00
Ivan Fraixedes
99c4359062 private/apigen: Avoid clashes of types names
The TypeScript generator create types of anonymous structs. When those
anonymous structs are in endpoints with the same name, but in different
endpoint groups, the generator panic because it create different types
with the same names.

This commit fixes that problem through using the endpoint group prefix
to create the types with different names.

Change-Id: Ibe87532609ce824b80951326f9777ed5b0cc2f7a
2023-10-05 07:30:09 +00:00
Ivan Fraixedes
3193ff9155 private/apigen: Don't force casing for API group name/prefix
The API generators rely on the Name and Prefix fields of the
EndpointGroup type to generate code.

Conventional naming code requires using upper or lower case for types,
functions, etc, however requiring the user to set this fields with the
correct casing seems cumbersome for them because they can be adjusted
depending where those values are used on the generated code.

This commit lifts the restriction for the user and adjust the casing of
them according to where they are used.

Change-Id: I700a879d13b4789b4d6ba0519b4d7508061eac73
2023-10-05 06:48:50 +00:00
Ethan Adams
6961d2e19d satellite/email: Add support for unauthenticated, cleartext SMTP connections
Change-Id: I11b4852122764c1ede188ca40d5edb14f2c4ee72
2023-10-04 16:55:55 +00:00
Ivan Fraixedes
47f344927d
private/apigen: Fix TS generator to apply naming conventions
Fix the TypeScript generator to generate code using the common
TypeScript conventions.

Closes https://github.com/storj/storj/issues/6360

Change-Id: I244896feea389670eca0df95d3ac58e25d600e14
2023-10-04 13:47:33 +02:00
Ivan Fraixedes
71c9547de5
private/apigen: Don't allow certain types for Request/Response
We cannot map certain types to TypeScript, hence we verify them in the
Endpoint.Validate method.

Even some types could be somehow mapped, we don't want to add more
complexity or allow types that don't make sense to be for a request or
response.

Change-Id: I51ecee286e637b1160e967d77f9ce6c7403ddfdc
2023-10-04 13:46:54 +02:00
Ivan Fraixedes
956109a097
private/apigen: Rename Endpoint fields
Rename the Endpoint fields MethodName and RequestName because they were
confusing for what they are used.

This commit also adds some validations for these fields values and other
validations for Endpoint and EndpointGroup to avoid generating invalid
code.

It also include some tests for these new validations.

Closes https://github.com/storj/storj/issues/6333

Change-Id: Iaabfc33935517889e3729c8b37be51a55eea366c
2023-10-04 13:46:54 +02:00
Ivan Fraixedes
ac86eb397a
private/apigen: Fix bug TypeScript optional class members
The TypesScript generator generates classes and checks for Go struct
field if the struct type is of a nullable type, however, it should check
if the field types is nullable because it doesn't make sense to check
that struct field for each field nor to assign to the field if it's
nullable or not depending on it.

Change-Id: Ia22a609a17752f520233c006cba17685fe142b32
2023-10-04 13:46:08 +02:00
Ivan Fraixedes
9d7ef17a26
private/apigen: Fix code generation for slices & arrays
Fix the API generator to generate valid TypeScript code when using
slices an arrays of any type (base types, struct types, anonymous struct
types, etc.).

Closes https://github.com/storj/storj/issues/6323

Change-Id: I580ae5305c58f65c2e4f4a35d14ca4ee509a9250
2023-10-02 17:05:35 +02:00
Ivan Fraixedes
48d7be7eab
private/apigen: Panic types defined in main package
The API generator was generating invalid code when types were defined in
a main package because the generated Go code was defining in import from
it.

This commit update the Go generator to panic with a explicit error
message if that situation happens.

The commit also add a new endpoint to the example with a named types
(i.e. no anonymous) to show that the Generator works fine with them.

Change-Id: Ieddd89c67048de50516f7ac7787d602660dc4a54
2023-09-26 14:04:33 +02:00
Ivan Fraixedes
7ab7ac49c8
private/apigen: Remove TypeScript errors from generated code
Remove the "duplicate identifier: 'path'" TypeScript errors from the
generated code.

Change-Id: I6b411a5cee720e2a16353034627954616c480f8a
2023-09-26 14:04:33 +02:00
Ivan Fraixedes
00484429d6
private/apigen: Generate valid TypeScript code with anonymous types
The API generator didn't generate valid TypeScript code when using
Go anonymous types.

This commit fixes that issue creating names for anonymous types.

Change-Id: Ice0748d8650686e3d3979523b8f218dc20eade5a
2023-09-26 14:04:33 +02:00
Ivan Fraixedes
a9901cc7d0
private/apigen: Add validations to improve usage
Add a few validations to panic with a nicer message or abort rather than
generating invalid code.

Also improve the panic message wrapping a standard error with errs2 at
the time that it's returned.

Change-Id: I1393933eb5f0bc3f86646bf4d0acfc64626efbe0
2023-09-26 14:04:33 +02:00
Ivan Fraixedes
822a13570e
private/apigen: Document Endpoint fields
Add document comments to tall the fields of the Endpoint struct.

Change-Id: Ibcd534f26474fc73fb378a9f30ce7d9401217557
2023-09-26 14:04:32 +02:00
Ivan Fraixedes
9a06de9058 private/apigen: Add test for Types.All
Add a test for Types.All method because we may need to adjust the logic
in future commits and we want to detect what has changed for good or
bad.

Change-Id: I1db4bf67db3c87513cb9aeb7b942c6c132fc4dd1
2023-09-25 21:15:06 +00:00
Ivan Fraixedes
926076bffd private/apigen/example: Add description to endpoint
Add the description field to the endpoint to show where it ends up when
generating code.

Change-Id: I415f088dbf795656ed1ef042ed41ebf39a517692
2023-09-19 18:31:20 +00:00
Andrew Harding
fd679c329c private/server: FreeBSD TCP fastopen support detection
Detects whether or not TCP fastopen is supported by running sysctl to
grab the current value of net.inet.tcp.fastopen.server_enable.

If not enabled, directs the operator to use sysctl to set it temporarily
and /etc/sysctl.conf to set it on-boot.

Setting the socket option always works whether it is enabled or not.

Verified that fastopen works on FreeBSD 13. Did not attempt on earlier
versions but support has been there since FreeBSD 10.

Change-Id: I2e0c457558a6fa7b7a1b18bc3c6684aff50b81a2
2023-09-18 21:17:42 +00:00
Andrew Harding
e4d6829971 private/server: Windows TCP fast-open support detection
Implements tryEnableFastOpen by creating a localhost socket and enabling
the TCP_FASTOPEN socket option. On builds where TCP_FASTOPEN isn't
available, setting the socket option fails and tryEnableFastOpen returns
false.

This was verified on a supporting build (latest Windows 10) and an
unsupporting build (Windows 8.1, couldn't find an ISO for an older
Windows 10 build).

Change-Id: I497117dc2f04acdd2b0cc836e20d12d69076b939
2023-09-18 14:15:30 +00:00
Márton Elek
e2006d821c satellite/overlay: change Reliable and KnownReliable
as GetParticipatingNodes and GetNodes, respectively.

We now want these functions to include offline and suspended nodes as
well, so that we can force immediate repair when pieces are out of
placement or in excluded countries. With that change, the old names no
longer made sense.

Change-Id: Icbcbad43dbde0ca8cbc80a4d17a896bb89b078b7
2023-09-02 23:34:50 +00:00
Moby von Briesen
b4c95a3b28 private/apigen/tsgen: Set query params better
The old way did not properly handle escaping, e.g. if the value of a
query param contained `&` or `=` inside it. By using
url.searchParams.set, we can safely add these types of arguments to the
path.

Change-Id: I62d3883b14f9d5a517e4a3d58f019014b46fd1b4
2023-08-31 20:09:29 -04:00
Egon Elbre
273ebd61d7 private/apigen/example: make it nicer
Change-Id: I1bd779090a902ed2b99b3993dc7cf61fc250f10f
2023-08-28 15:13:46 +00:00
Ivan Fraixedes
2d8f396eeb private/apigen: Make API base path configurable
Previously the base path for the API was hardcoded to `/api` and the
specified version.

This was not obvious that the generated code was setting that base path
and it was not flexible for serving the API under a different path than
`/api`.

We will likely need to set a different base path if we pretend to serve
the new back office API that we are going to implement alongside the
current admin API until the new back office is fully implemented and
verified that works properly.

This commit also fix add the base path of the endpoints to the
documentation because it was even more confusing for somebody that wants
to use the API having to find out them through looking to the generated
code.

Change-Id: I6efab6b6f3d295129d6f42f7fbba8c2dc19725f4
2023-08-28 14:35:01 +00:00
Ivan Fraixedes
d34c1b6825 private/apigen: Generate docs & typescript for example
Generate the documentation and the typescript code for the example of
that we have to generate an API through the API generator.

The JSON Go struct field tags are needed because the typescript
generator require them, otherwise it panics.

The test had to be adjusted according to match the tags so Go consider
the structs the same type, otherwise, it doesn't compile.

Change-Id: I3e4ebff9174885c50ca2058b86b7ec60e025945c
2023-08-25 12:30:21 +00:00
Moby von Briesen
d76f059c55 private/apigen: Update generated doc links
Make the link more human-friendly - to contain the text of the group and
endpoint names.

Also link back to list of endpoints from each endpoint.

Change-Id: Ia3e2ebe20b58b5f60ecefe9d35fb8fd90dd4b4d7
2023-08-25 11:51:57 +00:00
Moby von Briesen
7fbabbcf16 private/apigen: Add list of endpoints with links
This change makes it easier for someone reading the documentation to see
a full list of supported endpoints, and have direct links to the
details.

Change-Id: I46e2f809cfa2760845898eaa3d99db9066d435ef
2023-08-24 02:45:46 +00:00
Ivan Fraixedes
b0d072270b private/apigen: Add comment about panics
Add a comment in 2 methods about they panic as it's specified for the
`MustWriteDocs`.

Change-Id: I15d5fa420af37318d828bc633bed62988f8b207b
2023-08-23 18:50:37 +00:00
Egon Elbre
b5e1e5a9e2 private/apigen: use typescript in docs markdown
Fixes https://github.com/storj/storj/issues/6187

Change-Id: I34016513f29a9538343f0909951ab1a4605bb585
2023-08-17 15:33:21 +00:00
Egon Elbre
1f261bcc70 go.mod: bump lang to 1.19 and common
Change-Id: I8d91f97d786456da29ebe89a78412c50efbb8ccc
2023-08-10 18:41:15 +03:00
Wilfred Asomani
34e1caa55a satellite: add get user paged projects http endpoint
This change adds an endpoint to get a user's projects, similar to
the OwnedProjects GraphQL query.
The console.ProjectInfo struct has been renamed to UpsertProjectInfo
to more accurately reflect its use.

Issue: https://github.com/storj/storj/issues/6135

Change-Id: I802fe4694a5cc75a9df2b565476f6e6f473431d4
2023-08-08 14:02:53 +00:00
Wilfred Asomani
c934974652 satellite/{console,web}: update error handling
This change updates how API errors are handled and sent.

Change-Id: Ia4c71eeb6f2d009a47b59ce77a23f70b8b10f6dc
2023-07-27 21:01:01 +00:00
Clement Sam
4cb85186b2 storagenode/pieces: enable lazyfilewalker by default
Resolves https://github.com/storj/storj/issues/5861

Change-Id: I20e0a5b8a15ca966cbccd71369322a517a2c2130
2023-07-19 15:25:20 +00:00
Wilfred Asomani
4ee647a951 satellite: add request id to requests
This change adds request IDs to requests, logs them as part of audit
logs and sends to the client on error. This is to improve debugging
of customer issues.

Issue: https://github.com/storj/storj/issues/5898

Change-Id: I801514b547d28d810552d91aa7c8502051e552bf
2023-07-13 09:22:43 +00:00
Cameron
7e03ccfa46 satellite/console: optional separate web app server
This change creates the ability to run a server separate from the
console web server to serve the front end app. You can run it with
`satellite run ui`. Since there are now potentially two servers instead
of one, the UI server has the option to act as a reverse proxy to the
api server for local development by setting `--console.address` to the
console backend address and `--console.backend-reverse-proxy` to the
console backend's http url. Also, a feature flag has been implemented
on the api server to retain the ability to serve the front end app. It
is toggled with `--console.frontend-enable`.

github issue: https://github.com/storj/storj/issues/5843

Change-Id: I0d30451a20636e3184110dbe28c8a2a8a9505804
2023-07-11 12:17:35 -04:00
Michal Niewrzal
1f92e7acda satellite: move GC sender to Core peer
Having separate process/pod only for sending bloom filters once a week
is a bit waste. After reconfiguring production settings to use sender in
core we can remove also GC sender peer code.

https://github.com/storj/storj/issues/5979

Change-Id: I6efe3ec073f96545e1f70ad13843f8ccdf923ee8
2023-07-11 10:31:35 +00:00