Commit Graph

613 Commits

Author SHA1 Message Date
Ivan Fraixedes
386a978310 private/apigen: Validate param types
The Go generator only supports certain types as query and path
parameters and it panics on any an unsupported type.

The Document and TypeScript generator don't have any validation for
them. TypeScript generator generates code that compiles, however, it
won't work properly with all the types not supported by the Go
generator.

Because it doesn't make sense that some types may work on the TypeScript
generator, while the Go generator doesn't, doing the validation in the
Param constructor is better because it reports the issue without having
to run the Go generator and it gives a more understanding panic message.

TypeScript generator generates code that works properly with all the
types supported by the Go generator, hence, there isn't any change int
he TypeScript generator in this commit.

Change-Id: I03085283942a98341726a1560f511a46540df9f5
2023-10-10 12:37:20 +00:00
Ivan Fraixedes
99c4359062 private/apigen: Avoid clashes of types names
The TypeScript generator create types of anonymous structs. When those
anonymous structs are in endpoints with the same name, but in different
endpoint groups, the generator panic because it create different types
with the same names.

This commit fixes that problem through using the endpoint group prefix
to create the types with different names.

Change-Id: Ibe87532609ce824b80951326f9777ed5b0cc2f7a
2023-10-05 07:30:09 +00:00
Ivan Fraixedes
3193ff9155 private/apigen: Don't force casing for API group name/prefix
The API generators rely on the Name and Prefix fields of the
EndpointGroup type to generate code.

Conventional naming code requires using upper or lower case for types,
functions, etc, however requiring the user to set this fields with the
correct casing seems cumbersome for them because they can be adjusted
depending where those values are used on the generated code.

This commit lifts the restriction for the user and adjust the casing of
them according to where they are used.

Change-Id: I700a879d13b4789b4d6ba0519b4d7508061eac73
2023-10-05 06:48:50 +00:00
Ethan Adams
6961d2e19d satellite/email: Add support for unauthenticated, cleartext SMTP connections
Change-Id: I11b4852122764c1ede188ca40d5edb14f2c4ee72
2023-10-04 16:55:55 +00:00
Ivan Fraixedes
47f344927d
private/apigen: Fix TS generator to apply naming conventions
Fix the TypeScript generator to generate code using the common
TypeScript conventions.

Closes https://github.com/storj/storj/issues/6360

Change-Id: I244896feea389670eca0df95d3ac58e25d600e14
2023-10-04 13:47:33 +02:00
Ivan Fraixedes
71c9547de5
private/apigen: Don't allow certain types for Request/Response
We cannot map certain types to TypeScript, hence we verify them in the
Endpoint.Validate method.

Even some types could be somehow mapped, we don't want to add more
complexity or allow types that don't make sense to be for a request or
response.

Change-Id: I51ecee286e637b1160e967d77f9ce6c7403ddfdc
2023-10-04 13:46:54 +02:00
Ivan Fraixedes
956109a097
private/apigen: Rename Endpoint fields
Rename the Endpoint fields MethodName and RequestName because they were
confusing for what they are used.

This commit also adds some validations for these fields values and other
validations for Endpoint and EndpointGroup to avoid generating invalid
code.

It also include some tests for these new validations.

Closes https://github.com/storj/storj/issues/6333

Change-Id: Iaabfc33935517889e3729c8b37be51a55eea366c
2023-10-04 13:46:54 +02:00
Ivan Fraixedes
ac86eb397a
private/apigen: Fix bug TypeScript optional class members
The TypesScript generator generates classes and checks for Go struct
field if the struct type is of a nullable type, however, it should check
if the field types is nullable because it doesn't make sense to check
that struct field for each field nor to assign to the field if it's
nullable or not depending on it.

Change-Id: Ia22a609a17752f520233c006cba17685fe142b32
2023-10-04 13:46:08 +02:00
Ivan Fraixedes
9d7ef17a26
private/apigen: Fix code generation for slices & arrays
Fix the API generator to generate valid TypeScript code when using
slices an arrays of any type (base types, struct types, anonymous struct
types, etc.).

Closes https://github.com/storj/storj/issues/6323

Change-Id: I580ae5305c58f65c2e4f4a35d14ca4ee509a9250
2023-10-02 17:05:35 +02:00
Ivan Fraixedes
48d7be7eab
private/apigen: Panic types defined in main package
The API generator was generating invalid code when types were defined in
a main package because the generated Go code was defining in import from
it.

This commit update the Go generator to panic with a explicit error
message if that situation happens.

The commit also add a new endpoint to the example with a named types
(i.e. no anonymous) to show that the Generator works fine with them.

Change-Id: Ieddd89c67048de50516f7ac7787d602660dc4a54
2023-09-26 14:04:33 +02:00
Ivan Fraixedes
7ab7ac49c8
private/apigen: Remove TypeScript errors from generated code
Remove the "duplicate identifier: 'path'" TypeScript errors from the
generated code.

Change-Id: I6b411a5cee720e2a16353034627954616c480f8a
2023-09-26 14:04:33 +02:00
Ivan Fraixedes
00484429d6
private/apigen: Generate valid TypeScript code with anonymous types
The API generator didn't generate valid TypeScript code when using
Go anonymous types.

This commit fixes that issue creating names for anonymous types.

Change-Id: Ice0748d8650686e3d3979523b8f218dc20eade5a
2023-09-26 14:04:33 +02:00
Ivan Fraixedes
a9901cc7d0
private/apigen: Add validations to improve usage
Add a few validations to panic with a nicer message or abort rather than
generating invalid code.

Also improve the panic message wrapping a standard error with errs2 at
the time that it's returned.

Change-Id: I1393933eb5f0bc3f86646bf4d0acfc64626efbe0
2023-09-26 14:04:33 +02:00
Ivan Fraixedes
822a13570e
private/apigen: Document Endpoint fields
Add document comments to tall the fields of the Endpoint struct.

Change-Id: Ibcd534f26474fc73fb378a9f30ce7d9401217557
2023-09-26 14:04:32 +02:00
Ivan Fraixedes
9a06de9058 private/apigen: Add test for Types.All
Add a test for Types.All method because we may need to adjust the logic
in future commits and we want to detect what has changed for good or
bad.

Change-Id: I1db4bf67db3c87513cb9aeb7b942c6c132fc4dd1
2023-09-25 21:15:06 +00:00
Ivan Fraixedes
926076bffd private/apigen/example: Add description to endpoint
Add the description field to the endpoint to show where it ends up when
generating code.

Change-Id: I415f088dbf795656ed1ef042ed41ebf39a517692
2023-09-19 18:31:20 +00:00
Andrew Harding
fd679c329c private/server: FreeBSD TCP fastopen support detection
Detects whether or not TCP fastopen is supported by running sysctl to
grab the current value of net.inet.tcp.fastopen.server_enable.

If not enabled, directs the operator to use sysctl to set it temporarily
and /etc/sysctl.conf to set it on-boot.

Setting the socket option always works whether it is enabled or not.

Verified that fastopen works on FreeBSD 13. Did not attempt on earlier
versions but support has been there since FreeBSD 10.

Change-Id: I2e0c457558a6fa7b7a1b18bc3c6684aff50b81a2
2023-09-18 21:17:42 +00:00
Andrew Harding
e4d6829971 private/server: Windows TCP fast-open support detection
Implements tryEnableFastOpen by creating a localhost socket and enabling
the TCP_FASTOPEN socket option. On builds where TCP_FASTOPEN isn't
available, setting the socket option fails and tryEnableFastOpen returns
false.

This was verified on a supporting build (latest Windows 10) and an
unsupporting build (Windows 8.1, couldn't find an ISO for an older
Windows 10 build).

Change-Id: I497117dc2f04acdd2b0cc836e20d12d69076b939
2023-09-18 14:15:30 +00:00
Márton Elek
e2006d821c satellite/overlay: change Reliable and KnownReliable
as GetParticipatingNodes and GetNodes, respectively.

We now want these functions to include offline and suspended nodes as
well, so that we can force immediate repair when pieces are out of
placement or in excluded countries. With that change, the old names no
longer made sense.

Change-Id: Icbcbad43dbde0ca8cbc80a4d17a896bb89b078b7
2023-09-02 23:34:50 +00:00
Moby von Briesen
b4c95a3b28 private/apigen/tsgen: Set query params better
The old way did not properly handle escaping, e.g. if the value of a
query param contained `&` or `=` inside it. By using
url.searchParams.set, we can safely add these types of arguments to the
path.

Change-Id: I62d3883b14f9d5a517e4a3d58f019014b46fd1b4
2023-08-31 20:09:29 -04:00
Egon Elbre
273ebd61d7 private/apigen/example: make it nicer
Change-Id: I1bd779090a902ed2b99b3993dc7cf61fc250f10f
2023-08-28 15:13:46 +00:00
Ivan Fraixedes
2d8f396eeb private/apigen: Make API base path configurable
Previously the base path for the API was hardcoded to `/api` and the
specified version.

This was not obvious that the generated code was setting that base path
and it was not flexible for serving the API under a different path than
`/api`.

We will likely need to set a different base path if we pretend to serve
the new back office API that we are going to implement alongside the
current admin API until the new back office is fully implemented and
verified that works properly.

This commit also fix add the base path of the endpoints to the
documentation because it was even more confusing for somebody that wants
to use the API having to find out them through looking to the generated
code.

Change-Id: I6efab6b6f3d295129d6f42f7fbba8c2dc19725f4
2023-08-28 14:35:01 +00:00
Ivan Fraixedes
d34c1b6825 private/apigen: Generate docs & typescript for example
Generate the documentation and the typescript code for the example of
that we have to generate an API through the API generator.

The JSON Go struct field tags are needed because the typescript
generator require them, otherwise it panics.

The test had to be adjusted according to match the tags so Go consider
the structs the same type, otherwise, it doesn't compile.

Change-Id: I3e4ebff9174885c50ca2058b86b7ec60e025945c
2023-08-25 12:30:21 +00:00
Moby von Briesen
d76f059c55 private/apigen: Update generated doc links
Make the link more human-friendly - to contain the text of the group and
endpoint names.

Also link back to list of endpoints from each endpoint.

Change-Id: Ia3e2ebe20b58b5f60ecefe9d35fb8fd90dd4b4d7
2023-08-25 11:51:57 +00:00
Moby von Briesen
7fbabbcf16 private/apigen: Add list of endpoints with links
This change makes it easier for someone reading the documentation to see
a full list of supported endpoints, and have direct links to the
details.

Change-Id: I46e2f809cfa2760845898eaa3d99db9066d435ef
2023-08-24 02:45:46 +00:00
Ivan Fraixedes
b0d072270b private/apigen: Add comment about panics
Add a comment in 2 methods about they panic as it's specified for the
`MustWriteDocs`.

Change-Id: I15d5fa420af37318d828bc633bed62988f8b207b
2023-08-23 18:50:37 +00:00
Egon Elbre
b5e1e5a9e2 private/apigen: use typescript in docs markdown
Fixes https://github.com/storj/storj/issues/6187

Change-Id: I34016513f29a9538343f0909951ab1a4605bb585
2023-08-17 15:33:21 +00:00
Egon Elbre
1f261bcc70 go.mod: bump lang to 1.19 and common
Change-Id: I8d91f97d786456da29ebe89a78412c50efbb8ccc
2023-08-10 18:41:15 +03:00
Wilfred Asomani
34e1caa55a satellite: add get user paged projects http endpoint
This change adds an endpoint to get a user's projects, similar to
the OwnedProjects GraphQL query.
The console.ProjectInfo struct has been renamed to UpsertProjectInfo
to more accurately reflect its use.

Issue: https://github.com/storj/storj/issues/6135

Change-Id: I802fe4694a5cc75a9df2b565476f6e6f473431d4
2023-08-08 14:02:53 +00:00
Wilfred Asomani
c934974652 satellite/{console,web}: update error handling
This change updates how API errors are handled and sent.

Change-Id: Ia4c71eeb6f2d009a47b59ce77a23f70b8b10f6dc
2023-07-27 21:01:01 +00:00
Clement Sam
4cb85186b2 storagenode/pieces: enable lazyfilewalker by default
Resolves https://github.com/storj/storj/issues/5861

Change-Id: I20e0a5b8a15ca966cbccd71369322a517a2c2130
2023-07-19 15:25:20 +00:00
Wilfred Asomani
4ee647a951 satellite: add request id to requests
This change adds request IDs to requests, logs them as part of audit
logs and sends to the client on error. This is to improve debugging
of customer issues.

Issue: https://github.com/storj/storj/issues/5898

Change-Id: I801514b547d28d810552d91aa7c8502051e552bf
2023-07-13 09:22:43 +00:00
Cameron
7e03ccfa46 satellite/console: optional separate web app server
This change creates the ability to run a server separate from the
console web server to serve the front end app. You can run it with
`satellite run ui`. Since there are now potentially two servers instead
of one, the UI server has the option to act as a reverse proxy to the
api server for local development by setting `--console.address` to the
console backend address and `--console.backend-reverse-proxy` to the
console backend's http url. Also, a feature flag has been implemented
on the api server to retain the ability to serve the front end app. It
is toggled with `--console.frontend-enable`.

github issue: https://github.com/storj/storj/issues/5843

Change-Id: I0d30451a20636e3184110dbe28c8a2a8a9505804
2023-07-11 12:17:35 -04:00
Michal Niewrzal
1f92e7acda satellite: move GC sender to Core peer
Having separate process/pod only for sending bloom filters once a week
is a bit waste. After reconfiguring production settings to use sender in
core we can remove also GC sender peer code.

https://github.com/storj/storj/issues/5979

Change-Id: I6efe3ec073f96545e1f70ad13843f8ccdf923ee8
2023-07-11 10:31:35 +00:00
Márton Elek
97a89c3476 satellite: switch to use nodefilters instead of old placement.AllowedCountry
placement.AllowedCountry is the old way to specify placement, with the new approach we can use a more generic (dynamic method), which can check full node information instead of just the country code.

The 90% of this patch is just search and replace:

 * we need to use NodeFilters instead of placement.AllowedCountry
 * which means, we need an initialized PlacementRules available everywhere
 * which means we need to configure the placement rules

The remaining 10% is the placement.go, where we introduced a new type of configuration (lightweight expression language) to define any kind of placement without code change.

Change-Id: Ie644b0b1840871b0e6bbcf80c6b50a947503d7df
2023-07-07 16:55:45 +00:00
Clement Sam
7ac2031cac web/multinode: fix wrong free disk space in allocation on dashboard
There are many case where the keywords `free` and `available`
are confused in their usage.

For most cases, `free` space is the amount of free space left
on the whole disk, and not just in allocation while
`available` space is the amount of free space left in the
allocated disk space.

What the user/sno wants to see is not the free space but the
available space. To the SNO, free space is the free space
left in the allocated disk space.

Because of this confusion, the multinode dashboard displays
the `free` disk space instead of the free space in the
allocated disk space https://github.com/storj/storj/issues/5248
While the storagenode dashboard shows the correct free space
in the allocation.

This change fixes the wrong free disk space. I also added a
few comments to make a distinction between the `free`
and `available` fields in the `DiskSpace*` structs.

Change-Id: I11b372ca53a5ac05dc3f79834c18f85ebec11855
2023-07-05 11:24:24 +00:00
Michal Niewrzal
f2cd7b0928 satellite/overlay: refactor Reliable to be used with repair checker
Currently we are using Reliable to get missing pieces for repair
checker. The issue is that now checker is looking at more things than
just missing pieces (clumped/off, placement pieces) and using only node
ID is not enough. We have issue where we are skipping offline nodes from
clumped and off placement pieces check.

Reliable was refactored to get data (e.g. country, lastNet) about all
reliable nodes. List is split into online and offline. This data will be
cached for quick use by repair checker. It will be also possible to
check nodes metadata like country code or lastNet.

We are also slowly moving `RepairExcludedCountryCodes` config from
overlay to repair which makes more sens for it.

This this first part of changes.

https://github.com/storj/storj/issues/5998

Change-Id: If534342488c0e440affc2894a8fbda6507b8959d
2023-07-05 10:56:31 +02:00
Egon Elbre
edbea5efe1 go.mod: bump to pgx/v5
Change-Id: I31cf3bec1d7db94f0f612f6ed04b782f8b04d876
2023-06-14 18:32:54 +03:00
Moby von Briesen
8acb1ee5bf private/apigen: Support basic doc generation
Add some code to generate a basic markdown file documenting a generated
API. Generate this document for the API in
satellite/console/consoleweb/consoleapi/gen.

The documentation is not completely correct, as it may include some
values in the request body that are not actually usable by the
requester. This can be fixed by making sure all types used within the
generated API are properly annotated with `json` tags.

Issue: https://github.com/storj/storj-private/issues/244

Change-Id: I57b259967fb0db8f548b6598a10c825da15ba723
2023-06-13 08:48:06 +00:00
JT Olio
3fff61f04a payments: don't redefine compensation rates twice
Change-Id: Ic00abe3795a000d4f0284c99f270180123a2f663
2023-06-10 12:40:43 +00:00
JT Olio
1966340b2a storagenode: report fastopen support
Change-Id: Ic707bce32729540391d5139e7f3a82c693bcb834
2023-06-05 15:20:13 +00:00
JT Olio
b502310fe5 cmd/inspector: remove
this is a very old tool built in the very early days
of v3, when we didn't know how the network would be
used. this tool anticipated being able to query remote
nodes for internal state. we don't do that. i don't
think anyone uses this.

Change-Id: Ie1ded3ecbedb09313f2d6fc721039e0f15e4ee85
2023-05-19 09:11:29 +00:00
Michal Niewrzal
c0e7f463fe satellite/metabase: remove segmentsloop package
Last change to remove segments loop from codebase.

https://github.com/storj/storj/issues/5237

Change-Id: I77b12911b6b4e390a7385e6e8057c7587e74b70a
2023-05-18 19:08:29 +00:00
paul cannon
75d10fe4fa satellite/overlay: use UploadSelectionCache for GetNodesNetworkInOrder
The query for GetNodesNetworkInOrder is causing far too much load on the
database. Since it is not critical that the repair checker have
perfectly up-to-date node network information, we can use a cache
instead.

Change-Id: I07ad45bfdeb46529da093941a06c2da8a00ce878
2023-05-16 17:32:09 +00:00
Michal Niewrzal
2592aaef9c satellite/gc/bloomfilter: remove segments loop parts
We are switching completely to ranged loop.

https://github.com/storj/storj/issues/5368

Change-Id: I1a22ac4b242998e287b2b7d8167b64e850b61a0f
2023-05-15 11:46:26 +00:00
Michal Niewrzal
36e046375c satellite/repair/checker: remove segments loop parts
We are switching completely to ranged loop.

https://github.com/storj/storj/issues/5368

Change-Id: I8583549973cd36aa0e0c482c20d7a75cb7568ab3
2023-05-08 12:19:13 +00:00
Michal Niewrzal
1aa24b9f0d satellite/audit: remove segments loop parts
We are switching completely to ranged loop.

https://github.com/storj/storj/issues/5368

Change-Id: I9cec0ac454f40f19d52c078a8b1870c4d192bd7a
2023-04-24 15:52:11 +00:00
Michal Niewrzal
6ac5bf0d7c satellite/gracefulexit: remove segments loop parts
We are switching completely to ranged loop.

https://github.com/storj/storj/issues/5368

Change-Id: Ia3e2d7879d91f7f5ffa99b8e8f108380e3b39f31
2023-04-24 15:00:26 +00:00
Michal Niewrzal
6a55682bc6 satellite/accounting/nodetally: remove segments loop parts
We are switching completely to ranged loop.

https://github.com/storj/storj/issues/5368

Change-Id: I6176a129ba14cf83fb635048d09e6748276b52a1
2023-04-24 14:25:53 +00:00
Michal Niewrzal
fbfe5aaad7 satellite/metrics: remove code related to segments loop
We are switching completely to ranged loop.

Change-Id: I32120ef496addebec2de088fd10d0c1d02313c68
2023-04-20 13:47:22 +00:00