Commit Graph

8114 Commits

Author SHA1 Message Date
Wilfred Asomani
74b3617813 Revert "satellite/db: fix long loadtime for charges endpoint"
This reverts commit 676178299f.

Reason for revert:
The new query used by this commit performs a full table scan.
It's been reverted pending a fix for that.

Change-Id: Idc53954459aa6f5a692056232b8674b11d1928ce
2023-06-23 09:32:33 +00:00
Paul Willoughby
3180e09750 satellite/metainfo: increase default MaxEncryptedObjectKeyLength
Allow a longer encrypted key length to reduce 'key length is too big'
errors in gateway-mt.  Gateway is enforcing an unencrypted key length
of 1024 bytes but when encrypted some keys are exceeding the current
limit.

Updates https://github.com/storj/gateway-mt/issues/335

Change-Id: Ib02e2064c42e96b9d59936905832d8dd6068d2c7
2023-06-22 22:59:52 +00:00
Cameron
37e7eeb0e9 satellite/payments/accountfreeze: set grace period default to 15 days
Change-Id: Ied8f3758b579b83ebf04cba0fde9715c689bac4f
2023-06-22 18:53:03 +00:00
Clement Sam
1166fdfbab satellite/gc: add piece tracker ranged loop observer
Resolves https://github.com/storj/storj/issues/5798

Change-Id: I6fe2c57b3a247b085026feb8bee60c2d002db71b
2023-06-22 18:17:39 +00:00
Michal Niewrzal
2b2bca8e81 satellite/accounting/tally: save tallies in a batches
Because we are saving all tallies as a single SQL statement we finally
reached maximum message size. With this change we will call SaveTallies multiple times in batches.

https://github.com/storj/storj/issues/5977

Change-Id: I0c7dd27779b1743ede66448fb891e65c361aa3b0
2023-06-22 17:02:26 +00:00
paul cannon
355ea2133b satellite/audit: remove pieces when audits fail
When pieces fail an audit (hard fail, meaning the node acknowledged it
did not have the piece or the piece was corrupted), we will now remove
those pieces from the segment.

Previously, we did not do this, and some node operators were seeing the
same missing piece audited over and over again and losing reputation
every time.

This change will include both verification and reverification audits. It
will also apply to pieces found to be bad during repair, if
repair-to-reputation reporting is enabled.

Change-Id: I0ca7af7e3fecdc0aebbd34fee4be3a0eab53f4f7
2023-06-22 14:19:00 +00:00
Michal Niewrzal
203c6be25f satellite/repair/repairer: test repairing geofenced segment
Additional test case to cover situation where we are trying to
repair segment with specific placement set. We need to be sure
that segment won't be repaired into nodes that are outside
segment placement, even if that means that repair will fail.

Change-Id: I99d238aa9d9b9606eaf89cd1cf587a2585faee91
2023-06-22 13:21:05 +00:00
Michal Niewrzal
9e3fd4d514 satellite/overlay: delete unused method
Change-Id: I87828fcac4f4a9fb08c86af188aa6ea28c5c64af
2023-06-22 12:45:59 +00:00
Sean Harvey
ad9b0dfc40 satellite/admin: fix paid tier field on /api/apikeys/{apikey} endpoint
the field was not being set, so it would always show false in the
response whether the user was paid tier or not.

Updates storj/gateway-mt#321

Change-Id: I00ae751ac9118fc05b3c22bc6d2cb9600d0faaaf
2023-06-22 11:12:56 +00:00
Jeremy Wharton
25c21f03c3 web/satellite: add project role for expired invitations
Pending project members whose invitations have expired now appear with
the "Invite Expired" role in the Team page.

References #5752

Change-Id: Ic91289618ee02e65de29e986fa3205eccf39b267
2023-06-22 06:27:37 +00:00
Vitalii
adbd4fbab2 web/satellite: applied styling updates for gallery view
Add "geographic distribution" item into three-dot menu.
On small screens, instead of showing three-dot menu + geographic distribution, show three-dot menu + download.
Allow the user to use left and right arrow keys to navigate through previews in gallery view.
Update "Do not share download link with other people" notification according to new designs.
Add hover styles and tooltips to icons according to designs.
In the "View Details" modal, change "Saved in" to "Bucket".
(not related to gallery view) - the three-dot-menu in the file list has a border radius, but when you hover over the last component ("Delete"), the border radius goes away.

Issue:
https://github.com/storj/storj/issues/5968

Change-Id: I39bec2e630327d136cb2550dbbce4fcbf77399f9
2023-06-21 21:25:48 +00:00
Vitalii
1712e69f0c web/satellite: add data dimension label to charts on project dashboard
Added data dimension label to chart y-axis on project dashboard

Issue:
https://github.com/storj/storj/issues/5926

Change-Id: I03a3a12211b1a20f76e3a8291f349675956ff598
2023-06-21 20:30:02 +00:00
Lizzy Thomson
05f92fed11 web/satellite: show full access credentials when not hidden
Show full AG, API key or S3 credentials when not hidden

Issue https://github.com/storj/storj/issues/5848

Change-Id: I9e0903ed34b6f0068e9cef7f048553441ed98fc3
2023-06-21 19:41:26 +00:00
Vitalii
e06b94dcbc web/satellite: update project settings link styling
Updated link styling according to designs.

Issue:
https://github.com/storj/storj/issues/5933

Change-Id: Iea352272db4da315b12ca0d7c5076ce32ba49983
2023-06-21 18:04:02 +00:00
Wilfred Asomani
2caa5052ad satellite/console: add endpoint to get invite link
This change adds an endpoint that returns the invite link for invited
users.

Related to: #5762

Change-Id: I6432dfbe6405222b949fa02020aee2e01ab59c98
2023-06-21 16:08:50 +00:00
Michal Niewrzal
ac1ff0e7e2 satellite/accounting/tally: handle well bucket names with escape char
Some of tally queries are not passing bucket name as byte but as string.
If bucket contains some special characters encoding to bytea can fail.
This change makes sure all parts of tally passes bucket name correctly.

Change-Id: I7330d546b44d86a2e4614c814580e9e5262370ed
2023-06-21 15:04:14 +00:00
Jeremy Wharton
28b2384970 satellite/console/.../consoleql: add project invite expiration status
Each project member invitation returned from our GraphQL API now
contains a field indicating whether the invitation has expired. This is
required for us to enable functionality in the satellite frontend that
is dependent on this information.

References #5752

Change-Id: I4b71738e7a7373c690de188614f8c95009bc3989
2023-06-20 22:51:42 -05:00
Jeremy Wharton
d18f4f7d99 satellite/console: don't delete expired project invitations
This change removes instances of project invitation deletion due to
expiration because we now want such invitations to be accessible beyond
their expiration date. In the future, project members will be able to
view and resend expired invitations within the Team page in the
satellite frontend.

References #5752

Change-Id: If24a9637945874d719b894a66c06f6e0e9805dfa
2023-06-20 22:42:50 -05:00
Clement Sam
b6026b9ff3 storagenode/piecestore: fix ingress graph skewed by larger signed orders
Storagenodes are currently getting larger signed orders due to
a performance optimization in uplink, which now messes with the
ingress graph because the storagenode plots the graph using
the order amount instead of actually uploaded bytes, which this
change fixes.

The egress graph might have a similar issue if the order amount
is larger than the actually downloaded bytes but since we pay
for orders, whether fulfilled or unfulfilled, we continue using
the order amount for the egress graph.

Resolves https://github.com/storj/storj/issues/5853

Change-Id: I2af7ee3ff249801ce07714bba055370ebd597c6e
2023-06-20 13:28:08 +00:00
Michal Niewrzal
bbdeb1eeb8 satellite/gc/bloomfilter: enable sync observer by default
After deploying to most of production satellites we want to enable
it by default. No issues found after few weeks of running.

Change-Id: I4d83c65edaa95b0e50eeab6ac5e2c6cbb7206c1e
2023-06-20 10:42:05 +00:00
Vitalii
eb636f5b57 satellite/admin: add UI functionality to update user's or project's user agent
Implemented UI functionality to allow prod owner to update user's or project's user agent.

Issue:
https://github.com/storj/storj-private/issues/297

Change-Id: Ibe47e735f3c7a9fdf05b70735ebcecbac5c94ce8
2023-06-16 20:14:01 +00:00
Vitalii
1eee2fad69 satellite/{db, admin}: added endpoints to update user's and project's user_agent
Added backend (for now) implementation for updating user's and projects's user_agent using admin API.
Updating both user and project also updates bucket_metainfo and value_attribution tables.

Issue:
https://github.com/storj/storj-private/issues/297

Change-Id: I40244bbaa08b46834c1b1d0720e7d84d0c2a0330
2023-06-16 19:41:05 +00:00
Vitalii
fe0c3a13b4 web/satellite: updated styling of Upload hover component
Added box shadow and added icons for each object type.

Issue:
https://github.com/storj/storj/issues/5974

Change-Id: I5dd7c7cd5e0066d4da75f655feb162abad7bec3b
2023-06-16 17:19:15 +00:00
Michal Niewrzal
cb9a7bdc71 satellite/repair/repairer: make DialTimeout configurable
This change makes dial timeout configurable and change it also from
defatul 20s to 5s. Main motivation is that during repair we often loose
lots of time to dial which eventually will fail. New timeout should be
still enough to dial but we will move forward quicker to next node if
that one will fail.

Timeout is also applied directly as context timeout in case we will
use noise of tcp fast open one day.

Change-Id: I021bf459af49b11241e314fa1a7887c81d5214ea
2023-06-16 12:23:25 +00:00
Wilfred Asomani
dbd575e50b satellite/{web,console}: verify invite link
This change adds a new endpoint to verify the validity of an invite
link. It conditionally redirects to the login page with or without
whether the invite is valid.

Related: https://github.com/storj/storj/issues/5741

Change-Id: I587ef8ded67a9ea753e4edec1beeecd39c949922
2023-06-16 09:23:46 +00:00
Moby von Briesen
6268c75d3f satellite/console: Enable all projects dashboard by default
Now that the table view has been implemented, the all projects dashboard
is ready to be turned on everywhere.

https://github.com/storj/storj/issues/5872

Change-Id: Iead684bf7d326d36d4d323eb63a3ed520602b4dc
2023-06-16 00:33:21 +00:00
Wilfred Asomani
676178299f satellite/db: fix long loadtime for charges endpoint
This change addresses an issue where the /charges endpoint will take a
while to respond due to a project having a large number of buckets.
The queries responsible for this have been merged into a single query to
get all data needed at a go and potentially improve performance.
Benchmarks indicate that this new way is more performant than the
previous.

name                                 old ms/op  new ms/op
Postgres/sum_all_partner_usages          7.534      0.622
Postgres/individual_partner_usages       6.104      0.588
Cockroach/sum_all_partner_usages        14.813      3.057
Cockroach/individual_partner_usages     16.046      2.852

Issue: https://github.com/storj/storj-private/issues/277

Change-Id: Ibb7f867ab6610b3cb1ba203961f7d6aef6bfda4a
2023-06-15 14:49:30 +00:00
Vitalii
2a4c1fa41e web/satellite: add delimiters to large numbers on project dashboard
Added delimiters to large numbers on project dashboard based on user location.
So in US and EU there should be different strings.

See: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Number/toLocaleString

Issue:
https://github.com/storj/storj/issues/5928

Change-Id: I94a0a703c2b301de422c5103e71becfbf5aa7e79
2023-06-15 12:49:01 +00:00
Moby von Briesen
3bc300ef55 satellite/console: optionally serve vuetify POC from satellite
This change slightly modifies the logic for serving the vuetify
frontend. After this change, if the config `console.use-vuetify-project`
is set to `true`, the Vuetify UI will be served at the `/vuetifypoc`
prefix. The POC is only project dashboard right now; the existing
login/registration pages must be used, but once the cookie is set, the
POC will work correctly.

Change-Id: I7725f23a0d2b04f274bab36d8be3370116687d1b
2023-06-15 12:16:31 +00:00
Egon Elbre
edbea5efe1 go.mod: bump to pgx/v5
Change-Id: I31cf3bec1d7db94f0f612f6ed04b782f8b04d876
2023-06-14 18:32:54 +03:00
Márton Elek
981dd4cbdc go.mod: bump dependencies (storj.io/private, storj.io/uplink)
Change-Id: Ie5a6a14a39786dd745116c3b6e4fdbc8e890d2c3
2023-06-14 16:36:45 +03:00
Michal Niewrzal
2e0b687581 satellite/metainfo: drop piecedeletion package
We are not using this package anymore.

Change-Id: If32315d43d73c8deb096e93cb43c03881bd9aad1
2023-06-14 11:19:21 +00:00
Michal Niewrzal
6c08d5024e satellite/metainfo: stop sending delete requests to SN
We decided that we will stop sending explicit delete requests to nodes
and we will cleanup deleted with GC instead.

https://github.com/storj/storj/issues/5888

Change-Id: I65a308cca6fb17e97e3ba85eb3212584c96a32cd
2023-06-14 10:46:02 +00:00
Wilfred Asomani
4a49bc4b65 satellite/payments: fix account freeze chore race condition
This change fixes an issue where a formerly warned/frozen user will be
warned again even though they have made payment for the invoice that got
them frozen in the first place. A payment status check is now made
right before a warn/freeze event to make sure the invoice hasn't been
paid already.

Issue: https://github.com/storj/storj/issues/5931

Change-Id: I3f6ac1e224f40107d58dc8f7bdbce58bbbea0196
2023-06-14 10:10:56 +00:00
Jeremy Wharton
6b65b7e7d0 satellite/{console,satellitedb}: clean up obsolete project member code
This change removes the obsolete project member paging code.
Previously, we implemented functionality for including project
invitations in pages of project members. However, the satellite
frontend still expected API responses to use the old paging style, so
the related code could not be removed right away. Now that the frontend
has been updated, this code is no longer necessary.

References #5855

Change-Id: I12fdaaeb869977c4d87a0d50b9a7b11c68552c82
2023-06-13 22:13:04 +00:00
Jeremy Wharton
8ee7d104a0 web/satellite: show project invitations in the Team page
The Team page now displays project member invitations alongside project
members. These invitations can be removed in the same manner that
members can.

References #5855

Change-Id: Iade690757d4430deee3378066d7dc19766f53936
2023-06-13 19:00:33 +00:00
Jeremy Wharton
ed37d72e3f web/satellite: change order of projects and invitations in dashboard
This change places project invitations ahead of projects in the cards
view of the All Projects Dashboard in order to be consistent with our
designs.

References #5855

Change-Id: I31268b97a9af2279bf90a171d0958b5a4bcbf13b
2023-06-13 11:34:55 -05:00
Wilfred Asomani
62c29ee9de web/satellite: use new invite functionality
This change uses the new project invite endpoint in place of the former
that adds invited users directly to a project's members. LoginArea
is updated to make the region/email from an invite email link uneditable.
VInput.vue's composition api code has also been updated to match other
components.

Issue: https://github.com/storj/storj/issues/5741

Change-Id: Ia3f82f5675fba442bb079dc8659b5396a25b9f03
2023-06-13 16:00:39 +00:00
Wilfred Asomani
09a7d23003 satellite/console: add endpoint to invite users to project
This change adds a new endpoint that uses the new project invite flow's
 functionality instead of directly adding users to a project's members.

Issue: https://github.com/storj/storj/issues/5741

Change-Id: I6734f7e95be07086387fb133d6bdfd95e47cf4d9
2023-06-13 15:22:25 +00:00
Vitalii
782811c634 web/satellite: check if AG name is free to use at the beginning of creation flow
By this change we check if provided access grant name is free to use at the very beginning of the creation flow.

Issue:
https://github.com/storj/storj/issues/5693

Change-Id: I06583bf458cea977cb0a920d55df50f2d19e1599
2023-06-13 14:24:16 +00:00
Vitalii
cc085553c2 satellite/{console, db}: endpoint to get all API key names by project ID
Added new endpoint, service method and DB query to get all API key names by provided project ID.

Issue:
https://github.com/storj/storj/issues/5693

Change-Id: I62e4e8ae660bd81234b75aa159a472a5aa9d5a48
2023-06-13 13:52:30 +00:00
Egon Elbre
00d233bca5 satellite/satellitedb: remove duplicate NullUUID
Change-Id: I0fa583140ab14448c26c602838aa035d8372b965
2023-06-13 15:34:41 +03:00
Jeremy Wharton
d39424993b satellite/console: only return unexpired project invitations to user
This change causes users to be served only project member invitations
that have not expired. If expired invitations are encountered during
processing of an invitation listing request, they will be removed.

References #5855

Change-Id: I6f621305f4f0a993953eb40a4dbd2375493f02e3
2023-06-13 09:42:03 +00:00
Moby von Briesen
8acb1ee5bf private/apigen: Support basic doc generation
Add some code to generate a basic markdown file documenting a generated
API. Generate this document for the API in
satellite/console/consoleweb/consoleapi/gen.

The documentation is not completely correct, as it may include some
values in the request body that are not actually usable by the
requester. This can be fixed by making sure all types used within the
generated API are properly annotated with `json` tags.

Issue: https://github.com/storj/storj-private/issues/244

Change-Id: I57b259967fb0db8f548b6598a10c825da15ba723
2023-06-13 08:48:06 +00:00
Jeremy Wharton
5b1c22a1e7 satellite/console/consoleweb/consoleql: include invites in member pages
Project member invitations may now be requested through GraphQL
queries. This is necessary for the satellite frontend to display
invitations in the Team page.

References #5855

Change-Id: Ibc8526ba768fd82c1b1890201004ef0f066df2fc
2023-06-13 07:48:07 +00:00
Jeremy Wharton
0cbb0ee12e satellite/console: delete project invitations with project members
The console service method responsible for removing members from a
project has been extended to remove project member invitations as well.
This will allow invitations to be deleted through the satellite
frontend.

References #5855

Change-Id: I90ca042cc6fb9a75fcd9b391e317caabb1c828f2
2023-06-13 07:15:43 +00:00
Jeremy Wharton
2ab6e00966 satellite/satellitedb: add method to return project members and invites
A method has been implemented that allows for paged searching through
project members and project member invitations. In the future, this
will be used to display invitations in the Team page of the satellite
frontend.

References #5855

Change-Id: I0937c425f60f1318e55202bf30b44a33ff695414
2023-06-12 17:28:50 -05:00
Wilfred Asomani
3d1e429156 web/satellite: properly style all projects dashboard
This change matches the all projects dashboard styling to the figma as
closely as currently possible.

Issue: https://github.com/storj/storj/issues/5934

Change-Id: I93edd898fbcee954265737052a1967d318370924
2023-06-12 18:38:30 +00:00
Wilfred Asomani
8cdc5bd107 web/satellite: add invites to all projects dashboard table
This change adds projects invites to the projects table.

Related: https://github.com/storj/storj/issues/5925

Change-Id: I82fbf1e47ea383377369005956a9eec3c8ac6416
2023-06-12 18:07:37 +00:00
Moby von Briesen
771ec3237a satellite/console: Update CSP to include storjsatelliteshare.io
Change-Id: I8238f6e2644a0f4ee727d0b8da56e59733dd3252
2023-06-12 13:31:00 -04:00