Commit Graph

8211 Commits

Author SHA1 Message Date
Jeremy Wharton
df9a6e968e web/satellite: lint Vuetify files
This change enables linting for the Vuetify proof of concept code and
fixes the linting errors that were detected. Additionally, it migrates
the Vuetify components to the composition API.

Change-Id: Id8cc083954e3f4cb66a00ad2715a96c8747b592c
2023-07-17 20:32:59 +00:00
dlamarmorgan
abe1463a73 payments/stripe/invoices: add token payment to overdue invoice payment
Add an attempt to pay overdue invoices via storj token if the user has
a token balance.

Change-Id: I819b89e7cf9cdb7deb9a51eab5ca684b54418218
2023-07-17 12:59:33 -07:00
dlamarmorgan
c96c83e805 satellite/payments/stripe/service: add manual payment with token command
Add the ability to pay an individual users open invoices using their
storj token balance.

Change-Id: I6115f2b033fd77f109ded6f55b1f35fc77c71ff1
2023-07-17 19:24:36 +00:00
dlamarmorgan
0f4371e84c scripts/tests/{backwardcompatibility,integrations}: add test scripts
Change-Id: Ib83cd0f083bab7f560a200fd95e62e5b21e60c27
2023-07-17 18:39:18 +00:00
Wilfred Asomani
0a8115b149 satellite/{console,payments}: fix handling for autofreeze flow
This change adds an extra step to the auto freeze chore to attempt
payment before freezing/warning a user.
It also attempts payment after modifying user's cards whether the user
is frozen/warned or not.

Issue: https://github.com/storj/storj-private/issues/341

Change-Id: Ia9c0c5a2d37837bca5153fe720fef61f1385cb15
2023-07-17 17:37:11 +00:00
Michal Niewrzal
47a4d4986d satellite/repair: enable declumping by default
This feature flag was disabled by default to test it slowly. Its enabled
for some time on one production satellite and test satellites without
any issue. We can enable it by default in code.

Change-Id: If9c36895bbbea12bd4aefa30cb4df912e1729e4c
2023-07-17 15:02:35 +00:00
Michal Niewrzal
5272fd8497 satellite/metainfo: do full bucket validation only on create
We are doing full bucket name validation for many requests but
we should do this only while creating bucket. Other requests will be
covered only by basic name length validation. Less strict validation for
other requests will make bucket usable in case of invalid bucket names
in DB (we have such cases from the past).

https://github.com/storj/storj/issues/6044

Change-Id: I3a41050e3637787f788705ef15b5dc4df4d01fc6
2023-07-17 16:15:33 +02:00
Jeremy Wharton
95761908b5 web/satellite: update Vuetify proof of concept
The changes as of storj/vuetify-storj@c801fe6 have been pulled into the
Vuetify proof of concept.

Change-Id: I3db208836cff21287052615d36258fcf2d4c6169
2023-07-14 12:46:50 +00:00
Michal Niewrzal
5234727886 satellite/repair/repairer: fix flaky TestSegmentRepairPlacement
Sometimes DownloadSelectionCache doesn't keep up with all node
placement changes we are doing during this test.

Change-Id: Idbda6511e3324b560cee3be85f980bf8d5b9b7ef
2023-07-14 10:10:40 +00:00
Tomasz Melcer
5a1c3f7f19
storage/reputation: logging changes to node scores (#5877)
Useful for monitoring storage nodes using log-parsing tools, like swatchdog.

Co-authored-by: Clement Sam <clementsam75@gmail.com>
2023-07-13 17:03:18 +02:00
Wilfred Asomani
4ee647a951 satellite: add request id to requests
This change adds request IDs to requests, logs them as part of audit
logs and sends to the client on error. This is to improve debugging
of customer issues.

Issue: https://github.com/storj/storj/issues/5898

Change-Id: I801514b547d28d810552d91aa7c8502051e552bf
2023-07-13 09:22:43 +00:00
Cameron
e8fcdc10a4 satellite/metainfo: set user_agent in bucket_metainfos on bucket recreation
Before this change, if a user creates a bucket with a user_agent attributed then deletes and recreates it, the row in bucket_metainfos
will not have the user_agent. This is because we skip setting the field
in bucket_metainfos if the bucket already exists in value_attributions.
This can be problematic, as we return the bucket's user agent during the
ListBuckets operation, and the client may be expecting this value to be
populated.

This change ensures the bucket table user_agent is set when (re)creating a bucket. To avoid decreasing BeginObject performance, which also
updates attribution, a flag has been added to determine whether to
make sure the buckets table is updated: `forceBucketUpdate`.

Change-Id: Iada2f233b327b292ad9f98c73ea76a1b0113c926
2023-07-12 21:48:05 +00:00
Michal Niewrzal
99128ab551 satellite/metabase: reuse Pieces while looping segments
Segments loop implementation is using lots of memory to convert
alias pieces to pieces for each segment while iteration. To improve
situation this change is reusing Pieces between batch pages. This
should signifcantly reduce memory usage for ranged loop executions.

Change-Id: I469188779908facb19ad85c6bb7bc3657111cc9a
2023-07-12 09:29:34 +00:00
Jeremy Wharton
062ca285a0 web/satellite: add sharing option to dropdown in buckets page
This change adds a sharing option to the dropdown menu for bucket rows
in the Buckets page.

Resolves #5964

Change-Id: Ife0eb8f6cabbe85eaedae1d94d97694f3c677a3e
2023-07-11 22:03:42 +00:00
Egon Elbre
465941b345 satellite/{nodeselection,overlay}: use location.Set
location.Set is faster for comparisons.

Updates #6028

Change-Id: I764eb5cafc507f908e4168b16a7994cc7721ce4d
2023-07-11 17:16:30 +00:00
Cameron
7e03ccfa46 satellite/console: optional separate web app server
This change creates the ability to run a server separate from the
console web server to serve the front end app. You can run it with
`satellite run ui`. Since there are now potentially two servers instead
of one, the UI server has the option to act as a reverse proxy to the
api server for local development by setting `--console.address` to the
console backend address and `--console.backend-reverse-proxy` to the
console backend's http url. Also, a feature flag has been implemented
on the api server to retain the ability to serve the front end app. It
is toggled with `--console.frontend-enable`.

github issue: https://github.com/storj/storj/issues/5843

Change-Id: I0d30451a20636e3184110dbe28c8a2a8a9505804
2023-07-11 12:17:35 -04:00
Egon Elbre
9370bc4580 satellite/{nodeselection,overlay}: bump common and fix some potential issues
* Handle failed country code conversion.
* Avoid potential issues with a data-race due to shared slice.

Updates #6028

Change-Id: If7beef2619abd084e1f4109de2d323f834a6090a
2023-07-11 11:13:41 +00:00
Michal Niewrzal
1f92e7acda satellite: move GC sender to Core peer
Having separate process/pod only for sending bloom filters once a week
is a bit waste. After reconfiguring production settings to use sender in
core we can remove also GC sender peer code.

https://github.com/storj/storj/issues/5979

Change-Id: I6efe3ec073f96545e1f70ad13843f8ccdf923ee8
2023-07-11 10:31:35 +00:00
Vitalii
a9d979e4d7 web/satellite: update default multipart upload part size
Updated multipart upload part size to be 64MB or higher depending on file size.
Increased queue size from 4 to 5 (5 parts being uploaded at a time) because theoretically it can decrease uploading time, right?

Issue:
https://github.com/storj/storj/issues/5851

Change-Id: Ida5661fa0ed6bc5a0651afc05b5feb7e77791efe
2023-07-10 20:07:51 +03:00
Moby von Briesen
4108aa72ba satellite/console,web/satellite: Fix project limit checking
* Fixes backend to use only a user's owned projects to determine if the
  user has hit the project limit
* Makes frontend logic consistent (and simpler) for checking whether to
  send user to the "Create Project" modal or the "upgrade account or
  request limit increase" modal

Before this change, projects that a user is a member of would be
included in determining whether the user could create a project. Also,
the "create project" button in the projects menu in the navbar of the UI
did not enable a free tier user to create a new project, even if they
had not hit their limits.

Change-Id: Ia776eb627ca37b83f5bc63bed83ee83c9f7cc789
2023-07-10 15:51:00 +00:00
Jeremy Wharton
4e876fbdba web/satellite: update upload modal
The upload modal has been updated to more closely match our designs.
- A button has been added to cancel all in-progress uploads.
- The status text has been fixed to display the proper file count.
- Clicking completed items in the upload modal previews them.

Resolves #5973

Change-Id: Iaee5fe05be14b3a6f2de1a9c807eca5137c7d643
2023-07-10 15:12:31 +00:00
Moby von Briesen
bd4d57c604 satellite/payments: Exclude users who pay via storjscan from autofreeze
Add a configuration (default true) to exclude users who have made
storjscan payments from being auto-warned/frozen for an unpaid invoice.
This will allow us to reach out to these users and handle warning/freezing
manually. Auto account freeze still handles CC-only users.

Fixes https://github.com/storj/storj/issues/6027

Change-Id: I0c862785dad1c8febfa11100c0d30e621ce3ae9b
2023-07-10 13:39:01 +00:00
Jeremy Wharton
c79d1b0d2f {satellite/console,web/satellite}: show error for project invite dupes
This change fixes an issue where a new project member invitation would
silently replace an older one that has the same project ID and email if
the email did not belong to a registered user. Additionally, the
satellite frontend has been updated to display more descriptive error
messages for project member invitations.

Change-Id: I32b582c40c0028b8eedf2aed4b5bfb43501594b4
2023-07-10 12:56:02 +00:00
Jeremy Wharton
fbda13c752 {satellite/console,web/satellite}: trim emails when inviting members
This change trims whitespace from email addresses in project member
invitation requests.

Change-Id: Idd9116820897bf29f3eeba8cf95770b1aa14690c
2023-07-10 12:22:07 +00:00
Jeremy Wharton
0f9a0ba9cd web/satellite: fix project switch when removing project members
This change fixes an issue where the currently selected project would
be switched when removing project members.

Change-Id: I8138b4229eb7933d25a2fe84e5aa0b5846fc79b8
2023-07-10 11:46:49 +00:00
JT Olio
73d65fce9a cmd/satellite/billing: don't fail the overall process if an individual invoice fails
Change-Id: I36591a717ef97bdb417cc6d9218e22b2f91f249b
2023-07-10 11:13:23 +00:00
Michal Niewrzal
1d62dc63f5 satellite/repair/repairer: fix NumHealthyInExcludedCountries calculation
Currently, we have issue were while counting unhealthy pieces we are
counting twice piece which is in excluded country and is outside segment
placement. This can cause unnecessary repair.

This change is also doing another step to move RepairExcludedCountryCodes
from overlay config into repair package.

Change-Id: I3692f6e0ddb9982af925db42be23d644aec1963f
2023-07-10 12:01:19 +02:00
Igor
05f30740f5
docs/testplan: add project cowbell testplan (#6001) 2023-07-10 11:23:55 +02:00
Márton Elek
97a89c3476 satellite: switch to use nodefilters instead of old placement.AllowedCountry
placement.AllowedCountry is the old way to specify placement, with the new approach we can use a more generic (dynamic method), which can check full node information instead of just the country code.

The 90% of this patch is just search and replace:

 * we need to use NodeFilters instead of placement.AllowedCountry
 * which means, we need an initialized PlacementRules available everywhere
 * which means we need to configure the placement rules

The remaining 10% is the placement.go, where we introduced a new type of configuration (lightweight expression language) to define any kind of placement without code change.

Change-Id: Ie644b0b1840871b0e6bbcf80c6b50a947503d7df
2023-07-07 16:55:45 +00:00
Wilfred Asomani
e0b5476e78 web/satellite: fix long table content overflow
This change fixes an issue where long text content in the common table
will overflow and breaks layout.

Change-Id: I30f6e08488410359e0a97995f8d769b272b56c72
2023-07-07 12:24:37 +00:00
Jeremy Wharton
074457fa4e web/satellite: add folder sharing
This change allows users to generate links for sharing folders.
Previously, users were only able to do this with files and buckets.

Resolves #5644

Change-Id: I16dd8270337e3561b6bda895b46d3cc9be5f8041
2023-07-07 10:03:06 +00:00
Vitalii
5fc6eaab17 satellite/{console, web}: display accurate legacy free tier information in upgrade modal
Updated upgrade account modal to show user account free tier limits instead of hardcoded values.

Issue:
https://github.com/storj/storj/issues/5939

Change-Id: I26ffbe2571c5ca4b37f02bec5211bac986bedc6a
2023-07-07 09:23:36 +00:00
Márton Elek
70cdca5d3c
satellite: move satellite/nodeselection/uploadselection => satellite/nodeselection
All the files in uploadselection are (in fact) related to generic node selection, and used not only for upload,
but for download, repair, etc...

Change-Id: Ie4098318a6f8f0bbf672d432761e87047d3762ab
2023-07-07 10:32:03 +02:00
Márton Elek
8b4387a498 satellite/satellitedb: add tag information to nodes selected for upload/downloads
Change-Id: I0fa7daebcf83f7949726e5fffe68e0bdc6fd1d7a
2023-07-07 07:54:16 +00:00
Vitalii
ced8657caa web/satellite: removed unused images
Change-Id: Ifd9c691c69f50a8f346ac4ac2fa1433b00ea81b9
2023-07-06 20:31:24 +00:00
Vitalii
ece0cc5785 web/satellite: fix bottom spacing for all pages
Fixed bottom spacing for all pages. Basically removed bottom padding override in dashboard wrapper.
Removed a couple of unused components and icons.
Made pagination size changer dropdown to appear on top of selector to not extend page height.

This change fixes pagination issue on Team page:
https://github.com/storj/storj/issues/6012

Change-Id: I707dd1bf9d61b05742b7f97a757a2a8f5f9f93fd
2023-07-06 19:57:43 +00:00
JT Olio
a85c080509 docs/blueprints: certified nodes
Change-Id: I7c670d740e4c3d1035dee145ed65aaed4cbaba0c
2023-07-06 14:07:46 -04:00
paul cannon
a4d68b9b7e satellite/metabase: server-side copy copies metadata
..instead of using segment_copies and ancestor_stream_id, etc.

This bypasses reference counting entirely, depending on our mark+sweep+
bloomfilter garbage collection strategy to get rid of pieces once they
are no longer part of a segment.

This is only safe to do after we have stopped passing delete requests on
to storage nodes.

Refs: https://github.com/storj/storj/issues/5889
Change-Id: I37bdcffaa752f84fd85045235d6875b3526b5ecc
2023-07-06 14:40:59 +00:00
Márton Elek
ddf1f1c340 satellite/{nodeselection,overlay}: NodeFilters for dynamic placement implementations
Change-Id: Ica3a7b535fa6736cd8fb12066e615b70e1fa65d6
2023-07-06 12:08:01 +00:00
Vitalii
e3d2f09988 web/satellite: add support link to upgrade account STORJ token flow
Added support link to upgrade account STORJ token flow to tell user that they have to fill in another form to upgrade with tokens only.

Issue:
https://github.com/storj/storj/issues/5985

Change-Id: Ifb852b883c6bf092d5eec588e823925a8ea661c9
2023-07-06 08:38:18 +00:00
Ethan Adams
f819b6a210 satellite/entrypoint: Ignore unset variable errors while checking for VALID_EXECUTABLE
Otherwise core and ranged loop fail at startup with "/entrypoint: line 94: $1: unbound variable"

Change-Id: I45a318038cd937c11f6a00d506c339ba69ea07bf
2023-07-05 20:18:38 +00:00
Márton Elek
1525324384 satellite/uploadselection: avoid String conversation of location during node selection
Converting location to String is not free, better to avoid it.

81cb588c23/storj/location/countrycode.go (L32)

Thanks to Egon, who reported this issue.

See also: https://review.dev.storj.io/c/storj/common/+/10732

Change-Id: Ife348cffa59c020b46914a68be231c6eb75f06c9
2023-07-05 19:22:12 +00:00
Michal Niewrzal
2c3464081f satellite/metainfo: fix bucket name validation
Change-Id: Ifa400ec855ee978ff001fa3736a8a4c1c53fd18c
2023-07-05 14:42:31 +00:00
Márton Elek
6a3802de4f satellite,storagenode: propagate node tags with NodeCheckin
Change-Id: Ib1a602a8cf81204efa001b5d338914ea4218c39b
2023-07-05 13:45:42 +00:00
Clement Sam
a740f96f75 storagenode/pieces/lazyfilewalker: test zapwrapper
This add tests to the zapwrapper package and also adds a test
to verify the issue in https://github.com/storj/storj/issues/6006

Change-Id: Iec3f568e72683af71e1718017109a1ed52794b0b
2023-07-05 12:33:00 +00:00
Clement Sam
7ac2031cac web/multinode: fix wrong free disk space in allocation on dashboard
There are many case where the keywords `free` and `available`
are confused in their usage.

For most cases, `free` space is the amount of free space left
on the whole disk, and not just in allocation while
`available` space is the amount of free space left in the
allocated disk space.

What the user/sno wants to see is not the free space but the
available space. To the SNO, free space is the free space
left in the allocated disk space.

Because of this confusion, the multinode dashboard displays
the `free` disk space instead of the free space in the
allocated disk space https://github.com/storj/storj/issues/5248
While the storagenode dashboard shows the correct free space
in the allocation.

This change fixes the wrong free disk space. I also added a
few comments to make a distinction between the `free`
and `available` fields in the `DiskSpace*` structs.

Change-Id: I11b372ca53a5ac05dc3f79834c18f85ebec11855
2023-07-05 11:24:24 +00:00
Michal Niewrzal
21c1e66a85 satellite/overlay: refactor ReliabilityCache to keep more data
ReliabilityCache will be now using refactored overlay Reliable method.
This method will provide more info about nodes (e.g. country code) and
with this we are able to add two dedicated methods to classify pieces:
* OutOfPlacementPieces
* PiecesNodesLastNetsInOrder

With those new method we will fix issue where offline but reliable node
won't be checked for clumped pieces and off placement pieces.

https://github.com/storj/storj/issues/5998

Change-Id: I9ffbed9f07f4881c9db3bd0e5f0412f1a418dd82
2023-07-05 11:19:10 +02:00
Michal Niewrzal
f2cd7b0928 satellite/overlay: refactor Reliable to be used with repair checker
Currently we are using Reliable to get missing pieces for repair
checker. The issue is that now checker is looking at more things than
just missing pieces (clumped/off, placement pieces) and using only node
ID is not enough. We have issue where we are skipping offline nodes from
clumped and off placement pieces check.

Reliable was refactored to get data (e.g. country, lastNet) about all
reliable nodes. List is split into online and offline. This data will be
cached for quick use by repair checker. It will be also possible to
check nodes metadata like country code or lastNet.

We are also slowly moving `RepairExcludedCountryCodes` config from
overlay to repair which makes more sens for it.

This this first part of changes.

https://github.com/storj/storj/issues/5998

Change-Id: If534342488c0e440affc2894a8fbda6507b8959d
2023-07-05 10:56:31 +02:00
Márton Elek
500b6244f8
satellite/satellitedb: create table for node tags
Change-Id: I884bb740974e6b8241aa6b85faf266b85fe892d4
2023-07-05 09:38:53 +02:00
Clement Sam
1851d103f9 go.mod: bump storj.io/private
Updates https://github.com/storj/storj/issues/6006

Change-Id: I1f549d5642213e420f3e5d0df4ef972c77add713
2023-07-03 23:59:15 +00:00