Commit Graph

8247 Commits

Author SHA1 Message Date
Vitalii
2c56599ca0 web/satellite: revert uploads queue size
Revert upload parallelism queue size back to 4 segments at a time.
Increasing this number causes some memory issues for some users.

Issue:
https://github.com/storj/storj/issues/6120

Change-Id: Ic5bcd1f13ee625fdc2613ee23d6945c905f03142
2023-07-27 19:57:11 +03:00
Clement Sam
cc12a48c24 satellite/admin: extend admin API to allow setting and deleting geofence for projects
Issue: https://github.com/storj/storj-private/issues/357
Change-Id: Ib59319581641f1f5da71c629143e12f11eb04925
2023-07-27 11:40:26 +00:00
Michal Niewrzal
4cc167a6bd satellite/metabase: use better label for ignoring FTS queries
For some test queries we are using workaround to filter them out from
full table scan detection. To avoid confustion what is this all about
we are changing label to be more descriptive.

Change-Id: I41a744e8faf400e3e8de7e416d8f4242f9093fce
2023-07-26 20:01:38 +00:00
Michal Niewrzal
3d9c217627 satellite/metabase: add pending_objects table
This change adds only schema definition of pending_objects table and
small amount of supporting code which will be useful for testing later.

With this table we would like to achieve two major things:
* simplify `objects` table, before we will start working on object
versioning
* gain performance by removing need to filter `objects` results with `status` column, which is not indexed and we would like to avoid that

https://github.com/storj/storj/issues/6045

Change-Id: I6097ce1c644a8a3dad13185915fe01989ad41d90
2023-07-26 20:00:58 +00:00
Moby von Briesen
28737f5c62 web/satellite: Implement "sign out" in vuetify poc
Implements the "sign out" button in the vuetify app to clear all app
data and navigate to login.

For now, there is a hard refresh after the router navigates to login,
becuase the vuetify poc does not have its own login page to route to.
The hard refresh allows the user to be presented with the main app's
login page after signing out.

Change-Id: Idb1c1e6af8511ea5db6533e46b96b9c15693379f
2023-07-26 13:49:30 -04:00
Moby von Briesen
7487809476 web/satellite: unmock list buckets in vuetify app
This change implements the table on the buckets view of the vuetify app.
Searching and pagination are implemented, but sorting functionality
needs to be added in a separate change - we need to extend the
bucketsStore/backend functionality in order to support sorting.

Resolves https://github.com/storj/storj/issues/6062

Change-Id: I4e92e6facbd7b21f4ec081e41f7e568ddb3cea29
2023-07-26 13:49:30 -04:00
Michal Niewrzal
16c5b3c340 mod: bump uplink to v1.11.0
Change-Id: I88922421f250a261b78227a409f4879ae5221140
2023-07-26 15:01:18 +00:00
Vitalii
6d94d6a681 satellite/{console, accounting}: fix project bandwidth calculation which is shown on project dashboard
Another try to fix calculation of used bandwidth which is displayed on Project Dashboard.
This change sums up allocated-dead traffic for the last 3 days and settled traffic for the period which is earlier than 3 days ago.

Issue:
https://github.com/storj/storj-private/issues/293

Change-Id: I91e652eba69f81bd21e0d053ac170e2b926b3cb4
2023-07-26 06:14:48 +00:00
Vitalii
5db0fd8846 web/satellite: wire up vuetify access page to backend
Added logic to list, sort, search and paginate access grants table for Vuetify POC

Change-Id: I215a9ad4f894b6ac985cb6a3059fdcf007e3d914
2023-07-24 23:17:19 +00:00
Jeremy Wharton
28711e30ad web/satellite/vuetify-poc: add project ID to project-specific routes
A project ID path parameter has been incorporated into project-specific
routes. Previously, project IDs were stored only in local storage,
which made it impossible to bookmark or share links to pages within a
particular project.

Resolves #6076

Change-Id: I4f46f58a95dfc9b67d95fdb1c6a65b50fcafe02b
2023-07-24 22:25:29 +00:00
Jeremy Wharton
5a03e29fca web/satellite/vuetify-poc: unmock all projects dashboard data
The all projects dashboard of the Vuetify project has been updated to
display real project information rather than mock data. Additionally,
projects may now be selected and project invitations replied to through
the all projects dashboard.

Resolves #6036
Resolves #6038

Change-Id: I8aef0dc07c172e2bc8e2f7eb26a3d205bd56067f
2023-07-24 21:39:28 +00:00
Jeremy Wharton
9086078ac7 web/satellite: remove glob patterns from linting commands
The glob patterns in the linting commands have been removed in favor of
a simpler way of expressing which files should be linted.

Change-Id: I150a01725642b4bc445e2e157ddf86e50e3911a2
2023-07-24 10:33:07 -05:00
Michal Niewrzal
73a279235a satellite/accounting/live: get project totals in batches
After infrastructure changes redis instance is not neccessay close
to core instance (where tally is calculated) and round trips to get
data from redis can be very costly. From less than hour calculation can take few hours for larger satellite.

This change combines 'segment' and 'storage' usage requests into
batches to reduce latency impact on tally calculation.

https://github.com/storj/storj/issues/5800

Change-Id: I87e57ec09e88fd167060a4ed51dc8b0274a095c5
2023-07-24 14:13:04 +00:00
Jeremy Wharton
f30e0986b6 web/satellite/vuetify-poc: fix app store name conflict
The name of the Vuetify Pinia app store has been renamed so that it
does not conflict with the regular app store. Previously, whichever
store was created first would override the other due to them having
identical names.

Change-Id: I11cb3b70b7385320accd30644cb3ad6cfad7d00e
2023-07-22 20:51:50 -05:00
Cameron
b16c8ba2e4 web/satellite: add indeterminate progress bar
Display indeterminate progress bars in upload modal if progress is less
than 1.

Change-Id: Icdad8e59914985f3ed8fd25dd01dba7e9ff88cf0
2023-07-21 18:11:18 -04:00
Wilfred Asomani
8ed4c573db satellite/admin: add endpoint to unwarn user
This change enables the admin UI to remove the warning status of users.

resolves: storj-private/issues/342

Change-Id: Ib960ffb33fdabc045884ce7fa2c55c3553db0fb0
2023-07-21 17:10:09 +00:00
Vitalii
4ac8320f3c web/satellite: update wording for upgrade account token flow
Update wording to reflect latest auto upgrade account flow.

new wording:
Send more than $10 in STORJ Tokens to the following deposit address to upgrade to a Pro account.
Your account will be upgraded after your transaction receives 15 confirmations.
If your account is not automatically upgraded, please fill out this limit increase request form.

Issue:
https://github.com/storj/storj-private/issues/367

Change-Id: I46fcb9243722313b98e5c54e8194d6152f7e1631
2023-07-21 15:27:31 +00:00
Michal Niewrzal
d9525a0f27 cmd/tools/migrate-segment-copies: fix placement scan from DB
Placement can be null in DB and we need adjust scanning this column
from DB.

Additionally this change sets application name for DB connection.

Change-Id: I3c7d6294f4a3e5e441160b2fd4aeafffe705ec76
2023-07-21 13:12:08 +00:00
Vitalii
6e4044b245 satellite/payments: slightly refactor user upgrade observer of billing chore
Resolves post-merge comments from here
https://review.dev.storj.io/c/storj/storj/+/10780

Reworked some definitions. Added checks and comments.

Change-Id: I19c63804ad1f30a1ffd8cb87e96f43deed20a685
2023-07-21 03:43:42 +00:00
Wilfred Asomani
2c934d1cfd satellite/console: trigger invoice payment on card remove
This change triggers payment when a card is removed.

Issue: https://github.com/storj/storj/issues/6056

Change-Id: Ia1f2b49609ec9d111cbffb0757bec0b513abb79d
2023-07-20 15:46:50 +00:00
Egon Elbre
dc1509ee42 Makefile,web: bump to node@18
Change-Id: Ib50e18ecda9520a802a5a565e4dfe034ae41e98a
2023-07-20 16:44:13 +03:00
Artur M. Wolff
d45bc879c3 {Jenkinsfile, Makefile}: update Go to 1.20.6
Change-Id: I0a9b483ddcf2e0322840b1eb61982a21cea88e72
2023-07-20 09:20:30 +00:00
Egon Elbre
a058b7e982 web/satellite: fix npm for windows
`NODE_ENV=development xyz` approach does not work with Windows.

Change-Id: I3f0c4a6a2e9566df4923d5e6bc317c7d09573f18
2023-07-20 07:59:03 +00:00
Egon Elbre
7d1031feda satellite/accounting: fix duplicate code-block
Change-Id: Ic6de7337a5ff9a2fec73aa3d8a74ba5b9076a8c4
2023-07-20 07:58:53 +00:00
Jeremy Wharton
ca263c05bb web/satellite/vuetify-poc: go to all projects dashboard on logo click
This change navigates users to the all projects dashboard when the logo
in the navigation bar has been clicked.

Change-Id: Id63b616d12eea510b52f715f3003838541d2ab63
2023-07-19 12:06:29 -05:00
Clement Sam
4cb85186b2 storagenode/pieces: enable lazyfilewalker by default
Resolves https://github.com/storj/storj/issues/5861

Change-Id: I20e0a5b8a15ca966cbccd71369322a517a2c2130
2023-07-19 15:25:20 +00:00
paul cannon
9a871bf3bc go.mod: bump storj.io/common
In particular, to get commit 9db74ed9 in for the next storagenode build.

Change-Id: If07283bc72d66dfbd0ee91bc1cc8b2ce015871cd
2023-07-19 09:50:06 -05:00
Jeremy Wharton
afae5b578e web/satellite/vuetify-poc: allow navigation drawer to be toggled
This change allows the navigation drawer to be toggled by clicking the
the hamburger menu in the navigation bar. The hamburger menu has been
hidden in pages not containing a navigation drawer.

Resolves #6034

Change-Id: I48cfee1f48964c500c07f09f188c7077442261ab
2023-07-19 12:09:05 +00:00
Vitalii
5317135416 satellite/payments: fix config value for auto upgrade user tier flow
Fixed config value which indicates how many base units of US micro dollars are needed to auto upgrade user to paid tier.

Change-Id: I22821ac22fc3eaeeea21c6dec4e6912025df63aa
2023-07-19 10:43:07 +00:00
Wilfred Asomani
7cc873a62a satellite/payments: prevent removing other users' cards
This change patches a loophole that allowed accounts to remove cards
that belong to other users.

Closes #storj/storj-private#326

Change-Id: I33e9efe5c9cdb03aa48ad4c6b1d3283c396a7890
2023-07-19 09:44:06 +00:00
Michal Niewrzal
31bb6d54c7 cmd/tools: add tool to migrate segment copies metadata
We need migrate all existing segment copies to contain all the same
metadata as original segment. So far we were not duplicating stored
pieces but we are changing this behavior right now. We will use this
tool after enabling new way of doing server side copies.

Fixes https://github.com/storj/storj/issues/5890

Change-Id: Ia9ca12486f3c527abd28949eb438d1c4c7138d55
2023-07-18 15:12:51 +00:00
Egon Elbre
23631dc8bb satellite/accounting: fix TestProjectSegmentLimit*
Tally ensures that live accounting has the latest information,
however, when there are concurrent updates to live-accounting
it may by off by a few segments. Disable tally for those tests.

Change-Id: I6fa8a1794334bba093e18f29cb76e7b8d1244979
2023-07-18 14:26:05 +00:00
Egon Elbre
b1e7d70a86 satellite/payments/billing: fix test
`time.Now()` can return the same value when called sequentially.

Change-Id: I800c7696b919ad073e4558fb51c8d2eb4a04f05e
2023-07-18 16:25:42 +03:00
Vitalii
583ad54d86 satellite/{payments, console}: added functionality to get wallet's transactions (including pending)
Added new functionality to query storjscan for all wallet transactions (including pending).
Added new endpoint to query all wallet transactions.

Issue:
https://github.com/storj/storj/issues/5978

Change-Id: Id15fddfc9c95efcaa32aa21403cb177f9297e1ab
2023-07-18 11:09:29 +00:00
Vitalii
2ee0195eba satellite/payments: extend billing chore functionality to upgrade user
Added new observer for billing chore to check user's balance and upgrade their account if balance is more than or equal to needed amount for upgrade.
Added new config value which stands for needed amount of base units of US micro dollars needed to upgrade user.

Issue:
https://github.com/storj/storj/issues/5978

Change-Id: Ic3992cd3114397bfdd9e231ca090ff21ca66648b
2023-07-18 13:15:02 +03:00
Michal Niewrzal
0303920da7 satellite/metainfo: remove unused method
Change-Id: I08e307e6909cdc46951c5f3112d77a685e67fe2e
2023-07-18 08:45:29 +00:00
Jeremy Wharton
df9a6e968e web/satellite: lint Vuetify files
This change enables linting for the Vuetify proof of concept code and
fixes the linting errors that were detected. Additionally, it migrates
the Vuetify components to the composition API.

Change-Id: Id8cc083954e3f4cb66a00ad2715a96c8747b592c
2023-07-17 20:32:59 +00:00
dlamarmorgan
abe1463a73 payments/stripe/invoices: add token payment to overdue invoice payment
Add an attempt to pay overdue invoices via storj token if the user has
a token balance.

Change-Id: I819b89e7cf9cdb7deb9a51eab5ca684b54418218
2023-07-17 12:59:33 -07:00
dlamarmorgan
c96c83e805 satellite/payments/stripe/service: add manual payment with token command
Add the ability to pay an individual users open invoices using their
storj token balance.

Change-Id: I6115f2b033fd77f109ded6f55b1f35fc77c71ff1
2023-07-17 19:24:36 +00:00
dlamarmorgan
0f4371e84c scripts/tests/{backwardcompatibility,integrations}: add test scripts
Change-Id: Ib83cd0f083bab7f560a200fd95e62e5b21e60c27
2023-07-17 18:39:18 +00:00
Wilfred Asomani
0a8115b149 satellite/{console,payments}: fix handling for autofreeze flow
This change adds an extra step to the auto freeze chore to attempt
payment before freezing/warning a user.
It also attempts payment after modifying user's cards whether the user
is frozen/warned or not.

Issue: https://github.com/storj/storj-private/issues/341

Change-Id: Ia9c0c5a2d37837bca5153fe720fef61f1385cb15
2023-07-17 17:37:11 +00:00
Michal Niewrzal
47a4d4986d satellite/repair: enable declumping by default
This feature flag was disabled by default to test it slowly. Its enabled
for some time on one production satellite and test satellites without
any issue. We can enable it by default in code.

Change-Id: If9c36895bbbea12bd4aefa30cb4df912e1729e4c
2023-07-17 15:02:35 +00:00
Michal Niewrzal
5272fd8497 satellite/metainfo: do full bucket validation only on create
We are doing full bucket name validation for many requests but
we should do this only while creating bucket. Other requests will be
covered only by basic name length validation. Less strict validation for
other requests will make bucket usable in case of invalid bucket names
in DB (we have such cases from the past).

https://github.com/storj/storj/issues/6044

Change-Id: I3a41050e3637787f788705ef15b5dc4df4d01fc6
2023-07-17 16:15:33 +02:00
Jeremy Wharton
95761908b5 web/satellite: update Vuetify proof of concept
The changes as of storj/vuetify-storj@c801fe6 have been pulled into the
Vuetify proof of concept.

Change-Id: I3db208836cff21287052615d36258fcf2d4c6169
2023-07-14 12:46:50 +00:00
Michal Niewrzal
5234727886 satellite/repair/repairer: fix flaky TestSegmentRepairPlacement
Sometimes DownloadSelectionCache doesn't keep up with all node
placement changes we are doing during this test.

Change-Id: Idbda6511e3324b560cee3be85f980bf8d5b9b7ef
2023-07-14 10:10:40 +00:00
Tomasz Melcer
5a1c3f7f19
storage/reputation: logging changes to node scores (#5877)
Useful for monitoring storage nodes using log-parsing tools, like swatchdog.

Co-authored-by: Clement Sam <clementsam75@gmail.com>
2023-07-13 17:03:18 +02:00
Wilfred Asomani
4ee647a951 satellite: add request id to requests
This change adds request IDs to requests, logs them as part of audit
logs and sends to the client on error. This is to improve debugging
of customer issues.

Issue: https://github.com/storj/storj/issues/5898

Change-Id: I801514b547d28d810552d91aa7c8502051e552bf
2023-07-13 09:22:43 +00:00
Cameron
e8fcdc10a4 satellite/metainfo: set user_agent in bucket_metainfos on bucket recreation
Before this change, if a user creates a bucket with a user_agent attributed then deletes and recreates it, the row in bucket_metainfos
will not have the user_agent. This is because we skip setting the field
in bucket_metainfos if the bucket already exists in value_attributions.
This can be problematic, as we return the bucket's user agent during the
ListBuckets operation, and the client may be expecting this value to be
populated.

This change ensures the bucket table user_agent is set when (re)creating a bucket. To avoid decreasing BeginObject performance, which also
updates attribution, a flag has been added to determine whether to
make sure the buckets table is updated: `forceBucketUpdate`.

Change-Id: Iada2f233b327b292ad9f98c73ea76a1b0113c926
2023-07-12 21:48:05 +00:00
Michal Niewrzal
99128ab551 satellite/metabase: reuse Pieces while looping segments
Segments loop implementation is using lots of memory to convert
alias pieces to pieces for each segment while iteration. To improve
situation this change is reusing Pieces between batch pages. This
should signifcantly reduce memory usage for ranged loop executions.

Change-Id: I469188779908facb19ad85c6bb7bc3657111cc9a
2023-07-12 09:29:34 +00:00
Jeremy Wharton
062ca285a0 web/satellite: add sharing option to dropdown in buckets page
This change adds a sharing option to the dropdown menu for bucket rows
in the Buckets page.

Resolves #5964

Change-Id: Ife0eb8f6cabbe85eaedae1d94d97694f3c677a3e
2023-07-11 22:03:42 +00:00