Commit Graph

7191 Commits

Author SHA1 Message Date
Stefan Benten
c3171b4ba4
storagenode/retain: Move summary and start logs to info level (#4954)
We currently do not log the GC information/stats under normal circumstances.
This is not good for monitoring and troubleshooting.
2022-07-08 18:19:08 +02:00
Michał Niewrzał
43286dbc37 satellite/metainfo: use metaclient type instead of storj
This just cleanup change to unblock libuplink to reorganize types
which are aliases to storj types.

Change-Id: Id3edf13f1b0aef52d7606d545aa7a6594cf8d13f
2022-07-08 14:17:19 +02:00
Egon Elbre
96c3816313 build: add timeout to npm install
Add timeout to npm install and increase logging level.
npm install is still taking sometimes too long and it's not clear why,
verbose logging is not sufficient.

Change-Id: Ib72f9823f30c9744562e279c2a5481f096e38128
2022-07-07 17:59:16 +00:00
Egon Elbre
5c5e6f75e3 build: fail the build on timeout
When there's a timeout we need to handle it in `aborted` post action,
rather than failure.

Change-Id: I861dae2a95ed1818eed937df890f93865da12e75
2022-07-07 17:58:58 +00:00
kimbotsao
d09824d684 web/satellite: do not load stripe for environment unless enabled
Stripe script is only loaded when Stripe is enabled for adding
and using payments.

Change-Id: Ia2ed9dd138bf6a5a288adfb9259e7007822e634f
2022-07-07 16:46:15 +00:00
Egon Elbre
4e31c96836 satellite/console: don't use global log
Change-Id: I2464f2aca3cdb97c19de29dac59499cc9d5ff2bb
2022-07-06 16:50:04 +00:00
hovex023
6f4dc963b1
web/satellite: Billing history tab
* tabs and routing implemented
* added feature flag
* added billing history buildout
* Update to use real data for table population

Story: https://github.com/storj/storj/issues/4633

Co-authored-by: cl-mitch <mitch.george@compozelabs.com>
2022-07-06 11:42:16 -05:00
Egon Elbre
d0ca258925 build: increase npm install loglevel
The builds for npm still fail, however, `--timing` does not provide
sufficient data to debug the situation.

Change-Id: I7e618ba8cac775748ebea6145cd5c180d2dc7883
2022-07-06 18:13:53 +03:00
Lizzy Thomson
e26b1517ed web/satellite added middle truncation logic for filenames in file browser
Added middle truncation and extended filename on hover according to figma designs

Change-Id: I5559abc6e6d959491603c0d2d20ddfbfdc298eee
2022-07-06 13:43:22 +00:00
Cameron
023a1c6271 satellite/satellitedb: add public_id column and index to projects table
github issue: https://github.com/storj/storj/issues/4861

Change-Id: I3d1928ca51c825a4c1589db5ae55902e57f00c3b
2022-07-05 18:39:29 +00:00
Ivan Fraixedes
66f12992b0
cmd/storagenode: Don't abbreviate multinode in help
Don't abbreviate multinode in the command help message because there
isn't a need for it and the abbreviation isn't clear at all.

Change-Id: I7a1f2be6ae1f7d4b287c18c48b22c630549b731f
2022-07-05 19:29:12 +02:00
Clement Sam
ebd463b57c go.mod: update dependencies
Change-Id: I5360ea7b6487295cb7def759f417b31b133c34da
2022-07-05 18:41:39 +03:00
Artur M. Wolff
0f626a59c1 mod: bump storj.io/uplink
Change-Id: I6d51fd330ebb882564b6c0cd02d829de08082b46
2022-07-05 10:27:27 +00:00
Qweder93
417df4d73c satellite/metabase: don't return expired objects
Resolves: https://github.com/storj/team-metainfo/issues/112

Change-Id: If48f865826c22764fc1ff83745c34524cd396aac
2022-07-04 21:32:30 +03:00
Egon Elbre
d162788489 build: use npm install
`npm ci` deletes the node_modules directory, which also removes go.mod
from that folder.

Add --loglevel timing, so we can debug install slowness, whenever it
happens.

Change-Id: Ide613c4124bfdca9ae978876b2deed8abf86f987
2022-07-01 17:54:18 +03:00
Vitalii
53bddbdd0a web/satellite: fixes for bucket details page
Ask for encryption passphrase when user tries to navigate inside bucket using next flow: Buckets -> BucketDetails.
We don't ask for passphrase if user uses this flow: FileBrowser -> BucketDetails.
Don't allow empty passphrase in open bucket modal.
Fetch buckets using our API after new bucket creation flow (instead of S3 fetch).

Change-Id: Ia0894d6bb4a764c4ff0974fb16ed89bb82699807
2022-07-01 12:01:34 +00:00
Ivan Fraixedes
1fbc8f1f40 satellite/console/consoleweb: Delete old TODO comment
Delete an old TODO comment which doesn't apply anymore.

Closes https://github.com/storj/storj/issues/4958

Change-Id: Ie4be51afb6a39b3f5e5822ac04d1c8fd9d86ee3a
2022-07-01 11:31:19 +00:00
Egon Elbre
97e6e5a6a3 build: improve npm build
Add go.mod to node_modules folder, that way Go compiler doesn't
need to scan the node_module directories for any Go code.

Change-Id: I747909416490c847d6b4bfa3438fea66660fcd53
2022-07-01 10:51:53 +00:00
Egon Elbre
05e165283f storagenode/console/consoleapi: use fixed time.Now()
It seems the tests relied on time.Now(), which might cause some
discrepancies in calculations. Use a fixed time.Now() rather than
recalculating.

As a sidefix, remove "Test" prefix from t.Run. These are unnecessary.

Change-Id: I1de903fcf0fcf46fc8e3acf2463e17239b8e3cc6
2022-07-01 12:36:01 +03:00
Márton Elek
6c5edf436e build: execute web tests only when source is changed (verify)
Change-Id: I1036e8cfed8cc708ef28a41dcd69017028ed9ff4
2022-07-01 07:58:45 +00:00
Erik van Velzen
f23d5eb5a1 satellite/repair: remove superfluous conditional
Change-Id: If80ae0a1a4ee436763ed437fc77b0ed26db17a68
2022-06-30 18:09:17 +00:00
NickolaiYurchenko
5b550a192c web/satellite: access grant flow ui bugs fixed
Change-Id: If0e29728e3a410d629ffc6c39145ac0c9e572101
2022-06-30 10:00:26 +00:00
littleskunk
cf750716ea
satellite/console: enable new access grant flow (#4934) 2022-06-30 00:06:39 +02:00
Egon Elbre
b8006e192b satellite/audit: use a larger delay in the test
The MinDownloadTimeout 950ms and delay of 1s were quiet close, possibly
causing flaky behavior in TestVerifierSlowDownload.

Change-Id: I4f6c1554a118b21427357642abe39986fd0af38d
2022-06-28 17:03:23 +03:00
Ivan Fraixedes
0051298eec satellite/console: Classify errors activation tokens
Classify errors related to invalid tokens for activating user accounts
for returning 400 status code rather than 500 status code.

Don't log all the errors with "error" level, only the ones related to
internal server errors and the rest log them with "debug" level because
they pollute the production satellite errors with errors that are
misguiding.

Change-Id: Id2bd737edba8550ce08965b51b8bf2540bd13ca4
2022-06-28 11:07:57 +00:00
Egon Elbre
65b5a0fe82 satellite/{overlay,satellitedb}: add AS OF to download selection
This should reduce the load caused by download selection queries.

Change-Id: Ic1a89d9c3eb5418f0792eb20ec2aece18dc63f2c
2022-06-28 05:18:01 +00:00
Erik van Velzen
74f4f6e765 satellite/metabase: fix copy to ancestor location
Previously copying an object to it's ancestor location (copy of copy)
broke the object and all copies.

This fixes this by calling the existing delete method rather than a
custom one when there is an existing object at the copy destination.

The check for existing object at destination has been moved to an
earlier point in FinishCopy.

metabase.DeleteObject exposes a transaction parameter so that it can be
reused within metabase.

Closes https://github.com/storj/storj/issues/4707

Uplink test at https://review.dev.storj.io/c/storj/uplink/+/7557

Change-Id: I418fc3337fa9f30146ccc1db456af168ae41c326
2022-06-28 03:53:01 +02:00
Jeff Wendling
3a63939fc9 satellite/metabase: add defensive check on copy object
- instead of closing over the outer err variable, potentially
  overwriting some errors or something, declare local variables.

- double check that we got the number of rows we expected to get
  and error otherwise. this prevents a possible source of inserting
  bogus rows into the database.

Change-Id: I30662be2727afe0a90e4215a182fedc2648d1169
2022-06-27 12:30:44 -04:00
Jeff Wendling
e00423e2e9 cmd/uplink: disable deleting local files
Change-Id: Id3687f3346f8ec9b5da5459a57d91d1d30bf5566
2022-06-27 10:30:04 -04:00
Vitalii
8bd9a079e4 web/satellite: fix bucket open/delete issue
Fixed bucket open and delete issue.

Issue: https://github.com/storj/storj/issues/4932

Change-Id: Idbd981007e537ffb4dd085c68c03a0edfd1cebec
2022-06-27 11:54:42 +00:00
Erik van Velzen
4da61aa257 satellite/metabase: segment_copies full table scan
Part of the delete query cause a full table scan of segment_copies. This
slowed down the system. This change should have the same semantics but
improved performance.

Part of https://github.com/storj/storj/issues/4898

Change-Id: I4afe23df05467eafc9c91591f47a7251a0f3dd31
2022-06-27 11:03:15 +00:00
Vitalii
5f1390e1d5 web/satellite: fixes for navigation sidebar
Show free/pro badge in collapsed nav side bar.
Use getter to indicate which badge to show instead of calculating once after first render.
Fix project overview label for smaller screens. The problem was with new project dashboard route name.

Change-Id: I6ec340b14fe7cf11ba96a0d4ae6771e830c2ed94
2022-06-27 10:53:02 +03:00
Yaroslav Vorobiov
91ea659298 satellite/payments/storjscan: payment API type add USD value
Change-Id: I6e06a5557df795b4f91dd52b191384d7e6b4b07c
2022-06-24 22:30:30 +00:00
JT Olio
18666b078a metabase: support disabling serverside copy when already enabled
Change-Id: I99b0b5597537f69fbdd8cab479e059ba4bf310bd
2022-06-24 17:56:43 +00:00
Erik van Velzen
77fea6137f satellite/metabase: copy inside transaction
Read the source object and write the destination object in the same
transaction, to prevent breaking the object because it was deleted
simultaneously.

This is probably the root cause of the metainfo loop halting from
2022-06-21 onwards, where 2 objects lost their root_piece_id during
copying.

Part of https://github.com/storj/storj/issues/4930

Change-Id: I9c45d56a7bfb48ecd5f4906ee1cca42922901e90
2022-06-24 01:32:09 +02:00
Márton Elek
e3ac0ae2c2
build: fix } typo in Jenkinsfile
Change-Id: I30f172069e7b07efc682b7407e9c48e3cb8c2f9e
2022-06-23 09:25:51 +02:00
Márton Elek
204367b74d
build: premerge/verify Jenkinsfile (subsets of Jenkinsifle.public)
Change-Id: I33c65a37ecd8fcb89729b2a8db3906c6b8890ca3
2022-06-23 09:15:43 +02:00
NickolaiYurchenko
b4e486f3fc web/satellite: common table component applied to bucket page
Change-Id: I579ee22ea441e3084f0d56462676d48870087624
2022-06-22 15:57:40 +00:00
Ivan Fraixedes
6e8e86a0b7 satellite/accounting: Fix typos in doc comments
Change-Id: I497bebe152848e023d927337c61e3eee73b87697
2022-06-22 14:45:21 +00:00
Vitalii
0dba785798 satellite/admin: return only owned projects for get user endpoint
Returns only user's own projects when we hit GET user endpoint.

Fix for this issue
https://github.com/storj/storj/issues/4820

Change-Id: I546268fa3e5983a72f11f998803da5455c0035b4
2022-06-22 13:09:36 +03:00
Paul Willoughby
de125eeb51 cmd/uplink: Fix typo in flag description
Change-Id: I6db38ecec2549d288285a9865f97badfe808b892
2022-06-21 11:55:02 -06:00
Ivan Fraixedes
fc4b0e41f0 scripts: Change Storagenode arm32v6 to arm32v5
We don't build "multinode" nor "storagenode" docker images for armv6
architecture, instead we build for armv5.

Fix the script that publish the manifest for those images for a specific
tag to use armv5 for not failing when executing.

Change-Id: I7d859d8718240e1cd0dae6489e7e5c3b4068ff6e
2022-06-21 11:54:38 +00:00
Egon Elbre
94ae196189 testsuite/ui: move go.mod
I would like to start moving some of the integration tests from scripts
folder into testsuite. This allows integration tests to easily have
other and fewer dependencies.

This will also be useful for https://github.com/storj/storj/issues/4823.

Change-Id: Idbdb5aefc0e42d091ad7f74dbdc096281c1dfcba
2022-06-21 14:12:35 +03:00
Ivan Fraixedes
f626cd1868 satellite/accounting: Correctly cache project bandwidth
Satellite caches the project bandwidth in Redis when it doesn't have it
because was not set or the key expired, however, it doesn't perform the
check and set if not exists in a transaction. It also uses the increase
function which increases the value if it exists otherwise it sets it.

This provokes that multiple concurrent request to the same project may
increase the total project by multiples of the bandwidth usage
registered in the database rather than setting it because they may check
if the key exists before any other has executed the increase and then
the first one executing it will set the value but the others will
increased causing that Redis has a wrong bandwidth usage value which is
N magnitude of the real one and making the satellite to deny the
downloading if it surpasses the project limit.

This commit changes the "update"" project bandwidth usage by an "insert"
but using a Redis function that only sets the value if the key doesn't
exists for solving the increase issue but also not overriding the value
due to may contain updates of other downloading requests which aren't
already registered in the DB.

Change-Id: I33e2fe462930b2fdb4061fc94002bd3544476f94
2022-06-21 08:31:41 +00:00
René Smeekes
fd042a92b9
storagenode/reputation: clarify wording on suspension notification (#4921)
Fixes https://github.com/storj/storj/issues/4920
2022-06-20 22:07:12 +02:00
Cameron
b7fd6a70fb web/satellite: fix ui bugs in bucket navigation
On bucket details page, make bucket name breadcrumb link back to the
page for that bucket. Additionally, add a pointer cursor to the
"buckets" breadcrumb.

Also remove settings dropdown from bucket details page as the only item
is "View Bucket Details" which is already the current page.

github issue: https://github.com/storj/storj/issues/4909

Change-Id: Ic24abba7fb447ca5a8ac1e8ba244ba98dec2fc92
2022-06-20 15:53:32 +00:00
NickolaiYurchenko
f7a3be5d20 web/satellite: common table component created
Change-Id: Ie45a2c37257f3311c767adcea1a22f284b66f24e
2022-06-19 14:52:42 +00:00
Cameron
43ed35e025 testsuite/ui/satellite: add ui tests for new access grant flow
also fix typo in web/satellite CreateAccessModal.vue

github issue: https://github.com/storj/storj/issues/4825

Change-Id: I6f66e604ed3d0899aa2ec83a0638c43d4b5e1e42
2022-06-17 21:11:30 +00:00
prerna-parashar
6a1d7c8747
satellite/analytics: Added tracks calls for product activity metrics (#4907)
satellite/analytics: Added tracks calls for product activity metrics
2022-06-17 12:57:10 -07:00
paul cannon
4e81a60838 satellite/overlay: fix TestNodeSelectionGracefulExit
This test had an effective config.Reputation.AuditCount = 0, meaning all
nodes that had _any_ positive audit results were considered vetted.
Because of that, only one node in the test setup was "new". And that
node was marked as being in GE, so could not be returned by node
selection.

The reason the tests still worked is because of the node selection rule
that says "if there are no new nodes at all, just get all reputable
nodes to satisfy the request".

This commit makes it so half of the nodes are vetted and half new, which
makes the test somewhat more interesting (and means we aren't
concentrating too much on testing details of behavior when AuditCount is
0).

Change-Id: I09157b7dc20ecaddd2a6e60cfe146e9186e3603b
2022-06-17 18:28:04 +00:00