Commit Graph

6861 Commits

Author SHA1 Message Date
Erik van Velzen
4da61aa257 satellite/metabase: segment_copies full table scan
Part of the delete query cause a full table scan of segment_copies. This
slowed down the system. This change should have the same semantics but
improved performance.

Part of https://github.com/storj/storj/issues/4898

Change-Id: I4afe23df05467eafc9c91591f47a7251a0f3dd31
2022-06-27 11:03:15 +00:00
Vitalii
5f1390e1d5 web/satellite: fixes for navigation sidebar
Show free/pro badge in collapsed nav side bar.
Use getter to indicate which badge to show instead of calculating once after first render.
Fix project overview label for smaller screens. The problem was with new project dashboard route name.

Change-Id: I6ec340b14fe7cf11ba96a0d4ae6771e830c2ed94
2022-06-27 10:53:02 +03:00
Yaroslav Vorobiov
91ea659298 satellite/payments/storjscan: payment API type add USD value
Change-Id: I6e06a5557df795b4f91dd52b191384d7e6b4b07c
2022-06-24 22:30:30 +00:00
JT Olio
18666b078a metabase: support disabling serverside copy when already enabled
Change-Id: I99b0b5597537f69fbdd8cab479e059ba4bf310bd
2022-06-24 17:56:43 +00:00
Erik van Velzen
77fea6137f satellite/metabase: copy inside transaction
Read the source object and write the destination object in the same
transaction, to prevent breaking the object because it was deleted
simultaneously.

This is probably the root cause of the metainfo loop halting from
2022-06-21 onwards, where 2 objects lost their root_piece_id during
copying.

Part of https://github.com/storj/storj/issues/4930

Change-Id: I9c45d56a7bfb48ecd5f4906ee1cca42922901e90
2022-06-24 01:32:09 +02:00
Márton Elek
e3ac0ae2c2
build: fix } typo in Jenkinsfile
Change-Id: I30f172069e7b07efc682b7407e9c48e3cb8c2f9e
2022-06-23 09:25:51 +02:00
Márton Elek
204367b74d
build: premerge/verify Jenkinsfile (subsets of Jenkinsifle.public)
Change-Id: I33c65a37ecd8fcb89729b2a8db3906c6b8890ca3
2022-06-23 09:15:43 +02:00
NickolaiYurchenko
b4e486f3fc web/satellite: common table component applied to bucket page
Change-Id: I579ee22ea441e3084f0d56462676d48870087624
2022-06-22 15:57:40 +00:00
Ivan Fraixedes
6e8e86a0b7 satellite/accounting: Fix typos in doc comments
Change-Id: I497bebe152848e023d927337c61e3eee73b87697
2022-06-22 14:45:21 +00:00
Vitalii
0dba785798 satellite/admin: return only owned projects for get user endpoint
Returns only user's own projects when we hit GET user endpoint.

Fix for this issue
https://github.com/storj/storj/issues/4820

Change-Id: I546268fa3e5983a72f11f998803da5455c0035b4
2022-06-22 13:09:36 +03:00
Paul Willoughby
de125eeb51 cmd/uplink: Fix typo in flag description
Change-Id: I6db38ecec2549d288285a9865f97badfe808b892
2022-06-21 11:55:02 -06:00
Ivan Fraixedes
fc4b0e41f0 scripts: Change Storagenode arm32v6 to arm32v5
We don't build "multinode" nor "storagenode" docker images for armv6
architecture, instead we build for armv5.

Fix the script that publish the manifest for those images for a specific
tag to use armv5 for not failing when executing.

Change-Id: I7d859d8718240e1cd0dae6489e7e5c3b4068ff6e
2022-06-21 11:54:38 +00:00
Egon Elbre
94ae196189 testsuite/ui: move go.mod
I would like to start moving some of the integration tests from scripts
folder into testsuite. This allows integration tests to easily have
other and fewer dependencies.

This will also be useful for https://github.com/storj/storj/issues/4823.

Change-Id: Idbdb5aefc0e42d091ad7f74dbdc096281c1dfcba
2022-06-21 14:12:35 +03:00
Ivan Fraixedes
f626cd1868 satellite/accounting: Correctly cache project bandwidth
Satellite caches the project bandwidth in Redis when it doesn't have it
because was not set or the key expired, however, it doesn't perform the
check and set if not exists in a transaction. It also uses the increase
function which increases the value if it exists otherwise it sets it.

This provokes that multiple concurrent request to the same project may
increase the total project by multiples of the bandwidth usage
registered in the database rather than setting it because they may check
if the key exists before any other has executed the increase and then
the first one executing it will set the value but the others will
increased causing that Redis has a wrong bandwidth usage value which is
N magnitude of the real one and making the satellite to deny the
downloading if it surpasses the project limit.

This commit changes the "update"" project bandwidth usage by an "insert"
but using a Redis function that only sets the value if the key doesn't
exists for solving the increase issue but also not overriding the value
due to may contain updates of other downloading requests which aren't
already registered in the DB.

Change-Id: I33e2fe462930b2fdb4061fc94002bd3544476f94
2022-06-21 08:31:41 +00:00
René Smeekes
fd042a92b9
storagenode/reputation: clarify wording on suspension notification (#4921)
Fixes https://github.com/storj/storj/issues/4920
2022-06-20 22:07:12 +02:00
Cameron
b7fd6a70fb web/satellite: fix ui bugs in bucket navigation
On bucket details page, make bucket name breadcrumb link back to the
page for that bucket. Additionally, add a pointer cursor to the
"buckets" breadcrumb.

Also remove settings dropdown from bucket details page as the only item
is "View Bucket Details" which is already the current page.

github issue: https://github.com/storj/storj/issues/4909

Change-Id: Ic24abba7fb447ca5a8ac1e8ba244ba98dec2fc92
2022-06-20 15:53:32 +00:00
NickolaiYurchenko
f7a3be5d20 web/satellite: common table component created
Change-Id: Ie45a2c37257f3311c767adcea1a22f284b66f24e
2022-06-19 14:52:42 +00:00
Cameron
43ed35e025 testsuite/ui/satellite: add ui tests for new access grant flow
also fix typo in web/satellite CreateAccessModal.vue

github issue: https://github.com/storj/storj/issues/4825

Change-Id: I6f66e604ed3d0899aa2ec83a0638c43d4b5e1e42
2022-06-17 21:11:30 +00:00
prerna-parashar
6a1d7c8747
satellite/analytics: Added tracks calls for product activity metrics (#4907)
satellite/analytics: Added tracks calls for product activity metrics
2022-06-17 12:57:10 -07:00
paul cannon
4e81a60838 satellite/overlay: fix TestNodeSelectionGracefulExit
This test had an effective config.Reputation.AuditCount = 0, meaning all
nodes that had _any_ positive audit results were considered vetted.
Because of that, only one node in the test setup was "new". And that
node was marked as being in GE, so could not be returned by node
selection.

The reason the tests still worked is because of the node selection rule
that says "if there are no new nodes at all, just get all reputable
nodes to satisfy the request".

This commit makes it so half of the nodes are vetted and half new, which
makes the test somewhat more interesting (and means we aren't
concentrating too much on testing details of behavior when AuditCount is
0).

Change-Id: I09157b7dc20ecaddd2a6e60cfe146e9186e3603b
2022-06-17 18:28:04 +00:00
Vitalii
b25faf30ab web/satellite: rework navigation sidebar styling
Reworked styling to correspond to newest designs.

Resolves those issues:
https://github.com/storj/storj/issues/4847
https://github.com/storj/storj/issues/4848
https://github.com/storj/storj/issues/4845
https://github.com/storj/storj/issues/4850

Change-Id: Ie3510b435815834adff5160541b5d9d4859763eb
2022-06-17 17:55:47 +00:00
Jeremy Wharton
5ce7d980af private/apigen: Make API generation deterministic
This change fixes the issue where the API generator would produce
different Go code for the same API definition upon each invocation
due to the random nature of map iteration.

Change-Id: I6770a10faf06311c24f541611c25d0b2b0f8e521
2022-06-17 12:06:08 -05:00
Qweder93
cc0dc2e262 satellite/metabase: FinishMoveObject cheks for empty metadata
To avoid regression with old versions of uplink objects move we need to
remove FinishMoveObject check for key and nonce, in SQL
in FinishMoveObject do check if metadata is nil then
don't set key and nonce. The same UPDATE clause should return metadata and
if metadata != nil we should do the same validation for key and nonce
to avoid putting broken key and nonce while doing move

Resolves: https://github.com/storj/team-metainfo/issues/108

Change-Id: If723dfad899e9235f53559b71ee1c7fe49deb8b8
2022-06-17 08:24:24 +00:00
Vitalii
e79e79d2e1 web/satellite: fix linksharing issue
Bucket creation/open flow wasn't storing credentials correctly inside Vuex store.

Change-Id: Ifc2eb8ca79a672dbc44d95089f7d6cee2e71bb57
2022-06-16 20:57:43 +03:00
Yaroslav Vorobiov
13be4ad0dd satellite/payments/storjscan: add chore to periodically ask for new payments
Change-Id: Ia6ad182184596fe1fe1f9a40f909a3af4be8feef
2022-06-16 09:38:37 +01:00
Jeremy Wharton
bd0d717f4c private/apigen: Remove unnecessary log message when generating API code
A message intended to diagnose issues in a previous change was
accidentally allowed to remain in finalized code and merged into
the codebase.

Change-Id: I7df662a81e3790de065beeb6db54be5280ac2fa1
2022-06-15 15:58:45 -05:00
Jeremy Wharton
905654956a private/apigen: Resolve import issues
This change resolves several issues:
* Imports can no longer be duplicated
* The import statement now conforms to Storj's code style requirements
* Packages are imported as needed, eliminating extraneous imports

Change-Id: Ie021e06bb0d472c65aa3575db94bc81ccac108e3
2022-06-15 19:51:54 +00:00
Fadila Khadar
ffcd54719a satellite/satellitedb: use tx instead of db.db in transactions
Change-Id: Icee8871af76233651f007e03173660905eafc0e3
2022-06-14 17:35:44 +00:00
JT Olio
bb1eb1bae5 storj: bump private
Change-Id: I3dd80fc19a1412ac1d8264080cbf20bc4f040f13
2022-06-14 11:54:28 -04:00
Cameron
240b70b828 satellite/console: use new type UpdateUserRequest as arg to db users.Update
The users.Update method in the satellitedb package takes a console.User
as an argument. It reads some of the fields on this struct and assigns
the value to dbx.User_Update_Fields. However, you cannot optionally
update only some of the fields. They all will always be updated. This means
that if you only want to update FullName, you still need to read the
user info from the DB to avoid updating the rest of the fields to zero.
This is not good because concurrent updates can overwrite each other.

This change introduces a new struct type, UpdateUserRequest, which
contains pointers for all the fields that are updated by satellite db
users.Update. Now the update method will check if a field is nil before
assigning the value to be updated in the db, so you only need to set the
field you want updated. For nullable columns, the respective field is a
double pointer. This allows us to update a column to NULL if the outer
pointer is not nil, but the inner pointer is.

Change-Id: I27f842d283c2711e24d51dcab622e57eeb9157f1
2022-06-14 09:28:03 -04:00
Fadila Khadar
3ae325462c satellite/metabase: fix move transaction
Change-Id: Id1b6c721e7373dffd3f40aafa15109fc2cb82474
2022-06-14 11:04:44 +02:00
Erik van Velzen
b828b6ec34 satellite/metabase: fix copy transaction
Change-Id: I4921d3e6bc17b0c6aff00eba7c76266cb36e78a0
2022-06-13 19:47:07 +02:00
cl-mitch
2c391fa01b
web/satellite added billing navigation tabs (#4839)
tabs and routing implemented for new billing flow


Co-authored-by: hovex023 <97616907+hovex023@users.noreply.github.com>
Co-authored-by: Maximillian von Briesen <mobyvb@gmail.com>
2022-06-13 10:03:32 -05:00
Jeremy Wharton
58c5d44f44 satellite/console: integrate sessions into satellite UI
This change integrates the session management database functionality
with the web application. Claim-based authentication has been removed
in favor of session token-based authentication.

Change-Id: I62a4f5354a3ed8ca80272814aad2448f901eab1b
2022-06-13 08:02:02 +00:00
Artur M. Wolff
8c1caea5db satellite/contact: swap net.IP.IsPrivateIP with isPrivateIP
This change swaps net.IP.IsPrivateIP usages with custom isPrivateIP to
unbreak the build as we want to build for earlier than Go 1.17.

Change-Id: I44badbb487f35e43b8b0433ad0f3b9c87af718d4
2022-06-13 01:01:44 +02:00
JT Olio
098ee71e03 test-sim-versions: write what bash step is executing
Change-Id: If1231f69601c1774bc6ce116f45105cc1b2f4e22
2022-06-10 19:33:03 +00:00
Yaroslav Vorobiov
da97fafb0c satellite/payments/storjscan: add payments DB
Change-Id: Ia6f0f62912c9a60d2c513459959634179eb9506c
2022-06-10 13:44:27 +01:00
prerna-parashar
cc0518f473
satellite/analytics: Added segment.io page calls to track all the pages (#4880)
satellite/analytics: send analytics 'page visit' api requests when the user navigates around the UI
2022-06-09 11:54:23 -07:00
Cameron
b4c77d89a6 web/satellite: add free/pro tier badge to account area
Add a badge to the account area indicating whether the user is paid tier
or free tier.

github issue: https://github.com/storj/storj/issues/4747

Change-Id: I3d435113fada48d4bda6faeef27680eb2850427b
2022-06-09 18:02:57 +00:00
Cameron
5cfa7ca460 satellite/console: add cutoff to email reminders
There are multiple entries in the users table with the same email
address. This is because in the past users were able to register
multiple times if the email was not verified. This is no longer
the case. If a user tries to register with an unverified email
already in the DB, we send a verification email instead of
creating another entry. However, since these old entries in the
table with duplicate emails were never cleaned up, the email
reminder chore will send out email verification reminders to them.
A single person will get one separate email per entry in the DB
with their email and where status = 0.

Since the multiple entries with the same email problem was solved
a while ago, just add a constraint to GetUnverifiedNeedingReminder
to only select users created after a cutoff. Once the DB is
migrated to remove the duplicate emails, we can remove the cutoff.

github issue: https://github.com/storj/storj/issues/4853

Change-Id: I07d77d43109bcacc8909df61d4fb49165a99527c
2022-06-09 16:45:35 +00:00
JT Olio
c761acc0b5 web/ratelimiter: support disabling
Change-Id: I6f2d77016c4917389514b75b5eab64223cc69b2d
2022-06-09 15:33:35 +00:00
Michał Niewrzał
7e387af010 cmd/uplink: add buffering while writing to stdout
Current pipelining to stdout is synchronous so we don't have any
advantage from using --parallelism flag. This change adds buffer
while writing to stdout. Each part is first read into the buffer
and flushed only when all data was read from this part.

https://github.com/storj/uplink/issues/105

Change-Id: I07bec0f4864dc4fccb42224e450d85d4d196f2ee
2022-06-09 15:10:04 +00:00
Márton Elek
7608ea7af0 Makefile: make test should work with any directory name
docker-compose exec is more reliable as it doesn't depend on the name of the directory (the first part of the container name).

I tried to run the `make test` comman on Jenkins, but it failed as the checkout directory was storj_main (or storj_debug).

Change-Id: I85b060ebd0705a2a9fcaefa17033c61a0509c4a2
2022-06-09 13:37:56 +00:00
Márton Elek
35c022ebf9 satellite/accounting: don't use testplanet in rollup_test if not necessary
TestRollupNoDeletes is very flaky (passes locally but fails in the main branch build).

The exact reason is not clear, but stopping the loop seems to be async, the following lines may not stop the loops immediatelly which is a potential problem:

```
satellitePeer.Accounting.Rollup.Loop.Pause()
satellitePeer.Accounting.Tally.Loop.Pause()
```

Fortunatelly these test check only the database interfaces. Instead of testplanet.Run we can run only satellitedbtest.Run which is faster and more predictable (no background loops).

Other potential problem: comment claims that the default of DeleteTallies is false:

```
// In testplanet the setting config.Rollup.DeleteTallies defaults to false.
```

But it seems to be true (rollup.go):

```
DeleteTallies bool          `help:"option for deleting tallies after they are rolled up" default:"true"`
```

This is also fixed in the patch (as we need set it explicit), but TBH it can be fixed with testplanet, too.

Change-Id: Id7ec80d5c069bed2c556f4d001c71aa23fc5af23
2022-06-09 13:11:45 +00:00
Vitalii
cce0a18bf8 web/satellite: merge HeaderedInput and HeaderlessInput components
Merged two components into single one to have single source of truth.
Also this may fix some go-rod test problems.

Change-Id: Iffa86d1b3c24a0e2a551335ecda721a93ff616e3
2022-06-09 10:27:59 +00:00
Vitalii
289be4f1e9 web/satellite: removed confusing line smoothing for chart lines
Fix for this issue
https://github.com/storj/storj/issues/4564

Removed line smoothing for chart lines.

Change-Id: Idb1b9c6786307c48234487517a56fe1a8b76ae98
2022-06-09 08:40:45 +00:00
Jeremy Wharton
4e0acba9ba web/{multinode,satellite,storagenode}: Fix SVG styling
Our SVG loader was recently reworked so that snapshots of UI tests
are smaller, but this broke some SVG styles because SVG paths no longer
contain a scoped CSS attribute. This change adds deep selectors where
appropriate to repair the broken styles.

A scoped CSS attribute is added by Vue Loader to elements of components
with scoped styles to identify which component the element belongs to
and restrict CSS rules accordingly. Paths within imported SVGs no longer
contain this type of attribute, so the deep selector must be used to
style the SVG as if it were a child component.

Change-Id: If4beb903fa3b40aa208efa549b8fe23ccf2a547b
2022-06-08 21:54:59 +00:00
Jeremy Wharton
16d1296613 web/satellite: Fix new access grant flow dropdown
This change resolves an issue where clicking outside an access
grant deletion dropdown would produce errors instead of closing it.

Change-Id: I2e0d5bfc50e2abeffe2d181aa3319b3476138d07
2022-06-08 20:33:15 +00:00
Márton Elek
0831c8f390 makefile: use the ci-slim image for linting
Change-Id: I748d9fb6874a4b399985776e50202399467cfc2e
2022-06-08 14:03:10 +00:00
Egon Elbre
11dd0687c9 satellite/metabase: add missing monitoring
Change-Id: I8b1049aafc0d2c6029a68cac0e6fdc3d7b25b30d
2022-06-08 12:34:00 +00:00