Commit Graph

7266 Commits

Author SHA1 Message Date
Márton Elek
29aac75d80 build: provides earthfile for nightly build
Earthly is a build tool, it uses buildkitd to create reproducable and highly cacheable builds.

It is used by a new experimental nightly build to easily create storj-up images. (but can be used for any ad-hoc storj-up cluster to create the images).

To make the nightly more robust, I would prefer to commit the helper files (today I do a rebase every time, but sometimes it fails).

More detailed information about Earthly can be found at https://earthly.dev or https://www.youtube.com/watch?v=nChpMEdOaCQ

Change-Id: I683601e0558aca53b45ed3819c46c909534f8b15
2022-10-27 09:25:17 +00:00
Cameron
8c8688ca6b satellite/overlay: return email as part of NodeReputation
Make email accessible for email sending after reputation updates

Change-Id: I760feee0f6ca58b76a2955a04c0c366c618656bb
2022-10-26 17:33:22 +00:00
Cameron
4be042154c sdatellite/reputation: add overlay service to reputation service
Change-Id: I7b1af819f9e1989578335247730378a8658bfe7f
2022-10-26 17:03:29 +00:00
Lizzy Thomson
08052b38bd web/satellite: update date format
Update date range format to include abbreviated month name
Update correlated tests

Change-Id: I5772f4fbc1842c364efd34e2bcc4e37a81026326
2022-10-26 10:20:52 -06:00
Jeremy Wharton
eae71ae06b satellite/reputation: remove tautology in node status comparison
The procedure responsible for node reputation status comparison could
return an invalid result due to comparing a status timestamp against
itself rather than comparing it against another. This results in
unnecessary database updates that could be avoided otherwise.
This change modifies the procedure to resolve this issue.

Change-Id: Id147e1942e994e8bca4ced2a9358f2474927d6ec
2022-10-24 17:14:13 +00:00
Márton Elek
11df98a392
satellite: use evenkit instead of evenstat/top endpoint
We had multiple experiment so far to collect high cardinality data (mainly in aggregated form).

 1. we have a `/top` endpoint which aggregates events with upper bound
 2. we use same api (eventstat) to publish S3 gateway-mt agents to influxdb

This patch starts to replace theses api with jtolio/eventkit. Instead of aggregation all events can be sent to a collector host where we can do aggregation and/or persisting data.

Change-Id: Id6df4882b51d2dbd2be9401ee4199d14f3ff7186
2022-10-24 11:45:12 +02:00
Cameron
a2ca443e29 satellite/overlay: send Node Online emails
Send an email when a node goes from offline to online.

Change-Id: I82d3f9001b9b0669e096d784edf097ffdcb1697d
2022-10-20 18:56:35 +00:00
Egon Elbre
bcbf30a333 ci: make go compatibility check parallel
Change-Id: I40162548305dcd181cfb702f72830d2c0a887cf4
2022-10-20 15:41:03 +00:00
Erik van Velzen
2ed18a0ed6 satellite/metainfo/piecedeletion: configurable threshold
The threshold of piece deletions from the nodes during CommitObject
when overriding an existing object seemed to cause a race condition in
tests.

This change makes the threshold configurable so we can set it to maximum
so CommitObject waits until all pieces are removed from the nodes in the
test.

Change-Id: Idf6b52e71d0082a1cd87ad99a2edded6892d02a8
2022-10-20 16:58:05 +02:00
Wilfred Asomani
56128b972b web/satellite: clear payment account details on logout
this change clears the stored wallet, coupon and priceSummaryForSelectedProject on logout.

see: https://github.com/storj/storj/issues/5252

Change-Id: I2f51d727be97af9a0f26163bb3e93affa35dd2ee
2022-10-20 14:22:34 +00:00
Vitalii
149b59069f satellite/{web, console}: removed old object flow
Removed old flow along with a feature flag

Issue:
https://github.com/storj/storj/issues/5166

Change-Id: I85cfabbf94e910c9d5efb01ef850f6b888a13f71
2022-10-20 16:27:50 +03:00
JT Olio
58a9c55f36 mod: bump dependencies
- storj.io/common

Change-Id: Ib78154acc253a13683495abfdd96d702625fdce8
2022-10-19 17:01:53 +00:00
JT Olio
52ede93086 mod: bump dependencies
- storj.io/uplink

Change-Id: Ic8a180dbc2f08a9300f3d122a012d84e58fa74b0
2022-10-19 09:47:24 -04:00
NickolaiYurchenko
d6f8efb6e3 web/satellite: color palette added
colors moved from figma to style variables
colors from pallete changed from hex to use variables

Change-Id: I0a6adced8f598abc3e1102fcec68cf376ddb53cd
2022-10-19 11:16:26 +00:00
Michal Niewrzal
c63a401399 satellite/accounting/tally: bring back monkit call
We removed monkit call from "object" method because it was using
too much cpu and was visible on cpu profile but we should first try
optimized version of this call.

Change-Id: Ib76d8a2968a704ce47235c6dac6edad4e40bde48
2022-10-19 02:18:14 +00:00
Egon Elbre
2dccfdfb20 certificate/authorization: add golden test for Group marshaling
gob encoding/decoding private keys doesn't work in Go 1.19.
First let's add golden tests to ensure that the fix will work.

Change-Id: I6782c4083786e7c721d112e705c119e8e71610fc
2022-10-18 14:19:03 -04:00
Egon Elbre
4fd59fb3c9 go.mod: bump common
Change-Id: I1ea76bad3d1fbe0cb78b8ab23a1512c6007a166f
2022-10-18 13:21:49 -04:00
Yaroslav Vorobiov
34a88374fc web/satellite: remove add storj token deposit code
Removes unused front-end code to add storj token deposit since
coinpayments was disabled.

Change-Id: I4ce9a551727b523961e87307950125f69785b5d2
2022-10-17 14:27:43 +02:00
Yaroslav Vorobiov
e014e88cc7 satellite/payments: remove Deposit from Tokens interface
Change-Id: Ie04c35410baf8bf2c74cca0b7df1236a80f00e1b
2022-10-17 13:32:06 +02:00
Qweder93
fa287b8206 satellite/metabase: added CollectBucketTallies
One of two parts to stop using objects loop for bucket accounting,
this method collects bucket tallies from list of bucket locations

part1 of: https://github.com/storj/team-metainfo/issues/125

Change-Id: Id2d492582453e28463cddf1245622fb7f191050c
2022-10-15 18:31:06 +00:00
Egon Elbre
22c0b0ac5c cmd/tools/segment-verify: don't mark node immediately offline
Rather than marking node immediately offline, wait for more failures
until removing from the set.

Change-Id: I4363294a75d7d2844afc1f9c0025f664f933c2d7
2022-10-14 08:10:26 +00:00
Egon Elbre
a80a0ebeae cmd/tools/segment-verify: redial once rather than giving up
It's quite likely to hit some timelimit, rather than giving up
immediately, let's retry after the throttle amount.

Change-Id: I20944b058d771f5d4bfa0eea7a2c26cefcd74739
2022-10-14 01:18:31 +00:00
nadimhq
c256da5eca
docs/testplan: Testplan for Session Management (#4800) 2022-10-14 02:20:07 +02:00
Wilfred Asomani
3acd133c55 web/satellite: change token payment link to etherscan
This change uses the more accurate "View on Etherscan" for storjscan transactions on the native payment history table.

see: https://github.com/storj/storj/issues/5249

Change-Id: I07ba88f63417bb6a95f50be8e54f6bf0e8a4bd95
2022-10-13 21:39:07 +00:00
Cameron
a52f766273 satellite/overlay: add email-sending functionality to overlay service
We want to send emails to SNOs. Node status changes go through the
overlay service, so it's a good place to add the mail service.
Add the mailservice.Service, satellite address, and satellite name to
overlay service. Also add feature flag --overlay.send-node-emails

Change-Id: I3bd2cb3bf22f9724954ce2374f8b651b902b3a24
2022-10-13 18:01:05 +00:00
Lizzy Thomson
419fb9cf20 web/satellite: update color and add y-axis on bandwidth graph
change color of bandwidth graph to match figma designs, remove text explaining limits, and add y axis to display on usage/bandwidth graphs

fixes https://github.com/storj/storj/issues/5200

Change-Id: I3c8d08a70bb5e99b7a91ca12678be8ca8ff0e6bf
2022-10-13 17:04:38 +00:00
Fadila Khadar
35f74b78e0 satellite/metabase: IterateLoopSegments accepts ranges
Fixes: https://github.com/storj/storj/issues/5207

Change-Id: I7872696068320987825de2d381f57ea503736e89
2022-10-13 14:12:40 +00:00
Wilfred Asomani
fac638fc7d web/satellite: rewrite VInput with composition API
This change updates VInput with composition API, and fix resulting test failures.

see: https://github.com/storj/storj/issues/5115

Change-Id: Ic6870f4faef51de168ef077952b7ce6a39562e6e
2022-10-13 13:33:06 +00:00
Wilfred Asomani
b23b3e02e6 web/satellite: rewrite VButton with composition API
This change updates VButton with composition API, updates .eslintrc.js to ignore resulting import errors, and fix resulting test failures.

see: https://github.com/storj/storj/issues/5114

Change-Id: Iaf9e8323333c797936d3acb38dd32f6e27d1ed08
2022-10-13 12:54:39 +00:00
Erik van Velzen
0cfbd34da2 docs/blueprints: byte range multipart copy
Design proposal for approval to support S3's UploadPartCopy. Major
revision 2.

Change-Id: I9cbe934a773d3496101f32afb19492ea75f48061
2022-10-13 12:03:17 +00:00
Márton Elek
ea106f8894 go.mod: bump storj/private
Change-Id: I087b39c267c6492c4a5c85f470631b2f206cdf39
2022-10-13 11:23:57 +00:00
Cameron
98fed4bc30 {satellite/console,web/satellite}: get project salt from satellite
Add getSalt to projects api. Add action, GET_SALT, on Store
Projects module to make the api request and return the salt
string everywhere in the web app that generates an access grant.
The Wasm code which is used to create the access grant has been
changed to decode the salt as a base64 encoded string. The names
of the function calls in the changed Wasm code have also been
changed to ensure that access grant creation fails if JS access
grant worker code and Wasm code are not the same version.

https://github.com/storj/storj-private/issues/64

Change-Id: Ia2bc4cbadad84b066ca1882b042a3f0bb13c783a
2022-10-12 19:06:27 +00:00
Jeremy Wharton
1887660678 web/satellite: update custom HTML styling for registration page
Add styling for strong text in `customHtmlDescription` for
alternate signup landing pages in `registrationViewConfig`.

Change-Id: Ie688815713b84edb46532bc000152625ccb38ca7
2022-10-12 16:15:31 +00:00
Michal Niewrzal
e5ac8430c3 satellite/metainfo: delete pieces from nodes on object commit
We have new flow where existing object is deleted not on begin
object but on commit object. Deletion on commit object is still
missing deletion from storage nodes. This change adds this part
to the code.

Fixes https://github.com/storj/storj/issues/5222

Change-Id: Ibfd34665b2a055ec6c0d6e260c1a57e8a4c62b0e
2022-10-12 15:02:24 +00:00
Egon Elbre
dd60318147 cmd/tools/segment-verify: use resolved ip
Change-Id: I3662aaea3ff8721c415c038b2b5324d165b60975
2022-10-12 12:43:11 +00:00
Clement Sam
d800b51dd9 cmd/multinode: generate identity when not provided
Closes https://github.com/storj/storj/issues/4974

Change-Id: I7a07d846be7ac8f8f7d7e9ad61511ff84d2ab400
2022-10-12 12:01:14 +00:00
JT Olio
9c09a8dbd5 cmd/uplink: don't send TODO as the instance id
there's not really anything better to send. uplinks have
rotating node ids on each startup, so that's not right
here. i don't think anyone will use instance id for
uplinks so let's just fold and send nothing.

Change-Id: I2511605e95eba1816d662d385b28d5feab8c4eb0
2022-10-12 06:56:19 +00:00
Yaroslav Vorobiov
5cfd64579e private/blockchain: fix Hex to add 0x prefix
Change-Id: I47da9cea160fe9bbdf452956836e3d4b5b44646f
2022-10-11 19:15:24 +00:00
Egon Elbre
8b70f969b6 all: fix nolint directives
Change-Id: I261c8b12e4961e6401cc4024fa5abc35b1a5efa6
2022-10-11 18:31:20 +00:00
Malcolm Bouzi
773683221d web/satellite: add right border to desktop navbar to match designs.
Change-Id: I62e473fd86e16eacec2f367d232fc9f99b2f5e66
2022-10-11 17:42:03 +00:00
Vitalii
2e6b473aba web/satellite: improved modal experience
Made modals to close on background click.
Refactored new folder and object details modals to use VModal component.

Change-Id: Ib7c6ecb576507f2088c0c90fab262113ac7c0760
2022-10-11 16:49:11 +00:00
Michal Niewrzal
1c9f4d1824 satellite/metainfo: more tests for CopyObject
Change-Id: Ibb29f348a15f4afe5d75f9a81987cff4633ceb56
2022-10-11 15:57:23 +00:00
Egon Elbre
ff22fc7ddd all: fix deprecated ioutil commands
Change-Id: I59db35116ec7215a1b8e2ae7dbd319fa099adfac
2022-10-11 15:27:29 +00:00
Michal Niewrzal
11d1e623b5 satellite/metabase/segmentloop: don't do rate limiting if disabled
We have a code to  limit segments loop in case it will hit DB to hard
but so far we didn't use this loop feature  in production. This is a
simple change to avoid logic responsible for rate limiting and its
monitoring if limiting is disabled (RateLimit = 0)

Change-Id: I43e07b407c6e65cf252303159d052eef250d1bea
2022-10-11 10:55:30 +00:00
Egon Elbre
142a04f208 cmd/tools/segment-verify: add connection pool
Change-Id: If0f85edbf99438ac41c23fc7107fdab926288cc2
2022-10-11 09:06:44 +03:00
Egon Elbre
8916f2ee92 cmd/tools/segment-verify: allow ignoring specific nodes
This adds a new flag that allows to ignore some nodes completely.

Change-Id: I203d25f931262c809037e25e9c37e9a89bf47026
2022-10-10 20:14:38 +03:00
Egon Elbre
9e50d837e3 cmd/tools/segment-verify: add tool for summarizing log
Change-Id: I3177ab71dfd25e11adfedce32a530d83dda63bd6
2022-10-10 20:02:50 +03:00
Egon Elbre
5f01dad3a3 cmd/tools/segment-verify: add total progress indicator
Change-Id: Ib729abf6adbeba8d94e08c7e11497c6d5ddd5ec2
2022-10-10 20:02:30 +03:00
Michal Niewrzal
db1409eea6 satellite/metabase: use SUBSTRING with objects iterator
Until this change we were stripping prefix from object key on satellite side. Because of that we were transferring over network unnecessary data
from DB. This change adjusts iterator SQL queries to use SUBSTRING to
remove prefix on DB side and avoid sending it to satellite.

Benchmark against 'main':
unfortunately "time/op" is very unstable while doing local bench in this
case and sometimes  there is no difference in time and sometimes its up to 18%. I never saw results when old solution is faster then new one. Results for "alloc/op" and "allocs/op" are rather consistent.

name                                                 old time/op    new time/op    delta
NonRecursiveListing/Cockroach/listing_no_prefix-8      1.98ms ± 6%    2.05ms ±23%     ~     (p=1.000 n=9+10)
NonRecursiveListing/Cockroach/listing_with_prefix-8    3.97ms ± 8%    3.42ms ±20%  -13.86%  (p=0.005 n=10+10)
NonRecursiveListing/Cockroach/listing_only_prefix-8    8.42ms ±16%    7.58ms ± 5%   -9.91%  (p=0.002 n=10+10)

name                                                 old alloc/op   new alloc/op   delta
NonRecursiveListing/Cockroach/listing_no_prefix-8      16.7kB ± 0%    16.9kB ± 0%   +1.16%  (p=0.000 n=10+10)
NonRecursiveListing/Cockroach/listing_with_prefix-8    27.3kB ± 0%    28.2kB ± 0%   +3.31%  (p=0.000 n=10+10)
NonRecursiveListing/Cockroach/listing_only_prefix-8    60.0kB ± 0%    62.4kB ± 0%   +3.93%  (p=0.000 n=10+8)

name                                                 old allocs/op  new allocs/op  delta
NonRecursiveListing/Cockroach/listing_no_prefix-8         312 ± 0%       315 ± 0%   +0.96%  (p=0.000 n=10+10)
NonRecursiveListing/Cockroach/listing_with_prefix-8       526 ± 0%       541 ± 0%   +2.85%  (p=0.000 n=10+10)
NonRecursiveListing/Cockroach/listing_only_prefix-8     1.16k ± 0%     1.23k ± 0%   +5.24%  (p=0.000 n=10+10)

Change-Id: I23e501494ededafb2dd5ea903e8e4e313b42e956
2022-10-10 14:27:26 +00:00
nadimhq
c921cd5ccf
docs/testplan: Adding a testplan for Token Payment Processor (#4672)
* docs/testplan: Adding a testplan for Token Payment Processor

This testplan is going to cover the token payment processor, it will go over must haves and additional features that we can add on later.

* docs/testplan: Adding a testplan for Token Payment Processor

Updated final testplan for review

* docs/testplan: Adding a testplan for Token Payment Processor

Removed unnecessary tests related to mempool scanning

* Update token-payment-processor-testplan.md

Co-authored-by: littleskunk <jens.heimbuerge@googlemail.com>
2022-10-10 15:01:14 +02:00