Commit Graph

7294 Commits

Author SHA1 Message Date
Cameron
f06da25c3d satellite/overlay: add nodeevents.DB to satellite overlay service
Add nodeevents.DB to satellite overlay service so we can insert node
events into the nodeevents DB.

Change-Id: I642c0ccc9941ecdb08cb22d5c8cf701959a55156
2022-11-02 15:56:37 +00:00
Michal Niewrzal
d21bbab2b2 satellite: fix metabase configuration wiring
New flag 'MultipleVersions' was not correctly passed from metainfo
configuration to metabase configuration. Because configuration was
set correctly for unit tests we didn't catch it and issue was found
while testing on QA satellite.

This change reduce number of places where new metabase flags needs
to be propagated from metainfo configuration to avoid problems with
setting new flags in the future.

Fixes https://github.com/storj/storj/issues/5274

Change-Id: I74bc122649febefd87f665be2fba628f6bfd9044
2022-11-02 15:17:34 +00:00
Egon Elbre
947cdf0fb4 ci: fixes for new Go version
We need to make exceptions for older uplink versions, because it does
not compile with newer Go versions due to quic dependency.

Change-Id: I3e073694f0942029c56740f0689088058ee068c3
2022-11-02 15:17:46 +02:00
Qweder93
5e5d6ecf6c satellite/{accounting, buckets}: added feature that allows to use custom SQL query instead of object loop for buckets accounting
since amount of objects is growing and looping through all of them
starts taking lot of time, we are switching for SQL query to do it
in chunks of tallies per bucket. 2nd part of issue fix.

Closes https://github.com/storj/team-metainfo/issues/125

Change-Id: Ia26bcac0a7e2c6503df9ebbf4817a636841d3284
2022-11-02 09:01:33 +00:00
Ethan
9a09d8920e satellite/gc: Upload bloomfilters with prefix and update LATEST when complete
Change he bloomfilter generation process to prefix the objects with a date and update the LATEST object with the prefix.  The sender will read the LATEST file to get the prefix to process.

Change-Id: Iae0d3c49015d57f391d87789fb799a7d774066bf
2022-11-01 21:24:46 +00:00
Ethan
4efde65c9e satellite/gc: Optionally run the GC bloomfilter process once, instead of in a loop
The current deployment strategy requires that the GC bloomfilter generation process executes only once and exits.

Change-Id: I952991f126596aa165d1f2e9fce6f8548c21bdba
2022-11-01 18:19:40 +00:00
Wilfred Asomani
4edef9e05c web/satellite: add AB test for new upgrade banner
This change adds AB testing for a new upgrade banner and sends a hit event when "Upgrade" is clicked.

Change-Id: Ie463e224af5b0cb74a601b68eedb2b34f9089fd7
2022-11-01 16:39:52 +00:00
Cameron
11e229cc7f satellite/nodeevents: implement node events DB
Implement node events DB with Insert and GetLatestByEmailAndEvent. Get
was changed to GetLatestByEmailAndEvent so we can verify items are being
inserted into the table without needing the ID, which is not available
to us in the tests.

Change-Id: I4abe63631c44774cd7e795fbab0cbab4d801db4c
2022-11-01 16:03:14 +00:00
Egon Elbre
4b05beb3f5 cmd/satellite: add 'repair-segment' command
This mostly wiring up the necessary systems.

Change-Id: I9e03b3240235ca8e4cc51ddf6a41e27d9dbc198c
2022-11-01 14:31:39 +00:00
Erik van Velzen
e09c8198d5 web/satellite/static/emails: set dcs support address
Helene: support@storj.io is for node operators. Support for DCS accounts
should go to supportdcs@storj.io.

Fixes https://github.com/storj/customer-issues/issues/354

Change-Id: Icfd437004defcee6d49f19e58bdea16e7e99b7d8
2022-11-01 00:06:24 +00:00
Lizzy Thomson
23917e51f2 satellite: invalid coupon code returns 400 status code
fixes: https://github.com/storj/storj/issues/4782

Change-Id: Idcc00202ed91a10c280f706e14d4901bf2f61541
2022-10-31 13:43:09 -06:00
Cameron
f76abb3b3d satellite/satellitedb: add dbx read method to node_events
Change-Id: I9cfcb467051921a9b54370e426e618a76ac4197e
2022-10-31 16:31:31 +00:00
Egon Elbre
aeb645d32b all: replace deprecated ioutil
Change-Id: I60b0bbf5b68b066e2d44b8b99438594d600a3c2d
2022-10-31 15:50:41 +00:00
Erik van Velzen
c25391e976 satellite/metabase/rangedloop: uuid generation
Create helper function to generate ranges of UUIDs, for parallelizing
the segment loop.

Change-Id: I17dbc1d5effe27fc1a3491aa9ca56c692bd95df0
2022-10-31 16:05:41 +01:00
Jennifer Johnson
1da7520a88 cmd/tools/node-cleanup: swap deletion for atredis emails with last contact and created at timestamps
Change-Id: I4acde09e00ea4e5450dcb0fd26bbb2c3afa767f6
2022-10-31 14:13:56 +00:00
Igor
982b9a1844
scripts/draft-release.sh: add missing quotes 2022-10-31 14:15:49 +02:00
Cameron
80bfa55b8e satellite/satellitedb: edit node_events migration to have different PK
Change node_events schema to use an id column as primary key rather than
node id because there can be multiple events per node id.

Change-Id: I518d8ef9ea658764876483e282a4058d3c4910f4
2022-10-28 22:59:28 +00:00
Ethan
403003c294 storj: bump storj common package version
upgrade common to address connection leak

Change-Id: I4b8332120d803cace2856ef0248d73fa99469424
2022-10-28 21:38:47 +00:00
Cameron
6bf54bfaef satellite/satellitedb: add new table node_events
Add new table for node events. We can use this to notify node
operators of certain node events. Further, we can squash events for
multiple nodes with the same email into a single notification.

Change-Id: Icea6dd939df8fe4a98806bd79c014e21d239c43e
2022-10-28 19:16:25 +00:00
Clement Sam
d625eb85fd storagenode: use bytes instead of bytes*hour unit for used space graph
Closes https://github.com/storj/storj/issues/5146

Change-Id: I1b135da81a68193b5b50c761088d79471ca3a2fe
2022-10-28 18:42:45 +00:00
JT Olio
ec9d63d0ca mod: bump dependencies
bump storj.io/uplink to include small telemetry changes

Change-Id: Ia7767244e4eb8d1a86656a1e718576aa4b46218b
2022-10-28 14:53:42 +00:00
Malcolm Bouzi
c565c5e5fc web/satellite: update buckets page table to match designs.
see: https://github.com/storj/storj/issues/5175

Change-Id: I58de009f743196509ae39612a4201d36b7225e96
2022-10-28 10:09:24 -04:00
NickolaiYurchenko
d0cd39af7b web/satellite: session timer reset added while uploading files added
problem: when uploading large files we face a problem with session expire while waiting for file to upload
solution: added callback to progress event to reset session timer
timer is refreshed only when less than 3 min left

Change-Id: Ieab9250be61757762a884165902cd48f90048409
2022-10-28 09:23:50 +00:00
paul cannon
0c3dd44490 docs: audit-scaling: clarify process structure
and clarify some related implementation details.

Most notably, this change clarifies that the verification audit workers
and reverification audit workers are meant to live in a process or
processes separate from the satellite core, and outlines an extra queue
that will be used for communication with the core.

It's not entirely clear to me that this is the right approach; we would
save some fairly significant implementation time by leaving both types
of worker in the core. That would make it necessary to reconfigure and
restart the core when we wanted to change the number of verification
and/or reverification workers, and scaling would be limited to the
computational capacity of the core vm, but maybe those are acceptable
conditions.

Another option would be to leave the Verifier workers in the core, and
having a separate process for Reverifiers. That would be sort of a
middle way between the two above.

Change-Id: Ida12e423b94ef6088733b13d5cc58bdb78f2e93f
2022-10-27 20:00:36 +00:00
paul cannon
c54c45c9c7 satellite/audit: new ReverifyPiece implementation
ReverifyPiece() is not currently hooked up to anything, but is planned
to take the place of audit.(*Verifier).Reverify().

ReverifyPiece() works by downloading one piece in its entirety, rather
than pulling an entire stripe across many nodes.

Change-Id: Ie2c680f4d3c3b65273a72466a3f9f55c115b0311
2022-10-27 16:06:21 +00:00
paul cannon
9c67f62fe3 satellite/satellitedb: add table for reverify queue
This table will be used as a queue for pieces that need to be reverified
(a regular audit timed out on the owning node, so now that node is
contained and we need to validate the piece before un-containing it).

Refs: https://github.com/storj/storj/issues/5228

Change-Id: I5dcd26b6adced8674cbd81884c1543a61ea9d4c8
2022-10-27 15:28:47 +00:00
Wilfred Asomani
2dc2669e22 console/abTesting: add support for AB testing
This change adds support for AB testing using flagship.io

Change-Id: I3e12f5d6cd7248d69adc2c684e4bcff2aadda1df
2022-10-27 10:57:12 +00:00
Kaloyan Raev
689732188b satellite/metainfo: remove duplicate permission check in BeginCopyObject
BeginCopyObject checks twice for write permission in the destination
bucket. One check should be enough.

Change-Id: I3d5935d34f69cd48eaaf00d0117683edfdcefc05
2022-10-27 10:19:05 +00:00
Márton Elek
29aac75d80 build: provides earthfile for nightly build
Earthly is a build tool, it uses buildkitd to create reproducable and highly cacheable builds.

It is used by a new experimental nightly build to easily create storj-up images. (but can be used for any ad-hoc storj-up cluster to create the images).

To make the nightly more robust, I would prefer to commit the helper files (today I do a rebase every time, but sometimes it fails).

More detailed information about Earthly can be found at https://earthly.dev or https://www.youtube.com/watch?v=nChpMEdOaCQ

Change-Id: I683601e0558aca53b45ed3819c46c909534f8b15
2022-10-27 09:25:17 +00:00
Cameron
8c8688ca6b satellite/overlay: return email as part of NodeReputation
Make email accessible for email sending after reputation updates

Change-Id: I760feee0f6ca58b76a2955a04c0c366c618656bb
2022-10-26 17:33:22 +00:00
Cameron
4be042154c sdatellite/reputation: add overlay service to reputation service
Change-Id: I7b1af819f9e1989578335247730378a8658bfe7f
2022-10-26 17:03:29 +00:00
Lizzy Thomson
08052b38bd web/satellite: update date format
Update date range format to include abbreviated month name
Update correlated tests

Change-Id: I5772f4fbc1842c364efd34e2bcc4e37a81026326
2022-10-26 10:20:52 -06:00
Jeremy Wharton
eae71ae06b satellite/reputation: remove tautology in node status comparison
The procedure responsible for node reputation status comparison could
return an invalid result due to comparing a status timestamp against
itself rather than comparing it against another. This results in
unnecessary database updates that could be avoided otherwise.
This change modifies the procedure to resolve this issue.

Change-Id: Id147e1942e994e8bca4ced2a9358f2474927d6ec
2022-10-24 17:14:13 +00:00
Márton Elek
11df98a392
satellite: use evenkit instead of evenstat/top endpoint
We had multiple experiment so far to collect high cardinality data (mainly in aggregated form).

 1. we have a `/top` endpoint which aggregates events with upper bound
 2. we use same api (eventstat) to publish S3 gateway-mt agents to influxdb

This patch starts to replace theses api with jtolio/eventkit. Instead of aggregation all events can be sent to a collector host where we can do aggregation and/or persisting data.

Change-Id: Id6df4882b51d2dbd2be9401ee4199d14f3ff7186
2022-10-24 11:45:12 +02:00
Cameron
a2ca443e29 satellite/overlay: send Node Online emails
Send an email when a node goes from offline to online.

Change-Id: I82d3f9001b9b0669e096d784edf097ffdcb1697d
2022-10-20 18:56:35 +00:00
Egon Elbre
bcbf30a333 ci: make go compatibility check parallel
Change-Id: I40162548305dcd181cfb702f72830d2c0a887cf4
2022-10-20 15:41:03 +00:00
Erik van Velzen
2ed18a0ed6 satellite/metainfo/piecedeletion: configurable threshold
The threshold of piece deletions from the nodes during CommitObject
when overriding an existing object seemed to cause a race condition in
tests.

This change makes the threshold configurable so we can set it to maximum
so CommitObject waits until all pieces are removed from the nodes in the
test.

Change-Id: Idf6b52e71d0082a1cd87ad99a2edded6892d02a8
2022-10-20 16:58:05 +02:00
Wilfred Asomani
56128b972b web/satellite: clear payment account details on logout
this change clears the stored wallet, coupon and priceSummaryForSelectedProject on logout.

see: https://github.com/storj/storj/issues/5252

Change-Id: I2f51d727be97af9a0f26163bb3e93affa35dd2ee
2022-10-20 14:22:34 +00:00
Vitalii
149b59069f satellite/{web, console}: removed old object flow
Removed old flow along with a feature flag

Issue:
https://github.com/storj/storj/issues/5166

Change-Id: I85cfabbf94e910c9d5efb01ef850f6b888a13f71
2022-10-20 16:27:50 +03:00
JT Olio
58a9c55f36 mod: bump dependencies
- storj.io/common

Change-Id: Ib78154acc253a13683495abfdd96d702625fdce8
2022-10-19 17:01:53 +00:00
JT Olio
52ede93086 mod: bump dependencies
- storj.io/uplink

Change-Id: Ic8a180dbc2f08a9300f3d122a012d84e58fa74b0
2022-10-19 09:47:24 -04:00
NickolaiYurchenko
d6f8efb6e3 web/satellite: color palette added
colors moved from figma to style variables
colors from pallete changed from hex to use variables

Change-Id: I0a6adced8f598abc3e1102fcec68cf376ddb53cd
2022-10-19 11:16:26 +00:00
Michal Niewrzal
c63a401399 satellite/accounting/tally: bring back monkit call
We removed monkit call from "object" method because it was using
too much cpu and was visible on cpu profile but we should first try
optimized version of this call.

Change-Id: Ib76d8a2968a704ce47235c6dac6edad4e40bde48
2022-10-19 02:18:14 +00:00
Egon Elbre
2dccfdfb20 certificate/authorization: add golden test for Group marshaling
gob encoding/decoding private keys doesn't work in Go 1.19.
First let's add golden tests to ensure that the fix will work.

Change-Id: I6782c4083786e7c721d112e705c119e8e71610fc
2022-10-18 14:19:03 -04:00
Egon Elbre
4fd59fb3c9 go.mod: bump common
Change-Id: I1ea76bad3d1fbe0cb78b8ab23a1512c6007a166f
2022-10-18 13:21:49 -04:00
Yaroslav Vorobiov
34a88374fc web/satellite: remove add storj token deposit code
Removes unused front-end code to add storj token deposit since
coinpayments was disabled.

Change-Id: I4ce9a551727b523961e87307950125f69785b5d2
2022-10-17 14:27:43 +02:00
Yaroslav Vorobiov
e014e88cc7 satellite/payments: remove Deposit from Tokens interface
Change-Id: Ie04c35410baf8bf2c74cca0b7df1236a80f00e1b
2022-10-17 13:32:06 +02:00
Qweder93
fa287b8206 satellite/metabase: added CollectBucketTallies
One of two parts to stop using objects loop for bucket accounting,
this method collects bucket tallies from list of bucket locations

part1 of: https://github.com/storj/team-metainfo/issues/125

Change-Id: Id2d492582453e28463cddf1245622fb7f191050c
2022-10-15 18:31:06 +00:00
Egon Elbre
22c0b0ac5c cmd/tools/segment-verify: don't mark node immediately offline
Rather than marking node immediately offline, wait for more failures
until removing from the set.

Change-Id: I4363294a75d7d2844afc1f9c0025f664f933c2d7
2022-10-14 08:10:26 +00:00
Egon Elbre
a80a0ebeae cmd/tools/segment-verify: redial once rather than giving up
It's quite likely to hit some timelimit, rather than giving up
immediately, let's retry after the throttle amount.

Change-Id: I20944b058d771f5d4bfa0eea7a2c26cefcd74739
2022-10-14 01:18:31 +00:00