Commit Graph

4996 Commits

Author SHA1 Message Date
Cameron Ayer
a04495713d satellite/audit: add missing logs for audit failure conditions
Among other conditions, nodes fail audits by returning incorrect
data and by reaching the max reverify count, but we weren't logging
these events. This commit adds the missing logs.

Change-Id: I80749a7e95e8cb97bc8dd7dac1e523e223114b7f
2021-03-18 17:33:11 +00:00
Ivan Fraixedes
4c1098e571 Redis: Update Redis package to last major version
Update the Redis dependency to use the last major production version.
The last version accepts a context parameter in all the network methods
so it allows us to pass it through them.

Change-Id: I34121b2ec3c2728602115c724933ad24c9e6e4fd
2021-03-18 14:19:49 +00:00
Jeff Wendling
09adfc1205 mod: bump monkit to v3.0.10
Change-Id: Ieed0b4161b158edd7c8e7065f200675cfdfdebf8
2021-03-18 13:17:57 +00:00
Vitalii Shpital
cff1052bca web/satellite: save passphrase hash in local storage
WHAT:
store passphrase's hash in browser local storage

WHY:
to make user use the same passphrase

Change-Id: I3d5e601a0257a7c900e8a0ce3cdce953f1174830
2021-03-17 19:50:53 +02:00
Vitalii Shpital
9e20cfb6e0 web/satellite: create passphrase step for objects page
WHAT:
made a common component from AG's generate passphrase component to be reused here.
create passphrase step for objects page

WHY:
used for generating passphrase or setting user's own password for proceeding in upload flow

Change-Id: Iefa106c544a1c5ff04d591d3d400cf06c4c6d853
2021-03-17 15:12:33 +00:00
Vitalii Shpital
b5f83f463b web/satellite: routes and initial components for objects page
WHAT:
routes and components for new upload file feature

WHY:
upload file from browser feature. Step 1

Change-Id: If517fd4513757aeaee64764d1894f1af3e463b55
2021-03-17 15:12:28 +00:00
Egon Elbre
9491df76e7 satellite/satellitedb: optimize StoragenodeAccounting.SaveRollup
Change-Id: I758049872b4ea1ae22657dd9fcb47c228468b1d0
2021-03-16 18:11:40 +02:00
Vitalii Shpital
6a553ec9c5 web/satellite: change banner for beta satellites with URLs
WHAT:
beta satellite top banner's copy is changed to include support/feedback URLs

WHY:
so users using our beta satellite will be able to report feedback somewhere

Change-Id: Ibc349c8b3354b577275fcf1d2b75bfdd267729d9
2021-03-15 17:12:07 +00:00
Isaac Hess
1de6315b83 metainfo: Stub DownloadObject metainfo endpoint
This is the stub "not implemented" endpoint for the new DownloadObject
RPC.

Change-Id: Ifa724e20d89d839f2b6968d6283bc66a3bab8781
2021-03-15 10:52:57 -06:00
Yingrong Zhao
a36d854d8e bump monkit-jaeger to latest
Change-Id: I8c20769e667afd3e22ebd10e2e045d2e15fad7f7
2021-03-15 14:42:03 +00:00
Qweder93
0b060c8297 storagenode/console: estimated payout flaky test skip
Change-Id: Ibb20b5716df94f9430026ed697d57b6be9cc137d
2021-03-15 15:26:47 +02:00
Qweder93
487ad84b23 storagenode:consoleAPI flacky tests for estimatedPayout fixed
joinedAt date now is 2 months before time.Now which should fix calculations on early dates.

Change-Id: I6250ec2463c2dd609e811b74d908a02a3e64f19a
2021-03-13 20:33:25 +02:00
littleskunk
dcdfa10370
jenkins/build: rename release binaries inside zip archive (#4060)
Co-authored-by: Ivan Fraixedes <ivan@fraixed.es>
2021-03-11 12:35:32 +01:00
Cameron Ayer
8138939a2a satellite/satellitedb: remove bad migration step
Migration step 148 will cause errors because we missed some
references to the columns being dropped. Removing the step
altogether causes problems with backwards compatibility tests
because the change already exists in the latest release tag.
To circumvent, we change v148 to an empty migration.

Add methods FindTable and RemoveColumn in private/dbutil/dbschema

Change-Id: Ia527e95b88a88c5dc82800928ce6f8cfb879e334
2021-03-10 14:36:52 -05:00
NickolaiYurchenko
5616795f70 web/multinode: update node name modal
WHAT: markup and logic for set/update node name

WHY: to have abitity to change node name for future purposes like filtering or grouping

Change-Id: I44c679df43d34355efc4c6cff260faa1c3ff480e
2021-03-10 09:44:18 +00:00
Ivan Fraixedes
84b844a2a7 redis-server: Move testing type to specific testing pkg
Move a specific interface & types used for testing to be a private
subpackage with a name that clearly identifies it for testing purpose.

Change-Id: I646cf3b6f0a3b518a6f9a125998dc5a02df02db6
2021-03-10 06:09:46 +00:00
Yingrong Zhao
1f59a08e85 pkg/server: eliminate Retry packet from quic-go hanshake phase
In order to prevent traffic amplification attack, QUIC allows
application to perform an address validation during connection
establishment.
(https://tools.ietf.org/id/draft-ietf-quic-transport-34.html#name-address-validation)
However, this adds another round trip during connection establishment.
In storj network, it does client authentication before servers starts
sending significant amount of data to any client. We believe the traffic
amplification attack isn't going to be significant when turning off
address validation in QUIC. AND it will provide us a significant
performance boost during connection establishment.

Change-Id: I7f9a0ca5ca770b715d08b1e8ce3022fbb2b85d42
2021-03-09 22:05:35 +00:00
Egon Elbre
580eecf0bb mod: bump monkit to v3.0.9
Change-Id: I6b2e6276c4d83f06eab31481fc3826f8d1307dc4
2021-03-09 18:37:03 +02:00
Vitalii Shpital
703ca08b7b web/satellite: remove api keys related logic from client side
WHAT:
remove all the api keys related code

WHY:
it became redundant after access grants implementation

Change-Id: I36344d478d8d7524e3994ea2076491be4add1aa3
2021-03-04 20:43:04 +02:00
Ethan
7af245c3e7 satellite/payments: De-shadow customer page var in stripe mock
Fixes infinite loop when there are more records than the limit.

Change-Id: Ib61a8fcad6d08f4e0f1c5715d710acec607004b3
2021-03-03 20:51:43 -05:00
Jeff Wendling
376547c33c satellite/compensation: smaller txns for RecordPeriod
cockroach is having problems with huge transactions and
having them complete before timeouts or whatever, so
do smaller transactions.

because we can have partial recording of payments which
are not unique, we have to do a thing where we read and
check if it already exists before writing. this is not
concurrency safe.

Change-Id: Ia7d59499a43ce6d70cb2a23754edbdd1b643ef1a
2021-03-02 20:14:25 +00:00
Cameron Ayer
a44974a2f9 satellite/audit: fix pointless containment deletions
Previously if node was not found in containment, it was
given the status, 'skipped'.

We later try to delete skipped nodes from containment.

To fix this, add a new status called 'remove' to differentiate
nodes which should be skipped and nodes which should be deleted.

Change-Id: Ic09e62dc9723c89d0c9f968ce68c039114a9d74e
2021-03-02 13:40:18 -05:00
Malcolm Bouzi
a7ca78a519 web/satellite: Add user account tabs to signup ui
- toggle logic to trigger visual tab change
- toggling tabs renders appropriate fields
- error handling for professional account inputs
- successfully create professional user account

Goal is to be able to target users with specific professional focused communication.

Change-Id: Iffbeb712dac24ea1a83bb374740e0c1cd2100663
2021-03-02 15:02:40 +00:00
Natalie Villasana
c290e5ac9a satellite/orders: decrease FlushBatchSize default to 1000
The previous default FlushBatchSize of 10000 was causing major
slow down in select and insert statements on bucket_bandwidth_rollups.
We saw on the saltlake satellite that a FlushBatchSize of 1000 helped
reduce contention and query latency.

Change-Id: Ib95e73482219bc5aedc11925b1849fa5999774ba
2021-03-02 14:00:48 +00:00
JT Olio
932039ebdf blueprint: layer2 support for zksync
Change-Id: I130ce4c6d04b6f56acdbaefff507c694e8bf8260
2021-03-01 23:41:11 +00:00
Cameron Ayer
aeac6264cd sallite/satellitedb: add metric stray_nodes_dq_count
Add metric so we can see how many nodes are DQd due to
this.

Change-Id: Ie4bdd1375fb9bd948af14fed9a2962b783b6a526
2021-03-01 21:06:36 +00:00
Natalie Villasana
856db68fd9 satellite/gracefulexit: extend GE data cleanup to include exit_progress
The new 'consistency ge-cleanup-orphaned-data' cli command deleted
orphaned transfer queue items, but not entries in the
graceful_exit_progress table. This will delete orphaned entries
from the exit progress table too.

Change-Id: I5f927aac1f258490678deaf179be92ccfe10fcd8
2021-03-01 15:52:43 +00:00
Cameron Ayer
411a7ad0bc satellite/satellitedb: drop uptime_reputation_alpha and uptime_reputation_beta from nodes table
Change-Id: Ib46e783bf1a5c036394b4cac281382d0380bb1be
2021-03-01 15:30:51 +00:00
Moby von Briesen
0370d1553e storagenode/console: Fix TestStorageNodeApi flakiness
TestStorageNodeApi is failing due to very slight differences in
float values. This change rounds these values to 3 decimal places before
comparing them.

Change-Id: Ic7fae3a5e0a0a942c03d982bfa7b19357f2e3d2e
2021-02-26 15:12:35 -05:00
Stefan Benten
3a533640ce
Makefile: downgrade to go v1.15.7 (#4048) 2021-02-26 17:34:57 +01:00
Fadila Khadar
cf0c88c679 cmd/uplink: fix no result when listing single file
uplink ls sj://bucket/file.ext gives a result if file.ext exists.

Change-Id: Ife741926acacf510e4e098009ce1724386c7522f
2021-02-26 11:43:02 +00:00
Egon Elbre
4a2c6f6208 private/dbutil/pgutil: add UUIDArray helper
Change-Id: I469fa2381b7c1a731880a742787bb4842af9b902
2021-02-26 09:56:15 +02:00
Egon Elbre
1d28dc314c private/dbutil/pgutil: move type helpers to separate file
Change-Id: I6357894434b565b9e48dbe5ac9b87542dcd5fc38
2021-02-26 09:56:15 +02:00
Vitalii Shpital
4eb12eb70b web/satellite: create a project on AG name step
WHAT:
right now we've got accounts with PM but without projects.
So when user tries to create AG they get uuid error which means that there is no uuid of a project provided.
So we decided to create default project for such users before actual creating of AG.

WHY:
backwards compatibility

Change-Id: Icab857e711f1020fdcf8a12279153014d9daf11c
2021-02-25 17:41:27 +02:00
Qweder93
f2be856864 storagenode/payouts: estimated payouts calculations upgraded
Added recalculations of estimated income in current month in case
node joined in current month

Change-Id: Iac925fd8bcf2be8f1089e5f001a01448ba7650be
2021-02-25 14:11:44 +00:00
Michał Niewrzał
1af9400a23 satellite/satellitedb/dbx: remove unused methods
Turns out that many methods generated with dbx where not used at all.
Lets remove them.

As a next step we can think about dropping tables like:
* user_credit
* offer

Change-Id: Id6cda81a701348db2a6b8b26daa22ae9c4f87cb4
2021-02-25 13:50:50 +00:00
NickolaiYurchenko
badd6d12d4 web/satellite: distributed amount added to tables, content changed
Change-Id: Ib0eb3fa3f4f9d451f4b4048a6409d19968a41956
2021-02-24 16:01:40 +00:00
Vitalii Shpital
300e88f9a7 web/satellite: config flag for satellites in beta
WHAT:
config flag to indicate if satellite is in beta

WHY:
to avoid using hardcoded satellite names which may cause issues

Change-Id: If92eb7417c340bf343a9a91e2f6b11f0349020c5
2021-02-24 12:29:07 +02:00
Malcolm Bouzi
f3e75c754f satellite/console: add new professional tab fields to create user function in service
Change-Id: I746062ef6f938fa145a3eb29a491399c2cf35806
2021-02-23 18:25:49 +00:00
Vitalii Shpital
c3e13b9b38 web/satellite: change info banner for US2 satellites
WHAT:
added message for us2 users that their data may be deleted at any point of time

WHY:
to make users aware

Change-Id: I4369fc4ec780ab66e7355c664740f8f6f5a32392
2021-02-23 13:53:40 +00:00
Egon Elbre
0178ec7771 satellite/satellitedb: clearer testdata version parsing
Change-Id: I93a23625b30472e826b962dab31ff3e1081b1b8f
2021-02-23 10:22:19 +02:00
Egon Elbre
32afbd0d15 satellite/satellitedb: faster test database setup
Pregenerate the database schema we should use for most tests.
Currently, Cockroach is slow with regards to migration and it's
better if it happens in as few transactions as possible.

This reduces test time from ~21min to ~15min.

Change-Id: Ife8117053e6b9ecf3c93fe63677edf15d4d7c254
2021-02-22 21:13:00 +02:00
Vitalii Shpital
fce5c95eb2 web/satellite: logic for beta satellites terms confirmation
WHAT:
this logic is used on us2 sign up page to accept additional beta terms.

WHY:
to make US2 users know that their data can be removed.

Change-Id: I43f25bc945a5957bacc6662499830cb2dd6d4079
2021-02-22 14:54:51 +00:00
Egon Elbre
21a6dccd36 ci: set specific line coverage targets
Currently the auto-updating keeps moving the target upwards without
allowing for variations in the pipeline.

Let's try these settings:

* >=70% lines covered, to pass
* <=60% lines covered, to mark as unhealthy
* <=50% lines covered, to mark as unstable

Change-Id: Idc75e682345f6d62af34379362c75d321e564d2c
2021-02-22 15:13:18 +02:00
Cameron Ayer
549033f2e6 satellite/satellitedb: don't include DQd and exited nodes in DQStrayNodes
Don't update DQ time of already DQd nodes. Don't DQ nodes who exited.

Change-Id: I4528a9ba9f8e278987165ad337a9b34dadb9788b
2021-02-19 15:12:30 -05:00
Egon Elbre
3c49f0ff73 cmd/satellite: restore-trash, cleanup workers
Use a limiter instead of a custom work queue.

Change-Id: I1506daaaaeb2935f6011ddaeb83a0904abb69c98
2021-02-19 19:08:10 +02:00
Egon Elbre
630718c392 satellite/gracefulexit: fix test DeleteAllFinishedTransferQueueItems
Using random piece num may generate the exact same key or piecenum.
Instead use fixed piecenum and key.

Change-Id: I54b7bc1a6698149bf99608dd46501ea963cec084
2021-02-19 18:15:22 +02:00
Egon Elbre
62a2d05ae6 satellite/accounting: try fix test ResetLimitsFirstDayOfNextMonth
TestProjectUsage_ResetLimitsFirstDayOfNextMonth seems to be flaky.
It doesn't seem to reproduce easily, however it should flush orders from
storagenodes to satellites, that way the orders chore flushing on the
satellite makes sense.

Change-Id: I586d9d8ce10b5f6320e79324f6128b4f8c2cac5f
2021-02-19 17:29:15 +02:00
Egon Elbre
1137620baf satellite/satellitedb: move tests to their domains
Testing interfaces is slightly clearer when it's in the package needing
the database rather than each individual implementation.

Change-Id: I10334c214a205f7e510b939b4359a2214c4e060a
2021-02-19 17:29:15 +02:00
JT Olio
e23f423107 uplink: add api key back to inspect
Change-Id: I9aa46f3910a76a81a399597d7512056b218e55af
2021-02-19 14:46:35 +00:00