Commit Graph

2172 Commits

Author SHA1 Message Date
Cameron Ayer
a7cda642a5 satellite/repair: add logging for irreparable segments in checker
If the checker sees an irreparable segment, log out some info
so we can see what the problem is

Change-Id: I76eda5270214205f4fefc646d6c391cc13ddcafd
2021-09-02 12:35:29 -04:00
Moby von Briesen
6d876acfbf satellite/console: UpdateProject changes
* Disabled updating project for paid tier users who have 0 storage or
bandwidth limits set (these users have been disabled and should not be
able to increase their limits through the UI)
* Better error handling
* Removed unnecessary type conversions

Change-Id: I1e07a1909a4ba877718aa944848f36382f7dbbe2
2021-09-01 20:30:26 +00:00
Yaroslav Vorobiov
ee4361fe0d satellite/audit: fix segment stripes length calculation
GetRandomStripe function to randomly select a segment stripe to
audit was using `segment.EncryptedSize/segment.Redundancy.StripeSize()`.
Since integer divsion truncates it leads to skipping last stripe if
its size is less than stripe size. Use `Redundancy.StripeCount` to
get correct stripe count.

Change-Id: Ida09e035be30a21219ab3e1aedd66af8be707d1b
2021-09-01 13:25:20 +03:00
Ivan Fraixedes
2c50210c94 satellite/admin: Don't update project desc when empty
Don't update the project description if the request body has the
description field set to an empty string.

This follows the same convention used for updating an user's account.

Change-Id: I027047e609760e033cf4b233b1be352c6bf0ec8f
2021-09-01 04:53:16 +00:00
Yingrong Zhao
b64d8084e1 satellite/audit: fix metric reporting when fail to complete an audit
Change-Id: I39df8d4291db35afbba824281cb23438a91c45db
2021-08-31 17:02:30 +00:00
Yingrong Zhao
5304f672ad satellite/payments: fix invoice generation to include the last day of
the month

The Stripe API had a bug before that it wasn't calcualting the input
timestamp based on correct timezone. We had a workaround to not include
the last day of the month in our code when submitting to Stripe.
Now, Stripe has fixed the issue. We need to remove the workaround and
include the last day of the month into our invoice generation

Change-Id: Ic6364ed071be73a19f0b0b46f274a02fb2489db5
2021-08-31 16:25:18 +00:00
Moby von Briesen
2fafc0e16a satellite/console: Add CORS test
This includes basic testing around CORS headers for the register
endpoint.

Change-Id: Ieab2cc8a3c23625307e38026b2ab2c5f4984637e
2021-08-31 15:48:52 +00:00
Egon Elbre
707c4a7b0e satellite/metabase: use implicit tx when batch deleting
Change-Id: I2dc577b95209674b626a76633dedf5b5b1641db0
2021-08-30 15:04:35 +03:00
Cameron Ayer
51fdceafef satellite/repair: increment repair_too_many_nodes_failed with 0 for redash alerting
Change-Id: I990c8df7be30493705278b24954262834a1ed81f
2021-08-27 17:42:11 +00:00
Cameron Ayer
28cb690618 satellite/audit: log error and increment metric if shares cannot be verified
If we encounter an error during the infectious error correction, we just
add it to the errlist to be logged at the worker level.
We want to make sure we know about this if it happens. Give it its own
error log and increment a monkit metric.

Change-Id: Ie5946ae3cd97b766e3099af8ce160a686135ee27
2021-08-27 15:28:16 +00:00
Cameron Ayer
26f839a445 satellite/repair/repairer: if not enough nodes for repair order limits, increment metric and log as irreparable segment
Change-Id: I4bd46f28d64278c8d463e885ad221aafb6ce7cf3
2021-08-27 13:42:28 +00:00
Michał Niewrzał
2340429ea8 satellite/metabase/zombiedeletion: add missing test case
Additional test case where user is uploading 3 segments
each within 23h interval and when zombie deletion process
is executed then nothing is deleted because last segment
was uploaded in less then 24h.

Change-Id: I7426d6fe2c7e9b88c054a01408910c986bcf8d5f
2021-08-27 08:57:06 +00:00
Cameron Ayer
24e02b6352 satellite/{audit,orders}: if not enough nodes for audit order limits, increment metric and wrap error with ErrNotEnoughShares
Increment a metric so we can get alerts. Wrap the error so we can search
the logs for it.

Change-Id: I3827aa306c431009828014d9d9afff8dfc057ee6
2021-08-26 20:14:05 +00:00
Yingrong Zhao
445ebf86c0 satellite/satellitedb: allow setting rate limit for a project to 0
If a user starts to abuse our system, we need the ability to manually
disable their access

Change-Id: Ia31964cedb830323da69c1153a84d610cf8310cd
2021-08-26 13:45:01 +00:00
Michał Niewrzał
5c91ecd271 satellite/metainfo: endpoint cleanup
Two small cleanups:
* merging private commitObject, commitSegment,
makeInlineSegment with its public versions. We were
using it when pb.Pointer was still used.
* removing unused CreatePath method

Change-Id: Ib18b07473d91259335dab874559ef52412ab813d
2021-08-26 09:52:38 +00:00
Michał Niewrzał
c29734ef64 satellite/accounting/tally: remove commented metrics
This change removes metrics which were moved to metrics chore.

Change-Id: I5ddff428cae10ac5e94d8f45691d698958579a61
2021-08-26 09:29:55 +00:00
Moby von Briesen
7b4a09c1ea satellite/console: Allow basic headers in CORS preflight
Currently, post requests originating from www.storj.io are failing
because the content-type header is not permitted for the /register
endpoint.

Change-Id: I14f21c2a27d8f0c8bae3609a5bf4b1e69a72c119
2021-08-25 12:47:14 -04:00
Moby von Briesen
101bbc4414 satellite/console: Fix flaky TestMFA test
Change-Id: Id4fa072f94a4102f3df1bd3aa99984bdd0724497
2021-08-24 17:42:59 +00:00
Michał Niewrzał
b7555980c4 satellite/metainfo: add zombiedeletion chore
This chore will be deleting zombie objects and it's segments.

Change-Id: Ia5d92dd6c3f52e178d4d43d9603732310761e027
2021-08-24 08:22:06 +00:00
Jeremy Wharton
6a6cc28fc1 satellite/console,private/web: Rate limit coupon code application
Rate limits application of coupon codes by user ID to prevent
brute forcing. Refactors the rate limiter to allow limiting based
on arbitrary criteria and not just by IP.

Change-Id: I99d6749bd5b5e47d7e1aeb0314e363a8e7259dba
2021-08-23 17:00:31 +00:00
Michał Niewrzał
0344790c20 satellite/metabase: delete zombie object that has no new segments for specific period of time
Added options flag to define after which object won't be marked as inactive. All segments CreatedAt
time needs to be bellow this flag to treat object as inactive.

Change-Id: Ib5cffc776c6ee1b62b51eb8595438f968b42528c
2021-08-23 08:25:38 +00:00
Cameron Ayer
5a1a29a62e satellite/audit: fix containment bug where nodes not removed
When a node gets enough timeouts, it is supposed to be removed
from pending_audits and get an audit failure. We would give them
a failure, but we missed the removal. This change fixes it.

Change-Id: I2f7014e28d7d9b01a9d051f5bbb4f67c86c7b36b
2021-08-20 14:48:27 +00:00
Cameron Ayer
70296c5050 satellite/audit: change wording of audit worker error log
"audit failed" is already used when a node fails an audit. That makes
searching for this higher level audit worker error more difficult.
Additionally, the presence of errors from the audit worker doesn't
necessarily mean the audit failed. Reword the error message to
"error(s) during audit"

Change-Id: I0aab12c73c18d4bd962c5d8ac8a17cabcec022e6
2021-08-20 13:27:16 +00:00
Artur M. Wolff
e452f85163 satellite/metabase: sync batchSizeLimit and ListLimit constants
This change syncs batchSizeLimit and ListLimit constants to prevent
throwing away results returned while listing with a maximum returns
limit.

Change-Id: Ie2425542d945cb88653dcc34c079737bb32320d4
2021-08-20 11:01:46 +00:00
Artur M. Wolff
a5371353bf satellite/metabase/segmentloop: fix rate.NewLimiter call
This change fixes the call to rate.NewLimiter to account for the
"infinite" case.

Change-Id: Ib3f914ca33ad8b981157fd224f077e6ad8d8c644
2021-08-19 17:23:13 +00:00
Egon Elbre
704cad8da4 satellite/console/consoleweb: add endpoint tests
This currently contains tests for both graphql and regular endpoints.

Co-authored-by: Antonio Franco <antonio@storj.io>
Change-Id: I28d7e629b1caa114438d6fbc3abcc079a8ca10a6
2021-08-19 12:22:45 +03:00
Jeremy Wharton
96e39018c7 satellite/console,web/satellite: Allow disabling MFA with recovery code
This change allows users to disable multi-factor authentication
with a recovery code. Previously, users could only disable MFA
with a passcode.

Change-Id: Iec20bf7d8f6781182b81d1f17d9641491dbc8460
2021-08-18 14:34:17 +00:00
Moby von Briesen
ef9a5210a4 satellite/payments: Add command to ensure free tier coupons
This command is intended to be run as part of invoice generation - it
iterates over Stripe customers, and applies the free tier coupon to any
customer who doesn't already have a coupon.

This way, we can ensure that all customers have at least the free tier
coupon before and after invoice generation, in case a different coupon
has expired.

Change-Id: I33a4aff9174049f9e051de53ef65298ca65ed688
2021-08-18 13:48:44 +00:00
Jeremy Wharton
24b39b7559 web/satellite: Warn user if number of MFA recovery codes is low
Adds a banner at the top of the dashboard warning the user if he
has 3 or less MFA recovery codes.

Change-Id: I3271f1fe33501eaeeeca343bf60ca899e80f0fa1
2021-08-18 13:18:34 +00:00
dlamarmorgan
cc083dbdc9 web/satellite,satellite/console: Allow paid tier users to edit limits
Added editable fields to the project details page for Storage Limit and Bandwidth limit. Leveraged existing types when possible.

Added fixed checking into the limits to prevent reducing limits beyond current usage, as well as limiting usage to less than the default paid tier maximum.

Change-Id: I07ce53470919a8a9d4dce56ade6904ede8daf34c
2021-08-18 00:07:10 +00:00
Moby von Briesen
1fa0cfbfe0 satellite/console: Add CORS headers for register endpoint
Set the Access-Control-Allow-Origin header if a request is made to the
registration endpoint from storj.io or www.storj.io. This allows the
client to make the request with no issues.

Change-Id: I0cd747ececfe877fb5f834d8292307fc14e3db5c
2021-08-16 17:46:08 -04:00
Artur M. Wolff
2cd68bf4fb private/lrucache: import from common
Change-Id: Ia1f43d0440fef21122b071b05da59b4cf2689d6c
2021-08-16 10:04:32 +00:00
Bill Thorp
549e799bbe satellite: wrap notfound on failed deletes as DRPC errors
We're seeing BeginDeleteObject in metaclient returning object not found:
metabase: no rows deleted in the Gateway-MT mint tests.  There's a
client check for rpcStatus.NotFound, but the metabase endpoint isn't
wrapping the db error as a DRPC error.

Here's the chain:

  gateway.AbortMultipartUpload()
  project.AbortUpload()
  metainfoClient.BeginDeleteObject() <- understands DRPC errors
  endpoint.DeletePendingObject() <- where this code is
  db.DeletePendingObject() <- returns error

Change-Id: I93991de76487426df0a807b0d1e69fc975196a1a
2021-08-16 09:02:35 +00:00
Jeremy Wharton
ea772a8480 web/satellite: Allow users to see coupon in billing area
This change allows users to see the Stripe coupon applied to their
account in the billing area.

Change-Id: Ie1e810bfb2847f9b0c0bb827d5ca03c16cf5e818
2021-08-14 16:09:54 -05:00
Yingrong Zhao
6c34ff64ad satellite/satellitedb: remove referrence to audit information in
nodes and audit_history tables

This PR removes all code reference to audit_histories table and
```
audit_reputation_alpha, audit_reputation_beta,
unknown_audit_reputation_alpha, unknown_audit_reputation_beta,
```
columns from nodes table.

It also drops audit_histories table from the db since the code
that's referencing it currently are not being used.

Change-Id: Ifcda8db36afb3a333d487ff831f2fdefc8b02a4c
2021-08-13 21:11:28 +00:00
Jeremy Wharton
51ebc564d9 web/satellite,satellite/console: Overhaul password reset
Updates the password reset page to use the new theme.
Adds new endpoint '/api/v0/auth/reset-password'
for password reset.

Additionally, updates the link-clicking mail simulator to only
click links with a specified attribute. Otherwise, the password reset
cancellation link would be clicked before the password reset link
could be accessed, rendering testing impossible.

Change-Id: I8fde74ef7ad980880a7bf6558e3b9ed31509a393
2021-08-12 17:40:53 +00:00
Malcolm Bouzi
92c53afb84 satellite/{payments,console},web/satellite: Adds confirmation step if user already has coupon code applied and wants to replace it
Change-Id: I04d40d3b25bd67e29c043d651541ff300b5379ac
2021-08-11 20:04:23 +00:00
Artur M. Wolff
2f7e670996 satellite/metainfo: allow per-project rate limit of zero
Change-Id: I237c67dfa3d24ed4837175556f72b9c761644435
2021-08-11 11:45:58 +00:00
Cameron Ayer
dc69e1b16e satellite/repair: use mutex instead of channel to collect download errors
Change-Id: I3f958e9cc95126a25f73ccd105e614b51089edc5
2021-08-10 15:29:39 +00:00
Yingrong Zhao
077ec96d94 private/server: use quic implementation from storj.io/common
Change-Id: I820cf6444a3ddccee0d7c647dc84c80b2752068c
2021-08-10 13:32:21 +00:00
Yingrong Zhao
c074a5666b satellite/satellitedb: improve Update query for reputation
Change-Id: Iee140f726cd05c34028c7b532e1f855e2473ddbc
2021-08-10 13:06:13 +00:00
Cameron Ayer
a8f125c671 satellite:{audit,repair}: log additional info when we can't download enough pieces
When we can't complete an audit or repair, we need more information about
what happened during each individual share/piece download.

In audit, add the number of offline, unknown, contained, failed nodes to
the error log. In repair, combine the errors from each download and add
them to the error log.

Change-Id: Ic5d2a0f3f291f26cb82662bfb37355dd2b5c89ba
2021-08-09 22:57:49 +00:00
Michał Niewrzał
260a673327 satellite/metabase: don't delete pieces until query results are received
To optimize memory consumption we where consuming
segment data during processing results from delete
query. Turns out that there is a chance that query will be
rolled-back if something will go wrong while reading
results. In such case its possible to delete pices but
object/segment will be still in DB.
This change removed piece deletion from problematic
place. Pieces are still deleted in batches but are not
limited at the moment. To avoid memory issues object
deletion batch was decreased.

Change-Id: Icb3667220f9c25f64b73cf71d0cf3fdc7e5107c5
2021-08-08 03:43:17 +00:00
Clement Sam
6613133a1c satellite/metabase: return metadata from IterateObjectsAllVersionsWithStatus on demand
Change-Id: I6bb717cf941e2b5fd7c411bd1e5a77fad0835995
2021-08-06 16:20:39 +00:00
Ivan Fraixedes
a60e836baa satellite/admin: Rename endpoints paths entities
Rename the entities in the endpoint paths from singular after we have
agreed that it matches with API design best practices.

Change-Id: I5bdc086c64c6f93e45a13436b65863fc3e8d864e
2021-08-06 10:05:25 +00:00
Ivan Fraixedes
8d75a35e56 satellite/admin: Add new endpoint get list of API Keys
Add a new endpoint to get the list of API keys associated with a
project.

Change-Id: I9b5ed42e226786c03d853bbc8538344b40ee634f
2021-08-06 09:44:06 +00:00
Clement Sam
1f353f3231 segment/{metabase,repair}: change segment created_at column to not accept nulls
This change adds a NOT NULL constraint to the created_at column in the segment table.
All occurrences of CreatedAt as a pointer are changed to non pointer version (metabase, segment loop, etc)

Change-Id: I3efd476ebd1edd3327b69c9223d9edc800e1cc52
2021-08-06 08:16:28 +00:00
Yingrong Zhao
e00b573f8f satellite/overlay: fix UpdateCheckIn comment
Change-Id: I8584895249d7e5be6dbec79974fc9f77b7a1930c
2021-08-06 05:54:00 +00:00
Moby von Briesen
7f02e1f469 satellite/console: Remove config flag for MFA
MFA is complete and we are good to enable it in production. This change
removes the flag that disables MFA by default.

Change-Id: I2f985ae501171bdab505d664b43c8cfc248bad8d
2021-08-05 18:04:36 +00:00
Yingrong Zhao
e4cc965c39 satellite/satellitedb: replace explicit transaction with dbx query for
UpdateReputation

Change-Id: I7c139ededea83d4b58107536c3a031c4f92d6eb4
2021-08-05 17:09:49 +00:00