Commit Graph

1996 Commits

Author SHA1 Message Date
Egon Elbre
10372afbe4 ci: fix lint errors
Change-Id: Ib5893440807811f77175ccd347aa3f8ca9cccbdf
2021-05-17 13:37:31 +00:00
JT Olio
1852773e3e satellite/contact: rate limit node checkins
Change-Id: Ied386a2350aa073de46443e5259b56d49ec61dbf
2021-05-17 08:15:04 +00:00
Egon Elbre
8f15f975a2 satellite/overlay: improve contended update checkin
Improve UpdateCheckIn on a contended row:

  name                             old time/op  new time/op delta
  UpdateCheckInContended-100x-32   2.29s ±55%   0.17s ±61%  -92.45%  (p=0.008 n=5+5)

Change-Id: I053ab9f1cff136c306e5fb57f5e355cdc0269a8c
2021-05-16 20:41:12 +03:00
Cameron Ayer
dfe85beac2 satellite/satellitedb/dbx: fix duplicated lint line
Change-Id: Ia6677417ace208689eb6a814f2e93448fe57047c
2021-05-14 13:48:51 -04:00
Cameron Ayer
be87c80fe1 satellite/satellitedb: drop columns total_uptime_count and uptime_success_count
These columns on the nodes table are not referenced anywhere and are
ready to be dropped

Change-Id: I3f3652f635108419e93ced12b0ac1e58bba27a32
2021-05-14 11:49:12 -04:00
Cameron Ayer
3ea7aa2c7a satellite/repair/repairer: log piece hash verification failures
Piece hash verification failures during repair download are considered
audit failures, but we are not logging these occurrences. Now we log
them.

Change-Id: If456cebcfda6af7a659be3d1fc74448e681fb653
2021-05-14 15:03:15 +00:00
igor gaidaienko
6ee2210297 satellite/console: add test for time based permission
Add test with NotBefore and NotAfter restricted permission to verify that we don't have an access to bucket

Change-Id: I7ec98a5b02c0098ee7ec81034278398f4435f1cf
2021-05-14 11:39:12 +00:00
Egon Elbre
910eec8eee satellite/metainfo: remove MetabaseDB interface
Currently the interface is not useful. When we need to vary the
implementation for testing purposes we can introduce a local interface
for the service/chore that needs it, rather than using the large api.

Unfortunately, this requires adding a cleanup callback for tests, there
might be a better solution to this problem.

Change-Id: I079fe4dbe297b0ae08c10081a1cea4dfbc277682
2021-05-13 13:22:14 +00:00
Michał Niewrzał
547a6e9930 satellite/metabase: add DeletePart method
Method will delete all segments for given part.

Change-Id: I11bdfdf30176749aee03cea0d1d2fa467f7cebea
2021-05-11 22:45:43 +02:00
Egon Elbre
2ae80690cb satellite/metabase/metabasetest: remove STORJ_TEST_DATABASES
Initially metabase was developed separately and it was useful to have a
separate environment flag for tests, however, it's more convenient to
use the same as rest of the testsuite.

Change-Id: Ia4d79be27ce5911cbae68d57cdf0b30f63459444
2021-05-11 13:31:01 +00:00
Egon Elbre
0858c3797a satellite/{metabase,satellitedb}: deduplicate AS OF SYSTEM TIME code
Currently we were duplicating code for AS OF SYSTEM TIME in several
places. This replaces the code with using a method on
dbutil.Implementation.

As a consequence it's more useful to use a shorter name for
implementation - 'impl' should be sufficiently clear in the context.

Similarly, using AsOfSystemInterval and AsOfSystemTime to distinguish
between the two modes is useful and slightly shorter without causing
confusion.

Change-Id: Idefe55528efa758b6176591017b6572a8d443e3d
2021-05-11 12:40:36 +03:00
Michał Niewrzał
033006403f satellite/metainfo: fix setting object ZombieDeletionDeadline
So far we were setting ZombieDeletionDeadline alwasy as nil and because of that DB default was never set. This change adds separate query for inserting object if deadline is not set.

Change-Id: I3d6a16570e7c74b5304e13edad8c7adcd021340c
2021-05-11 07:26:42 +00:00
Ivan Fraixedes
7fb86617fc satellite/satellitedb: Use CRDB AS OF SYSTEM & batch for GE
Use the 'AS OF SYSTEM TIME' Cockroach DB clause for the Graceful Exit
(a.k.a GE) queries that count the delete the GE queue items of nodes
which have already exited the network.

Split the subquery used for deleting all the transfer queue items of
nodes which has exited when CRDB is used and batch the queries because
CRDB struggles when executing in a single query unlike Postgres.

The new test which has been added to this commit to verify the CRDB
batch logic for deleting all the transfer queue items of the exited
nodes has raised that the Enqueue method has to run in baches when CRDB
is used otherwise CRDB has return the error "driver: bad connection"
when a big a amount of items are passed to be enqueued. This error
didn't happen with the current test implementation it was with an
initial one that it was creating a big amount of exited nodes and
transfer queue items for those nodes.

Change-Id: I6a099cdbc515a240596bc93141fea3182c2e50a9
2021-05-07 13:09:19 -04:00
Egon Elbre
6161436d8b satellite/metabase/metabasetest: sort results
Deletion results are not guaranteed to be sorted, hence we need to sort
them before comparing.

Change-Id: I6b571dd812d4256327c6c64fff74c780c62b2948
2021-05-07 12:36:02 +03:00
Egon Elbre
6e6051b172 satellite/metabase: drop alias migration code
We have migrated all of the satellites and we shouldn't keep dead-code
around.

Change-Id: I539d6766cfafa2f278ff7767ceb2d39f6777ace3
2021-05-07 12:13:28 +03:00
Egon Elbre
2af7e4ef26 satellite/metabase/metaloop: use database time
The system and database time may drift. We should use database time for
absolute "as of system time" to ensure that it's not newer than the
current database time. When the "as of system time" is in the future,
then the query will fail.

Change-Id: I5423f6aaad966ca03a76b5ff805bfba932e44a51
2021-05-07 09:08:04 +00:00
Ivan Fraixedes
38c41d0943 satellite/admin: Fix typo in README
Fix a typo in the README of the Satellite Admin API.

Change-Id: I682a8e2310082b5fc14e69347f0796cc0b0fc0f8
2021-05-06 19:18:12 +00:00
JT Olio
a89a2b4b43 satellite/billing: make stripe invoice generation work with multiregion satellites
multiregion satellites have complex database connection strings
largely due to using a different backend for the repair queue than
cockroach.

billing stuff didn't work right with this.

Change-Id: Ie8759a8c47e71347c3a190abfc9d53945d7b8855
2021-05-06 11:51:03 -06:00
Egon Elbre
69b149a66f mod: bump uplink
uplink stopped using zap, hence some of the private methods needed to be
changed.

Change-Id: Iac1fae45a40cd3f1649b9f672bf8c250344986d5
2021-05-06 14:48:36 +00:00
Ivan Fraixedes
2f8969ae35
satellite/admin: Add a ToC to README
Add an auto-generated table of content to the README for easy to find
and browser the documentation of the available API end points.

Change-Id: Id94d904cefd30449234224072ddc50a181aaba04
2021-05-06 07:25:15 +02:00
Ivan Fraixedes
1504303363
satellite/admin: Add clarifications in README
Add some clarifications to the README of the satellite admin API.

Change-Id: I2f56c8778872a098442ecd601c2b8dcfe1ac19a2
2021-05-06 06:49:11 +02:00
Michał Niewrzał
0d3865950f satellite/metabase: rename delete_expired.go to delete_objects.go
Logic for deleting specific object type can be reused so file name can be more general.

Change-Id: I75182c0d5e1b653eaa454c895f347dd33b4bebbc
2021-05-05 08:57:32 +00:00
Egon Elbre
d2033c2f52 satellite/nodeselection/uploadselection: rename package
Currently nodeselection package only contained state for uploads, move
these to a subpackage, such that we can make another "downloadselection"
for downloads. Then move selection logic from overlay to nodeselection.

Change-Id: I0fc42bcae3a29db2728dae9f3863b1e95bf5165b
2021-05-04 15:50:00 +00:00
Egon Elbre
5269596c7d satellite/{metabase,metainfo}: use ObjectStream as argument
Change-Id: I5a7f096002b4d7a6162b24d7a64346b058c4c89c
2021-05-04 15:30:59 +00:00
littleskunk
d06206488f
satellite/rolluparchive: archiveAge use 90d prod default for storj-sim 2021-05-04 18:30:39 +03:00
Ethan
dba9321483 satellite/metabase: Remove pending_index
Remove pending_index until more performance testing can be done.

Change-Id: I31d29b1f37998eb62aee8bd42c7dcb79859446d2
2021-05-04 09:49:39 -04:00
Michał Niewrzał
e8ef689929 satellite/metabase: add DeleteZombieObjects method
We will use this method to delete objects where zombie_deletion_deadline passed.

Change-Id: Id24322e40dad5b8bceb0f84080922c746c0b802d
2021-05-04 07:08:16 +00:00
Cameron Ayer
bb343d9028 satellite/satellitedb: don't remove offline nodes from containment
When audits are being recorded, we automatically add some SQL to remove
the node from the pending audits table in case it exists. They are
removed from pending audits even if the node was offline for the audit.
This is not the correct behavior.

Add statement to record audit results in reverify tests to ensure no
more false positives.

Change-Id: I186ae68bc5e7962ef6c5defbebc1d95e63596a17
2021-05-03 16:05:55 +00:00
Egon Elbre
a8533042a3 mod: bump uplink
Renamed a private package in uplink and some of the tests need to be
updated.

Change-Id: I8735fc16c2066ecba669618f0c1dadf68273704d
2021-04-30 16:53:54 +03:00
Moby von Briesen
1b736104cb satellite/payments: Apply free tier coupons before preparing invoices
We are already adding the free tier coupons at the end of
InvoiceApplyCoupons, but there is a case where we will charge customers
who do not currently have a coupon the next time invoices are generated.
By applying the free tier coupon before preparing invoice project
records, we cover this case.

Once every customer has a coupon, it will be safe to remove this
functionality and only apply new coupons at the end of invoicing.

Change-Id: I65afbe5c0b84e63eeb1a0221e8d95311d87641a0
2021-04-30 13:26:25 +02:00
littleskunk
396cd5a683
satellite/payments: reduce object fee to 0 (#4104) 2021-04-29 22:31:32 +02:00
Egon Elbre
2c657f594e satellite/metainfo: don't rely on exact error name
Change-Id: I7975a00c32891a43b4f1e6dc4f5847201844f2ec
2021-04-29 18:35:56 +00:00
Ivan Fraixedes
5f29a20937 satellite/satellitedb: Add missing indexes
The DBs of our production satellites have some indexes that we didn't
have in the migrations because at that time we weren't able to add them
because our migration test was not able to deal with Cockroach indexes
with the STORING clause.
We have recently modified the storj.io/private/dbutil/pgutil package to
support the CRDB STRORING clause, so we are adding the missing indexes
to our migrations for being able to have them if we have to recover a DB
from scratch or we deploy a new DB satellite.

Change-Id: I686ff84e5b4c02d9615f50fa531261363affefb8
2021-04-29 15:34:46 +00:00
Egon Elbre
961e841bd7 all: fix error naming
errs.Class should not contain "error" in the name, since that causes a
lot of stutter in the error logs. As an example a log line could end up
looking like:

    ERROR node stats service error: satellitedbs error: node stats database error: no rows

Whereas something like:

    ERROR nodestats service: satellitedbs: nodestatsdb: no rows

Would contain all the necessary information without the stutter.

Change-Id: I7b7cb7e592ebab4bcfadc1eef11122584d2b20e0
2021-04-29 15:38:21 +03:00
Moby von Briesen
4c0817bcfb satellite/payments: Populate new coupons during invoice generation
The previously configured never-expiring coupon does not refill every
month. Eventually, even though it never expires, it will run out. This
commit makes several small changes to address this issue for the free
tier:
* Change the config for the promotional coupon to be $1.65 for 1 month
(the change from $10 to $1.65 is due to our recent pricing changes)
* Update PopulatePromotionalCoupons (PPC for brevity) to add promotional
coupons to users with expired and consumed coupons (all users with a
project and no active coupons should get a new coupon when PPC is called)
* Call PPC at the end of the `create-invoice-coupons` stage of invoice
generation - after current coupons are processed and expired/exhausted.
* Remove legacy admin functionality for PPC from satellite/console - we
do not currently use it, but if we did, it should be in satellite/admin
instead.

Change-Id: I77727b97bef972df32ebb23cdc05055827076e2a
2021-04-28 18:07:05 +00:00
Moby von Briesen
b317f28fdb web/satellite: Update registration files
Allows us to remove the following files from satellite branding
repo, with an up-to-date single source of truth now in storj/storj:
* web/satellite/src/common/registrationSuccess.html
* web/satellite/src/common/registrationSuccess.scss
* web/satellite/src/views/register/registerArea.html
* web/satellite/src/views/register/registerArea.scss

The registrationSuccess files have been removed from all satellites in
the branding repository. The registerArea files have been removed only
from production satellites in the branding repository.

Importantly, this change enables the "resend email" functionality on
production satellites - previously, this functionality was available in
storj/storj, but not our branding repository.

Removes the config for VerificationPageURL, which redirected users away
from the satellite app to storj.io after creating an account. In order
for the email resend button to work, we cannot leave the app.

Adds a new config value for partner satellites, which replaces the
partner satellite names config. The new config includes name and
address. It is validated on setup/run to ensure it can be parsed.

Change-Id: I67db0702d9b9641f1a37b599f2929d56f3c33aca
2021-04-28 16:16:16 +00:00
Fadila Khadar
5af898fcc9 satellite/metabase: add migration step for index on pending objects
Add an index on (project_id, bucket_name) for objects with pending status

Change-Id: Ibd3c949a99e49e8e6bb884564331ec16f9184942
2021-04-28 09:54:10 +00:00
Michał Niewrzał
02460fcc4c satellite/metainfo/expireddeletion: change chore interval to 24h
We want run deletion query more often to avoid long running DB operations.

Change-Id: I72c63bb8ec05cc3ab2e7ab63479ba6cdf830386a
2021-04-27 12:30:23 +00:00
Vitalii Shpital
606a501441 satellite/console: add tardigradeshare.io domain to media-src CSP whitelist
WHAT:
added tardigradeshare.io domain to media-src CSP whitelist

WHY:
we still support it

Change-Id: Id02daa0712cc84f5cb7edaafb21a143859531f26
2021-04-27 11:17:50 +00:00
Malcolm Bouzi
c1fdf1ad71 satellite/satellitedb: add have_sales_contact column on users
For business accounts we need to track the sales contact.
It will be a question to business accounts during onboarding.

Change-Id: I8d101ce1b52091478dfb0ddd875e1cc717d765d3
2021-04-26 13:37:26 +00:00
Michał Niewrzał
7944df20d6 storj: use multipart API
Change-Id: I10b401434e3e77468d12ecd225b41689568fd197
2021-04-26 13:15:09 +00:00
Egon Elbre
948529a600 satellite/metabase/metabasetest: move test utils to a separate package
Change-Id: Iaa607996478497d5c97c6a6973d099b315b3abf0
2021-04-26 13:12:58 +03:00
Egon Elbre
7802ab714f pkg/,private/: merge with private package
Initially there were pkg and private packages, however for all practical
purposes there's no significant difference between them. It's clearer to
have a single private package - and when we do get a specific
abstraction that needs to be reused, we can move it to storj.io/common
or storj.io/private.

Change-Id: Ibc2036e67f312f5d63cb4a97f5a92e38ae413aa5
2021-04-23 16:37:28 +03:00
Egon Elbre
c641ddcb54 pkg/lrucache: rename package
cache is really common variable and type name and we have already used
the package name alias in multiple places.

Change-Id: I6435785b7549b541d533de59ec94557b9bd11e04
2021-04-23 16:37:28 +03:00
Egon Elbre
8c62788b24 satellite/metabase/metaloop: stop timer properly
There's a rare chance that `Stop` returns false, however doesn't have
time triggered. Use a non-blocking drain to remove the token.

Change-Id: I1ae18a197424017f0ca76656602709a029b56bfd
2021-04-23 16:35:21 +03:00
Vitalii Shpital
d32515fa5d satellite/console: add storjshare.io domain to media-src CSP whitelist
WHAT:
whitelist .storjshare.io domain for media-src CSP

WHY:
to enable video preview for linksharing

Change-Id: Ib673602d31ca116e7ce1cee0eba17099a55d7dbc
2021-04-23 16:11:41 +03:00
Egon Elbre
a2e20c93ae private/dbutil: use dbutil and tagsql from storj.io/private
Initially we duplicated the code to avoid large scale changes to
the packages. Now we are past metainfo refactor we can remove the
duplication.

Change-Id: I9d0b2756cc6e2a2f4d576afa408a15273a7e1cef
2021-04-23 14:36:52 +03:00
Egon Elbre
dd5eb2616b satellite/metrics: join for monitoring
We don't want metrics to trigger the metaloop on it's own.

Change-Id: I648272a0e3043173b0a377697db1f172712bdda5
2021-04-23 12:21:48 +03:00
Egon Elbre
c1fbecb96b satellite/metabase/metaloop: add Monitor
We need some chores to join without triggering the loop.

For example it's fine to run metrics, only when something else is
running.

Change-Id: I9d8bd16f59c28c540c8d72971bc4e233a8660c02
2021-04-23 12:20:22 +03:00
Egon Elbre
fff21b330d cmd/metabase-verify: tool for verifying metabase state
Currently the tool verifies:
* validity of plain_offset
* whether plain_size is smaller than encrypted_size

Change-Id: I9ec4fb5ead3356a196392c26ca377fcdb367138e
2021-04-23 07:50:36 +00:00