Commit Graph

1960 Commits

Author SHA1 Message Date
Fadila Khadar
5af898fcc9 satellite/metabase: add migration step for index on pending objects
Add an index on (project_id, bucket_name) for objects with pending status

Change-Id: Ibd3c949a99e49e8e6bb884564331ec16f9184942
2021-04-28 09:54:10 +00:00
Michał Niewrzał
02460fcc4c satellite/metainfo/expireddeletion: change chore interval to 24h
We want run deletion query more often to avoid long running DB operations.

Change-Id: I72c63bb8ec05cc3ab2e7ab63479ba6cdf830386a
2021-04-27 12:30:23 +00:00
Vitalii Shpital
606a501441 satellite/console: add tardigradeshare.io domain to media-src CSP whitelist
WHAT:
added tardigradeshare.io domain to media-src CSP whitelist

WHY:
we still support it

Change-Id: Id02daa0712cc84f5cb7edaafb21a143859531f26
2021-04-27 11:17:50 +00:00
Malcolm Bouzi
c1fdf1ad71 satellite/satellitedb: add have_sales_contact column on users
For business accounts we need to track the sales contact.
It will be a question to business accounts during onboarding.

Change-Id: I8d101ce1b52091478dfb0ddd875e1cc717d765d3
2021-04-26 13:37:26 +00:00
Michał Niewrzał
7944df20d6 storj: use multipart API
Change-Id: I10b401434e3e77468d12ecd225b41689568fd197
2021-04-26 13:15:09 +00:00
Egon Elbre
948529a600 satellite/metabase/metabasetest: move test utils to a separate package
Change-Id: Iaa607996478497d5c97c6a6973d099b315b3abf0
2021-04-26 13:12:58 +03:00
Egon Elbre
7802ab714f pkg/,private/: merge with private package
Initially there were pkg and private packages, however for all practical
purposes there's no significant difference between them. It's clearer to
have a single private package - and when we do get a specific
abstraction that needs to be reused, we can move it to storj.io/common
or storj.io/private.

Change-Id: Ibc2036e67f312f5d63cb4a97f5a92e38ae413aa5
2021-04-23 16:37:28 +03:00
Egon Elbre
c641ddcb54 pkg/lrucache: rename package
cache is really common variable and type name and we have already used
the package name alias in multiple places.

Change-Id: I6435785b7549b541d533de59ec94557b9bd11e04
2021-04-23 16:37:28 +03:00
Egon Elbre
8c62788b24 satellite/metabase/metaloop: stop timer properly
There's a rare chance that `Stop` returns false, however doesn't have
time triggered. Use a non-blocking drain to remove the token.

Change-Id: I1ae18a197424017f0ca76656602709a029b56bfd
2021-04-23 16:35:21 +03:00
Vitalii Shpital
d32515fa5d satellite/console: add storjshare.io domain to media-src CSP whitelist
WHAT:
whitelist .storjshare.io domain for media-src CSP

WHY:
to enable video preview for linksharing

Change-Id: Ib673602d31ca116e7ce1cee0eba17099a55d7dbc
2021-04-23 16:11:41 +03:00
Egon Elbre
a2e20c93ae private/dbutil: use dbutil and tagsql from storj.io/private
Initially we duplicated the code to avoid large scale changes to
the packages. Now we are past metainfo refactor we can remove the
duplication.

Change-Id: I9d0b2756cc6e2a2f4d576afa408a15273a7e1cef
2021-04-23 14:36:52 +03:00
Egon Elbre
dd5eb2616b satellite/metrics: join for monitoring
We don't want metrics to trigger the metaloop on it's own.

Change-Id: I648272a0e3043173b0a377697db1f172712bdda5
2021-04-23 12:21:48 +03:00
Egon Elbre
c1fbecb96b satellite/metabase/metaloop: add Monitor
We need some chores to join without triggering the loop.

For example it's fine to run metrics, only when something else is
running.

Change-Id: I9d8bd16f59c28c540c8d72971bc4e233a8660c02
2021-04-23 12:20:22 +03:00
Egon Elbre
fff21b330d cmd/metabase-verify: tool for verifying metabase state
Currently the tool verifies:
* validity of plain_offset
* whether plain_size is smaller than encrypted_size

Change-Id: I9ec4fb5ead3356a196392c26ca377fcdb367138e
2021-04-23 07:50:36 +00:00
Cameron Ayer
a0c5da6643 satellite/satellitedb: in stray nodes DQ, don't DQ nodes where last_contact_success = '0001-01-01 00:00:00+00'
When nodes check in for the very first time, if the satellite can't ping
them back, they are inserted into the nodes table with
last_contact_success of '0001-01-01 00:00:00+00'. If the stray nodes
chore runs before the node can fix their problem, they are DQd.

Solution: when DQing stray nodes, dont DQ where last_contact_success =
'0001-01-01 00:00:00+00'::timestamptz

Change-Id: I477a02d5ef85b2c930ed6b7d99a4d1995169bca8
2021-04-22 10:13:13 -04:00
Egon Elbre
4c9ed64f75 satellite/metabase/metaloop: move loop under metabase
Currently the loop handling is heavily related to the metabase rather
than metainfo.

metainfo over time has become related to the "public API" for accessing
the metabase data.

Currently updates monkit.lock, because monkit monitoring does not handle
ScopeNamed correctly. Needs a followup change to monitoring check.

Change-Id: Ie50519991d718dfb872ec9a0176a82e732c97584
2021-04-22 12:58:09 +03:00
Egon Elbre
ebc981291a satellite/accounting: use correct error in tests
rpcstatus codes are not part of the public uplink API
and the tests should not use them.

Change-Id: I9c96c557bf2dac5d6fff200fb217378524fd14ff
2021-04-22 07:25:42 +00:00
Yingrong Zhao
43f5888052 satellite/contact: only test PingBack failure case
The original test caused the testplanet to timeout due to contact
service stuck in a loop. We can change it to only test the failure case for
PingBack method instead of closing the TCP port on storagenodes.

Change-Id: Ic96aee637b39ae95050c6902c2bf9ca51fb586c3
2021-04-21 14:20:36 -04:00
Ethan
a9c9f080bb satellite/metaloop: Use AS OF SYSTEM TIME when querying the objects table
(*loopIterator).doNextQuery is not using AS OF SYSTEM TIME because the value is 0

Change-Id: I3c671c1b2aa32bd6379b5cdea6e16246b4def9fd
2021-04-21 17:18:52 +00:00
Ivan Fraixedes
2537bbf543
satellite/gracefulexit: Try avoiding randomly test failure
The test function fails randomly in the CI when runs with CRDB. There
isn't currently an explanation why the expectation of number of nodes
which exited 4 minutes ago reports 4 nodes rather than 5 and the only
clue that we have now to see if it gets remedied is to give 2 minutes
rather than 1 to the node that exited close to the time passed function
which makes the test to randomly fail.

Change-Id: I3a731e3eb7f19caebdf29713150727f2cf3e0e0a
2021-04-21 17:40:07 +02:00
Egon Elbre
267506bb20 satellite/metabase: move package one level higher
metabase has become a central concept and it's more suitable for it to
be directly nested under satellite rather than being part of metainfo.

metainfo is going to be the "endpoint" logic for handling requests.

Change-Id: I53770d6761ac1e9a1283b5aa68f471b21e784198
2021-04-21 15:54:22 +03:00
Kaloyan Raev
2ee3030275 all: remove code related to PointerDB
Change-Id: I6675c9597f87019020f6233b83ab2f1119d2bc46
2021-04-21 12:35:31 +00:00
Yingrong Zhao
a3c437a7bf satellite/contact,storagenode/contact: try ping back to nodes through
QUIC

We want to encourage storagenodes to open their udp port. This PR
changes contact service in satellite to try to connect to nodes through
QUIC. If satellite can't reach nodes through quic, it will send an error
message back to nodes. On the nodes side, it will always log out error
message from check in if the error message is not empty.
Whether satellite can reach nodes through quic has no affect on nodes'
uptime check.

Change-Id: I5ebf80f921c4a6504997d83c8bd45226da9d3703
2021-04-20 19:25:37 +00:00
prerna-parashar
613a95530b
satellite/analytics: Add analytics for "path selected" in onboarding step (#4086) 2021-04-19 09:44:25 -07:00
Egon Elbre
6a805b2891 satellite/metainfo/metabase: use pgx.Batch to delete expired segments
Change-Id: I51eeaadeeef0ca2faabac4b38158278aea177eb8
2021-04-19 11:19:29 +03:00
Egon Elbre
b2be1f1629 satellite/metainfo/metabase: fix delete expiration
The cursor was not being used in the batch deletion.

The stream ID was not being used while deleting, which could in rare
circumstaces delete a newly uploaded object.

Use the stream id in deletion, rather than passing that information from
one query to another.

Change-Id: I03271c6e72747e345dfb0bb70989f29e835efd8e
2021-04-19 11:19:29 +03:00
Vitalii Shpital
b57819f590 satellite: take pricing from the config instead of hardcoding
WHAT:
take pricing from config instead of hardcoding

WHY:
bakeoff

Change-Id: Id8209f0905a9105c1f5796165e279acf31563c65
2021-04-16 07:59:31 +00:00
JT Olio
35d8a840d3 storj/storj: more domain changes
Change-Id: I643c38bdae6dc26c9346147d80a83ae9dde2eeae
2021-04-15 20:51:43 +00:00
JT Olio
3b09d6c308 storj/storj: update support request links
Change-Id: I7592df7fe4630508784b6cc92b751d3b567e911e
2021-04-14 20:52:45 +00:00
JT Olio
f3c8cac1ee storj/storj: more domain changes
Change-Id: I91ae3bc36be38c27269f9e6e678fa3616b8e4d53
2021-04-14 20:49:12 +00:00
Brandon Iglesias
32d76652c3
satellite/payments: Change customer pricing (#4085)
Co-authored-by: littleskunk <jens.heimbuerge@googlemail.com>
Co-authored-by: JT Olio <hello@jtolio.com>
Co-authored-by: Igor <38665104+ihaid@users.noreply.github.com>
2021-04-14 12:14:58 -06:00
Fadila Khadar
bde367ae73 satellite/gc: check on bloom filter creation date
Check that the bloom filter creation date is earlier than the
metainfo loop system time used for db scanning.

Change-Id: Ib0f47c124f5651deae0fd7e7996abcdcaac98fb4
2021-04-14 16:40:37 +00:00
Michał Niewrzał
c3f8e06798 satellite/metainfo/metainfo: simplify expired objects deletion query
Change-Id: Iba5fc498527338ed6d2c5dd779c508ec8b6ce443
2021-04-13 19:34:32 +00:00
JT Olio
afcc55fb47 storj/storj: some domain changes we can make now
Change-Id: I5622edea9dfcad788cac65e1754b331b9fed6154
2021-04-13 17:12:06 +00:00
Vitalii Shpital
0b59a165e9 web/satellite: remove google tag manager from satellite GUI
WHAT:
removed google tag manager from satellite GUI

WHY:
redundant

Change-Id: I9332b770b9050395bd2b44c2f3f3f410b5891fa4
2021-04-13 11:23:47 +00:00
Michał Niewrzał
027554a15c satellite/metainfo: add tests for commit segment validation
Change-Id: I012bf2f7481003591cc873fe738a7cd8f525db29
2021-04-12 16:33:17 -06:00
Egon Elbre
3236f91ae0 satellite/metainfo: reenable piece hash verification
Change-Id: I9312d8fde0c1c392cbad82b947222b4b6e0dbbdd
2021-04-12 15:12:42 -06:00
Michał Niewrzał
6e79923996 satellite/metainfo: bring back validation while committing segment
During metainfo refactor we disabled some validation as it was designed to validate pointer. Now part of this validation is restored. This is first part.

Change-Id: I6132f922fe23d60118bbccfdb77fd93c3c81afed
2021-04-12 15:12:42 -06:00
Vitalii Shpital
3ade87eb25 satellite/console: added tardigradeshare.io and storjshare.io domains to CSP's whitelist
WHAT:
updated CSP's whitelist to include linksharing domains

WHY:
to make linksharing work

Change-Id: Ie252f5f231a8f1069dd76f41bec4b1d36bb779a8
2021-04-12 22:42:35 +03:00
prerna-parashar
d2705c1143
satellite/analytics: Added analytics for "passphrase created", "account verified" and "external_link_clicked" (#4078) 2021-04-12 09:58:36 -07:00
Egon Elbre
2a4a70908f cmd/satellite: remove metrics from gc
Currently metrics could trigger metainfo loop on it's own on GC pod,
however running only metrics adds a lot of load, we should only run it
together with other essential chores.

There's no easy way to join that way in metainfo loop, for now, let's
remove it from GC. Satellite Core will still run the metrics.

Change-Id: Ie513a9d2f86add0aa319d08567c6cf3542d73d4e
2021-04-10 12:24:09 +03:00
littleskunk
6236deffb9
satellite/web: enable feature flag for new file browser and onboarding workflow (#4079) 2021-04-09 20:44:05 +02:00
Malcolm Bouzi
9591e75574 satellite/console: add feature flag for onboarding overview step pathway rendering
Change-Id: I2edfeb868e356b714c8d2535332713c1f64d35f9
2021-04-09 12:35:55 -04:00
Egon Elbre
ec7af97a17 satellite/metainfo: document migrated objects
Document the fields that migrated objects have missing, it's easy to
forget that they might not exist.

Avoid downloading the segment, if we're not sure whether it's the
correct one. We'll later improve the code with an heuristic to get a
best guess, which segment to download.

Change-Id: I12395c17bbf0edf25e0d00c8d072fce6085e303b
2021-04-09 15:28:27 +00:00
Moby von Briesen
51d667a65e satellite/analytics: Add anonymous ID to TrackCreateUser
If a visitor to the website (run through the reverse proxy) consented to
cookies, read the ID stored in that cookie and send it along with the
Identify/Track calls sent to Segment upon account creation. This allows
us to connect referral information gathered when visitors land on our
website with account activity, helping us improve our onboarding flow.

Change-Id: I0ece717ab5bba67901e50a9b4229c1d4ed7e46b7
2021-04-09 14:19:35 +00:00
Vitalii Shpital
b907aab0ad web/satellite: use config value for linksharing
WHAT:
use config value for linksharing instead of harded one

WHY:
better testing

Change-Id: I16fdb5b23198716b76d5332a1113daaa0eb86030
2021-04-09 14:56:26 +03:00
Vitalii Shpital
a3e6604883 satellite/console: config flag for enabling/disabling CSP
WHAT:
config flag for enabling/disabling CSP

WHY:
for easier testing

Change-Id: If04a98071d2b147d18b95809ef6bd4c95d6a2d1d
2021-04-09 14:44:04 +03:00
Moby von Briesen
c4a950a40c web/satellite: Remove client-side Segment analytics
We can be more precise and conservative by using the backend
satellite/analytics service. We also no longer need client-side Segment
scripts.

Change-Id: Ic5fb18bea2d388b586ad773e26027d69bde87294
2021-04-08 17:36:06 +00:00
prerna-parashar
16c98e1ecd
satellite/analytics: Add analytics for user signed in, project created, and access grant created (#4073)
* satellite/analytics: Add analytics for user signed in, project created and access grant created events


Co-authored-by: Moby von Briesen <mobyvb@gmail.com>
2021-04-08 10:34:23 -07:00
Michał Niewrzał
6b88a675c5 metabase-createdat-migration: add migration tool
We recently added create_at column to segments table.
Old segments needs to get this value from objects table.
This tool will iterate over all objects and update corresponding
segments if create_at column is not set.

Change-Id: Ib5aedc384637e739ee9af84454af0639e2559416
2021-04-08 14:20:09 +00:00