Commit Graph

5660 Commits

Author SHA1 Message Date
Egon Elbre
ec7af97a17 satellite/metainfo: document migrated objects
Document the fields that migrated objects have missing, it's easy to
forget that they might not exist.

Avoid downloading the segment, if we're not sure whether it's the
correct one. We'll later improve the code with an heuristic to get a
best guess, which segment to download.

Change-Id: I12395c17bbf0edf25e0d00c8d072fce6085e303b
2021-04-09 15:28:27 +00:00
Moby von Briesen
51d667a65e satellite/analytics: Add anonymous ID to TrackCreateUser
If a visitor to the website (run through the reverse proxy) consented to
cookies, read the ID stored in that cookie and send it along with the
Identify/Track calls sent to Segment upon account creation. This allows
us to connect referral information gathered when visitors land on our
website with account activity, helping us improve our onboarding flow.

Change-Id: I0ece717ab5bba67901e50a9b4229c1d4ed7e46b7
2021-04-09 14:19:35 +00:00
Vitalii Shpital
b907aab0ad web/satellite: use config value for linksharing
WHAT:
use config value for linksharing instead of harded one

WHY:
better testing

Change-Id: I16fdb5b23198716b76d5332a1113daaa0eb86030
2021-04-09 14:56:26 +03:00
Vitalii Shpital
a3e6604883 satellite/console: config flag for enabling/disabling CSP
WHAT:
config flag for enabling/disabling CSP

WHY:
for easier testing

Change-Id: If04a98071d2b147d18b95809ef6bd4c95d6a2d1d
2021-04-09 14:44:04 +03:00
Moby von Briesen
c4a950a40c web/satellite: Remove client-side Segment analytics
We can be more precise and conservative by using the backend
satellite/analytics service. We also no longer need client-side Segment
scripts.

Change-Id: Ic5fb18bea2d388b586ad773e26027d69bde87294
2021-04-08 17:36:06 +00:00
prerna-parashar
16c98e1ecd
satellite/analytics: Add analytics for user signed in, project created, and access grant created (#4073)
* satellite/analytics: Add analytics for user signed in, project created and access grant created events


Co-authored-by: Moby von Briesen <mobyvb@gmail.com>
2021-04-08 10:34:23 -07:00
Kaloyan Raev
a264a4422b cmd: remove metainfo-migrator and metainfo-verifier
We already merged the multipart-upload branch to main. These two tools
make only sense if we are migrating a satellite from Pointer DB to
Metabase. There is one remaining satellite to migrate, but these tools
should be used from the respective release branch instead of from main.

Removing these tools from main will:
1) Avoid the mistake to use them from the main branch instead of from
the respective release branch.
2) Allow to finally remove any code related to the old Pointer DB.

Change-Id: Ied66098c5d0b8fefeb5d6e92b5e0ef5c6603df5d
2021-04-08 14:50:29 +00:00
Michał Niewrzał
6b88a675c5 metabase-createdat-migration: add migration tool
We recently added create_at column to segments table.
Old segments needs to get this value from objects table.
This tool will iterate over all objects and update corresponding
segments if create_at column is not set.

Change-Id: Ib5aedc384637e739ee9af84454af0639e2559416
2021-04-08 14:20:09 +00:00
Egon Elbre
372bb1140e mod: bump uplink and private
Change-Id: I12d540c47a47a9b16e8a9fa7e97cafbaf215e12f
2021-04-08 14:02:19 +00:00
Moby von Briesen
0a512108a6 web/satellite: Remove noPaywallInfoBar
With the new free tier, there is no need for this banner anymore.

Change-Id: Id8efee29567dac342b712b7cc732235f6a1e764b
2021-04-07 22:51:14 +00:00
Malcolm Bouzi
3088d14116
Overview refactor (#4076) 2021-04-08 01:32:40 +03:00
Vitalii Shpital
6ae2351389 web/satellite: import file browser component
WHAT:
import and instantiate file browser component

WHY:
to operate over folders and objects

Change-Id: Ib6fb4fdc2668d2f274df3d1b23f8cc0bb6a361ea
2021-04-07 22:53:20 +03:00
Jennifer Johnson
444b1f4757 storagenode/piecestore: update endpoint_test TestDownload to avoid index out of range runtime errors
Change-Id: Ib3c166a1db84f4caafa970eb800a169d35ba3a54
2021-04-07 13:34:35 -04:00
Jennifer Johnson
71072eb593 storagenode/pieces: send piece deletions to trash
This is a temporary precaution to avoid incorrectly auditing nodes for pieces that were deleted between database backups if we have to restore from a previous backup.

Here we send pieces to trash rather than directly deleting them from storage nodes so we can restore from trash after a db restoration.

Change-Id: Icd979d2a9a755e7428190c0129c9bc969649d544
2021-04-07 16:52:10 +00:00
Moby von Briesen
7e4e1040f2 satellite/console: Add endpoint for clientside analytics events
This is a very simple endpoint which allows the satellite UI client to
notify the console server that an event has occurred. We will use this
to track when users have completed certain tasks that can't be tracked
server-side (e.g. generating gateway credentials, setting a passphrase)

As part of this change, one client side event is implemented to use the
endpoint - when the user clicks the button to create gateway credentials
after making a new access grant.

Change-Id: Ic8fa729f1c84474788e1de84c18532aef8e8fa3c
2021-04-07 14:23:26 +00:00
Yingrong Zhao
4330a4bc4b pkg/quic: set KeepAlive to true
Change-Id: I84e06219a265c82e634092c46731f75703915d48
2021-04-07 13:45:03 +00:00
Egon Elbre
ffc8aca8b0 satellite/metainfo: update bandwidth usage in DownloadObject
Change-Id: I3c77fd46c57903435bd9618bb6cf33647217adbf
2021-04-07 14:17:59 +03:00
Egon Elbre
98b7e1a991 satellite/metainfo: fix returning []{nil} in DownloadObject
When object doesn't contain segments the implementation would have
returned []*pb.SegmentDownloadResponse{nil} instead of nil.

Change-Id: If38f6d3d9d119f514f63ad1a8762055f657f3004
2021-04-07 09:55:01 +03:00
Egon Elbre
47be501188 satellite/metainfo: add DownloadObject endpoint
Add endpoint for getting object information, list segments in a range
and download the first segment in the range.

Change-Id: I056d697ae87c9aa34e7deccba8713902db260457
2021-04-06 19:47:36 +03:00
Egon Elbre
17817d84e9 satellite/metainfo: add PlainOffset to SegmentDownloadResponse
Change-Id: Ib68f2fd0ad86556f443a01a514243247dc7989a3
2021-04-06 15:21:00 +03:00
Vitalii Shpital
288a7e8ee7 web/satellite: split AG wizard's result step into two
WHAT:
split AG wizard's result step into AG view and s3 gateway credentials view

WHY:
better user experience

Change-Id: I848c5fc99488258bdc9ff1210ba7ba1540f34941
2021-04-06 10:42:13 +00:00
Yingrong Zhao
f3df411b41 bump storj.io/common to latest
Change-Id: I1396008c6b7cae246653694126f1a54716a8bfa4
2021-04-05 19:07:02 -04:00
Egon Elbre
d2101bd017 satellite/metainfo: add Position to SegmentDownloadResponse
Change-Id: I538c4cf2a9d81788f0df2581defe166871d72cd7
2021-04-05 19:27:34 +03:00
Moby von Briesen
04294e3e1c satellite/payments: Update coupon defaults for free tier
The new default promotional coupon is $10/month, and doesn't expire.

This change also migrates the coupon.duration column over to the new
coupon.billing_periods, and switches to rely completely on
billing_periods.

Change-Id: Ic3341e9fa4040449bab5e66ca4ee2640b095cf3d
2021-04-05 10:12:26 -04:00
Yingrong Zhao
e0c3a7a47d bump quic-go to latest
Change-Id: I2716a6670e2eb0b226aa0a3510b0c1ec8e927ef6
2021-04-05 09:28:17 -04:00
Jeff Wendling
2462ac5917 metainfo+satellite/satellitedb: use HasBucket when possible
Change-Id: I8318f5581eef559f0dfbf20d07ea1fe36f81ce87
2021-04-02 13:20:36 -04:00
Cameron Ayer
523dd823db satellite/satellitedb: remove references to total_uptime_count and uptime_success_count columns
These columns are no longer used

Change-Id: I0400544e4e98752c9b90662c037ab3a1884c234b
2021-04-02 13:22:12 +00:00
Michał Niewrzał
a5224e7a6c satellite/metainfo/metaloop: use segment CreatedAt and RepairedAt
Repair checker expects to have information about CreatedAt and RepairedAt fields to calculate segment age metric.

Change-Id: I6b41df880d77133be541e14d10d91cc75759b339
2021-04-02 08:46:54 +00:00
Michał Niewrzał
71d0adf90b satellite/metainfo: avoid error on inserting existing attribution
Currently we can have an error about duplicated entry while inserting into value_attributions table. This change is changing simple insert into insert that is doing nothing on conflict.

Change-Id: I3efd8dc0b63115e8e2ed8f4196ccf969ee942295
2021-04-02 08:18:17 +00:00
Egon Elbre
5c038c4325 satellite/metainfo/metabase: add stream range for listing segments
Change-Id: I32833e805a1046b9752b04888f830b51809a1efd
2021-04-01 15:27:10 +00:00
Moby von Briesen
c334fd090e satellite/payments: Convert coupon duration to be nullable
* Add a nullable billing_periods column in the coupons table
* Add nullable billing_periods column to the currently unused
coupon_codes table
* Drop the duration column from the coupon_codes table
* Replace duration config type so that the default promotional coupon
can be configured to never expire

Zero downtime migration plan:
* Add billing_periods column to coupons and coupon_codes tables (this change)
* After one release, remove all references to the old duration column,
replacing with references to billing_periods. At this point, we can also
change the defult promotional coupon to never expire and migrate over
values from the old duration column.
* After another release, drop the duration column.

Change-Id: I374e8dc9fab9f81b4a5bc681771955662d4c007a
2021-04-01 16:28:23 +02:00
Qweder93
c4293f5e52 storagenode/payouts: DQ satellites removed from estimated payouts
Change-Id: Ib3fb287bbdece5386a2a6f257b54411ad5b742be
2021-04-01 11:17:27 +00:00
Michał Niewrzał
0b7c288155 test-sim: improve integration test scripts
* integration tests have now a trap that will display where script failed, line and message
* functions from uplink tests moved to utils.sh to reuse them

Change-Id: Ib2311775fd70ce784aa986328969d75eefc5ac36
2021-03-31 16:48:53 +00:00
Tome Boshevski
f6c9b8d4f1
web/satellite: access grant wizard copy edit (#4066) 2021-03-31 18:03:26 +03:00
Yingrong Zhao
7dfb89a871 bump quic-go to latest release
Change-Id: Ic5e93f7a6ffe0bb46fc8f8bf1ce310eac4aa1182
2021-03-31 13:28:41 +00:00
Michał Niewrzał
a5bc38b197 build: replace 'multipart-upload' branch occurences
We removed `multipart-upload` branches but we have few leftovers.

Change-Id: I74f637748a248cb286cd190a3a8f895ef85f3940
2021-03-31 11:06:47 +02:00
JT Olio
86c41790ce satellite/metainfo/metaloop: add observability
we want to know a lot more about what's going on during
the operation of the metainfo loop. this patchset adds
more instrumentation to previously unmonitored but
interesting functions, and adds metrics that keep track
of how far through a specific loop we are. it also
adds mon:lock annotations, especially to the metainfo
loop run task, which recently changed, silently broke
some queries, and thus failed to alert us to spiking
run time issues.

Change-Id: I4358e2f2293d8ebe30eef497ba4e423ece929041
2021-03-30 14:32:05 -06:00
Ivan Fraixedes
f3176adbce private/migrate: Add missing validation table call
Add a missing call to the validation table name method to ensure that
it cannot be SQL injection.

Change-Id: Iee10b09e15c07448f0803ab0be653b35fadb4ee4
2021-03-30 16:21:48 +00:00
Fadila Khadar
8d4df93d45 satellite/metainfo: remove TODO
remove TODO that we won't do.

Change-Id: I6ab3ddf2d9285006bed0fc5700da8c6738aa7792
2021-03-30 13:55:54 +00:00
Egon Elbre
794dc6d6d5 satellite/metainfo/metabase: add PlainOffset to segment listing
Change-Id: I8bbb5fcebf9f171033938eaae9f415b3a4b5aaa0
2021-03-30 16:20:21 +03:00
Egon Elbre
c4cb79ff1f satellite/metainfo/metabase: update segment plain offset, when needed
CommitObject was not updating plain_offset on segments.

Change-Id: I7e56c873bb15b893bf298b73e0349eb18d855a37
2021-03-30 16:20:21 +03:00
NickolaiYurchenko
f2718734f6 web/storagenode: bug fixes
WHAT:
- fix for wrong total held calcutation on refresh
- payout history year restrictions depends on selected satellite fixed
- payout history total changed if distributed amount > 0
- dark mode info icon colors changed
- option menu closing issue fixed

Change-Id: Ia289d51f081b782ae601722b7c05c3307f56541d
2021-03-30 14:43:51 +03:00
NickolaiYurchenko
b91ba7057e web/storagenode: added zksync wallet explorer button
WHAT: link to zkscan if zksync is opted in

WHY: to have ability to see wallet status redirected from dashboard

Change-Id: Ic2d084c1e551f461307fb5a1d5eb1741ed7cdf85
2021-03-30 08:47:11 +00:00
Michał Niewrzał
5dbc91622f metainfo-migration: fix flaky TestMigrator_SingleSegmentObj test
Test had two issues:
* sub test was using parent test to fail (ctx.Check)
* pointerDB and metabaseDB were using the same unique DB and pointerDB was closing/deleting it first

Change-Id: I5741b76d518663d80c4e6448c76e4ee9dd86c8e1
2021-03-29 17:31:58 +00:00
Moby von Briesen
23c8331d1f satellite/analytics: Add flag for disabling analytics (default disabled)
Just in case we need to have an easy flag for turning analytics on or
off.

Change-Id: I219071200e0a4c156ba92cbbaa4de4ab75b2cb68
2021-03-29 16:23:47 +00:00
Jeff Wendling
a65aecfd98 compensation: always generate invoices for every node
instead of only generating invoices for nodes that had some
activity, we generate it for every node so that we can find
and pay terminal nodes that did not meet thresholds before
we recognized them as terminal.

Change-Id: Ibb3433e1b35f1ddcfbe292c034238c9fa1b66c44
2021-03-29 14:15:45 +00:00
Kaloyan Raev
035c393da0 satellite: update tests to pass etag.Reader to multipart.PutObjectPart
Change-Id: Ibe99357945ae7a91f5b5d4f87b83d425c9fa84a5
2021-03-29 13:18:11 +00:00
Egon Elbre
74b243d1ca certificate: move protobuf to storj/storj repository
Change-Id: I4689318304b54121a65e28860430d39d94ac5bd6
2021-03-29 12:59:49 +00:00
Michał Niewrzał
141444f6d6 satellite/repair/repairer: fix segmentAge metric
Change-Id: I146b3163aa1bfab5ee060298e6bf9822ca6820a0
2021-03-29 12:29:47 +00:00
Kaloyan Raev
4b14839e14 make: bump-dependencies should update uplink from main branch
Change-Id: I140b15d8d67e1283af820b4a54b86c2b714e82ca
2021-03-29 11:48:39 +00:00