Commit Graph

5322 Commits

Author SHA1 Message Date
Egon Elbre
3236f91ae0 satellite/metainfo: reenable piece hash verification
Change-Id: I9312d8fde0c1c392cbad82b947222b4b6e0dbbdd
2021-04-12 15:12:42 -06:00
Michał Niewrzał
6e79923996 satellite/metainfo: bring back validation while committing segment
During metainfo refactor we disabled some validation as it was designed to validate pointer. Now part of this validation is restored. This is first part.

Change-Id: I6132f922fe23d60118bbccfdb77fd93c3c81afed
2021-04-12 15:12:42 -06:00
Vitalii Shpital
6ddd22e55a web/satellite: remove redundant FAQ link from no buckets view
WHAT:
removed explanatory link "Why can't I upload from browser?"

WHY:
we can upload from browser now

Change-Id: I84d4aaf6f55566c2dddf8dd345d0e074bf8fd338
2021-04-12 22:48:40 +03:00
Vitalii Shpital
3ade87eb25 satellite/console: added tardigradeshare.io and storjshare.io domains to CSP's whitelist
WHAT:
updated CSP's whitelist to include linksharing domains

WHY:
to make linksharing work

Change-Id: Ie252f5f231a8f1069dd76f41bec4b1d36bb779a8
2021-04-12 22:42:35 +03:00
Vitalii Shpital
40b41e081e web/satellite: onboarding overview step updated
WHAT:
redundant file splitting removed
More Integrations button link updated
blank links security attributes added

WHY:
bakeoff

Change-Id: I50eed4f916335eec708f5a8e020d352177569aee
2021-04-12 22:38:41 +03:00
prerna-parashar
d2705c1143
satellite/analytics: Added analytics for "passphrase created", "account verified" and "external_link_clicked" (#4078) 2021-04-12 09:58:36 -07:00
JT Olio
037c7799b5 storj/storj: bump storj/common
This only pulls in https://review.dev.storj.io/c/storj/common/+/4474

Change-Id: I69dde0d66cd899ef4c91bc74a3c481ff2bf36a50
2021-04-12 07:36:15 -06:00
Vitalii Shpital
6f5f2dd36d web/satellite: clear objects vuex state on destroy. Always fetch buckets list
WHAT:
clear objects vuex state on leaving /objects/... route
always fetch buckets when entering buckets view

WHY:
storing/caching bug fix for buckets view

Change-Id: I04f3ee5cf2e762471f248ecbaee3dc68ed28ac5c
2021-04-12 09:55:37 +00:00
Egon Elbre
2a4a70908f cmd/satellite: remove metrics from gc
Currently metrics could trigger metainfo loop on it's own on GC pod,
however running only metrics adds a lot of load, we should only run it
together with other essential chores.

There's no easy way to join that way in metainfo loop, for now, let's
remove it from GC. Satellite Core will still run the metrics.

Change-Id: Ie513a9d2f86add0aa319d08567c6cf3542d73d4e
2021-04-10 12:24:09 +03:00
littleskunk
6236deffb9
satellite/web: enable feature flag for new file browser and onboarding workflow (#4079) 2021-04-09 20:44:05 +02:00
Malcolm Bouzi
9591e75574 satellite/console: add feature flag for onboarding overview step pathway rendering
Change-Id: I2edfeb868e356b714c8d2535332713c1f64d35f9
2021-04-09 12:35:55 -04:00
Egon Elbre
8f13dcacb9 mod: bump uplink
This starts using uplink with fixes:

* range downloading a migrated object
* downloading a zero length range

Change-Id: I4a95d4cdfe86ef44959dd81921ccad28ba7d7af5
2021-04-09 15:43:53 +00:00
Egon Elbre
ec7af97a17 satellite/metainfo: document migrated objects
Document the fields that migrated objects have missing, it's easy to
forget that they might not exist.

Avoid downloading the segment, if we're not sure whether it's the
correct one. We'll later improve the code with an heuristic to get a
best guess, which segment to download.

Change-Id: I12395c17bbf0edf25e0d00c8d072fce6085e303b
2021-04-09 15:28:27 +00:00
Moby von Briesen
51d667a65e satellite/analytics: Add anonymous ID to TrackCreateUser
If a visitor to the website (run through the reverse proxy) consented to
cookies, read the ID stored in that cookie and send it along with the
Identify/Track calls sent to Segment upon account creation. This allows
us to connect referral information gathered when visitors land on our
website with account activity, helping us improve our onboarding flow.

Change-Id: I0ece717ab5bba67901e50a9b4229c1d4ed7e46b7
2021-04-09 14:19:35 +00:00
Vitalii Shpital
b907aab0ad web/satellite: use config value for linksharing
WHAT:
use config value for linksharing instead of harded one

WHY:
better testing

Change-Id: I16fdb5b23198716b76d5332a1113daaa0eb86030
2021-04-09 14:56:26 +03:00
Vitalii Shpital
a3e6604883 satellite/console: config flag for enabling/disabling CSP
WHAT:
config flag for enabling/disabling CSP

WHY:
for easier testing

Change-Id: If04a98071d2b147d18b95809ef6bd4c95d6a2d1d
2021-04-09 14:44:04 +03:00
Moby von Briesen
c4a950a40c web/satellite: Remove client-side Segment analytics
We can be more precise and conservative by using the backend
satellite/analytics service. We also no longer need client-side Segment
scripts.

Change-Id: Ic5fb18bea2d388b586ad773e26027d69bde87294
2021-04-08 17:36:06 +00:00
prerna-parashar
16c98e1ecd
satellite/analytics: Add analytics for user signed in, project created, and access grant created (#4073)
* satellite/analytics: Add analytics for user signed in, project created and access grant created events


Co-authored-by: Moby von Briesen <mobyvb@gmail.com>
2021-04-08 10:34:23 -07:00
Kaloyan Raev
a264a4422b cmd: remove metainfo-migrator and metainfo-verifier
We already merged the multipart-upload branch to main. These two tools
make only sense if we are migrating a satellite from Pointer DB to
Metabase. There is one remaining satellite to migrate, but these tools
should be used from the respective release branch instead of from main.

Removing these tools from main will:
1) Avoid the mistake to use them from the main branch instead of from
the respective release branch.
2) Allow to finally remove any code related to the old Pointer DB.

Change-Id: Ied66098c5d0b8fefeb5d6e92b5e0ef5c6603df5d
2021-04-08 14:50:29 +00:00
Michał Niewrzał
6b88a675c5 metabase-createdat-migration: add migration tool
We recently added create_at column to segments table.
Old segments needs to get this value from objects table.
This tool will iterate over all objects and update corresponding
segments if create_at column is not set.

Change-Id: Ib5aedc384637e739ee9af84454af0639e2559416
2021-04-08 14:20:09 +00:00
Egon Elbre
372bb1140e mod: bump uplink and private
Change-Id: I12d540c47a47a9b16e8a9fa7e97cafbaf215e12f
2021-04-08 14:02:19 +00:00
Moby von Briesen
0a512108a6 web/satellite: Remove noPaywallInfoBar
With the new free tier, there is no need for this banner anymore.

Change-Id: Id8efee29567dac342b712b7cc732235f6a1e764b
2021-04-07 22:51:14 +00:00
Malcolm Bouzi
3088d14116
Overview refactor (#4076) 2021-04-08 01:32:40 +03:00
Vitalii Shpital
6ae2351389 web/satellite: import file browser component
WHAT:
import and instantiate file browser component

WHY:
to operate over folders and objects

Change-Id: Ib6fb4fdc2668d2f274df3d1b23f8cc0bb6a361ea
2021-04-07 22:53:20 +03:00
Jennifer Johnson
444b1f4757 storagenode/piecestore: update endpoint_test TestDownload to avoid index out of range runtime errors
Change-Id: Ib3c166a1db84f4caafa970eb800a169d35ba3a54
2021-04-07 13:34:35 -04:00
Jennifer Johnson
71072eb593 storagenode/pieces: send piece deletions to trash
This is a temporary precaution to avoid incorrectly auditing nodes for pieces that were deleted between database backups if we have to restore from a previous backup.

Here we send pieces to trash rather than directly deleting them from storage nodes so we can restore from trash after a db restoration.

Change-Id: Icd979d2a9a755e7428190c0129c9bc969649d544
2021-04-07 16:52:10 +00:00
Moby von Briesen
7e4e1040f2 satellite/console: Add endpoint for clientside analytics events
This is a very simple endpoint which allows the satellite UI client to
notify the console server that an event has occurred. We will use this
to track when users have completed certain tasks that can't be tracked
server-side (e.g. generating gateway credentials, setting a passphrase)

As part of this change, one client side event is implemented to use the
endpoint - when the user clicks the button to create gateway credentials
after making a new access grant.

Change-Id: Ic8fa729f1c84474788e1de84c18532aef8e8fa3c
2021-04-07 14:23:26 +00:00
Yingrong Zhao
4330a4bc4b pkg/quic: set KeepAlive to true
Change-Id: I84e06219a265c82e634092c46731f75703915d48
2021-04-07 13:45:03 +00:00
Egon Elbre
ffc8aca8b0 satellite/metainfo: update bandwidth usage in DownloadObject
Change-Id: I3c77fd46c57903435bd9618bb6cf33647217adbf
2021-04-07 14:17:59 +03:00
Egon Elbre
98b7e1a991 satellite/metainfo: fix returning []{nil} in DownloadObject
When object doesn't contain segments the implementation would have
returned []*pb.SegmentDownloadResponse{nil} instead of nil.

Change-Id: If38f6d3d9d119f514f63ad1a8762055f657f3004
2021-04-07 09:55:01 +03:00
Egon Elbre
47be501188 satellite/metainfo: add DownloadObject endpoint
Add endpoint for getting object information, list segments in a range
and download the first segment in the range.

Change-Id: I056d697ae87c9aa34e7deccba8713902db260457
2021-04-06 19:47:36 +03:00
Egon Elbre
17817d84e9 satellite/metainfo: add PlainOffset to SegmentDownloadResponse
Change-Id: Ib68f2fd0ad86556f443a01a514243247dc7989a3
2021-04-06 15:21:00 +03:00
Vitalii Shpital
288a7e8ee7 web/satellite: split AG wizard's result step into two
WHAT:
split AG wizard's result step into AG view and s3 gateway credentials view

WHY:
better user experience

Change-Id: I848c5fc99488258bdc9ff1210ba7ba1540f34941
2021-04-06 10:42:13 +00:00
Yingrong Zhao
f3df411b41 bump storj.io/common to latest
Change-Id: I1396008c6b7cae246653694126f1a54716a8bfa4
2021-04-05 19:07:02 -04:00
Egon Elbre
d2101bd017 satellite/metainfo: add Position to SegmentDownloadResponse
Change-Id: I538c4cf2a9d81788f0df2581defe166871d72cd7
2021-04-05 19:27:34 +03:00
Moby von Briesen
04294e3e1c satellite/payments: Update coupon defaults for free tier
The new default promotional coupon is $10/month, and doesn't expire.

This change also migrates the coupon.duration column over to the new
coupon.billing_periods, and switches to rely completely on
billing_periods.

Change-Id: Ic3341e9fa4040449bab5e66ca4ee2640b095cf3d
2021-04-05 10:12:26 -04:00
Yingrong Zhao
e0c3a7a47d bump quic-go to latest
Change-Id: I2716a6670e2eb0b226aa0a3510b0c1ec8e927ef6
2021-04-05 09:28:17 -04:00
Jeff Wendling
2462ac5917 metainfo+satellite/satellitedb: use HasBucket when possible
Change-Id: I8318f5581eef559f0dfbf20d07ea1fe36f81ce87
2021-04-02 13:20:36 -04:00
Cameron Ayer
523dd823db satellite/satellitedb: remove references to total_uptime_count and uptime_success_count columns
These columns are no longer used

Change-Id: I0400544e4e98752c9b90662c037ab3a1884c234b
2021-04-02 13:22:12 +00:00
Michał Niewrzał
a5224e7a6c satellite/metainfo/metaloop: use segment CreatedAt and RepairedAt
Repair checker expects to have information about CreatedAt and RepairedAt fields to calculate segment age metric.

Change-Id: I6b41df880d77133be541e14d10d91cc75759b339
2021-04-02 08:46:54 +00:00
Michał Niewrzał
71d0adf90b satellite/metainfo: avoid error on inserting existing attribution
Currently we can have an error about duplicated entry while inserting into value_attributions table. This change is changing simple insert into insert that is doing nothing on conflict.

Change-Id: I3efd8dc0b63115e8e2ed8f4196ccf969ee942295
2021-04-02 08:18:17 +00:00
Egon Elbre
5c038c4325 satellite/metainfo/metabase: add stream range for listing segments
Change-Id: I32833e805a1046b9752b04888f830b51809a1efd
2021-04-01 15:27:10 +00:00
Moby von Briesen
c334fd090e satellite/payments: Convert coupon duration to be nullable
* Add a nullable billing_periods column in the coupons table
* Add nullable billing_periods column to the currently unused
coupon_codes table
* Drop the duration column from the coupon_codes table
* Replace duration config type so that the default promotional coupon
can be configured to never expire

Zero downtime migration plan:
* Add billing_periods column to coupons and coupon_codes tables (this change)
* After one release, remove all references to the old duration column,
replacing with references to billing_periods. At this point, we can also
change the defult promotional coupon to never expire and migrate over
values from the old duration column.
* After another release, drop the duration column.

Change-Id: I374e8dc9fab9f81b4a5bc681771955662d4c007a
2021-04-01 16:28:23 +02:00
Qweder93
c4293f5e52 storagenode/payouts: DQ satellites removed from estimated payouts
Change-Id: Ib3fb287bbdece5386a2a6f257b54411ad5b742be
2021-04-01 11:17:27 +00:00
Michał Niewrzał
0b7c288155 test-sim: improve integration test scripts
* integration tests have now a trap that will display where script failed, line and message
* functions from uplink tests moved to utils.sh to reuse them

Change-Id: Ib2311775fd70ce784aa986328969d75eefc5ac36
2021-03-31 16:48:53 +00:00
Tome Boshevski
f6c9b8d4f1
web/satellite: access grant wizard copy edit (#4066) 2021-03-31 18:03:26 +03:00
Yingrong Zhao
7dfb89a871 bump quic-go to latest release
Change-Id: Ic5e93f7a6ffe0bb46fc8f8bf1ce310eac4aa1182
2021-03-31 13:28:41 +00:00
Michał Niewrzał
a5bc38b197 build: replace 'multipart-upload' branch occurences
We removed `multipart-upload` branches but we have few leftovers.

Change-Id: I74f637748a248cb286cd190a3a8f895ef85f3940
2021-03-31 11:06:47 +02:00
JT Olio
86c41790ce satellite/metainfo/metaloop: add observability
we want to know a lot more about what's going on during
the operation of the metainfo loop. this patchset adds
more instrumentation to previously unmonitored but
interesting functions, and adds metrics that keep track
of how far through a specific loop we are. it also
adds mon:lock annotations, especially to the metainfo
loop run task, which recently changed, silently broke
some queries, and thus failed to alert us to spiking
run time issues.

Change-Id: I4358e2f2293d8ebe30eef497ba4e423ece929041
2021-03-30 14:32:05 -06:00
Ivan Fraixedes
f3176adbce private/migrate: Add missing validation table call
Add a missing call to the validation table name method to ensure that
it cannot be SQL injection.

Change-Id: Iee10b09e15c07448f0803ab0be653b35fadb4ee4
2021-03-30 16:21:48 +00:00