Commit Graph

5344 Commits

Author SHA1 Message Date
Egon Elbre
ffc8aca8b0 satellite/metainfo: update bandwidth usage in DownloadObject
Change-Id: I3c77fd46c57903435bd9618bb6cf33647217adbf
2021-04-07 14:17:59 +03:00
Egon Elbre
98b7e1a991 satellite/metainfo: fix returning []{nil} in DownloadObject
When object doesn't contain segments the implementation would have
returned []*pb.SegmentDownloadResponse{nil} instead of nil.

Change-Id: If38f6d3d9d119f514f63ad1a8762055f657f3004
2021-04-07 09:55:01 +03:00
Egon Elbre
47be501188 satellite/metainfo: add DownloadObject endpoint
Add endpoint for getting object information, list segments in a range
and download the first segment in the range.

Change-Id: I056d697ae87c9aa34e7deccba8713902db260457
2021-04-06 19:47:36 +03:00
Egon Elbre
17817d84e9 satellite/metainfo: add PlainOffset to SegmentDownloadResponse
Change-Id: Ib68f2fd0ad86556f443a01a514243247dc7989a3
2021-04-06 15:21:00 +03:00
Vitalii Shpital
288a7e8ee7 web/satellite: split AG wizard's result step into two
WHAT:
split AG wizard's result step into AG view and s3 gateway credentials view

WHY:
better user experience

Change-Id: I848c5fc99488258bdc9ff1210ba7ba1540f34941
2021-04-06 10:42:13 +00:00
Yingrong Zhao
f3df411b41 bump storj.io/common to latest
Change-Id: I1396008c6b7cae246653694126f1a54716a8bfa4
2021-04-05 19:07:02 -04:00
Egon Elbre
d2101bd017 satellite/metainfo: add Position to SegmentDownloadResponse
Change-Id: I538c4cf2a9d81788f0df2581defe166871d72cd7
2021-04-05 19:27:34 +03:00
Moby von Briesen
04294e3e1c satellite/payments: Update coupon defaults for free tier
The new default promotional coupon is $10/month, and doesn't expire.

This change also migrates the coupon.duration column over to the new
coupon.billing_periods, and switches to rely completely on
billing_periods.

Change-Id: Ic3341e9fa4040449bab5e66ca4ee2640b095cf3d
2021-04-05 10:12:26 -04:00
Yingrong Zhao
e0c3a7a47d bump quic-go to latest
Change-Id: I2716a6670e2eb0b226aa0a3510b0c1ec8e927ef6
2021-04-05 09:28:17 -04:00
Jeff Wendling
2462ac5917 metainfo+satellite/satellitedb: use HasBucket when possible
Change-Id: I8318f5581eef559f0dfbf20d07ea1fe36f81ce87
2021-04-02 13:20:36 -04:00
Cameron Ayer
523dd823db satellite/satellitedb: remove references to total_uptime_count and uptime_success_count columns
These columns are no longer used

Change-Id: I0400544e4e98752c9b90662c037ab3a1884c234b
2021-04-02 13:22:12 +00:00
Michał Niewrzał
a5224e7a6c satellite/metainfo/metaloop: use segment CreatedAt and RepairedAt
Repair checker expects to have information about CreatedAt and RepairedAt fields to calculate segment age metric.

Change-Id: I6b41df880d77133be541e14d10d91cc75759b339
2021-04-02 08:46:54 +00:00
Michał Niewrzał
71d0adf90b satellite/metainfo: avoid error on inserting existing attribution
Currently we can have an error about duplicated entry while inserting into value_attributions table. This change is changing simple insert into insert that is doing nothing on conflict.

Change-Id: I3efd8dc0b63115e8e2ed8f4196ccf969ee942295
2021-04-02 08:18:17 +00:00
Egon Elbre
5c038c4325 satellite/metainfo/metabase: add stream range for listing segments
Change-Id: I32833e805a1046b9752b04888f830b51809a1efd
2021-04-01 15:27:10 +00:00
Moby von Briesen
c334fd090e satellite/payments: Convert coupon duration to be nullable
* Add a nullable billing_periods column in the coupons table
* Add nullable billing_periods column to the currently unused
coupon_codes table
* Drop the duration column from the coupon_codes table
* Replace duration config type so that the default promotional coupon
can be configured to never expire

Zero downtime migration plan:
* Add billing_periods column to coupons and coupon_codes tables (this change)
* After one release, remove all references to the old duration column,
replacing with references to billing_periods. At this point, we can also
change the defult promotional coupon to never expire and migrate over
values from the old duration column.
* After another release, drop the duration column.

Change-Id: I374e8dc9fab9f81b4a5bc681771955662d4c007a
2021-04-01 16:28:23 +02:00
Qweder93
c4293f5e52 storagenode/payouts: DQ satellites removed from estimated payouts
Change-Id: Ib3fb287bbdece5386a2a6f257b54411ad5b742be
2021-04-01 11:17:27 +00:00
Michał Niewrzał
0b7c288155 test-sim: improve integration test scripts
* integration tests have now a trap that will display where script failed, line and message
* functions from uplink tests moved to utils.sh to reuse them

Change-Id: Ib2311775fd70ce784aa986328969d75eefc5ac36
2021-03-31 16:48:53 +00:00
Tome Boshevski
f6c9b8d4f1
web/satellite: access grant wizard copy edit (#4066) 2021-03-31 18:03:26 +03:00
Yingrong Zhao
7dfb89a871 bump quic-go to latest release
Change-Id: Ic5e93f7a6ffe0bb46fc8f8bf1ce310eac4aa1182
2021-03-31 13:28:41 +00:00
Michał Niewrzał
a5bc38b197 build: replace 'multipart-upload' branch occurences
We removed `multipart-upload` branches but we have few leftovers.

Change-Id: I74f637748a248cb286cd190a3a8f895ef85f3940
2021-03-31 11:06:47 +02:00
JT Olio
86c41790ce satellite/metainfo/metaloop: add observability
we want to know a lot more about what's going on during
the operation of the metainfo loop. this patchset adds
more instrumentation to previously unmonitored but
interesting functions, and adds metrics that keep track
of how far through a specific loop we are. it also
adds mon:lock annotations, especially to the metainfo
loop run task, which recently changed, silently broke
some queries, and thus failed to alert us to spiking
run time issues.

Change-Id: I4358e2f2293d8ebe30eef497ba4e423ece929041
2021-03-30 14:32:05 -06:00
Ivan Fraixedes
f3176adbce private/migrate: Add missing validation table call
Add a missing call to the validation table name method to ensure that
it cannot be SQL injection.

Change-Id: Iee10b09e15c07448f0803ab0be653b35fadb4ee4
2021-03-30 16:21:48 +00:00
Fadila Khadar
8d4df93d45 satellite/metainfo: remove TODO
remove TODO that we won't do.

Change-Id: I6ab3ddf2d9285006bed0fc5700da8c6738aa7792
2021-03-30 13:55:54 +00:00
Egon Elbre
794dc6d6d5 satellite/metainfo/metabase: add PlainOffset to segment listing
Change-Id: I8bbb5fcebf9f171033938eaae9f415b3a4b5aaa0
2021-03-30 16:20:21 +03:00
Egon Elbre
c4cb79ff1f satellite/metainfo/metabase: update segment plain offset, when needed
CommitObject was not updating plain_offset on segments.

Change-Id: I7e56c873bb15b893bf298b73e0349eb18d855a37
2021-03-30 16:20:21 +03:00
NickolaiYurchenko
f2718734f6 web/storagenode: bug fixes
WHAT:
- fix for wrong total held calcutation on refresh
- payout history year restrictions depends on selected satellite fixed
- payout history total changed if distributed amount > 0
- dark mode info icon colors changed
- option menu closing issue fixed

Change-Id: Ia289d51f081b782ae601722b7c05c3307f56541d
2021-03-30 14:43:51 +03:00
NickolaiYurchenko
b91ba7057e web/storagenode: added zksync wallet explorer button
WHAT: link to zkscan if zksync is opted in

WHY: to have ability to see wallet status redirected from dashboard

Change-Id: Ic2d084c1e551f461307fb5a1d5eb1741ed7cdf85
2021-03-30 08:47:11 +00:00
Michał Niewrzał
5dbc91622f metainfo-migration: fix flaky TestMigrator_SingleSegmentObj test
Test had two issues:
* sub test was using parent test to fail (ctx.Check)
* pointerDB and metabaseDB were using the same unique DB and pointerDB was closing/deleting it first

Change-Id: I5741b76d518663d80c4e6448c76e4ee9dd86c8e1
2021-03-29 17:31:58 +00:00
Moby von Briesen
23c8331d1f satellite/analytics: Add flag for disabling analytics (default disabled)
Just in case we need to have an easy flag for turning analytics on or
off.

Change-Id: I219071200e0a4c156ba92cbbaa4de4ab75b2cb68
2021-03-29 16:23:47 +00:00
Jeff Wendling
a65aecfd98 compensation: always generate invoices for every node
instead of only generating invoices for nodes that had some
activity, we generate it for every node so that we can find
and pay terminal nodes that did not meet thresholds before
we recognized them as terminal.

Change-Id: Ibb3433e1b35f1ddcfbe292c034238c9fa1b66c44
2021-03-29 14:15:45 +00:00
Kaloyan Raev
035c393da0 satellite: update tests to pass etag.Reader to multipart.PutObjectPart
Change-Id: Ibe99357945ae7a91f5b5d4f87b83d425c9fa84a5
2021-03-29 13:18:11 +00:00
Egon Elbre
74b243d1ca certificate: move protobuf to storj/storj repository
Change-Id: I4689318304b54121a65e28860430d39d94ac5bd6
2021-03-29 12:59:49 +00:00
Michał Niewrzał
141444f6d6 satellite/repair/repairer: fix segmentAge metric
Change-Id: I146b3163aa1bfab5ee060298e6bf9822ca6820a0
2021-03-29 12:29:47 +00:00
Kaloyan Raev
4b14839e14 make: bump-dependencies should update uplink from main branch
Change-Id: I140b15d8d67e1283af820b4a54b86c2b714e82ca
2021-03-29 11:48:39 +00:00
Egon Elbre
d57873fd45 satellite/overlay: remove Inspector
Currently overlay.Inspector had two rpc methods and both of them were
unimplemented.

Change-Id: I1a2ecc7b7113898fa234a1c1fe451c8cc9e2ee81
2021-03-29 12:26:10 +03:00
Egon Elbre
86e698f572 pb: use *UnimplementedServer to avoid breaking API changes
Change-Id: I99a34eeb37ac4453411f273511710562a519f57a
2021-03-29 12:26:10 +03:00
Michał Niewrzał
c8a3e0ff96 tests: fix calculation of previous month for billing integration test
Change-Id: I413629520aa52518395189c2ea56000cab67c948
2021-03-29 10:38:59 +02:00
Cameron Ayer
05f8d2d0b1 satellite/satellitedb: filter offline suspended nodes from selection
Change-Id: I5a6f413453332238d579a7bf50eb30e9156f96c2
2021-03-27 23:36:46 +00:00
Cameron Ayer
1a51049ac0 satellite/{overlay,satellitedb}: add flag to toggle suspending nodes for offline audits
This change introduces a new config flag,
--overlay.audit-history.offline-suspension-enabled,
to toggle suspending nodes for offline audits.

If the flag is set to true, nodes will be suspended if they meet the
requirements.

If the flag is false, nodes will not be suspended. If they are already
suspended and/or under review, these will be cleared.

Change-Id: Ibeba759c42d6e504f6b7598120d4fd4dab85ca74
2021-03-27 16:28:27 +00:00
Cameron Ayer
eb44dc21b4 satellite/satellitedb: select stray nodes for DQ in separate tx from update
Previously we would select a limited number of nodes for DQ in a
CTE and run the update on that set in a single transaction. This
could lead to locking on the table, so instead we select and update
in separate transactions.

Change-Id: I1e802c0845e829eeadcee4fa382f58462515fdb1
2021-03-27 00:00:23 +00:00
JT Olio
2ddbaf1eb5 satellite/wasm: support restricting full access grants to paths
Change-Id: Id6d4fa41db068d32e7c0d542d9d8805fba927fc6
2021-03-26 19:49:21 +00:00
Malcolm Bouzi
527b7eb56b web/satellite: ui changes to registration and billing pages to enable user to add promo codes to their account
- add Credit History table to billing acount page and set up ui for a user adding promo codes
- implement promo codes ui in registration form
- add feature flag to handle if coupon code ui should be rendered

Change-Id: I9fdeef7cffc7901958d3f9be335e1115b2471a2e
2021-03-26 11:57:11 -04:00
Michał Niewrzał
9bc2a9a4b7 rollingupgrade: change how metainfo-migration tool is installed
Change-Id: Iad1285a770f76c9cf09a872a4bf1cd39969ee117
2021-03-26 15:03:39 +00:00
Michał Niewrzał
3e91d22ce6 satellite/metainfo/metabase: update repair_at with UpdateSegmentPieces
We need to update repair_at column while repairing segments.

Change-Id: I9770fe2e11a7e4b81a39470c50bb1d2aa6cfae29
2021-03-26 11:47:25 +00:00
Moby von Briesen
2a68e0767e scripts/tests/testversions: Fix testversions Episode 2
The previous fix did not remove the comment at the front of the config
values that were modified.

Change-Id: I05729c30a5398d049c6254afec01df00230d8deb
2021-03-25 18:06:13 -04:00
Moby von Briesen
63cd807c9f scripts/tests/testversions: Increase bandwidth/storage limits for testversions
A recent change made the default usage/storage limits for projects 50gb
rather than 500gb. This increases the default limit for testversions.

Change-Id: Ibea05c0d0760662e447b6455d560a2a640801c6c
2021-03-25 14:43:28 -04:00
Moby von Briesen
3db52491ec satellite/analytics: Add analytics service to satellite
* Set up basic structure of new service.
* Implement a basic analytics track event for user creation.

Change-Id: Ica8c785540b1ef9d848404af307a22f21d33c6aa
2021-03-25 13:28:51 -04:00
Vitalii Shpital
c4b2d76d1c web/satellite: buckets view for objects page
WHAT:
buckets management view for objects page

WHY:
to be able to create and delete buckets

Change-Id: I6df986b52928433f7a0a4c4772d3064c4f1a1516
2021-03-25 16:37:55 +02:00
Vitalii Shpital
ebf6bee0d4 web/satellite: import and setup AWS s3 client
WHAT:
import and setup S3 client that will be used to manage buckets

WHY:
to manage buckets

Change-Id: I27f5c03082e687aab32346eedd889a2218407238
2021-03-25 13:50:18 +00:00
Vitalii Shpital
346b85b66d web/satellite: generate gateway credentials for objects page
WHAT:
generate gateway credentials that will be used to instantiate s3 client

WHY:
for s3 client that will be used to manage buckets

Change-Id: I6d654e48c41925b72e11ec3edde3dc54f5290d42
2021-03-25 13:11:51 +00:00