Commit Graph

5480 Commits

Author SHA1 Message Date
Moby von Briesen
c334fd090e satellite/payments: Convert coupon duration to be nullable
* Add a nullable billing_periods column in the coupons table
* Add nullable billing_periods column to the currently unused
coupon_codes table
* Drop the duration column from the coupon_codes table
* Replace duration config type so that the default promotional coupon
can be configured to never expire

Zero downtime migration plan:
* Add billing_periods column to coupons and coupon_codes tables (this change)
* After one release, remove all references to the old duration column,
replacing with references to billing_periods. At this point, we can also
change the defult promotional coupon to never expire and migrate over
values from the old duration column.
* After another release, drop the duration column.

Change-Id: I374e8dc9fab9f81b4a5bc681771955662d4c007a
2021-04-01 16:28:23 +02:00
Qweder93
c4293f5e52 storagenode/payouts: DQ satellites removed from estimated payouts
Change-Id: Ib3fb287bbdece5386a2a6f257b54411ad5b742be
2021-04-01 11:17:27 +00:00
Michał Niewrzał
0b7c288155 test-sim: improve integration test scripts
* integration tests have now a trap that will display where script failed, line and message
* functions from uplink tests moved to utils.sh to reuse them

Change-Id: Ib2311775fd70ce784aa986328969d75eefc5ac36
2021-03-31 16:48:53 +00:00
Tome Boshevski
f6c9b8d4f1
web/satellite: access grant wizard copy edit (#4066) 2021-03-31 18:03:26 +03:00
Yingrong Zhao
7dfb89a871 bump quic-go to latest release
Change-Id: Ic5e93f7a6ffe0bb46fc8f8bf1ce310eac4aa1182
2021-03-31 13:28:41 +00:00
Michał Niewrzał
a5bc38b197 build: replace 'multipart-upload' branch occurences
We removed `multipart-upload` branches but we have few leftovers.

Change-Id: I74f637748a248cb286cd190a3a8f895ef85f3940
2021-03-31 11:06:47 +02:00
JT Olio
86c41790ce satellite/metainfo/metaloop: add observability
we want to know a lot more about what's going on during
the operation of the metainfo loop. this patchset adds
more instrumentation to previously unmonitored but
interesting functions, and adds metrics that keep track
of how far through a specific loop we are. it also
adds mon:lock annotations, especially to the metainfo
loop run task, which recently changed, silently broke
some queries, and thus failed to alert us to spiking
run time issues.

Change-Id: I4358e2f2293d8ebe30eef497ba4e423ece929041
2021-03-30 14:32:05 -06:00
Ivan Fraixedes
f3176adbce private/migrate: Add missing validation table call
Add a missing call to the validation table name method to ensure that
it cannot be SQL injection.

Change-Id: Iee10b09e15c07448f0803ab0be653b35fadb4ee4
2021-03-30 16:21:48 +00:00
Fadila Khadar
8d4df93d45 satellite/metainfo: remove TODO
remove TODO that we won't do.

Change-Id: I6ab3ddf2d9285006bed0fc5700da8c6738aa7792
2021-03-30 13:55:54 +00:00
Egon Elbre
794dc6d6d5 satellite/metainfo/metabase: add PlainOffset to segment listing
Change-Id: I8bbb5fcebf9f171033938eaae9f415b3a4b5aaa0
2021-03-30 16:20:21 +03:00
Egon Elbre
c4cb79ff1f satellite/metainfo/metabase: update segment plain offset, when needed
CommitObject was not updating plain_offset on segments.

Change-Id: I7e56c873bb15b893bf298b73e0349eb18d855a37
2021-03-30 16:20:21 +03:00
NickolaiYurchenko
f2718734f6 web/storagenode: bug fixes
WHAT:
- fix for wrong total held calcutation on refresh
- payout history year restrictions depends on selected satellite fixed
- payout history total changed if distributed amount > 0
- dark mode info icon colors changed
- option menu closing issue fixed

Change-Id: Ia289d51f081b782ae601722b7c05c3307f56541d
2021-03-30 14:43:51 +03:00
NickolaiYurchenko
b91ba7057e web/storagenode: added zksync wallet explorer button
WHAT: link to zkscan if zksync is opted in

WHY: to have ability to see wallet status redirected from dashboard

Change-Id: Ic2d084c1e551f461307fb5a1d5eb1741ed7cdf85
2021-03-30 08:47:11 +00:00
Michał Niewrzał
5dbc91622f metainfo-migration: fix flaky TestMigrator_SingleSegmentObj test
Test had two issues:
* sub test was using parent test to fail (ctx.Check)
* pointerDB and metabaseDB were using the same unique DB and pointerDB was closing/deleting it first

Change-Id: I5741b76d518663d80c4e6448c76e4ee9dd86c8e1
2021-03-29 17:31:58 +00:00
Moby von Briesen
23c8331d1f satellite/analytics: Add flag for disabling analytics (default disabled)
Just in case we need to have an easy flag for turning analytics on or
off.

Change-Id: I219071200e0a4c156ba92cbbaa4de4ab75b2cb68
2021-03-29 16:23:47 +00:00
Jeff Wendling
a65aecfd98 compensation: always generate invoices for every node
instead of only generating invoices for nodes that had some
activity, we generate it for every node so that we can find
and pay terminal nodes that did not meet thresholds before
we recognized them as terminal.

Change-Id: Ibb3433e1b35f1ddcfbe292c034238c9fa1b66c44
2021-03-29 14:15:45 +00:00
Kaloyan Raev
035c393da0 satellite: update tests to pass etag.Reader to multipart.PutObjectPart
Change-Id: Ibe99357945ae7a91f5b5d4f87b83d425c9fa84a5
2021-03-29 13:18:11 +00:00
Egon Elbre
74b243d1ca certificate: move protobuf to storj/storj repository
Change-Id: I4689318304b54121a65e28860430d39d94ac5bd6
2021-03-29 12:59:49 +00:00
Michał Niewrzał
141444f6d6 satellite/repair/repairer: fix segmentAge metric
Change-Id: I146b3163aa1bfab5ee060298e6bf9822ca6820a0
2021-03-29 12:29:47 +00:00
Kaloyan Raev
4b14839e14 make: bump-dependencies should update uplink from main branch
Change-Id: I140b15d8d67e1283af820b4a54b86c2b714e82ca
2021-03-29 11:48:39 +00:00
Egon Elbre
d57873fd45 satellite/overlay: remove Inspector
Currently overlay.Inspector had two rpc methods and both of them were
unimplemented.

Change-Id: I1a2ecc7b7113898fa234a1c1fe451c8cc9e2ee81
2021-03-29 12:26:10 +03:00
Egon Elbre
86e698f572 pb: use *UnimplementedServer to avoid breaking API changes
Change-Id: I99a34eeb37ac4453411f273511710562a519f57a
2021-03-29 12:26:10 +03:00
Michał Niewrzał
c8a3e0ff96 tests: fix calculation of previous month for billing integration test
Change-Id: I413629520aa52518395189c2ea56000cab67c948
2021-03-29 10:38:59 +02:00
Cameron Ayer
05f8d2d0b1 satellite/satellitedb: filter offline suspended nodes from selection
Change-Id: I5a6f413453332238d579a7bf50eb30e9156f96c2
2021-03-27 23:36:46 +00:00
Cameron Ayer
1a51049ac0 satellite/{overlay,satellitedb}: add flag to toggle suspending nodes for offline audits
This change introduces a new config flag,
--overlay.audit-history.offline-suspension-enabled,
to toggle suspending nodes for offline audits.

If the flag is set to true, nodes will be suspended if they meet the
requirements.

If the flag is false, nodes will not be suspended. If they are already
suspended and/or under review, these will be cleared.

Change-Id: Ibeba759c42d6e504f6b7598120d4fd4dab85ca74
2021-03-27 16:28:27 +00:00
Cameron Ayer
eb44dc21b4 satellite/satellitedb: select stray nodes for DQ in separate tx from update
Previously we would select a limited number of nodes for DQ in a
CTE and run the update on that set in a single transaction. This
could lead to locking on the table, so instead we select and update
in separate transactions.

Change-Id: I1e802c0845e829eeadcee4fa382f58462515fdb1
2021-03-27 00:00:23 +00:00
JT Olio
2ddbaf1eb5 satellite/wasm: support restricting full access grants to paths
Change-Id: Id6d4fa41db068d32e7c0d542d9d8805fba927fc6
2021-03-26 19:49:21 +00:00
Malcolm Bouzi
527b7eb56b web/satellite: ui changes to registration and billing pages to enable user to add promo codes to their account
- add Credit History table to billing acount page and set up ui for a user adding promo codes
- implement promo codes ui in registration form
- add feature flag to handle if coupon code ui should be rendered

Change-Id: I9fdeef7cffc7901958d3f9be335e1115b2471a2e
2021-03-26 11:57:11 -04:00
Michał Niewrzał
9bc2a9a4b7 rollingupgrade: change how metainfo-migration tool is installed
Change-Id: Iad1285a770f76c9cf09a872a4bf1cd39969ee117
2021-03-26 15:03:39 +00:00
Michał Niewrzał
3e91d22ce6 satellite/metainfo/metabase: update repair_at with UpdateSegmentPieces
We need to update repair_at column while repairing segments.

Change-Id: I9770fe2e11a7e4b81a39470c50bb1d2aa6cfae29
2021-03-26 11:47:25 +00:00
Moby von Briesen
2a68e0767e scripts/tests/testversions: Fix testversions Episode 2
The previous fix did not remove the comment at the front of the config
values that were modified.

Change-Id: I05729c30a5398d049c6254afec01df00230d8deb
2021-03-25 18:06:13 -04:00
Moby von Briesen
63cd807c9f scripts/tests/testversions: Increase bandwidth/storage limits for testversions
A recent change made the default usage/storage limits for projects 50gb
rather than 500gb. This increases the default limit for testversions.

Change-Id: Ibea05c0d0760662e447b6455d560a2a640801c6c
2021-03-25 14:43:28 -04:00
Moby von Briesen
3db52491ec satellite/analytics: Add analytics service to satellite
* Set up basic structure of new service.
* Implement a basic analytics track event for user creation.

Change-Id: Ica8c785540b1ef9d848404af307a22f21d33c6aa
2021-03-25 13:28:51 -04:00
Vitalii Shpital
c4b2d76d1c web/satellite: buckets view for objects page
WHAT:
buckets management view for objects page

WHY:
to be able to create and delete buckets

Change-Id: I6df986b52928433f7a0a4c4772d3064c4f1a1516
2021-03-25 16:37:55 +02:00
Vitalii Shpital
ebf6bee0d4 web/satellite: import and setup AWS s3 client
WHAT:
import and setup S3 client that will be used to manage buckets

WHY:
to manage buckets

Change-Id: I27f5c03082e687aab32346eedd889a2218407238
2021-03-25 13:50:18 +00:00
Vitalii Shpital
346b85b66d web/satellite: generate gateway credentials for objects page
WHAT:
generate gateway credentials that will be used to instantiate s3 client

WHY:
for s3 client that will be used to manage buckets

Change-Id: I6d654e48c41925b72e11ec3edde3dc54f5290d42
2021-03-25 13:11:51 +00:00
Fadila Khadar
1c696168c5 satellite/metainfo/metabase: delete objects and segments in one query
We were using two queries to delete one or more objects and its/their
segments in DeleteObjectExactVersion, DeletePendingObject, DeleteObjectLatestVersion,
DeleteObjectAnyStatusAllVersions,DeleteObjectsAllVersions.

This change delete objects and their segments in one query.

Change-Id: Ib2c0eb501f00b091ee32519e02155350c4dcb8b0
2021-03-25 11:13:18 +00:00
Michał Niewrzał
0ef0a3f967 rollingupgrade: add migration to metabase
For time of transition from pointerdb to metabase we need add migration step to rollingupgrade tests and comment few cases.

Change-Id: Ib12ae6aa14be35f9bf4ff3efb55cfc6957d4ceba
2021-03-25 10:52:35 +00:00
Vitalii Shpital
6ddcacbe78 web/satellite: initial setup of bucket's view of objects page
WHAT:
setup of objects store module,
setup of initial bucket's view,
generate special API key that will be used to generate gateway credentials

WHY:
initial setup of future bucket's management view

Change-Id: I0078869b95c04c0b142b2e112e93ff2332e8e90f
2021-03-25 09:48:16 +00:00
Kaloyan Raev
8e1aa4bb74 satellite/metainfo: etag support for committing and listing segments
Change-Id: I9a4cdf0cffa887efc856b191ee0939c11e8391e7
2021-03-25 10:08:21 +02:00
Kaloyan Raev
ec929ca1da satellite/metainfo/metabase: add support for encrypted ETag
Change-Id: I142d5f759a452b12bf2ce67e6c939846a5e86ce7
2021-03-25 10:01:16 +02:00
Kaloyan Raev
cc42dfcc78 satellite/metainfo: add EncryptionParameters to SatStreamID
We have a use case for this in ListSegments. ListSegments is going to
return the EncryptedETag along with EncryptedKey and EncryptedKeyNonce.
It also must return the EncryptionParameters.

Since the EncryptionParameters are in the objects DB table, it would be
more efficient for ListSegment to avoid querying that DB table, but take
it from the SatStreamID.

Change-Id: I16c98641c0fe0c98e3303329d0da6ef137ca55cf
2021-03-25 07:54:34 +00:00
Ivan Fraixedes
c5cb4dce4d redis: Rename functions prefixed with New by Open
Rename the functions that are prefixed with 'New' which connect with
Redis by 'Open' to  make clear that they perform network operations.

Change-Id: I1351e89a642e8e2c2586626646315ad0fb2c6242
2021-03-25 06:09:27 +00:00
JT Olio
903b1b5ffb satellite ui: don't call the gateway mt the gateway mt
users don't know what that is yet.

Change-Id: Ifb3b3e6bd51a75326a581cbd784ee364d7c8400b
2021-03-24 18:37:54 +00:00
Kaloyan Raev
1156d8b4ed mod: update drpc to v0.0.20
Change-Id: I0ac6ff9bebe5b16d4fb63b30242c87c899e7d358
2021-03-24 19:06:56 +02:00
Qweder93
866a3e3ad3 storagenode/console: estimated payout tests fix
Added check if days paste <1 and node joined in current month -> days paste = 1

Change-Id: Ice1ccd6c869f629a87da585850b9b4f3729ec65c
2021-03-24 18:05:37 +02:00
Moby von Briesen
27bcb46718 satellite/console: change default project and usage limits
This is one step for implementing the free tier:
* Change the default project limit from 10 to 3
* Move storage and bandwidth project usage limits from the metainfo
package to the console package (otherwise there is a cyclical
dependency, and metainfo doesn't use these values anyway)
* Change the default storage usage limit per project from 500gb to 50gb
* Change the default bandwidth usage limit per project from 500gb to 50gb
* Migrate the database so that old users and projects continue to have
the old defaults (10 projects/500gb usage)

Change-Id: Ice9ee6a738bc6410da18c336c672d3fcd0cab1b9
2021-03-24 14:24:27 +00:00
NickolaiYurchenko
034eb2c214 storagenode: wallet features
WHAT: updated dashboard type to return wallet features from storagenode api

Change-Id: Ifdbdeae8b9ae239eaebe32d0a093557154f777ad
2021-03-24 09:08:36 +00:00
Vitalii Shpital
3e37d1e71c satellite/console: delete api key by name and project id endpoint
WHAT:
new endpoint to be able to delete apiKey/accessGrant by name and project id

WHY:
it will be called to delete special pregenerated access grant which will be used to generate gateway credentials for file browser component or bucket management

Change-Id: I7467ebaab27a7da33efd062536c6da41e6ed4c30
2021-03-23 20:21:07 +00:00
Egon Elbre
f19ef4afe5 satellite/metainfo/metaloop: move loop to a separate package
Change-Id: I94c931a27c1af6062185ec62688624ec02050f11
2021-03-23 15:37:34 +00:00