Commit Graph

954 Commits

Author SHA1 Message Date
JT Olio
f3c8cac1ee storj/storj: more domain changes
Change-Id: I91ae3bc36be38c27269f9e6e678fa3616b8e4d53
2021-04-14 20:49:12 +00:00
Fadila Khadar
bde367ae73 satellite/gc: check on bloom filter creation date
Check that the bloom filter creation date is earlier than the
metainfo loop system time used for db scanning.

Change-Id: Ib0f47c124f5651deae0fd7e7996abcdcaac98fb4
2021-04-14 16:40:37 +00:00
JT Olio
afcc55fb47 storj/storj: some domain changes we can make now
Change-Id: I5622edea9dfcad788cac65e1754b331b9fed6154
2021-04-13 17:12:06 +00:00
Kaloyan Raev
a264a4422b cmd: remove metainfo-migrator and metainfo-verifier
We already merged the multipart-upload branch to main. These two tools
make only sense if we are migrating a satellite from Pointer DB to
Metabase. There is one remaining satellite to migrate, but these tools
should be used from the respective release branch instead of from main.

Removing these tools from main will:
1) Avoid the mistake to use them from the main branch instead of from
the respective release branch.
2) Allow to finally remove any code related to the old Pointer DB.

Change-Id: Ied66098c5d0b8fefeb5d6e92b5e0ef5c6603df5d
2021-04-08 14:50:29 +00:00
Michał Niewrzał
6b88a675c5 metabase-createdat-migration: add migration tool
We recently added create_at column to segments table.
Old segments needs to get this value from objects table.
This tool will iterate over all objects and update corresponding
segments if create_at column is not set.

Change-Id: Ib5aedc384637e739ee9af84454af0639e2559416
2021-04-08 14:20:09 +00:00
Moby von Briesen
c334fd090e satellite/payments: Convert coupon duration to be nullable
* Add a nullable billing_periods column in the coupons table
* Add nullable billing_periods column to the currently unused
coupon_codes table
* Drop the duration column from the coupon_codes table
* Replace duration config type so that the default promotional coupon
can be configured to never expire

Zero downtime migration plan:
* Add billing_periods column to coupons and coupon_codes tables (this change)
* After one release, remove all references to the old duration column,
replacing with references to billing_periods. At this point, we can also
change the defult promotional coupon to never expire and migrate over
values from the old duration column.
* After another release, drop the duration column.

Change-Id: I374e8dc9fab9f81b4a5bc681771955662d4c007a
2021-04-01 16:28:23 +02:00
Michał Niewrzał
5dbc91622f metainfo-migration: fix flaky TestMigrator_SingleSegmentObj test
Test had two issues:
* sub test was using parent test to fail (ctx.Check)
* pointerDB and metabaseDB were using the same unique DB and pointerDB was closing/deleting it first

Change-Id: I5741b76d518663d80c4e6448c76e4ee9dd86c8e1
2021-03-29 17:31:58 +00:00
Jeff Wendling
a65aecfd98 compensation: always generate invoices for every node
instead of only generating invoices for nodes that had some
activity, we generate it for every node so that we can find
and pay terminal nodes that did not meet thresholds before
we recognized them as terminal.

Change-Id: Ibb3433e1b35f1ddcfbe292c034238c9fa1b66c44
2021-03-29 14:15:45 +00:00
Egon Elbre
d57873fd45 satellite/overlay: remove Inspector
Currently overlay.Inspector had two rpc methods and both of them were
unimplemented.

Change-Id: I1a2ecc7b7113898fa234a1c1fe451c8cc9e2ee81
2021-03-29 12:26:10 +03:00
JT Olio
2ddbaf1eb5 satellite/wasm: support restricting full access grants to paths
Change-Id: Id6d4fa41db068d32e7c0d542d9d8805fba927fc6
2021-03-26 19:49:21 +00:00
Ivan Fraixedes
c5cb4dce4d redis: Rename functions prefixed with New by Open
Rename the functions that are prefixed with 'New' which connect with
Redis by 'Open' to  make clear that they perform network operations.

Change-Id: I1351e89a642e8e2c2586626646315ad0fb2c6242
2021-03-25 06:09:27 +00:00
Egon Elbre
f19ef4afe5 satellite/metainfo/metaloop: move loop to a separate package
Change-Id: I94c931a27c1af6062185ec62688624ec02050f11
2021-03-23 15:37:34 +00:00
Michał Niewrzał
50b08a5b00 metainfo-migrator: handle 'omit' conn string
Uplink tests are running storj/storj tests only against postgres. Without this change integration test on uplink will fail as 'omit' is not supported in migrator tests code.

Change-Id: Ic72406f52439e98683d050508fb42aa41632e183
2021-03-23 13:33:22 +00:00
Michał Niewrzał
237782813b Merge remote-tracking branch 'origin/multipart-upload'
Change-Id: If6c5a450b238adab55d1e0dea67d01e5f5768a9f
2021-03-23 09:44:49 +01:00
Yingrong Zhao
bf4fdc7717 cmd/uplink: add --client.enable-quic flag
Change-Id: I19e81ecb98688fd0a36174fd06c363e9c40b3866
2021-03-19 01:24:16 +00:00
Ivan Fraixedes
4c1098e571 Redis: Update Redis package to last major version
Update the Redis dependency to use the last major production version.
The last version accepts a context parameter in all the network methods
so it allows us to pass it through them.

Change-Id: I34121b2ec3c2728602115c724933ad24c9e6e4fd
2021-03-18 14:19:49 +00:00
Michał Niewrzał
e0f07ef04e metainfo-migrator: ignore invalid segment keys
Old pointerdbs can have key with just project id, segment index and bucket, without object key. We need to ignore such keys.

Change-Id: I80a466a94e317a229da236fe6bc9e762e6f7ced6
2021-03-10 17:23:36 +00:00
Michał Niewrzał
80ae7787ee metainfo-migration: add simple retry logic for queries
Change-Id: I2fd93493f9f540a39583d4a2a2a22d25453cb0e5
2021-03-10 14:00:18 +01:00
Michał Niewrzał
67e26aafcd Merge remote-tracking branch 'origin/main' into multipart-upload
Change-Id: I9b183323cb470185be22f7c648bb76917d2e6fca
2021-03-10 08:53:38 +01:00
Michał Niewrzał
110c16ddc6 metainfo-loop-benchmark: run benchmark on the top of storj.io/private/process
We want to have access to monitoring tools from `storj.io/private/process`.

Change-Id: I565840ba1ae981c8dee53d588c5185d5981e9728
2021-03-09 11:26:45 +01:00
Michał Niewrzał
4d08f06133 metainfo-loop-benchmark: add memory statistics
Change-Id: I7557c09bc80f7ef7f9718212c74443023a69c5a8
2021-03-09 09:39:20 +01:00
Michał Niewrzał
ce4024a521 metainfo-migrator: handle objects with missing segments
We can have objects with missing segments. Such objects will be removed by segment reaper but between executions we can have it. We should not interrupt migration but collect such objects to cleanup migrated DB later.

Change-Id: I5cc4a66395c1773a6430f34cc25a0f2449133f80
2021-03-08 13:43:26 +00:00
Natalie Villasana
856db68fd9 satellite/gracefulexit: extend GE data cleanup to include exit_progress
The new 'consistency ge-cleanup-orphaned-data' cli command deleted
orphaned transfer queue items, but not entries in the
graceful_exit_progress table. This will delete orphaned entries
from the exit progress table too.

Change-Id: I5f927aac1f258490678deaf179be92ccfe10fcd8
2021-03-01 15:52:43 +00:00
Fadila Khadar
cf0c88c679 cmd/uplink: fix no result when listing single file
uplink ls sj://bucket/file.ext gives a result if file.ext exists.

Change-Id: Ife741926acacf510e4e098009ce1724386c7522f
2021-02-26 11:43:02 +00:00
Fadila Khadar
8bbabe57db cmd/uplink: allow deletion of pending objects
Basic command for deleting a pending object by object key.

Change-Id: I47f68acf79cb0568a42a65b6d8195ca309b185a7
2021-02-25 14:23:56 +00:00
Fadila Khadar
e96ed289e5 cmd/uplink: add --pending for listing pending objects
Change-Id: I19c1965117c386f63e7b1c71ba70402d312329f0
2021-02-25 13:50:10 +00:00
Michał Niewrzał
d995fb497f Merge remote-tracking branch 'origin/main' into multipart-upload
Change-Id: I367da03351ab80f7343332420490dde9282aa47a
2021-02-23 12:31:31 +01:00
Egon Elbre
3c49f0ff73 cmd/satellite: restore-trash, cleanup workers
Use a limiter instead of a custom work queue.

Change-Id: I1506daaaaeb2935f6011ddaeb83a0904abb69c98
2021-02-19 19:08:10 +02:00
JT Olio
e23f423107 uplink: add api key back to inspect
Change-Id: I9aa46f3910a76a81a399597d7512056b218e55af
2021-02-19 14:46:35 +00:00
Egon Elbre
4c0ea717eb satellite/metainfo: remove unneeded dependencies from Loop
metainfo.Loop doesn't require buckets nor pointerdb anymore.

Also:
* fix comments
* update full iterator limit to 2500

Change-Id: I6604402868f5c34079197c407f969ac8015e63c5
2021-02-19 15:11:16 +02:00
Egon Elbre
a25e35f0b0 cmd/metainfo-loop-benchmark: add benchmark
Change-Id: I0745cfcf9f8c9d73fd025dcba6ee8a7480273fe2
2021-02-19 10:59:41 +02:00
JT Olio
ac8c70e20d cmd/satellite: restore trash connection close
Change-Id: I96ca0ba653005f519a2d5e5e7eeb9e2c9f894056
2021-02-18 11:25:28 -07:00
JT Olio
b2ed7edd30 cmd/satellite: restore-trash parallel workers
Change-Id: Ic7466b21c20bda334e7ba4268a494e96b6528ac1
2021-02-18 19:11:19 +02:00
Michał Niewrzał
69151965b9 cmd/satellite: add logic to send RestoreTrash to nodes
Change-Id: I69815fa766a40a7266787e96bf9662854ae56937
2021-02-18 09:19:30 -07:00
JT Olio
3ae3389ddc cmd/satellite: restore-trash command
Change-Id: I80fc932c12147692d49cde277784871ac611fcad
2021-02-18 09:19:22 -07:00
Michał Niewrzał
12402eb729 Merge remote-tracking branch 'origin/main' into multipart-upload
Change-Id: I38adf8218c1415c7ea1910f8bd6bed13544b0f03
2021-02-17 08:50:38 +01:00
Michał Niewrzał
79d6294dbe metainfo-migration: fix data race
Change-Id: Idc266509fe7a6a717689f090a1409db09c8e9daa
2021-02-16 09:53:17 +01:00
Kaloyan Raev
9bcffa3d02 cmd/metainfo-verifier: verifies if metainfo migration is done correctly
Change-Id: Id47f32fe098b5f4ac3b5e6829f5dd21e3a98c6da
2021-02-12 14:31:02 +00:00
Michał Niewrzał
341a4b7f59 metainfo-migration: extend test case
Change-Id: I9d1ac41a18bb08200ef20f7ff2f8df5531140f99
2021-02-11 19:28:41 +01:00
Yaroslav Vorobiov
2e1455bc55 multinode/multinodedb: add sqlite3 support
Change-Id: I023ffb75f836de2c33eb7dbee52072e1622448bd
2021-02-11 16:36:05 +00:00
Michał Niewrzał
5a23d9c167 metainfo-migrator: improve performance
Change-Id: I258605737c37dfdd8f0070dbadf0e3a9c6358cc3
2021-02-11 17:33:22 +01:00
Michał Niewrzał
908a96ae30 Merge remote-tracking branch 'origin/main' into multipart-upload
Change-Id: I075aaff42ca3f5dc538356cedfccd5939c75e791
2021-02-11 11:48:23 +01:00
JT Olio
1ac17cfac5 uplink: link.tardigradeshare.io URLs should have an /s/ right after the domain
should be
https://link.tardigradeshare.io/s/<access>/<bucket>/<path>

legacy URLs have the /s/ missing but a redirect is issued

Change-Id: Ic2a3dc092ff68d7706fd888a9fbfc27716877c6e
2021-02-09 14:38:22 -07:00
Yaroslav Vorobiov
1e328f3c30 satellite/compensation: add wallet features to invoice csv
Change-Id: I90a72c0a4c8d07604096913b6680263b6defc0a2
2021-02-08 22:13:48 +02:00
Yingrong Zhao
3b49d3cddf satellite: remove referral program related code
This PR removes all back-end related referral program code including the
marketing portal.

We will have a separate PR for front-end code and database migration to
drop `offers` and `usercredits` table

Change-Id: If59f952cddfe0558a7dc03a0eac7cc1081517f88
2021-02-08 13:52:50 +00:00
Michał Niewrzał
9a60011774 Merge remote-tracking branch 'origin/main' into multipart-upload
Change-Id: Ia90f29be432e207c4125f7f955c912978eabe59a
2021-02-04 09:38:08 +01:00
Jeff Wendling
759bdd6794 satellite/compensation: add total-paid and total-distributed to invoices
Change-Id: Id4414867917cbf8aad77795f764d6381e88d9a34
2021-02-02 18:14:31 +00:00
Kaloyan Raev
6f3d0c4ad5 Merge remote-tracking branch 'origin/main' into multipart-upload
Conflicts:
	go.mod
	go.sum
	satellite/repair/repair_test.go
	satellite/repair/repairer/segments.go

Change-Id: Ie51a56878bee84ad9f2d31135f984881a882e906
2021-02-02 19:19:04 +02:00
Caleb Case
22bc69ad6d
cmd/uplink: Add more details to access inspect
Change-Id: Ia7e600c214d6097b48b9c868532b8fbe21b0035c
2021-02-02 10:25:26 -05:00
Ivan Fraixedes
d93944c57b satellite/orders: Delete unused methods & DB tables
Delete satellite order methods and DB tables which aren't used anymore
after we have done a refactoring on the orders to stuck bucket
information in the orders' encrypted metadata.

There are also configuration parameters and a satellite chore that
aren't needed anymore after the orders refactoring.

Change-Id: Ida3682b95921df70792284b42c96d2508bf8ca9c
2021-02-01 18:01:29 +00:00