Commit Graph

5231 Commits

Author SHA1 Message Date
Egon Elbre
f19ef4afe5 satellite/metainfo/metaloop: move loop to a separate package
Change-Id: I94c931a27c1af6062185ec62688624ec02050f11
2021-03-23 15:37:34 +00:00
JT Olio
972d40df6f pkg/server: use drpcmigrate listenmux
Change-Id: I569367f04958ef836c6b02190385d0c936fc8268
2021-03-23 09:18:26 -06:00
Egon Elbre
54c2ace483 mod: update drpc to v0.0.19
Change-Id: Ia2e0e0a371368700c596ce89e24748ea4b5c016a
2021-03-23 17:13:28 +02:00
Michał Niewrzał
50b08a5b00 metainfo-migrator: handle 'omit' conn string
Uplink tests are running storj/storj tests only against postgres. Without this change integration test on uplink will fail as 'omit' is not supported in migrator tests code.

Change-Id: Ic72406f52439e98683d050508fb42aa41632e183
2021-03-23 13:33:22 +00:00
Michał Niewrzał
9d52451464 satellite/metainfo/expireddeletion: add config value to set list limit
We want to control how many expired objects we will query at once.

Change-Id: Iba7c79a6236c6b720b53bb8f078d55514690d8e5
2021-03-23 09:51:56 +00:00
Michał Niewrzał
237782813b Merge remote-tracking branch 'origin/multipart-upload'
Change-Id: If6c5a450b238adab55d1e0dea67d01e5f5768a9f
2021-03-23 09:44:49 +01:00
Moby von Briesen
d2b365bf9f satellite/payments: Disable paywall (free tier)
We are implementing the free tier, which will give all new users 3
projects, 50gb storage, and 50gb bandwidth per project. All users will
receive a recurring coupon to cover this amount of usage.

With the free tier, we no longer need a paywall. Users will not need to
enter a payment method unless they want to increase their project or
usage limits.

Change-Id: If3b026e91858e5f557a2758e366616cecc8f21c7
2021-03-22 17:11:20 -04:00
Cameron Ayer
37602d86d3 satellite/satellitedb: drop columns uptime_reputation_alpha and uptime_reputation_beta
Change-Id: I1e0c5f2f96a920070afb76851da98208654b02cf
2021-03-22 14:33:17 -04:00
Cameron Ayer
864ad70fe2 satellite/overlay/straynodes: set --stray-nodes.enable-dq release default to true
Since we will enable this on all satellites, just set default to true

Change-Id: Ibc86a0afd0b0f57e86bd067abb9cdf06c295a467
2021-03-22 17:25:09 +00:00
Cameron Ayer
2607b16070 satellite/{overlay/straynodes,satellitedb}: rework DQNodesLastSeenBefore to return DQd node IDs and last contact successes
We would like to log Node IDs and last contact successes of nodes DQd
in this manner. We would also like to avoid returning an unbounded list
of items from the db. Therefore we change the query to select a limited
number of nodes that meet the DQ conditions and iterate until 0 rows are
returned. Each column of the query is already indexed.

Change-Id: Iaec2d9b56e7202b7c2028ba21750d40c8dd506ee
2021-03-22 13:01:30 -04:00
Fadila Khadar
a8b5f8f990 satellite/metainfo/metabase: select then delete expired objects
We were deleting expired objects by directly executing a delete query.
With this change, we first select the objects to be deleted and then
delete them (as recommended by cockroachdb for deleting using a non indexed
column).

Change-Id: Ied150fbdc7031a343a74e0b9dab316598188ef66
2021-03-22 17:39:13 +01:00
NickolaiYurchenko
2525e15842 web/multinode: favicon added
Change-Id: I74c0bf8e27e47e5c8da06cd4ffb985a483bd47c2
2021-03-22 12:50:53 +00:00
Ivan Fraixedes
536990dcf6 Jenkins: Run integration test Redis unavailability
Make Jenkins to run the integration tests that verifies that the
satellite can operates when Redis is unavailable.

Change-Id: Idace3ffb84c788f2af2c6e24eec1d63fb40c263a
2021-03-22 09:56:30 +00:00
Ivan Fraixedes
fa8b0a29ea scripts: Do not use Docker for test-sim Redis test
Do not use Docker for running a Redis server in the test-sim that we
check that Satellite can still operate when Redis is unavailable because
we have to run this test in Jenkins and we don't want to empower a
container to be able to connect to the Docker socket of the host machine
for running sibling containers.

Change-Id: I6180e8ed804968c8ccb0783ed334acab38af9a0f
2021-03-22 08:43:53 +00:00
Michał Niewrzał
27ae0d1f15 satellite/metainfo/metabase: add NewRedundancy parameter for UpdateSegmentPieces method
At some point we might try to change original segment RS values and set Pieces according to the new values. This change adds add NewRedundancy parameter for UpdateSegmentPieces method to give ability to do that. As a part of change NewPieces are validated against NewRedundancy.

Change-Id: I8ea531c9060b5cd283d3bf4f6e4c320099dd5576
2021-03-22 08:12:56 +00:00
Moby von Briesen
78e132cb74 satellite/satellitedb: Add coupon_codes table and allow optional link to coupons table
The coupon_codes table will allow for administrators to create new promo
codes associated with coupon information (amount, duration, etc...).

A user will be able to enter a promo code (aka coupon code) in order to
apply a new coupon to their account. The coupon in the coupons table is
linked to the template defined in the coupon_codes table.

Change-Id: I50e49fa92afbc6aa9d01d8a895c069efb59e472b
2021-03-19 20:46:28 +00:00
Fadila Khadar
e613c641d7 satellite/metainfo/metabase: benchmark expired deletion
Change-Id: Icc2e8227b8a96002563ff69f3e9cd8e71369f0d1
2021-03-19 11:30:00 +00:00
Michał Niewrzał
1c02158620 satellite/metainfo/metabase: add RepairedAt to RawSegment
Change-Id: I3f8bbed0f815f8394fc6ab16a914cae276fbab56
2021-03-19 11:06:59 +00:00
Vitalii Shpital
c3ae122aa7 web/satellite: enter passphrase step for objects page
WHAT:
enter passphrase step for users who has already created passphrase

WHY:
to let users proceed to upload step

Change-Id: I084aec5b863981978cf190f99ee95154fbed9aab
2021-03-19 10:59:32 +00:00
Yingrong Zhao
bf4fdc7717 cmd/uplink: add --client.enable-quic flag
Change-Id: I19e81ecb98688fd0a36174fd06c363e9c40b3866
2021-03-19 01:24:16 +00:00
Cameron Ayer
a04495713d satellite/audit: add missing logs for audit failure conditions
Among other conditions, nodes fail audits by returning incorrect
data and by reaching the max reverify count, but we weren't logging
these events. This commit adds the missing logs.

Change-Id: I80749a7e95e8cb97bc8dd7dac1e523e223114b7f
2021-03-18 17:33:11 +00:00
Kaloyan Raev
be8a70309d satellite/metainfo/metabase: rename etag column to encrypted_etag
Change-Id: I79662d9a2cec2da6ea9bba250f0e71978119558d
2021-03-18 18:34:29 +02:00
Ivan Fraixedes
4c1098e571 Redis: Update Redis package to last major version
Update the Redis dependency to use the last major production version.
The last version accepts a context parameter in all the network methods
so it allows us to pass it through them.

Change-Id: I34121b2ec3c2728602115c724933ad24c9e6e4fd
2021-03-18 14:19:49 +00:00
Jeff Wendling
09adfc1205 mod: bump monkit to v3.0.10
Change-Id: Ieed0b4161b158edd7c8e7065f200675cfdfdebf8
2021-03-18 13:17:57 +00:00
Kaloyan Raev
36bc17ef5b satellite/metainfo/metabase: add etag column to segments table
We will use it keep track of ETag for object parts.

Change-Id: Ib610397afd44d71a1b5fe4301940de6f36ba39ae
2021-03-18 10:40:55 +00:00
Vitalii Shpital
cff1052bca web/satellite: save passphrase hash in local storage
WHAT:
store passphrase's hash in browser local storage

WHY:
to make user use the same passphrase

Change-Id: I3d5e601a0257a7c900e8a0ce3cdce953f1174830
2021-03-17 19:50:53 +02:00
Vitalii Shpital
9e20cfb6e0 web/satellite: create passphrase step for objects page
WHAT:
made a common component from AG's generate passphrase component to be reused here.
create passphrase step for objects page

WHY:
used for generating passphrase or setting user's own password for proceeding in upload flow

Change-Id: Iefa106c544a1c5ff04d591d3d400cf06c4c6d853
2021-03-17 15:12:33 +00:00
Vitalii Shpital
b5f83f463b web/satellite: routes and initial components for objects page
WHAT:
routes and components for new upload file feature

WHY:
upload file from browser feature. Step 1

Change-Id: If517fd4513757aeaee64764d1894f1af3e463b55
2021-03-17 15:12:28 +00:00
Michał Niewrzał
fa083a7f05 Merge remote-tracking branch 'origin/main' into multipart-upload
Change-Id: Ib5ce5965b77b81c254d08c27ab30c7eccefbd4c6
2021-03-17 15:37:17 +01:00
Egon Elbre
9491df76e7 satellite/satellitedb: optimize StoragenodeAccounting.SaveRollup
Change-Id: I758049872b4ea1ae22657dd9fcb47c228468b1d0
2021-03-16 18:11:40 +02:00
Vitalii Shpital
6a553ec9c5 web/satellite: change banner for beta satellites with URLs
WHAT:
beta satellite top banner's copy is changed to include support/feedback URLs

WHY:
so users using our beta satellite will be able to report feedback somewhere

Change-Id: Ibc349c8b3354b577275fcf1d2b75bfdd267729d9
2021-03-15 17:12:07 +00:00
Isaac Hess
1de6315b83 metainfo: Stub DownloadObject metainfo endpoint
This is the stub "not implemented" endpoint for the new DownloadObject
RPC.

Change-Id: Ifa724e20d89d839f2b6968d6283bc66a3bab8781
2021-03-15 10:52:57 -06:00
Michał Niewrzał
65957c7525 satellite/metainfo/expireddeletion: temporarily log errors instead failing whole system
At the moment we are trying to optimize deletion queries but its hard to verify deletion performance. Until we are sure that the queries are good we will just log errors instead shutting down whole satellite core.

Change-Id: I5625251d4518c35f0d46d6bf37b2f3ea7950675e
2021-03-15 16:00:20 +01:00
Yingrong Zhao
a36d854d8e bump monkit-jaeger to latest
Change-Id: I8c20769e667afd3e22ebd10e2e045d2e15fad7f7
2021-03-15 14:42:03 +00:00
Qweder93
0b060c8297 storagenode/console: estimated payout flaky test skip
Change-Id: Ibb20b5716df94f9430026ed697d57b6be9cc137d
2021-03-15 15:26:47 +02:00
Kaloyan Raev
c4f0289904 satellite/metainfo: ListSegments returns CreatedAt in response
If a non-nil value is read from created_at column of the segments table,
it will be set to the CreatedAt field if SegmentListItem.

Change-Id: I02691d8e11fad12c1b0e4c443bdebb568016ffe3
2021-03-15 11:45:25 +00:00
Kaloyan Raev
6e661da0a0 satellite/metainfo/metabase: read created_at from DB
All SQL queries for reading segments are updated to read the created_at
column where appropriate.

Change-Id: Icd7c7672fa71e992673078598b28229bb898c728
2021-03-15 11:37:14 +00:00
Qweder93
487ad84b23 storagenode:consoleAPI flacky tests for estimatedPayout fixed
joinedAt date now is 2 months before time.Now which should fix calculations on early dates.

Change-Id: I6250ec2463c2dd609e811b74d908a02a3e64f19a
2021-03-13 20:33:25 +02:00
Michał Niewrzał
887f3b04e1 satellite/metainfo/metabase: pieces vs redundancy scheme validation while commiting segment
Additional validation for pieces based on RS.

Change-Id: Ib25bfc0e38ca78e2c3194a3223d59257489049fb
2021-03-11 16:12:52 +00:00
littleskunk
dcdfa10370
jenkins/build: rename release binaries inside zip archive (#4060)
Co-authored-by: Ivan Fraixedes <ivan@fraixed.es>
2021-03-11 12:35:32 +01:00
Fadila Khadar
018b0684b8 satellite/metabase: different queries for delete_expired
use custom queries for pg and crdb when deleting expired objects.

Change-Id: I2e230b3b34d5128c877ed8f8f77d83b9a935df42
2021-03-11 09:54:03 +00:00
Kaloyan Raev
bdb33b381b satellite/metainfo/metabase: add created_at and repaired_at to segments
The created_at columns is first added without a default value to avoid
setting the current time to existing segments.

Change-Id: Ic2fe3da238422e2949e6f3016fbac04eb89ba037
2021-03-10 19:59:47 +00:00
Cameron Ayer
8138939a2a satellite/satellitedb: remove bad migration step
Migration step 148 will cause errors because we missed some
references to the columns being dropped. Removing the step
altogether causes problems with backwards compatibility tests
because the change already exists in the latest release tag.
To circumvent, we change v148 to an empty migration.

Add methods FindTable and RemoveColumn in private/dbutil/dbschema

Change-Id: Ia527e95b88a88c5dc82800928ce6f8cfb879e334
2021-03-10 14:36:52 -05:00
Michał Niewrzał
e0f07ef04e metainfo-migrator: ignore invalid segment keys
Old pointerdbs can have key with just project id, segment index and bucket, without object key. We need to ignore such keys.

Change-Id: I80a466a94e317a229da236fe6bc9e762e6f7ced6
2021-03-10 17:23:36 +00:00
Michał Niewrzał
80ae7787ee metainfo-migration: add simple retry logic for queries
Change-Id: I2fd93493f9f540a39583d4a2a2a22d25453cb0e5
2021-03-10 14:00:18 +01:00
NickolaiYurchenko
5616795f70 web/multinode: update node name modal
WHAT: markup and logic for set/update node name

WHY: to have abitity to change node name for future purposes like filtering or grouping

Change-Id: I44c679df43d34355efc4c6cff260faa1c3ff480e
2021-03-10 09:44:18 +00:00
Egon Elbre
5e954ad487 satellite/metainfo/metabase: optimize ConvertAliasesToPieces
old time/op    new time/op    delta
    7.56µs ± 5%    4.93µs ± 2%  -34.75%  (p=0.000 n=5+15)

  old alloc/op   new alloc/op   delta
    6.86kB ± 0%    3.85kB ± 0%  -43.87%  (p=0.000 n=5+18)

  old allocs/op  new allocs/op  delta
      19.0 ± 0%      17.0 ± 0%  -10.53%  (p=0.000 n=5+18)

Change-Id: Iedf24087766b3bd90934f2daa7ac186c3503a341
2021-03-10 08:39:23 +00:00
Michał Niewrzał
67e26aafcd Merge remote-tracking branch 'origin/main' into multipart-upload
Change-Id: I9b183323cb470185be22f7c648bb76917d2e6fca
2021-03-10 08:53:38 +01:00
Ivan Fraixedes
84b844a2a7 redis-server: Move testing type to specific testing pkg
Move a specific interface & types used for testing to be a private
subpackage with a name that clearly identifies it for testing purpose.

Change-Id: I646cf3b6f0a3b518a6f9a125998dc5a02df02db6
2021-03-10 06:09:46 +00:00
Yingrong Zhao
1f59a08e85 pkg/server: eliminate Retry packet from quic-go hanshake phase
In order to prevent traffic amplification attack, QUIC allows
application to perform an address validation during connection
establishment.
(https://tools.ietf.org/id/draft-ietf-quic-transport-34.html#name-address-validation)
However, this adds another round trip during connection establishment.
In storj network, it does client authentication before servers starts
sending significant amount of data to any client. We believe the traffic
amplification attack isn't going to be significant when turning off
address validation in QUIC. AND it will provide us a significant
performance boost during connection establishment.

Change-Id: I7f9a0ca5ca770b715d08b1e8ce3022fbb2b85d42
2021-03-09 22:05:35 +00:00