Commit Graph

5497 Commits

Author SHA1 Message Date
Vitalii Shpital
c4b2d76d1c web/satellite: buckets view for objects page
WHAT:
buckets management view for objects page

WHY:
to be able to create and delete buckets

Change-Id: I6df986b52928433f7a0a4c4772d3064c4f1a1516
2021-03-25 16:37:55 +02:00
Vitalii Shpital
ebf6bee0d4 web/satellite: import and setup AWS s3 client
WHAT:
import and setup S3 client that will be used to manage buckets

WHY:
to manage buckets

Change-Id: I27f5c03082e687aab32346eedd889a2218407238
2021-03-25 13:50:18 +00:00
Vitalii Shpital
346b85b66d web/satellite: generate gateway credentials for objects page
WHAT:
generate gateway credentials that will be used to instantiate s3 client

WHY:
for s3 client that will be used to manage buckets

Change-Id: I6d654e48c41925b72e11ec3edde3dc54f5290d42
2021-03-25 13:11:51 +00:00
Fadila Khadar
1c696168c5 satellite/metainfo/metabase: delete objects and segments in one query
We were using two queries to delete one or more objects and its/their
segments in DeleteObjectExactVersion, DeletePendingObject, DeleteObjectLatestVersion,
DeleteObjectAnyStatusAllVersions,DeleteObjectsAllVersions.

This change delete objects and their segments in one query.

Change-Id: Ib2c0eb501f00b091ee32519e02155350c4dcb8b0
2021-03-25 11:13:18 +00:00
Michał Niewrzał
0ef0a3f967 rollingupgrade: add migration to metabase
For time of transition from pointerdb to metabase we need add migration step to rollingupgrade tests and comment few cases.

Change-Id: Ib12ae6aa14be35f9bf4ff3efb55cfc6957d4ceba
2021-03-25 10:52:35 +00:00
Vitalii Shpital
6ddcacbe78 web/satellite: initial setup of bucket's view of objects page
WHAT:
setup of objects store module,
setup of initial bucket's view,
generate special API key that will be used to generate gateway credentials

WHY:
initial setup of future bucket's management view

Change-Id: I0078869b95c04c0b142b2e112e93ff2332e8e90f
2021-03-25 09:48:16 +00:00
Kaloyan Raev
8e1aa4bb74 satellite/metainfo: etag support for committing and listing segments
Change-Id: I9a4cdf0cffa887efc856b191ee0939c11e8391e7
2021-03-25 10:08:21 +02:00
Kaloyan Raev
ec929ca1da satellite/metainfo/metabase: add support for encrypted ETag
Change-Id: I142d5f759a452b12bf2ce67e6c939846a5e86ce7
2021-03-25 10:01:16 +02:00
Kaloyan Raev
cc42dfcc78 satellite/metainfo: add EncryptionParameters to SatStreamID
We have a use case for this in ListSegments. ListSegments is going to
return the EncryptedETag along with EncryptedKey and EncryptedKeyNonce.
It also must return the EncryptionParameters.

Since the EncryptionParameters are in the objects DB table, it would be
more efficient for ListSegment to avoid querying that DB table, but take
it from the SatStreamID.

Change-Id: I16c98641c0fe0c98e3303329d0da6ef137ca55cf
2021-03-25 07:54:34 +00:00
Ivan Fraixedes
c5cb4dce4d redis: Rename functions prefixed with New by Open
Rename the functions that are prefixed with 'New' which connect with
Redis by 'Open' to  make clear that they perform network operations.

Change-Id: I1351e89a642e8e2c2586626646315ad0fb2c6242
2021-03-25 06:09:27 +00:00
JT Olio
903b1b5ffb satellite ui: don't call the gateway mt the gateway mt
users don't know what that is yet.

Change-Id: Ifb3b3e6bd51a75326a581cbd784ee364d7c8400b
2021-03-24 18:37:54 +00:00
Kaloyan Raev
1156d8b4ed mod: update drpc to v0.0.20
Change-Id: I0ac6ff9bebe5b16d4fb63b30242c87c899e7d358
2021-03-24 19:06:56 +02:00
Qweder93
866a3e3ad3 storagenode/console: estimated payout tests fix
Added check if days paste <1 and node joined in current month -> days paste = 1

Change-Id: Ice1ccd6c869f629a87da585850b9b4f3729ec65c
2021-03-24 18:05:37 +02:00
Moby von Briesen
27bcb46718 satellite/console: change default project and usage limits
This is one step for implementing the free tier:
* Change the default project limit from 10 to 3
* Move storage and bandwidth project usage limits from the metainfo
package to the console package (otherwise there is a cyclical
dependency, and metainfo doesn't use these values anyway)
* Change the default storage usage limit per project from 500gb to 50gb
* Change the default bandwidth usage limit per project from 500gb to 50gb
* Migrate the database so that old users and projects continue to have
the old defaults (10 projects/500gb usage)

Change-Id: Ice9ee6a738bc6410da18c336c672d3fcd0cab1b9
2021-03-24 14:24:27 +00:00
NickolaiYurchenko
034eb2c214 storagenode: wallet features
WHAT: updated dashboard type to return wallet features from storagenode api

Change-Id: Ifdbdeae8b9ae239eaebe32d0a093557154f777ad
2021-03-24 09:08:36 +00:00
Vitalii Shpital
3e37d1e71c satellite/console: delete api key by name and project id endpoint
WHAT:
new endpoint to be able to delete apiKey/accessGrant by name and project id

WHY:
it will be called to delete special pregenerated access grant which will be used to generate gateway credentials for file browser component or bucket management

Change-Id: I7467ebaab27a7da33efd062536c6da41e6ed4c30
2021-03-23 20:21:07 +00:00
Egon Elbre
f19ef4afe5 satellite/metainfo/metaloop: move loop to a separate package
Change-Id: I94c931a27c1af6062185ec62688624ec02050f11
2021-03-23 15:37:34 +00:00
JT Olio
972d40df6f pkg/server: use drpcmigrate listenmux
Change-Id: I569367f04958ef836c6b02190385d0c936fc8268
2021-03-23 09:18:26 -06:00
Egon Elbre
54c2ace483 mod: update drpc to v0.0.19
Change-Id: Ia2e0e0a371368700c596ce89e24748ea4b5c016a
2021-03-23 17:13:28 +02:00
Michał Niewrzał
50b08a5b00 metainfo-migrator: handle 'omit' conn string
Uplink tests are running storj/storj tests only against postgres. Without this change integration test on uplink will fail as 'omit' is not supported in migrator tests code.

Change-Id: Ic72406f52439e98683d050508fb42aa41632e183
2021-03-23 13:33:22 +00:00
Michał Niewrzał
9d52451464 satellite/metainfo/expireddeletion: add config value to set list limit
We want to control how many expired objects we will query at once.

Change-Id: Iba7c79a6236c6b720b53bb8f078d55514690d8e5
2021-03-23 09:51:56 +00:00
Michał Niewrzał
237782813b Merge remote-tracking branch 'origin/multipart-upload'
Change-Id: If6c5a450b238adab55d1e0dea67d01e5f5768a9f
2021-03-23 09:44:49 +01:00
Moby von Briesen
d2b365bf9f satellite/payments: Disable paywall (free tier)
We are implementing the free tier, which will give all new users 3
projects, 50gb storage, and 50gb bandwidth per project. All users will
receive a recurring coupon to cover this amount of usage.

With the free tier, we no longer need a paywall. Users will not need to
enter a payment method unless they want to increase their project or
usage limits.

Change-Id: If3b026e91858e5f557a2758e366616cecc8f21c7
2021-03-22 17:11:20 -04:00
Cameron Ayer
37602d86d3 satellite/satellitedb: drop columns uptime_reputation_alpha and uptime_reputation_beta
Change-Id: I1e0c5f2f96a920070afb76851da98208654b02cf
2021-03-22 14:33:17 -04:00
Cameron Ayer
864ad70fe2 satellite/overlay/straynodes: set --stray-nodes.enable-dq release default to true
Since we will enable this on all satellites, just set default to true

Change-Id: Ibc86a0afd0b0f57e86bd067abb9cdf06c295a467
2021-03-22 17:25:09 +00:00
Cameron Ayer
2607b16070 satellite/{overlay/straynodes,satellitedb}: rework DQNodesLastSeenBefore to return DQd node IDs and last contact successes
We would like to log Node IDs and last contact successes of nodes DQd
in this manner. We would also like to avoid returning an unbounded list
of items from the db. Therefore we change the query to select a limited
number of nodes that meet the DQ conditions and iterate until 0 rows are
returned. Each column of the query is already indexed.

Change-Id: Iaec2d9b56e7202b7c2028ba21750d40c8dd506ee
2021-03-22 13:01:30 -04:00
Fadila Khadar
a8b5f8f990 satellite/metainfo/metabase: select then delete expired objects
We were deleting expired objects by directly executing a delete query.
With this change, we first select the objects to be deleted and then
delete them (as recommended by cockroachdb for deleting using a non indexed
column).

Change-Id: Ied150fbdc7031a343a74e0b9dab316598188ef66
2021-03-22 17:39:13 +01:00
NickolaiYurchenko
2525e15842 web/multinode: favicon added
Change-Id: I74c0bf8e27e47e5c8da06cd4ffb985a483bd47c2
2021-03-22 12:50:53 +00:00
Ivan Fraixedes
536990dcf6 Jenkins: Run integration test Redis unavailability
Make Jenkins to run the integration tests that verifies that the
satellite can operates when Redis is unavailable.

Change-Id: Idace3ffb84c788f2af2c6e24eec1d63fb40c263a
2021-03-22 09:56:30 +00:00
Ivan Fraixedes
fa8b0a29ea scripts: Do not use Docker for test-sim Redis test
Do not use Docker for running a Redis server in the test-sim that we
check that Satellite can still operate when Redis is unavailable because
we have to run this test in Jenkins and we don't want to empower a
container to be able to connect to the Docker socket of the host machine
for running sibling containers.

Change-Id: I6180e8ed804968c8ccb0783ed334acab38af9a0f
2021-03-22 08:43:53 +00:00
Michał Niewrzał
27ae0d1f15 satellite/metainfo/metabase: add NewRedundancy parameter for UpdateSegmentPieces method
At some point we might try to change original segment RS values and set Pieces according to the new values. This change adds add NewRedundancy parameter for UpdateSegmentPieces method to give ability to do that. As a part of change NewPieces are validated against NewRedundancy.

Change-Id: I8ea531c9060b5cd283d3bf4f6e4c320099dd5576
2021-03-22 08:12:56 +00:00
Moby von Briesen
78e132cb74 satellite/satellitedb: Add coupon_codes table and allow optional link to coupons table
The coupon_codes table will allow for administrators to create new promo
codes associated with coupon information (amount, duration, etc...).

A user will be able to enter a promo code (aka coupon code) in order to
apply a new coupon to their account. The coupon in the coupons table is
linked to the template defined in the coupon_codes table.

Change-Id: I50e49fa92afbc6aa9d01d8a895c069efb59e472b
2021-03-19 20:46:28 +00:00
Fadila Khadar
e613c641d7 satellite/metainfo/metabase: benchmark expired deletion
Change-Id: Icc2e8227b8a96002563ff69f3e9cd8e71369f0d1
2021-03-19 11:30:00 +00:00
Michał Niewrzał
1c02158620 satellite/metainfo/metabase: add RepairedAt to RawSegment
Change-Id: I3f8bbed0f815f8394fc6ab16a914cae276fbab56
2021-03-19 11:06:59 +00:00
Vitalii Shpital
c3ae122aa7 web/satellite: enter passphrase step for objects page
WHAT:
enter passphrase step for users who has already created passphrase

WHY:
to let users proceed to upload step

Change-Id: I084aec5b863981978cf190f99ee95154fbed9aab
2021-03-19 10:59:32 +00:00
Yingrong Zhao
bf4fdc7717 cmd/uplink: add --client.enable-quic flag
Change-Id: I19e81ecb98688fd0a36174fd06c363e9c40b3866
2021-03-19 01:24:16 +00:00
Cameron Ayer
a04495713d satellite/audit: add missing logs for audit failure conditions
Among other conditions, nodes fail audits by returning incorrect
data and by reaching the max reverify count, but we weren't logging
these events. This commit adds the missing logs.

Change-Id: I80749a7e95e8cb97bc8dd7dac1e523e223114b7f
2021-03-18 17:33:11 +00:00
Kaloyan Raev
be8a70309d satellite/metainfo/metabase: rename etag column to encrypted_etag
Change-Id: I79662d9a2cec2da6ea9bba250f0e71978119558d
2021-03-18 18:34:29 +02:00
Ivan Fraixedes
4c1098e571 Redis: Update Redis package to last major version
Update the Redis dependency to use the last major production version.
The last version accepts a context parameter in all the network methods
so it allows us to pass it through them.

Change-Id: I34121b2ec3c2728602115c724933ad24c9e6e4fd
2021-03-18 14:19:49 +00:00
Jeff Wendling
09adfc1205 mod: bump monkit to v3.0.10
Change-Id: Ieed0b4161b158edd7c8e7065f200675cfdfdebf8
2021-03-18 13:17:57 +00:00
Kaloyan Raev
36bc17ef5b satellite/metainfo/metabase: add etag column to segments table
We will use it keep track of ETag for object parts.

Change-Id: Ib610397afd44d71a1b5fe4301940de6f36ba39ae
2021-03-18 10:40:55 +00:00
Vitalii Shpital
cff1052bca web/satellite: save passphrase hash in local storage
WHAT:
store passphrase's hash in browser local storage

WHY:
to make user use the same passphrase

Change-Id: I3d5e601a0257a7c900e8a0ce3cdce953f1174830
2021-03-17 19:50:53 +02:00
Vitalii Shpital
9e20cfb6e0 web/satellite: create passphrase step for objects page
WHAT:
made a common component from AG's generate passphrase component to be reused here.
create passphrase step for objects page

WHY:
used for generating passphrase or setting user's own password for proceeding in upload flow

Change-Id: Iefa106c544a1c5ff04d591d3d400cf06c4c6d853
2021-03-17 15:12:33 +00:00
Vitalii Shpital
b5f83f463b web/satellite: routes and initial components for objects page
WHAT:
routes and components for new upload file feature

WHY:
upload file from browser feature. Step 1

Change-Id: If517fd4513757aeaee64764d1894f1af3e463b55
2021-03-17 15:12:28 +00:00
Michał Niewrzał
fa083a7f05 Merge remote-tracking branch 'origin/main' into multipart-upload
Change-Id: Ib5ce5965b77b81c254d08c27ab30c7eccefbd4c6
2021-03-17 15:37:17 +01:00
Egon Elbre
9491df76e7 satellite/satellitedb: optimize StoragenodeAccounting.SaveRollup
Change-Id: I758049872b4ea1ae22657dd9fcb47c228468b1d0
2021-03-16 18:11:40 +02:00
Vitalii Shpital
6a553ec9c5 web/satellite: change banner for beta satellites with URLs
WHAT:
beta satellite top banner's copy is changed to include support/feedback URLs

WHY:
so users using our beta satellite will be able to report feedback somewhere

Change-Id: Ibc349c8b3354b577275fcf1d2b75bfdd267729d9
2021-03-15 17:12:07 +00:00
Isaac Hess
1de6315b83 metainfo: Stub DownloadObject metainfo endpoint
This is the stub "not implemented" endpoint for the new DownloadObject
RPC.

Change-Id: Ifa724e20d89d839f2b6968d6283bc66a3bab8781
2021-03-15 10:52:57 -06:00
Michał Niewrzał
65957c7525 satellite/metainfo/expireddeletion: temporarily log errors instead failing whole system
At the moment we are trying to optimize deletion queries but its hard to verify deletion performance. Until we are sure that the queries are good we will just log errors instead shutting down whole satellite core.

Change-Id: I5625251d4518c35f0d46d6bf37b2f3ea7950675e
2021-03-15 16:00:20 +01:00
Yingrong Zhao
a36d854d8e bump monkit-jaeger to latest
Change-Id: I8c20769e667afd3e22ebd10e2e045d2e15fad7f7
2021-03-15 14:42:03 +00:00