Commit Graph

5376 Commits

Author SHA1 Message Date
Michal Niewrzal
9bdc8ebf75 satellite/metainfo: adjust GetObject to handle
EncryptedMetadataEncryptedKey

Change-Id: If159fdcc9f37e9449a7d8459a277366d6d6b5802
2020-11-17 16:38:10 +00:00
Egon Elbre
72fed3b3fb satellite/metainfo/metabase: add slow non-recursive listing
Change-Id: Ide60640400caea6474465af5224010fde60826c7
2020-11-17 16:47:29 +02:00
Ivan Fraixedes
fa95c6bbb9
storagenode/orders/ordersfile: Fix error message wrong var
Fix the error message reported by a wrong order size due to passing the
wrong variable to the interpolation pattern.

Change-Id: Ic0059615c60cfa33a26d4aeb0ebda5e586f0df05
2020-11-17 15:22:27 +01:00
Egon Elbre
5e7d47b9cd satellite/metainfo: code cleaning
Rename iterator tests to iterator_test.go.
Add Delimiter to metabase package.

Change-Id: Icea6542b54176d716daba4afad901713f1dcac90
2020-11-17 15:50:40 +02:00
Egon Elbre
2e426ef0a8 satellite/metainfo/metabase: iterator compatibility
Old iterator returns object keys without prefixes, this helps to reduce
the bandwidth from the database. The endpoint also doesn't send the
prefixes.

Change-Id: I77d85dae671ee3a16abe75db14e19674e80abaf4
2020-11-17 15:50:39 +02:00
Kaloyan Raev
402a581b86 ci: fail build if metabase tests fail
Change-Id: Ic263ad2dcf777d1fb7c03bbddc9edd09a2165c83
2020-11-17 13:30:15 +00:00
Fadila Khadar
c9bbd83f96 satellite/metainfo/metabase: align object status to those already in use
Change-Id: I33b2ce7282838e1172b7cb92d4e76169c96fddcb
2020-11-17 14:07:42 +01:00
NickolaiYurchenko
ac94333422 web/storagenode: notifications components unit tests
Change-Id: I2b0249eff73572b3bf401c5f920440f095bc3978
2020-11-17 12:00:00 +00:00
Ivan Fraixedes
9740da6508 storagenode/orders: Don't panic if size is over MaxInt32
`make` built function to build a new slice with a negative
length panics.
`make` length parameter is of `int` type.

These changes avoid that `make` panics on 32 bits architecture due to
the fact that `int` type is a `int32` an uint32 value can be over the
maximum `int32`, and when that happens the length parameter value
becomes negative and makes `make` to panic.

Change-Id: Ife9ab5993916d6dcf5584b37c208272269cb2b45
2020-11-17 10:35:21 +00:00
Michal Niewrzal
b7df41a0a1 satellite/metainfo/metabase: add EncryptedMetainfoEncryptedKey parameter
to metabase

* EncryptedMetainfoEncryptedKey added to CommitSegment and
UpdateMetadata request
* EncryptedMetainfoEncryptedKey returned with GetObject response and all
delete responses
* EncryptedMetainfoEncryptedKey returned with object iterator results

Change-Id: I917541ab5f3e1863bc8f238d17a15fbf72a23025
2020-11-17 10:19:12 +00:00
Qweder93
c409194d43 storagenode/payouts: estimation payout heldamount rounding removed
Change-Id: I9fdc7cda15de0df8875436b0b376f0e6479d3aeb
2020-11-17 10:06:11 +00:00
Moby von Briesen
0ec685b173 satellite/{satellitedb, repair/{queue, checker}}: Use new column "segmentHealth" instead of "numHealthy" in injured segments queue
We plan to add support for a new Reed-Solomon scheme soon, but our
repair queue orders segments by least number of healthy pieces first.
With a second RS scheme, fewer healthy pieces will not necessarily
correlate to lower health.

This change just adds the new column in a migration. A separate change
will add the new health function.

Right now, since we only support one RS scheme, behavior will not
change. Number of healthy pieces is being inserted as "segment health"
until the new health function is merged.

Segment health is calculated with a new priority function created in
commit 3e5640359. In order to use the function, a new config value is
added, called NodeFailureRate, representing the approximate probability
of any individual node going down in the duration of one checker run.

Change-Id: I51c4202203faf52528d923befbe886dbf86d02f2
2020-11-16 21:18:09 +00:00
Egon Elbre
afc9545ff1 cmd/storj-sim: add "tool wait-for <address>"
For coordinating with other processes it can be useful to wait until
another process is accepting requests on an address.

Change-Id: Id623ed815149f14f9f0344e2f396ab70fc4dec6a
2020-11-16 20:38:56 +00:00
littleskunk
9ab824d3e6
jenkins/rollingupgrade: sleep 5 seconds between old api startup and database migration (#3971) 2020-11-16 21:25:11 +01:00
Cameron Ayer
48d8114b3f satellite/contact: treat pingback failure as error
If the satellite fails to pingback the storage node during CheckIn
an error message is returned to the node in the response, but the actual
error value returned is nil. We are only checking the error. This means
the node has no feedback about the failure, and the node also does not
attempt to retry the connection.

Change-Id: Iaed00e422ba91af573e72255cc6671ea97928eae
2020-11-16 18:26:37 +00:00
crawter
f311722854 multinode/db: nodes repository tests added
Change-Id: Ia5172f249c18540683f66ef244c2c6d39aa3da0a
2020-11-16 20:03:10 +02:00
VitaliiShpital
51fa52e636 web/satellite: access grant type, api, store module, mock
Change-Id: I4c27ca8ac0df2d348e945d3266a56bd26f7d444a
2020-11-16 16:10:58 +00:00
VitaliiShpital
51a712f9e8 satellite/console: get all bucket names endpoint and service method
WHAT:
new endpoint for fetching all bucket names

WHY:
used by new access grant flow

Change-Id: I356a3381359665fd2726120139b34b1e611fe3c4
2020-11-16 17:51:40 +02:00
Egon Elbre
402cfcb7c6 satellite/metainfo/metabase: add prefix to iterator
Change-Id: I2dd7b7917aed9def0be3f131ca60ab12f2077d83
2020-11-16 17:38:59 +02:00
Moby von Briesen
db480e6e1b storagenode/orders: Improve performance of handling corrupt orders.
This change fixes two things which can make reading from a corrupted
orders file inefficient.
* When a corrupted order is detected, but the underlying error is an
UnexpectedEOF (as opposed to a pb.Unmarshal error, for instance), there
is no point in attempting to read from the file another time to find an
additional uncorrupted order - we will continue to get UnexpectedEOF
errors until we seek to the very end of the file and get a normal EOF.
Instead, when UnexpectedEOF occurs, log and send metrics as with other
types of corruption, but do not attempt to read again.
* When a corrupted order is detected, instead of seeking forward only
one byte for the next attempt, seek forward by the size of entryHeader.
This cuts down on the number of iterations needed to find an uncorrupted
order after detecting a corrupted one.

Change-Id: Ie1a613127e29d29318584ec7f60e8f7554f73487
2020-11-16 14:08:36 +00:00
Egon Elbre
7aba265db0 satellite/metainfo/metabase: add status option to iteration
Change-Id: Ie9d1a3704d8d08e3421ba2244a1f3be57d15ed8f
2020-11-16 15:41:20 +02:00
Kaloyan Raev
a015f41927 satellite/metainfo: ListObjects to use Metabase API
Change-Id: If75d04b9a44f08515be717a85f397b66f8a5c56f
2020-11-16 13:27:17 +02:00
Egon Elbre
1726b39ed2 scripts: remove thrift mod replace
Currently our code is only using github version of the code, so there
shouldn't be need for the exception.

Change-Id: I0c6e8a8465ab7b525d4b5d1b29e4e5298384286d
2020-11-16 12:00:09 +02:00
Michal Niewrzal
7c384c8293 Merge 'master' branch
Change-Id: I1eefd5a56449e577820977d61fa4a22bdd4fc230
2020-11-16 10:02:54 +01:00
Michal Niewrzal
cd7d911b26 satellite/metainfo: implement ListSegments with metabase
Change-Id: I78fb2400fed0bb01c5e05bd5bbcddef5e4a3bdf0
2020-11-16 08:33:35 +00:00
Ivan Fraixedes
c2ba5a9905 Makefile: Update Go version security patch
Update Go version to use the last path release 1.15.5 due to a security
fix.
https://groups.google.com/u/1/g/golang-nuts/c/c-ssaaS7RMI/m/5iS6JRtOAwAJ

Change-Id: I748df29e2309408ba1567ebf72652803ee4ec5bc
2020-11-15 00:36:54 +01:00
Jessica Grebenschikov
f558cc825e satellite/orders: add storagenode_bw_phase2 table and dont delete tallies for longer
It turns out we need to make 2 more changes in order for the new order submission phase 3 to get deployed.

This PR makes 2 changes:
1) when the rollup service deletes tallies, we now keep tallies around until orders expire (vs 1 day like before).
2) the reported rollup chore will now write the storagenode_bandwidth_rollups to a new table _phase2 as an intermediary step so it doesn't conflict with phase 3 order settlement.

These changes need to be deployed for 2 days before we can turn on phase 3 of the new orders settlement workflow.

Change-Id: Iafbff577ba7d55f8f17b7db857311b2ce799de60
2020-11-13 17:15:24 +00:00
Egon Elbre
c0b5e7ce3e ci: ensure cockroach doesn't pollute repo
Cockroach 20.2 started automatically profiling, this is a workaround to
disable it and ensure it doesn't create any folders and files in the
repository.

Change-Id: Ib65de01ea1fc619160d710c01602ced3a3a3492e
2020-11-13 16:07:01 +00:00
Malcolm Bouzi
2e6ffd9af6
web/satellite:access grant empty state (#3970) 2020-11-13 18:06:34 +02:00
Kaloyan Raev
ff32268310 satellite/metainfo/metabase: DeleteExpiredObjects
Change-Id: I16293ce1a33f714aa005022daf3d40670cc66b04
2020-11-13 15:31:31 +02:00
Fadila Khadar
a749ac9f47 satellite/metainfo/metabase: iterateObjectsAllVersions
Iterate through all objects in a bucket
recursive, no prefix support

Change-Id: Ieaad9fbd0b5c1593554db9d543f4ee04851bac28
2020-11-13 13:50:08 +01:00
Yaroslav Vorobiov
1b4bfbb9d2 multinode/console: nodes addition and removal
Change-Id: I60c685953a8d0e24f78b1414c34a28d4b87863b0
2020-11-12 20:26:08 +02:00
crawter
e6967720cd cmd/multinode: create schema command added, run command bug fixed
Change-Id: Ief76fc4a878441e5f112bd79810c66e8d85d7acb
2020-11-12 18:00:18 +00:00
NickolaiYurchenko
f8d3a977fa web/storagenode: PayoutPeriodCalendar.vue unit tests
Change-Id: I6a41611e28993577eb72426b941cf272ae8da46f
2020-11-12 18:50:16 +02:00
Jessica Grebenschikov
226e13e616 satellite/cosole: add tests for wasm access code
Change-Id: I78f71b2f0bef03b6e87cd7d79ccaef5f45393b55
2020-11-12 08:03:36 -08:00
NickolaiYurchenko
259f4ebcf1 web/storagenode: EstimationArea.vue unit tests
Change-Id: I5b7606f3deb0b3b8cccf6ec1026d06f3558fd808
2020-11-12 13:21:10 +00:00
Egon Elbre
8182d8a726 satellite/metainfo/metabase: use txutil.WithTx in delete
Change-Id: I75965c4dcf57479ace2367f5d3069d785628e86a
2020-11-12 14:38:36 +02:00
Egon Elbre
bc460cd62d satellite/metainfo/metabase: use txutil.WithTx in commit
Change-Id: I84347291179c7042e22bda703a1c9bd80a1513a0
2020-11-12 14:38:33 +02:00
Michal Niewrzal
9d25b3a7d3 satellite/metainfo/metabase: add ListSegments request
Change-Id: Ie777a0f791685beab3973e76b10a9ce76398e7d7
2020-11-12 11:46:17 +01:00
Yingrong Zhao
54c5d564a1 scripts/tests/testversions: fix older uplink setup
This PR does follwing changes:
    1. Change oldest uplink version in the test to v0.35.3
        When the test is first created, we decided to support uplink
        version starting from v0.17.1, however with many API changes,
        older uplinks are not usable with latest version of the network
        anymore. One of the reasons being older uplinks uses deprecated
        endpoint. Therefore, we will change the oldest uplink version to
        the one that's using only new endpoints.
    2. Disable tls certificate verification in uplink
    3. Use storj-sim version control server instead of production one
    4. Skip uplink version v1.3.x due to bug in that release

Change-Id: I926a6bb9829cb7181ee752437cdcb67e59197fe0
2020-11-11 17:00:01 -05:00
paul cannon
3e56403599 satellite/repair: add a repair health function
This will be used to rank segments in need of repair for attention by
the repair workers.

Change-Id: I5b70650cec933696b4c6d73bb7efb97e3efdf24a
2020-11-11 18:48:51 +00:00
Michal Niewrzal
fd707392e7 satellite/metainfo: set RedundancyScheme for begin segment
response

Multipart upload needs it for uploading parts.

Change-Id: I34886a9dabd17ecb7d55d4c6e19914a1c821f638
2020-11-11 17:25:52 +00:00
Jeff Wendling
31533ed1a1 satellite/console/wasm: remove storj.io/uplink deependency
Change-Id: Iee95389e4ba24618e31aff7be44d05377b2e2419
2020-11-11 16:51:14 +00:00
Malcolm Bouzi
592d0bd6bc
web/satellite: access grant routing (#3966) 2020-11-11 18:41:46 +02:00
Cameron Ayer
5a337c48ec {cmd,private,storagenode}: create storage dir verification during setup
Previously, we created a new file to use for directory verification
every time the storage node starts. This is not helpful if the storage node
points to the wrong directory when restarting. Now we will only create the file
on setup. Now the file should be created only once and will be verified at
runtime.

Change-Id: Id529f681469138d368e5ea3c63159befe62b1a5b
2020-11-11 11:01:36 -05:00
crawter
4ce00c7caa cmd/multinode: run and setup commands added
Change-Id: If7b39c392a9a5617315cefaeafffddab845cf071
2020-11-11 14:48:16 +00:00
VitaliiShpital
5e0106f1fe web/satellite: web worker for wasm
WHAT:
web worker for compiling and instantiation of web assembly module

WHY:
Currently webassembly requires unsafe-eval, however we don't want to
add it to main site due to CSP. The workaround for this is to instantiate
wasm code inside a web worker.

Change-Id: I0c3c9cafa3a0c344761cf6dd86bf96248f1103ca
2020-11-11 16:24:06 +02:00
Kaloyan Raev
9dcd0936d7 satellite/metainfo: MakeInlineSegment to use metabase
Change-Id: Ib32c1d9e91e3f13b4e44abefb2b5d0352b64bd79
2020-11-11 15:24:21 +02:00
Cameron Ayer
07acf0e574 cmd/storagenode: add docker env variable to toggle running setup
Previously, we ran setup if no config file was found in the expected dir.
However, there may be situations where a previously set up node's files
may be unreachable. In this case, we would prefer to exit with an error
rather than assume this node needs to be initialized.

The solution here is to add a new env variable to call the setup command.
If SETUP == true, the node will setup, but not run. If SETUP != true,
the node will run and not setup.

If a previously set up node runs with SETUP, it will return an error.
If a node runs without an initial SETUP, it will return an error.

Change-Id: Id2c796ec3d43f2add5e5f34fb777a563eae59f2f
2020-11-11 13:11:19 +00:00
Cameron Ayer
da9f1f0611 satellite/repair: add monkit counter for segments below minimum required
The current monkit reporting for "remote_segments_lost" is not usable for
triggering alerts, as it has reported no data. To allow alerting, two new
metrics "checker_segments_below_min_req" and "repairer_segments_below_min_req"
will increment by zero on each segment unless it is below the minimum
required piece count. The two metrics report what is found by the checker
and the repairer respectively.

Change-Id: I98a68bb189eaf68a833d25cf5db9e68df535b9d7
2020-11-11 12:48:23 +00:00