JakeHillion/storj
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
4,082 Commits 5 Branches 0 Tags 94 MiB
c272872d5d
Commit Graph

4082 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Egon Elbre
1024bf9ce1 all: simplify uuid usage 
Instead of uuid.Parse, use uuid.FromString.
This removes a bunch of pointer management logic.

Change-Id: Id25bd174eb43c71d00b450158a198abafd8958f2
 2020-04-02 13:45:19 +00:00
Michal Niewrzal
fe2340285f storagenode/console/consoleserver: fix TestConsole tests 
https://storjlabs.atlassian.net/browse/SG-145

Change-Id: Idee6050cea5fd53ec631a7c64fb18f1ad7f16af0
 2020-04-02 13:17:04 +00:00
Michal Niewrzal
c178a08cb8 satellite/metainfo: add max segment size and max inline size to 
BeginObject response

We want to control inline segment size and segment size on satellite
side. We need to return such information to uplink like with redundancy
scheme.

Change-Id: If04b0a45a2757a01c0cc046432c115f475e9323c
 2020-04-02 12:41:28 +00:00
Michal Niewrzal
4a79b609e9 satellite/metainfo: fix panic when we batch BeginObjectDelete without 
all permissions

Without read and list permissions BeginObjectDelete won't return error
if occurs. This was breaking Batch processing because there was
assumption that without error response will be always not nil.

https://storjlabs.atlassian.net/browse/SM-590

Change-Id: I0fc9539e429110a660eb28725b266d5e4771d198
 2020-04-02 12:20:19 +00:00
Egon Elbre
8f73fb7a32 all: simplify uuid usage 
uuid.UUID implements driver.Value so it can be directly used as a
scannable result.

Replace uses of dbutil.BytesToUUID with uuid.FromBytes.

Change-Id: I51a670185ceb3cc2199d5aa2b76bc3fc191ca8fe
 2020-04-02 05:48:58 +00:00
Yingrong Zhao
f663906357 storagenode/contact: call return value from mon.Task() on function finish 
Change-Id: I5e6462acb99ac1d28b5d2518d5db8a4afe593d11
 2020-04-01 23:26:14 +00:00
Egon Elbre
90319dbec1 scripts: fix test-sim-backwards 
Change-Id: I9d6644d4b493f5f6f60a960cacfaac2b5b828a5f
 2020-04-02 00:13:42 +02:00
Jeff Wendling
ffe7a3c211 satellite/compensation: make surge percent an int64 for strictcsv 
Change-Id: I1783bf73ee68ca9beb8a03f5928873fab0bbe95d
 2020-04-01 14:06:33 +00:00
Egon Elbre
644df8dcdc private/version: minimal fix for tag-release.sh 
Previous split to a storj.io/private repository broke tag-release.sh
script. This is the minimal temporary fix to make things work.

This links the build information to specified variables and sets them
inline. This approach, of course, is very fragile.

Change-Id: I73db2305e6c304146e5a14b13f1d917881a7455c
 2020-04-01 13:46:45 +00:00
NickolaiYurchenko
9a72b5fde9 web/storagenode: byte*h to byte*month conversion 
Change-Id: Iec95e0d6d946426df38e5725af3a889c6d81b23a
 2020-04-01 13:03:16 +00:00
Michal Niewrzal
d444fbadea scripts: cleanup rolling upgrade test 
* add script for easy rolling upgrade test local execution
* remove unneeded binaries building for rolling upgrade and versions
tests
* unify build process for Jenkins and local execution for rolling
upgrade and versions tests

Change-Id: Ic11211b83f3f447494bbd5827d2af77ea4b20dfe
 2020-04-01 12:30:08 +00:00
Kaloyan Raev
473cd4695f bump storj.io/uplink to v1.0.3 
Change-Id: I9466214260d139eed155c2d8fd725572cd5f0425
 2020-04-01 11:12:59 +00:00
Egon Elbre
a416b03941 satellite/accounting: fix TestProjectBandwidthTotal 
Test was inserting for past 4 days, however the test was summing up for
the current month.

Change-Id: I509afdc6a76b314a6bb90652ab70cd2c2bab1288
 2020-04-01 11:50:18 +03:00
Jeff Wendling
9bd0bd0c24 private/currency: add strictcsv support to microunit 
Change-Id: Iad2f6a07f189f2faa1d13bdb82dfa320921f6938
 2020-03-31 14:57:04 -06:00
Egon Elbre
0a69da4ff1 all: switch to storj.io/common/uuid 
Change-Id: I178a0a8dac691e57bce317b91411292fb3c40c9f
 2020-03-31 19:16:41 +03:00
Egon Elbre
1d79228ed0 satellite/metainfo: support uplink useragent 
Adds support for parsing user agent and specifying uplink version and
it's library dependencies.

Change-Id: Ibaddde4deb93e153ac05c91b676c5b5f1ae1aa37
 2020-03-31 15:11:31 +00:00
Qweder93
1fb37915c3 heldamount date picking hot fix 
Change-Id: I0cd0787e20e8af35a44bf05fa5c177434f563572
 2020-03-31 15:26:26 +03:00
Qweder93
dc32f1da55 storagenode/cache/heldamount added, errNoRows ignored 
Change-Id: If6b675e622d6c1324c0893c43cca93dc5323cd78
 2020-03-31 11:35:58 +00:00
Michal Niewrzal
0374e30678 scripts: fix storj-sim installation function for rolling upgrade tests 
Change-Id: Ifb749e0a41c57b90cc03293c2588b2ceb6a0a15a
 2020-03-31 09:11:00 +00:00
Jeff Wendling
e2ff2ce672 satellite: compensation package and commands 
Change-Id: I7fd6399837e45ff48e5f3d47a95192a01d58e125
 2020-03-30 14:08:14 -06:00
littleskunk
23e5a0471f
satellite/audit: clean up logging (#3832) 
Co-authored-by: Ivan Fraixedes <ivan@fraixed.es>
 2020-03-30 12:09:50 -06:00
Jess G
a73416e208
revert to use the config for storagesync (#3837) 
Change-Id: I11920b54736e7e548f5638a9ba5f6380a94d99c4
 2020-03-30 13:43:38 -04:00
Jennifer Johnson
d77f3b8786 satellitedb/migrate: set vetted_at backfill to now.day 
Change-Id: Ib2b12be43dbd3f3705b1891bc703ae15abb75e09
 2020-03-30 16:50:23 +00:00
Egon Elbre
439aba922a satellite/overlay: reduce overhead of GetNodes 
Instead of filtering on the client side it's better to filter on the
database side.

Change-Id: I845fbbe5ed28c2ffdb0b8a3f789b59c094fd1069
 2020-03-30 18:36:23 +03:00
Egon Elbre
cb781d66c7 satellite/overlay: optimize FindStorageNodes 
Reduce the number of fields returned from the query.

Benchmark results in `satellite/overlay`:

benchstat before.txt after2.txt
name                               old time/op  new time/op  delta
SelectStorageNodes-32              7.85ms ± 1%  6.27ms ± 1%  -20.18%  (p=0.002 n=10+4)
SelectNewStorageNodes-32           8.21ms ± 1%  6.61ms ± 0%  -19.53%  (p=0.002 n=10+4)
SelectStorageNodesExclusion-32     17.2ms ± 1%  15.9ms ± 1%   -7.55%  (p=0.002 n=10+4)
SelectNewStorageNodesExclusion-32  17.8ms ± 2%  16.1ms ± 0%   -9.38%  (p=0.002 n=10+4)
FindStorageNodes-32                48.4ms ± 1%  45.1ms ± 0%   -6.69%  (p=0.002 n=10+4)
FindStorageNodesExclusion-32       79.2ms ± 1%  76.1ms ± 1%   -3.89%  (p=0.002 n=10+4)

Benchmark results from `satellite/overlay` after making them parallel:

benchstat before-parallel.txt after2-parallel.txt
name                               old time/op  new time/op  delta
SelectStorageNodes-32               548µs ± 1%   353µs ± 1%  -35.60%  (p=0.029 n=4+4)
SelectNewStorageNodes-32            562µs ± 0%   368µs ± 0%  -34.51%  (p=0.029 n=4+4)
SelectStorageNodesExclusion-32     1.02ms ± 1%  0.84ms ± 0%  -18.08%  (p=0.029 n=4+4)
SelectNewStorageNodesExclusion-32  1.03ms ± 1%  0.86ms ± 2%  -16.22%  (p=0.029 n=4+4)
FindStorageNodes-32                3.11ms ± 0%  2.79ms ± 1%  -10.27%  (p=0.029 n=4+4)
FindStorageNodesExclusion-32       4.75ms ± 0%  4.43ms ± 1%   -6.56%  (p=0.029 n=4+4)

Change-Id: I1d85e2764eb270f4c2b1998303ccfc1179d65b26
 2020-03-30 18:36:23 +03:00
Stefan Benten
028ce16020
cla: Adding sixcorners to the CLA approved list (#3836) 2020-03-30 18:33:06 +03:00
Egon Elbre
a6540dc3ef satellite/overlay: remove unused KeyLock 
Change-Id: Ie99f97772824ceafb3d97453545fc6e96be2fb6f
 2020-03-30 16:47:48 +03:00
Ivan Fraixedes
de903a652d satellite/accounting: Test no repair traffic in billing 
Add a test for checking that the billing doesn't include traffic due to
audits and repairs.

Change-Id: I1ce1199ec87c4440082c42b100124aeb14200f41
 2020-03-30 10:48:35 +00:00
littleskunk
048ca4558f
satellite/repair: clean up logging (#3833) 
Co-authored-by: Michal Niewrzal <michal@storj.io>
 2020-03-30 11:59:56 +02:00
Yingrong Zhao
831668478a scripts/tests: fix gateway installation 
Change-Id: I1629644f543505c27d4edb8f7bbe97d037cdc8a8
 2020-03-29 19:24:05 +00:00
NickolaiYurchenko
4122be154e web/storagenode: payout page adaptation 
Change-Id: Iaa9fc69567cc6a3daf11829d0434f12098098f5b
 2020-03-29 18:39:30 +00:00
NickolaiYurchenko
5c6c797737 web/storagenode: payout info logic implementation 
Change-Id: Ief1768f3987fdbf687fda5a90aa3c1c2320b17a1
 2020-03-29 20:01:37 +03:00
NickolaiYurchenko
6cac755916 web/storagenode: notifications page adaptations 
Change-Id: I958cde37fdfb6790453f56c9fa1db04b0372cb97
 2020-03-29 16:41:50 +00:00
JT Olio
f28100b73f bump storj.io/private 
Change-Id: I4ddd5c34521602967b89bd18e2a71a6f1e29f436
 2020-03-27 21:57:35 +00:00
Egon Elbre
c970969503 satellite/overlay: add benchmark for node selection 
Change-Id: I15b767a78b662f8276e656b3fb73a15ec59e76c8
 2020-03-27 23:09:29 +02:00
Egon Elbre
480ea1e4b5 satellite/repair/repairer: fix temporary file handling 
Change-Id: Ice1a467510737b3375c018ae37b16431c7dffe9e
 2020-03-27 21:36:23 +02:00
Egon Elbre
e1a443b04a private/testplanet: allow modifying created database 
Instead of providing the database from outside to testplanet create it
inside and then allow wrapping and modifying it. This is more convenient
to use.

Change-Id: I9b8f69e6e0a19ff984b4e2bfe927c9100c77bc6c
 2020-03-27 19:14:48 +00:00
Moby von Briesen
a933bcc99a satellite/repair/repairer/ec.go: add option for downloading pieces onto disk instead of in memory during repair 
Add flag to satellite repairer, "InMemoryRepair" that allows the
satellite to decide whether to download the entire segment being
repaired into memory (this is what the satellite already does), or to
download it into temporary files on disk that will be read from in the
upload phase of repair.

This should help with handling high repair traffic on satellites that
cannot afford to spend 64mb of memory per repair worker.

Updates tests to test repair for both in memory and to disk.

Change-Id: Iddf591e165621497c98533d45bfea3c28b08a194
 2020-03-27 16:41:00 +00:00
Egon Elbre
e8f18a2cfe private/testplanet: expose storagenode and satellite Config 
Change-Id: I80fe7ed8ef7356948879afcc6ecb984c5d1a6b9d
 2020-03-27 17:01:25 +02:00
NickolaiYurchenko
b47381138d web/storagenode: added hash to bundle name 
Change-Id: Iab71e7b8b5cb4d0b9d37bdf0d9688410ee1ed549
 2020-03-27 14:44:53 +02:00
igor gaidaienko
9d3d411ca1 satellite/accounting: Add test billing inline segments 
New test added to be sure that we bill customers for inline segments

Change-Id: I5eb0ee7b475a7b57ecebaad214ece8c5f1cf8c4d
 2020-03-27 10:36:53 +00:00
Michal Niewrzal
75961fe8ce cmd/gateway: remove gateway command from repository and adjust `make 
install-sim`

Change-Id: Ie55989ec0d702f6037843a8b6664f4f4f6347446
 2020-03-27 07:42:42 +00:00
JT Olio
5511827662 satellite/orders: don't log expired order limits 
we still need to come up with a better plan to get storage nodes
to stop doing this, but in the meantime, we know this is happening,
just stop logging it and keep some stats instead.

Change-Id: Icb6bcba275e0e955c54b1a90da2b37219fff2349
 2020-03-26 22:31:10 -06:00
Ethan
df462d7265 satellite/accounting: Add index on bucket_bandwidth_rollups to minimize full table scans 
https://storjlabs.atlassian.net/browse/SM-545

Change-Id: I5599a72a991d70236f17beca027e9bc032777177
 2020-03-26 19:53:50 +00:00
Natalie Villasana
8e0ca0e6f5
satellite/gc: update release default for gc to run separately (#3830) 2020-03-26 14:44:18 -04:00
VitaliiShpital
23da9228b3 satellite/console: email used error handling for registration 
Change-Id: Ifd3f2ce065ebd3c5e538c5c1eeaa76137b243b78
 2020-03-26 17:42:33 +00:00
VitaliiShpital
506582e208 web/satellite: project overview page removed, dashboard page implemented 
Change-Id: Id56425fe38296250e5814a0f02a554cdd73b2330
 2020-03-26 17:15:14 +00:00
Jeff Wendling
97e980cd8a private/dbutil: add database name to configure as a tag 
storagenodes have like 10 or more databases. without this
tag they all get sent as the same value, stomping on each
other.

Change-Id: Ib12019684d6ea8f2a5b83df584056dfa79e3c4b3
 2020-03-26 16:50:15 +00:00
crawter
f879bfcf70 storagenode/console/server/heldamount - endpoint tests added 
Change-Id: I4a324e808fd18e55bad28c1cce2ea90c41dc659e
 2020-03-26 15:50:33 +00:00
Yingrong Zhao
fecd36f6b6 bump uplink version to latest 
Change-Id: I57c7dae60e4b521bf6f5a07b107f5c2ef18fc168
 2020-03-26 09:03:53 +00:00