JakeHillion/storj
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
3,708 Commits 5 Branches 0 Tags 94 MiB
cb781d66c7
Commit Graph

3708 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Egon Elbre
cb781d66c7 satellite/overlay: optimize FindStorageNodes 
Reduce the number of fields returned from the query.

Benchmark results in `satellite/overlay`:

benchstat before.txt after2.txt
name                               old time/op  new time/op  delta
SelectStorageNodes-32              7.85ms ± 1%  6.27ms ± 1%  -20.18%  (p=0.002 n=10+4)
SelectNewStorageNodes-32           8.21ms ± 1%  6.61ms ± 0%  -19.53%  (p=0.002 n=10+4)
SelectStorageNodesExclusion-32     17.2ms ± 1%  15.9ms ± 1%   -7.55%  (p=0.002 n=10+4)
SelectNewStorageNodesExclusion-32  17.8ms ± 2%  16.1ms ± 0%   -9.38%  (p=0.002 n=10+4)
FindStorageNodes-32                48.4ms ± 1%  45.1ms ± 0%   -6.69%  (p=0.002 n=10+4)
FindStorageNodesExclusion-32       79.2ms ± 1%  76.1ms ± 1%   -3.89%  (p=0.002 n=10+4)

Benchmark results from `satellite/overlay` after making them parallel:

benchstat before-parallel.txt after2-parallel.txt
name                               old time/op  new time/op  delta
SelectStorageNodes-32               548µs ± 1%   353µs ± 1%  -35.60%  (p=0.029 n=4+4)
SelectNewStorageNodes-32            562µs ± 0%   368µs ± 0%  -34.51%  (p=0.029 n=4+4)
SelectStorageNodesExclusion-32     1.02ms ± 1%  0.84ms ± 0%  -18.08%  (p=0.029 n=4+4)
SelectNewStorageNodesExclusion-32  1.03ms ± 1%  0.86ms ± 2%  -16.22%  (p=0.029 n=4+4)
FindStorageNodes-32                3.11ms ± 0%  2.79ms ± 1%  -10.27%  (p=0.029 n=4+4)
FindStorageNodesExclusion-32       4.75ms ± 0%  4.43ms ± 1%   -6.56%  (p=0.029 n=4+4)

Change-Id: I1d85e2764eb270f4c2b1998303ccfc1179d65b26
 2020-03-30 18:36:23 +03:00
Stefan Benten
028ce16020
cla: Adding sixcorners to the CLA approved list (#3836) 2020-03-30 18:33:06 +03:00
Egon Elbre
a6540dc3ef satellite/overlay: remove unused KeyLock 
Change-Id: Ie99f97772824ceafb3d97453545fc6e96be2fb6f
 2020-03-30 16:47:48 +03:00
Ivan Fraixedes
de903a652d satellite/accounting: Test no repair traffic in billing 
Add a test for checking that the billing doesn't include traffic due to
audits and repairs.

Change-Id: I1ce1199ec87c4440082c42b100124aeb14200f41
 2020-03-30 10:48:35 +00:00
littleskunk
048ca4558f
satellite/repair: clean up logging (#3833) 
Co-authored-by: Michal Niewrzal <michal@storj.io>
 2020-03-30 11:59:56 +02:00
Yingrong Zhao
831668478a scripts/tests: fix gateway installation 
Change-Id: I1629644f543505c27d4edb8f7bbe97d037cdc8a8
 2020-03-29 19:24:05 +00:00
NickolaiYurchenko
4122be154e web/storagenode: payout page adaptation 
Change-Id: Iaa9fc69567cc6a3daf11829d0434f12098098f5b
 2020-03-29 18:39:30 +00:00
NickolaiYurchenko
5c6c797737 web/storagenode: payout info logic implementation 
Change-Id: Ief1768f3987fdbf687fda5a90aa3c1c2320b17a1
 2020-03-29 20:01:37 +03:00
NickolaiYurchenko
6cac755916 web/storagenode: notifications page adaptations 
Change-Id: I958cde37fdfb6790453f56c9fa1db04b0372cb97
 2020-03-29 16:41:50 +00:00
JT Olio
f28100b73f bump storj.io/private 
Change-Id: I4ddd5c34521602967b89bd18e2a71a6f1e29f436
 2020-03-27 21:57:35 +00:00
Egon Elbre
c970969503 satellite/overlay: add benchmark for node selection 
Change-Id: I15b767a78b662f8276e656b3fb73a15ec59e76c8
 2020-03-27 23:09:29 +02:00
Egon Elbre
480ea1e4b5 satellite/repair/repairer: fix temporary file handling 
Change-Id: Ice1a467510737b3375c018ae37b16431c7dffe9e
 2020-03-27 21:36:23 +02:00
Egon Elbre
e1a443b04a private/testplanet: allow modifying created database 
Instead of providing the database from outside to testplanet create it
inside and then allow wrapping and modifying it. This is more convenient
to use.

Change-Id: I9b8f69e6e0a19ff984b4e2bfe927c9100c77bc6c
 2020-03-27 19:14:48 +00:00
Moby von Briesen
a933bcc99a satellite/repair/repairer/ec.go: add option for downloading pieces onto disk instead of in memory during repair 
Add flag to satellite repairer, "InMemoryRepair" that allows the
satellite to decide whether to download the entire segment being
repaired into memory (this is what the satellite already does), or to
download it into temporary files on disk that will be read from in the
upload phase of repair.

This should help with handling high repair traffic on satellites that
cannot afford to spend 64mb of memory per repair worker.

Updates tests to test repair for both in memory and to disk.

Change-Id: Iddf591e165621497c98533d45bfea3c28b08a194
 2020-03-27 16:41:00 +00:00
Egon Elbre
e8f18a2cfe private/testplanet: expose storagenode and satellite Config 
Change-Id: I80fe7ed8ef7356948879afcc6ecb984c5d1a6b9d
 2020-03-27 17:01:25 +02:00
NickolaiYurchenko
b47381138d web/storagenode: added hash to bundle name 
Change-Id: Iab71e7b8b5cb4d0b9d37bdf0d9688410ee1ed549
 2020-03-27 14:44:53 +02:00
igor gaidaienko
9d3d411ca1 satellite/accounting: Add test billing inline segments 
New test added to be sure that we bill customers for inline segments

Change-Id: I5eb0ee7b475a7b57ecebaad214ece8c5f1cf8c4d
 2020-03-27 10:36:53 +00:00
Michal Niewrzal
75961fe8ce cmd/gateway: remove gateway command from repository and adjust `make 
install-sim`

Change-Id: Ie55989ec0d702f6037843a8b6664f4f4f6347446
 2020-03-27 07:42:42 +00:00
JT Olio
5511827662 satellite/orders: don't log expired order limits 
we still need to come up with a better plan to get storage nodes
to stop doing this, but in the meantime, we know this is happening,
just stop logging it and keep some stats instead.

Change-Id: Icb6bcba275e0e955c54b1a90da2b37219fff2349
 2020-03-26 22:31:10 -06:00
Ethan
df462d7265 satellite/accounting: Add index on bucket_bandwidth_rollups to minimize full table scans 
https://storjlabs.atlassian.net/browse/SM-545

Change-Id: I5599a72a991d70236f17beca027e9bc032777177
 2020-03-26 19:53:50 +00:00
Natalie Villasana
8e0ca0e6f5
satellite/gc: update release default for gc to run separately (#3830) 2020-03-26 14:44:18 -04:00
VitaliiShpital
23da9228b3 satellite/console: email used error handling for registration 
Change-Id: Ifd3f2ce065ebd3c5e538c5c1eeaa76137b243b78
 2020-03-26 17:42:33 +00:00
VitaliiShpital
506582e208 web/satellite: project overview page removed, dashboard page implemented 
Change-Id: Id56425fe38296250e5814a0f02a554cdd73b2330
 2020-03-26 17:15:14 +00:00
Jeff Wendling
97e980cd8a private/dbutil: add database name to configure as a tag 
storagenodes have like 10 or more databases. without this
tag they all get sent as the same value, stomping on each
other.

Change-Id: Ib12019684d6ea8f2a5b83df584056dfa79e3c4b3
 2020-03-26 16:50:15 +00:00
crawter
f879bfcf70 storagenode/console/server/heldamount - endpoint tests added 
Change-Id: I4a324e808fd18e55bad28c1cce2ea90c41dc659e
 2020-03-26 15:50:33 +00:00
Yingrong Zhao
fecd36f6b6 bump uplink version to latest 
Change-Id: I57c7dae60e4b521bf6f5a07b107f5c2ef18fc168
 2020-03-26 09:03:53 +00:00
JT Olio
4d91533a3f
Update README.md (#3828) 2020-03-25 19:53:11 -05:00
Jennifer Johnson
b75cbc8e24 satellite,storagenode: remove references to free bandwidth 
Change-Id: I42a6597544804fa9235e89ec656ebc365eb522e5
 2020-03-25 22:28:34 +00:00
Egon Elbre
c715c75fea pkg/server: add counters for grpc calls 
This will help to determine how many grpc calls are made to the
satellite.

Also remove the grpc funcs that have been added to upstream.

Change-Id: I91878f4fd10f9bfe601c94222c102eaaf4d35963
 2020-03-25 21:38:13 +02:00
Yingrong Zhao
b7b19289d1 bump storj.io/common to latest 
Change-Id: I16e337660ce8e1ef332cc842dbf4cfa067b9b98b
 2020-03-25 09:08:40 -04:00
Kaloyan Raev
0a50d6bdce bump storj.io/uplink to v1.0.1 
Change-Id: I55d44aee1a02fe662408a3c48a3e97d6721994af
 2020-03-25 10:41:34 +02:00
Yingrong Zhao
a731472496 bump storj.io/common to latest and storj.io/drpc to v0.0.11 
Change-Id: I7a6e823b441eeff4621dfdf2d6577be76c9761c8
 2020-03-24 15:17:10 -04:00
Michal Niewrzal
2a482e8bc4 private/version/checker: remove code which was moved to 
`storj/private/debug` package

Change-Id: I44dfecd6ab875fb33851a22cf10b3064da9bfd65
 2020-03-24 17:07:33 +00:00
Egon Elbre
bd9a998abd pkg/server: remove dead code 
Change-Id: I4501ac7de727e2ec6908a7624808b4e596a68a23
 2020-03-24 18:30:22 +02:00
Michal Niewrzal
3a3648c80b scripts: add script for running versions tests locally 
Change-Id: Iaa2a2d085b7edd46f63a0d79b4e731ea72412953
 2020-03-24 15:44:00 +00:00
Michal Niewrzal
fdf40a7526 storj: remove storj/private/version package which was moved to 
`storj/private` repo

Change-Id: I81c3f5b9d5e4fe7bca760999eb045ee9734e5e2e
 2020-03-24 14:31:33 +00:00
Stefan Benten
bdec51658e
cmd/storj-sim: Increase storj-sim max-alpha-usage (#3824) 2020-03-24 14:48:25 +01:00
Michal Niewrzal
aed8dea625 satellite/accounting: Add test not billing after deletion 
New test added to be sure that we don`t bill for the data after deleting it.

Change-Id: Ifb5931ce28f6b0294aeb16311164675a17f11917
 2020-03-24 12:55:49 +00:00
Stefan Benten
173cb1e484
Changing LogLevel to Warn (#3822) 
This is not a process error and can cause false alarm for monitoring systems
 2020-03-24 13:46:28 +01:00
Egon Elbre
a9fb7b7694 storagenode/trust: fix go 1.14 failures 
Change-Id: I6d147f8e5907666ffb3a3e487fa179e377c70841
 2020-03-24 11:04:30 +00:00
Egon Elbre
326c0cebde storage/boltdb: update to etcd/bbolt v1.3.4 
bbolt v1.3.4 has pointer usage fixes.

Change-Id: I5e0fc4782711d01c09ced579f25a4f8fbc8de85c
 2020-03-24 12:33:34 +02:00
Michal Niewrzal
f0aeda3091 storj: remove from storj/pkg packages moved to storj/private repo 
* debug
* traces
* cfgstruct
* process

Package `storj/private/version` will be removed as a separate change.

Change-Id: Iadc40faa782e6225513b28218952f02d9c240a9f
 2020-03-24 09:56:29 +01:00
Isaac Hess
ddf87286c5 cmd/statreceiver: Remove statreceiver code 
Change-Id: Ic572a1623a2008c49d584b769d13e07e4470bce0
 2020-03-23 13:00:13 -06:00
Egon Elbre
6a7571f73e cmd/s3-benchmark: move to storj.io/benchmark 
Change-Id: Idca2b836bdf876ca28eb5cabc9bfae1d576e4a3e
 2020-03-23 19:09:42 +02:00
VitaliiShpital
d7558db5ed web/satellite: succesful registration flow reworked 
Change-Id: I29f7b0fd2286c32ada39bc1e86d58feda624c645
 2020-03-23 16:09:00 +00:00
VitaliiShpital
fe39845a8c web/satellite: refreshing billing history on entering billing page implemented 
Change-Id: I986d6bc5ca1277c78ead7cf6323d59cb73045e25
 2020-03-23 15:26:08 +00:00
VitaliiShpital
f61b77ea39 web/satellite: request limit link added on project details page 
Change-Id: I4e57bb2f9dc610e9103ca21face20a615481bf40
 2020-03-23 15:03:40 +00:00
VitaliiShpital
859fce3af7 web/satellite: browser caching bug fixed 
Change-Id: I5b97a90d28c754cc893852909dcc87290a739f59
 2020-03-23 14:41:20 +00:00
Michal Niewrzal
e258de518c cmd/storj-sim: remove identity dir from gateway configuration 
We are not generating identity for gateway during setup anymore, its
generated on-fly by libuplink. With this change we can remove
`--identity-dir` from gateway.

Change-Id: I63298115d4399564b2f29541b8dc16e3b3acdcf2
 2020-03-23 14:11:51 +00:00
Kaloyan Raev
6c44512c15 satellite/metainfo: increase stream ID expiration to 48h 
Otherwise, it is not possible to upload very large files like 1 TB.

Change-Id: Iadda1ae91174125736684850b906d5bb6d19f0a9
 2020-03-23 15:47:39 +02:00