JakeHillion/storj
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

cmd/uplink: add buffering while reading from stdin

Browse Source 
This change is similar to
https://review.dev.storj.io/c/storj/storj/+/7687 but applied when
uploading from stdin with parallelism > 1.

Currently, the paralellism from stdin scales up to 3 or 4, but not
greater than that. If we buffer the content from stdin more aggressively
the parallelism scales to higher levels and reaches the performance of
reading directly from a file.

Change-Id: I1f447686a88074882709992ee6d52dd262e220fb
This commit is contained in:
Kaloyan Raev 2022-12-23 11:57:28 +02:00 committed by Storj Robot
parent b683f83994
commit 56896353b6
1 changed files with 2 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
cmd/uplink/cmd_cp.go
View File

@ -390,8 +390,8 @@ func (c *cmdCp) parallelCopy(
} }
var readBufs *ulfs.BytesPool var readBufs *ulfs.BytesPool
if p > 1 && dest.Std() { if p > 1 && (source.Std() || dest.Std()) {
// Create the read buffer pool only for downloads to stdout with parallelism > 1. // Create the read buffer pool only for uploads from stdin and downloads to stdout with parallelism > 1.
readBufs = ulfs.NewBytesPool(int(chunkSize)) readBufs = ulfs.NewBytesPool(int(chunkSize))
} }