Commit Graph

7983 Commits

Author SHA1 Message Date
Egon Elbre
a7d34bea4b
Add missing copyright headers (#352) 2018-09-13 16:30:28 +03:00
Alexander Leitner
4397b90349
Only create index if not exists (#347) 2018-09-12 12:23:21 -04:00
Matt Robinson
0b947355da A bunch of small fixes to make OPsy things better (#346) 2018-09-12 11:02:53 -04:00
Kaloyan Raev
8f4b29abd0
Lazy segment ranger (#343) 2018-09-12 09:06:44 -04:00
Michal Niewrzal
e90c2140b8 Show file size for uplink ls command (#344)
* Show file size for uplink ls command

* align size to right

* Fix stream store list method to handle meta.Size flag
2018-09-12 09:03:31 -04:00
Egon Elbre
10d9420f24
command-line benchmark and visualization (#342) 2018-09-11 18:35:46 +03:00
Egon Elbre
4486e265bb
Cleanup after tests (#341) 2018-09-11 16:57:12 +03:00
Egon Elbre
e7e2d4d7c9
use golangci-lint and add missing error checks (#340) 2018-09-11 16:13:25 +03:00
Alexander Leitner
8f07c4d746
Keep track of used space (#338)
* Keep track of space at all times

* Create index on expires

* Updated mock psclient
2018-09-11 08:40:45 -04:00
Egon Elbre
6ee6f0fdf5
Remove duplicated mock store implementations (#337) 2018-09-11 10:27:12 +03:00
Jennifer Li Johnson
d0f87f0de1
Overlay Bulk Lookup (#279)
* implements bulk node lookup from overlay cache
2018-09-11 00:52:14 -04:00
Egon Elbre
7a906285e3 Small cleanup at top-level (#336)
* cleanup some top-level packages

* remove unused index.html

* update indirect tags
2018-09-10 07:23:08 -07:00
Egon Elbre
00ac266bda Remove zap dependency from boltdb (#334)
* Remove zap dependency from boltdb

* Avoid pulling in testing in main binary
2018-09-10 05:52:53 -04:00
Alexander Leitner
2eb660d4b7 Bandwidth allocation pipeline data (#276)
* Moving retrieve into multiple goroutines

* Make sure we pass nil errors into err channel

* restore tests

* incorporate locks in retrieve.go

* deserialize data only if we have something to deserealize when receiving bandwidth allocation in server store

* Adding logic for retrieve to be more efficient

* Add channel?

* hmm

* implement Throttle concurrency primitive

* using throttle

* Remove unused variables

* Egon comments addressed

* Get ba total correct

* Consume without waiting

* incrementally increase signing size

* Get downloads working with throttle

* Removed logging

* Make sure we handle errors properly

* Fix tests
>
>
Co-authored-by: Kaloyan <kaloyan@storj.io>

* Can't Fatalf in goroutine

* Add missing returns to tests

* add capacity to channel, smarter allocations

* rename things and don't use size as limit

* replace things with sync2.Throttle

* fix compilation errors

* add note about security

* fix ordering

* Max length is actually 64 bytes for piece ID

* Max length is actually 64 bytes for piece ID

* fix limit

* error comes from pending allocs, so no need to relog

* Optimize throughput

* TODO

* Deleted allocation manager

* Return when someone sends a smaller bandwidth allocation than the previous message

* review comments
2018-09-10 03:18:41 -06:00
JT Olio
01507929a4 thepaul signed the cla (#332) 2018-09-10 10:54:11 +03:00
Kaloyan Raev
20863e6bdf Fix recursive listing and deletion in S3 gateway (#333)
* Fix recursive listing and deletion in S3 gateway

* Prepend the prefix only on recursive listing

* Fix tests
2018-09-09 15:51:33 -06:00
Cameron
555e7f1897 CLI commands with bucket store API (#304)
* add mb command

* forgot colon

* add command descriptions

* use utils.ParseURL in commands

* return error message instead of minio.BucketAlreadyExists in mb

* ls command with bucket store functionality

* rb command with bucket store functionality

* rm command with bucket store functionality

* newline

* use print rather than errs for messages, add no buckets messsage

* cp command with bucket store functionality

* remove deprecated getStorjObjects function

* defer utils.LogClose(f) on instead of defer f.Close()

* Check for no buckets after for loop

* add checks for unspecified bucket in bucket store methods

* fix incorrect return types

* add no path error messages in object store methods

* split copy into helpers

* srcObj scheme check in download

* print buckets instead of appending to slice

* check if destObj.Host != srcObj.Host

* better method of handling destination name if not specified

* uplink rename

* final cleanups

* trailing slash fixes

* linting

* more linting

* helpful error messages

* Adjust startAfter after merging #328

* Improve output messages

* Improved error check for empty bucket and path

* No page limit on client side. Rely on server side limit.

* Better time formatting

* Fix paths in recursive list results
2018-09-09 11:31:26 -06:00
JT Olio
fd94300c53
move stacktraces to debug level, disable debug by default (#329)
* move stacktraces to debug level, disable debug by default

* configure zap better

* var to const
2018-09-09 05:27:09 -06:00
Kaloyan Raev
f62a3c383c
Filter prefix from list results (#328) 2018-09-09 09:34:23 +03:00
JT Olio
e0226790e6
miniogw: simplify output (#330) 2018-09-08 12:53:52 -06:00
Kaloyan Raev
2a1a52acc2 Non-recursive listing in Minio Gateway (#327)
* Non-recursive listing in Minio Gateway

* Cleaner code

* Fix tests
2018-09-08 11:10:58 -06:00
Alexander Leitner
c68cd1232d Prevent sql injection for farmer (#322)
* Validate incoming ID

* Pass tests, cleanup code

* Smarter regex

* Update server.go
2018-09-08 11:06:44 -06:00
Egon Elbre
4671ac6c5b Abort trying to upload segments when reader fails (#326)
* abort trying to upload segments when reader fails

* also return the error when it fails
2018-09-08 09:41:40 -06:00
Egon Elbre
a575a8e293
Safer and faster psdb (#324) 2018-09-08 18:34:55 +03:00
Michal Niewrzal
dfc52e4394 [WIP] Piece Rangers should connect lazily to storage nodes (second approach) (#321)
* Piece Rangers should connect lazily to storage nodes

* code reorganization

* fixed unit tests

* remove if
2018-09-08 16:52:19 +03:00
Matt Robinson
75c64035a2 Normalize the Docker Bits (#325)
* Normalize the dockerfiles and entrypoints

* fix formattting and docker-compose file

* add missing env var to satellite dockerfile

* Remove build args no longer in use

* More changes to make it all work
2018-09-07 15:46:08 -04:00
JT Olio
8f3398b349
merge uplink and storj (#323)
also remove testing package from import dependency chain
(testing defines some flags)
2018-09-07 09:01:04 -06:00
JT Olio
1ea3f9f03b eestream: better error reporting when downloads fail (#295)
* eestream: better error reporting when downloads fail

* Fix failing tests

* Make the check in hasEnoughShares a little bit cleaner
2018-09-07 10:37:13 -04:00
Egon Elbre
0a2839b2d7
Add IsPrefix support to piecestore (#313) 2018-09-07 17:20:15 +03:00
Egon Elbre
706a6ac26e
Use better test ports (#311)
Other interface or ports may be blocked or require permissions when running tests.
2018-09-07 12:42:04 +03:00
Egon Elbre
79354bf0ab
Enforce lookup limit for storage (#312) 2018-09-07 12:00:00 +03:00
Egon Elbre
ff57fa5c00
use better redis server setup with fallback (#310) 2018-09-07 10:33:33 +03:00
JT Olio
8da996d61a infectious: pull in optimizations (#320) 2018-09-07 10:32:54 +03:00
Egon Elbre
8a4151f397
Fix Redis Iterators (#318)
Redis SCAN may return same item multiple times, add a basic check to avoid duplicate entries.
2018-09-06 23:50:22 +03:00
nfarah86
f58bb3e41d removed bin (#319) 2018-09-06 16:31:50 -04:00
nfarah86
766a82c51f
Create secret key (#309)
* initial aws creds for uplink

* uplink  random key works

* fixed pr per  suggestion
2018-09-06 11:04:41 -07:00
aligeti
75110cf2da
check for unsupported flags while running uplink (#316)
* uplink handle invalid or more than needed arguments

* updated the error reporting

* rebase from master

* check for unsupported flags
2018-09-06 12:43:08 -04:00
Brandon Iglesias
f168e19fb4 adding mniewrzal to the CLA bot (#315)
* adding mniewrzal to the CLA bot

* adding stefanbenten
2018-09-06 17:23:57 +03:00
Matt Robinson
2c315db84b Fix the binaries and Docker Images (#306)
* Finally fix the binary builds for a couple os/arch combinations

* Make the file executable before adding it to the zip

* This cache hack wasn't suppose to make it into the PR

* Guard against calling the binary target by itself

* Order matters on OS X

* Vendored my image

* No 386 builds until storage is fixed.
2018-09-05 16:40:47 -04:00
JT Olio
a4d1070d68
storage/ec: fix some Get socket leaks (#308)
* storage/ec: fix some Get socket leaks

* fix linters
2018-09-05 10:08:39 -07:00
aligeti
2686476364
uplink handle invalid or more than needed arguments (#305)
* uplink handle invalid or more than needed arguments

* updated the error reporting
2018-09-05 12:46:48 -04:00
Egon Elbre
83df0ee1b0
Implement ListV2 with storage rework (#303)
1. Added KeyValueStore.Iterate for implementing the different List, ListV2 etc. implementations. This allows for more efficient use of memory depending on the situation.
2. Implemented an inmemory teststore for running tests. This should allow to replace MockKeyValueStore in most places.
3. Rewrote tests
4. Pulled out logger from bolt implementation so it can be used for all other storage implementations.
5. Fixed multiple things in bolt and redis implementations.
2018-09-05 19:10:35 +03:00
Cameron
6251b8e090 extract bucketstore code from NewGateway into helper (#300)
* extract bucketstore code from NewGateway into helper

* fix lines breaks
2018-09-05 19:03:20 +03:00
Egon Elbre
7ecca9988a
add reed-solomon encoding benchmark (#307) 2018-09-05 17:33:20 +03:00
Matt Robinson
0a78e741ec
Build and upload binaries (#296)
* First pass at building binaries

* Finished building images

* Reworked some bits to save binaries in a better place.

* First pass at uploading binaries

* Builds for freebsd fail currently

* Add ignore file for docker builds to not invalidate the cache as often

* Docker image for 1.11-alpine exists now

* Name windows binaries correctly

* The makefile is used in tests.

* Take a swing at updating the tests for 1.11

* Switch to something with glibc so the race detector works.

* Remove unused .PHONY targets
2018-08-31 11:21:44 -04:00
aligeti
743ec65204
hc to satellite rename (#282)
* hc to satellite rename

* rebase with master and fix spelling

* missed out rename

* rebase merge fix
2018-08-29 14:32:41 -04:00
JT Olio
2bba44fa1c
Update README.md (#299)
* Update README.md

* cla mention

* fix a-names

* fix a-names again
2018-08-29 09:28:29 -06:00
Cameron
ea9f5b4e63
CopyObject (#242)
* add CopyObject method

* use utils.LogClose

* extract common code from GetObject to getObject helper

* remove rr.Range from getObject helper, create helper putObject

* return rr, err in getObject helper

* extract code from PutObject into putObject helper

* remove commented out text

* remove other commented out code

* WIP trying to get storj cp command to work with copyObject

* fix typo in rb and now it works

* use rr.Size() instead of srcInfo.Size

* Revert "WIP trying to get storj cp command to work with copyObject"

This reverts commit e256b9f9a0fda728d41eb5b9d7a98b5446825842.

* add CopyObject test

* rebase and fix merge conflicts

* check error in gateway-storj test

* fix typo
2018-08-29 11:00:05 -04:00
Egon Elbre
23231a29b1
Reduce linter concurrency (#297)
Limit concurrency to reduce memory usage.
2018-08-28 18:00:18 +03:00
Bryan White
746b63f685
CA and identity commands (#235)
* wip ca/ident cmds

* minor improvements and commenting

* combine id and ca commands and add $CONFDIR

* add `NewIdenity` test

* refactor `NewCA` benchmarks

* linter fixes
2018-08-28 00:23:48 +02:00