Commit Graph

314 Commits

Author SHA1 Message Date
Bryan White
8edfd43abb
certificates config and --certs-dir fixes (#1093) 2019-01-22 13:35:48 +01:00
Egon Elbre
bbf81f2479 Consolidate identity management to identity cli commands (#1083)
* Consolidate identity management:

Move identity cretaion/signing out of storagenode setup command.

* fixes

* linters

* Consolidate identity management:

Move identity cretaion/signing out of storagenode setup command.

* fixes

* sava backups before saving signed certs

* add "-prebuilt-test-cmds" test flag

* linters

* prepare cli tests for travis

* linter fixes

* more fixes

* linter gods

* sp/sdk/sim

* remove ca.difficulty

* remove unused difficulty

* return setup to its rightful place

* wip travis

* Revert "wip travis"

This reverts commit 56834849dcf066d3cc0a4f139033fc3f6d7188ca.

* typo in travis.yaml

* remove tests

* remove more

* make it only create one identity at a time for consistency

* add config-dir for consitency

* add identity creation to storj-sim

* add flags

* simplify

* fix nolint and compile

* prevent overwrite and pass difficulty, concurrency, and parent creds

* goimports
2019-01-18 11:36:58 +01:00
paul cannon
681d77c41a
use _real_ cert-path to get node ID for metrics (#1058)
apparently, the presence of an "Identity" attribute in a config struct
does not imply that anything will fill it in or use it.

i tested all of these executables manually and individually this time.
2019-01-16 15:43:29 -05:00
paul cannon
248ee6438f
use Node ID for metrics ID (#1052)
This change removes automatic metrics reporting for everything going
through process.Exec(), and re-adds metrics reporting for those commands
which are expected to be long-lived. Other commands (which may have been
intermittently sending metrics before this, if they ran unusually long)
will no longer send any metrics.

For commands where it makes sense, a node ID is used as the metrics ID.
2019-01-15 10:02:54 -05:00
Kaloyan Raev
a63abf8fab
Save only user-specific flags to storage node's config.yaml (#1051) 2019-01-15 15:55:33 +02:00
Maximillian von Briesen
43ed9dea86
Remove overwrite flags (#1040)
remove overwrite flags from uplink, satellite, storagenode, captplanet, bootstrap

https://storjlabs.atlassian.net/browse/V3-1013
2019-01-14 10:57:58 -05:00
Philip Hutchins
b5f51014d3
Adding additional required values for satellite configuration (#1036)
* Adding other top level vars to be parsed into config

* Adding additional vars for parsing

* Fixing shell script conditionals in remaining entrypoints

* Adding null default for remaining env variable if statements
2019-01-11 22:47:18 -05:00
Michal Niewrzal
58b16c2ada
Fix for uplink setup command (#989) 2019-01-08 14:30:07 +01:00
Michal Niewrzal
4e857ea133
Add setup tag for config structs (#968)
* Fix ignored setup arguments

* fix linter errors

* small params cleanup

* fix integration tests

* cleanup in configs

* Add `setup` tag for config structs

* fix broken if statement

* cleanup captplanet config

* remove reduntant return

* add missing Signer config

* review comments

* local variable renamed

* remove unused var
2019-01-07 12:06:10 +01:00
Michal Niewrzal
4be58e2a76
Fix ignored setup arguments (#944)
* Fix ignored setup arguments

* fix linter errors

* small params cleanup

* fix integration tests

* cleanup in configs

* Restart Travis
2019-01-07 10:48:16 +01:00
JT Olio
2c916a04c3 pkg/provider: split into pkg/server, pkg/identity (#953) 2019-01-02 12:23:25 +02:00
Matt Robinson
5224e92646
Fix detection of existing config in docker images (#929)
* If the directory is somehow persistent, but the config file itself isn't created yet, create it

* Other improvements to thd storagenode Docker image

* Other improvements to thd satellite Docker image

* Other improvements to thd gateway Docker image

* Other improvements to thd uplink Docker image
2018-12-21 11:56:57 -05:00
Michal Niewrzal
f0662675bc
Add '--dir' param for all CLI parts (replace --base-path) (#833)
* Add '--dir' param for all CLI parts (replace --base-path)

* FindDirParam method moved

* fix compilation error

* make param global

* remove unused fields

* rename param

* remove config flag

* goimports
2018-12-14 22:14:59 +01:00
Kaloyan Raev
4cc49a8fa3
Better error handling of stream.Upload.Close (#811) 2018-12-10 16:35:22 +02:00
Kaloyan Raev
a4dec2da93
Fix "key not found" error on uploading files with uplink cli (#807) 2018-12-10 12:13:30 +02:00
Kaloyan Raev
72a20af569
Migrate Minio Gateway to the new Metainfo API (#775) 2018-12-07 20:31:29 +02:00
Bryan White
228aa34ff6
TLS extension processing (#771) 2018-12-07 14:44:25 +01:00
JT Olio
b6c936340b
clean up uplink config section (#786) 2018-12-06 17:18:11 -07:00
Kaloyan Raev
b7c45c7c16
Migrate Uplink CLI to Metainfo API (#745) 2018-12-04 18:46:02 +02:00
aligeti
c11a4385ae
upload filename fix (#760) 2018-12-04 10:02:12 -05:00
Michal Niewrzal
06c196d28c
Allow config generation into empty directory (#752)
* Allow config generation into empty directory

* unifed messages

* move main method to fpath

* fix linter problems
2018-12-04 12:42:16 +01:00
Michal Niewrzal
1206ef9126
Use OS related default setup directory (#741)
* Use OS related default setup directory

* added missing comment

* adjust test-captplanet.sh

* ApplicationDir moved to fpath
2018-12-03 16:51:56 +01:00
Egon Elbre
3e461df470
Linting protobuf files (#676) 2018-11-30 17:02:01 +02:00
Natalie Villasana
934e9c8dac
fixes Linux default base path for uplink (#730) 2018-11-28 14:17:58 -05:00
Natalie Villasana
d8b343ff3a
Fix default basepath location (#684) 2018-11-27 14:09:25 -05:00
Kaloyan Raev
8c56198090
Use designated error classes for common bucket and object errors (#691) 2018-11-21 13:17:28 +02:00
Michal Niewrzal
7e40caafcd
Write access for uplink mount command (#587)
* Write access for uplink mount command

* fix linter problems

* Fix closing wrong variable and debug logs

* goimports

* fix problem with missing getattr for large files

* better map initialization

* fixes after comments

* add support for making directory

* add support for removing directories

* optimize OpenDir and Rmdir methods

* use storj.JoinPaths

* fix linter issues

* remove useless error check
2018-11-21 11:19:54 +01:00
Michal Niewrzal
b1cab0fc5a
Fix for uplink cat writing additional bytes (#655)
* Fix for uplink cat writing additional bytes

* rename progress parameter
2018-11-15 16:21:58 +01:00
Kaloyan Raev
5cdb03515f
Configure path encryption in metainfo API (#644) 2018-11-14 11:26:18 +02:00
Michal Niewrzal
a62b413de2
Uplink mount better prefix handling (#541)
* Uplink mount better prefix handling

* fix formatting
2018-10-31 21:38:32 +01:00
Michal Niewrzal
79e1ed5d1f
Fix uplink mount for latest api (#539)
Fix uplink mount for changes in store.List
2018-10-26 16:22:20 +02:00
JT Olio
fdfb5f1da9
uplink: fix mount and cli/gateway logical merge conflict (#537)
Change-Id: Ib1661e1c9d3f06783854b2293579ab715d0f1c20
2018-10-25 16:00:15 -06:00
JT Olio
cd51787827 split gateway from uplink
Change-Id: I7504a630cb85539750e32780c1c2347742a9c9c4
Reviewed-on: https://review.gerrithub.io/430644
Reviewed-by: Kaloyan Raev <kaloyan@storj.io>
Tested-by: Kaloyan Raev <kaloyan@storj.io>
2018-10-25 15:01:44 -06:00
Kaloyan Raev
99640225fd
Refactor Path type (#522)
The old paths.Path type is now replaced with the new storj.Path.

storj.Path is simply an alias to the built-in string type. As such it can be used just as any string, which simplifies a lot working with paths. No more conversions paths.New and path.String().

As an alias storj.Path does not define any methods. However, any functions applying to strings (like those from the strings package) gracefully apply to storj.Path too. In addition we have a few more functions defined:

    storj.SplitPath
    storj.JoinPaths
    encryption.EncryptPath
    encryption.DecryptPath
    encryption.DerivePathKey
    encryption.DeriveContentKey

All code in master is migrated to the new storj.Path type.

The Path example is also updated and is good for reference: /pkg/encryption/examples_test.go

This PR also resolve a nonce misuse issue in path encryption: https://storjlabs.atlassian.net/browse/V3-545
2018-10-25 23:28:16 +03:00
Michal Niewrzal
b3c1ea1852
Add 'mount' command for uplink cli (read-only access) (#345)
* Add 'mount' command for uplink cli

* limit build to unix

* small changes

* Removed in-memory files map, fix in read function

* unmount on interrupt

* add delete support

* some Read improvements

* update to latest api version

* removed Println and better error handling

* fix linter errors

* fix linter errors

* disable async read + review comments

* minor fixes

* remove mutex

* code cleanup

* add prefix support

* simplifying if statement
2018-10-25 12:44:34 +02:00
Michal Niewrzal
b2caac4323
Option for generating sample TLS certs for Minio GW (#525)
* Option for generating sample TLS certs for Minio GW

* os.Link errors handled

* rename option and change cert template
2018-10-24 22:17:28 +02:00
Michal Niewrzal
6ffded3186
Uplink cp upload improvements (#480)
* Uplick cp upload improvements

* fix formatting
2018-10-16 14:10:41 +02:00
Stefan Benten
586ed1b58d
Improve path handling + filepath handling framework (#423)
* Initial Layout

* Commit to test File Handling OS independed

* Hide struct properties to prevent manual interaction

* Fix Linting Errors

* 1st Working Windows Version

* Add missing Error Handling

* Fix Linting Errors

* Remove dependencies

* Further Improvements

* Remove commented code

* Improve comments and error messages

* No pointers to FPath

* Improve comment

* Do not filepath.ToSlash URL path

* Extract helper functions for parsing local path and Storj path

* Minor Improvements based on PR Comments

* Fix Linting Error and make Regex private

* Improve Layout

* Rework FPath and add tests

* Add more tests cases for windows

* Use for-loop instead of goto

* Use FPath in all uplink commands

* Add guard checks

* Add Test Cases and add comments
2018-10-15 16:10:20 +02:00
Michal Niewrzal
a4b5978259
Don't override file if remote file doesn't exist (#445) 2018-10-09 14:36:36 +02:00
aligeti
cf329b1f37
Handle CTRL+C and clean up the partial segments and pieces (#381)
* merged the lasted master changes

* debug working of handling ctrl+c

* Handling of clean up of partially uploaded segments and pieces

* code cleanup per code comment

* updates based on code review comments
2018-10-04 09:52:12 -04:00
JT Olio
d8c0f18059 uplink: encryption key setup (#414) 2018-10-04 15:36:26 +03:00
Michal Niewrzal
39b3a09456
Fix uplink cp progress bar (#394)
* Fix uplink cp progress bar

* disable progress for logging above warn

* remove log level check
2018-09-30 22:25:58 +02:00
Matt Robinson
d2dcd8a136 Revert break for local files (#366) 2018-09-20 14:44:28 -04:00
Alexander Leitner
5a4f5e55ff
Add progress bar (#358)
* Add progress bar

* Turn progress bar on and off

* fix dependencies
2018-09-20 14:41:42 -04:00
Matt Robinson
e1f44ec782 Make the next shirt, better (#362) 2018-09-18 09:26:59 -06:00
Kaloyan Raev
ee5f32b0f7
ConcatRanger closes segment readers as soon as data is read (#349)
* ConcatRanger closes a segment readers as soon as data is read

* Fix linter errors

* Fix data race

* Nicer call to CombineErrors
2018-09-14 10:10:43 -04:00
Stefan Benten
05ce4e5269
Fix path handling (#354) 2018-09-14 15:14:04 +02:00
aligeti
3571886fe7 check uplink setup (#339)
* check uplink setup

* code review comments changes
2018-09-13 13:34:01 -04:00
Matt Robinson
0b947355da A bunch of small fixes to make OPsy things better (#346) 2018-09-12 11:02:53 -04:00
Michal Niewrzal
e90c2140b8 Show file size for uplink ls command (#344)
* Show file size for uplink ls command

* align size to right

* Fix stream store list method to handle meta.Size flag
2018-09-12 09:03:31 -04:00
Egon Elbre
e7e2d4d7c9
use golangci-lint and add missing error checks (#340) 2018-09-11 16:13:25 +03:00
Cameron
555e7f1897 CLI commands with bucket store API (#304)
* add mb command

* forgot colon

* add command descriptions

* use utils.ParseURL in commands

* return error message instead of minio.BucketAlreadyExists in mb

* ls command with bucket store functionality

* rb command with bucket store functionality

* rm command with bucket store functionality

* newline

* use print rather than errs for messages, add no buckets messsage

* cp command with bucket store functionality

* remove deprecated getStorjObjects function

* defer utils.LogClose(f) on instead of defer f.Close()

* Check for no buckets after for loop

* add checks for unspecified bucket in bucket store methods

* fix incorrect return types

* add no path error messages in object store methods

* split copy into helpers

* srcObj scheme check in download

* print buckets instead of appending to slice

* check if destObj.Host != srcObj.Host

* better method of handling destination name if not specified

* uplink rename

* final cleanups

* trailing slash fixes

* linting

* more linting

* helpful error messages

* Adjust startAfter after merging #328

* Improve output messages

* Improved error check for empty bucket and path

* No page limit on client side. Rely on server side limit.

* Better time formatting

* Fix paths in recursive list results
2018-09-09 11:31:26 -06:00
JT Olio
e0226790e6
miniogw: simplify output (#330) 2018-09-08 12:53:52 -06:00
Matt Robinson
75c64035a2 Normalize the Docker Bits (#325)
* Normalize the dockerfiles and entrypoints

* fix formattting and docker-compose file

* add missing env var to satellite dockerfile

* Remove build args no longer in use

* More changes to make it all work
2018-09-07 15:46:08 -04:00
JT Olio
8f3398b349
merge uplink and storj (#323)
also remove testing package from import dependency chain
(testing defines some flags)
2018-09-07 09:01:04 -06:00
nfarah86
f58bb3e41d removed bin (#319) 2018-09-06 16:31:50 -04:00
nfarah86
766a82c51f
Create secret key (#309)
* initial aws creds for uplink

* uplink  random key works

* fixed pr per  suggestion
2018-09-06 11:04:41 -07:00
aligeti
75110cf2da
check for unsupported flags while running uplink (#316)
* uplink handle invalid or more than needed arguments

* updated the error reporting

* rebase from master

* check for unsupported flags
2018-09-06 12:43:08 -04:00
Matt Robinson
2c315db84b Fix the binaries and Docker Images (#306)
* Finally fix the binary builds for a couple os/arch combinations

* Make the file executable before adding it to the zip

* This cache hack wasn't suppose to make it into the PR

* Guard against calling the binary target by itself

* Order matters on OS X

* Vendored my image

* No 386 builds until storage is fixed.
2018-09-05 16:40:47 -04:00
aligeti
2686476364
uplink handle invalid or more than needed arguments (#305)
* uplink handle invalid or more than needed arguments

* updated the error reporting
2018-09-05 12:46:48 -04:00
Matt Robinson
0a78e741ec
Build and upload binaries (#296)
* First pass at building binaries

* Finished building images

* Reworked some bits to save binaries in a better place.

* First pass at uploading binaries

* Builds for freebsd fail currently

* Add ignore file for docker builds to not invalidate the cache as often

* Docker image for 1.11-alpine exists now

* Name windows binaries correctly

* The makefile is used in tests.

* Take a swing at updating the tests for 1.11

* Switch to something with glibc so the race detector works.

* Remove unused .PHONY targets
2018-08-31 11:21:44 -04:00
Egon Elbre
0f5a2f4ef5 Enable more linters (#272)
* enable more linters

* Run gofmt -s

* run goimports

* run unconvert

* fix naked return

* fix misspellings

* fix ineffectual assigments

* fix missing declaration

* don't use deprecated grpc.Errof

* check errors in tests

* run gofmt -w -r "assert.Nil(err) -> assert.NoError(err)"

* fix directory permissions

* don't use nil Context

* simplify boolean expressions

* use bytes.Equal instead of bytes.Compare

* merge variable declarations, remove redundant returns

* fix some golint errors

* run goimports

* handle more errors

* delete empty TestMain

* delete empty TestMain

* ignore examples for now

* fix lint errors

* remove unused values

* more fixes

* run gofmt -w -s .

* add more comments

* fix naming

* more lint fixes

* try switching travis to go1.11

* fix unnecessary conversions

* fix deprecated methods

* use go1.10 and disable gofmt/goimports for now

* switch to 1.10

* don't re-enable gofmt and goimports

* switch covermode to atomic because of -race

* gofmt
2018-08-27 11:28:16 -06:00
JT Olio
560e9fd9ec
uplink: make ul/uplink consistent (#286)
* uplink: make ul/uplink consistent

* pstore: make test more robust
2018-08-24 14:02:42 -06:00
aligeti
4edba19986 [Ready for review]Rename gw uplink (#273)
Rename gw uplink
2018-08-24 10:22:24 -06:00