Commit Graph

490 Commits

Author SHA1 Message Date
Kaloyan Raev
175c30048d
Use SaltedKeyFromPassphrase in Uplink CLI and Gateway (#2637)
Co-authored-by: Jeff Wendling <leterip@gmail.com>
2019-07-29 10:17:49 +03:00
Egon Elbre
5d0816430f
rename all the things (#2531)
* rename pkg/linksharing to linksharing
* rename pkg/httpserver to linksharing/httpserver
* rename pkg/eestream to uplink/eestream
* rename pkg/stream to uplink/stream
* rename pkg/metainfo/kvmetainfo to uplink/metainfo/kvmetainfo
* rename pkg/auth/signing to pkg/signing
* rename pkg/storage to uplink/storage
* rename pkg/accounting to satellite/accounting
* rename pkg/audit to satellite/audit
* rename pkg/certdb to satellite/certdb
* rename pkg/discovery to satellite/discovery
* rename pkg/overlay to satellite/overlay
* rename pkg/datarepair to satellite/repair
2019-07-28 08:55:36 +03:00
Egon Elbre
0cdeae1922 add missing error handling (#2630) 2019-07-25 17:01:44 +02:00
Stefan Benten
d84b987717 Disable CAWhitelist config when signing identities(#2628)
Ensure we don't enforce a signed Peer Identity during the Authorize command
2019-07-25 08:57:21 +02:00
Stefan Benten
6e5e261244
Fix missing Version config parameter (#2627) 2019-07-24 16:11:17 -04:00
Yingrong Zhao
6778caf846
satellite/console: add referral link logic (#2576)
* setup referral route

* referredBy

* add user id

* modify user query

* separate optional field from userInfo

* get current reward on init of satellite gui

* remove unsed code

* fix format

* only apply 0 credit on registration

* only pass required information for rewards

* fix time parsing

* fix test and linter

* rename method

* add todo

* remove user referral logic

* add null check and fix format

* get current offer

* remove partnerID on CreateUser struct

* fix storj-sim user creation

* only redeem credit when there's an offer

* fix default offer configuration

* fix migration

* Add helper function for get correct credit duration

* add comment

* only store userid into user_credit table

* add check for partner id to set correct offer type

* change free credit to use invitee credits

* remove unecessary code
2019-07-19 14:22:10 -04:00
Brandon Iglesias
b4c51e9744
update the uplink CLI copy (#2593) 2019-07-18 13:01:04 -04:00
Jess G
3af9250659
update irreparableDB.GetLimited query to use where instead of offset (#2585)
* update query to use where instead of offset, update tests

* update cmd/inspector irreparable

* add comment for offset
2019-07-18 09:21:21 -07:00
Simon Guindon
91f0adef10
Add the ability to set dial and request timeouts from the cmd/uplink CLI to libuplink. (#2439)
* Added the ability to pass timeout settings from cmd/uplink to libuplink.

* Removed commented out code.

* Updated 2min timeouts for the uplink CLI.

* Removed comment.

* Made transport defaultDialTimeout and defaultRequestTimeout public

* Added comments to describe where these defaults apply.

* Added a new defaults to libuplink and added tests.

* Added a new defaults to libuplink and added tests.
2019-07-18 11:13:59 -04:00
Andrew Harding
416fa80e85
Link Sharing Service (#2431)
Link sharing service. See `docs/design/link-sharing-service.md` for the design and `cmd/linksharing/README.md` for operational instructions.
2019-07-18 06:26:09 -06:00
Egon Elbre
f6f65a80d7
storagenode/trust: implement fetching peer identity without kademlia and endpoint (#2584) 2019-07-17 21:14:44 +03:00
Stefan Benten
4ebaae26d1
Disabling Dashboard Link, as its localhost and not working yet (#2579) 2019-07-16 17:03:03 -04:00
Jess G
f11bf46a11
Jg/1967 mv bucket metadata uplink (#2505)
* add bucketstore, add init uplink bucket

* update uplink to use bucket rpc

* fix tests

* wrap metainfo client errors

* add allowedBucket struct, fix tests

* update comment

* add paging

* updates per CR

* add test for pagination

* fix lint

* fix uplink test so its easier tyo understand

* fix gateway pagination bug

* changes per cr

* fix bug w/allowedBuckets, add test to catch
2019-07-12 08:57:02 -04:00
Stefan Benten
ccef5eee46
Add proper Version Handling to Identity, Gateway and Uplink Binary (#2471) 2019-07-08 10:45:20 -04:00
Alexander Leitner
88732188cb
Update inspector timestamp to time.time (#2464)
* Update inspector timestamp
2019-07-08 10:06:12 -04:00
Yaroslav Vorobiov
7aca0eb284 storagenode/dashboard: show console address (#2456) 2019-07-06 15:40:58 +02:00
aligeti
ae8b9698f9
Rename/remove EncryptionScheme -> EncryptionParame… (#2363)
* rename/remove EncryptionScheme -> EncryptionParameters
2019-07-03 14:07:44 -04:00
Egon Elbre
0c52f40d97
cmd/internal/assets: package for embedding resources (#2374) 2019-07-01 17:11:23 +03:00
Jeff Wendling
1a65e42d33 rename EncryptionCtx back to EncryptionAccess (#2368)
* rename EncryptionCtx back to EncryptionAccess

Change-Id: I5e58915a59979ad6f2e83d36e191b2bbf3ba2ba2

* missed some stuff

Change-Id: Ib65fb186d7f854c0406e5fd7d11498e8941da59e

* oops protolock

Change-Id: I85a5ab4bafb9dd6a804d3dcd17a70811702f07e4

* retrigger

Change-Id: I2d0e3f02b4dbae0299b090e9617662b4437980b0
2019-06-27 23:18:24 -06:00
Jeff Wendling
efcdaa43a3
lib/uplink: encryption context (#2349)
* lib/uplink: encryption context

Change-Id: I5c23dca3286a46b713b30c4997e9ae6e630b2280

* lib/uplink: bucket operation examples

Change-Id: Ia0f6e69f365dcff0cf11c731f51b30842bce053b

* lib/uplink: encryption key sharing test cases

Change-Id: I3a172d565f33f4e591402cdcb9460664a7cc7fbe

* fix encrypted path prefix restriction issue

Change-Id: I8f3921f9d52aaf4b84039de608b8cbbc88769554

* implement panics in libuplink encryption code

todo on cipher suite selection as well as an api concern

Change-Id: Ifa39eb3cc4b3443f7d96f9304df9b2ac4ec4085d

* implement GetProjectInfo api call to get salt

Change-Id: Ic5f6b3be9ea35df48c1aa214ab5d355fb328e2cf

* some fixes and accessors for encryption store

Change-Id: I3bb61f6712a037900e2a96e72ad4029ec1d3f718

* general fixes to builds/tests/etc

Change-Id: I9930fa96acb3b221d9a001f8e274af5729cc8a47

* java bindings changes

Change-Id: Ia2bd4c9c69739c8d3154d79616cff1f36fb403b6

* get libuplink examples passing

Change-Id: I828f09a144160e0a5dd932324f78491ae2ec8a07

* fix proto.lock file

Change-Id: I2fbbf4d0976a7d0473c2645e6dcb21aaa3be7651

* fix proto.lock again

Change-Id: I92702cf49e1a340eef6379c2be4f7c4a268112a9

* fix golint issues

Change-Id: I631ff9f43307a58e3b25a58cbb4a4cc2495f5eb6

* more linting fixes

Change-Id: I51f8f30b367b5bca14c94b15417b9a4c9e7aa0ce

* bug fixed by structs bump

Change-Id: Ibb03c691fce7606c35c08721b3ef0781ab48a38a

* retrigger

Change-Id: Ieee0470b6a2d07168a1578552e8e7f271ae93a13

* retrigger

Change-Id: I753d63853171e6a436c104ce176048892eb974c5

* semantic merge conflict

Change-Id: I9419448496de90340569047a6a16a1b858a7978a

* update total to match prod defaults

Change-Id: I693d55c1ebb28b5803ee1d26e9e198decf82308b

* retrigger

Change-Id: I28b74d5d6202f61aa3866fe407d423f6a0a14b9e

* retrigger

Change-Id: I6fd054885c715f602e2cef623fd464c42e88742c

* retrigger

Change-Id: I6a01bae88c72406d4ed5a8f13bf8a2b3c650bd2d
2019-06-27 17:36:51 +00:00
Egon Elbre
2128b460b4
cmd/uplink/cmd: don't create benchmark data on init (#2351) 2019-06-26 22:08:14 +03:00
Egon Elbre
b6ad3e9c9f
internal/testrand: new package for random data (#2282) 2019-06-26 13:38:51 +03:00
ethanadams
0e528bc56e
Add attribution report to the satellite CLI (#2288)
* added satalite partner value attribution report. WIP

* WIP

* basic attribution report test completed. still a WIP

* cleanup

* fixed projectID conversion

* report display cleanup

* cleanup .added more test data

* added partnerID to query results

* fixed lint issues

* fix import order

* suggestions from PR review

* updated doc to reflect implementation

* clarification comments in the report SQL

* Changed based on PR suggestion

* More changes based on PR suggestions

* Changes based on PR suggestions

* reordered tests to make consistant with previous 2

* small comments cleanup

* More PR suggestions

* fixed lint issue and removed printf

* fixed var name

* Updates based on PR suggestions

* fixed message

* fixed test

* changes required after merge from master
2019-06-25 16:58:38 -04:00
Egon Elbre
c7679b9b30
Fix some leaks and add notes about close handling (#2334) 2019-06-25 23:00:51 +03:00
Stefan Benten
57ef352b3c
Update Wizard to allow more easily addition of satellites and move package to cmd folder (#2340) 2019-06-25 20:25:31 +02:00
Stefan Benten
2cc01c7899 Rename Satellites in the Wizard (#2339)
* Rename Satellites

* Fixing Typo and changing names for gateway

* Unify Setup wizard across gateway and uplink
2019-06-25 11:29:52 -06:00
Alexander Bender
c42046601f Dockerfile changes for marketing resources. (#2329) 2019-06-25 15:41:19 +02:00
Jennifer Li Johnson
e285fe1997
Don't require encryption keys for project or bucket management (#2291) 2019-06-23 22:06:14 -04:00
Bill Thorp
8f47fca5d3
Remove audit / uptime ratio fields (#2247)
* removed ratios
2019-06-21 13:14:53 -04:00
Maximillian von Briesen
d103dd2c4e
Include information about disqualified nodes in the storage node payment report (#2239) 2019-06-21 10:21:15 -04:00
Yehor Butko
e5fd0287e4
V3-1819 Storage node operator server and service started (#2112)
* V3-1819 Storage node operator server and service started
2019-06-20 14:52:32 +03:00
Egon Elbre
35e71e5f95
cmd/storj-sim: ensure prefix writer can be used concurrently (#2243) 2019-06-19 22:30:48 +03:00
Egon Elbre
9d670b33b3
reduce expected disk size for storj-sim and testplanet (#2249) 2019-06-19 18:29:45 +03:00
Maximillian von Briesen
ad8cad4909
Expand the inspector tool to provide node id's for each segment, rather than just numeric totals (#2205) 2019-06-18 18:22:14 -04:00
Matt Robinson
c481e071b2 Rework gateway container image to build like the rest (#2230) 2019-06-18 10:29:22 -04:00
Egon Elbre
f4f776d092 Use mail.test as domain in emails (#2224) 2019-06-18 02:28:40 +02:00
Bill Thorp
f378125c8b
draft of removing stats from inspector (#2226) 2019-06-17 16:48:04 -04:00
Matt Robinson
2fdcd8c632 Give the uplink image the same business (#2168) 2019-06-13 18:14:53 +02:00
Bill Thorp
703e5fee49
mostly removed things (#2199) 2019-06-13 12:06:37 -04:00
Egon Elbre
1a1a084477
testcontext: sanitize folder name (#2195) 2019-06-13 15:46:08 +03:00
JT Olio
ff7a9030eb lib/uplink: expose restrict on api keys (#2189) 2019-06-12 23:35:57 +02:00
Egon Elbre
fdddaa2a47 Fix marketingweb code (#2177)
* set to only listen on 127.0.0.1, move static files to same location, better template handling

* handle error

* fix path in storj-sim

* revert template handling changes

* code shouldn't panic on invalid tempalte

* do not rewrite once writing has started

* write correct error code

* use filepath for path handling

* revert change

* fix

* fix mod tidy

* use correct error code for not found, avoid infinite loop on failure
2019-06-12 09:42:39 -04:00
Jess G
d84cd719ff
add const stripesPerBlock const to calc blocksize (#2163)
* add const stripesPerBlock, update comments in Pad, add speed to progressbar

* change size of erasure share size, update comments

* missing copyright

* update tests with stripesPerBlock
2019-06-11 10:14:05 -07:00
Yingrong Zhao
af66d9c6e4
Open a new port on satellite for admin GUI (#1901)
* Set up new port 8090 for in offers

Clean up commented code

Rename offers to offersweb

Remove unused code

Add todos for adding front-end templates

Add middleware for only allow local access

Add comment

Fix linting error

Remove commented code

Update storj-sim

Check request IP against Host IP

Use net pakcage to retrieve IP address

Rename service to marketing

* Add wrapper for all errors

* fix conflicts

* update the config file

* fix linting error

* remove unused packages

* remove global runtime var and add flag to storj-sim for mar static dir

* remove debugging lines

* add new config for test data and check if static dir flag is set before passing to mux

* change 'console' to 'marketing' for test data config

* fix linting errors

* update config flag

* Trigger Jenkins

* Trigger CLA
2019-06-11 11:00:59 -04:00
Stefan Benten
74484fc57e
Enforce our Minimum Requirements for Node Operators and sanity check them (#2155) 2019-06-10 12:14:50 +02:00
Ivan Fraixedes
f5227abd36
uplink: enc.encryption-key flag is only available for setup command (#2090)
* uplink: Mark encryption key config field for setup
  Set the "setup" property to the `EncryptionConfig.EncrptionKey` for
  avoiding to save it in the configuration file.
  This field is only meant for using in the command line parameters which
  need to use a different encryption key than the one present in the key
  file or use it when there is not set any encryption key file path.
* cmd/uplink: Setup non-interactive accept enc key
  Change the uplink CLI setup command non-interactive to save the
  encryption key into a file when it's passed through the flag
  --enc.encryption-key
  Previous to this change it wasn't possible to create an key file despite
  of that the flag was provided, so it was useless on the setup command.
* cmd/uplink: Reuse logic to read pwd from terminal
  Reuse the logic which is already implemented in the pkg/cfgstruct for
  reading a password from the terminal on interactive mode, rather than
  duplicating it in the setup command.
* cmd/gateway: Use encryption key file flags
  The cmd/gateway was still using the `enc.key` configuration field which
  doesn't exist anymore and its setup command wasn't using the
  `enc.key-filepath` with combination of the `enc.encryption-key` for
  generating a file with the encryption key.
  This commit update the cmd/gateway appropriately and move to the uplink
  package the function used by cmd/uplink to save the encryption key for
  allowing to also be used by the cmd/gateway without duplicating the
  logic.
* cmd/storj-sim: Adapt gateway config cmd changes
  Adapt the cmd/storj-sim to correctly pass the parameters to the
  cmd/gateway setup and run command.
* scripts: Don't pass the --enc.encryption-key flag
  uplink configuration has changed to only support the
  `--enc.encryption-key` flag for setup commands and consequently the
  cmd/uplink and cmd/gateway don't accept this flag over other commands,
  hence the test for the uplink had to be updated for no passing the
  flag on the multiples calls that the test do to cmd/uplink.
* uplink: Remove func which aren't useful anymore
  Remove the function which allows to user or load an encryption key
  because it isn't needed anymore since the `--enc.encryption-key` flag is
  only available for the setup command.
  Consequently remove its usage from cmd/uplink and cmd/gateway, because
  such flag will always be empty because in case that's passed Cobra will
  return an error due to a "unknown flag".
2019-06-07 18:14:40 +02:00
Jess G
bf3d168cf6
makes sure all uplink cli configs get passed to libuplink, add stripeSize (#2103)
* makes sure all uplink cli configs get passed to libuplink, add stripSize

* update comment

* update defaults for uplink config blocksize

* changes per CR, update uplink config defaults

* pass shareSize from uplink config

* move block size validation to kvmeta pkg

* fix tests

* shareSize default 1k, rm config option blocksize

* rm printing err to stdout
2019-06-06 11:55:10 -07:00
Egon Elbre
28a1201590 pkg/bwagreements: remove service (#2138) 2019-06-06 15:57:58 +02:00
JT Olio
9c5708da32 pkg/*: add monkit task to missing places (#2109) 2019-06-04 13:36:27 +02:00
Jeff Wendling
e74cac52ab
Command line flags features and cleanup (#2068)
* change BindSetup to be an option to Bind
* add process.Bind to allow composite structures
* hack fix for noprefix flags
* used tagged version of structs

Before this PR, some flags were created by calling `cfgstruct.Bind` and having their fields create a flag. Once the flags were parsed, `viper` was used to acquire all the values from them and config files, and the fields in the struct were set through the flag interface.

This doesn't work for slices of things on config structs very well, since it can only set strings, and for a string slice, it turns out that the implementation in `pflag` appends an entry rather than setting it.

This changes three things:

1. Only have a `Bind` call instead of `Bind` and `BindSetup`, and make `BindSetup` an option instead.
2. Add a `process.Bind` call that takes in a `*cobra.Cmd`, binds the struct to the command's flags, and keeps track of that struct in a global map keyed by the command.
3. Use `viper` to get the values and load them into the bound configuration structs instead of using the flags to propagate the changes.

In this way, we can support whatever rich configuration we want in the config yaml files, while still getting command like flags when important.
2019-05-29 17:56:22 +00:00