Commit Graph

85 Commits

Author SHA1 Message Date
Jeff Wendling
efcdaa43a3
lib/uplink: encryption context (#2349)
* lib/uplink: encryption context

Change-Id: I5c23dca3286a46b713b30c4997e9ae6e630b2280

* lib/uplink: bucket operation examples

Change-Id: Ia0f6e69f365dcff0cf11c731f51b30842bce053b

* lib/uplink: encryption key sharing test cases

Change-Id: I3a172d565f33f4e591402cdcb9460664a7cc7fbe

* fix encrypted path prefix restriction issue

Change-Id: I8f3921f9d52aaf4b84039de608b8cbbc88769554

* implement panics in libuplink encryption code

todo on cipher suite selection as well as an api concern

Change-Id: Ifa39eb3cc4b3443f7d96f9304df9b2ac4ec4085d

* implement GetProjectInfo api call to get salt

Change-Id: Ic5f6b3be9ea35df48c1aa214ab5d355fb328e2cf

* some fixes and accessors for encryption store

Change-Id: I3bb61f6712a037900e2a96e72ad4029ec1d3f718

* general fixes to builds/tests/etc

Change-Id: I9930fa96acb3b221d9a001f8e274af5729cc8a47

* java bindings changes

Change-Id: Ia2bd4c9c69739c8d3154d79616cff1f36fb403b6

* get libuplink examples passing

Change-Id: I828f09a144160e0a5dd932324f78491ae2ec8a07

* fix proto.lock file

Change-Id: I2fbbf4d0976a7d0473c2645e6dcb21aaa3be7651

* fix proto.lock again

Change-Id: I92702cf49e1a340eef6379c2be4f7c4a268112a9

* fix golint issues

Change-Id: I631ff9f43307a58e3b25a58cbb4a4cc2495f5eb6

* more linting fixes

Change-Id: I51f8f30b367b5bca14c94b15417b9a4c9e7aa0ce

* bug fixed by structs bump

Change-Id: Ibb03c691fce7606c35c08721b3ef0781ab48a38a

* retrigger

Change-Id: Ieee0470b6a2d07168a1578552e8e7f271ae93a13

* retrigger

Change-Id: I753d63853171e6a436c104ce176048892eb974c5

* semantic merge conflict

Change-Id: I9419448496de90340569047a6a16a1b858a7978a

* update total to match prod defaults

Change-Id: I693d55c1ebb28b5803ee1d26e9e198decf82308b

* retrigger

Change-Id: I28b74d5d6202f61aa3866fe407d423f6a0a14b9e

* retrigger

Change-Id: I6fd054885c715f602e2cef623fd464c42e88742c

* retrigger

Change-Id: I6a01bae88c72406d4ed5a8f13bf8a2b3c650bd2d
2019-06-27 17:36:51 +00:00
Kaloyan Raev
ae36979804 Take advantage of Is and IsFunc from zeebo/errs (#2310) 2019-06-26 09:38:07 +02:00
Maximillian von Briesen
ad8cad4909
Expand the inspector tool to provide node id's for each segment, rather than just numeric totals (#2205) 2019-06-18 18:22:14 -04:00
Faris Huskovic
7795ba8169
update UI to reflect final mockups (#2175)
* update UI to reflect final mockups

* fix line-height unit and remove important from selectors

* update file names and ids for clarity

* shorten 'label' in ids

* correct file-naming conventions

* add copyright to files

* check if break in imports is causing lint error

* resolve lint

* tidy go mod

* fix shorthands
2019-06-17 21:57:04 -04:00
JT Olio
7d5e425d89
monkit: start reporting 10th percentile, remove init.ializers from everything (#2173)
Change-Id: I2bb51f95493965bc14fb7134597fc9f274344807
2019-06-14 08:58:04 -06:00
aligeti
8e8a5eb89c
initial value attribution check development (#2091)
* initial value attribution check development

* code reveiew updates
2019-06-12 21:35:37 -04:00
Egon Elbre
fdddaa2a47 Fix marketingweb code (#2177)
* set to only listen on 127.0.0.1, move static files to same location, better template handling

* handle error

* fix path in storj-sim

* revert template handling changes

* code shouldn't panic on invalid tempalte

* do not rewrite once writing has started

* write correct error code

* use filepath for path handling

* revert change

* fix

* fix mod tidy

* use correct error code for not found, avoid infinite loop on failure
2019-06-12 09:42:39 -04:00
Yingrong Zhao
af66d9c6e4
Open a new port on satellite for admin GUI (#1901)
* Set up new port 8090 for in offers

Clean up commented code

Rename offers to offersweb

Remove unused code

Add todos for adding front-end templates

Add middleware for only allow local access

Add comment

Fix linting error

Remove commented code

Update storj-sim

Check request IP against Host IP

Use net pakcage to retrieve IP address

Rename service to marketing

* Add wrapper for all errors

* fix conflicts

* update the config file

* fix linting error

* remove unused packages

* remove global runtime var and add flag to storj-sim for mar static dir

* remove debugging lines

* add new config for test data and check if static dir flag is set before passing to mux

* change 'console' to 'marketing' for test data config

* fix linting errors

* update config flag

* Trigger Jenkins

* Trigger CLA
2019-06-11 11:00:59 -04:00
Kaloyan Raev
099cf921d3 Fix flaky audit test (#2093)
* Show full unexpected error when assertion fails

* restart jenkins

* restart jenkins

* restart jenkins

* restart jenkins

* Even better display of the unexpected error

* Fix flakiness using kaloyan-raev/errs fork

* Add more logs

* restart jenkins

* restart jenkins

* Better print for unexpected errors

* "error" instead of "err"

* restart jenkins

* Switch to zeebo/errs v1.2.0

* restart jenkins

* restart jenkins

* restart jenkins

* restart jenkins

* restart jenkins

* restart jenkins

* Remove some extra logs

* restart jenkins
2019-06-05 12:33:35 -06:00
Egon Elbre
3b02607c1d go.mod: update grpc to latest version (#2096) 2019-06-04 14:38:31 +02:00
Yaroslav Vorobiov
6809129e6f
Console add stripe service (#2080) 2019-06-03 16:46:57 +03:00
Jeff Wendling
e74cac52ab
Command line flags features and cleanup (#2068)
* change BindSetup to be an option to Bind
* add process.Bind to allow composite structures
* hack fix for noprefix flags
* used tagged version of structs

Before this PR, some flags were created by calling `cfgstruct.Bind` and having their fields create a flag. Once the flags were parsed, `viper` was used to acquire all the values from them and config files, and the fields in the struct were set through the flag interface.

This doesn't work for slices of things on config structs very well, since it can only set strings, and for a string slice, it turns out that the implementation in `pflag` appends an entry rather than setting it.

This changes three things:

1. Only have a `Bind` call instead of `Bind` and `BindSetup`, and make `BindSetup` an option instead.
2. Add a `process.Bind` call that takes in a `*cobra.Cmd`, binds the struct to the command's flags, and keeps track of that struct in a global map keyed by the command.
3. Use `viper` to get the values and load them into the bound configuration structs instead of using the flags to propagate the changes.

In this way, we can support whatever rich configuration we want in the config yaml files, while still getting command like flags when important.
2019-05-29 17:56:22 +00:00
Egon Elbre
2679b9046f update dependencies for bug fixes (#1993) 2019-05-19 16:58:30 +02:00
Alexander Leitner
b0f67820f6 Use libuplink in the cmd/uplink files (#1703)
* Use libuplink in the cmd/uplink files

Co-authored-by: "Dylan Lott" lott.dylan@gmail.com
2019-04-26 15:00:25 -06:00
Egon Elbre
60c4c10c79
storagenode: delete psserver (#1837) 2019-04-26 08:17:18 +03:00
Yaroslav Vorobiov
b38b87cb14
Console usage rollup api (#1664) 2019-04-04 17:56:20 +03:00
Michal Niewrzal
cab8fa349f
Use sha256-simd for hashing (#1656) 2019-04-04 10:12:08 +02:00
Egon Elbre
e016105a72 update some packages so we don't depend on launchpad (#1644) 2019-04-02 21:43:24 +03:00
Alexander Leitner
59b3eb190e
API endpoint checking object and segment health from the satellite (#1546)
* Generate protobuf for checking object and segment health

* Initial cmd client for health check

* Register the health inspector on the satellite peer
2019-04-02 10:55:58 -04:00
Natalie Villasana
a09f568fdb
audit service won't hang on bad storage node conns (#1503) 2019-03-19 13:37:26 -04:00
Cameron
c7ffbe1c28
Add ability to view irreparable segments on satellite (#1448)
* define irreparable inspector protobuf

* add IrreparableDB method GetLimited

* fill out irreparable inspector API

* add IrreparableInspector server to satellite, fix small error

* refactor IrreparableDB to use pb.IrreparableSegment instead of irreparable.RemoteSegmentInfo
2019-03-15 16:21:52 -04:00
Egon Elbre
0b5e7d34b7 update golang.org/x packages and grpc (#1363) 2019-02-26 16:49:56 +01:00
paul cannon
c35b93766d
Unite all cryptographic signing and verifying (#1244)
this change removes the cryptopasta dependency.

a couple possible sources of problem with this change:

 * the encoding used for ECDSA signatures on SignedMessage has changed.
   the encoding employed by cryptopasta was workable, but not the same
   as the encoding used for such signatures in the rest of the world
   (most particularly, on ECDSA signatures in X.509 certificates). I
   think we'll be best served by using one ECDSA signature encoding from
   here on, but if we need to use the old encoding for backwards
   compatibility with existing nodes, that can be arranged.

 * since there's already a breaking change in SignedMessage, I changed
   it to send and receive public keys in raw PKIX format, instead of
   PEM. PEM just adds unhelpful overhead for this case.
2019-02-07 14:39:20 -06:00
Bryan White
7b7e6c43f8
better batch-generation (#1219) 2019-02-06 09:04:12 +01:00
Kaloyan Raev
dd76829d10
Improve logic for cutting the long tail during upload (#909) 2019-02-05 12:54:25 +02:00
Dylan Lott
1385a2d6bd
Storagenode online confirmation ping (#1144)
* initial commit of dashboard connection tools

* Currently working against default configs

* Update configs and remove hardcoded addresses

* Update config handling

* adds external address param for tunneling nodes

* remove unnecessary address check logic
2019-01-25 10:46:09 -07:00
Alexander Leitner
c4afec9328
quick log cleanup for storage node (#1121)
* Clean up logs
2019-01-24 11:09:37 -05:00
Egon Elbre
b6c61cdd55
Use storagenode.Peer for storagenode (#1107) 2019-01-23 12:39:03 +02:00
Egon Elbre
2d00f882a3
Kademlia Dialer (#1092) 2019-01-18 10:00:56 -05:00
Dylan Lott
82b9b54695
storagenode dashboard is working and passing tests locally (#1072)
* storagenode dashboard is working and passing tests locally

* linter fixes

* linter fixes

* moar linter fixes god

* review fixes and updates

* linter fixes
2019-01-16 16:09:13 -05:00
Egon Elbre
fe9dd0ba21
Upgrade monkit (#996) 2019-01-08 20:10:07 +02:00
Egon Elbre
cb74d91cb0
Upgrade monkit-hw (#984) 2019-01-07 21:11:11 +02:00
Egon Elbre
0ca03b41e2
Use generated locking database implementation (#947) 2019-01-02 19:53:27 +02:00
Bryan White
249244536a
CSR Service (part 2): cert signing rpc (#950)
* CSR Service:

+ implement certificate sign rpc method
+ implement certificate signer client/server
+ refactor `AuthorizationDB#Create`
+ refactor `NewTestIdentity`
+ add `AuthorizationDB#Claim`
+ add `Token#Equal`
+ fix `Authorizations#Marshal` when marshaling identities and certificates
+ tweak `Authorization#String` format
+ cert debugging improvements (jsondiff)
+ receive context arg in `NewTestIdentity`
+ misc. fixes
2019-01-02 12:39:17 -05:00
Egon Elbre
0a717ac4fe
Upgrade errs package (#912) 2018-12-20 00:48:06 +02:00
Egon Elbre
a5aaf54e7d
Upgrade hrtime package (#869) 2018-12-17 13:41:40 +02:00
Egon Elbre
f8caefcdeb
Update go.sum (#878) 2018-12-17 13:07:28 +02:00
Bryan White
2016ce9fd6
Certificate revocation (#836)
* wip certificate revocation

* refactor tests

* wip testing

* testing

* review fixes

* integration fix attempt #1

* review fixes

* integration fix attempt #2

* linter fixes

* add copywrite

* integration fix attemp #3

* more testing

* more tests

* go mod tidy

* review fixes

* linter fixes
2018-12-13 21:01:43 +01:00
Egon Elbre
e2d22b4b58 Update monkit-hw to not poll /proc on non-linux (#853) 2018-12-13 08:55:06 -07:00
JT Olio
362f447d9f
cmd/statreceiver: lua-scriptable stat receiver (#636)
* cmd/statreceiver: lua-scriptable stat receiver

Change-Id: I3ce0fe3f1ef4b1f4f27eed90bac0e91cfecf22d7

* some updates

Change-Id: I7c3485adcda1278fce01ae077b4761b3ddb9fb7a

* more comments

Change-Id: I0bb22993cd934c3d40fc1da80d07e49e686b80dd

* linter fixes

Change-Id: Ied014304ecb9aadcf00a6b66ad28f856a428d150

* catch errors

Change-Id: I6e1920f1fd941e66199b30bc427285c19769fc70

* review feedback

Change-Id: I9d4051851eab18970c5f5ddcf4ff265508e541d3

* errorgroup improvements

Change-Id: I4699dda3022f0485fbb50c9dafe692d3921734ff

* too tricky

the previous thing was better for memory with lots of errors at a time
but https://play.golang.org/p/RweTMRjoSCt is too much of a foot gun

Change-Id: I23f0b3d77dd4288fcc20b3756a7110359576bf44
2018-12-11 11:24:31 -07:00
Egon Elbre
4224be5bb1
install specific gogo (#757) 2018-12-05 11:06:57 +02:00
Egon Elbre
3e461df470
Linting protobuf files (#676) 2018-11-30 17:02:01 +02:00
Bryan White
2a0c4e60d2
preparing for use of customtype gogo extension with NodeID type (#693)
* preparing for use of `customtype` gogo extension with `NodeID` type

* review changes

* preparing for use of `customtype` gogo extension with `NodeID` type

* review changes

* wip

* tests passing

* wip fixing tests

* more wip test fixing

* remove NodeIDList from proto files

* linter fixes

* linter fixes

* linter/review fixes

* more freaking linter fixes

* omg just kill me - linterrrrrrrr

* travis linter, i will muder you and your family in your sleep

* goimports everything - burn in hell travis

* goimports update

* go mod tidy
2018-11-29 19:39:27 +01:00
Dylan Lott
07b4f3d7e3
updates the refresh interval in captplanet test script (#695)
* updates the refresh interval in captplanet test script

* remove sleep 25 line because unnecessary
2018-11-20 16:08:52 -07:00
Jennifer Li Johnson
93c5f385a8
Enable checker in captplanet and staging (#643)
* enable checker

* add option to use mock overlay in checker

* adds logs to checker

* appease linter
2018-11-20 10:54:22 -05:00
Yaroslav Vorobiov
c442205b3a
account api: infrastructure, login, register, getUser (#611) 2018-11-14 12:50:15 +02:00
Egon Elbre
6d401a4351
Check for go.mod validity (#605) 2018-11-09 15:32:35 +02:00
Yehor Butko
54cd9491c6
V3-664 Extend User repository. Reimplemented with dbx (#591)
* V3-664 Extend User repository. Reimplemented with dbx

* structure updated

* redundant packages removed, structure simplified

* fixing goimports

* removed constuctor from user struct

* separated types declarations from database file

* test file renamed

* fixes according to review

* fixing goimports

* fixing goimports
2018-11-08 16:19:42 +02:00
Egon Elbre
47d056d627
testcontext package (#545) 2018-10-29 16:16:36 +02:00
paul cannon
e2c0dd437a
offer PostgreSQL storage for pointerdb (#440)
..although it ought to work for other storage.KeyValueStore needs as
well. it's just optimized to work pretty well for a largish hierarchy of
paths.

This includes the addition of "long benchmarks" for KeyValueStore
testing. These will only be run when -test-bench-long is added to the
test flags. In these benchmarks, a large corpus of paths matching a
natural ("real-life") hierarchy is read from paths.data.gz (which you
can get from https://github.com/storj/path-test-corpus) and imported
into a particular KeyValueStore. Recursive and non-recursive queries are
run on it to detect performance problems that arise only at scale.

This also includes alternate implementation of the postgreskv client,
which works in a less-bizarre way for non-recursive queries, but suffers
from poor performance in tests such as the long benchmarks. Once this
alternate impl is committed to the tree, we can remove it again; I just
want it to be available for future reference.
2018-10-25 12:11:28 -05:00