Commit Graph

108 Commits

Author SHA1 Message Date
Jeff Wendling
c1fbfea7fa drpc: bump to latest version
Change-Id: I8426c2dd7f6263050c746c2724524ff687c7298a
2019-10-04 15:09:10 -06:00
Jeff Wendling
098cbc9c67 all: use pkg/rpc instead of pkg/transport
all of the packages and tests work with both grpc and
drpc. we'll probably need to do some jenkins pipelines
to run the tests with drpc as well.

most of the changes are really due to a bit of cleanup
of the pkg/transport.Client api into an rpc.Dialer in
the spirit of a net.Dialer. now that we don't need
observers, we can pass around stateless configuration
to everything rather than stateful things that issue
observations. it also adds a DialAddressID for the
case where we don't have a pb.Node, but we do have an
address and want to assert some ID. this happened
pretty frequently, and now there's no more weird
contortions creating custom tls options, etc.

a lot of the other changes are being consistent/using
the abstractions in the rpc package to do rpc style
things like finding peer information, or checking
status codes.

Change-Id: Ief62875e21d80a21b3c56a5a37f45887679f9412
2019-09-25 15:37:06 -06:00
Michal Niewrzal
68d281db44 autoupdater: use blang/semver (#3063) 2019-09-20 17:22:40 +03:00
Egon Elbre
149a2f8d3f
go.mod: update to grpc 1.23.1 and update golang.org/x (#3072) 2019-09-19 15:41:19 +03:00
Jeff Wendling
0dcbd3dc08 bootstrap/satellite/certificate/storagenode: register drpc services
Change-Id: Id29f14b76a8c9cb2be31001b9a7a4356a4bda183
2019-09-12 15:09:46 -06:00
Jeff Wendling
007662d49e pkg/server: serve drpc listeners along with grpc
The fundamental problem is that both drpc and grpc servers
want to close the listener and they both want to ignore the
error from Accept after the listener is closed. There's no
way to do this in a race free way. Fortunately, the mux
hands out listeners that can be independently closed. That
means they can both do their own shutdown logic where they
ignore the error, and then after they're closed, the code
orchestrating the servers can close the listeners.

The final weird bit is that the server's Close method is
required to wait until the Run method has exited (or at
least enough for the listeners to definitely be closed)
because tests depend on that behavior, so we have to add
some channels/mutexes/onces to ensure that Run has exited
and that a new call can't start after Close is called.

Change-Id: I7c4ef293f7963f83138815f51824fd5b8d09ce15
2019-09-12 19:18:30 +00:00
Egon Elbre
0d4fd6ceeb
cmd/uplink: fix progressbar data race (#2987) 2019-09-10 11:51:49 +03:00
Jeff Wendling
60eba990eb use-drpc: use protoc-gen-drpc to generate protobufs
Change-Id: I5c23256068e30864022dba5137c499796ab9d6ad
2019-09-06 13:28:27 -06:00
paul cannon
bf5a16b823 jenkins: work around git.apache.org failures (#2944) 2019-09-04 11:30:59 +03:00
Egon Elbre
b3a3d8df06
go.mod: update grpc-go (#2778) 2019-08-14 21:43:21 +03:00
Egon Elbre
7fcbec48f7 go.mod: update grpc-go and go (#2777) 2019-08-14 06:18:10 -04:00
Egon Elbre
9c4d2eb5bf go.mod: use pseudo version (#2741) 2019-08-08 11:09:36 -05:00
Egon Elbre
ad5a1d4a02
go.mod: use custom grpc that doesn't have tracing support (#2739) 2019-08-08 17:24:28 +03:00
Jeff Wendling
21a3bf89ee cmd/uplink: use scopes to open (#2501)
What: Change cmd/uplink to use scopes

It moves the fields that will be subsumed by scopes into an explicit legacy section and hides their configuration flags.

Why: So that it can read scopes in from files and stuff
2019-08-05 11:01:20 -06:00
Egon Elbre
ebbf0e1462
uplink/storage: don't import mock in production code (#2687) 2019-08-02 11:01:36 +03:00
Egon Elbre
9ba8b53ed5 pkg/auth: use grpc.WithPerRPCCredentials (#2670) 2019-07-31 13:57:13 +02:00
Michal Niewrzal
6f2b85603d
Metainfo RPC segment methods (part 1) (#2567) 2019-07-22 16:45:18 +02:00
aligeti
29b576961f
value attribution merge fix and more test cases (#2588)
* value attribution merge fix and more test cases
2019-07-19 11:17:34 -04:00
Yaroslav Vorobiov
7aca0eb284 storagenode/dashboard: show console address (#2456) 2019-07-06 15:40:58 +02:00
Michal Niewrzal
61dfa61e3a
Add timestamp and piece size to piece hash (#2198) 2019-07-03 18:14:37 +02:00
Egon Elbre
f1e670c9be go.mod: update grpc to 1.22.0 (#2444) 2019-07-03 14:25:10 +02:00
Egon Elbre
9a1e1d84f8 go.mod: enforce go 1.12 (#2399) 2019-07-01 13:57:05 +02:00
Faris Huskovic
ebd976ec28
satellite/marketing: Create New Offer (#2186)
* update UI to reflect final mockups

* implement create handler and render offers table data to UI

* fix line-height unit and remove important from selectors

* update file names and ids for clarity

* shorten 'label' in ids

* localize global vars, fix endpoint names, remove unnecessary receiver, fix comments

* fix unnecessary initialization of pointer

* correct file-naming conventions

* register timeConverter in an init func for safety and remove unnecessary important from css

* consolidate create endpoints and add comments

* register timeConverter in init func

* add copyright to files

* introduce require pkg

* add proper http server unit test

* update linting and create offers concurrently in unit test

* fix getOffers comment

* add copy-right to unit-test

* fix data-races

* fix linting

* remove converter in NewServer

* requested changes in progress

* add require for checking status code

* renamed template file

* fix 400 handler

* fix missing copyright and remove extra line

* fix build

* run goroutine for testing parallel

* evaluate reqType with switch stmt and promp for credit amount in cents

* fix lint issue

* add default case

* remove unnecessary var

* fix range scope error

* remove empty lines and use long form for struct field

* fix merge conflicts

* fix template reference

* fix modal id

* not delete package

* add currency formatting and requested changes

* markup formatting

* lean out currency logic and move wait outside loop

* pass ToDollars func to home template

* fix lint
2019-06-28 10:34:10 -04:00
Jeff Wendling
efcdaa43a3
lib/uplink: encryption context (#2349)
* lib/uplink: encryption context

Change-Id: I5c23dca3286a46b713b30c4997e9ae6e630b2280

* lib/uplink: bucket operation examples

Change-Id: Ia0f6e69f365dcff0cf11c731f51b30842bce053b

* lib/uplink: encryption key sharing test cases

Change-Id: I3a172d565f33f4e591402cdcb9460664a7cc7fbe

* fix encrypted path prefix restriction issue

Change-Id: I8f3921f9d52aaf4b84039de608b8cbbc88769554

* implement panics in libuplink encryption code

todo on cipher suite selection as well as an api concern

Change-Id: Ifa39eb3cc4b3443f7d96f9304df9b2ac4ec4085d

* implement GetProjectInfo api call to get salt

Change-Id: Ic5f6b3be9ea35df48c1aa214ab5d355fb328e2cf

* some fixes and accessors for encryption store

Change-Id: I3bb61f6712a037900e2a96e72ad4029ec1d3f718

* general fixes to builds/tests/etc

Change-Id: I9930fa96acb3b221d9a001f8e274af5729cc8a47

* java bindings changes

Change-Id: Ia2bd4c9c69739c8d3154d79616cff1f36fb403b6

* get libuplink examples passing

Change-Id: I828f09a144160e0a5dd932324f78491ae2ec8a07

* fix proto.lock file

Change-Id: I2fbbf4d0976a7d0473c2645e6dcb21aaa3be7651

* fix proto.lock again

Change-Id: I92702cf49e1a340eef6379c2be4f7c4a268112a9

* fix golint issues

Change-Id: I631ff9f43307a58e3b25a58cbb4a4cc2495f5eb6

* more linting fixes

Change-Id: I51f8f30b367b5bca14c94b15417b9a4c9e7aa0ce

* bug fixed by structs bump

Change-Id: Ibb03c691fce7606c35c08721b3ef0781ab48a38a

* retrigger

Change-Id: Ieee0470b6a2d07168a1578552e8e7f271ae93a13

* retrigger

Change-Id: I753d63853171e6a436c104ce176048892eb974c5

* semantic merge conflict

Change-Id: I9419448496de90340569047a6a16a1b858a7978a

* update total to match prod defaults

Change-Id: I693d55c1ebb28b5803ee1d26e9e198decf82308b

* retrigger

Change-Id: I28b74d5d6202f61aa3866fe407d423f6a0a14b9e

* retrigger

Change-Id: I6fd054885c715f602e2cef623fd464c42e88742c

* retrigger

Change-Id: I6a01bae88c72406d4ed5a8f13bf8a2b3c650bd2d
2019-06-27 17:36:51 +00:00
Kaloyan Raev
ae36979804 Take advantage of Is and IsFunc from zeebo/errs (#2310) 2019-06-26 09:38:07 +02:00
Maximillian von Briesen
ad8cad4909
Expand the inspector tool to provide node id's for each segment, rather than just numeric totals (#2205) 2019-06-18 18:22:14 -04:00
Faris Huskovic
7795ba8169
update UI to reflect final mockups (#2175)
* update UI to reflect final mockups

* fix line-height unit and remove important from selectors

* update file names and ids for clarity

* shorten 'label' in ids

* correct file-naming conventions

* add copyright to files

* check if break in imports is causing lint error

* resolve lint

* tidy go mod

* fix shorthands
2019-06-17 21:57:04 -04:00
JT Olio
7d5e425d89
monkit: start reporting 10th percentile, remove init.ializers from everything (#2173)
Change-Id: I2bb51f95493965bc14fb7134597fc9f274344807
2019-06-14 08:58:04 -06:00
aligeti
8e8a5eb89c
initial value attribution check development (#2091)
* initial value attribution check development

* code reveiew updates
2019-06-12 21:35:37 -04:00
Egon Elbre
fdddaa2a47 Fix marketingweb code (#2177)
* set to only listen on 127.0.0.1, move static files to same location, better template handling

* handle error

* fix path in storj-sim

* revert template handling changes

* code shouldn't panic on invalid tempalte

* do not rewrite once writing has started

* write correct error code

* use filepath for path handling

* revert change

* fix

* fix mod tidy

* use correct error code for not found, avoid infinite loop on failure
2019-06-12 09:42:39 -04:00
Yingrong Zhao
af66d9c6e4
Open a new port on satellite for admin GUI (#1901)
* Set up new port 8090 for in offers

Clean up commented code

Rename offers to offersweb

Remove unused code

Add todos for adding front-end templates

Add middleware for only allow local access

Add comment

Fix linting error

Remove commented code

Update storj-sim

Check request IP against Host IP

Use net pakcage to retrieve IP address

Rename service to marketing

* Add wrapper for all errors

* fix conflicts

* update the config file

* fix linting error

* remove unused packages

* remove global runtime var and add flag to storj-sim for mar static dir

* remove debugging lines

* add new config for test data and check if static dir flag is set before passing to mux

* change 'console' to 'marketing' for test data config

* fix linting errors

* update config flag

* Trigger Jenkins

* Trigger CLA
2019-06-11 11:00:59 -04:00
Kaloyan Raev
099cf921d3 Fix flaky audit test (#2093)
* Show full unexpected error when assertion fails

* restart jenkins

* restart jenkins

* restart jenkins

* restart jenkins

* Even better display of the unexpected error

* Fix flakiness using kaloyan-raev/errs fork

* Add more logs

* restart jenkins

* restart jenkins

* Better print for unexpected errors

* "error" instead of "err"

* restart jenkins

* Switch to zeebo/errs v1.2.0

* restart jenkins

* restart jenkins

* restart jenkins

* restart jenkins

* restart jenkins

* restart jenkins

* Remove some extra logs

* restart jenkins
2019-06-05 12:33:35 -06:00
Egon Elbre
3b02607c1d go.mod: update grpc to latest version (#2096) 2019-06-04 14:38:31 +02:00
Yaroslav Vorobiov
6809129e6f
Console add stripe service (#2080) 2019-06-03 16:46:57 +03:00
Jeff Wendling
e74cac52ab
Command line flags features and cleanup (#2068)
* change BindSetup to be an option to Bind
* add process.Bind to allow composite structures
* hack fix for noprefix flags
* used tagged version of structs

Before this PR, some flags were created by calling `cfgstruct.Bind` and having their fields create a flag. Once the flags were parsed, `viper` was used to acquire all the values from them and config files, and the fields in the struct were set through the flag interface.

This doesn't work for slices of things on config structs very well, since it can only set strings, and for a string slice, it turns out that the implementation in `pflag` appends an entry rather than setting it.

This changes three things:

1. Only have a `Bind` call instead of `Bind` and `BindSetup`, and make `BindSetup` an option instead.
2. Add a `process.Bind` call that takes in a `*cobra.Cmd`, binds the struct to the command's flags, and keeps track of that struct in a global map keyed by the command.
3. Use `viper` to get the values and load them into the bound configuration structs instead of using the flags to propagate the changes.

In this way, we can support whatever rich configuration we want in the config yaml files, while still getting command like flags when important.
2019-05-29 17:56:22 +00:00
Egon Elbre
2679b9046f update dependencies for bug fixes (#1993) 2019-05-19 16:58:30 +02:00
Alexander Leitner
b0f67820f6 Use libuplink in the cmd/uplink files (#1703)
* Use libuplink in the cmd/uplink files

Co-authored-by: "Dylan Lott" lott.dylan@gmail.com
2019-04-26 15:00:25 -06:00
Egon Elbre
60c4c10c79
storagenode: delete psserver (#1837) 2019-04-26 08:17:18 +03:00
Yaroslav Vorobiov
b38b87cb14
Console usage rollup api (#1664) 2019-04-04 17:56:20 +03:00
Michal Niewrzal
cab8fa349f
Use sha256-simd for hashing (#1656) 2019-04-04 10:12:08 +02:00
Egon Elbre
e016105a72 update some packages so we don't depend on launchpad (#1644) 2019-04-02 21:43:24 +03:00
Alexander Leitner
59b3eb190e
API endpoint checking object and segment health from the satellite (#1546)
* Generate protobuf for checking object and segment health

* Initial cmd client for health check

* Register the health inspector on the satellite peer
2019-04-02 10:55:58 -04:00
Natalie Villasana
a09f568fdb
audit service won't hang on bad storage node conns (#1503) 2019-03-19 13:37:26 -04:00
Cameron
c7ffbe1c28
Add ability to view irreparable segments on satellite (#1448)
* define irreparable inspector protobuf

* add IrreparableDB method GetLimited

* fill out irreparable inspector API

* add IrreparableInspector server to satellite, fix small error

* refactor IrreparableDB to use pb.IrreparableSegment instead of irreparable.RemoteSegmentInfo
2019-03-15 16:21:52 -04:00
Egon Elbre
0b5e7d34b7 update golang.org/x packages and grpc (#1363) 2019-02-26 16:49:56 +01:00
paul cannon
c35b93766d
Unite all cryptographic signing and verifying (#1244)
this change removes the cryptopasta dependency.

a couple possible sources of problem with this change:

 * the encoding used for ECDSA signatures on SignedMessage has changed.
   the encoding employed by cryptopasta was workable, but not the same
   as the encoding used for such signatures in the rest of the world
   (most particularly, on ECDSA signatures in X.509 certificates). I
   think we'll be best served by using one ECDSA signature encoding from
   here on, but if we need to use the old encoding for backwards
   compatibility with existing nodes, that can be arranged.

 * since there's already a breaking change in SignedMessage, I changed
   it to send and receive public keys in raw PKIX format, instead of
   PEM. PEM just adds unhelpful overhead for this case.
2019-02-07 14:39:20 -06:00
Bryan White
7b7e6c43f8
better batch-generation (#1219) 2019-02-06 09:04:12 +01:00
Kaloyan Raev
dd76829d10
Improve logic for cutting the long tail during upload (#909) 2019-02-05 12:54:25 +02:00
Dylan Lott
1385a2d6bd
Storagenode online confirmation ping (#1144)
* initial commit of dashboard connection tools

* Currently working against default configs

* Update configs and remove hardcoded addresses

* Update config handling

* adds external address param for tunneling nodes

* remove unnecessary address check logic
2019-01-25 10:46:09 -07:00
Alexander Leitner
c4afec9328
quick log cleanup for storage node (#1121)
* Clean up logs
2019-01-24 11:09:37 -05:00