Commit Graph

134 Commits

Author SHA1 Message Date
Michal Niewrzal
86f194769f uplink: adjust to changes in storj/uplink
This change is adjusting code base to changes in storj/uplink.

https://review.dev.storj.io/c/storj/uplink/+/643

Change-Id: Ieca87f9f5983e391bf4b4fec8b9d5491fd32bfa1
2020-01-20 22:06:19 +00:00
JT Olio
c01cbe0130 satellitedb: save out all db-touching traces
Change-Id: Ib1e192221f9da813fd9cbb55f620a047b82c9523
2020-01-14 18:47:45 -05:00
crawter
41d5e86306 satellite/payments: coupon addition removed
Change-Id: I92781d9133603fdefd58b19a6f0ac6b1c6df3ac6
2020-01-14 16:24:48 +00:00
Yingrong Zhao
ee87846f0b satellite/contact: add placeholder for GetTime endpoint
Change-Id: I42f8479708f0558350c2280a398d84d145e8118f
2020-01-14 06:38:47 +00:00
Egon Elbre
b252fcdc01 mod: bump uplink dependency
Change-Id: I67bee25c7cae28445e5091ed519359cfb5ea1af5
2020-01-09 15:38:02 +00:00
VitaliiShpital
a4e5c18877 satellite/payments: mock methods added to endpoint to match pb PaymentsServer
Change-Id: Ic8ff44cbe0b2368021a5d83cf86ce0dd2b670fd7
2020-01-09 14:30:21 +02:00
Egon Elbre
8d8d57c3b5 mod: update sqlite module to v2.0.2
This updates SQLite amalgamation from 3.29.0 to 3.30.1.

The module contains fixes for races.

Change-Id: Ic6a06a43ba404de0091d8a2f7444a8f4b1d5d54c
2020-01-08 21:21:15 +02:00
Egon Elbre
082ec81714
uplink: move to storj.io/uplink (#3746) 2020-01-08 15:40:19 +02:00
Egon Elbre
00c0c51b1c cmd/uplink: fix TestSetGetMeta flakiness
testrand.Path was also returning folders which has different behavior
for cp.

Change-Id: Ia53a2709bf3e768b3b7063a6137ec474c2622cb2
2020-01-08 12:25:25 +00:00
Egon Elbre
05e2b0ca86 mod: bump common and some other packages
Change-Id: Icf73220f5182f5a2fd1431471825948c300c0bea
2020-01-07 18:09:59 +02:00
Jeff Wendling
2549c601e9 all: bump storj.io/common dependency
Change-Id: I2e9ba0a76380d99d8650ae6921ed0a3ffe436536
2020-01-06 22:41:06 +00:00
Simon Guindon
e1e7cebe49 satellite/metainfo: added rate limiting support to the metainfo loop.
As per discussed we decided to rate limit how fast we iterate through
the metainfo database in the metainfo loop. This puts in place a
mechanism for rate limiting and burst limiting if need be in the future.

The default for this rate limiting is still no limits so it stays the
same as our previous functionality.

Change-Id: I950f7192962b0e49f082d2c4284e2d52b0a925c7
2020-01-03 15:00:29 -05:00
Egon Elbre
6615ecc9b6 common: separate repository
Change-Id: Ibb89c42060450e3839481a7e495bbe3ad940610a
2019-12-27 14:11:15 +02:00
Egon Elbre
ef8bc88328 ci: use external repository
Change-Id: If26a005df45f6067240511d603fb4dd613f92b79
2019-12-19 12:05:49 +00:00
Jennifer Johnson
7c5f777a4f satellitedb/repairqueue: runs a different implementation of the query within Select() for postgres vs cockroach
Change-Id: Ie34dbdb9d870d7d9f8f269702b6b3bad0c55b98e
2019-12-04 21:32:02 +00:00
JT Olio
031ba86de5
argon2: choose a steady parallelism value (#3630)
* argon2: choose a steady parallelism value

Change-Id: I6006da7d7980cda88f5f08ee759612df23a8132d

* whoops, not cruft

Change-Id: Ied9039f9a9be1d0f6ff3c7d5c4839a83fc7b4b1f

* fix broken test file

Change-Id: I07288cd6cef32ba387f2f003febff5c297e50997

* fix linting error

Change-Id: Icdbda8b709cc100a86f3859303c40edb8dff1e0f
2019-11-22 14:00:04 -07:00
Jeff Wendling
f3b20215b0 pkg/{rpc,server,tlsopts}: pick larger defaults for buffer sizes
these may not be optimal but they're probably better based on
our previous testing. we can tune better in the future now that
the groundwork is there.

Change-Id: Iafaee86d3181287c33eadf6b7eceb307dda566a6
2019-11-18 21:22:49 +00:00
Jeff Wendling
ecd2ef4a21 all: build release fully dprc and test in mixed mode
Change-Id: I3bded3edf25a0b113601c8b12ecf1337f596649b
2019-11-15 10:03:18 -07:00
Jeff Wendling
aaead2a29e
makefile: bump drpc and build storj-sim binaries like release (#3506)
Change-Id: I8cfd99226aefc2cfb5b18929799f56b3905cae53
2019-11-06 12:55:20 -07:00
Jess G
8d92c288e2
satellitedb: separate migration into subcommand (#3436)
* separate sadb migration, add version check

* update checkversion to do same validation as migration

* changes per CR

* add sa migration to storj-sim

* add different debug port in storj-sim for migration

* add wait for exit for storj-sim migration

* update sa docker entrypoint to support migration

* storj-sim satellite parts all wait for migration

* upgrade golang-migrate/migrate to v4 because bug

* fix go mod tidy
2019-11-02 13:09:07 -07:00
Jeff Wendling
d820b5abf0 drpc: bump version
Change-Id: I4c368a0200ea07dc4e1b1d14a750e64377e513dc
2019-11-01 23:27:02 +00:00
Jeff Wendling
506d9715da
drpc: version bump (#3386)
Change-Id: Ied42b1bbbe8bff0b3e5c10d29f895b70617373e6
2019-10-29 12:13:57 -06:00
Jeff Wendling
51d5d8656a pkg/rpc: drpc connection pooling
keep a pool of connections open when dialing for drpc. this
makes it so that long lived clients (like lib/uplink's Project)
don't continue to use a bad connection forever. it also allows
for concurrent rpcs.

Change-Id: If649b286050e4f09c413fadc3e1ce88f5fc6e600
2019-10-22 18:15:24 -06:00
Yingrong Zhao
743a0fc38b storagenode/cmd: create start graceful exit CLI (#3202) 2019-10-11 09:58:12 -04:00
Isaac Hess
89c68890b3
Bump drpc version (#3235) 2019-10-10 13:48:47 -06:00
Yehor Butko
0cc23add5b
satellite/payments - payment account setup (#3187) 2019-10-10 20:12:23 +03:00
Jeff Wendling
c1fbfea7fa drpc: bump to latest version
Change-Id: I8426c2dd7f6263050c746c2724524ff687c7298a
2019-10-04 15:09:10 -06:00
Jeff Wendling
098cbc9c67 all: use pkg/rpc instead of pkg/transport
all of the packages and tests work with both grpc and
drpc. we'll probably need to do some jenkins pipelines
to run the tests with drpc as well.

most of the changes are really due to a bit of cleanup
of the pkg/transport.Client api into an rpc.Dialer in
the spirit of a net.Dialer. now that we don't need
observers, we can pass around stateless configuration
to everything rather than stateful things that issue
observations. it also adds a DialAddressID for the
case where we don't have a pb.Node, but we do have an
address and want to assert some ID. this happened
pretty frequently, and now there's no more weird
contortions creating custom tls options, etc.

a lot of the other changes are being consistent/using
the abstractions in the rpc package to do rpc style
things like finding peer information, or checking
status codes.

Change-Id: Ief62875e21d80a21b3c56a5a37f45887679f9412
2019-09-25 15:37:06 -06:00
Michal Niewrzal
68d281db44 autoupdater: use blang/semver (#3063) 2019-09-20 17:22:40 +03:00
Egon Elbre
149a2f8d3f
go.mod: update to grpc 1.23.1 and update golang.org/x (#3072) 2019-09-19 15:41:19 +03:00
Jeff Wendling
0dcbd3dc08 bootstrap/satellite/certificate/storagenode: register drpc services
Change-Id: Id29f14b76a8c9cb2be31001b9a7a4356a4bda183
2019-09-12 15:09:46 -06:00
Jeff Wendling
007662d49e pkg/server: serve drpc listeners along with grpc
The fundamental problem is that both drpc and grpc servers
want to close the listener and they both want to ignore the
error from Accept after the listener is closed. There's no
way to do this in a race free way. Fortunately, the mux
hands out listeners that can be independently closed. That
means they can both do their own shutdown logic where they
ignore the error, and then after they're closed, the code
orchestrating the servers can close the listeners.

The final weird bit is that the server's Close method is
required to wait until the Run method has exited (or at
least enough for the listeners to definitely be closed)
because tests depend on that behavior, so we have to add
some channels/mutexes/onces to ensure that Run has exited
and that a new call can't start after Close is called.

Change-Id: I7c4ef293f7963f83138815f51824fd5b8d09ce15
2019-09-12 19:18:30 +00:00
Egon Elbre
0d4fd6ceeb
cmd/uplink: fix progressbar data race (#2987) 2019-09-10 11:51:49 +03:00
Jeff Wendling
60eba990eb use-drpc: use protoc-gen-drpc to generate protobufs
Change-Id: I5c23256068e30864022dba5137c499796ab9d6ad
2019-09-06 13:28:27 -06:00
paul cannon
bf5a16b823 jenkins: work around git.apache.org failures (#2944) 2019-09-04 11:30:59 +03:00
Egon Elbre
b3a3d8df06
go.mod: update grpc-go (#2778) 2019-08-14 21:43:21 +03:00
Egon Elbre
7fcbec48f7 go.mod: update grpc-go and go (#2777) 2019-08-14 06:18:10 -04:00
Egon Elbre
9c4d2eb5bf go.mod: use pseudo version (#2741) 2019-08-08 11:09:36 -05:00
Egon Elbre
ad5a1d4a02
go.mod: use custom grpc that doesn't have tracing support (#2739) 2019-08-08 17:24:28 +03:00
Jeff Wendling
21a3bf89ee cmd/uplink: use scopes to open (#2501)
What: Change cmd/uplink to use scopes

It moves the fields that will be subsumed by scopes into an explicit legacy section and hides their configuration flags.

Why: So that it can read scopes in from files and stuff
2019-08-05 11:01:20 -06:00
Egon Elbre
ebbf0e1462
uplink/storage: don't import mock in production code (#2687) 2019-08-02 11:01:36 +03:00
Egon Elbre
9ba8b53ed5 pkg/auth: use grpc.WithPerRPCCredentials (#2670) 2019-07-31 13:57:13 +02:00
Michal Niewrzal
6f2b85603d
Metainfo RPC segment methods (part 1) (#2567) 2019-07-22 16:45:18 +02:00
aligeti
29b576961f
value attribution merge fix and more test cases (#2588)
* value attribution merge fix and more test cases
2019-07-19 11:17:34 -04:00
Yaroslav Vorobiov
7aca0eb284 storagenode/dashboard: show console address (#2456) 2019-07-06 15:40:58 +02:00
Michal Niewrzal
61dfa61e3a
Add timestamp and piece size to piece hash (#2198) 2019-07-03 18:14:37 +02:00
Egon Elbre
f1e670c9be go.mod: update grpc to 1.22.0 (#2444) 2019-07-03 14:25:10 +02:00
Egon Elbre
9a1e1d84f8 go.mod: enforce go 1.12 (#2399) 2019-07-01 13:57:05 +02:00
Faris Huskovic
ebd976ec28
satellite/marketing: Create New Offer (#2186)
* update UI to reflect final mockups

* implement create handler and render offers table data to UI

* fix line-height unit and remove important from selectors

* update file names and ids for clarity

* shorten 'label' in ids

* localize global vars, fix endpoint names, remove unnecessary receiver, fix comments

* fix unnecessary initialization of pointer

* correct file-naming conventions

* register timeConverter in an init func for safety and remove unnecessary important from css

* consolidate create endpoints and add comments

* register timeConverter in init func

* add copyright to files

* introduce require pkg

* add proper http server unit test

* update linting and create offers concurrently in unit test

* fix getOffers comment

* add copy-right to unit-test

* fix data-races

* fix linting

* remove converter in NewServer

* requested changes in progress

* add require for checking status code

* renamed template file

* fix 400 handler

* fix missing copyright and remove extra line

* fix build

* run goroutine for testing parallel

* evaluate reqType with switch stmt and promp for credit amount in cents

* fix lint issue

* add default case

* remove unnecessary var

* fix range scope error

* remove empty lines and use long form for struct field

* fix merge conflicts

* fix template reference

* fix modal id

* not delete package

* add currency formatting and requested changes

* markup formatting

* lean out currency logic and move wait outside loop

* pass ToDollars func to home template

* fix lint
2019-06-28 10:34:10 -04:00
Jeff Wendling
efcdaa43a3
lib/uplink: encryption context (#2349)
* lib/uplink: encryption context

Change-Id: I5c23dca3286a46b713b30c4997e9ae6e630b2280

* lib/uplink: bucket operation examples

Change-Id: Ia0f6e69f365dcff0cf11c731f51b30842bce053b

* lib/uplink: encryption key sharing test cases

Change-Id: I3a172d565f33f4e591402cdcb9460664a7cc7fbe

* fix encrypted path prefix restriction issue

Change-Id: I8f3921f9d52aaf4b84039de608b8cbbc88769554

* implement panics in libuplink encryption code

todo on cipher suite selection as well as an api concern

Change-Id: Ifa39eb3cc4b3443f7d96f9304df9b2ac4ec4085d

* implement GetProjectInfo api call to get salt

Change-Id: Ic5f6b3be9ea35df48c1aa214ab5d355fb328e2cf

* some fixes and accessors for encryption store

Change-Id: I3bb61f6712a037900e2a96e72ad4029ec1d3f718

* general fixes to builds/tests/etc

Change-Id: I9930fa96acb3b221d9a001f8e274af5729cc8a47

* java bindings changes

Change-Id: Ia2bd4c9c69739c8d3154d79616cff1f36fb403b6

* get libuplink examples passing

Change-Id: I828f09a144160e0a5dd932324f78491ae2ec8a07

* fix proto.lock file

Change-Id: I2fbbf4d0976a7d0473c2645e6dcb21aaa3be7651

* fix proto.lock again

Change-Id: I92702cf49e1a340eef6379c2be4f7c4a268112a9

* fix golint issues

Change-Id: I631ff9f43307a58e3b25a58cbb4a4cc2495f5eb6

* more linting fixes

Change-Id: I51f8f30b367b5bca14c94b15417b9a4c9e7aa0ce

* bug fixed by structs bump

Change-Id: Ibb03c691fce7606c35c08721b3ef0781ab48a38a

* retrigger

Change-Id: Ieee0470b6a2d07168a1578552e8e7f271ae93a13

* retrigger

Change-Id: I753d63853171e6a436c104ce176048892eb974c5

* semantic merge conflict

Change-Id: I9419448496de90340569047a6a16a1b858a7978a

* update total to match prod defaults

Change-Id: I693d55c1ebb28b5803ee1d26e9e198decf82308b

* retrigger

Change-Id: I28b74d5d6202f61aa3866fe407d423f6a0a14b9e

* retrigger

Change-Id: I6fd054885c715f602e2cef623fd464c42e88742c

* retrigger

Change-Id: I6a01bae88c72406d4ed5a8f13bf8a2b3c650bd2d
2019-06-27 17:36:51 +00:00