Commit Graph

1847 Commits

Author SHA1 Message Date
Yaroslav Vorobiov
eb5dc4ba2b
Add node stats GRPC endpoint to the satellite (#2281) 2019-06-25 13:40:32 +03:00
Egon Elbre
6502143e79
fix import ordering (#2322) 2019-06-25 12:46:29 +03:00
Ivan Fraixedes
fa802dc429
pkg/audit: Improve Reporter.ReporterAudits doc (#2266)
Add what the method returns because from the method name and the
signature name, it wasn't clear why and when a report is returned.
2019-06-25 11:23:41 +02:00
Jeff Wendling
c35c8e4c24 allow reading bucket metadata with restricted keys (#2321)
Change-Id: I47d3a2f5f02744ae6c51d54963cdf2dff24134e2
2019-06-24 17:36:09 -06:00
JT Olio
8c57434ded
pkg/process/metrics: add an instance prefix (#2190)
* pkg/process/metrics: add an instance prefix

the distinction between which satellite is sending which
data should go in the instance field, not the suffix or application
fields. (un)fortunately, the instance id is deliberately not
configurable because we don't want it to be easy to accidentally
have multiple applications collide with the same instance id.

so we're currently stuffing the human readable instance in the
suffix. :(

perhaps a reasonable tradeoff would be an optional instance
prefix that allows operators to put their domain name in
the instance

Change-Id: I6fcc8498be908c5740439cc00f77474ad151febd

* linting

Change-Id: I9f9a44fa9a2634ef5e4f89548d42d57ce9e4450e
2019-06-24 16:45:37 -06:00
Cameron
b3da72c21c
Fix TestVoucherService (#2317)
* change parameters in update stats to fix test issue
2019-06-24 18:29:24 -04:00
Kaloyan Raev
7bf5a09638 Use ErrNodeOffline when returning error for offline node (#2312) 2019-06-25 00:15:38 +02:00
Fadila
70f28ae41b
[v3-1952 tests 4 & 5] DQ nodes should not be used for download or upload (#2272) 2019-06-25 00:04:06 +02:00
Yingrong Zhao
bbedff12a6 satellite: rearrange marketing package (#2268)
* move offer out of marketing package and remove marketing package

* fix imports

* fix rename errors

* remove offer service

* change package name from offers to rewards

* fix linting

* remove unused code and use appropriate comment
2019-06-24 16:51:54 -04:00
Jeff Wendling
30f790a040
Create and use an encryption.Store (#2293)
* add path implementation

This commit adds a pkg/paths package which contains two types,
Encrypted and Unencrypted, to statically enforce what is contained
in a path. It's part of a refactoring of the code base to be more
clear about what is contained in a storj.Path at all the layers.

Change-Id: Ifc4d4932da26a97ea99749b8356b4543496a8864

* add encryption store

This change adds an encryption.Store type to keep a collection
of root keys for arbitrary locations in some buckets. It allows
one to look up all of the necessary information to encrypt paths,
decrypt paths and decrypt list operations.

It adds some exported functions to perform encryption on paths
using a Store.

Change-Id: I1a3d230c521d65f0ede727f93e1cb389f8be9497

* add shim around streams store

This commit changes no functionality, but just reorganizes the code
so that changes can be made directly to the streams store
implementation without affecting callers.

It also adds a Path type that will be used at the interface boundary
for the streams store so that it can be sure that it's getting well
formed paths that it expects.

Change-Id: I50bd682995b185beb653b00562fab62ef11f1ab5

* refactor streams to use encryption store

This commit changes the streams store to use the path type as
well as the encryption store to handle all of it's encryption
and decryption.

Some changes were made to how the default key is returned in
the encryption store to have it include the case when the bucket
exists but no paths matched. The path iterator could also be
simplified to not report if a consume was valid: that information
is no longer necessary.

The kvmetainfo tests were changed to appropriately pass the
subtests *testing.T rather than having the closure it executes
use the parent one. The test framework now correctly reports
which test did the failing.

There are still some latent issues with listing in that listing
for "a/" and listing for "a" are not the same operation, but we
treat them as such. I suspect that there are also issues with
paths like "/" or "//foo", but that's for another time.

Change-Id: I81cad4ba2850c3d14ba7e632777c4cac93db9472

* use an encryption store at the upper layers

Change-Id: Id9b4dd5f27b3ecac863de586e9ae076f4f927f6f

* fix linting failures

Change-Id: Ifb8378879ad308d4d047a0483850156371a41280

* fix linting in encryption test

Change-Id: Ia35647dfe18b0f20fe13763b28e53294f75c38fa

* get rid of kvmetainfo rootKey

Change-Id: Id795ca03d9417e3fe9634365a121430eb678d6d5

* Fix linting failure for return with else

Change-Id: I0b9ffd92be42ffcd8fef7ea735c5fc114a55d3b5

* fix some bugs adding enc store to kvmetainfo

Change-Id: I8e765970ba817289c65ec62971ae3bfa2c53a1ba

* respond to review feedback

Change-Id: I43e2ce29ce2fb6677b1cd6b9469838d80ec92c86
2019-06-24 19:23:07 +00:00
JT Olio
eb1b1c434d scripts/tag-release.sh: libuplink release tagging (#2256)
* scripts/tag-release.sh: libuplink release tagging

a couple of arch review meetings ago we discussed how to make sure
that it is much easier to get the release defaults for binaries,
libraries, and so on. we already imperfectly solved the binary
problem with the release.sh script, but (until now!) have not solved
the problem of getting release defaults for people building from
source.

the solution we seemed to all prefer was to make sure our tagged
version commits check the release state into the source code.

this script aides in tagging commits with version tags and
updating the source defaults. it still plays nicely with
release.sh and our other build processes.

after this is merged we should configure github/go modules to
prefer people use one of our tags instead of master (which will
keep dev defaults).

Change-Id: I36c5c33a1bc90ec1685f59b05dde779090e252b6

* gofmt release.go

Change-Id: I6e968eff86230496e9cbddecd767ca8d8ff36ba4

* regex for version tag

Change-Id: Icaa6d753ffc962115d961bcabe9daed89b16430c

* added some docs

Change-Id: Ide624fab794ce849e3a3e7254fb038251bba0c71
2019-06-24 19:53:17 +02:00
Yingrong Zhao
5fb2e01912
web/satellite: create ReferralStats component (#2309)
* create referral stats component

*  add title

* add copyright

* use correct font
2019-06-24 13:40:59 -04:00
JT Olio
e17908a266
metainfo: add project info rpc for getting a project salt using an api key (#2311)
The API exposed in #2302 needs to be able to get a project-level salt (https://github.com/storj/storj/pull/2302/files#diff-72b734f4a2e408a544fef071d2565fc8R200), see https://github.com/storj/storj/blob/master/docs/design/password-key-derivation.md also

I realized this morning that if we need it later this week, we need to get it into today's release.

Please describe the tests: tests will come as part of #2302

Please describe the performance impact: none
2019-06-24 11:15:45 -06:00
Maximillian von Briesen
fd6a4d96f2
change uptime dq threshold to 0.4 (#2313)
* change uptime dq threshold to 0.4

* update config lock
2019-06-24 12:18:32 -04:00
Nikolay Yurchenko
20041d43ba hovers on icons, apikey name errormessage added (#2307) 2019-06-24 18:50:07 +03:00
Stefan Benten
01beaa289a
Mask IP Addresses to subnets (#2305) 2019-06-24 17:33:18 +02:00
Yehor Butko
96bc0ccfa4
SNO Dshboard initial api endpoint added (#2284)
* initial api endpoint added
2019-06-24 18:15:31 +03:00
Fadila
8226024ca8
Do not use disqualified nodes when asking for get order limits (#2303)
Add checks about disqualification when creating order limits
2019-06-24 16:46:10 +02:00
Brandon Iglesias
fc11a17df7
initial design doc for SN graceful exit (#2077)
* initial design doc for SN graceful exit

* updates to the document mostly syntax and typos

* updates based on the first acceptance meeting

* SN will get paid for download data from clients

* updates based on our second acceptance meeting

* typo updates

* updates to things and stuff based on comments

* updated based on comment

* syntax changes/ typo fix
2019-06-24 10:28:53 -04:00
Kaloyan Raev
75d71f70c6 Stop pinging disqualified nodes (#2306) 2019-06-24 14:39:47 +02:00
Michal Niewrzal
fdeb834801
Bucket name validation (#2244) 2019-06-24 11:52:25 +02:00
Jennifer Li Johnson
e9e68c8420
kademlia audit gating design draft (#1941) 2019-06-23 23:12:31 -04:00
Jennifer Li Johnson
e285fe1997
Don't require encryption keys for project or bucket management (#2291) 2019-06-23 22:06:14 -04:00
paul cannon
8948459166
do ip filtering in a more correct way? (#2301)
This doesn't solve much of the performance difficulty but ought to be
lots more correct in terms of proper node selection semantics.
2019-06-23 16:16:45 -05:00
Bill Thorp
d1e9829e84 preserve reputation migration (#2295)
Adds a migration step to pull in old reputation success / total counts into modern alpha / beta scores

If audit success count is less than 50, audit alpha will be set to 50
If uptime success count is less than 100, uptime alpha will be set to 100
This helps us deal with cases where nodes have not been audited or checked for uptime yet, in which case alpha/beta values of 0/0 would cause a node to be considered disqualified.

A node with audit alpha/beta of 50/0 will be disqualified on the 19th check
A node with uptime alpha/beta of 100/0 will be disqualified on the 44th check

This does not affect brand new nodes (nodes that were not in the database before this change). The alpha/beta values for those nodes will be set to 1/0 as before
2019-06-22 21:18:54 -04:00
Cameron
1283036e37
add storage node voucher request service (#2158)
* add voucher service on storage node

* config field tag syntax, go routines for requests

* hook up voucher service in storagenode/peer.go

* add voucher config to testplanet

* add voucher config to testplanet

* add voucher response status INVALID, ACCEPTED, REJECTED

* add a test for vouchers service

* handle no row from GetValid, test it

* add trust pool to voucher service

* use trusted list to get satellites

* verify vouchers upon receipt

* test VerifyVoucher
2019-06-21 18:48:52 -04:00
Jeff Wendling
81f1bc19dc add functions for password/root key derivation (#2294)
This commit adds two functions that implement the algorithms
described in the password key derivation design document. They
will be used during setup to derive bucket level root keys or
default passwords to use when buckets do not have their own
independent key.

Change-Id: Ie7fb2d8d549ba7465d0722716a2c1ac0ad907286
2019-06-21 18:21:16 -04:00
aligeti
d8e62bc06e
support value attribution endpoint (#2231)
Support Value attribution Endpoint
2019-06-21 15:14:34 -04:00
Bryan White
9304817927
Uplink C bindings part 4 (#2260) 2019-06-21 20:44:00 +02:00
aligeti
043d603cbe
satellite rs config check with validation check set to false default (#2229)
* satellite rs config check with validation check
2019-06-21 14:15:58 -04:00
Bill Thorp
8f47fca5d3
Remove audit / uptime ratio fields (#2247)
* removed ratios
2019-06-21 13:14:53 -04:00
Egon Elbre
69c2cbacdf internal/testplanet: improve test speed (#2280) 2019-06-21 18:41:39 +02:00
Ivan Fraixedes
3d6b25a043
[v3-1952 test 1 & 3] pkg/audit: Add DQ test for too many failed audits (#2265)
* pkg/audit: Add DQ test for too many failed audits
  
  Add an integration test which checks that a node which fails several
  audits gets disqualified but not before it reaches the audit reputation
  disqualification cut-off.

* internal/testplanet: Set DQ cut-off config values
  Set the values of the Overlay cache DQ cut-off configuration parameters
  used by testplanet.
2019-06-21 18:27:19 +02:00
Egon Elbre
23e081f0c7 storagenode: delete piece when upload is cancelled (#2286)
* storagenode: delete piece when upload is cancelled

* don't delete when piece info has been committed
2019-06-21 18:16:39 +02:00
ethanadams
4f2e893e68
Fix the way project_id is stored in bucket_storage_tallies and bucket_bandwidth_rollups (#2283)
* fixing issues where projectID is stored as the byte representation of a UUID string, instead of bytes of the UUID

* added test for spitBucketID
2019-06-21 11:38:37 -04:00
Maximillian von Briesen
bfcfe39313
Enable and fix statdb tests (#2270) 2019-06-21 11:23:15 -04:00
Natalie Villasana
169fc9594c
pkg/audit: fix maxRetries bug (#2271) 2019-06-21 11:10:03 -04:00
Egon Elbre
86ec9b9570
jenkins: capability to use leakcheck (#2179) 2019-06-21 17:57:38 +03:00
Maximillian von Briesen
d103dd2c4e
Include information about disqualified nodes in the storage node payment report (#2239) 2019-06-21 10:21:15 -04:00
Bryan White
09e55ca28e
jenkins timeout workaround (#2285) 2019-06-21 16:04:34 +02:00
Egon Elbre
5b030062c0
internal/testplanet: split planet.go file to avoid package naming conflicts (#2279) 2019-06-21 16:39:43 +03:00
Bryan White
5f47b7028d Uplink C bindings part 3 (#2258)
* add object upload and download
2019-06-21 08:24:06 -04:00
littleskunk
a5baebfa65 reject invalid orders (#2262) 2019-06-21 13:44:14 +02:00
JT Olio
aa25c4458f kvmetainfo: merge with storage/buckets (#2277) 2019-06-21 13:29:31 +02:00
JT Olio
568b000e9b satellite: make order expiration configurable (#2251) 2019-06-21 13:38:40 +03:00
JT Olio
76b54458e9 satellite: send external address in order limits (#2278) 2019-06-21 12:19:52 +03:00
Kaloyan Raev
24918e072b
Include Node ID and Piece ID in piecestore errors (#2261) 2019-06-21 12:12:52 +03:00
JT Olio
06006effb3 lib/uplink: minor fixes (#2257) 2019-06-20 23:50:13 +02:00
Egon Elbre
d435d4859d internal/testplanet: add missing consoleserver.Config (#2269) 2019-06-20 22:27:36 +02:00
Jess G
daf166d423
fix tally test flakiness (#2250)
* fix test flakiness

* update tests to rm uplink.upload

* rm metainfo change

* add test case of no objects in bucket
2019-06-20 12:15:13 -07:00