Commit Graph

1612 Commits

Author SHA1 Message Date
Kaloyan Raev
de8070730a Save hash of expected share from missing audited node (#2025) 2019-05-23 16:07:19 -04:00
Jeff Wendling
b0ca12e31e
log errors only when an error happens (#2029) 2019-05-23 17:20:01 +00:00
Jeff Wendling
47a2f5e8b6
Password based key derivation design (#2020)
* initial password key derivation design

* some typos/clarifications

* point out that clients can ignore bucket entropy

* clarification about offline table generation

* change proposal to only do network requests when inputting passwords

* make algorithm have shorter lines

* fix algorithm example

* be explicit about allowing already-formed root keys

* go back to having a salt in the project/bucket metadata

* some proofreading/clarifications

* feedback and clarifications

* clarify definition of root key
2019-05-23 17:05:16 +00:00
Maximillian von Briesen
da91d22376 properly check last iteration of checker (#2040) 2019-05-23 18:14:08 +02:00
Ivan Fraixedes
69cf6caa62
cmd/uplink: Fix error on interactive setup when --config-dir flag is used. (#2033)
Uplink CLI was returning an error when the `--config-dir` flag was
used with interactive `setup` command because the directory used for
storing the encryption key file was the default configuration directory
rather than the value set in the flag.

This changes make the setup process to use the same directory, than the
one informed by the `--config-dir` flag value, for storing encryption
key file.

On the other hand, the default value for the encryption key file path
which was set with the purpose of showing it in the help message has
been removed because the cfgstruct doesn't show it unless that the flag
is set to the Cobra root CMD, so it was useless.

Also show the absolute path where the encryption key file has been saved when
the interactive setup command ends satisfactorily as it's equally done
with the configuration file.
2019-05-23 16:49:28 +02:00
Natalie Villasana
8b31c4b91f
remove unnecessary containmentDB wrapper (#2027) 2019-05-23 10:37:23 -04:00
Natalie Villasana
ef4849b53b
skip flaky TestProjectUsageBandwidth (#2035) 2019-05-23 10:27:57 -04:00
Maximillian von Briesen
b4f18226db
Send number of files as part of durability stats (#2030) 2019-05-22 18:50:43 -04:00
JT Olio
24787adb5b
cmd/statreceiver: allow for packet filtering to packet destinations (#2019)
What: allow packetfilter to work on packet destinations instead of only on metric destinations. this will allow us to filter what applications get sent to rothko.

Why: currently rothko is drowning in storj-sim data and it'd be nice to filter out.
2019-05-22 16:19:32 -06:00
Yingrong Zhao
f9045f8385 Create offer table in satellitedb for free credit program (#1975) 2019-05-22 23:41:55 +02:00
Matt Robinson
5d5d82d2f7 Show all of the claims (#2009) 2019-05-22 23:31:32 +02:00
Maximillian von Briesen
45a2253628 Send durability stats after iterating over all segments (#2028) 2019-05-22 17:17:52 -04:00
Cameron
4058c29ca4
filter duplicate node IPs (#1890)
* add last_ip field to dbx model node, generate dbx

* add last_ip to node proto, generate pb

* migrate

* resolve address in transport.DialNode, update lastIp in cache.UpdateAddress

* use net.SplitHostPort to isolate host address from port

* define DistinctIPs flag

* add test for GetIP

* select last_ip when querying for nodes

* if distinctIPs flag == true, query for nodes with distinct IPs

* some basic tests

* change last_ip to field 14 in proto

* remove comments

* check err

* change distinctIPs to distinctIP

* exclude IPs from newNodes in query for reputable nodes

* add index on last_ip

* only add to excludedIPs if flag is true

* test half new nodes returns distinct IPs

* fix alignment

* add test

* rework ip filter query, add retry logic, add switch for database driver

* add retry to SelectNewNodes

* change discovery intervals so IPs don't get overwritten

* remove TestGetIP

* edit updating node stats in test

* split exclude into nodeIDs and IPs

* separate non-distinct IP query into other function

* trigger checks

* remove else block
2019-05-22 16:06:27 -04:00
Jeff Wendling
b3e6236142
add uplink-contexts design doc (#1959)
* add uplink-contexts design doc

* respond to some feedback and rename context to scope

* feedback summaries

* review fixes

* review feedback

* more review feedback

* fix types in go structs
2019-05-22 19:49:46 +00:00
Cameron
a883f87dc3
define vouchers proto (#2026) 2019-05-22 15:15:59 -04:00
Natalie Villasana
acfb59ae48
adds db layer functions for containment (#1972) 2019-05-22 10:50:22 -04:00
Ivan Fraixedes
69d8b9f828
Change where the encryption key is being stored for uplink (#1967)
* uplink: Add a new flag to set the filepath of the file which is used for 
  saving the encryption key and rename the one that hold the encryption key and 
  establish that it has priority over the key stored in the file to make the 
  configuration usable without having a huge refactoring in test-sim.
* cmd/uplink: Adapt the setup subcommand for storing the user input key to a file 
  and adapt the rest of the subcommands for reading the key from the key-file when 
  the key isn't explicitly set with a command line flag.
* cmd/gateway: Adapt it to read the encryption key from the key-file or use the 
  one passed by a command line flag.
* pkg/process: Export the default configuration filename so other packages which 
  use the same value can reference to it rather than having it hardcoded.
* Adapt several integrations (scripts, etc.) to consider the changes applied in uplink and cmd packages.
2019-05-22 15:57:12 +02:00
Bill Thorp
a6c4019288
using DB time only (#2018)
* using DB time only, using UTC
2019-05-21 12:50:55 -04:00
Matt Robinson
4544542b43 I think this fixes the weird '-heads-' in the version tags (#2010) 2019-05-21 16:36:08 +02:00
Egon Elbre
9c23c2d427 db: set max idle connections higher to avoid redialing all the time (#1991) 2019-05-21 17:30:06 +03:00
Michal Niewrzal
c43e8daf6b
Mountable Drive design doc (#2016) 2019-05-21 16:16:31 +02:00
littleskunk
878e79dd79 Bugfix: Repair increase success counter (#2015) 2019-05-21 14:23:00 +02:00
Matt Robinson
cbb323ad52 Give the bootstrap image the business. (#2005) 2019-05-21 13:57:20 +02:00
JT Olio
32b3f8fef0 cmd/storagenode: pull more things into releaseDefaults (#1980) 2019-05-21 13:48:47 +02:00
Yingrong Zhao
4319b637bf
Design Doc for referral & free tier program (#1942)
* init doc

* Add link to offer types table

* Correct title

* Update Credits interface

* Add another question

* Add steps for configure segment.io for different satellite

* Update database design

Change database schema

* Address comments

* Add explaination on how to restrict access to the private port

* rename file

* Add credits_used column and offer update limit

* Change credits to be integer

* Remove extra blank lines
2019-05-20 20:21:32 -04:00
Bill Thorp
6522579ecb better repairer logging (#2006)
* logging and delete only repairs with no errors

* removing delete logi~c
2019-05-21 00:05:28 +02:00
3bl3gamer
46ab933363 Fixed NodeID and PieceID marshaling from JSON (#1995) 2019-05-20 19:48:09 +03:00
littleskunk
8e023b8bbf improve logging (#2004) 2019-05-20 16:18:16 +02:00
Bill Thorp
09065b8dec call GetRemotePieces once (#2003) 2019-05-20 15:22:03 +02:00
littleskunk
c974e0ce8a
Store repaired Segments and improve Repair Condition (#2000)
* repair no cutoff longtail

* commit repair pieces even if not hitting success threshold

* commit repair pieces even if not hitting success threshold

* remove useless condition

* better error message
2019-05-20 12:50:13 +02:00
Stefan Benten
020a6c7804 Add 3bl3gamer (#2001) 2019-05-20 12:23:46 +02:00
littleskunk
d2c95c1d62 improve repair logs (#1999) 2019-05-20 10:37:46 +02:00
Egon Elbre
42562429f5 Optimize KnownUnreliableOrOffline SQL query (#1968) 2019-05-19 17:10:46 +02:00
Egon Elbre
2679b9046f update dependencies for bug fixes (#1993) 2019-05-19 16:58:30 +02:00
Egon Elbre
1103fa63c0
disable flaky TestSegmentStoreRepair (#1994) 2019-05-17 23:13:37 +03:00
Bill Thorp
91721f63ba
Bt/repair no nodes (#1974)
* handle cases where repair is equal to total
2019-05-17 15:02:40 -04:00
Bill Thorp
b23afb7aaa
remove downloader from verifier (#1983)
* remove downloader from verifier
2019-05-17 14:48:32 -04:00
paul cannon
5d24af9b41 Revert "storage/postgreskv: use batch size 1000 (#1988)" (#1992)
This reverts commit f988543764.
2019-05-17 14:41:15 -04:00
Egon Elbre
f988543764 storage/postgreskv: use batch size 1000 (#1988) 2019-05-17 19:09:04 +02:00
littleskunk
910eb5d2c7 improve logging (#1987) 2019-05-17 19:02:39 +02:00
Egon Elbre
859e04f8f3 satellite/console: use require.NoError in tests (#1986) 2019-05-17 09:31:14 -04:00
Egon Elbre
748b2b05a4 satellite/mailservice: wait for goroutines to finish (#1985)
* satellite/mailservice: wait for goroutines to finish

* fix tests
2019-05-17 14:29:35 +02:00
Natalie Villasana
edd0a880ec Removes 0 offline nodes assertion from verifier test (#1982) 2019-05-17 15:24:23 +03:00
Fadila
e6c30f419e
storage node garbage collection design doc (#1943)
* Initial version

* small update

* add probability of false positive table

* put the table at the right place

* updates during meeting

* adds some updates about using bloom filters for gc deleted data

* small fixes

* adds implementation steps and basic gc service protobuf

* adds super brief abstract

* rm gendered language

* removes DataDeletionReport section, adds rationale for not using 2 bloom filters

* Remove parts about deletion

* rearrangement
2019-05-16 19:26:19 +02:00
Yaroslav Vorobiov
4463153f46
Console rollup period fix (#1833) 2019-05-16 17:38:55 +03:00
Maximillian von Briesen
cc020dfdea
Create containment mode database table and migrate scripts (#1970) 2019-05-16 10:11:15 -04:00
aligeti
60cf1dafb0
repair segment reassess it missing pieces just before repair (#1939)
* repair segment reaccess it missing pieces just before repair to see if it actually needs repair
2019-05-16 09:49:10 -04:00
Marc Schubert
0531d11434 Bd/doc text2 (#1973)
Correcting old path to the new one

uplink run is not a valid command, ls is.

the gateway does not share the config file with uplink!
2019-05-16 15:04:06 +02:00
Brandon Iglesias
46e19954ba
updating words to be accurate on GUI (#1971) 2019-05-16 08:41:56 -04:00
Ivan Fraixedes
a651cea2f9
pkg/macaroon: fix NewCaveat doc comment 2019-05-16 13:22:38 +02:00