Commit Graph

1854 Commits

Author SHA1 Message Date
JT Olio
24787adb5b
cmd/statreceiver: allow for packet filtering to packet destinations (#2019)
What: allow packetfilter to work on packet destinations instead of only on metric destinations. this will allow us to filter what applications get sent to rothko.

Why: currently rothko is drowning in storj-sim data and it'd be nice to filter out.
2019-05-22 16:19:32 -06:00
Yingrong Zhao
f9045f8385 Create offer table in satellitedb for free credit program (#1975) 2019-05-22 23:41:55 +02:00
Matt Robinson
5d5d82d2f7 Show all of the claims (#2009) 2019-05-22 23:31:32 +02:00
Maximillian von Briesen
45a2253628 Send durability stats after iterating over all segments (#2028) 2019-05-22 17:17:52 -04:00
Cameron
4058c29ca4
filter duplicate node IPs (#1890)
* add last_ip field to dbx model node, generate dbx

* add last_ip to node proto, generate pb

* migrate

* resolve address in transport.DialNode, update lastIp in cache.UpdateAddress

* use net.SplitHostPort to isolate host address from port

* define DistinctIPs flag

* add test for GetIP

* select last_ip when querying for nodes

* if distinctIPs flag == true, query for nodes with distinct IPs

* some basic tests

* change last_ip to field 14 in proto

* remove comments

* check err

* change distinctIPs to distinctIP

* exclude IPs from newNodes in query for reputable nodes

* add index on last_ip

* only add to excludedIPs if flag is true

* test half new nodes returns distinct IPs

* fix alignment

* add test

* rework ip filter query, add retry logic, add switch for database driver

* add retry to SelectNewNodes

* change discovery intervals so IPs don't get overwritten

* remove TestGetIP

* edit updating node stats in test

* split exclude into nodeIDs and IPs

* separate non-distinct IP query into other function

* trigger checks

* remove else block
2019-05-22 16:06:27 -04:00
Jeff Wendling
b3e6236142
add uplink-contexts design doc (#1959)
* add uplink-contexts design doc

* respond to some feedback and rename context to scope

* feedback summaries

* review fixes

* review feedback

* more review feedback

* fix types in go structs
2019-05-22 19:49:46 +00:00
Cameron
a883f87dc3
define vouchers proto (#2026) 2019-05-22 15:15:59 -04:00
Natalie Villasana
acfb59ae48
adds db layer functions for containment (#1972) 2019-05-22 10:50:22 -04:00
Ivan Fraixedes
69d8b9f828
Change where the encryption key is being stored for uplink (#1967)
* uplink: Add a new flag to set the filepath of the file which is used for 
  saving the encryption key and rename the one that hold the encryption key and 
  establish that it has priority over the key stored in the file to make the 
  configuration usable without having a huge refactoring in test-sim.
* cmd/uplink: Adapt the setup subcommand for storing the user input key to a file 
  and adapt the rest of the subcommands for reading the key from the key-file when 
  the key isn't explicitly set with a command line flag.
* cmd/gateway: Adapt it to read the encryption key from the key-file or use the 
  one passed by a command line flag.
* pkg/process: Export the default configuration filename so other packages which 
  use the same value can reference to it rather than having it hardcoded.
* Adapt several integrations (scripts, etc.) to consider the changes applied in uplink and cmd packages.
2019-05-22 15:57:12 +02:00
Bill Thorp
a6c4019288
using DB time only (#2018)
* using DB time only, using UTC
2019-05-21 12:50:55 -04:00
Matt Robinson
4544542b43 I think this fixes the weird '-heads-' in the version tags (#2010) 2019-05-21 16:36:08 +02:00
Egon Elbre
9c23c2d427 db: set max idle connections higher to avoid redialing all the time (#1991) 2019-05-21 17:30:06 +03:00
Michal Niewrzal
c43e8daf6b
Mountable Drive design doc (#2016) 2019-05-21 16:16:31 +02:00
littleskunk
878e79dd79 Bugfix: Repair increase success counter (#2015) 2019-05-21 14:23:00 +02:00
Matt Robinson
cbb323ad52 Give the bootstrap image the business. (#2005) 2019-05-21 13:57:20 +02:00
JT Olio
32b3f8fef0 cmd/storagenode: pull more things into releaseDefaults (#1980) 2019-05-21 13:48:47 +02:00
Yingrong Zhao
4319b637bf
Design Doc for referral & free tier program (#1942)
* init doc

* Add link to offer types table

* Correct title

* Update Credits interface

* Add another question

* Add steps for configure segment.io for different satellite

* Update database design

Change database schema

* Address comments

* Add explaination on how to restrict access to the private port

* rename file

* Add credits_used column and offer update limit

* Change credits to be integer

* Remove extra blank lines
2019-05-20 20:21:32 -04:00
Bill Thorp
6522579ecb better repairer logging (#2006)
* logging and delete only repairs with no errors

* removing delete logi~c
2019-05-21 00:05:28 +02:00
3bl3gamer
46ab933363 Fixed NodeID and PieceID marshaling from JSON (#1995) 2019-05-20 19:48:09 +03:00
littleskunk
8e023b8bbf improve logging (#2004) 2019-05-20 16:18:16 +02:00
Bill Thorp
09065b8dec call GetRemotePieces once (#2003) 2019-05-20 15:22:03 +02:00
littleskunk
c974e0ce8a
Store repaired Segments and improve Repair Condition (#2000)
* repair no cutoff longtail

* commit repair pieces even if not hitting success threshold

* commit repair pieces even if not hitting success threshold

* remove useless condition

* better error message
2019-05-20 12:50:13 +02:00
Stefan Benten
020a6c7804 Add 3bl3gamer (#2001) 2019-05-20 12:23:46 +02:00
littleskunk
d2c95c1d62 improve repair logs (#1999) 2019-05-20 10:37:46 +02:00
Egon Elbre
42562429f5 Optimize KnownUnreliableOrOffline SQL query (#1968) 2019-05-19 17:10:46 +02:00
Egon Elbre
2679b9046f update dependencies for bug fixes (#1993) 2019-05-19 16:58:30 +02:00
Egon Elbre
1103fa63c0
disable flaky TestSegmentStoreRepair (#1994) 2019-05-17 23:13:37 +03:00
Bill Thorp
91721f63ba
Bt/repair no nodes (#1974)
* handle cases where repair is equal to total
2019-05-17 15:02:40 -04:00
Bill Thorp
b23afb7aaa
remove downloader from verifier (#1983)
* remove downloader from verifier
2019-05-17 14:48:32 -04:00
paul cannon
5d24af9b41 Revert "storage/postgreskv: use batch size 1000 (#1988)" (#1992)
This reverts commit f988543764.
2019-05-17 14:41:15 -04:00
Egon Elbre
f988543764 storage/postgreskv: use batch size 1000 (#1988) 2019-05-17 19:09:04 +02:00
littleskunk
910eb5d2c7 improve logging (#1987) 2019-05-17 19:02:39 +02:00
Egon Elbre
859e04f8f3 satellite/console: use require.NoError in tests (#1986) 2019-05-17 09:31:14 -04:00
Egon Elbre
748b2b05a4 satellite/mailservice: wait for goroutines to finish (#1985)
* satellite/mailservice: wait for goroutines to finish

* fix tests
2019-05-17 14:29:35 +02:00
Natalie Villasana
edd0a880ec Removes 0 offline nodes assertion from verifier test (#1982) 2019-05-17 15:24:23 +03:00
Fadila
e6c30f419e
storage node garbage collection design doc (#1943)
* Initial version

* small update

* add probability of false positive table

* put the table at the right place

* updates during meeting

* adds some updates about using bloom filters for gc deleted data

* small fixes

* adds implementation steps and basic gc service protobuf

* adds super brief abstract

* rm gendered language

* removes DataDeletionReport section, adds rationale for not using 2 bloom filters

* Remove parts about deletion

* rearrangement
2019-05-16 19:26:19 +02:00
Yaroslav Vorobiov
4463153f46
Console rollup period fix (#1833) 2019-05-16 17:38:55 +03:00
Maximillian von Briesen
cc020dfdea
Create containment mode database table and migrate scripts (#1970) 2019-05-16 10:11:15 -04:00
aligeti
60cf1dafb0
repair segment reassess it missing pieces just before repair (#1939)
* repair segment reaccess it missing pieces just before repair to see if it actually needs repair
2019-05-16 09:49:10 -04:00
Marc Schubert
0531d11434 Bd/doc text2 (#1973)
Correcting old path to the new one

uplink run is not a valid command, ls is.

the gateway does not share the config file with uplink!
2019-05-16 15:04:06 +02:00
Brandon Iglesias
46e19954ba
updating words to be accurate on GUI (#1971) 2019-05-16 08:41:56 -04:00
Ivan Fraixedes
a651cea2f9
pkg/macaroon: fix NewCaveat doc comment 2019-05-16 13:22:38 +02:00
Yaroslav Vorobiov
2d2301d5ff
Console buckets page (#1847) 2019-05-16 13:43:46 +03:00
Nikolay Yurchenko
ae24778ef7
Billing markup and project page rework (#1963) 2019-05-15 18:01:41 +03:00
Kaloyan Raev
d59c00b47c
Fix download error threshold (#1966) 2019-05-15 15:49:13 +03:00
Bogdan Artemenko
38acc4dba6
Resend account activation email (#1961) 2019-05-15 11:28:36 +03:00
Natalie Villasana
2692c83cfe
adds audit system's containment mode design draft (#1929) 2019-05-14 18:22:08 -04:00
Jeff Wendling
791ec89c5e make caveat nonces random (#1964) 2019-05-14 16:08:52 -05:00
Jeff Wendling
15e74c8c3d uplink share subcommand (#1924)
* cmd/uplink: add share command to restrict an api key

This commit is an early bit of work to just implement restricting
macaroon api keys from the command line. It does not convert
api keys to be macaroons in general.

It also does not apply the path restriction caveats appropriately
yet because it does not encrypt them.

* cmd/uplink: fix path encryption for shares

It should now properly encrypt the path prefixes when adding
caveats to a macaroon.

* fix up linting problems

* print summary of caveat and require iso8601

* make clone part more clear
2019-05-14 12:15:12 -06:00
Nikolay Yurchenko
eac18352f3
headers in team and apiKeys pages aligned (#1893) 2019-05-14 18:29:11 +03:00