Commit Graph

1622 Commits

Author SHA1 Message Date
Jess G
6f23aeef18
add performance benchmarks (#2021)
* update uplink cmd args

* add uplod/download benchmark, add script to run benchmarks

* fix lint err

* changes per CR comments

* export env var for linux

* fix lint

* add byte count to bench test
2019-05-24 13:25:53 -07:00
Fadila
8f078d2841 Delete pending audit entries from containment DB when files get deleted (#2050) 2019-05-24 15:56:08 -04:00
Ivan Fraixedes
2cdc55d345 cmd/uplink: Fix a faulty test (#2054)
The test was not considering the proper length of a slice generated with
a random length.
2019-05-24 10:55:46 -07:00
Ivan Fraixedes
5c50aac661 pkg/audit: Readdress ignored Verifier tests (#2048) 2019-05-24 19:57:07 +03:00
Jeff Wendling
1bd52b9f90 server side macaroons (#1945)
What: Adds macaroon support to the server side

Why: So that api keys are now macaroons
2019-05-24 10:51:27 -06:00
Dennis Coyle
b15b6edc58
Add ethanadams to CLA Bot (#2052) 2019-05-24 10:56:58 -04:00
aligeti
ac2dc8684e
added warning info when total BW < used BW (#2042) 2019-05-24 09:53:29 -04:00
Michal Niewrzal
7193b16e92
Java/Android libuplink bindings (#1918) 2019-05-24 10:13:01 +02:00
littleskunk
73a039afa7
named log (#2034) 2019-05-24 00:47:05 +02:00
Natalie Villasana
e862382a6f rename audit.RecordAuditsInfo to audit.Report (#2044)
* rename RecordAuditsInfo to Report

* change comment
2019-05-24 00:32:19 +02:00
Kaloyan Raev
de8070730a Save hash of expected share from missing audited node (#2025) 2019-05-23 16:07:19 -04:00
Jeff Wendling
b0ca12e31e
log errors only when an error happens (#2029) 2019-05-23 17:20:01 +00:00
Jeff Wendling
47a2f5e8b6
Password based key derivation design (#2020)
* initial password key derivation design

* some typos/clarifications

* point out that clients can ignore bucket entropy

* clarification about offline table generation

* change proposal to only do network requests when inputting passwords

* make algorithm have shorter lines

* fix algorithm example

* be explicit about allowing already-formed root keys

* go back to having a salt in the project/bucket metadata

* some proofreading/clarifications

* feedback and clarifications

* clarify definition of root key
2019-05-23 17:05:16 +00:00
Maximillian von Briesen
da91d22376 properly check last iteration of checker (#2040) 2019-05-23 18:14:08 +02:00
Ivan Fraixedes
69cf6caa62
cmd/uplink: Fix error on interactive setup when --config-dir flag is used. (#2033)
Uplink CLI was returning an error when the `--config-dir` flag was
used with interactive `setup` command because the directory used for
storing the encryption key file was the default configuration directory
rather than the value set in the flag.

This changes make the setup process to use the same directory, than the
one informed by the `--config-dir` flag value, for storing encryption
key file.

On the other hand, the default value for the encryption key file path
which was set with the purpose of showing it in the help message has
been removed because the cfgstruct doesn't show it unless that the flag
is set to the Cobra root CMD, so it was useless.

Also show the absolute path where the encryption key file has been saved when
the interactive setup command ends satisfactorily as it's equally done
with the configuration file.
2019-05-23 16:49:28 +02:00
Natalie Villasana
8b31c4b91f
remove unnecessary containmentDB wrapper (#2027) 2019-05-23 10:37:23 -04:00
Natalie Villasana
ef4849b53b
skip flaky TestProjectUsageBandwidth (#2035) 2019-05-23 10:27:57 -04:00
Maximillian von Briesen
b4f18226db
Send number of files as part of durability stats (#2030) 2019-05-22 18:50:43 -04:00
JT Olio
24787adb5b
cmd/statreceiver: allow for packet filtering to packet destinations (#2019)
What: allow packetfilter to work on packet destinations instead of only on metric destinations. this will allow us to filter what applications get sent to rothko.

Why: currently rothko is drowning in storj-sim data and it'd be nice to filter out.
2019-05-22 16:19:32 -06:00
Yingrong Zhao
f9045f8385 Create offer table in satellitedb for free credit program (#1975) 2019-05-22 23:41:55 +02:00
Matt Robinson
5d5d82d2f7 Show all of the claims (#2009) 2019-05-22 23:31:32 +02:00
Maximillian von Briesen
45a2253628 Send durability stats after iterating over all segments (#2028) 2019-05-22 17:17:52 -04:00
Cameron
4058c29ca4
filter duplicate node IPs (#1890)
* add last_ip field to dbx model node, generate dbx

* add last_ip to node proto, generate pb

* migrate

* resolve address in transport.DialNode, update lastIp in cache.UpdateAddress

* use net.SplitHostPort to isolate host address from port

* define DistinctIPs flag

* add test for GetIP

* select last_ip when querying for nodes

* if distinctIPs flag == true, query for nodes with distinct IPs

* some basic tests

* change last_ip to field 14 in proto

* remove comments

* check err

* change distinctIPs to distinctIP

* exclude IPs from newNodes in query for reputable nodes

* add index on last_ip

* only add to excludedIPs if flag is true

* test half new nodes returns distinct IPs

* fix alignment

* add test

* rework ip filter query, add retry logic, add switch for database driver

* add retry to SelectNewNodes

* change discovery intervals so IPs don't get overwritten

* remove TestGetIP

* edit updating node stats in test

* split exclude into nodeIDs and IPs

* separate non-distinct IP query into other function

* trigger checks

* remove else block
2019-05-22 16:06:27 -04:00
Jeff Wendling
b3e6236142
add uplink-contexts design doc (#1959)
* add uplink-contexts design doc

* respond to some feedback and rename context to scope

* feedback summaries

* review fixes

* review feedback

* more review feedback

* fix types in go structs
2019-05-22 19:49:46 +00:00
Cameron
a883f87dc3
define vouchers proto (#2026) 2019-05-22 15:15:59 -04:00
Natalie Villasana
acfb59ae48
adds db layer functions for containment (#1972) 2019-05-22 10:50:22 -04:00
Ivan Fraixedes
69d8b9f828
Change where the encryption key is being stored for uplink (#1967)
* uplink: Add a new flag to set the filepath of the file which is used for 
  saving the encryption key and rename the one that hold the encryption key and 
  establish that it has priority over the key stored in the file to make the 
  configuration usable without having a huge refactoring in test-sim.
* cmd/uplink: Adapt the setup subcommand for storing the user input key to a file 
  and adapt the rest of the subcommands for reading the key from the key-file when 
  the key isn't explicitly set with a command line flag.
* cmd/gateway: Adapt it to read the encryption key from the key-file or use the 
  one passed by a command line flag.
* pkg/process: Export the default configuration filename so other packages which 
  use the same value can reference to it rather than having it hardcoded.
* Adapt several integrations (scripts, etc.) to consider the changes applied in uplink and cmd packages.
2019-05-22 15:57:12 +02:00
Bill Thorp
a6c4019288
using DB time only (#2018)
* using DB time only, using UTC
2019-05-21 12:50:55 -04:00
Matt Robinson
4544542b43 I think this fixes the weird '-heads-' in the version tags (#2010) 2019-05-21 16:36:08 +02:00
Egon Elbre
9c23c2d427 db: set max idle connections higher to avoid redialing all the time (#1991) 2019-05-21 17:30:06 +03:00
Michal Niewrzal
c43e8daf6b
Mountable Drive design doc (#2016) 2019-05-21 16:16:31 +02:00
littleskunk
878e79dd79 Bugfix: Repair increase success counter (#2015) 2019-05-21 14:23:00 +02:00
Matt Robinson
cbb323ad52 Give the bootstrap image the business. (#2005) 2019-05-21 13:57:20 +02:00
JT Olio
32b3f8fef0 cmd/storagenode: pull more things into releaseDefaults (#1980) 2019-05-21 13:48:47 +02:00
Yingrong Zhao
4319b637bf
Design Doc for referral & free tier program (#1942)
* init doc

* Add link to offer types table

* Correct title

* Update Credits interface

* Add another question

* Add steps for configure segment.io for different satellite

* Update database design

Change database schema

* Address comments

* Add explaination on how to restrict access to the private port

* rename file

* Add credits_used column and offer update limit

* Change credits to be integer

* Remove extra blank lines
2019-05-20 20:21:32 -04:00
Bill Thorp
6522579ecb better repairer logging (#2006)
* logging and delete only repairs with no errors

* removing delete logi~c
2019-05-21 00:05:28 +02:00
3bl3gamer
46ab933363 Fixed NodeID and PieceID marshaling from JSON (#1995) 2019-05-20 19:48:09 +03:00
littleskunk
8e023b8bbf improve logging (#2004) 2019-05-20 16:18:16 +02:00
Bill Thorp
09065b8dec call GetRemotePieces once (#2003) 2019-05-20 15:22:03 +02:00
littleskunk
c974e0ce8a
Store repaired Segments and improve Repair Condition (#2000)
* repair no cutoff longtail

* commit repair pieces even if not hitting success threshold

* commit repair pieces even if not hitting success threshold

* remove useless condition

* better error message
2019-05-20 12:50:13 +02:00
Stefan Benten
020a6c7804 Add 3bl3gamer (#2001) 2019-05-20 12:23:46 +02:00
littleskunk
d2c95c1d62 improve repair logs (#1999) 2019-05-20 10:37:46 +02:00
Egon Elbre
42562429f5 Optimize KnownUnreliableOrOffline SQL query (#1968) 2019-05-19 17:10:46 +02:00
Egon Elbre
2679b9046f update dependencies for bug fixes (#1993) 2019-05-19 16:58:30 +02:00
Egon Elbre
1103fa63c0
disable flaky TestSegmentStoreRepair (#1994) 2019-05-17 23:13:37 +03:00
Bill Thorp
91721f63ba
Bt/repair no nodes (#1974)
* handle cases where repair is equal to total
2019-05-17 15:02:40 -04:00
Bill Thorp
b23afb7aaa
remove downloader from verifier (#1983)
* remove downloader from verifier
2019-05-17 14:48:32 -04:00
paul cannon
5d24af9b41 Revert "storage/postgreskv: use batch size 1000 (#1988)" (#1992)
This reverts commit f988543764.
2019-05-17 14:41:15 -04:00
Egon Elbre
f988543764 storage/postgreskv: use batch size 1000 (#1988) 2019-05-17 19:09:04 +02:00
littleskunk
910eb5d2c7 improve logging (#1987) 2019-05-17 19:02:39 +02:00