Commit Graph

892 Commits

Author SHA1 Message Date
Maximillian von Briesen
50a0bffa0a
Improve audit cursor NextStripe (#1821) 2019-04-25 11:43:26 -04:00
Bryan White
fd9d244de7
add info log for successfull authorizations (#1804) 2019-04-25 12:47:40 +02:00
Michal Niewrzal
fe3dfc1587
Move pointerdb.Service to satellite (#1826) 2019-04-25 10:46:32 +02:00
Michal Niewrzal
a585b97363
Download is hanging when one node is not responsive (#1764)
Change closes download connection in parallel and sets a timeout for communication between uplink and storage node
2019-04-25 10:17:26 +02:00
Michal Niewrzal
dcea59205d
Uplink CLI setup welcome message (#1735) 2019-04-24 15:17:32 +02:00
Egon Elbre
ee3bf4a102 only run selection code once (#1825) 2019-04-24 06:35:50 -04:00
Egon Elbre
c284cfde30
ensure TestParallel doesn't deadlock on error (#1808) 2019-04-24 13:15:46 +03:00
Bill Thorp
cd4a3e06d8
wired up IsHealthy to config (#1820)
* wired up IsHealthy to config
2019-04-23 18:45:50 -04:00
Fadila
8ddf481b33 Checker: invalid and offline nodes search update (#1812)
* simplified invalid and offline login into getMissingPieces
2019-04-23 16:54:39 -04:00
Bill Thorp
2029c558af
updated audit configs (#1814)
updated audit configs
2019-04-23 16:47:11 -04:00
Cameron
9f6b010748 rollup interval fixes (#1739)
* set all intervals to UTC in rollupStats map, only delete latest day after both rollups

* clean up usage of interval, use intervalEndTime rather than createdAt

* change some variable names, add comments

* add flag for tally deletion

* adds deletetallies flag to testplanet

* space

* Removes println:

* adds test for deletes false
2019-04-23 15:21:30 -04:00
Kaloyan Raev
da0bc5adb5
Selecting non-reputable nodes should not result in excessive selection (#1680) 2019-04-23 18:23:51 +03:00
Egon Elbre
f7ed63a119
handle database error checks properly (#1796) 2019-04-23 14:13:57 +03:00
Egon Elbre
bdd0d778eb
interface tests belong to the interface, not the implementation (#1794) 2019-04-23 11:47:16 +03:00
Jennifer Li Johnson
253cbe5fa0
updates rollup default interval (#1800) 2019-04-22 15:18:16 -04:00
Egon Elbre
05732416dc disable other overlay services that might update online in graveyard test (#1798) 2019-04-22 11:22:52 -04:00
Egon Elbre
f49b0acc5b repair until queue is empty (#1716)
* repair until queue is empty
2019-04-22 11:16:21 -04:00
Egon Elbre
274fd53ace
Skip verifier tests early (#1793) 2019-04-22 15:14:59 +03:00
Bryan White
b6d0157b5b expose identity version flag on relevant identity cli commands (#1776) 2019-04-22 13:58:57 +03:00
Natalie Villasana
8d1f614662 removes unused queue code, moves queue_test.go to repairqueue_test.go in satellitedb dir (#1783) 2019-04-22 13:35:52 +03:00
Kaloyan Raev
8fc5fe1d6f
Refactor pb.Node protobuf (#1785) 2019-04-22 12:07:50 +03:00
Fadila
123bf291f2 Inspector draw routing table (#1732) 2019-04-22 11:34:11 +03:00
Maximillian von Briesen
b29d875326 Enable TestGetShareTimeout (#1774) 2019-04-22 08:54:08 +03:00
JT Olio
2744a26b60
pkg/cfgstruct: tie defaults to releases (#1787)
* tie defaults to releases

this change makes it so that by default, the flag defaults are
chosen based on whether the build was built as a release build or
an ordinary build. release builds by default get release defaults,
whereas ordinary builds by default get dev defaults.

any binary can have its defaults changed by specifying

 --defaults=dev

or

 --defaults=release

Change-Id: I6d216aa345d211c69ad913159d492fac77b12c64

* make release defaults more clear

this change extends cfgstruct structs to support either
a 'default' tag, or a pair of 'devDefault' and 'releaseDefault'
tags, but not both, for added clarity

Change-Id: Ia098be1fa84b932fdfe90a4a4d027ffb95e249c6

* clarify cfgstruct.DefaultsFlag

Change-Id: I55f2ff9080ebbc0ce83abf956e085242a92f883e
2019-04-19 12:17:30 -06:00
JT Olio
2f9aa7e7be pkg/process: add --debug.trace-out option to binaries (#1788)
this adds a flag to all pkg/process binaries where if set,
will write a process-level trace for all monkit-instrumented
functions as an svg to the target path.

usage like:

  uplink cp --debug.trace-out=trace.svg file sj://bucket/file

Change-Id: I8f6bbfc488f0f1ac270cb28a61347c6b9698cea7
2019-04-19 11:34:29 -04:00
Philip Hutchins
a840948671
Moving failure for putting pieces to Debug as this is expected behavior (#1777) 2019-04-18 08:35:44 -04:00
Egon Elbre
c5b6027d0e
move collect to a more appropriate package (#1771) 2019-04-18 12:04:42 +03:00
Bill Thorp
9dc4e82437
removed commented code, removed unnecessary pointer (#1766)
* removed commented code
2019-04-16 15:55:28 -04:00
Bill Thorp
17a227e6e9
refactor injuredsegments db so that we can't have duplicates (#1717)
made repairqueue not use a true queue, forbid duplicates
2019-04-16 14:14:09 -04:00
JT Olio
d70f6e3760
libuplink: remove encryption key from project opening (#1761)
What: This change moves project-level bucket metadata encryption information to the volatile section, because it is unlikely to remain in future releases

Why: Ultimately, the web user interface will allow bucket management (creation, removal, etc), but not object management as that requires an encryption key for sure and we don't want to have users give the satellite their encryption keys.

At a high level, a (*Project) type should map to all of the things you can do inside the web user interface within a project, which by necessity cannot have an encryption key. So, we really don't want an encryption key in the non-volatile section of this library.
2019-04-16 11:29:33 -04:00
Kaloyan Raev
d6850f8691
Migrate Minio Gateway to libuplink (#1652) 2019-04-15 23:13:02 +03:00
Michal Niewrzal
991cbad66a
Fix download when missing piece (#1753) 2019-04-12 16:57:24 -04:00
Egon Elbre
0eee46524d update linter to v1.16 (#1741)
* update to v1.16

* use full version number

* fix linter issues

* fix comment

* nicer comment

* restart travis

* restart travis
2019-04-11 13:32:40 -04:00
paul cannon
0ae0de75bc use SerializableMeta to store bucket attributes (#1658) 2019-04-10 18:27:04 -04:00
Natalie Villasana
311fd2227a
optimizes deleting obsolete paths (#1734) 2019-04-10 11:26:12 -04:00
Bryan White
1257ce2588
remove fmt print statements (#1733) 2019-04-10 11:10:22 -04:00
Bryan White
c29dbc6e7c
remove limits from certificate auth info and export subcommands (#1709) 2019-04-10 09:44:38 -04:00
JT Olio
46ab2f0d8f tlsopts: fix helper (#1515) 2019-04-10 09:07:51 -04:00
Egon Elbre
ea8a0fd9ba
use one testplanet where we can (#1725) 2019-04-10 08:10:08 -04:00
JT Olio
27ec69eb79 add ability to inspect version to all binaries (#1685)
* add ability to inspect version to all binaries

Change-Id: I09d83b0e6ddd145dc02ae6619f842fcc7c3a0469

* add monitoring

Change-Id: I14ceca4cd7706f4c33fa67f7b34ebb8297fc76ad

* review comments

Change-Id: I255c5559be78b92cba4b23e22cf92f459bbd16b7

* switch to atomic

Change-Id: Ibdab950c59b974f64b129d783b283abe025ab0ff

* linting

Change-Id: Ia2407172e64c2cc92e78ad8b48c371e61ae67278
2019-04-10 08:38:26 +02:00
Stefan Benten
bae4c820ee
Add Version Information into KAD Network and SatelliteDB & Change Selection Process (#1648)
* Initial Webserver Draft for Version Controlling

* Rename type to avoid confusion

* Move Function Calls into Version Package

* Fix Linting and Language Typos

* Fix Linting and Spelling Mistakes

* Include Copyright

* Include Copyright

* Adjust Version-Control Server to return list of Versions

* Linting

* Improve Request Handling and Readability

* Add Configuration File Option
Add Systemd Service file

* Add Logging to File

* Smaller Changes

* Add Semantic Versioning and refuses outdated Software from Startup (#1612)

* implements internal Semantic Version library

* adds version logging + reporting to process

* Advance SemVer struct for easier handling

* Add Accepted Version Store

* Fix Function

* Restructure

* Type Conversion

* Handle Version String properly

* Add Note about array index

* Set temporary Default Version

* Add Copyright

* Adding Version to Dashboard

* Adding Version Info Log

* Renaming and adding CheckerProcess

* Iteration Sync

* Iteration V2

* linting

* made LogAndReportVersion a go routine

* Refactor to Go Routine

* Add Context to Go Routine and allow Operation if Lookup to Control Server fails

* Handle Unmarshal properly

* Linting

* Relocate Version Checks

* Relocating Version Check and specified default Version for now

* Linting Error Prevention

* Refuse Startup on outdated Version

* Add Startup Check Function

* Straighten Logging

* Dont force Shutdown if --dev flag is set

* Create full Service/Peer Structure for ControlServer

* Linting

* Straighting Naming

* Finish VersionControl Service Layout

* Improve Error Handling

* Change Listening Address

* Move Checker Function

* Remove VersionControl Peer

* Linting

* Linting

* Create VersionClient Service

* Renaming

* Add Version Client to Peer Definitions

* Linting and Renaming

* Linting

* Remove Transport Checks for now

* Move to Client Side Flag

* Remove check

* Linting

* Transport Client Version Intro

* Adding Version Client to Transport Client

* Add missing parameter

* Adding Version Check, to set Allowed = true

* Set Default to true, testing

* Restructuring Code

* Uplink Changes

* Add more proper Defaults

* Renaming of Version struct

* Dont pass Service use Pointer

* Set Defaults for Versioning Checks

* Put HTTP Server in go routine

* Add Versioncontrol to Storj-Sim

* Testplanet Fixes

* Linting

* Add Error Handling and new Server Struct

* Move Lock slightly

* Reduce Race Potentials

* Remove unnecessary files

* Linting

* Add Proper Transport Handling

* small fixes

* add fence for allowed check

* Add Startup Version Check and Service Naming

* make errormessage private

* Add Comments about VersionedClient

* Linting

* Remove Checks that refuse outgoing connections

* Remove release cmd

* Add Release Script

* Linting

* Update to use correct Values

* Change Timestamp handling

* Adding Protobuf changes back in

* Adding SatelliteDB Changes and adding Storj Node Version to PB

* Add Migration Table

* Add Default Stats for Creation

* Move to BigInt

* Proper SQL Migration

* Ensure minimum Version is passed to the node selection

* Linting...

* Remove VersionedClient and adjust smaller changes from prior merge

* Linting

* Fix PB Message Handling and Query for Node Selection

* some future-proofing type changes

Change-Id: I3cb5018dcccdbc9739fe004d859065992720caaf

* fix a compiler error

Change-Id: If66bb92d8b98e31cd618ecec9c6448ab9b037fa5

* Comment on Constant for Overlay

* Remove NOT NULL and add epoch call as function

* add versions to bootstrap and satellites

Change-Id: I436944589ea5f21600cdd997742a84fe0b16e47b

* Change Update Migration

* Fix DB Migration

* Increase Timeout temporarily, to see whats going on

* Remove unnecessary const and vars
Cleanup Function calls from deprecated NodeVersion struct

* Updated Protopuf, removed depcreated Code from Inspector

* Implement NodeVersion into InfoResponse

* Regenerated locked.go

* Linting

* Fix Tests

* Remove unnecessary constant

* Update Function and Flag Description

* Remove Empty Stat Creation

* return properly with error

* Remove unnecessary struct

* simplify migration step

* Update Inspector to return Version Info

* Update local Endpoint Version Handling

* Reset Travis Timeout

* Add Default for CommitHash

* single quotes
2019-04-10 08:04:24 +02:00
JT Olio
ffdb2e7728
actually skip the data repair test (#1728)
Change-Id: I76286fc6cc5129d8be50d45a684a3e0dce9c0cc6
2019-04-09 23:29:05 -06:00
JT Olio
61ec92f2e8
disable datarepair test for now (#1727)
Change-Id: I1854817ea051ab621936f587b198de2da07c9960
2019-04-09 22:31:55 -06:00
Maximillian von Briesen
ed8fc126aa
add error in ec client Repair() if not enough successful nodes by end (#1722) 2019-04-09 18:46:43 -04:00
Cameron
32192aca10
convert times to UTC before entering as utimestamp (#1708) 2019-04-09 15:12:58 -04:00
Cameron
62deae6a0a
fix bucketID bug in repair (#1719) 2019-04-09 13:20:00 -04:00
Bryan White
08b8d84248
Identity versioning fix (#1721) 2019-04-09 13:01:45 -04:00
Jess G
55bfe507e5
add tests for tally and accounting db (#1675)
* add tally calcAtRestData test

* add account db tests

* move tests from sadb to accounting pkg

* add func to calcEncSize

* remove unneeded pad changes

* update comment

* add changes to comments
2019-04-09 09:48:35 -04:00
Jess G
5dfe28a8c3
use allocated instead of settled (#1700)
* use allocated instead of settled

* add expansion factor

* changes per CR
2019-04-08 14:35:54 -07:00
Maximillian von Briesen
3fb4813227
Fix data repair checker missing pieces list (#1705) 2019-04-08 15:46:23 -04:00