Commit Graph

219 Commits

Author SHA1 Message Date
Michal Niewrzal
da2eaa7085
scripts: dev script to start postgres before tests (#3344) 2019-10-27 12:02:10 -07:00
Yingrong Zhao
fa1ac24e19
satellite/gracefulexit: add failure threshold check (#3329)
* add overall failure percentage check and inactive time frame check before sending a response to sno

* update comment

* delete node from transfer queue if it has been inactive for too long

* fix linting error

* add test config value

* fix nil pointer

* add config value into testplanet

* add unit test for overall failure threshold

* move timeframe threshold to chore

* update protolock

* add chore test

* add per peiece failure count logic

* change config name from EndpointMaxFailures to MaxFailuresPerPiece

* address comments

* fix linting error

* add error handling for no row returned from progress table

* fix test for graceful exit chore on storagenode

* fix typo InActive -> Inactive

* improve readability for failure threshold calculation

* update config lock

* change error handling for GetProgress in graceful exit endpoint on the satellite side

* return proper rpc error in endpoint

* add check in chore test for checking finish timestamp and queue
2019-10-24 12:24:42 -04:00
Egon Elbre
3c438f31bd
satellite/satellitedb: remove sqlite support (#3296) 2019-10-19 00:27:57 +03:00
littleskunk
2a5526fcc4
satellite/repair: reduce timeout (#3302) 2019-10-18 13:43:24 +02:00
Natalie Villasana
855fca003d satellite/metrics: create a metrics chore (#3263)
* add metrics counter and chore

* updates metrics observer interval release default and dev default to 15min

* add more specific check for remote pointers

* add Counter field to metrics chore, add counter tests

* rm redundant ObjectCount suffix

* make pointer check easier to read

* change metrics.Config.Interval to ChoreInterval

* rm unneeded var

* fix comment

* update satellite config lock
2019-10-16 14:08:33 -04:00
Cameron
76ad83f12c
satellite/accounting: add redis support to live accounting (#3213)
* set up redis support in live accounting

* move live.Service interface into accounting package and rename to Cache, pass into satellite

* refactor Cache to store one int64 total, add IncrBy method to redis client implementation

* add monkit tracing to live accounting
2019-10-16 12:50:29 -04:00
Natalie Villasana
cf430d2d73
scripts: add check-monitoring script to detect changes to monkit calls (#3114) 2019-10-15 13:00:14 -04:00
Jennifer Li Johnson
b185dbbee2
satellite/discovery: remove discovery related code (#3175) 2019-10-14 10:57:01 -04:00
littleskunk
96aeedcdee
OrderLimit/GracePeriod: Increase time window from 1h to 24h (#3255)
* OrderLimit/GracePeriod: Increase time window from 1h to 24h

* update satellite config lock
2019-10-13 17:40:24 +02:00
Ethan Adams
a1275746b4
satellite/gracefulexit: Implement the 'process' endpoint on the satellite (#3223) 2019-10-11 17:18:05 -04:00
Isaac Hess
9256399872
CI: test drpc and grpc (#3163)
* wip: test drpc

* Add parallel intregration test

* Add jenkinsfile.drpc

* Remove unnecessary jenkinsfile items

* testing: GOFLAGS=-drpc (#3236)

* Use GOFLAGS

* add debug

* revert tags

* revert changes

* move goflags to the correct place

* add sanity check
2019-10-11 08:30:06 -06:00
Ethan Adams
4c4519f0be
satellite/gracefulexit: add transfer queue for pieces (#3174)
initial impl of transfer queue
updated docs represent the new design how we handle durability during exit
2019-10-07 16:38:05 -04:00
Jeff Wendling
c1fbfea7fa drpc: bump to latest version
Change-Id: I8426c2dd7f6263050c746c2724524ff687c7298a
2019-10-04 15:09:10 -06:00
Jennifer Li Johnson
7ceaabb18e
Delete Bootstrap and Kademlia (#2974) 2019-10-04 16:48:41 -04:00
Stefan Benten
1db4251234 Satellite/repair: Add Repair Threshold Override to allow earlier repair (#3151) 2019-10-02 14:58:37 +02:00
Egon Elbre
ef5e0dce20
scripts: ignore .build directory for size checks (#3153) 2019-10-02 15:22:35 +03:00
Maximillian von Briesen
08ed50bcaa
satellite/metainfo: add commit interval to prevent long delays between order limit creation and segment commit (#3149) 2019-10-01 12:55:02 -04:00
Bogdan Artemenko
423d35fb3f
satellite/console: Added support URLs and other fields to config file (#3090) 2019-09-27 10:48:53 -06:00
Matt Robinson
02f68d68d6 Put -s and -w in the right spot (#3135) 2019-09-27 17:38:02 +02:00
Bryan White
c8aa821ccb
pkg/certificates: move certificate package to root (#3107) 2019-09-26 09:11:05 -07:00
Stefan Benten
c71f3a3f4a internal/version: Change default endpoint to query (#3126)
* change default domain name

change default domain name to point to the new version control

* Update satellite-config.yaml.lock
2019-09-25 22:55:38 +02:00
Bryan White
515799267f fix certificates auth export command (#3110)
* fix certificates auth export command

* actually fix command config logic

* fix test-certificates.sh

* simplify
2019-09-24 10:38:18 -06:00
Jennifer Li Johnson
d2502bb51b Adds tests for kad replacement and restores kad operator configs (#3094)
* test that all nodes can check in with all satellites

* keep kademlia config

* add untrusted satellite test

* use getversion

* remove kademlia config changes in test-sim-backwards.sh

* add kademlia flags back to storj-sim storagenode

* reset kademlia flags in storagenode entrypoint
2019-09-20 16:02:23 -04:00
Jennifer Li Johnson
724bb44723
Remove Kademlia dependencies from Satellite and Storagenode (#2966)
What:

cmd/inspector/main.go: removes kad commands
internal/testplanet/planet.go: Waits for contact chore to finish
satellite/contact/nodesservice.go: creates an empty nodes service implementation
satellite/contact/service.go: implements Local and FetchInfo methods & adds external address config value
satellite/discovery/service.go: replaces kad.FetchInfo with contact.FetchInfo in Refresh() & removes Discover()
satellite/peer.go: sets up contact service and endpoints
storagenode/console/service.go: replaces nodeID with contact.Local()
storagenode/contact/chore.go: replaces routing table with contact service
storagenode/contact/nodesservice.go: creates empty implementation for ping and request info nodes service & implements RequestInfo method
storagenode/contact/service.go: creates a service to return the local node and update its own capacity
storagenode/monitor/monitor.go: uses contact service in place of routing table
storagenode/operator.go: moves operatorconfig from kad into its own setup
storagenode/peer.go: sets up contact service, chore, pingstats and endpoints
satellite/overlay/config.go: changes NodeSelectionConfig.OnlineWindow default to 4hr to allow for accurate repair selection
Removes kademlia setups in:

cmd/storagenode/main.go
cmd/storj-sim/network.go
internal/testplane/planet.go
internal/testplanet/satellite.go
internal/testplanet/storagenode.go
satellite/peer.go
scripts/test-sim-backwards.sh
scripts/testdata/satellite-config.yaml.lock
storagenode/inspector/inspector.go
storagenode/peer.go
storagenode/storagenodedb/database.go
Why: Replacing Kademlia

Please describe the tests:
• internal/testplanet/planet_test.go:

TestBasic: assert that the storagenode can check in with the satellite without any errors
TestContact: test that all nodes get inserted into both satellites' overlay cache during testplanet setup
• satellite/contact/contact_test.go:

TestFetchInfo: Tests that the FetchInfo method returns the correct info
• storagenode/contact/contact_test.go:

TestNodeInfoUpdated: tests that the contact chore updates the node information
TestRequestInfoEndpoint: tests that the Request info endpoint returns the correct info
Please describe the performance impact: Node discovery should be at least slightly more performant since each node connects directly to each satellite and no longer needs to wait for bootstrapping. It probably won't be faster in real time on start up since each node waits a random amount of time (less than 1 hr) to initialize its first connection (jitter).
2019-09-19 15:56:34 -04:00
Michal Niewrzal
1c72e80e40 uplink/satellite: fix for case when inline segment is last one (#3062)
* uplink/satellite: fix when inline seg is last one

* review comments
2019-09-19 01:18:14 +02:00
Jennifer Li Johnson
ce3203e910
update NodeSelectionConfig.OnlineWindow to 4hr default (#3082) 2019-09-18 14:57:57 -04:00
Maximillian von Briesen
684b07b2c1
scripts/protobuf.go: update drpc version for protobuf generation (#3059) 2019-09-17 13:14:38 -04:00
Andrew Harding
f550ab5d1c
Uplink "import" command (#2981)
* uplink import cmd

* pkg/process: fix import order

* fix golangci-lint failures

* remove "help" from the satellite config lock file
2019-09-13 12:33:30 -06:00
Ethan Adams
731016cd85
Increase file size limit to 650 KB (#3034) 2019-09-12 13:54:44 -04:00
Ivan Fraixedes
cc8a47324a
scripts: Fix warn message update sat config lock (#3029)
Fix the warning message to indicate the Slack channel where the
satellite configuration changes must be posted.
2019-09-12 18:10:22 +02:00
Kaloyan Raev
208327835f
Script for deploying the Docker manifest for watchtower (#3015) 2019-09-12 17:38:48 +03:00
Natalie Villasana
aa3567187e
satellite/audit: worker now verifies and reverifies (#2965) 2019-09-11 18:37:01 -04:00
Bryan White
6c80f01bf0
pkg/certificates: add authorization endpoint and refactor (#2971) 2019-09-11 10:36:44 +02:00
Egon Elbre
7589ca796f
cmd/storj-sim: allow overriding executables (#2976)
The backward compatibility test upgrades only half of the storage nodes and
tests with both the release and new uplink.
2019-09-09 22:13:38 +03:00
Jeff Wendling
60eba990eb use-drpc: use protoc-gen-drpc to generate protobufs
Change-Id: I5c23256068e30864022dba5137c499796ab9d6ad
2019-09-06 13:28:27 -06:00
Natalie Villasana
6d363fb756
satellite/audit: create the audit queue, chore, and worker (#2888) 2019-09-05 11:40:52 -04:00
Bryan White
1fc0c63a1d
{cmd,pkg}/certificates: service refactor (#2938) 2019-09-05 17:11:21 +02:00
Egon Elbre
c4ee7eb51c
jenkins: build and test web/satellite (#2908) 2019-09-05 16:32:46 +03:00
Bryan White
62df8ddb0c cmd/certificates: auth export improvements & certificates test script (#2897) 2019-09-04 19:58:38 +02:00
Cameron
af5fb8e9c5
satellite/vouchers: deprecate voucher endpoint, return 'please upgrade' error (#2940)
* voucher endpoint returns 'please upgrade' error, test
2019-09-04 13:21:02 -04:00
littleskunk
05e9916b9c
storagenode/buildscript: Overwrite existing docker manifest file (#2934) 2019-09-04 18:55:03 +02:00
paul cannon
bf5a16b823 jenkins: work around git.apache.org failures (#2944) 2019-09-04 11:30:59 +03:00
Yingrong Zhao
10a896bf73
web/marketing: static asset path (#2872)
* use relative path instead of absolute path

* add template func baseURL

* add a method

* update storj-sim

* add comment
2019-08-30 18:43:53 -04:00
Egon Elbre
66ec727a37 jenkins: backwards-compatibility, don't overwrite installed binaries (#2892) 2019-08-28 09:57:34 -04:00
Matt Robinson
1b2a2d045f
[build]: Adjust arm63v6 and aarch64 images to match convention (#2845)
* Adjust arm32v6 and aarch64 docker images to match the hello-world image

* Update from master, fix potential bug in push-images target, and update storagenode deploy to handle arm64 image
2019-08-28 08:18:56 -04:00
Matt Robinson
f404aad878
Add deploy script for storagenodes (#2889) 2019-08-27 16:27:34 -04:00
Cameron
599324c364
satellite/dbcleanup: delete expired serials from satellite (#2867)
Creates a new chore, dbcleanup, which can be used for routine deletion of items from the satellite database and adds functionality for deletion of expired serial numbers
2019-08-27 13:12:38 -04:00
Egon Elbre
c309bd3fec
lint: add linting for errs package (#2881) 2019-08-27 19:07:12 +03:00
Egon Elbre
00b2e1a7d7 all: enable staticcheck (#2849)
* by having megacheck in disable it also disabled staticcheck

* fix closing body

* keep interfacer disabled

* hide bodies

* don't use deprecated func

* fix dead code

* fix potential overrun

* keep stylecheck disabled

* don't pass nil as context

* fix infinite recursion

* remove extraneous return

* fix data race

* use correct func

* ignore unused var

* remove unused consts
2019-08-22 13:40:15 +02:00
Natalie Villasana
243cedb628
satellite/audit: implement reservoir struct and RemoteSegment observer method (#2744) 2019-08-21 11:49:27 -04:00