Commit Graph

25 Commits

Author SHA1 Message Date
Michał Niewrzał
ac058e5ecc metainfo-migration: basic pointerdb->metabase migrator
Change-Id: If183b9c99ce816521fcae6e7189a6f85ddd4eb48
2021-01-12 12:59:53 +00:00
Egon Elbre
1726b39ed2 scripts: remove thrift mod replace
Currently our code is only using github version of the code, so there
shouldn't be need for the exception.

Change-Id: I0c6e8a8465ab7b525d4b5d1b29e4e5298384286d
2020-11-16 12:00:09 +02:00
Michal Niewrzal
2be93df566 scripts: reduce file size for backward compatibility tests
Change-Id: Ibcf460f908b6e551fe0ca68248affb7ddf49592f
2020-05-27 13:21:55 +00:00
Jeff Wendling
bb28851964 test-sim-backwards: refactor, reduce node count to 9 in mixed mode
This catches the bug that hit prod where we missed that a migration
was necessary for a column, but only on uploads, and only for nodes
that had not checked in yet.

So, this does more uploads, and stops a node from starting so that
we trigger that specific scenario. Hopefully it will catch other
similar ones in the future.

I confirmed that this locally caught the bug when the release under
test was v0.34.10 and HEAD did not include the fix for it.

Change-Id: If7d41e8241d6a042fa524b4aff956b0264ecb128
2020-05-01 04:56:06 +00:00
Egon Elbre
90319dbec1 scripts: fix test-sim-backwards
Change-Id: I9d6644d4b493f5f6f60a960cacfaac2b5b828a5f
2020-04-02 00:13:42 +02:00
JT Olio
3b66ba6f02 scripts/tests: uplink no longer respects --client.segment-size
this is going to make all the tests slower but it is what it is

test-sim-aws.sh is removed because it was moved to storj/gateway repo.

Change-Id: I10727e747a4c3740b1c9054ce7d17313b4fa310b
2020-03-22 17:48:57 +00:00
Matt Robinson
bd4982e249
test/backwards-compatibility: Exclude rc tags from testing (#3787)
Change-Id: Id486709306c5c75a262ea17410ae641be53df8ed

Co-authored-by: Ivan Fraixedes <ivan@fraixed.es>
Co-authored-by: littleskunk <jens.heimbuerge@googlemail.com>
2020-03-18 00:12:05 +01:00
Michal Niewrzal
54e38b8986 pkg/miniogw: gateway implementation with new libuplink
Change-Id: I170c3a68cfeea33b528eeb27e6aecb126ecb0365
2020-02-21 16:20:38 +01:00
Egon Elbre
dbf46c4aa7 satellite/admin: administrative endpoint
Admin server allows creating basic REST and html API-s
for different administrative tasks.

Change-Id: I3dc1786abe1c87350eed60ec90e48130f44e63cf
2020-02-12 12:12:50 +02:00
Andrew Harding
cb89496569 storagenode/trust: wire up list into pool
- also updated ping chore to pick up trust changes
- fixed small typo in blueprint
- fixed flags for storj-sim
- wired up changes to testplanet

Change-Id: I02982f3a63a1b4150b82a009ee126b25ed51917d
2019-12-13 20:32:50 +00:00
Egon Elbre
ee6c1cac8a
private: rename internal to private (#3573) 2019-11-14 21:46:15 +02:00
Egon Elbre
994a69cfdc jenkins: use lower segment size for back comp test (#3097) 2019-11-06 05:53:38 -08:00
Egon Elbre
3b18c864dc
test/backwards-compatibility: fix port change (#3509) 2019-11-06 13:46:54 +02:00
Jess G
5abb91afcf
satellite: change the Peer name to Core (#3472)
* change satellite.Peer name to Core

* change to Core in testplanet

* missed a few places

* keep shared stuff in peer.go to stay consistent with storj/docs
2019-11-04 11:01:02 -08:00
Jennifer Li Johnson
76b64b79ba
cmd/identity: allow using redis for RevocationDB (#3259) 2019-11-01 13:27:47 -04:00
Jess G
e96d615013
satellite: remove satellite API code from peer (#3414)
* rm dup api code from sa peer, update storj-sim

* fix for backwards compat tests

* use env var instead of localhost

* changes per CR

* fix env var name

* skip peer for setup
2019-10-30 12:23:09 -07:00
Jennifer Li Johnson
d2502bb51b Adds tests for kad replacement and restores kad operator configs (#3094)
* test that all nodes can check in with all satellites

* keep kademlia config

* add untrusted satellite test

* use getversion

* remove kademlia config changes in test-sim-backwards.sh

* add kademlia flags back to storj-sim storagenode

* reset kademlia flags in storagenode entrypoint
2019-09-20 16:02:23 -04:00
Jennifer Li Johnson
724bb44723
Remove Kademlia dependencies from Satellite and Storagenode (#2966)
What:

cmd/inspector/main.go: removes kad commands
internal/testplanet/planet.go: Waits for contact chore to finish
satellite/contact/nodesservice.go: creates an empty nodes service implementation
satellite/contact/service.go: implements Local and FetchInfo methods & adds external address config value
satellite/discovery/service.go: replaces kad.FetchInfo with contact.FetchInfo in Refresh() & removes Discover()
satellite/peer.go: sets up contact service and endpoints
storagenode/console/service.go: replaces nodeID with contact.Local()
storagenode/contact/chore.go: replaces routing table with contact service
storagenode/contact/nodesservice.go: creates empty implementation for ping and request info nodes service & implements RequestInfo method
storagenode/contact/service.go: creates a service to return the local node and update its own capacity
storagenode/monitor/monitor.go: uses contact service in place of routing table
storagenode/operator.go: moves operatorconfig from kad into its own setup
storagenode/peer.go: sets up contact service, chore, pingstats and endpoints
satellite/overlay/config.go: changes NodeSelectionConfig.OnlineWindow default to 4hr to allow for accurate repair selection
Removes kademlia setups in:

cmd/storagenode/main.go
cmd/storj-sim/network.go
internal/testplane/planet.go
internal/testplanet/satellite.go
internal/testplanet/storagenode.go
satellite/peer.go
scripts/test-sim-backwards.sh
scripts/testdata/satellite-config.yaml.lock
storagenode/inspector/inspector.go
storagenode/peer.go
storagenode/storagenodedb/database.go
Why: Replacing Kademlia

Please describe the tests:
• internal/testplanet/planet_test.go:

TestBasic: assert that the storagenode can check in with the satellite without any errors
TestContact: test that all nodes get inserted into both satellites' overlay cache during testplanet setup
• satellite/contact/contact_test.go:

TestFetchInfo: Tests that the FetchInfo method returns the correct info
• storagenode/contact/contact_test.go:

TestNodeInfoUpdated: tests that the contact chore updates the node information
TestRequestInfoEndpoint: tests that the Request info endpoint returns the correct info
Please describe the performance impact: Node discovery should be at least slightly more performant since each node connects directly to each satellite and no longer needs to wait for bootstrapping. It probably won't be faster in real time on start up since each node waits a random amount of time (less than 1 hr) to initialize its first connection (jitter).
2019-09-19 15:56:34 -04:00
Egon Elbre
7589ca796f
cmd/storj-sim: allow overriding executables (#2976)
The backward compatibility test upgrades only half of the storage nodes and
tests with both the release and new uplink.
2019-09-09 22:13:38 +03:00
paul cannon
bf5a16b823 jenkins: work around git.apache.org failures (#2944) 2019-09-04 11:30:59 +03:00
Egon Elbre
66ec727a37 jenkins: backwards-compatibility, don't overwrite installed binaries (#2892) 2019-08-28 09:57:34 -04:00
Bryan White
863675605a scripts/test-sim-backwards: usability improvements (#2816)
* + add git worktree cleanup
+ specify commit in git worktree add

* fail backwards compatibility test if no postgres

* fix typo

* rm dir after worktree rm
2019-08-19 12:13:38 -07:00
Jess G
309ad2cb29
jenkins: Backwards Compatibility use Postgres (#2810)
* add postgres support

* fix indentation

* create db

* rm prod configs file
2019-08-18 07:51:33 -07:00
Jess G
e1d8503f50
jenkins: add backwards compatibility test (#2774)
* add jenkins runs backwards compat test, fix sa config issue

* try to use current branch

* add jenkins stage condition of branch name

* try when with branch env var

* only run test on master branch

* fix != to ==

* add comment, fix indentation

* run for all branches

* make comment more specific
2019-08-14 08:34:12 -07:00
Jess G
2a464bfdb5
Jg/1622 add backwards compatibility test (#2656)
* add stage to jenkins, add script for backwards compat tests

* debug backwards compat tests

* add setup in a script

* set env vars

* add more env ars

* mv to one script to debug

* debug api key problem

* add tmp dir

* rm print statements
2019-08-06 16:03:14 -07:00