Commit Graph

2642 Commits

Author SHA1 Message Date
Michal Niewrzal
1c72e80e40 uplink/satellite: fix for case when inline segment is last one (#3062)
* uplink/satellite: fix when inline seg is last one

* review comments
2019-09-19 01:18:14 +02:00
Cameron
ccdd435610
defer client.close() (#3084) 2019-09-18 16:17:04 -04:00
Jess G
695de9dcd7
rm noisy debug logs that we dont need (#3083) 2019-09-18 12:43:57 -07:00
Jennifer Li Johnson
ce3203e910
update NodeSelectionConfig.OnlineWindow to 4hr default (#3082) 2019-09-18 14:57:57 -04:00
Simon Guindon
a2b1e9fa95
storagenode/storagenodedb: refactor both data access objects and migrations to support multiple DB connections (#3057)
* Split the info.db database into multiple DBs using Backup API.

* Remove location. Prev refactor assumed we would need this but don't.

* Added VACUUM to reclaim space after splitting storage node databases.

* Added unique names to SQLite3 connection hooks to fix testplanet.

* Moving DB closing to the migration step.

* Removing the closing of the versions DB. It's already getting closed.

* Swapping the database connection references on reconnect.

* Moved sqlite closing logic away from the boltdb closing logic.

* Moved sqlite closing logic away from the boltdb closing logic.

* Remove certificate and vouchers from DB split migration.

* Removed vouchers and bumped up the migration version.

* Use same constructor in tests for storage node databases.

* Use same constructor in tests for storage node databases.

* Adding method to access underlining SQL database connections and cleanup

* Adding logging for migration diagnostics.

* Moved migration closing database logic to minimize disk usage.

* Cleaning up error handling.

* Fix missing copyright.

* Fix linting error.

* Add test for migration 21 (#3012)

* Refactoring migration code into a nicer to use object.

* Refactoring migration code into a nicer to use object.

* Fixing broken migration test.

* Removed unnecessary code that is no longer needed now that we close DBs.

* Removed unnecessary code that is no longer needed now that we close DBs.

* Fixed bug where an invalid database path was being opened.

* Fixed linting errors.

* Renamed VersionsDB to LegacyInfoDB and refactored DB lookup keys.

* Renamed VersionsDB to LegacyInfoDB and refactored DB lookup keys.

* Fix migration test. NOTE: This change does not address new tables satellites and satellite_exit_progress

* Removing v22 migration to move into it's own PR.

* Removing v22 migration to move into it's own PR.

* Refactored schema, rebind and configure functions to be re-useable.

* Renamed LegacyInfoDB to DeprecatedInfoDB.

* Cleaned up closeDatabase function.

* Renamed storageNodeSQLDB to migratableDB.

* Switched from using errs.Combine() to errs.Group in closeDatabases func.

* Removed constructors from storage node data access objects.

* Reformatted usage of const.

* Fixed broken test snapshots.

* Fixed linting error.
2019-09-18 12:17:28 -04:00
Egon Elbre
186e67e056 pkg/transport: set default timeout to 10 minutes (#3075) 2019-09-18 11:56:23 -04:00
Maximillian von Briesen
574c96c350
satellite/metainfo: Verify storagenode signature on satellite upload (#2985) 2019-09-18 09:50:33 -04:00
Isaac Hess
fd20fa38c6
internal/dbutil/sqliteutil: add MigrateTablesToDatabase (#3064)
* internal/dbutil/sqliteutil: add migrator

* internal/dbutil/sqliteutil: Fix errors and tablename
2019-09-17 15:42:40 -06:00
Jess G
7c203b4884
add satelliteSystem to testplanet and update tests (#3066) 2019-09-17 13:14:49 -07:00
Natalie Villasana
cc70cd2329
satellite/repair: add metric trackers for segment age before repair (#3056) 2019-09-17 15:18:48 -04:00
Maximillian von Briesen
684b07b2c1
scripts/protobuf.go: update drpc version for protobuf generation (#3059) 2019-09-17 13:14:38 -04:00
Michal Niewrzal
d65386f69e
auto-updater: unpack/check binary, restart service (#2968) 2019-09-17 00:19:56 -07:00
Natalie Villasana
4aaf525bd3
satellite/audit: set devDefaults for ChoreInterval and QueueInterval to 1m (#3058) 2019-09-16 16:36:33 -04:00
Ivan Fraixedes
febe32bc7a pkg/miniogw: Add a missed stack trace error (#3035)
Add the stack trace to an error returned by one of the functions for
being able to track down the origin in case the error happens and gets
logged.
2019-09-16 23:00:50 +03:00
Simon Guindon
91d54af705
Add satellites database business objects. (#3055)
* Add satellites database business objects.

* Fixed linting error.
2019-09-16 13:54:53 -04:00
Yingrong Zhao
b37ea864b1
satellite/repair: delete pieces that failed piece hashes verification from pointer (#3051)
* add test

* add implementation

* remove todo comments

* modifies cooment

* fix linting

* typo oops
2019-09-16 13:13:24 -04:00
Maximillian von Briesen
7ada5d4152
satellite/metainfo: make piece hashes nil before storing pointer in metainfo.UpdatePieces() (#3050) 2019-09-16 12:11:12 -04:00
Matt Robinson
0c2ae7786e storagenode/dashboard: Enable storagenode dashboard in docker images (#3024) 2019-09-16 15:09:59 +02:00
Jess G
d3ef574b20 pkg/pb: minor changes to contact.proto (#3048)
* minor fixes to contact proto

* simply and rm nodeAddr object from client
2019-09-13 19:37:32 -05:00
Isaac Hess
5a50042c77
uplink/storage/streams: Add test for interrupted deletes (#3040)
* uplink/storage/streams: Add test for interrupted deletes

* uplink/storage/streams: Fix linting errors
2019-09-13 13:08:15 -06:00
Andrew Harding
f550ab5d1c
Uplink "import" command (#2981)
* uplink import cmd

* pkg/process: fix import order

* fix golangci-lint failures

* remove "help" from the satellite config lock file
2019-09-13 12:33:30 -06:00
Ethan Adams
886041e0ba
satellite/satellitedb: add new graceful exit tables and add graceful exit fields to nodes (#3033)
DB schema changes for satellite to support Graceful Exit
2019-09-13 12:57:32 -04:00
Yingrong Zhao
95aa33c964
satellite/repair/repairer: update audit status as failed after failing piece hash verification (#2997)
* update audit status as failed for nodes that failed piece hash verification

* remove comment

* fix lint error

* add test

* fix format

* use named return value for Get

* add comments

* add more better comment

* format
2019-09-13 12:21:20 -04:00
Vitalii Shpital
46ada7b6b1
web/storagenode: error processing added (#3046) 2019-09-13 19:06:04 +03:00
Yaroslav Vorobiov
9d1e19cd92
satellite/console/server: fix usage rollup report (#3047) 2019-09-13 18:38:29 +03:00
Yehor Butko
8b54e329f3
web/satellite: added trailing coma rule to tslint (#3044) 2019-09-13 17:58:18 +03:00
Egon Elbre
7240e6cbb2
satellite: remove remote/inline file from BucketTally (#3041) 2019-09-13 16:51:41 +03:00
Egon Elbre
ca058e606f
storagenode/orders: fix data race in settle (#3042) 2019-09-13 15:50:39 +03:00
Cameron
44bcdd222f storagenode/contact: test node info is updated (#3039) 2019-09-13 07:53:48 -04:00
Ivan Fraixedes
ccbf73ecc7
uplink/ecclient: Remove unneeded atomic operation (#3036)
Atomic operations are only needed when a variable can be accessed
concurrently, so when it isn't the case there is no need to use them.
2019-09-13 12:47:35 +02:00
Nikolay Yurchenko
77e7f182db
notification height fixed (#3020) 2019-09-13 13:25:52 +03:00
Ivan Fraixedes
8a48500ba4
uplink/ecclient: Report success in debug level (#3037)
Packages shouldn't be chatty when the things go as expected unless the
DEBUG log level is set.
2019-09-13 12:04:12 +02:00
Yehor Butko
4266f8c364
web/satellite: linter updated, analytics artifacts removed (#3043) 2019-09-13 12:48:27 +03:00
Yehor Butko
a342c3abad
web/satellite: analytics removed, buckets header foreground improved (#3027) 2019-09-13 11:34:36 +03:00
Jeff Wendling
0dcbd3dc08 bootstrap/satellite/certificate/storagenode: register drpc services
Change-Id: Id29f14b76a8c9cb2be31001b9a7a4356a4bda183
2019-09-12 15:09:46 -06:00
Jeff Wendling
007662d49e pkg/server: serve drpc listeners along with grpc
The fundamental problem is that both drpc and grpc servers
want to close the listener and they both want to ignore the
error from Accept after the listener is closed. There's no
way to do this in a race free way. Fortunately, the mux
hands out listeners that can be independently closed. That
means they can both do their own shutdown logic where they
ignore the error, and then after they're closed, the code
orchestrating the servers can close the listeners.

The final weird bit is that the server's Close method is
required to wait until the Run method has exited (or at
least enough for the listeners to definitely be closed)
because tests depend on that behavior, so we have to add
some channels/mutexes/onces to ensure that Run has exited
and that a new call can't start after Close is called.

Change-Id: I7c4ef293f7963f83138815f51824fd5b8d09ce15
2019-09-12 19:18:30 +00:00
Jeff Wendling
477b47f554 pkg/server: use a listenmux with nothing registered
Change-Id: I25577b4afb907f4f8b57fc0428de6e6ea4ce9ba9
2019-09-12 19:12:53 +00:00
Ethan Adams
731016cd85
Increase file size limit to 650 KB (#3034) 2019-09-12 13:54:44 -04:00
Egon Elbre
8ef57a2af3
satellite/satellitedb: use noreturn (#3022) 2019-09-12 20:31:50 +03:00
Natalie Villasana
a085b05ec5 satellitePeer -> satellite rename consistency in repair test (#3032) 2019-09-12 13:16:39 -04:00
Cameron
ab1147afb6
storagenode/pieces: fix race condition in cache service (#2972)
* add mutex lock to PersistCacheTotals, move lock around copyCacheTotals to inside function
2019-09-12 12:42:39 -04:00
Maximillian von Briesen
82a651ac3a
satellite/contact: Populate PeerIdentities table in satellitedb (#2998)
* Add peer identities db dependency to contact service
* Update peer identities db on contact checkin
2019-09-12 12:33:04 -04:00
Ivan Fraixedes
cc8a47324a
scripts: Fix warn message update sat config lock (#3029)
Fix the warning message to indicate the Slack channel where the
satellite configuration changes must be posted.
2019-09-12 18:10:22 +02:00
Vitalii Shpital
500ed27e01
web/satellite: error while project creation fixed (#3023) 2019-09-12 18:04:49 +03:00
Vitalii Shpital
489047a96e
web/satellite: notifications for billing page implemented (#3004) 2019-09-12 17:49:31 +03:00
Kaloyan Raev
208327835f
Script for deploying the Docker manifest for watchtower (#3015) 2019-09-12 17:38:48 +03:00
Bogdan Artemenko
bb6086aeab
web/satellite: API keys paged backend (#2839) 2019-09-12 17:19:30 +03:00
Bogdan Artemenko
0c1798407c
web/satellite: API Keys Pagination implementation (#3019) 2019-09-12 17:10:50 +03:00
Vitalii Shpital
5383b8be82
web/storagenode: frontend bug fixes (#3021) 2019-09-12 16:46:22 +03:00
littleskunk
1d8cd526e0 storj-sim: correct storagenode dashboard config (#3010) 2019-09-12 15:20:52 +03:00