Commit Graph

3113 Commits

Author SHA1 Message Date
Isaac Hess
6aeddf2f53
storagenode/pieces: Add Trash and RestoreTrash to piecestore (#3575)
* storagenode/pieces: Add Trash and RestoreTrash to piecestore

* Add index for expiration trash
2019-11-20 09:28:49 -07:00
Kaloyan Raev
6d728d6ea0
storagenode/collect: delete piece 24 hours after expiration (#3613) 2019-11-20 17:02:57 +02:00
Matt Robinson
b5707d1e5d
scripts: make update-tools.sh more verbose (#3572)
* Make update-tools.sh more verbose

* Was checking the wrong filehandle
2019-11-20 09:41:06 -05:00
Vitalii Shpital
61c8bcc9a6
web/storagenode: egress chart implemented (#3574) 2019-11-20 16:37:57 +02:00
Vitalii Shpital
49694b2c7a
web/satellite: successful reset password page styling bug fixed (#3612) 2019-11-20 16:28:09 +02:00
Yehor Butko
9ca547acb1
web/satellite: project charges (#3611) 2019-11-20 15:46:22 +02:00
Vitalii Shpital
16f0e998b1
web/satellite: project selection dropdown bug fixed (#3605) 2019-11-20 12:55:35 +02:00
JT Olio
fe8d556b4e
add sourcerer hall-of-fame to README
https://github.com/sourcerer-io/hall-of-fame
2019-11-19 16:08:16 -07:00
Rafael Antonio Ribeiro Gomes
da39c71d35
storagenode: add new metric satellite.request (#3610)
* storagenode: add new metric satellite.request

* storagenode: metrics fixed

* switch from Counter to Meter
2019-11-19 18:11:31 -03:00
Jess G
e9c3194c82
satellitedb: merge migration into one step (#3551)
* merge migration

* rm migration versions

* rm unneeded migration test data

* create index w/postgres + crdb compatible syntax

* add default to offers.invitee_credit_duration_days

* changes so that schema matches from master to branch

* change to be crdb compatible

* add check to confirm db version

* mv version check to migration

* update tests

* add minversion to sadb migration, update tests

* confirm min version for all dbs in a migration

* add validate migration to sadb

* fix lint err

* rm min version check from migrate

* change sadb check

* hard code min db version

* fix comment
2019-11-19 12:52:57 -08:00
Yaroslav Vorobiov
3d94c3fc99
satellite/payments: stripe client use satellite logger, give credits when applying transaction to balance (#3603) 2019-11-19 19:56:18 +02:00
Maximillian von Briesen
8653dda2b1 satellite/audit: do not contain nodes for unknown errors (#3592)
* skip unknown errors (wip)

* add tests to make sure nodes that time out are added to containment

* add bad blobs store

* call "Skipped" "Unknown"

* add tests to ensure unknown errors do not trigger containment

* add monkit stats to lockfile

* typo

* add periods to end of bad blobs comments
2019-11-19 17:30:28 +01:00
Ivan Fraixedes
8e1e4cc342
piecestore: Fix invalid comment and typos (#3604) 2019-11-19 16:30:48 +01:00
Ivan Fraixedes
46ad27c5f2
docs/blueprints: Change adapt endpoint by creating (#3601)
Replace a sentence which indicates that we're creating and new endpoint
in the storage node side rather than adapting the current one.
2019-11-19 14:28:19 +01:00
Michal Niewrzal
5964502ce0
uplink/metainfo: remove GetObject from download Batch (#3596) 2019-11-19 04:58:26 -08:00
Nikolai Siedov
24318d74b3
storagenode/console: show satellite url in satellite selection (#3602) 2019-11-19 14:16:56 +02:00
Ivan Fraixedes
b5b4543df3
docs/blueprints: Deletion performance (#3520)
Creates the blueprint document for changing the mechanism to delete pieces and improve its performance.
2019-11-19 08:16:35 +01:00
Jeff Wendling
53176dcb0e pkg/rpc/rpcstatus: do not depend on grpc/drpc build mode
if your server is built to make drpc connections, clients can
still connect with grpc. thus, your responses to grpc clients
must still look the same, so we have to have all of our status
wrapping include codes for both the drpc and grpc servers to
return the right thing.

Change-Id: If99fa0e674dec2e20ddd372a827f1c01b4d305b2
2019-11-18 15:51:58 -07:00
Jeff Wendling
f3b20215b0 pkg/{rpc,server,tlsopts}: pick larger defaults for buffer sizes
these may not be optimal but they're probably better based on
our previous testing. we can tune better in the future now that
the groundwork is there.

Change-Id: Iafaee86d3181287c33eadf6b7eceb307dda566a6
2019-11-18 21:22:49 +00:00
Natalie Villasana
3a0b71ae5b
pkg/pb/referralmanager: update RedeemTokenRequest, rm ReserveToken (#3593) 2019-11-18 16:17:21 -05:00
littleskunk
c52c7275ad
satellite/repair: reduce upload timeout (#3597) 2019-11-18 18:52:56 +01:00
Michal Niewrzal
8ea09d5556
cmd/segment-reaper: add iteration over DB (#3576) 2019-11-18 09:13:20 -08:00
Michal Niewrzal
ec41a51bbb
metainfo/loop: refactor iterator to be reusable (#3578) 2019-11-18 07:26:48 -08:00
Nikolay Yurchenko
fdcf328469
short name field removed from registration page (#3584) 2019-11-18 13:56:02 +02:00
Nikolai Siedov
3fe518d547
satellite: added ability to inject stripe public key post build (#3560) 2019-11-18 13:38:43 +02:00
Kaloyan Raev
f4a626bbc6 storagenode-updater: re-enable auto-update storagenode-updater (#3588)
storagenode-updater: re-enable auto-update for storagenode-updater
2019-11-18 08:26:50 +01:00
littleskunk
8b3444e088
satellite/nodeselection: don't select nodes that haven't checked in for a while (#3567)
* satellite/nodeselection: dont select nodes that havent checked in for a while

* change testplanet online window to one minute

* remove satellite reconfigure online window = 0 in repair tests

* pass timestamp into UpdateCheckIn

* change timestamp to timestamptz

* edit tests to set last_contact_success to 4 hours ago

* fix syntax error

* remove check for last_contact_success > last_contact_failure in IsOnline
2019-11-15 23:43:06 +01:00
Jeff Wendling
ecd2ef4a21 all: build release fully dprc and test in mixed mode
Change-Id: I3bded3edf25a0b113601c8b12ecf1337f596649b
2019-11-15 10:03:18 -07:00
Maximillian von Briesen
2524cc5d42
pkg/pb: remove unneeded proto imports (#3585) 2019-11-15 11:55:32 -05:00
Vitalii Shpital
b79fad6591
web/satellite: redirecting condition for universal login page added (#3550) 2019-11-15 18:26:16 +02:00
Egon Elbre
91859f1ccf
satellite/metainfo: fix broken test (#3580) 2019-11-15 17:44:23 +02:00
Yaroslav Vorobiov
a36e9b504e satellite/payments: switch to using STORJ tokens (#3582) 2019-11-15 17:27:13 +02:00
Yaroslav Vorobiov
53c6741ba6
satellite/payments: add API for retrieving conversion ratio, convert tokens to USD before applying to balance (#3530) 2019-11-15 16:59:39 +02:00
Michal Niewrzal
ead8af3c16
storagenode/updater: recover command in Service Manager (#3425) 2019-11-15 06:37:29 -08:00
Yehor Butko
a8e4e9cb03
satellite/payments: project usage charges (#3512) 2019-11-15 16:27:44 +02:00
Ivan Fraixedes
aa7b5b7c53
satellite/metainfo: avoid large conditional block (#3579)
Large conditional blocks are hard to read.
When the conditional block only has one branch it's easy to understand
the logic of the function to early return switching the condition.
2019-11-15 14:33:09 +01:00
Nikolai Siedov
0d35505fe1
SNOboard/console: router changed for gorillaMux, caching added (#3577) 2019-11-15 14:36:43 +02:00
Michal Niewrzal
bc16cb5d24
libuplink: remove additional GetBucket for upload/download (#3568) 2019-11-15 02:06:17 -08:00
Ivan Fraixedes
c193dee9ae
uplink/storage/streams: Fix upload error clean up (#3555)
The number of the last segment uploaded returned by upload wasn't
correct in all the return statements. The Put method calls upload but
with the returned values wasn't even certain of cleaning up correctly
the segments uploaded when an error happens.

This commit moves the logic of cleaning up inside of the upload method
because it's easier to understand than only doing inside of it the clean
up when a context cancellation happens. It also fixes the number of
segments to be deleted by cancelHandler when an error happens.

These changes should avoid trying to delete segments which haven't been
uploaded because of the wrong index value or because of being inline.
2019-11-15 10:37:20 +01:00
Isaac Hess
2166c2a21b
storage/filestore: Add Trash and RestoreTrash to Blobs (#3529)
* storage/filestore: Add Trash and RestoreTrash to Blobs

* Change restore to be satellite-specific

* Fix comment

* Fix merge rename conflict
2019-11-14 15:19:15 -07:00
Egon Elbre
ee6c1cac8a
private: rename internal to private (#3573) 2019-11-14 21:46:15 +02:00
Vitalii Shpital
f1d0d0d68f
web/satellite: static pages styling fixed (#3549) 2019-11-14 19:51:14 +02:00
Yingrong Zhao
d2a8ab5d7f pkg/pb: add referral manager protobuf definition (#3561) 2019-11-14 12:33:00 -05:00
Michal Niewrzal
4912922631
cmd: skeleton for segment-reaper detect command (#3562) 2019-11-14 07:57:27 -08:00
Egon Elbre
1a54007f1c
storagenode/storagenodedb: dont log opening of each database (#3571) 2019-11-14 17:08:16 +02:00
Matt Robinson
b2a7a9f4c4 scripts: add script to update tools (#3570) 2019-11-14 16:53:32 +02:00
Bryan White
b56cc21710 cmd/storagenode-updater: make updater test windows compatible (#3542) 2019-11-14 16:03:49 +02:00
Kaloyan Raev
77ed047428
installer/windows: unit test mocking file system (#3543) 2019-11-14 14:02:03 +02:00
Egon Elbre
1e64006e32 lint: add staticcheck as a separate step (#3569) 2019-11-14 10:31:30 +02:00
paul cannon
d5963d81d0
storage/postgreskv: fix ultra-slow nonrecursive list (#3564)
This is based on Jeff's most excellent work to identify why
non-recursive listing under postgreskv was phenomenally slow. It turns
out PostgreSQL's query planner was actually using two sequential scans
of the pathdata table to do its job. It's unclear for how long that has
been happening, but obviously it won't scale any further.

The main change is propagating bucket association with pathnames through
the CTE so that the query planner lets itself use the pathdata index on
(bucket, fullpath) for the skipping-forward part.

Jeff also had some changes to the range ends to keep NULL from being
used- I believe with the intent of making sure the query planner was
able to use the pathdata index. My tests on postgres 9.6 and 11
indicate that those changes don't make any appreciable difference in
performance or query plan, so I'm going to leave them off for now to
avoid a careful audit of the semantic differences.

There is a test included here, which only serves to check that the new
version of the function is indeed active. To actually ensure that no
sequential scans are being used in the query plan anymore, our tests
would need to be run against a test db with lots of data already loaded
in it, and that isn't feasible for now.

Change-Id: Iffe9a1f411c54a2f742a4abb8f2df0c64fd662cb
2019-11-13 17:52:14 -06:00