Commit Graph

891 Commits

Author SHA1 Message Date
paul cannon
8ce9ce7f0f satellite/gracefulexit: wait for errgroup to return
credit to Yingrong

Change-Id: I538371040d4dcdf6e943c61e8454320fd57b7526
2020-01-28 19:26:43 +00:00
Michal Niewrzal
90fc1922d0 satellite/metainfo: override bucket RS values with satellite config
Satellite now is keeping RS values for uplink but old uplinks were using
default bucket settings. Because of that we need to override buckets
settings with satellite settings to avoid breaking older uplinks.

Change-Id: Ia1068db70e4adbf741c5e81d27d9e39799049c22
2020-01-28 15:51:04 +00:00
Jennifer Johnson
2209924d41 satellite/satellitedb: use arrays and batch inserts for SaveTallies query
Cockroachdb is more performant with multi-row inserts

Change-Id: Ie1ce2a9da0be1df4e66e72fc9cae49cbd95023f3
2020-01-27 16:54:20 -05:00
Egon Elbre
227e03dea1 satellite/satellitedb: insert using arrays
Using dynamic query strings is error prone, prefer arrays.

Change-Id: I303fbf21c6a54795bd9f399371943b5c51e6f863
2020-01-27 21:27:28 +00:00
Jeff Wendling
d09bd4a749 satellite/satellitedb/dbx: regenerate with paged composite key fixes
before dbx would generate a compilcated blob of conditions that
encoded a row comparison, which only optimized to an index seek
on cockroachdb. this means that sqlite and postgres both had
quadratic behavior on paged queries of this form. instead, use
the implicit row construction feature supported in all of the
databases to do paged support so that they all optimize well.

Change-Id: Iac8703929ba2a59ee3ffa619b916d12663422887
2020-01-27 12:43:16 -07:00
Yingrong Zhao
f3fcbe256c satellite/metainfo: revert combine CommitSegment and CommitObject in batch
This reverts commit 8772867855.

for uplink versions v0.25.0 through v0.30.7, there's a bug with multiplesegment upload
where the last segment is inline caused by this commit.

Change-Id: If375e186b23265586caf08991c25980e99f3cc1a
2020-01-27 13:26:33 -05:00
NikolaiYurchenko
9bcb81108f web/satellite: verification email change
Change-Id: I0293ef4411b55e42bb372b230d797d6798eda515
2020-01-27 15:55:52 +02:00
Michal Niewrzal
ca32ffbfc5 satellite/metainfo: move deletion before upload to satellite
Change is adding object deletion to BeginObject request (before upload).
Now when satellite controls deletion we can move deletion before upload
to satellite. This change improves two things:
* no need for additional request to delete object before upload (need
one more change to storj/uplink)
* fix an issue with lack of permissions to upload if caveat allows only
for writing (e.g. disallow deletes but allows to write)

https://storjlabs.atlassian.net/browse/V3-3362

Change-Id: Ic453146298cdd302df290c532123731a3f99e38e
2020-01-27 12:48:10 +00:00
paul cannon
a0a94a9ac7 satellite/satellitedb: insert into reported_serials w/ arrays
Change-Id: Icb682de09ded3e3159e3590594dcf13f2e7f40f0
2020-01-24 18:36:21 -06:00
littleskunk
90cf78e6f2 satellite/coinpayments: fix migration
The old migration was not working. It was updateding pending (status 0)
and failed (status -1) to completed (status 100).

Change-Id: I808ff3cc692fe6c698ce26a8b411b134e67b752b
2020-01-25 00:12:35 +00:00
littleskunk
a6c6440ab7 satellite/order: decrease expire time from 7 days to 2 days
For the last few month we had no issues with order submission. I would
call it stable and now it is time to risk a lower expire time. This will
increase the database performance on the satellite and it will reduce
the delay for billing.

The long term goal is 6h but for that step we need to change graceful
exit first. At the moment storage nodes would get disuqlaified for not
transfering alle pieces in less than 6 hours.

Change-Id: I421a2c2421c5374c4e706e2338f1c2161fedc14c
2020-01-24 23:37:39 +00:00
Jeff Wendling
26e33e7e07 satellite/gracefulexit: make orders with right bucket id and action
paths are organized as follows:

    project_id/segment_index/bucket_name/encrypted_key

so by picking parts[0] and parts[1], we were using the segment
index instead of the bucket name, causing bandwidth to be
accounted for incorrectly. additionally, we were using the
PUT action instead of the PUT_GRACEFUL_EXIT action, causing
the data to be charged incorrectly. we use PUT_REPAIR for
now because nodes won't accept uploads with PUT_GRACEFUL_EXIT
and our tables need migrations to handle rollups with it.

Change-Id: Ife2aff541222bac930c35df8fcf76e8bac5d60b2
2020-01-24 19:27:38 +00:00
Cameron Ayer
494fead7af satellitedb/orders: fix comma bug in SQL stmt
Change-Id: Ibc6024eeeb5aa4de3909c0cec2d01ac0a01c809f
2020-01-24 13:58:32 -05:00
Ivan Fraixedes
f5c9597d29 golangci: Enable new linter added to last release
Enable a new golangci-lint linter that has been added to the last
release. It reports a very little number of issues so they are fix it in
this commit.

Change-Id: I74fef4779c3f592aae19103fd9f70103586fe24e
2020-01-24 18:09:37 +00:00
Ivan Fraixedes
d5a60aec58
satellite/metainfo: Delete segments in reverse order
Change DeleteObjectPieces for deleting the segments' pointers of an
object in a reverse order.

Last segment: L
N: total number of segments

Deleting in reverse order is: L, n-1 to 0

Deleting in reverse order makes BeginDeleteObject usable to delete
partially uploaded objects that were interrupted (e.g. upload
cancellation).

With this change, the uplink upload cancellation, can be changed to use
BeginDeleteObject to cleanup already uploaded segments without having to
retrieve orders and dial every single node which stored a piece.

Ticket: https://storjlabs.atlassian.net/browse/V3-3525
Change-Id: Ieca6fd3801c4b71671811cb5f08a99d5146928a6
2020-01-24 16:05:12 +02:00
Jeff Wendling
665ed3b6b1 satellite/satellitedb: fix issue with shared memory on range for bucket rollups
A uuid.UUID is an array of bytes, and slicing it refers to the
underlying value, much like taking the address. Because range
in Go reuses the same value for every loop iteration, this means
that later iterations would overwrite earlier stored project
ids. We fix that by making a copy of the value before slicing it
for every loop iteration.

Change-Id: Iae3f11138d11a176ce360bd5af2244307c74fdad
2020-01-23 21:57:02 -07:00
ccase
a9e4c6f66d
satellite/satellitedb/dbx: Remove bashism from gen.sh
Change-Id: Ia698edae99d7ff0c73fa457b4a3c0a7b5f0bbec5
2020-01-23 17:09:07 -05:00
Isaac Hess
44de90ecc8 storagenode/pieces: Rename vars and update comments
A few variables were not renamed to the new standard piecesTotal and
piecesContentSize, so it was unclear which value was being used. These
have been updated, and some comments made more thorough.

Change-Id: I363bad4dec2a8e5c54d22c3c4cd85fc3d2b3096c
2020-01-23 11:00:24 -07:00
Isaac Hess
14fd6a9ef0 storagenode/pieces: Track total piece size
This change updates the storagenode piecestore apis to expose access to
the full piece size stored on disk. Previously we only had access to
(and only kept a cache of) the content size used for all pieces. This
was inaccurate when reporting the amount of disk space used by nodes.

We now have access to the total content size, as well as the total disk
usage, of all pieces. The pieces cache also keeps a cache of the total
piece size along with the content size.

Change-Id: I4fffe7e1257e04c46021a2e37c5adc6fe69bee55
2020-01-23 11:00:24 -07:00
Isaac Hess
40a890639d satellite/orders: Flush all pending bandwidth rollup writes on shutdown
Currently we risk losing pending bandwidth rollup writes even on a clean
shutdown. This change ensures that all pending writes are actually
written to the db when shutting down the satellite.

Change-Id: Ideab62fa9808937d3dce9585c52405d8c8a0e703
2020-01-23 08:12:41 -07:00
Isaac Hess
960e103082 satellite/orders: Rename orders_write_cache to rollups_write_cache
Change-Id: Icffca37e40bb8b2927b38d97728575321c2ad90c
2020-01-23 08:12:41 -07:00
Isaac Hess
0548c3f6bf satellite/orders: RollupsWriteCache has a single method to reset cache
Change-Id: I3ae18115dccd7ac8369313bd96951b9da6464cf3
2020-01-23 08:12:41 -07:00
Egon Elbre
c6f94ce9e4 satellite/metainfo: remove support for boltdb based pointerDB
By previous changes we can now remove testplanet.New and
also remove metainfo boltdb support.

Change-Id: I5bdfbbbb45967492728e705b34b2fedb4f28c381
2020-01-23 13:54:00 +02:00
Egon Elbre
5a4745eddb all: remove usages of testplanet.New
Ensure that tests use testplanet.Run, so we always require running
against all database backends.

Change-Id: I6b0209e6a4912cf3328bd35b2c31bb8598930acb
2020-01-22 22:42:57 +02:00
Egon Elbre
76fdb5d863 storage: add configurable lookup limits
Currently storage tests were tied to the default lookup limit.
By increasing the limits, the tests will take longer and sometimes
cause a large number of goroutines to be started.

This change adds configurable lookup limit to all storage backends.

Also remove boltdb.NewShared, since it's not used any more.

Change-Id: I1a052f149da471246fac5745da133c3cfc27582e
2020-01-22 21:35:56 +02:00
Egon Elbre
fc2766eefc private/testplanet: flatten migration for running tests
Currently Cockroach DB setup takes a significant amount of time.
This flattens the database setup into a single query,
which improves the test time significantly.

The migration tests still test each migration separately.

Change-Id: Iaca16f34a6af3926fa2b5ebf618f939fd59460b3
2020-01-22 15:09:11 +00:00
Michal Niewrzal
6502454947 satellite/metainfo: move RS configuration to satellite
With this change RS configuration will be set on satellite. Uplink with
get RS values with BeginObject request and will use it. For backward
compatibility and to avoid super large change redundancy scheme stored
with bucket is not touched. This can be done in future.

Change-Id: Ia5f76fc10c37e2c44e4f7b8754f28eafe1f97eff
2020-01-22 09:33:53 +00:00
Jeff Wendling
75314a4364 satellite/satellitedb: fix roundToNextDay to handle timezones appropriately
Since incoming times may be in any time zone, and we want the output
to be in UTC and for them to have 00:00:00 hours, minutes and seconds
we first convert the incoming timestamp to UTC before doing the
truncate to the day and adding a day.

Because the old code always returned a timestamp that was in the
future, this is just for efficiency.

Change-Id: Ie692d47bca8691e73852c822d5c56cf8773d99b4
2020-01-21 21:02:16 +00:00
Egon Elbre
0c0b47823d satellite: use require.WithinDuration
Noticed that assert/require has WithinDuration for comparing
time.Time-s.

Change-Id: Ia340896443f610d38799b7ef245b5775eecfc92b
2020-01-21 19:43:53 +02:00
Ethan
21a5d70a83 satellite/metainfo: Rate limiting - API requests
Limits how many times metainfo APIs can be called per second by project ID. If limit is exceeded, the API will return Unauthorized/Too Many requests.

Limit per second and the size of the limiter cache per project are configurable, as well as whether the limiter is enabled.

Tests added/updated for the new rate_limit field in projects table.
Tests added for exceeding limits and disableing limiter.

Change-Id: Ic8ad102de3b690a475809d4f684156d5715f20fa
2020-01-21 14:25:04 +00:00
Michal Niewrzal
8772867855 satellite/metainfo: combine CommitSegment and CommitObject in batch
This change is a special case for batch processing. If in batch request
CommitSegment and CommitObject are one after another we can execute
these request as one. This will avoid current logic where we are saving
pointer for CommitSegment and later we are deleting this pointer and
saving it once again as under last segment path for CommitObject.

Change-Id: If170e78c8410f5ba5916cbff6a29b9221db9ce2e
2020-01-21 09:30:55 +00:00
Michal Niewrzal
86f194769f uplink: adjust to changes in storj/uplink
This change is adjusting code base to changes in storj/uplink.

https://review.dev.storj.io/c/storj/uplink/+/643

Change-Id: Ieca87f9f5983e391bf4b4fec8b9d5491fd32bfa1
2020-01-20 22:06:19 +00:00
Yaroslav
c636b06191 satellite/console: use cookie based auth scheme
Change-Id: I143b56f49fa9028ec172db8c29fd93577c3e7878
2020-01-20 21:35:23 +02:00
stefanbenten
b678b55f83 satellite/metainfo: improve metainfo logging
Change-Id: Ibc06393bfb996e1cad9bb76c451e3faf2393b478
2020-01-20 18:49:55 +00:00
Egon Elbre
c1c878efcf all: fix import groupings
check-imports was broken and didn't complain about things.

Change-Id: I38adafd16b4aba86f0eb4f53427b4393f9a6c710
2020-01-20 17:47:44 +00:00
Yaroslav
33790e0f7e satellite/console: handle graphql errors properly
Change-Id: Id62b4f10cff9e9caba64ec940f558530eadbabbc
2020-01-20 15:20:24 +00:00
Egon Elbre
f3b4bf2b7c satellite/satellitedb/satellitedbtest: pass ctx as an argument
ctx is created in most tests, instead pass in as argument
to reduce code duplication.

Change-Id: I466c51c008392001129c8b007c9d6b3619935ac4
2020-01-20 16:35:42 +02:00
Egon Elbre
1279eeae39 private/tagsql,storage: fixes to context cancellation
Replace all the remaining uses of sql.DB with tagsql.DB to
fix issues with context cancellation.

Introduce tagsql.Open which helps to get rid of all tagsql.Wrap-s.
Use tagsql in cockroachkv and postgreskv.

Change-Id: I8946d203341cb85a25976896fc7881e1f704e779
2020-01-20 15:44:39 +02:00
Egon Elbre
ba2fce814c satellite/satellitedb: better coupons query
Change-Id: Iaf180b99c57443550418b46dfd8300f921e93bec
2020-01-20 15:05:10 +02:00
Egon Elbre
10d932fd65 lib/uplinkc: fix test flakiness by setting MaxTimeSkew
Not having a skew caused an issue where:

1. Uplink calls "begin segment", where segment isn't committed to the
database.
2. Uplink stores piece X to the storage node A with timestamp 1.
3. Satellite runs garbage collection with timestamp 2.
4. Satellite sends retain request to storage node A with timestamp 2.
5. Storage node A deletes piece X, because 1 < 2.
6. Uplink calls "commit segment" with storage node A in it.
7. Download of segment fails, because A doesn't have piece X.

In production this is not an issue since the MaxTimeSkew is 72h by
default.

Change-Id: Id87ca3ddc44103dcd85d031b1367168c014b8e7b
2020-01-20 12:44:42 +00:00
crawter
c4cbc6ff2f satellite/payments: promotional coupons generation functional added
Change-Id: Ie0df256503114ca377d81bf7c8b26cc90a1f5b26
2020-01-20 11:01:55 +00:00
Egon Elbre
a4026f97b8 satellite: fix test time comparisons
Correct way to compare time that may have an error is to use InDelta.

Change-Id: I0140892119c44c63fa042bbc7292ab91bb33a350
2020-01-20 10:17:20 +00:00
Egon Elbre
d5438036b5 {satellite,storagnode}/gracefulexit: reduce logging
Change-Id: I9f274ede77a582fc43ef14a47bf9341d4e3083df
2020-01-19 22:36:13 +02:00
Egon Elbre
cf7b22c466 satellite/satellitedb: add missing err check
Change-Id: I502838f78f1871315597b488602c0f1112612981
2020-01-19 19:24:12 +00:00
Yaroslav
48303bc0a9 satellite/console/limits: add content-type to response, fix error
Change-Id: I63dffb5372456569b100f97b04eacc1fc19bddf5
2020-01-19 15:42:55 +00:00
Egon Elbre
c207cd08fc satellite/satellitedb: gracefulexit, add missing Errs check
Change-Id: Iba4ba84fd57b3a0a0d15f13006566076045d6c11
2020-01-19 15:24:12 +00:00
Egon Elbre
1abfe42142 satellite: use tagsql
Change-Id: I2170dee409fb0c2fe85913ddd36e7811a3b853ed
2020-01-19 14:39:16 +02:00
Egon Elbre
59d06644b9 private/migrate: switch to tagsql
Also added temporary types withRebind and withTagTx,
which will be later removed. Currently they help to avoid
changing the whole codebase at the same time.

Change-Id: I7f07ba8f4709a23a463bfa67464628665a05808f
2020-01-19 14:39:16 +02:00
stefanbenten
f4097d518c satellite: reduce logging of node status
Change-Id: I6618cf4bf31b856acd7a28b54011a943c03ab22a
2020-01-18 17:47:59 +00:00
Yaroslav
d8368d0b30 satellite/payments: coinpayments add completed status, treat received status as pending, add balance for completed transactions only
Change-Id: I20494bdddfda6d4f37ba2c5b6f7955cd29a6d798
2020-01-17 17:26:34 +00:00