Commit Graph

3735 Commits

Author SHA1 Message Date
Jeff Wendling
97e980cd8a private/dbutil: add database name to configure as a tag
storagenodes have like 10 or more databases. without this
tag they all get sent as the same value, stomping on each
other.

Change-Id: Ib12019684d6ea8f2a5b83df584056dfa79e3c4b3
2020-03-26 16:50:15 +00:00
crawter
f879bfcf70 storagenode/console/server/heldamount - endpoint tests added
Change-Id: I4a324e808fd18e55bad28c1cce2ea90c41dc659e
2020-03-26 15:50:33 +00:00
Yingrong Zhao
fecd36f6b6 bump uplink version to latest
Change-Id: I57c7dae60e4b521bf6f5a07b107f5c2ef18fc168
2020-03-26 09:03:53 +00:00
JT Olio
4d91533a3f
Update README.md (#3828) 2020-03-25 19:53:11 -05:00
Jennifer Johnson
b75cbc8e24 satellite,storagenode: remove references to free bandwidth
Change-Id: I42a6597544804fa9235e89ec656ebc365eb522e5
2020-03-25 22:28:34 +00:00
Egon Elbre
c715c75fea pkg/server: add counters for grpc calls
This will help to determine how many grpc calls are made to the
satellite.

Also remove the grpc funcs that have been added to upstream.

Change-Id: I91878f4fd10f9bfe601c94222c102eaaf4d35963
2020-03-25 21:38:13 +02:00
Yingrong Zhao
b7b19289d1 bump storj.io/common to latest
Change-Id: I16e337660ce8e1ef332cc842dbf4cfa067b9b98b
2020-03-25 09:08:40 -04:00
Kaloyan Raev
0a50d6bdce bump storj.io/uplink to v1.0.1
Change-Id: I55d44aee1a02fe662408a3c48a3e97d6721994af
2020-03-25 10:41:34 +02:00
Yingrong Zhao
a731472496 bump storj.io/common to latest and storj.io/drpc to v0.0.11
Change-Id: I7a6e823b441eeff4621dfdf2d6577be76c9761c8
2020-03-24 15:17:10 -04:00
Michal Niewrzal
2a482e8bc4 private/version/checker: remove code which was moved to
`storj/private/debug` package

Change-Id: I44dfecd6ab875fb33851a22cf10b3064da9bfd65
2020-03-24 17:07:33 +00:00
Egon Elbre
bd9a998abd pkg/server: remove dead code
Change-Id: I4501ac7de727e2ec6908a7624808b4e596a68a23
2020-03-24 18:30:22 +02:00
Michal Niewrzal
3a3648c80b scripts: add script for running versions tests locally
Change-Id: Iaa2a2d085b7edd46f63a0d79b4e731ea72412953
2020-03-24 15:44:00 +00:00
Michal Niewrzal
fdf40a7526 storj: remove storj/private/version package which was moved to
`storj/private` repo

Change-Id: I81c3f5b9d5e4fe7bca760999eb045ee9734e5e2e
2020-03-24 14:31:33 +00:00
Stefan Benten
bdec51658e
cmd/storj-sim: Increase storj-sim max-alpha-usage (#3824) 2020-03-24 14:48:25 +01:00
Michal Niewrzal
aed8dea625 satellite/accounting: Add test not billing after deletion
New test added to be sure that we don`t bill for the data after deleting it.

Change-Id: Ifb5931ce28f6b0294aeb16311164675a17f11917
2020-03-24 12:55:49 +00:00
Stefan Benten
173cb1e484
Changing LogLevel to Warn (#3822)
This is not a process error and can cause false alarm for monitoring systems
2020-03-24 13:46:28 +01:00
Egon Elbre
a9fb7b7694 storagenode/trust: fix go 1.14 failures
Change-Id: I6d147f8e5907666ffb3a3e487fa179e377c70841
2020-03-24 11:04:30 +00:00
Egon Elbre
326c0cebde storage/boltdb: update to etcd/bbolt v1.3.4
bbolt v1.3.4 has pointer usage fixes.

Change-Id: I5e0fc4782711d01c09ced579f25a4f8fbc8de85c
2020-03-24 12:33:34 +02:00
Michal Niewrzal
f0aeda3091 storj: remove from storj/pkg packages moved to storj/private repo
* debug
* traces
* cfgstruct
* process

Package `storj/private/version` will be removed as a separate change.

Change-Id: Iadc40faa782e6225513b28218952f02d9c240a9f
2020-03-24 09:56:29 +01:00
Isaac Hess
ddf87286c5 cmd/statreceiver: Remove statreceiver code
Change-Id: Ic572a1623a2008c49d584b769d13e07e4470bce0
2020-03-23 13:00:13 -06:00
Egon Elbre
6a7571f73e cmd/s3-benchmark: move to storj.io/benchmark
Change-Id: Idca2b836bdf876ca28eb5cabc9bfae1d576e4a3e
2020-03-23 19:09:42 +02:00
VitaliiShpital
d7558db5ed web/satellite: succesful registration flow reworked
Change-Id: I29f7b0fd2286c32ada39bc1e86d58feda624c645
2020-03-23 16:09:00 +00:00
VitaliiShpital
fe39845a8c web/satellite: refreshing billing history on entering billing page implemented
Change-Id: I986d6bc5ca1277c78ead7cf6323d59cb73045e25
2020-03-23 15:26:08 +00:00
VitaliiShpital
f61b77ea39 web/satellite: request limit link added on project details page
Change-Id: I4e57bb2f9dc610e9103ca21face20a615481bf40
2020-03-23 15:03:40 +00:00
VitaliiShpital
859fce3af7 web/satellite: browser caching bug fixed
Change-Id: I5b97a90d28c754cc893852909dcc87290a739f59
2020-03-23 14:41:20 +00:00
Michal Niewrzal
e258de518c cmd/storj-sim: remove identity dir from gateway configuration
We are not generating identity for gateway during setup anymore, its
generated on-fly by libuplink. With this change we can remove
`--identity-dir` from gateway.

Change-Id: I63298115d4399564b2f29541b8dc16e3b3acdcf2
2020-03-23 14:11:51 +00:00
Kaloyan Raev
6c44512c15 satellite/metainfo: increase stream ID expiration to 48h
Otherwise, it is not possible to upload very large files like 1 TB.

Change-Id: Iadda1ae91174125736684850b906d5bb6d19f0a9
2020-03-23 15:47:39 +02:00
Egon Elbre
06c499ccf4 ci: remove .golangci.yml
Currently .golangic.yml duplicates what's in ci repository.
Keeping things in sync is somewhat annoying, it's better to
let people setup a symlink to the appropriate file.

Change-Id: I12dd839f51a7e9cf8d39eaaa32e5eeab7ce70910
2020-03-23 15:06:57 +02:00
Kaloyan Raev
98ac16fb5f cmd/wizard: improve prompt for satellite
Change-Id: Iff83ccf30c7b4059cc970c14d29e7d11e497c4bb
2020-03-23 13:38:24 +02:00
Michal Niewrzal
3781733816 pkg/process: replace storj/pkg/telemetry with common/telemetry
* storj/pkg/telemetry package replaced with common/telemetry
* storj/pkg/telemetry deleted

https://storjlabs.atlassian.net/browse/SM-440

Change-Id: I67921b05d653d75bdb90c93bd80d022aa9fd2d9f
2020-03-23 11:14:48 +01:00
JT Olio
3b66ba6f02 scripts/tests: uplink no longer respects --client.segment-size
this is going to make all the tests slower but it is what it is

test-sim-aws.sh is removed because it was moved to storj/gateway repo.

Change-Id: I10727e747a4c3740b1c9054ce7d17313b4fa310b
2020-03-22 17:48:57 +00:00
Jessica Grebenschikov
aeab599d21 satellitedb: removed unused id on storagenode_storage_tallies table, add index on node_id
The goal of this change is to improve the storagenode_storage_tallies table by removing the unneeded id column that is not being used but only taking up space, and also to add an index on a different column that needs it. Removing and adding a column seems simple, but ended up being more complicated because of some cockroachdb limitations.

The cockroachdb limitation when trying to remove a column from a table and create a new primary key are:
1. only allows primary key creation at table creation time (docs: https://www.cockroachlabs.com/docs/stable/primary-key.html)
2. table drop or rename is performed async and cannot be done in a transaction (issue: https://github.com/cockroachdb/cockroach/issues/12123, https://github.com/cockroachdb/cockroach/issues/22868)

To address these differences between cockroachdb  and Postgres, this PR performs different migrations for the two database. The Postgres migration is straight forward and what you would expect, but the cockroach migration has two main changes:
1. To change a primary key, use the recommended process from the cockroachdb docs to create a new table with the new primary key you want and then migrate the data.
2. In order to do 1, we needed to do the new table renaming in a separate transaction from the data migration.

Ref: SM-65

Change-Id: Idc9aee3ab57aa4d5570e3d2980afea853cd966bf
2020-03-20 14:39:44 -07:00
Isaac Hess
05da31a97b statreceiver: Add name of buffer for better logging/debugging
Change-Id: If845908543c951111ac13ac6262b4e7c50dcd22c
2020-03-20 15:00:20 +00:00
Egon Elbre
1b6ab173a8 private/context2: moved to storj.io/common/context2
Change-Id: Ic1dd1ed645ff3e1057c9b2b143e2c3ddf29d678e
2020-03-20 14:39:46 +00:00
Michal Niewrzal
d994584e88 cmd/storj-sim: remove removed flags from storj-sim gateway setup
Change-Id: I27d585740e8441b41bce5010480cc8e4f5deb9ce
2020-03-20 12:09:55 +01:00
Jennifer Johnson
9b78473c0c satellitedb: adds vetted_at nullable timestamp to nodes table
Change-Id: I42d5a396b4eecbad26b683c6aee51e043d2ff034
2020-03-20 01:37:28 +00:00
JT Olio
b2590cf283
bump uplink to 1.0.0 (#3816)
What: bumps uplink to 1.0.0
Why: we just released it!
2020-03-19 15:38:27 -06:00
Jennifer Johnson
699b635e5d satellite/overlay: rename newNodePercentage to newNodeFraction
Change-Id: Ie66de91f88183b44de0773589e83e4ade9aa997a
2020-03-19 20:09:32 +00:00
Egon Elbre
0781a91ff4 linksharing: move to storj.io/linksharing
Change-Id: If3bca64f80de9c244414e2aff28bb8b3bf9e87a2
2020-03-19 16:49:00 +00:00
Egon Elbre
eb1d8aab96 satellite/metainfo/pointerverification: service for verifying pointers
This implements a service for pointer verification. This makes the
slightly clearer, because it's not part of metainfo.

It also adds a peer identity cache which reduces database calls and peer
identity decoding.

Change-Id: I45da40460d579c6f5fd74c69bccea215157aafda
2020-03-19 16:27:38 +00:00
Qweder93
8597e6b512 storagenode/console/api period payment api extended
Change-Id: I18ec331c6a684e3a9351e3c917bacdb8b8f18c28
2020-03-19 16:51:31 +02:00
Qweder93
0df586c3a8 satellitedb/heldamount updated, tests added + storagenode console updated
Change-Id: I10f568a426d0fc42069d025de2accbef5b26dc0c
2020-03-19 15:37:45 +02:00
crawter
fde5c3542b storagenode/console/api: period payStub api extended
Change-Id: I624bbf7a9640f9df97789bea109201cbfb556753
2020-03-19 14:42:02 +02:00
Kaloyan Raev
10b032e484 libuplink: return deleted bucket/object (step 4)
Switch back to the original DeleteBucket and DeleteObject methods.

Next step: remove the DeleteBucketReturnDeleted and
DeleteObjectReturnDeleted from storj.io/uplink.

Change-Id: I273a305326d411e51ce354ce72fcc6ecadf4dd5f
2020-03-19 13:32:07 +02:00
Kaloyan Raev
78b253c774 libuplink: return deleted bucket/object (step 2)
step 1 in https://review.dev.storj.io/c/storj/uplink/+/1236

Now the old libuplink uses the temporary DeleteBucketReturnDeleted and
DeleteObjectReturnDeleted methods. This way, in the next step, we will
be able to change the DeleteBucket and DeleteObject methods to return
the deleted bucket/object.

Change-Id: I2e638be1960bca6ce1456c92849fcdd6d93e5252
2020-03-18 17:26:23 +00:00
Jessica Grebenschikov
5142874144 satellite/gc: move garbage collection to its own process
Change-Id: I7235aa83f7c641e31c62ba9d42192b2232dca4a5
2020-03-18 16:44:01 +00:00
Michal Niewrzal
19685ad81a go.mod: bump common and uplink versions
Change-Id: I9894a1bdbc11b9785d5c0744675d720545f398c0
2020-03-18 15:25:54 +00:00
Egon Elbre
09e0f3de63 satellite/metainfo/piecedeletion: add Service
Change-Id: Id7e32ed569701fa0be66f9527c43a67052994570
2020-03-18 14:50:08 +00:00
Jeff Wendling
115f4559e5 satellite/orders: more efficient processing of orders
by doing an indexed anti-join we're able to reduce the time to
select the pending orders by over 10x on postgres. this should
help us process pending orders much more quickly.

it probably won't do as good a job on cockroach because it does
not do an indexed anti-join and instead does a hash join after
scanning the entire consumed serials table. we should either
remove orders entirely or try to make that more efficient
when necessary.

Change-Id: I8ca0535acd21c51e74955b24c9b86d20e4f2ff9c
2020-03-18 09:03:30 +00:00
Matt Robinson
bd4982e249
test/backwards-compatibility: Exclude rc tags from testing (#3787)
Change-Id: Id486709306c5c75a262ea17410ae641be53df8ed

Co-authored-by: Ivan Fraixedes <ivan@fraixed.es>
Co-authored-by: littleskunk <jens.heimbuerge@googlemail.com>
2020-03-18 00:12:05 +01:00