Commit Graph

5156 Commits

Author SHA1 Message Date
Ivan Fraixedes
0f70574e3c
satellite/accounting/tally: Don't abort tally when cache is down
Tally shouldn't abort its cycle if the accounting cache return an error
because it isn't an essential requirement to update it.

Change-Id: I78fd2bd9cf253ddedfb9ada80c0fa2ddf438f647
2021-01-19 15:13:01 +01:00
Egon Elbre
8e47adfad5 satellite/metainfo/metabase: optimize a query
Change-Id: I6395225f68cd8fbc8e3078c80ac6cb079c3c6a4f
2021-01-19 15:37:03 +02:00
Qweder93
6ba8f6c8a9 storanode, satellite: payout renamed to payouts, expected estimation payouts added, console api for audits reworked
Change-Id: I4aa5e99bffaa87d0a800a429a4c83aa498ad4b7b
2021-01-18 10:56:03 +00:00
Yaroslav Vorobiov
1ed7227521 versioncontrol: add process url resolver
Change-Id: Iddd0d6cd976e10cbed3def85af72259d3c14e717
2021-01-15 17:11:08 +00:00
Ivan Fraixedes
678b07b314
satellite: Fix typos & code formatting
Fix some typos in the doc comments and readdress some code formatting
applied automatically.

Change-Id: I605b4eff2e7c6c58227ecf16be4c1d26f5322eb6
2021-01-15 16:40:26 +01:00
Moby von Briesen
c24f84914c satellite/console: Add ability to list projects by owner ID
Listing projects by owner ID also includes the number of members in each
project.

Change-Id: I53a09674b60c199ef378943851bb0f164e92e4e2
2021-01-15 14:22:22 +00:00
Kaloyan Raev
424d2787eb go.mod: bump deps to uplink v1.4.5
Change-Id: Ic5a4efc78bb9c140a4104fd96c514477e225cba4
2021-01-14 17:51:29 +02:00
Michał Niewrzał
38beecc7ad satellite/metainfo: Override pending object on upload
On upload we need to override pending and committed object. This change is adjusting DeleteObjectAllVersions to delete both.

Change-Id: Ib66c2af207c618119f7bf0de7fa9d3e5145d8641
2021-01-14 15:16:02 +00:00
Michał Niewrzał
95320912b4 satellite/metainfo: simplify metainfo delete test
Simplifies tests for pending and committed objects.

Change-Id: I0c683198f8fce62898142c6c343aca985d91a77f
2021-01-14 15:06:06 +00:00
Kaloyan Raev
6dff40f5c5 Merge remote-tracking branch 'origin/main' into multipart-upload
Conflicts:
	go.mod
	go.sum
	satellite/metainfo/metainfo.go

Change-Id: Ib5c49f3c911c58319855a171f9ce73657da976d9
2021-01-14 14:33:59 +02:00
Egon Elbre
85fb964afe satellite/{metainfo,overlay}: improvements to GetObjectIPs
* Deduplicate NodeID list prior to fetching IPs.
* Use NodeSelectionCache for fetching reliable IPs.
* Return number of segements, reliable pieces and all pieces.

Change-Id: I13e679caab275488b4037624b840a4068dad9589
2021-01-14 09:12:45 +00:00
NickolaiYurchenko
889d2eaaea web/multinode: svg type error on build fix
Change-Id: If6249d36bd42790cdf37f5f99def5fadc956a935
2021-01-13 18:21:19 +00:00
Egon Elbre
d11c2b709e go.mod: bump storj.io/common
* Add missing endpoints.
* Fix deprecated packages and funcs.

Change-Id: I756090c46a4d15eabf6d413a593cdc64c5809bc7
2021-01-13 14:51:08 +00:00
Ivan Fraixedes
a4d06b9b1e satellite/metainfo: Don't response errors when Redis down
For being able to have resilient multi-region satellites we cannot stop
processing uploads/download client request when Redis isn't responding
properly.

These changes avoid to stop the processing of the client requests when
we cannot check if the client exceeds its storage or bandwidth limits
and we cannot update its used storage/bandwidth limits because Redis is
not responding successfully or the satellite database returns an error.

Change-Id: Ia7f12c07fc9ffdfad0e7ff052ff3fd81eca0f0e3
2021-01-13 14:30:44 +00:00
Ivan Fraixedes
a73c59bbdd
satellite/console/consoleweb: Change status codes usage limits
Respond to the HTTP clients which request the project usage limits with
different status codes depending of the error class returned by the
satellite/accounting Service.

Change-Id: I6f486ea55517f616c7cec81dbbe77e997484180f
2021-01-13 15:00:12 +01:00
Qweder93
a3b1059fe0 multinode/console: list node updated with total earned
Change-Id: I53a6ebd98acb7741f60d61178451575e7677c730
2021-01-12 23:01:49 +00:00
Cameron Ayer
0184d33e96 satellite/satellitedb: set default 0 on uptime columns
This is the first step in the removal of uptime columns on the
nodes table. These columns are no longer used:

uptime_success_count
total_uptime_count
uptime_reputation_alpha
uptime_reputation_beta

In order to avoid breaking backwards compatibility, we need to
remove all references to these columns before removing the columns
themselves from the database. However, since uptime_success_count
and total_uptime_count are NOT NULLABLE, we can't remove them from
the insert statements in the overlay. So we can't remove the columns
because of the references, and we can't remove the references because
the columns can't be null. What a pickle. To remedy this, we will set a
default on the columns. Then we should be able to remove them from the
insert statements

Change-Id: I75f6c56fb7897835bbf29869f86f39de1d9dd345
2021-01-12 17:44:37 +00:00
Ivan Fraixedes
ce26616647
satellite/accounting/live: Use Redis client directly
We have to adapt the live accounting to allow the packages that use it
to differentiate about errors for being able to ignore them and make our
satellite resilient to Redis downtime.

For differentiating errors we should make changes in the live accounting
but also in the storage/redis.Client, however, we may need to do some
dirty workarounds or break other parts of the implementation that
depends on it.

On the other hand we want to get rid of the storage/redis.Client because
it has more functionality that the one that we are using and some
process has been started to remove it.

Hence, we have refactored the live accounting to directly use the Redis
client library for later on (in a future commit) adapt the satellite for
being resilient to Redis downtime.

Last but not least, a test for expired bandwidth keys have been added
and with it a bug was spotted and fix it.

Change-Id: Ibd191522cd20f6a9a15e5ccb7beb83a678e530ff
2021-01-12 15:33:29 +01:00
Michał Niewrzał
ac058e5ecc metainfo-migration: basic pointerdb->metabase migrator
Change-Id: If183b9c99ce816521fcae6e7189a6f85ddd4eb48
2021-01-12 12:59:53 +00:00
JT Olio
1ad69b9f96 satellite: make external address calc more robust
Change-Id: Icbdc6fb4e3fc85076dcb7bcc4b3ec36baad308d2
2021-01-11 16:47:25 +00:00
Cameron Ayer
0403e99a5b satellite/{overlay,satellitedb}: remove unused methods for old downtime tracking
GetSuccessfulNodeNotCheckedInSince and GetOfflineNodesLimited are overlay methods
which were only used by the previous downtime tracking system which has been removed.
These methods should also be removed.

Change-Id: Idb829d742e1f987e095604423fff656fe581183e
2021-01-11 15:21:28 +00:00
Qweder93
cf56071bb4 storagenode/payout: error message from empty receipt removed
Change-Id: Iff77ca3a839404b541c687f52dd3ffd891723968
2021-01-11 16:02:07 +02:00
Egon Elbre
0df0e980a0 satellite/metainfo/metabase: fix FixedSegmentSize calculation
Commiting multipart object with same sized segments should always
end with FixedSegmentSize = -1.

Change-Id: Iedeafb763b62b2513de0a753dce20ddad5b32d90
2021-01-11 14:47:49 +02:00
Egon Elbre
24833465e6 satellite/metainfo/metabase: avoid magic constant
Change-Id: I4f01e38f67e18ae9cb9845a8e75a987acba66427
2021-01-11 10:22:21 +00:00
Michał Niewrzał
ec88d21a3c Merge 'main' branch.
Change-Id: I6e8162d1a6caf75e89c9f9c9f9522730aebf83ae
2021-01-11 10:26:58 +01:00
Michał Niewrzał
71ca18134b satellite/metainfo: reject downloading a segment from multipart object
Non-multipart uplink implementation is always trying to download object
by downloading last segment first (PartNumber=0, Index=-1) but this
approach won't work with multipart object. We need to reject such old
style request with reasonable message.

Change-Id: I9221e019933565a8d25136bdfef3e054320bac3d
2021-01-11 08:04:33 +00:00
Jessica Grebenschikov
1709117b0d satellite/console/wasm: add more unit tests
Change-Id: Ie134f8a08d690ce013039ed1a4e484f8b6a1a6d5
2021-01-08 18:50:29 +00:00
Jeff Wendling
2d2359667d satellite/orders: remove unused satelliteAddress field
Change-Id: I58091769472688433c48becc8dfc9029bddd87aa
2021-01-08 12:25:39 -05:00
Egon Elbre
ba5461562d satellite/orders: remove satellite address
SatelliteAddress in OrderLimit is not being used anymore and some
satellite addresses may consume too much bytes.

Change-Id: Ic7a0efe5b6211c2f3b91af67b293cde98b29d074
2021-01-08 16:57:36 +00:00
Egon Elbre
51731db121 satellite/orders: use smaller encrypted metadata
Avoid using project uuid string representation, because
it uses more bandwidth.

This reduces the encrypted metadata size from 118 -> 97 bytes.

Change-Id: Ic53a81b83acc065f24f28cd404f9c0b1fe592594
2021-01-08 16:40:31 +00:00
Yaroslav Vorobiov
6507f3ebc6 multinode/console: trusted satellites list api
Change-Id: I97bb9efb1d6cb7d456df0b86e66417c31018b762
2021-01-08 14:50:12 +02:00
Isaac Hess
db6b64c2ba Makefile: Custom gateway location for storj-sim
When running make install-sim we currently create a .build directory and
init a go module there. This change allows a user to pass in a custom
location for this to occur. If nothing is passed in it defaults to
current behavior.

Change-Id: I432dfc7bae412d8a8454a6b8b3dffece84a41147
2021-01-07 13:14:48 -07:00
Michał Niewrzał
da316c27aa satellite/metainfo/metabase: disable PlainSize validation
We need to accept segments with PlainSize == 0 to be backward compatible
for old uplinks.

Change-Id: I926abe751e9e0ad7e06e42a5301ab78a3f23aafa
2021-01-07 14:14:20 +00:00
JT Olio
35c61bd0e2 satellite: mark gateway credentials feature beta
Change-Id: I378d703bd17b842d56f7ca2c8bf24b47c4a07264
2021-01-06 15:18:43 -07:00
JT Olio
8907180e81 satellite: pass contact.external-address config to web ui
Change-Id: I54978aa34aa9eb98876fab6460a5737d718d6135
2021-01-06 10:11:20 -07:00
Kaloyan Raev
4fc61f7afa satellite/metainfo/metabase: allow commmitting objects > 2 GiB
The total_plain_size and total_encrypted_size columns in the objects
table were set as INT4, which limits the size of committed objects to
just 2 GiB.

This patch migrates the DB to change the type of these fields to INT8.

Change-Id: Iad7e7b44a652e6c5b8e17b80588637bb48390fe6
2021-01-06 18:23:03 +02:00
Bill Thorp
c0bc2123f5 uplink/cmd: fix uplink access register ability to use saved accesses
Jens noticed that 'uplink access register' wasn't working with named
accesses.

This was because GetNamedAccess was harcoded to use inspectCfg, which
in the case of 'uplink access register' wasn't being bound to the
config file.

Change-Id: I49403b45af28ad33408cfc5ec6545a395f0f080d
2021-01-06 15:08:51 +00:00
Egon Elbre
590bc529d2 cmd/storj-sim: fix local binary lookup
Change-Id: I6404fd05119c2a7113e4b0d7593a555327bef35f
2021-01-06 16:26:22 +02:00
Egon Elbre
b33d7a318e cmd/storj-sim: add -failfast
Fail all the processes immediately when one of the processes fails. This
is to make it more obvious that one of them has failed.

To disable failfast, use `-failfast=false`.

Change-Id: I2bbedf12fb653e42739d00273aa9ae515d34eda6
2021-01-06 16:26:22 +02:00
Kaloyan Raev
1e44be2900 go.mod: bump deps to uplink v1.4.4
Change-Id: I203259f0f324284ac2991e2cabf4bd471cd8ad5d
2021-01-06 13:07:17 +00:00
VitaliiShpital
2ad560e2dc web/satellite: satellite address field was added to continue in CLI step
WHAT:
satellite node url is now shown on continue in CLI step

WHY:
to make it usable

Change-Id: Iaf5fcbe3b9cfbebff904a16781ecd0225f3d4ba5
2021-01-05 21:32:17 +02:00
VitaliiShpital
6f065a8331 web/satellite: fix for all bucket names selecting
WHAT:
if all buckets are selected then we pass an empty array to web worker instead of all existing bucket names

WHY:
bug fix

Change-Id: I59b9e9ec1deb0abb3edeba7e2303e2b1d578935d
2021-01-05 16:13:00 +00:00
NickolaiYurchenko
f925d99d39 web/multinode: MyNodes page markup
Change-Id: Ie1bf57cde2dee6b72238ce7188a12dfb741bc031
2021-01-05 15:54:03 +00:00
NickolaiYurchenko
857155f5ac web/multinode: navigation panel added
Change-Id: Ia8d56cfce14476e1f4b59f4d13f239dda4a87b4f
2021-01-05 15:33:27 +00:00
NickolaiYurchenko
69a491aa6f web/multinode: add new node modal added
Change-Id: I337b39be065665cd531310171757ad8aab6be799
2021-01-05 15:10:54 +00:00
Yaroslav Vorobiov
5a43c86b68 multinode/console: list node satellite infos
Change-Id: Ic6cb8d1a6fd7637fdb7bf49e040c43ac30ab1bbf
2021-01-05 14:49:58 +00:00
Yaroslav Vorobiov
fb00d099cf multinode/console: list node infos
Change-Id: I5cac49feff2bac6fbd7ac61dfccffd672da8e8c0
2021-01-05 14:49:11 +00:00
NickolaiYurchenko
16a8b1da35 web/multinode: onboarding pages and common components
Change-Id: I71910108ff1cd40b7718dba1944589a2cc4852ef
2021-01-05 16:23:40 +02:00
NickolaiYurchenko
7703b4b691 web/multinode: common dropdown component added
Change-Id: I9e8b76abb73c561939daf9758ace0f82eed4f443
2021-01-05 13:17:31 +00:00
Egon Elbre
9cb4466eb0 cmd/storj-sim: use dev setup by default for consistency
Fixes bug when using release binaries together with storj-sim.

Change-Id: I077bedc1486ac85aa1f04fcc0ed4098cd313f2fc
2021-01-05 13:47:30 +02:00