Commit Graph

4870 Commits

Author SHA1 Message Date
Egon Elbre
b7a0739219 satellite/overlay: use DownloadSelectionCache for getting node IPs
Change-Id: Ib8f4eedb2bf465767050693a1e961b37a294ca06
2021-01-29 16:47:10 +02:00
Egon Elbre
54e01d37f9 satellite/overlay: add DownloadSelectionCache
Change-Id: Ic0779280172325f8d03f55a2e9673722f72bdd44
2021-01-29 16:47:06 +02:00
Cameron Ayer
89e682b4d7 satellite/repair/checker: add 29/80/130-52 to default repair overrides
Change-Id: I2e5a7538fdf33f3869fcb65fc88f7abb10faad79
2021-01-28 16:55:16 -05:00
Yingrong Zhao
52d6852e58 pkg/server: add retry logic for random port assignment
Change-Id: I70464e344a79dce8eadb9513d2a990faf3b2cca8
2021-01-28 14:44:22 -06:00
Brandon Iglesias
fff10b041c
quick update to readme (#4033)
Co-authored-by: Stefan Benten <mail@stefan-benten.de>
2021-01-28 10:16:57 -05:00
Egon Elbre
19e3dc4ec0 satellite/overlay: rename NodeSelectionCache to UploadSelectionCache
It wasn't obvious that NodeSelectionCache was only for uploads.

Change-Id: Ifeeaa6fdb50a4b7916245b48d8634d70ac54459c
2021-01-28 14:56:53 +02:00
Egon Elbre
a97b5c8552 go.mod: bump quic to master, such it compiles with Go 1.16
Change-Id: I4af35cd561955094a34fe3c96b0c532f325ac505
2021-01-28 11:13:02 +02:00
Kaloyan Raev
4d32bdaefb satellite/satellitedb: drop bucket_metainfos_name_project_id_key index
This index is obsolete and duplicates a similiar (project_id, name)
index on the same table.

Moreover, it might confuse CockroachDB which of the two index to use,
which may might affect DB performance.

Change-Id: If8d1df8347714942cea9dca82864ba5f4973bed3
2021-01-28 09:06:22 +02:00
Yingrong Zhao
824fd6f131 pkg/quic: add backward compatibility for qtls
Change-Id: I2560074ece5b61d9ddb236269172325e1b5de83e
2021-01-27 15:24:50 -05:00
Jeff Wendling
ca86820b8b satellite/snopayouts: use dbx + some refactorings
Change-Id: I8f3973d2377f071bcea2f61e0fc21d913ffa7ea8
2021-01-27 17:59:16 +00:00
Jeff Wendling
66e15fb7f1 satellite/compensation: remove ytd paid amounts
they aren't right and we aren't using them.

Change-Id: I5ca024e38d055696696886278863e941b5bc51bf
2021-01-27 17:31:01 +00:00
Yingrong Zhao
02845e7b8f pkg/server,private/testplanet: start to listen on quic
This PR introduces a new listener that can listen for quic traffic on
both storagenodes and satellites.

Change-Id: I5eb5bc82c37dde20d3be2ec8fa5f69c18fae0af0
2021-01-27 11:03:42 -05:00
Stefan Benten
f18cb24522
.clabot: add harrymaurya05 and gregoirevda (#4032)
Adding both as they signed the CLA.
2021-01-27 15:34:23 +01:00
Caleb Case
d2148edcdb
cmd/uplink/cmd/setup.go: Access Grant vs API Key error messaging
Provide a clearer error message to users who confuse the API Key with
the Access Grant and suggest the right command to them.

Change-Id: If73ae8cde140b68a19f4cfc3f59bb88a3b74c9c1
2021-01-26 14:09:59 -05:00
crawter
93e8413617 web/mnd: my nodes page ui
Change-Id: I96403a6ae6ce232eee08208895e03c6a7aed74cb
2021-01-26 17:45:07 +00:00
crawter
9820145e14 web/mnd: nodes domain, api and store
Change-Id: I022c5153dfc85a25eebce6e8ba91b97e906736cb
2021-01-26 17:25:57 +00:00
crawter
135d846aff mnd/console/server: endpoint with index.html added
Change-Id: Ic5154feaa995bf5c26c851024079f7e82612f306
2021-01-26 17:16:11 +00:00
Malcolm Bouzi
24d60384c5
satellite/satellitedb: add columns for professional users (#4028)
Co-authored-by: Egon Elbre <egonelbre@gmail.com>
2021-01-26 11:38:53 -05:00
Isaac Hess
c92bda7e75 tally monkit: change location to monitor piecesize
When we observed the value for total piecesizes stored in the network,
we were doing it after converting them to byte-hours, rather than using
the actual piece sizes. This fixes that issue.

Change-Id: I1564d21b519f70eb59f298d97dbd777baf127723
2021-01-26 15:37:02 +00:00
littleskunk
0b2568d712
satellite/overlay/straynodes: increase development duration without contact
Stopping storj-sim for over 5 minutes caused nodes to be disqualified. Set development max duration without contact to 300 days.
2021-01-26 12:24:39 +02:00
Michał Niewrzał
3fc0d2a83e satellite/metainfo: add testing method from multipart-upload branch
We wanto have single uplink branch for standard and multipart-upload satellite but some tests are using helper methods from multipart. This change adds methods used by uplink test.

Change-Id: I82352ed56674ff7e8743b58061ba594018e78e3b
2021-01-26 09:13:12 +00:00
Michał Niewrzał
50bea3ab1a satellite/metainfo: adjust tests to changes in uplink
One of uplink method changed its signature and we need to fix test on satellite side.

Change-Id: Ib89ea6aa25c57bac11bc3e0e9c2c89a4b9debd7c
2021-01-26 08:54:35 +00:00
Moby von Briesen
8263f18321 satellite/console: Add graphql query for owned projects
Change-Id: If47183d46cb7552ecdddbb3e536c36d958fad6d0
2021-01-25 17:43:04 +00:00
Qweder93
c139cbd76b storagenode/payouts: fix CurrentMonthExpectations timezone handling. Estimations based on node's join date.
On servers with non-UTC it would have calculated a different month boundary.
If node joined in current month calculations will be related on amount of days node've been working.

Change-Id: Ie572b197f50c6cdff5a044a53dfb5b9138f82f24
2021-01-25 19:03:30 +02:00
Ivan Fraixedes
49c8e94480
scripts: Add test Satellite working w/o Redis
Create a storj-sim test that checks that uplinks operations works when
satellite runs and can connect to Redis and when it cannot connect to
simulate a Redis downtime. Also verifies that the satellite can start
despite of Redis being downtime.

This test currently doesn't pass and it will be the one used to verify
the work that has to be done to make sure that the satellite allow the
clients to perform their operations despite of Redis being unavailable.
We require these changes before we deploy any customer face satellite on
a multi-region architecture.

NOTE that this test will be added later on to Jenkins to run this test
every time that we apply changes and at that time we'll see if it has to
be adjusted for being able to run on Jenkins because as it's now it may
not work because the scripts start and stop a Redis docker container.

Change-Id: I22acb22f0ca594583e36b45c88f8c03bac73b329
2021-01-25 16:02:59 +01:00
Egon Elbre
c44368c489 cmd/storj-sim: fix port assignment for multiple satellites
Currently first satellite GC would've conflicted with second satellites
public RPC port. Instead assign "satellite workers" a new peer class.

Change-Id: Id6bdaa17243556482e88da708c5147149788f6be
2021-01-22 14:08:39 +02:00
Yingrong Zhao
ae03a0540c pkg/quic: add quic implementation
Due to the issues with the licensing and go version requirement by the
quic-go library, we can not introduce the library into libuplink at the
moment. Therefore, we decided to put our quic integration code into
storj/storj. It will unblock us from rolling out quic to storagenodes.

Change-Id: If9d765da10c45947a4e3a3a11e4679bab69bcf08
2021-01-21 21:28:30 +00:00
Jessica Grebenschikov
b7d8dee5e9 satellite/console/wasm: add js tests
Change-Id: I8b1b0e81500836e0408e0517edb6c696698ab5f7
2021-01-21 20:18:03 +00:00
Jessica Grebenschikov
1f1d9fce58 satellite/console/wasm: add test to confirm wasm size isnt growing
Change-Id: I975a9f8ac3f6b98cc213140fdd7a99557efe14c8
2021-01-21 15:48:49 +00:00
VitaliiShpital
6c3b7a233e web/satellite: add beta messaging for EU and US2 sats
WHAT:
add a banner with a message saying: gateway MT is in beta for EU and US2 sats

WHY:
user experience?

Change-Id: I0964c4499cdaf12dbd9dba082910fff4ff6f9a12
2021-01-21 15:05:18 +00:00
Malcolm Bouzi
fa2dae0aca
Pin Project Owner in Team tab to top of members list (#4021)
* web/satellite: Pin Project Owner in Team tab to top of members list

* web/satellite:Pin Project Owner to top of members list
- input missing semicolon

* web/satellite: Pin Project Owner to top of members list
- refactor project owner variable instantion
- check for existance of owner before returning members list

* web/satellite: Pin project owner to top of members list
- changed returned members list variable name
- add comment explaining unshift logic
2021-01-21 10:03:02 -05:00
Stefan Benten
ff102d6ffe
Makefile: update to go v1.15.7 (#4026) 2021-01-20 19:08:35 +01:00
Moby von Briesen
0a48071854 satellite/console: Add pagination fields for ListProjectsByOwnerID
Add ProjectsCursor type for pagination
Add PageCount, CurrentPage, and TotalCount ProjectsPage
This allows us to mimic the logic of GetBucketTotals and the
implementation of BucketUsages in graphql for the new ProjectsByOwnerID
functionality.

Change-Id: I4e1613859085db65971b44fcacd9813d9ddad8eb
2021-01-20 16:15:29 +00:00
Brandon Iglesias
e6c0383477
adding hypernet to the list (#4025) 2021-01-20 10:52:29 -05:00
Cameron Ayer
d14607a5f7 satellite/{contact,nodestats,overlay,satellitedb}: remove references to total_uptime_count and uptime_success_count columns
Change-Id: I1f92022909bc564e9b1e31bf937fdfe7c16554de
2021-01-19 15:43:02 -05:00
Isaac Hess
1e0d57bf01 uplink: Add timeout to RegisterAccess
We add a timeout for the http client used to register the access with
the auth service. We have a hard-coded common default for now.

Change-Id: I50207ad83c9221b7cb61f39310e24b140b95673b
2021-01-19 20:14:58 +00:00
Cameron Ayer
75d828200c private,satellite: add chore to dq stray nodes
Full scope:
private/testplanet,satellite/{overlay,satellitedb}

Description:
In most cases, downtime tracking with audits will eventually lead
to DQ for nodes who are unresponsive. However, if a stray node has no
pieces, it will not be audited and will thus never be disqualified.
This chore will check for nodes who have not successfully been contacted
in some set time and DQ them.

There are some new flags for toggling DQ of stray nodes and the timeframes
for running the chore and how long nodes can go without contact.

Change-Id: Ic9d41fdbf214736798925e728245180fb3c55615
2021-01-19 14:21:56 -05:00
Ivan Fraixedes
2e34b631b1
cmd/satellite: Allow core & API without live accounting cache
Allow the satellite commands which uses the live accounting cache (core
and API) to run when at the time that its instantiated there is an error
connecting to the backend.

This prevent that if live accounting backend is down we can run these
services because:

1. The services must run despite of the cache backend being down
   although it may be degraded.
2. We may need to start new replicas of the services or the services in
   a different place while we are troubleshooting and fixing the cache
   backend system.
3. Our services may restart when the cache backend or the network
   connecting to it fails momentarily.

Change-Id: Ic93f9571bc0865c9488d64ab1356376fae797efc
2021-01-19 15:53:06 +01:00
Ivan Fraixedes
0f70574e3c
satellite/accounting/tally: Don't abort tally when cache is down
Tally shouldn't abort its cycle if the accounting cache return an error
because it isn't an essential requirement to update it.

Change-Id: I78fd2bd9cf253ddedfb9ada80c0fa2ddf438f647
2021-01-19 15:13:01 +01:00
Qweder93
6ba8f6c8a9 storanode, satellite: payout renamed to payouts, expected estimation payouts added, console api for audits reworked
Change-Id: I4aa5e99bffaa87d0a800a429a4c83aa498ad4b7b
2021-01-18 10:56:03 +00:00
Yaroslav Vorobiov
1ed7227521 versioncontrol: add process url resolver
Change-Id: Iddd0d6cd976e10cbed3def85af72259d3c14e717
2021-01-15 17:11:08 +00:00
Ivan Fraixedes
678b07b314
satellite: Fix typos & code formatting
Fix some typos in the doc comments and readdress some code formatting
applied automatically.

Change-Id: I605b4eff2e7c6c58227ecf16be4c1d26f5322eb6
2021-01-15 16:40:26 +01:00
Moby von Briesen
c24f84914c satellite/console: Add ability to list projects by owner ID
Listing projects by owner ID also includes the number of members in each
project.

Change-Id: I53a09674b60c199ef378943851bb0f164e92e4e2
2021-01-15 14:22:22 +00:00
Kaloyan Raev
424d2787eb go.mod: bump deps to uplink v1.4.5
Change-Id: Ic5a4efc78bb9c140a4104fd96c514477e225cba4
2021-01-14 17:51:29 +02:00
Egon Elbre
85fb964afe satellite/{metainfo,overlay}: improvements to GetObjectIPs
* Deduplicate NodeID list prior to fetching IPs.
* Use NodeSelectionCache for fetching reliable IPs.
* Return number of segements, reliable pieces and all pieces.

Change-Id: I13e679caab275488b4037624b840a4068dad9589
2021-01-14 09:12:45 +00:00
NickolaiYurchenko
889d2eaaea web/multinode: svg type error on build fix
Change-Id: If6249d36bd42790cdf37f5f99def5fadc956a935
2021-01-13 18:21:19 +00:00
Egon Elbre
d11c2b709e go.mod: bump storj.io/common
* Add missing endpoints.
* Fix deprecated packages and funcs.

Change-Id: I756090c46a4d15eabf6d413a593cdc64c5809bc7
2021-01-13 14:51:08 +00:00
Ivan Fraixedes
a4d06b9b1e satellite/metainfo: Don't response errors when Redis down
For being able to have resilient multi-region satellites we cannot stop
processing uploads/download client request when Redis isn't responding
properly.

These changes avoid to stop the processing of the client requests when
we cannot check if the client exceeds its storage or bandwidth limits
and we cannot update its used storage/bandwidth limits because Redis is
not responding successfully or the satellite database returns an error.

Change-Id: Ia7f12c07fc9ffdfad0e7ff052ff3fd81eca0f0e3
2021-01-13 14:30:44 +00:00
Ivan Fraixedes
a73c59bbdd
satellite/console/consoleweb: Change status codes usage limits
Respond to the HTTP clients which request the project usage limits with
different status codes depending of the error class returned by the
satellite/accounting Service.

Change-Id: I6f486ea55517f616c7cec81dbbe77e997484180f
2021-01-13 15:00:12 +01:00
Qweder93
a3b1059fe0 multinode/console: list node updated with total earned
Change-Id: I53a6ebd98acb7741f60d61178451575e7677c730
2021-01-12 23:01:49 +00:00