Commit Graph

7114 Commits

Author SHA1 Message Date
Kaloyan Raev
e709112eeb satellite/metainfo: allow list permission for GetObject and GetObjectIPs
GetObject and GetObjectIPs are invoked by the Linksharing service to
display the shared object and its map. These two endpoint currently
require read permission.

There is a use case where an object can be shared with an access grant
that has only list permission. In such a case, the expectation is that
the linksharing service would still display the metadata of the shared
object (name, size, map), but the content would be still inaccessible.
See https://github.com/storj/gateway-mt/issues/209 for details.

This change allows GetObject and GetObjectIPs to require either read or
list permission to support the described use case.

Change-Id: I3477edc7bf8990e9848482890da047094c875d09
2022-11-07 10:59:43 +00:00
Jeremy Wharton
b34b5c4177 satellite/payments/stripecoinpayments: do not reference nil invoice
When a customer has no pending line items, an invoice will not be
generated for them and the Stripe client method responsible for
creating new invoices returns nil. This change adds a nil check to
account for this possibility to ensure that no panics are caused
by attempted processing of the invoice.

Change-Id: Id184d027d7447f0ef876db58601ab6cf63927fc5
2022-11-05 18:09:52 -05:00
Michal Niewrzal
b2dc8211d6 satellite/gracefulexit: observer cleanup
Some changes to make code cleaner and easier to adopt to new ranged
loop.

* removed unneeded mutex
* reorganize constructor args
* avoid creating the same redundancy scheme for each segment piece

Change-Id: I81f3f6597147fc515516949db3ce796a60b1c8a0
2022-11-05 12:10:44 +00:00
Moby von Briesen
cc858f4e91 satellite/console: Turn new billing screens on by default
Set default value for console.new-billing-screen to true.

Change-Id: I0f7f9663af4b4dc384d2bda946b4853593151cfa
2022-11-04 19:41:54 +00:00
Vitalii
a342edd1c2 web/multinode: fixed linter
Fixed web linter

Issue:
https://github.com/storj/storj/issues/5158

Change-Id: Ibc0b8e9f241e77e833645d9fa114bf5742c9f81c
2022-11-04 19:06:21 +00:00
Vitalii
8268933b56 web/storagenode: fix linter
Fixed web linter.

Issue:
https://github.com/storj/storj/issues/5158

Change-Id: Ia63e3e6e7352a99e902c9ed08a4bcfd75059e943
2022-11-04 18:32:26 +00:00
Clement Sam
10f59f8633 web/storagenode: update new suspension and audit threshold on dashboard
The new audit scoring sets disqualification threshold to <96:
https://forum.storj.io/t/new-audit-scoring-is-live/19466/24.

This changes applies this update to the dashboard,
same colors but different thresholds now:
<96 = disqualification
<99 = warning

previously:
<60 = disqualification
<95 = warning

Closes https://github.com/storj/storj/issues/5113

Change-Id: Iaf61e5034101fe5d4d70df6b72eb5f4ccb706e5b
2022-11-04 17:47:05 +00:00
Cameron
cb0c359b81 satellite/overlay: insert DQ node events for stray nodes
Change-Id: I99da11e506ab7f6bcebdb08a5815078a3297c932
2022-11-04 15:48:17 +00:00
Cameron
74ddfab810 satellite/overlay: insert DQ event into node events in overlay.DisqualifyNode
Also, return node email from overlaycache db DisqualifyNode to be used
in node events insertion

Change-Id: I41534cf01351c1690c3966a8055c5fe6fcf0d6a6
2022-11-04 15:18:31 +00:00
Cameron
f42e5049c9 satellite/satellitedb: add log line for node events insertions
Change-Id: I3271885cfd130128c240c9e9993e835fedb1ab7f
2022-11-03 16:09:23 -04:00
Jeremy Wharton
e34826c168 docs/blueprints: add automatic account freezing/unfreezing blueprint
This blueprint describes a way to allow user accounts to be
automatically restricted after failing to pay and unrestricted after
payment has been collected.

Change-Id: I4b64eb8ef6bf83603be7ad7d7bcb78f2d885c2a4
2022-11-03 16:56:28 +00:00
Egon Elbre
5ee27fc42a mod: bump storj.io/uplink
Change-Id: I9692c3c8dd266b29d23a782f6aff3594a6423de3
2022-11-03 14:06:17 +00:00
Erik van Velzen
337b72f310 satellite/metabase/rangedloop: uuid range pairs
Pair uuid's to create ranges. Will be used to parallelize the segment
loop.

Part of https://github.com/storj/storj/issues/5223

Change-Id: I73e2fb8a2cd379b840864449b6251b48feeb7b66
2022-11-03 11:15:56 +00:00
Cameron
68fe26ebe5 satelite/overlay: insert reputation events into node events
Insert reputation event into node events if reputation change occurs.

Change-Id: If1c5526092cb6834fe2faa6aa6e0306d4d88a4b7
2022-11-02 18:32:20 +00:00
Márton Elek
603bc019e1 build: use specific tag for nightly deployments
This patch is required to fix the nightly deployment:

 * We need to use the exact docker image tag what we built earlier
 * Migration should be full instead of snapshot (snapshot couldn't update existing, but older dbs)

Change-Id: Id2a2070638072a7b0021326326b0d53533817168
2022-11-02 17:57:20 +00:00
Vitalii
7ac1e38e28 web/satellite: reworked new Add coupon modal to use common VModal component
Reworked new Add coupon modal to use common VModal component to improve UX.

Issue:
https://github.com/storj/storj/issues/5104

Change-Id: Idd186ffd30a529761d15052bee4fac7cee4f5814
2022-11-02 19:16:25 +02:00
Vitalii
7f285c2eee web/satellite: reworked old add coupon code modal to use common VModal component
Reworked old Add coupon code modal to use common VModal component to improve UX.

Issue:
https://github.com/storj/storj/issues/5104

Change-Id: I4e1b38cfa2a88bacb4bcc58b837bc5b4319aca3d
2022-11-02 19:04:49 +02:00
Vitalii
db864a8605 web/satellite: reworked delete bucket modal to use common VModal component
Reworked delete bucket modal to use common VModal component to improve UX.

Issue:
https://github.com/storj/storj/issues/5104

Change-Id: I74acd07a0cb3f7e0231f88bf1255de9ac000ace5
2022-11-02 19:04:49 +02:00
Vitalii
8152920a51 web/satellite: reworked share object modal to use common VModal component
Reworked share object modal to use common VModal component to improve UX.

ISSUE:
https://github.com/storj/storj/issues/5104

Change-Id: Ie07550bb3e116651b08721298994eb0c867cf6f4
2022-11-02 19:04:49 +02:00
Cameron
865974950d satellite/overlay: insert node online events into node events table
Instead of sending emails at the time the node is seen to be back
online, we have decided to send the event to the node events table,
which will initiate the email sending process at some point.

Change-Id: Id756209498112579de8e78ee20ad2df54571a617
2022-11-02 16:26:19 +00:00
Cameron
f06da25c3d satellite/overlay: add nodeevents.DB to satellite overlay service
Add nodeevents.DB to satellite overlay service so we can insert node
events into the nodeevents DB.

Change-Id: I642c0ccc9941ecdb08cb22d5c8cf701959a55156
2022-11-02 15:56:37 +00:00
Michal Niewrzal
d21bbab2b2 satellite: fix metabase configuration wiring
New flag 'MultipleVersions' was not correctly passed from metainfo
configuration to metabase configuration. Because configuration was
set correctly for unit tests we didn't catch it and issue was found
while testing on QA satellite.

This change reduce number of places where new metabase flags needs
to be propagated from metainfo configuration to avoid problems with
setting new flags in the future.

Fixes https://github.com/storj/storj/issues/5274

Change-Id: I74bc122649febefd87f665be2fba628f6bfd9044
2022-11-02 15:17:34 +00:00
Egon Elbre
947cdf0fb4 ci: fixes for new Go version
We need to make exceptions for older uplink versions, because it does
not compile with newer Go versions due to quic dependency.

Change-Id: I3e073694f0942029c56740f0689088058ee068c3
2022-11-02 15:17:46 +02:00
Qweder93
5e5d6ecf6c satellite/{accounting, buckets}: added feature that allows to use custom SQL query instead of object loop for buckets accounting
since amount of objects is growing and looping through all of them
starts taking lot of time, we are switching for SQL query to do it
in chunks of tallies per bucket. 2nd part of issue fix.

Closes https://github.com/storj/team-metainfo/issues/125

Change-Id: Ia26bcac0a7e2c6503df9ebbf4817a636841d3284
2022-11-02 09:01:33 +00:00
Ethan
9a09d8920e satellite/gc: Upload bloomfilters with prefix and update LATEST when complete
Change he bloomfilter generation process to prefix the objects with a date and update the LATEST object with the prefix.  The sender will read the LATEST file to get the prefix to process.

Change-Id: Iae0d3c49015d57f391d87789fb799a7d774066bf
2022-11-01 21:24:46 +00:00
Ethan
4efde65c9e satellite/gc: Optionally run the GC bloomfilter process once, instead of in a loop
The current deployment strategy requires that the GC bloomfilter generation process executes only once and exits.

Change-Id: I952991f126596aa165d1f2e9fce6f8548c21bdba
2022-11-01 18:19:40 +00:00
Wilfred Asomani
4edef9e05c web/satellite: add AB test for new upgrade banner
This change adds AB testing for a new upgrade banner and sends a hit event when "Upgrade" is clicked.

Change-Id: Ie463e224af5b0cb74a601b68eedb2b34f9089fd7
2022-11-01 16:39:52 +00:00
Cameron
11e229cc7f satellite/nodeevents: implement node events DB
Implement node events DB with Insert and GetLatestByEmailAndEvent. Get
was changed to GetLatestByEmailAndEvent so we can verify items are being
inserted into the table without needing the ID, which is not available
to us in the tests.

Change-Id: I4abe63631c44774cd7e795fbab0cbab4d801db4c
2022-11-01 16:03:14 +00:00
Egon Elbre
4b05beb3f5 cmd/satellite: add 'repair-segment' command
This mostly wiring up the necessary systems.

Change-Id: I9e03b3240235ca8e4cc51ddf6a41e27d9dbc198c
2022-11-01 14:31:39 +00:00
Erik van Velzen
e09c8198d5 web/satellite/static/emails: set dcs support address
Helene: support@storj.io is for node operators. Support for DCS accounts
should go to supportdcs@storj.io.

Fixes https://github.com/storj/customer-issues/issues/354

Change-Id: Icfd437004defcee6d49f19e58bdea16e7e99b7d8
2022-11-01 00:06:24 +00:00
Lizzy Thomson
23917e51f2 satellite: invalid coupon code returns 400 status code
fixes: https://github.com/storj/storj/issues/4782

Change-Id: Idcc00202ed91a10c280f706e14d4901bf2f61541
2022-10-31 13:43:09 -06:00
Cameron
f76abb3b3d satellite/satellitedb: add dbx read method to node_events
Change-Id: I9cfcb467051921a9b54370e426e618a76ac4197e
2022-10-31 16:31:31 +00:00
Egon Elbre
aeb645d32b all: replace deprecated ioutil
Change-Id: I60b0bbf5b68b066e2d44b8b99438594d600a3c2d
2022-10-31 15:50:41 +00:00
Erik van Velzen
c25391e976 satellite/metabase/rangedloop: uuid generation
Create helper function to generate ranges of UUIDs, for parallelizing
the segment loop.

Change-Id: I17dbc1d5effe27fc1a3491aa9ca56c692bd95df0
2022-10-31 16:05:41 +01:00
Jennifer Johnson
1da7520a88 cmd/tools/node-cleanup: swap deletion for atredis emails with last contact and created at timestamps
Change-Id: I4acde09e00ea4e5450dcb0fd26bbb2c3afa767f6
2022-10-31 14:13:56 +00:00
Igor
982b9a1844
scripts/draft-release.sh: add missing quotes 2022-10-31 14:15:49 +02:00
Cameron
80bfa55b8e satellite/satellitedb: edit node_events migration to have different PK
Change node_events schema to use an id column as primary key rather than
node id because there can be multiple events per node id.

Change-Id: I518d8ef9ea658764876483e282a4058d3c4910f4
2022-10-28 22:59:28 +00:00
Ethan
403003c294 storj: bump storj common package version
upgrade common to address connection leak

Change-Id: I4b8332120d803cace2856ef0248d73fa99469424
2022-10-28 21:38:47 +00:00
Cameron
6bf54bfaef satellite/satellitedb: add new table node_events
Add new table for node events. We can use this to notify node
operators of certain node events. Further, we can squash events for
multiple nodes with the same email into a single notification.

Change-Id: Icea6dd939df8fe4a98806bd79c014e21d239c43e
2022-10-28 19:16:25 +00:00
Clement Sam
d625eb85fd storagenode: use bytes instead of bytes*hour unit for used space graph
Closes https://github.com/storj/storj/issues/5146

Change-Id: I1b135da81a68193b5b50c761088d79471ca3a2fe
2022-10-28 18:42:45 +00:00
JT Olio
ec9d63d0ca mod: bump dependencies
bump storj.io/uplink to include small telemetry changes

Change-Id: Ia7767244e4eb8d1a86656a1e718576aa4b46218b
2022-10-28 14:53:42 +00:00
Malcolm Bouzi
c565c5e5fc web/satellite: update buckets page table to match designs.
see: https://github.com/storj/storj/issues/5175

Change-Id: I58de009f743196509ae39612a4201d36b7225e96
2022-10-28 10:09:24 -04:00
NickolaiYurchenko
d0cd39af7b web/satellite: session timer reset added while uploading files added
problem: when uploading large files we face a problem with session expire while waiting for file to upload
solution: added callback to progress event to reset session timer
timer is refreshed only when less than 3 min left

Change-Id: Ieab9250be61757762a884165902cd48f90048409
2022-10-28 09:23:50 +00:00
paul cannon
0c3dd44490 docs: audit-scaling: clarify process structure
and clarify some related implementation details.

Most notably, this change clarifies that the verification audit workers
and reverification audit workers are meant to live in a process or
processes separate from the satellite core, and outlines an extra queue
that will be used for communication with the core.

It's not entirely clear to me that this is the right approach; we would
save some fairly significant implementation time by leaving both types
of worker in the core. That would make it necessary to reconfigure and
restart the core when we wanted to change the number of verification
and/or reverification workers, and scaling would be limited to the
computational capacity of the core vm, but maybe those are acceptable
conditions.

Another option would be to leave the Verifier workers in the core, and
having a separate process for Reverifiers. That would be sort of a
middle way between the two above.

Change-Id: Ida12e423b94ef6088733b13d5cc58bdb78f2e93f
2022-10-27 20:00:36 +00:00
paul cannon
c54c45c9c7 satellite/audit: new ReverifyPiece implementation
ReverifyPiece() is not currently hooked up to anything, but is planned
to take the place of audit.(*Verifier).Reverify().

ReverifyPiece() works by downloading one piece in its entirety, rather
than pulling an entire stripe across many nodes.

Change-Id: Ie2c680f4d3c3b65273a72466a3f9f55c115b0311
2022-10-27 16:06:21 +00:00
paul cannon
9c67f62fe3 satellite/satellitedb: add table for reverify queue
This table will be used as a queue for pieces that need to be reverified
(a regular audit timed out on the owning node, so now that node is
contained and we need to validate the piece before un-containing it).

Refs: https://github.com/storj/storj/issues/5228

Change-Id: I5dcd26b6adced8674cbd81884c1543a61ea9d4c8
2022-10-27 15:28:47 +00:00
Wilfred Asomani
2dc2669e22 console/abTesting: add support for AB testing
This change adds support for AB testing using flagship.io

Change-Id: I3e12f5d6cd7248d69adc2c684e4bcff2aadda1df
2022-10-27 10:57:12 +00:00
Kaloyan Raev
689732188b satellite/metainfo: remove duplicate permission check in BeginCopyObject
BeginCopyObject checks twice for write permission in the destination
bucket. One check should be enough.

Change-Id: I3d5935d34f69cd48eaaf00d0117683edfdcefc05
2022-10-27 10:19:05 +00:00
Márton Elek
29aac75d80 build: provides earthfile for nightly build
Earthly is a build tool, it uses buildkitd to create reproducable and highly cacheable builds.

It is used by a new experimental nightly build to easily create storj-up images. (but can be used for any ad-hoc storj-up cluster to create the images).

To make the nightly more robust, I would prefer to commit the helper files (today I do a rebase every time, but sometimes it fails).

More detailed information about Earthly can be found at https://earthly.dev or https://www.youtube.com/watch?v=nChpMEdOaCQ

Change-Id: I683601e0558aca53b45ed3819c46c909534f8b15
2022-10-27 09:25:17 +00:00
Cameron
8c8688ca6b satellite/overlay: return email as part of NodeReputation
Make email accessible for email sending after reputation updates

Change-Id: I760feee0f6ca58b76a2955a04c0c366c618656bb
2022-10-26 17:33:22 +00:00