Commit Graph

3186 Commits

Author SHA1 Message Date
Jeremy Wharton
7a17dc5e07 satellite/{console,satellitedb}: add methods for project invite table
This change immplements methods for interacting with the project member
invitations table.

Resolves #5766

Change-Id: I0090c50f9fde5bcdae4ebdaa72cdcaa84d341b54
2023-04-26 11:26:12 -05:00
Jeremy Wharton
defb9eae82 satellite/satellitedb: add table for project invitations
A table for storing pending project member invitations has been added.
This table is required to satisfy our new project invitation UX revamp.

References #5766

Change-Id: I6f948de66ed5b4dc81532564958ff7f48533cad2
2023-04-26 11:26:12 -05:00
dlamarmorgan
efcae857ba satellite/main,stripe/{client,service}: stripe balance invoice item cmd
Add a new billing command that will convert stripe customer balances into invoice items so that the charges can be processed normally by the invoicing workflow.

Change-Id: Iaa8350e7aca80a0f14e94eb8ef8b7d6ce0b5b3b8
2023-04-25 21:08:23 +00:00
dlamarmorgan
5fc493b276 metainfo: per-project and user-agent eventkit-based usage tracking
Add eventkit events for each satellite RPC type, tagged with user agent and public project id.

Change-Id: I838401683298d4594495de18563c55031e82f881
2023-04-25 15:13:43 +00:00
Vitalii
4d998970d4 satellite/admin: rework update user limits functionality
Fixed nil pointer dereference panic.
Updated naming conventions so that PUT request and GET response bodies are the same (bandwidth, storage and segment).
Allowed usage of notations like 150GB, 2TB for storage and bandwidth limits.
Updated tests.

Issue:
https://github.com/storj/storj/issues/5674

Change-Id: I7ac27c00721a9b4bf507afa34cb05c4475a809ad
2023-04-25 13:38:59 +00:00
Wilfred Asomani
260b71e70c satellite/{console,accountfreeze}: test freeze effects
This change adds more tests to the autofreeze chore and the freeze
service according to the testplan linked in the issue below.

Issue: https://github.com/storj/storj/issues/5738

Change-Id: Ib2afaa283961b2e7ef6fb6e5613ee083ac7d79eb
2023-04-25 13:04:37 +00:00
Michal Niewrzal
1aa24b9f0d satellite/audit: remove segments loop parts
We are switching completely to ranged loop.

https://github.com/storj/storj/issues/5368

Change-Id: I9cec0ac454f40f19d52c078a8b1870c4d192bd7a
2023-04-24 15:52:11 +00:00
Michal Niewrzal
6ac5bf0d7c satellite/gracefulexit: remove segments loop parts
We are switching completely to ranged loop.

https://github.com/storj/storj/issues/5368

Change-Id: Ia3e2d7879d91f7f5ffa99b8e8f108380e3b39f31
2023-04-24 15:00:26 +00:00
Michal Niewrzal
6a55682bc6 satellite/accounting/nodetally: remove segments loop parts
We are switching completely to ranged loop.

https://github.com/storj/storj/issues/5368

Change-Id: I6176a129ba14cf83fb635048d09e6748276b52a1
2023-04-24 14:25:53 +00:00
Egon Elbre
f40a0cb7ba satellite/*: use typed lrucache and ReadCache
Change-Id: Ieee535dd8735a95dd196a77413e4a25a6a72342c
2023-04-21 10:49:08 +00:00
Michal Niewrzal
fbfe5aaad7 satellite/metrics: remove code related to segments loop
We are switching completely to ranged loop.

Change-Id: I32120ef496addebec2de088fd10d0c1d02313c68
2023-04-20 13:47:22 +00:00
Egon Elbre
2405bc8f3b satellite/metabase: stop using the common error type
Updates https://github.com/storj/storj/issues/5291

Change-Id: I7b57a4b454d3619cb5d8ae4cd92f818ad2839c8b
2023-04-19 19:18:18 +03:00
Moby von Briesen
3679e29a93 satellite: Remove remaining references to "partner ID"
This field is deprecated in favor of UserAgent; Removing these
references is the final step necessary before dropping the columns from
the database.

https: //github.com/storj/storj/issues/5432
Change-Id: I3a6619170dcf382f82dc8eddb73b6547eaf636f0
2023-04-19 15:37:10 +00:00
Egon Elbre
41931970fd satellite/*: changes to stop using storj.ListDirection
Currently this duplicates the constants, because we cannot yet bump
common nor uplink.

Change-Id: I81818df8a9092e0b358eb946c31f2a42ac9dbf0a
2023-04-19 15:30:42 +03:00
Egon Elbre
54beea82da satellite/metabase: define a local ErrObjectNotFound
Updates https://github.com/storj/storj/issues/5291

Change-Id: If847cd95a9b500fb18535d20b46b1b1714021584
2023-04-19 12:03:01 +00:00
Márton Elek
9fbad53bbe
satellite/gracefulexit: extend GE tests using multple hash algo
Change-Id: I0b7accd5c0ae9ebb7adc7e5b8545bab639d70d36
2023-04-19 11:05:23 +02:00
Wilfred Asomani
9ac5183c7e satellite/consoleweb: improve freeze-status endpoint
This change adds a freeze status endpoint that gets whether a user is
frozen or warned.

Change-Id: I0bc498a128af969177c2cfcfda9b9ee3d8968305
2023-04-18 19:57:05 +00:00
Egon Elbre
daf5264f48 satellite/console/consoleweb: create a fallback error.html
Currently when error.html is not available, we should still start the
server and fallback to some simpler implementation. This template does
not require any external assets.

Change-Id: I76b660db988987e1e9ebadd966f60e149f26ff24
2023-04-18 14:36:51 +00:00
Michal Niewrzal
114eda6e87 satellite/metainfo: remove sleep from upload limit test
lrucache is now using time2 package and we can make expiration
test without using time.Sleep.

https://github.com/storj/storj/issues/5788

Change-Id: I48f2693c3db78fcf4e30e618bb3304be3625100c
2023-04-18 09:38:21 +02:00
dlamarmorgan
d402ed4b71 satellite/payments/stripe/service.go: fix typo in credit note memo
Remove the extra prepended '0x' from the wallet address.
Remove prepended '0x' from the transaction ID.

Change-Id: Id215536915fba62cc348aa2c3356ecc7898d68a2
2023-04-17 20:35:58 +00:00
Jeff Wendling
df9cc4d10f all: bump storj.io/{common,uplink,drpc}
Change-Id: Ie98a7eacc11a492a20c76636f8044c81b7ed580d
2023-04-17 19:59:30 +00:00
Wilfred Asomani
771d2269ab satellite/analytics: separate hubspot form for personal vs business
This change separates hubspot form submission for personal and business
accounts, with new company name and storage needs fields.

Issue: https://github.com/storj/storj-private/issues/220

Change-Id: Ieb0fb64f87614c7327dc5f894140fb8a54ededa0
2023-04-17 16:28:56 +00:00
Michal Niewrzal
3f543163c7 satellite/accounting: fix full table scan GetProjectObjectsSegments
New SQL queries for GetProjectObjectsSegments turns out to introduce
full table scan. This is fix for this problem.

Change-Id: Ieac22aafeb780168523a97e27c9283c9ac6a24c8
2023-04-17 14:18:52 +00:00
Jeremy Wharton
2193392e6b satellite/console/consoleweb: suppress index.html loading errors
This change reverses behavior added by 45d5a93 that made the server
return a 500 status code when the index.html file for the satellite
frontend couldn't be loaded. The presence of this file was previously
intentionally optional.

Change-Id: I875a171a37b735c3523eb5b13d83f084f1781053
2023-04-17 08:49:53 -05:00
Jeremy Wharton
45d5a93085 satellite/console/consoleweb: remove templating for index.html
Previously, we evaluated index.html as a template in order to insert
frontend config values into meta tags. Now that the frontend fetches
its config through the satellite API, this is no longer necessary.

Resolves #5494

Change-Id: Ic98507c5e16cd80317bd9c31d4b55abda0dd7e34
2023-04-14 14:09:42 -05:00
Jeremy Wharton
b3b619efc5 satellite/admin: use system-given port in OAuth test to fix flakiness
The test for the admin API's OAuth authorization behaviour has been
modified to use a random available port given by the system rather than
a hardcoded one. This prevents the test from accidentally using a port
that is already in use.

Change-Id: Iae017b2f397ae53f1a006bae1d0578d2ddfd0875
2023-04-14 13:14:19 -05:00
Jeremy Wharton
d62dd0b8e7 web/satellite: use frontend config in Vue components
References to the meta tag config values in Vue components have been
modified to instead refer to the frontend config fetched through the
satellite API.

References #5494

Change-Id: I00ecf81d4a0ba6bd07c827cecb2c689d923d67c0
2023-04-14 16:24:05 +00:00
Jeremy Wharton
34e6c5048b web/satellite,satellite/console/consoleweb: update static error pages
Our static error pages have been unified and updated to reflect our new
designs.

Change-Id: Ib7cf76e87c56180dd997a91a2829187dac99baff
2023-04-14 15:48:15 +00:00
Wilfred Asomani
4ee22e0ed8 satellite/admin: add tests to admin auth
This change tests authorization of the admin api.

Issue: https://github.com/storj/storj/issues/5699

Change-Id: Iecfe4c27a70ab1b48aeb5ed3251b51a3406140e8
2023-04-14 11:13:14 +00:00
Egon Elbre
eecb055dfd satellite/buckets: move Bucket definition
Move Bucket struct definition.

Updates https://github.com/storj/storj/issues/5291

Change-Id: I6bfc5ce287793ea479f2cb8b17878ba3cf6b63e0
2023-04-13 17:55:40 -04:00
Jeremy Wharton
34b2a369f7 satellite/payments/stripe: apply egress discount to project charges
The project charge information used by the satellite frontend to
compute project cost estimates has been updated to account for
configured egress discounts.

Resolves storj/storj-private#215

Change-Id: Ic90b015d65f5bea104ac96fb0cea545b3f9f1f8f
2023-04-13 17:34:50 +00:00
JT Olio
16ccffb6f7 satellite/metainfo: test to confirm that retried pieces can't be submitted with originals
closes https://github.com/storj/storj/issues/5709

Change-Id: I2030825d85987d97ae8c10f02ab1f70086cbcb97
2023-04-13 14:48:32 +00:00
Michal Niewrzal
a21afeddd1 satellite/payments/stripe: avoid full table scan while listing
Query to list (with pages) stripe customers were doing full table scan
because Offset clause was used. This refactoring changed listing to
use cursor instead Offset.

Change-Id: I14688e6c533bc932ba0d209a061562f080b4cf54
2023-04-13 12:36:31 +02:00
Jeremy Wharton
c92fdba972 satellite/payments/stripe: apply egress discount to invoice items
Invoicing has been modified to account for an egress discount ratio,
which specifies the fraction of the amount of data stored that egress
usage should be discounted.

The egress discount ratio measures discounted egress units per storage
unit-months, so a ratio of 0.5 with 2 MB-months of storage would
discount 1 MB of egress.

Resolves storj/storj-private#224

Change-Id: I43d7d6719391c303712c082709aef77249c65f62
2023-04-12 15:28:00 +00:00
Jeremy Wharton
6e9dae06d5 satellite/payments: add egress discount ratio to price override config
This change allows for specifying the ratio of free egress per unit
of storage within a price override configuration.

References storj/storj-private#215
References storj/storj-private#224

Change-Id: Ib1c79f77ec8bb11dd5b2f9dace13800b0b3ce942
2023-04-12 14:52:19 +00:00
Jeremy Wharton
59eab37b95 satellite/console/consoleweb: remove bucket usage report page
No component has referenced this page since 9dab10e and we do not
anticipate this changing, so this page can be safely removed.

Resolves #5768

Change-Id: I57acb5e4d0977d74df46aaf67606a19ec0f10bcf
2023-04-11 18:17:03 +00:00
Egon Elbre
8fba740332 storagenode/blobstore/testblobs: don't error checks in BadDB
We automatically start a chore to check whether the blobstore is
writeable and readable, however, we don't want to fail the tests due to
that reason. Usually we want to test some other failure.

There probably should be some nicer way to achieve this, but this is an
easier fix.

Change-Id: I77ada75329f88d3ea52edd2022e811e337c5255a
2023-04-11 16:14:31 +03:00
Andrew Harding
8c5924d6ea satellite/payments/billing: add bonus transactions
- only applies to storjscan transactions
- applies a 10% bonus by default
- bonus transactions have a distinct source "type" to allow for
  filtering on the frontend

Fixes: https://github.com/storj/storj/issues/5702

Change-Id: I32d65f776c58bcb41227ff5bc77a8e4cb62a9add
2023-04-10 16:26:56 +00:00
Wilfred Asomani
5b65e10563 satellite/console: enable session timeout configuration
This another account endpoint; patch /auth/account/settings. to handle
changing a user's settings, including their session timeout config.

Issue: https://github.com/storj/storj/issues/5560

Change-Id: I747b4e919cf7cef7c867ac9d282837ef51bed67e
2023-04-07 21:05:36 +00:00
paul cannon
915f3952af satellite/repair: repair pieces on the same last_net
We avoid putting more than one piece of a segment on the same /24
network (or /64 for ipv6). However, it is possible for multiple pieces
of the same segment to move to the same network over time. Nodes can
change addresses, or segments could be uploaded with dev settings, etc.
We will call such pieces "clumped", as they are clumped into the same
net, and are much more likely to be lost or preserved together.

This change teaches the repair checker to recognize segments which have
clumped pieces, and put them in the repair queue. It also teaches the
repair worker to repair such segments (treating clumped pieces as
"retrievable but unhealthy"; i.e., they will be replaced on new nodes if
possible).

Refs: https://github.com/storj/storj/issues/5391
Change-Id: Iaa9e339fee8f80f4ad39895438e9f18606338908
2023-04-06 17:34:25 +00:00
Michal Niewrzal
8a50a3baa3 satellite/payments: rename 'stripecoinpayments' package to 'stripe'
Automatic rename. May require some more cleanups later.

Change-Id: I18220a4278056d25c41fb137832bb81f2b876ac1
2023-04-06 16:51:43 +00:00
Michal Niewrzal
31f5e2cb65 satellite/metainfo: limit uploads to the same location
We would like to have ability to limit burst uploads to the single
object (the same location). This change we are limiting such upload to
one per second.

Change-Id: Ib9351df1017cbc07d7fc2f846c2dbdbfcd3a360c
2023-04-06 15:49:11 +00:00
Egon Elbre
48256c91b5 storage: move errors to better locations
Change-Id: Ia44570949a8f6bb50220dc838c5b6aa21e851a4d
2023-04-06 17:26:29 +03:00
Egon Elbre
f5020de57c storagenode/blobstore: move blob store logic
The blobstore implementation is entirely related to storagenode, so the
rightful place is together with the storagenode implementation.

Fixes https://github.com/storj/storj/issues/5754

Change-Id: Ie6637b0262cf37af6c3e558556c7604d9dc3613d
2023-04-05 18:06:20 +00:00
Cameron
f550f7e159 web/satellite: sort api keys
add ability to sort api keys on the access management page by ascending
or descending name or creation date. Additionally, edit api keys query
when ordering by name to order by lower(name) so names starting with
capital letters are not treated differently from lower case when
ordering.

Change-Id: I81dbb87587a24fb7097313f76bad116b1f20d306
2023-04-05 11:55:17 -04:00
Michal Niewrzal
c4c391e154 satellite/accounting/live: replace address parsing with redis util
With this change we are replacing parsing code with existing go-redis
util.

We also switch redis client to version 9.

Change-Id: Ie4a651e3ae6960e68958c690873925d319b70e10
2023-04-05 13:20:11 +00:00
paul cannon
9e6955cc17 satellite/repair: fix flaky TestFailedDataRepair and friends
The following tests should be made less flaky by this change:

- TestFailedDataRepair
- TestOfflineNodeDataRepair
- TestUnknownErrorDataRepair
- TestMissingPieceDataRepair_Succeed
- TestMissingPieceDataRepair
- TestCorruptDataRepair_Succeed
- TestCorruptDataRepair_Failed

This follows on to a change in commit 6bb64796. Nearly all tests in the
repair suite used to rely on events happening in a certain order. After
some of our performance work, those things no longer happen in that
expected order every time. This caused much flakiness.

The fix in 6bb64796 was sufficient for the tests operating directly on
an `*ECRepairer` instance, but not for the tests that make use of the
repairer by way of the repair queue and the repair worker. These tests
needed a different way to indicate the number of expected failures. This
change provides that different way.

Refs: https://github.com/storj/storj/issues/5736
Refs: https://github.com/storj/storj/issues/5718
Refs: https://github.com/storj/storj/issues/5715
Refs: https://github.com/storj/storj/issues/5609
Change-Id: Iddcf5be3a3ace7ad35fddb513ab53dd3f2f0eb0e
2023-04-04 18:08:52 +00:00
Jeremy Wharton
f2ae202024 satellite/payments,web/satellite: separate UI cost estimates by partner
Components related to project usage costs have been updated to show
different estimations for each partner, and the satellite has been
updated to send the client the information it needs to do this.

Previously, project costs in the satellite frontend were estimated
using only the price model corresponding to the partner that the user
registered with. This caused users who had a project containing
differently-attributed buckets to see an incorrect price estimation.

Resolves storj/storj-private#186

Change-Id: I2531643bc49f24fcb2e5f87e528b552285b6ff20
2023-04-04 15:56:16 +00:00
Michal Niewrzal
4c05293d8b satellite: set names for lrucache metrics
Fixes https://github.com/storj/storj/issues/5685

Change-Id: I9c19b7644b0cd3297448982b920c97abcdce9119
2023-04-04 11:55:20 +00:00
Michal Niewrzal
4db3d33d3e satellite/accounting: check bucket existance with GetProjectObjectsSegments
Check if project have any buckets before trying to get stats from
bucket_storage_tallies table.

Change-Id: If87159e29236a1816c3fb99c2d889ff735737100
2023-04-03 20:06:11 +00:00