Commit Graph

6001 Commits

Author SHA1 Message Date
Yingrong Zhao
fcde92d38d web/satellite: display segment count in billing
Change-Id: I472d65967c656889b8ffeb799cb46bc18b2a9b15
2021-10-27 19:51:45 +00:00
Yingrong Zhao
d441c8da15 satellite: use segment count for billing
Change-Id: Iafaf41cb6cde4cb2abdac3f75ce0fead838cb758
2021-10-27 19:21:50 +00:00
Yingrong Zhao
52f8c8175e satellite/satellitedb: add segments column into invoiceprojectrecords
table and drop not null constraint on objects column

Since, we want to move from charging our customers by object count to
segment count, this PR prepares the database to be able to record segments count
instead of objects count for satellite's billing system

Change-Id: Ie91ef354e78d24a268bc1cdc4327c182f733321e
2021-10-27 14:51:51 -04:00
paul cannon
1951450c57 cmd/satellite: add register-lost-segments command
Usage: from a host for the affected satellite, issue this command
and supply the number of segments that have been permanently lost. For
example, to indicate 2 segments permanently lost:

    satellite register-lost-segments 2

If the unthinkable happens and this is necessary to use, it is presumed
that we will also remove the non-recoverable segments from the metainfo
db, so they will not continue to appear as 'temporarily unavailable' to
the repair checker.

The influxQL query for this will probably look something like:

    SELECT sum(total) AS "sum_total"
    FROM "v3_stats_new"."autogen"."lost-segments"
    WHERE time > :dashboardTime: AND "scope" = 'segment-durability'
    GROUP BY time(:interval:), "application", "instance"
    FILL(null)

Or use the Flux language in order to get the benefit of the
cumulativeSum function:

    from(bucket: "v3_stats_new/autogen")
      |> range(start: dashboardTime)
      |> filter(fn: (r) => r._measurement == "lost-segments"
                           and r.scope == "segment-durability"
                           and r._field == "total")
      |> group(columns: ["application", "instance"])
      |> cumulativeSum()

Change-Id: I73d364937705aa815af7520ab79c00bc2aea09f6
2021-10-26 15:03:52 +00:00
Vitalii Shpital
5bbb69886a web/satellite: improved responsiveness of new nav sidebar
Since we don't have main content responsiveness I decided to make sidebar shrink earlier (1280px).
Added event listener for dropdowns to close if navigation sidebar is being scrolled vertically.

Change-Id: I15ac2ae652aedfc06fd331585bb64b69076c0eed
2021-10-26 13:37:05 +00:00
Egon Elbre
1409288c49 mod: bump semantic version to go 1.17
This enables module graph pruning and lazy module loading.

Change-Id: I5d63a686d4f4eda93b23576169bb03847292f8c7
2021-10-26 11:58:13 +00:00
Márton Elek
20d03bebdb satellite/nodeselection: flexible interface to includes nodes in selection
This commit doesn't change any behavior, just organize the code in
different way to make it easier to implement different Criterias
to include nodes. Today we use NodeID and Subnet based selection
but later Criteria can be extended with different kind of
placement rules (like geofencing).

The change nodeselection is used by segment allocaton (upload) and repair
and excludes nodes from an in-memory selection.

Resolves https://github.com/storj/storj/issues/4240

Change-Id: I0c1955fe16a045e3b76d7e50b2e1f4575a7ff095
2021-10-26 11:01:33 +00:00
Michał Niewrzał
de38e2e7d8 satellite/metainfo: stop using sat StreamID Redundancy
We needed Redundancy insided sat StreamID when uplink was defining RS values. Now it can be removed.

Change-Id: Id37187493eaa00cf29cb0262a050d71add3deb96
2021-10-26 10:23:11 +00:00
Vitalii Shpital
c7e2570123 web/satellite: tab adaptation for new navigation sidebar
Tab (768px) adaptation for new navigation sidebar.

Change-Id: I53b6b7b8bb25df8975d7ab4b5145c72170bb2e1f
2021-10-26 08:04:41 +00:00
paul cannon
eb68dbad41 satellite/satellitedb: fix ordering in listPendingTransitionShim
listPendingTransitionShim is a temporary transition shim intended to
make existing API processes keep working when a future DB schema change
is executed. For more explanation, see the message on commit c053bdbd70.

However, the shim has a small bug: it is missing the ORDER BY clause
that appears in the original ListPending method. This transition shim
code won't ever run until we make the DB schema change, so this bug
hasn't hurt anything yet; it's just important that we fix it before the
DB schema change happens.

Change-Id: I5953651583ee236500c2c07141dfc9d690a95118
2021-10-25 19:33:53 +00:00
Malcolm Bouzi
d90a1b09a2 satellite/console,satellitedb: add signup promo code column to users
This update is to set up users being able to register with a promo code added to their account in place of the free tier coupon.

Change-Id: I7badf87937b12664f145520b6dcc4b26fe750407
2021-10-25 18:08:00 +00:00
dlamarmorgan
01cfbde56b satellite/console: increase free tier project bandwidth and storage from 50 GB to 150 GB and reduce free tier max projects from 3 to 1
To make our free tier limits more clear, we will reduce the number of projects allowed from 3 to 1, and increase the storage and bandwidth limit of the free tier from 50 Gb to 150 GB. The total allotments across all projects for a given user are unchanged, just reduced to a single project.

Change-Id: Ic8dddb135f2b83a3f36e2b9fdcb477e351ec137b
2021-10-25 17:34:19 +00:00
crawter
3c7e289395 satellite/matainfo: handle metabase errors in metainfo endpoint in common way
We should improve the way how we are handling metabase errors in
metainfo endpoint.
https://storjlabs.atlassian.net/browse/PG-316

Change-Id: I1da6f333546cabf34d6eb1de8e94a3ef455d75d5
2021-10-25 16:30:24 +00:00
Qweder93
65d139fb96 private/testplanet: create metabaseDB after reconfigure apply
Moved CreateMetabaseDB after applying reconfigure changes to affect them during testing

Change-Id: I51034a6671728645dcc52a0d8c808b9dfc2fba7c
2021-10-25 15:55:45 +00:00
Vitalii Shpital
6c2cb98d2f web/satellite: fix onb CLI setup uplink download link
Fixed a download link for CLI setup screen of onboarding flow.

This is a fix for this github issue
https://github.com/storj/storj/issues/4220

Change-Id: I5ab8f69f30c7c83098004556aebb5318916cd9e6
2021-10-25 14:16:08 +00:00
Clement Sam
85b49bb27c cmd/uplink: add ranged download to uplink cli
Change-Id: Ib274df024a8ffc5db2d5c99f8f363efa3b43723f
2021-10-25 13:13:33 +00:00
Vitalii Shpital
a281adddd5 web/satellite: auto-create first demo bucket for the user
Create first bucket ('demo-bucket') for the user if they have no buckets.
Fixed bucket name validation to include '.' and '-'.

Change-Id: I21ddad63f214b6482ebaad95474ff0b6cf16254f
2021-10-25 15:31:54 +03:00
NickolaiYurchenko
0ed3ef0fe4 web/satellite: registation success page moved
page moved to separate route to be able to track conversion metrics

Change-Id: I1506ac416056d652b9292d62c29f79bd75567c6a
2021-10-25 08:51:41 +00:00
Ivan Fraixedes
df44e8152d
satellite/admin: Move declaration before first usage
Move a variable declaration just before where is used for making easy to
follow the implementation when reading it.

Change-Id: I3e93597956352037a63c5496b11576fdd2c600a6
2021-10-24 18:38:41 +02:00
Vitalii Shpital
3e040767b4 testsuite/ui/satellite: fix tests to pass with new navigation
Fixed tests that were failing due to new navigation structure.

Change-Id: Ia48458bf98d464e366d2468897d4012d8e0890fc
2021-10-22 22:26:26 +00:00
Stefan Benten
df6cf60ffe
go.mod: update to minimum supported go version (#4239) 2021-10-22 21:12:13 +02:00
Fadila Khadar
f654c3ae4c satellite/gracefulexit: drop unused column in graceful_exit_progress
We don't need column uses_segment_transfer_queue in graceful_exit_progress
as now all exiting nodes are using graceful_exit_segment_transfer_queue and
table graceful_exit_transfer_queue has been dropped.

Change-Id: I4b7c087433f04138cf09bcf8ad3d8de2c185502a
2021-10-22 16:41:27 +00:00
Moby von Briesen
1b22525002 satellite/payments: Fix discount when listing invoices.
The error message Stripe's API was sending:
"This property cannot be expanded (total_discount_amounts).
You may want to try expanding 'data.total_discount_amounts' instead."

Change-Id: I9f8cea4107d826d837755be2c3c04675a36f3c37
2021-10-22 10:16:27 -04:00
Egon Elbre
56f35e9ea9 mod: bump private repo
Contains a fix to ignore cockroach DROP error while testing.

Change-Id: I74b02240f570c30a66f6f0b938ce036392c05e06
2021-10-22 15:02:48 +03:00
Qweder93
50baefa10e satellite/metabase: limit maximum number of parts and size
Multipart upload limits added. Last part has no size limit.
Max number of parts: 10000, min part size: 5 MiB

Change-Id: Ic2262ce25f989b34d92f662bde720d4c4d0dc93d
2021-10-22 10:16:37 +00:00
Vitalii Shpital
49492d28fc testsuite/ui/satellite: test for new navigation sidebar
Added tests for new navigation sidebar.

Change-Id: If07615625ecb21de060e7c3853e6d1d675edb8e8
2021-10-22 08:35:57 +00:00
Egon Elbre
f2d8e97d97 satellite/satellitedb: simplify select nodes query construction
Change-Id: I07009b28762d4485929a2a999e8f4be8179bee51
2021-10-22 07:41:07 +00:00
Monty Anderson
3989107031 web/satellite: add feature flag for new browser
Change-Id: If3773b9df7f79d4c5e094140448d60ec060cdaeb
2021-10-22 03:49:08 +00:00
Clement Sam
f82b696858 mod: bump common dependency
Change-Id: I24ba0ef8d0ca25a34e28de1064a2f1213787ccc5
2021-10-21 16:33:48 +00:00
Michał Niewrzał
b32fbc0f93 satellite/metainfo: speedup deletion tests
Speedup is done by reducing number of testplanet instances
for tests without changing main test logic.

Change-Id: Ic3849485d37b8ca55c013a45b7191dce65b88b04
2021-10-21 15:54:27 +00:00
Egon Elbre
187941ff86 satellite/{metabase,metainfo}: add some missing monitoring
Change-Id: Ie0018f6385bb145d0a514dfe2792f3f299b55b79
2021-10-21 09:15:04 +00:00
Egon Elbre
bdadc4b460 private/testplanet: add stack traces related to the planet
It's useful to see only the goroutines that are related to the current
testplanet.

Change-Id: Id77c62dec42c09d44c523b019f7bdc940905fcf1
2021-10-20 23:12:07 +03:00
crawter
9ed826eabb cmd/uplink/mv: moving files between folders
https://storjlabs.atlassian.net/browse/PG-305

we should extend method move of cmd/uplink/cmd/mv.go
if both parameters end with slash -
i should list all files and call move method in loop

parameters:
uplink mv sj://bucket/a/prefix/ sj://new-bucket/a/new-prefix/

Change-Id: Ic24c2af83153ea60ec74393e65736af094877151
2021-10-20 18:16:08 +00:00
Jeff Wendling
987cb6ab11 cmd/uplinkng: allow removing pending objects
Change-Id: I12457e7d6fb28492ed4b6c5816f78aa7820fed6b
2021-10-20 12:25:25 -04:00
Yingrong Zhao
2df41028a3 storagenode/piecestore: add logs for restore trash endpoint
When a satellite operator runs restore trash command, it's hard for node
operator to know whether their node has received the request. Adding
logs so this process is visible from node operators perspective.

Change-Id: I08c670b1b0c5971e6b73e038a0935cb0caaca63d
2021-10-20 15:37:28 +00:00
Vitalii Shpital
9773197c61 web/satellite: new nav account area and vertical layout fixes
Added account area and dropdown for new navigation sidebar.
Fixed vertical layout for new and old nav structures depending on top info bars amount.

Change-Id: I238ba1f899d27d7fd9b865b0c17b0bfd7cff47ad
2021-10-20 13:57:14 +00:00
Vitalii Shpital
0aab339d0d web/satellite: links and dropdowns for new navigation sidebar
Added nav links, resources and quick start dropdowns.

Change-Id: I12f2afddd899621749bb00d0fed58f84a45701a3
2021-10-20 13:56:27 +00:00
Vitalii Shpital
b619ce6913 web/satellite: new nav project dropdown
Added new project dropdown for new navigation sidebar.

Change-Id: Idbfd37cfff2218b8bfda803bdf143b8eab691200
2021-10-20 13:56:19 +00:00
Vitalii Shpital
0009447773 satellite/console, web/satellite: feature flag for new navigation structure
Change-Id: I6f1fd43ea8154714187c7547ae93de0efe3482e6
2021-10-20 13:51:25 +00:00
Moby von Briesen
9c8b6c3fc3 testsuite/ui: Add coupon tests
Add some testing around adding/replacing promotional coupons and what
the user sees in the UI depending on their coupon status.

Change-Id: Ice6c0a0644d05af0c30c87a93ba963c0bb09e32d
2021-10-19 19:08:35 -04:00
Brandon Iglesias
8d18033a99
satellite/rewards: adding SeaweedFS to partners list (#4230) 2021-10-19 21:30:31 +02:00
Mya Pitzeruse
d250f9dfee
satellite/core: fix label for orders chore (#4228) 2021-10-19 18:44:15 +02:00
Egon Elbre
51f488fc4b satellite/satellitedb: fix selection query with AOST
Union query for both reputable and new nodes didn't properly work. The
top level query is required to have an `AS OF SYSTEM TIME` statement as
well.

Change-Id: I8ee6dd5b700c2b1ed2aa562962bfa72be7eec30a
2021-10-19 16:59:40 +03:00
Michał Niewrzał
4a146000cc satellite/metainfo: read from DB only needed columns fro bucket
Uplink needs only part of columns we are reading from DB.
To improve performance we should read only those that are
realy needed.

Change-Id: Ib39259318169c46afe5fa4c6ce2184da82e960c8
2021-10-19 11:53:42 +00:00
Michał Niewrzał
8293c9659e satellite/metabase: speedup iterator tests
Main speedup is from reducing number of cases to iterate
in 'boundaries' test case.

Change-Id: I5bbf1311790f3d0e553d59e6a73e1cf3fa601a55
2021-10-19 11:52:07 +00:00
Michał Niewrzał
f5f2398431 satellite/metabase: remove code related to part deletion
It is not used currently.

Change-Id: I32e9fa357814a3159ee8659a607cd6986f361fbf
2021-10-19 10:11:49 +00:00
Moby von Briesen
4ad7056bf7 satellite/payments: Add old invoice list functionality
3b751a35c Removed our old coupon functionality, and slightly reworked
the invoice List() function in the stripecoinpayments package.

It turns out, this is causing some issues when trying to delete users.
This change keeps the new functionality, which is used in the satellite
UI, but under a new name, ListWithDiscounts()

Change-Id: I6a62a1de480e09d005dd22d75aa1e024fd2ed3a0
2021-10-18 18:18:18 -04:00
Yingrong Zhao
35e4a87e60 satellite/repair: ignore expired segments at the beginning of the repair
work

Since we have changed the repair worker to also mark a node as audit
failure if they return a not found error, we should ignore expired
segments when possible

Change-Id: Ie6a677e1d7b234e93965c736d05950440236653c
2021-10-18 18:15:39 +00:00
Clement Sam
29599dd7cd testsuite/ui/multinode: add first node test
Change-Id: Ibeaee92e6d1a41e408659a5045f52e1908f73089
2021-10-18 16:07:01 +00:00
dlamarmorgan
4bbf667ad1 satellite/{satellitedb,attribution,console}: value attribution changes that add userAgent field to buckets table and all tables that have partner_id
Change-Id: I36a13bb651b86bfc14fe5a0a2258f719e6cd2b48
2021-10-18 13:56:19 +00:00