Commit Graph

6159 Commits

Author SHA1 Message Date
Egon Elbre
431f55d53b ci: cross-compile storagenode-updater
Change-Id: I1db757b64e61f185ba710bf669f2e62441b9ab71
2021-11-01 11:21:06 +00:00
Jeremy Wharton
136899eeea web/satellite: Trim whitespace around login credentials
Trims whitespace around the email address, MFA passcode,
and MFA recovery code before sending a login request to the satellite.

Change-Id: I145e6a5991270b0e49ac3f09ab3c9d1c9f970699
2021-10-30 19:39:13 +00:00
Jeff Wendling
4d023f773f cmd/storagenode-updater: update to correct version
Currently if the nodes is below the minimum version it will immediately
update to the suggested version, regardless if its eligible from the seed
or not. This change corrects the behaviour to update to the minimum
version only and then properly respect/wait for the rollout to include it.

Updates based on logic here: https://review.dev.storj.io/c/storj/private/+/6187

Change-Id: Ic6c91c48ae9b8a116378b2573fbfca7e7bd5cc32
2021-10-29 20:38:30 +00:00
Jeff Wendling
f94d8a200c cmd/uplinkng: mkdir config file folder
If the directory doesn't exist, then the first run in a non-migration
setting will error because it cannot create the config file. This
change creates the directory.

Change-Id: I439159c00047e9ae20e139318dad5a047c853253
2021-10-29 11:07:55 +00:00
Stefan Benten
8c1a149de2 build: bump private dependency
Enables proper version printing.

Change-Id: I37e12d6823f86f9275ac9f64abf324003019bd18
2021-10-29 12:25:33 +13:00
mya
9749f9756c
cmd/uplinkng: port expanded option to show additional metadata (#4229) 2021-10-28 11:51:33 -05:00
Egon Elbre
edb8d656de satellite/metainfo: adjust piecedeletion timeouts
Currently slower storagenodes can slow down deletion queue.
To make piece deletion faster reduce the maximum time spent in
either dialing or piece deletion requests.

With this change:
* dial timeout is 3s
* request timeout is 15s
* fail threshold is set to 10min

Similarly, we'll mark storage node as failed when the timeout occurs.
The timeout usually indicates that the storagenode is overwhelmed.
Garbage collection will ensure that the pieces get deleted eventually.

Change-Id: Iec5de699f5917905f5807140e2c3252088c6399b
2021-10-28 13:37:01 +03:00
Michał Niewrzał
d3a0364f21 Makefile: add uplinkng to build binaries
We want uplinkng binaries easily accessible.

Change-Id: I83e1d90f9413ebd97e1a51d0435548bc0cac8ad8
2021-10-28 09:47:36 +02:00
Yingrong Zhao
fcde92d38d web/satellite: display segment count in billing
Change-Id: I472d65967c656889b8ffeb799cb46bc18b2a9b15
2021-10-27 19:51:45 +00:00
Yingrong Zhao
d441c8da15 satellite: use segment count for billing
Change-Id: Iafaf41cb6cde4cb2abdac3f75ce0fead838cb758
2021-10-27 19:21:50 +00:00
Yingrong Zhao
52f8c8175e satellite/satellitedb: add segments column into invoiceprojectrecords
table and drop not null constraint on objects column

Since, we want to move from charging our customers by object count to
segment count, this PR prepares the database to be able to record segments count
instead of objects count for satellite's billing system

Change-Id: Ie91ef354e78d24a268bc1cdc4327c182f733321e
2021-10-27 14:51:51 -04:00
paul cannon
1951450c57 cmd/satellite: add register-lost-segments command
Usage: from a host for the affected satellite, issue this command
and supply the number of segments that have been permanently lost. For
example, to indicate 2 segments permanently lost:

    satellite register-lost-segments 2

If the unthinkable happens and this is necessary to use, it is presumed
that we will also remove the non-recoverable segments from the metainfo
db, so they will not continue to appear as 'temporarily unavailable' to
the repair checker.

The influxQL query for this will probably look something like:

    SELECT sum(total) AS "sum_total"
    FROM "v3_stats_new"."autogen"."lost-segments"
    WHERE time > :dashboardTime: AND "scope" = 'segment-durability'
    GROUP BY time(:interval:), "application", "instance"
    FILL(null)

Or use the Flux language in order to get the benefit of the
cumulativeSum function:

    from(bucket: "v3_stats_new/autogen")
      |> range(start: dashboardTime)
      |> filter(fn: (r) => r._measurement == "lost-segments"
                           and r.scope == "segment-durability"
                           and r._field == "total")
      |> group(columns: ["application", "instance"])
      |> cumulativeSum()

Change-Id: I73d364937705aa815af7520ab79c00bc2aea09f6
2021-10-26 15:03:52 +00:00
Vitalii Shpital
5bbb69886a web/satellite: improved responsiveness of new nav sidebar
Since we don't have main content responsiveness I decided to make sidebar shrink earlier (1280px).
Added event listener for dropdowns to close if navigation sidebar is being scrolled vertically.

Change-Id: I15ac2ae652aedfc06fd331585bb64b69076c0eed
2021-10-26 13:37:05 +00:00
Egon Elbre
1409288c49 mod: bump semantic version to go 1.17
This enables module graph pruning and lazy module loading.

Change-Id: I5d63a686d4f4eda93b23576169bb03847292f8c7
2021-10-26 11:58:13 +00:00
Márton Elek
20d03bebdb satellite/nodeselection: flexible interface to includes nodes in selection
This commit doesn't change any behavior, just organize the code in
different way to make it easier to implement different Criterias
to include nodes. Today we use NodeID and Subnet based selection
but later Criteria can be extended with different kind of
placement rules (like geofencing).

The change nodeselection is used by segment allocaton (upload) and repair
and excludes nodes from an in-memory selection.

Resolves https://github.com/storj/storj/issues/4240

Change-Id: I0c1955fe16a045e3b76d7e50b2e1f4575a7ff095
2021-10-26 11:01:33 +00:00
Michał Niewrzał
de38e2e7d8 satellite/metainfo: stop using sat StreamID Redundancy
We needed Redundancy insided sat StreamID when uplink was defining RS values. Now it can be removed.

Change-Id: Id37187493eaa00cf29cb0262a050d71add3deb96
2021-10-26 10:23:11 +00:00
Vitalii Shpital
c7e2570123 web/satellite: tab adaptation for new navigation sidebar
Tab (768px) adaptation for new navigation sidebar.

Change-Id: I53b6b7b8bb25df8975d7ab4b5145c72170bb2e1f
2021-10-26 08:04:41 +00:00
paul cannon
eb68dbad41 satellite/satellitedb: fix ordering in listPendingTransitionShim
listPendingTransitionShim is a temporary transition shim intended to
make existing API processes keep working when a future DB schema change
is executed. For more explanation, see the message on commit c053bdbd70.

However, the shim has a small bug: it is missing the ORDER BY clause
that appears in the original ListPending method. This transition shim
code won't ever run until we make the DB schema change, so this bug
hasn't hurt anything yet; it's just important that we fix it before the
DB schema change happens.

Change-Id: I5953651583ee236500c2c07141dfc9d690a95118
2021-10-25 19:33:53 +00:00
Malcolm Bouzi
d90a1b09a2 satellite/console,satellitedb: add signup promo code column to users
This update is to set up users being able to register with a promo code added to their account in place of the free tier coupon.

Change-Id: I7badf87937b12664f145520b6dcc4b26fe750407
2021-10-25 18:08:00 +00:00
dlamarmorgan
01cfbde56b satellite/console: increase free tier project bandwidth and storage from 50 GB to 150 GB and reduce free tier max projects from 3 to 1
To make our free tier limits more clear, we will reduce the number of projects allowed from 3 to 1, and increase the storage and bandwidth limit of the free tier from 50 Gb to 150 GB. The total allotments across all projects for a given user are unchanged, just reduced to a single project.

Change-Id: Ic8dddb135f2b83a3f36e2b9fdcb477e351ec137b
2021-10-25 17:34:19 +00:00
crawter
3c7e289395 satellite/matainfo: handle metabase errors in metainfo endpoint in common way
We should improve the way how we are handling metabase errors in
metainfo endpoint.
https://storjlabs.atlassian.net/browse/PG-316

Change-Id: I1da6f333546cabf34d6eb1de8e94a3ef455d75d5
2021-10-25 16:30:24 +00:00
Qweder93
65d139fb96 private/testplanet: create metabaseDB after reconfigure apply
Moved CreateMetabaseDB after applying reconfigure changes to affect them during testing

Change-Id: I51034a6671728645dcc52a0d8c808b9dfc2fba7c
2021-10-25 15:55:45 +00:00
Vitalii Shpital
6c2cb98d2f web/satellite: fix onb CLI setup uplink download link
Fixed a download link for CLI setup screen of onboarding flow.

This is a fix for this github issue
https://github.com/storj/storj/issues/4220

Change-Id: I5ab8f69f30c7c83098004556aebb5318916cd9e6
2021-10-25 14:16:08 +00:00
Clement Sam
85b49bb27c cmd/uplink: add ranged download to uplink cli
Change-Id: Ib274df024a8ffc5db2d5c99f8f363efa3b43723f
2021-10-25 13:13:33 +00:00
Vitalii Shpital
a281adddd5 web/satellite: auto-create first demo bucket for the user
Create first bucket ('demo-bucket') for the user if they have no buckets.
Fixed bucket name validation to include '.' and '-'.

Change-Id: I21ddad63f214b6482ebaad95474ff0b6cf16254f
2021-10-25 15:31:54 +03:00
NickolaiYurchenko
0ed3ef0fe4 web/satellite: registation success page moved
page moved to separate route to be able to track conversion metrics

Change-Id: I1506ac416056d652b9292d62c29f79bd75567c6a
2021-10-25 08:51:41 +00:00
Ivan Fraixedes
df44e8152d
satellite/admin: Move declaration before first usage
Move a variable declaration just before where is used for making easy to
follow the implementation when reading it.

Change-Id: I3e93597956352037a63c5496b11576fdd2c600a6
2021-10-24 18:38:41 +02:00
Vitalii Shpital
3e040767b4 testsuite/ui/satellite: fix tests to pass with new navigation
Fixed tests that were failing due to new navigation structure.

Change-Id: Ia48458bf98d464e366d2468897d4012d8e0890fc
2021-10-22 22:26:26 +00:00
Stefan Benten
df6cf60ffe
go.mod: update to minimum supported go version (#4239) 2021-10-22 21:12:13 +02:00
Fadila Khadar
f654c3ae4c satellite/gracefulexit: drop unused column in graceful_exit_progress
We don't need column uses_segment_transfer_queue in graceful_exit_progress
as now all exiting nodes are using graceful_exit_segment_transfer_queue and
table graceful_exit_transfer_queue has been dropped.

Change-Id: I4b7c087433f04138cf09bcf8ad3d8de2c185502a
2021-10-22 16:41:27 +00:00
Moby von Briesen
1b22525002 satellite/payments: Fix discount when listing invoices.
The error message Stripe's API was sending:
"This property cannot be expanded (total_discount_amounts).
You may want to try expanding 'data.total_discount_amounts' instead."

Change-Id: I9f8cea4107d826d837755be2c3c04675a36f3c37
2021-10-22 10:16:27 -04:00
Egon Elbre
56f35e9ea9 mod: bump private repo
Contains a fix to ignore cockroach DROP error while testing.

Change-Id: I74b02240f570c30a66f6f0b938ce036392c05e06
2021-10-22 15:02:48 +03:00
Qweder93
50baefa10e satellite/metabase: limit maximum number of parts and size
Multipart upload limits added. Last part has no size limit.
Max number of parts: 10000, min part size: 5 MiB

Change-Id: Ic2262ce25f989b34d92f662bde720d4c4d0dc93d
2021-10-22 10:16:37 +00:00
Vitalii Shpital
49492d28fc testsuite/ui/satellite: test for new navigation sidebar
Added tests for new navigation sidebar.

Change-Id: If07615625ecb21de060e7c3853e6d1d675edb8e8
2021-10-22 08:35:57 +00:00
Egon Elbre
f2d8e97d97 satellite/satellitedb: simplify select nodes query construction
Change-Id: I07009b28762d4485929a2a999e8f4be8179bee51
2021-10-22 07:41:07 +00:00
Monty Anderson
3989107031 web/satellite: add feature flag for new browser
Change-Id: If3773b9df7f79d4c5e094140448d60ec060cdaeb
2021-10-22 03:49:08 +00:00
Clement Sam
f82b696858 mod: bump common dependency
Change-Id: I24ba0ef8d0ca25a34e28de1064a2f1213787ccc5
2021-10-21 16:33:48 +00:00
Michał Niewrzał
b32fbc0f93 satellite/metainfo: speedup deletion tests
Speedup is done by reducing number of testplanet instances
for tests without changing main test logic.

Change-Id: Ic3849485d37b8ca55c013a45b7191dce65b88b04
2021-10-21 15:54:27 +00:00
Egon Elbre
187941ff86 satellite/{metabase,metainfo}: add some missing monitoring
Change-Id: Ie0018f6385bb145d0a514dfe2792f3f299b55b79
2021-10-21 09:15:04 +00:00
Egon Elbre
bdadc4b460 private/testplanet: add stack traces related to the planet
It's useful to see only the goroutines that are related to the current
testplanet.

Change-Id: Id77c62dec42c09d44c523b019f7bdc940905fcf1
2021-10-20 23:12:07 +03:00
crawter
9ed826eabb cmd/uplink/mv: moving files between folders
https://storjlabs.atlassian.net/browse/PG-305

we should extend method move of cmd/uplink/cmd/mv.go
if both parameters end with slash -
i should list all files and call move method in loop

parameters:
uplink mv sj://bucket/a/prefix/ sj://new-bucket/a/new-prefix/

Change-Id: Ic24c2af83153ea60ec74393e65736af094877151
2021-10-20 18:16:08 +00:00
Jeff Wendling
987cb6ab11 cmd/uplinkng: allow removing pending objects
Change-Id: I12457e7d6fb28492ed4b6c5816f78aa7820fed6b
2021-10-20 12:25:25 -04:00
Yingrong Zhao
2df41028a3 storagenode/piecestore: add logs for restore trash endpoint
When a satellite operator runs restore trash command, it's hard for node
operator to know whether their node has received the request. Adding
logs so this process is visible from node operators perspective.

Change-Id: I08c670b1b0c5971e6b73e038a0935cb0caaca63d
2021-10-20 15:37:28 +00:00
Vitalii Shpital
9773197c61 web/satellite: new nav account area and vertical layout fixes
Added account area and dropdown for new navigation sidebar.
Fixed vertical layout for new and old nav structures depending on top info bars amount.

Change-Id: I238ba1f899d27d7fd9b865b0c17b0bfd7cff47ad
2021-10-20 13:57:14 +00:00
Vitalii Shpital
0aab339d0d web/satellite: links and dropdowns for new navigation sidebar
Added nav links, resources and quick start dropdowns.

Change-Id: I12f2afddd899621749bb00d0fed58f84a45701a3
2021-10-20 13:56:27 +00:00
Vitalii Shpital
b619ce6913 web/satellite: new nav project dropdown
Added new project dropdown for new navigation sidebar.

Change-Id: Idbfd37cfff2218b8bfda803bdf143b8eab691200
2021-10-20 13:56:19 +00:00
Vitalii Shpital
0009447773 satellite/console, web/satellite: feature flag for new navigation structure
Change-Id: I6f1fd43ea8154714187c7547ae93de0efe3482e6
2021-10-20 13:51:25 +00:00
Moby von Briesen
9c8b6c3fc3 testsuite/ui: Add coupon tests
Add some testing around adding/replacing promotional coupons and what
the user sees in the UI depending on their coupon status.

Change-Id: Ice6c0a0644d05af0c30c87a93ba963c0bb09e32d
2021-10-19 19:08:35 -04:00
Brandon Iglesias
8d18033a99
satellite/rewards: adding SeaweedFS to partners list (#4230) 2021-10-19 21:30:31 +02:00
Mya Pitzeruse
d250f9dfee
satellite/core: fix label for orders chore (#4228) 2021-10-19 18:44:15 +02:00