Commit Graph

7372 Commits

Author SHA1 Message Date
Wilfred Asomani
d76f764b72 satellite/web: replace cancel with back buttons
This replaces "Cancel" with "Back" on Clear, Create and Switch steps.
It also removes margin-top from the button rows in Clear, Create steps.

Issue: https://github.com/storj/storj/issues/5463

Change-Id: Ib2ef0dba9b91e82a7cf5b042fad0162128fa071a
2023-01-18 17:51:18 +00:00
Vitalii
65a090d111 satellite/console: enable project level passphrase by default
Enable project level passphrase by default.

Issue:
https://github.com/storj/storj/issues/5456

Change-Id: I8c08f2d81a3ce1b5e0134172b7440309b850c5c2
2023-01-18 15:32:29 +02:00
JT Olio
b022f371d2 blueprint: tcp fastopen
Change-Id: I20fc843b916b5ae0b72acfa5a03cf75f211739c0
2023-01-17 20:35:47 +00:00
JT Olio
14c91aa05d blueprint: noise over tcp simplification
the noise over tcp blueprint had two unresolved questions:

 * what do we do for IPv6?
 * what do we do for UDP support?

this change punts those two questions for later and explicitly
rejects solving them as part of noise over TCP for now.

this change also punts renaming NodeURL to reduce codebase
churn.

Change-Id: Ib209942bae61716b2ff6cea852c336af072d8bc5
2023-01-17 16:06:05 +00:00
Jeremy Wharton
6142b1cd12 web/satellite: show overridden usage prices in the satellite UI
This change allows users who register with a partner that has
different project usage prices to see the correct prices in the
satellite UI.

Resolves storj/storj-private#90

Change-Id: I06bde50db474b25396671a27e282ef5637efe85b
2023-01-17 15:01:57 +00:00
Jeremy Wharton
5d656e66bf satellite/payments/stripecoinpayments: implement invoice price override
This change allows for overriding project usage prices for a specific
partner so that users who sign up with that partner do not need their
invoices to be manually adjusted.

Relates to storj/storj-private#90

Change-Id: Ia54a9cc7c2f8064922bbb15861f974e5dea82d5a
2023-01-17 14:32:10 +00:00
Yaroslav Vorobiov
5644fb1a7e satellite/accounting/nodetally: add ranged loop
Add node tally ranged loop observer and partial.
Add node tally randed observer to range loop peer.
Add config flag to select which loop to use for node tally.
Update satellite core to use segement/ranged loop based on a flag.
Duplicate existing node tally test but using ranged loop.

Change-Id: I6786f1a16933463fab5f79601bf438203a7a5f9e
2023-01-17 13:50:18 +01:00
Egon Elbre
90b7076d26 storagenode/pieces: fix log line
Change-Id: I8dba6b0f3d6af3140dfa503c8d6b33e6808d004f
2023-01-17 11:04:47 +02:00
Vitalii
079728f725 web/satellite: improved project level passphrase experience
Clicking continue in web toggles create project level passphrase which then redirects to project dashboard.
Added new create bucket modal.
Updated open bucket modal.
Updated project dashboard and buckets view to work correctly with no buckets state and no passphrase state.

Issue:
https://github.com/storj/storj/issues/5455

Change-Id: If6ddac7d3365854a02b2bb8898e4742e9d2c31c1
2023-01-17 08:15:06 +00:00
JT Olio
df9bbcecad blueprint: noise over tcp
Change-Id: I3871e540313100a7198d62a455c745f128ac51fb
2023-01-13 16:24:56 +00:00
NickolaiYurchenko
a80bbb9d16 web/satellite: navigation label fixed, add token address truncation changed
console error fixed;
https://github.com/storj/storj/issues/5462
https://github.com/storj/storj/issues/5469

Change-Id: I33e9f5ae09a166fdd8b21ef811fc078d14c10663
2023-01-13 15:42:02 +00:00
NickolaiYurchenko
86fb188be6 web/satellite: Setting.vue migrated to use composition api
added useLoading composable to get rid or repetative same code usage;

Change-Id: I3524efdb1919759046bbf11fe3510f8044adc828
2023-01-13 17:04:30 +02:00
NickolaiYurchenko
a7fc884a2c web/satellite: browser header buttons styles fixed
https://github.com/storj/storj/issues/5459

Change-Id: I75368aabd0faac3d6662b95a31bce4896e6a8341
2023-01-13 14:39:32 +02:00
Andrew Harding
c5b5695bca satellite/metabase/rangedloop: clean up observerstats init
Small cleanups of the observer stats init code:
1. Use sync.Once for race free addition to the monitoring chain
   (purely defensive)
2. Set the observer durations before adding to the monitoring chain on
   first use.
3. observerDurations slice does not need to be initialized to non-nil

Change-Id: I9ae8ec96debc7d52c4ee5d22762a89f21bb2e38c
2023-01-13 10:40:30 +00:00
Egon Elbre
dc6e752dff satellite/satellitedb/dbx: document project.dbx
Updates https://github.com/storj/team-delivery/issues/1

Change-Id: I28bafdc4af4149489439fb5ab6e5bbe1d8f5186b
2023-01-13 11:24:34 +02:00
NickolaiYurchenko
2e28e59a5e web/satellite: cant open bucket on name click bug fixed
https://github.com/storj/storj/issues/5467

Change-Id: I0c91167ddd1d01b15238dbebf10f49dc0cc6c340
2023-01-12 19:03:43 +00:00
Malcolm Bouzi
583ea51fb1 satellite/console: Prevent duplicate project names for the same user.
issue: https://github.com/storj/storj/issues/5247

Change-Id: Ie51f24dca5b9e5064cb06ae238055177bec5975a
2023-01-12 17:18:09 +00:00
Michal Niewrzal
0185bba90a cmd: cleanup segment verify/repair tools
* use the same DB application name for satellite and metabase
* use noop orders DB implementation to avoid storing allocated bandwidth
in DB

Change-Id: I20e88c694d38240fe1a20c45719e210cfb76402c
2023-01-12 15:27:07 +00:00
NickolaiYurchenko
e0f2087245 web/satellite: EncryptForm component migrated to use composition api
Change-Id: Iae41d7bd7b44f53c43b4bf2ab552b5f746915469
2023-01-12 14:35:49 +00:00
Michal Niewrzal
a2a9dafa33 satellite/orders: don't store allocated bandwidth in
bucket_bandwidth_rollups table

We have performance problems with updating bucket_bandwidth_rollups. To
improve situation we can stop storing allocated bandwidth in this table.
This should reduce large number of updates which are comming from
metainfo endpoints, repair workers and audit.

Next step will be to drop `allocated` column completely from
bucket_bandwidth_rollups.

Allocated GET bandwidth is all we need and we are keeping it in
bucket_bandwidth_rollups table.

Change-Id: Ifdd26a89ba8262acbca6d794a6c02883ad0c0c9b
2023-01-12 13:21:02 +00:00
Michal Niewrzal
2876d5f20b satellite/satellitedb: drop project_bandwidth_rollups table
We are not using this table.

Change-Id: I3e4d5d0c2d8522b319caf48fea460b1cc35d5630
2023-01-12 10:23:48 +00:00
Cameron
5da2544e62 satellite/console: support public id as generated api parameter
On generated console api endpoints allow either the project ID or the
public ID to be used as the ID parameter.

github issue: https://github.com/storj/storj/issues/5412

Change-Id: Ic9901ed273931a50ae12f20142a3c4938dfcc8c0
2023-01-11 22:03:35 +00:00
Erik van Velzen
ed910b6087
satellite/metabase/rangedloop: continue after error (#5430)
When an observer errors we still want to finish the other observers.

This changes store the error and continues the loop, skipping
the observer which errored out and setting the duration metric to -1.

When the error occurs in the process stage, it does continue the other
ranges of the same observer. It removes the observer entirely after the process
stage. To improve this would make it more complex due to race
conditions.

Closes https://github.com/storj/storj/issues/5389

Change-Id: I528432c491d4340817d6950f1200ee2b9e703309
2023-01-11 22:23:17 +01:00
Moby von Briesen
3d1007ae18 private/apigen: Change order of operations in api generation
Move the IsAuthenticated check until after initial parameter
parsing/validation. IsAuthenticated will be more expensive than
parsing/validation, so we should fail before auth if possible.

Change-Id: I96a020892eabcb750e8ec9ecc1d8b7d9bf8bf573
2023-01-11 16:54:53 +00:00
Erik van Velzen
2d863759b0 satellite/metabase/rangedloop: add AsOfSystemTime
Add option AsOfSystemTime to segment provider to make it equivalent to
the old segment loop.

There's no comment on what it does because it's pretty complex and
makes no sense, but we can improve it later.

closes https://github.com/storj/storj/issues/5434

Change-Id: I8f09b03803e681e2fd41008c5dba67804b0f37a1
2023-01-11 16:22:18 +00:00
prerna-parashar
6e9567cb25
satellite/analytics: Track Project Details Updates (#5451) 2023-01-11 21:25:41 +05:30
Clement Sam
b5d0021fb6 cmd/storagenode-updater: restart storagenode after update on BSD unix derivatives
On BSD, the storagenode-updater falls back to renaming the storagenode without
doing anything to restart the service.
Like the approach we have for linux, this change finds the process ID of the
storagenode using pgrep and sends an interrupt signal to the process.

Closes https://github.com/storj/storj/issues/5333

Change-Id: Icced90ea3e831528804784c2170a3b8b14952e8c
2023-01-11 12:38:26 +00:00
Michal Niewrzal
282aaf8945 satellite/metabase: fix GetStreamPieceCountByNodeID full table scan
Previous version of SQL query was causing full table scan.

Output of EXPLAIN:
---
distribution: local
vectorized: true

• lookup join
│ table: segments@segments_pkey
│ equality: (?column?) = (stream_id)
│ pred: remote_alias_pieces IS NOT NULL
│
└── • union
    │
    ├── • values
    │     size: 1 column, 1 row
    │
    └── • scan
          missing stats
          table: segment_copies@segment_copies_pkey
          spans: [/'\xff135285155378d980b8c49148cef3ca' - /'\xff135285155378d980b8c49148cef3ca']
---

Change-Id: I708d1df204ac2d33cefe80b23594442b193424d2
2023-01-10 23:35:22 +00:00
Clement Sam
5fdfc6c4f3 cmd/{storagenode,satellite}: remove unused docker compose YAML files
Change-Id: I8d35cc723fa6b25d2baff58d848e0888a863cf4c
2023-01-10 18:17:42 +00:00
NickolaiYurchenko
60cd245764 web/satellite: access grant CreateForm modal migrated and slightly refactored
Change-Id: Icbe0c1189fcf077450988873bd0916b5518df4a8
2023-01-10 17:42:02 +00:00
Moby von Briesen
e26654c2a9 satellite/satellitedb: Drop unused tables
The `coupons`, `offers`, `coupon_usages`, `coupon_codes`, and
`user_credits` tables are not used anymore. They exist due to
legacy billing code which has already been removed.

Change-Id: Ie93af90a06e5247a74015af2a78a223d69249565
2023-01-10 17:00:15 +00:00
Moby von Briesen
6d1f2fe307 satellite/satellitedb/dbx/gen: use go instead of sh
The shell script has different behavior on different operating systems,
and now has enough logic that it makes sense to put it into a go script.
This change replaces dbx/gen.sh with dbx/gen/main.go, which accomplishes
the same thing.

Change-Id: Ibb53ebf9c2475377aae1165cfe245140d7162026
2023-01-10 16:30:39 +00:00
Egon Elbre
b3448606fa satellite/satellitedb/dbx: document project_bucket.dbx
Updates https://github.com/storj/team-delivery/issues/1

Change-Id: I2931af073b7e2e014ce3881e6e5b6e05618ec49b
2023-01-10 17:28:29 +02:00
Lizzy Thomson
c3c239d1f7 web/satellite: update billing history logic
Update billing history logic to exclude empty status invoices

Fixes https://github.com/storj/storj/issues/5427

Change-Id: If600f3dbce511b1c427d8fcde577dd145e693bb1
2023-01-09 17:47:45 +00:00
Egon Elbre
1688e4965b satellite/satellitedb/dbx: split into multiple files
The current satellitedb.dbx has grown quite big over time.
Split the single file into multiple smaller ones.

Change-Id: I8d6ca851b834ac60820aff565906f04aab661875
2023-01-09 16:24:31 +02:00
NickolaiYurchenko
d806c5c49b web/satellite: CreditCardContainer.vue padding bug on 'Default' label fixed
Change-Id: I714614a5d92e4f3d994f03da94dd053e751bbf50
2023-01-06 18:09:38 +00:00
paul cannon
246c193145 cmd/tools/segment-verify: add timeout to VerifyWithExists
We have to wait until the slowest node is done being tested before we
can move on to the next of segments. Since the slowest node can be
arbitrarily slow, we'll set a timeout and treat too-slow nodes as
temporarily offline.

Change-Id: I80fe865dd4e8f826700430fb0140c2d3aefca381
2023-01-06 17:25:30 +00:00
paul cannon
23acee2df0 cmd/tools/segment-verify: better handle timeouts
When we are verifying pieces by downloading the first byte, if we
encounter a timeout, treat the node as if we failed to connect to it,
and log the error once instead of twice.

Change-Id: I70602d554183c98f1213f3ffb1bfec41100ea0e7
2023-01-06 16:52:28 +00:00
paul cannon
3a9ad48345 cmd/tools/segment-verify: fix problem-pieces.csv
This csv file was being closed as soon as the service was created.
All subsequent writes to the closed file handle produced errors,
which were logged but otherwise ignored.

Instead, we would like the file to remain open and writable, until
the service is destroyed.

Change-Id: Ib29944d25b2f5b2d0f90fdbdcde44fea8d769321
2023-01-06 16:23:34 +00:00
Wilfred Asomani
0773b4efbc satellite/console: support id and publicId
Update get usage-limits, daily-usage and salt endpoints to support
both project-ID and project-PublicID.

Issue: https://github.com/storj/storj/issues/5411

Change-Id: Iff0114a295d1a479b141bfffbfb31599844d1fc0
2023-01-06 15:42:45 +00:00
Wilfred Asomani
05c83a5539 satellite/{console,web}: support delete by ID and publicID
Update the delete API key by name and projectID to support project-ID
and project-publicID.

Issue: https://github.com/storj/storj/issues/5410

Change-Id: I3bd11b9c3ae1ad6ce662dfc18b42779d2e4edf9b
2023-01-06 13:40:10 +00:00
Erik van Velzen
23b92da490
satellite/metabase/rangedloop: live reporting (#5366)
Add an observer to monitor ranged segment loop progress.

Tested by running the segment loop in storj-up and navigating to
http://<container>:11111/mon/stats and there is the entry:

rangedloop-live,scope=storj.io/storj/satellite/metabase/rangedloop numSegments=364523630000.000000

part of https://github.com/storj/storj/issues/5223

Change-Id: If3d2774d2f17f51eac86f47c6dda1fb8ad696dfe
2023-01-06 09:49:14 +01:00
Andrew Harding
5efb08cd7b cmd/uplink: skip abort on committed multipart writes
The copyFile method has some safeguards to ensure that the multipart
write is aborted. This is accomplished by always calling abort on the
MultiWriteHandle when the copy is finished, whether or not there was a
failure or it was successfully committed. If the copy was committed,
then this RPC is a no-op on the metainfo server.

Regardless, the calls to abort to constitute an additional RPC to the
satellite for no benefit. This is exacerbated by the fact that the code
currently ends up calling abort twice.

This change updates the libuplink-backed MultiWriteHandle implementation
to not call abort if the write is committed and vice-versa. This
eliminates the two wasteful RPC calls.

Change-Id: I13679234f6f473e9a93179e6791fb57eac512f25
2023-01-05 18:30:23 +00:00
Lizzy Thomson
591615b90b satellite/satellitedb: remove references to last_verification_reminder
Removing all references to column last_verification_reminder which is to be removed, due to new column verification_reminders

Issue: https://github.com/storj/storj/issues/4560

Change-Id: I7c9a426e946c7aed58e62c1eef80629daf6b1272
2023-01-05 08:58:05 -07:00
Erik van Velzen
1d4411f166
satellite/metabase/rangedloop: cancellation (#5364)
Support interruption of the ranged segment loop through context.

Part of https://github.com/storj/storj/issues/5223

Change-Id: Iae0260e250f8ea33affed95c6592a1f42df384eb
2023-01-05 16:32:30 +01:00
Qweder93
8c69ee62fc {cmd/storj-sim, satellite/rangedloop}: added rangedloop to storj-sim, removed identity
added in storj-sim rangedloop for each satellite, to verify it works for metrics oveserver,
removed identity from rangedloop peer as we never use it, added logs on service run, added loop
to service instead of endless for loop, interval value to config

Closes: https://github.com/storj/storj/issues/5414

Change-Id: Ibc3b06071b68feda4a35b45da2bbe36e22a02fc8
2023-01-05 11:29:00 +00:00
Cameron
5138bb1e4f satellite/console: add public ID field to graphql Project
Add public ID field to graphql Project so it can be used on the front
end. Additionally public_id needed to be added to the ListByOwnerID sql
query which is called by graphql OwnedProjectsQuery.

github issue: https://github.com/storj/storj/issues/5408

Change-Id: I2ec04363c20493dc0f9c70b6d1610f724f18ec2f
2023-01-04 15:00:20 -05:00
paul cannon
6e1554652a cmd/tools/segment-verify: handle dq'd nodes
Previously, if any pieces are still on disqualified nodes, this tool
would treat those pieces as fine (if the disqualified node is still
online) or temporarily unavailable (if the disqualified node is
offline). Instead, we should treat such pieces as lost.

This also fixes a slight problem with the code that handles a broken
alias. This is not likely to happen, but if we do see an alias that is
not in the alias map, we return an error instead of nil.

Change-Id: Ib4e2e729ef0535dd7bd9ce2f621680d9f959891c
2023-01-04 17:54:03 +00:00
nadimhq
63639e862b
docs/testplan: Testplan for Scaling Audit Worker (#5335)
This testplan is going to cover the changes to allow for scaling audit workers. It will go over the scaling audit worker design doc found under blueprints.

Co-authored-by: Antonio Franco (He/Him) <antonio@storj.io>
2023-01-04 18:49:53 +01:00
Ethan Adams
d29abed2aa cmd/satellite: Add run auditor to satellite entrypoint.
Needed to support new auditor processess

Change-Id: I6687a667c123c69a335317216affad3a14ab7b9c
2023-01-04 15:31:06 +00:00