Commit Graph

7794 Commits

Author SHA1 Message Date
Egon Elbre
e9bc06608e certificate/certificatepb: add definitions for migration
Also fixing the timestamp in .proto which was actually int64.

Change-Id: I65e3172793a49bf8bfd946b199016204ed72f221
2023-01-25 10:28:36 +02:00
JT Olio
e40191afd6 storj: upgrade to use latest storj/common NodeAddress
Change-Id: I5987391bcfe5f6dfd7b525698c337a4cbda9b76e
2023-01-25 01:37:26 +00:00
Vitalii
c98ef89311 web/satellite: fixed multiple passphrases notification for object browser
Fixed notifications link to toggle manage passphrase modal instead of redirecting to buckets screen.

Issue:
https://github.com/storj/storj/issues/5452

Change-Id: I84ce67da7445c6316eb0658606c0129bb4af680e
2023-01-24 16:21:35 +00:00
Erik van Velzen
eab595397f cmd/uplink: add --analytics flag
Add a flag to enable/disable analytics so uplink can be run
non-interactively. Also when run non-interactively for the first time
it will not error any more but instead default to disable analytics.

Part of https://github.com/storj/storj/issues/5126

Change-Id: I07ac8a040664334efcb4e2536f26c330c1751a6f
2023-01-24 15:33:55 +00:00
Yaroslav Vorobiov
3639c5ee1e satellite/payments/stripecoinpayments: skip paying invoices with due date
When running pay invoices command skip invoices that has due date set, so
customers can still pay it themselves until due date.

https://github.com/storj/storj/issues/5453

Change-Id: I8e557062491ab0c8246b28bc5ca57e845eb32e29
2023-01-24 15:12:29 +01:00
Clement Sam
95960572b3 storagenode/piecestore: improve logs for incoming requests
- Adds "Remote Address" field to all INFO logs related to GET,
PUT, and DELETE requests
- Adds Offset and Size fields to all info logs related to GET
requests

Resolves https://github.com/storj/storj/issues/5404

Change-Id: I5dab1867619385362e5f1e0455dfab17d295a37a
2023-01-24 10:23:34 +00:00
Erik van Velzen
cb01aca13d cmd/uplink: add docker image
Add a docker image for uplink-cli and push it to docker hub.
We used to have this before the change to uplinkng. I'm not
sure if the pushing works, we'll see after merge.

To test, build an image with `make uplink-image`, read the tag from the
output and run normal uplink-cli commands using
`docker run -it storjlabs/uplink:df9bbceca-uplink-docker-go1.18.8-amd64 [command]`

Part of https://github.com/storj/uplink/issues/109

Change-Id: I8a10aab2b778951ff42a22ba2f252c581eb66b65
2023-01-24 09:48:02 +00:00
paul cannon
740cb0d9c7 cmd/tools/segment-verify: fix read-csv subcommand
We were reading in a segment's stream ID and position, and assuming that
was enough for the downloader. But of course, the downloader needs
AliasPieces filled in. So now we request each segment record from the
metabase and fill in the VerifySegment records entirely.

Change-Id: If85236388eb99a65e2cb739aa976bd49ee2b2c89
2023-01-24 09:08:03 +00:00
Michal Niewrzal
8850fde9f5 satellite/metabase/metabasetest: detect full scan table queries
This is automated test around metabase tests. It's detecting queries
which were performing full table scan during test execution.

After merging this and checking that this is not problematic in any way
we will enable this also for testplanet tests.

One query was adjusted to avoid full table scan and pass new check.

https://github.com/storj/storj/issues/5471

Change-Id: I2d176f0c25deda801e8a731b75954f83d18cc1ce
2023-01-23 19:40:20 +00:00
NickolaiYurchenko
bf5b378836 web/satellite: CreditCardContainer and billing Overview migrated to use composition api
Change-Id: I8e0a027671aa996f42edf916c9b0434e773de618
2023-01-23 18:53:25 +00:00
Yaroslav Vorobiov
b86ce0d527 satellite/payments/storjscan: return 404 when there is no wallet
When there is no wallet in the database for a particular customer
return 404 http response status code instread of internal server error.
Change web/satellite payments API to return empty wallet on 404 response
code instead of throwing an error.

Change-Id: Ib44914f9ed002382258968fb81846f2b97dee0fe
2023-01-23 18:07:33 +00:00
Vitalii
d1c230fe4d web/satellite: fix Vuex state clearing on session timeout
Fixed Vuex state clearing on session timeout.
Now state is being cleared right after redirect to login and before logout API call.

Change-Id: I3cddb85735899f0913a273a49335730a4097435e
2023-01-23 17:58:26 +02:00
Qweder93
d6a948f59d satellite/repair : implemented ranged loop observer
implemented observer and partial, created new structures to keep mon
metrics remain in same way as in segment loop

Change-Id: I209c126096c84b94d4717332e56238266f6cd004
2023-01-23 14:23:03 +00:00
Egon Elbre
c320835a38 private/migrate: add version to failed migration step
Change-Id: I98984636f2f7df794d54e5c65cf649e7f0beeded
2023-01-23 15:29:35 +02:00
Wilfred Asomani
250f505f4d web/satellite: fix new folder button in file browser
This change opens the "new folder" modal on click of the "new folder"
button in the file browser

Change-Id: I063d644c70ad1eff005eb509d97b95cf30aca00a
2023-01-23 11:19:12 +00:00
NickolaiYurchenko
b9c0bbce2f web/satellite: access page responsiveness added
navigation logo shrink fixed by updating icon;
https://github.com/storj/storj/issues/5461
https://github.com/storj/storj/issues/5458

Change-Id: I6794cbf1c63786978336810f4c8151dc7825c7c8
2023-01-23 12:06:03 +02:00
NickolaiYurchenko
5a4f089750 web/satellite: Billing history and Coupons components migrated to use composition api
Change-Id: I589af5376af5d94a15069c1d6ff4fbe2ce1c6fb0
2023-01-20 16:16:23 +00:00
Michal Niewrzal
984da95543 Jenkinsfile: collect full table scan queries from tests
This change is first attempt to detect queries which are introducing
bad performance with full table scans. CRDB have special query to list
all queries which performs FTS.

With this change we are collecting all full table scan queries executed while unit tests. Results are combined from all started CRDB nodes into single results file. Such output can be later reviewed. This approach is not perfect but it's just one piece what we would like to build to detect FTS before pushing it into production.

part of https://github.com/storj/storj/issues/5471

Change-Id: I615fd9e624a136bb0870f9ff9c10e4eb5918339c
2023-01-20 12:32:47 +00:00
Michal Niewrzal
4cbb1ed296 satellite/orders: log bandwidth values we are dropping
When we have problem with inserting bandwidth amounts into cache
or DB we are logging information about it but log entries are not
very detailed. This change adds bandwidth amounts to the log entry.

https://github.com/storj/storj/issues/5470

Change-Id: I55ccad837d17b141501d3def1dec7ad5f3acdb0b
2023-01-20 09:28:25 +00:00
JT Olio
8d69837f02 storagenode/piecestore: handle order if provided in first message
we may in the future want to accept orders as part of the initial
request message
(e.g. https://review.dev.storj.io/c/storj/uplink/+/9246).

this change is forward compatible but continues to work with
existing clients.

Change-Id: I475ad50d6cbfee8a1f843383230698e4ef9b9e54
2023-01-20 08:53:22 +00:00
Cameron
7ffa9ef914 satellite/payments: add config for partner package plans
Create a config to specify one-time prices and corresponding coupon
ids for partners.

github issue: https://github.com/storj/storj-private/issues/118

Change-Id: I67b26e7208b12ba8f0e6dc1b164dd9545b09cac0
2023-01-20 02:33:36 +00:00
paul cannon
c3b5c18d00 cmd/tools/segment-verify: learn to take a CSV list of segments as input
This will allow us to retry some specific segments from
segments-retry.csv with particularly high counts of "retry" pieces.

Change-Id: I48fd419cc0350a3be4c9e77ce8d28871565b7f97
2023-01-18 20:53:27 +00:00
Wilfred Asomani
d76f764b72 satellite/web: replace cancel with back buttons
This replaces "Cancel" with "Back" on Clear, Create and Switch steps.
It also removes margin-top from the button rows in Clear, Create steps.

Issue: https://github.com/storj/storj/issues/5463

Change-Id: Ib2ef0dba9b91e82a7cf5b042fad0162128fa071a
2023-01-18 17:51:18 +00:00
Vitalii
65a090d111 satellite/console: enable project level passphrase by default
Enable project level passphrase by default.

Issue:
https://github.com/storj/storj/issues/5456

Change-Id: I8c08f2d81a3ce1b5e0134172b7440309b850c5c2
2023-01-18 15:32:29 +02:00
JT Olio
b022f371d2 blueprint: tcp fastopen
Change-Id: I20fc843b916b5ae0b72acfa5a03cf75f211739c0
2023-01-17 20:35:47 +00:00
JT Olio
14c91aa05d blueprint: noise over tcp simplification
the noise over tcp blueprint had two unresolved questions:

 * what do we do for IPv6?
 * what do we do for UDP support?

this change punts those two questions for later and explicitly
rejects solving them as part of noise over TCP for now.

this change also punts renaming NodeURL to reduce codebase
churn.

Change-Id: Ib209942bae61716b2ff6cea852c336af072d8bc5
2023-01-17 16:06:05 +00:00
Jeremy Wharton
6142b1cd12 web/satellite: show overridden usage prices in the satellite UI
This change allows users who register with a partner that has
different project usage prices to see the correct prices in the
satellite UI.

Resolves storj/storj-private#90

Change-Id: I06bde50db474b25396671a27e282ef5637efe85b
2023-01-17 15:01:57 +00:00
Jeremy Wharton
5d656e66bf satellite/payments/stripecoinpayments: implement invoice price override
This change allows for overriding project usage prices for a specific
partner so that users who sign up with that partner do not need their
invoices to be manually adjusted.

Relates to storj/storj-private#90

Change-Id: Ia54a9cc7c2f8064922bbb15861f974e5dea82d5a
2023-01-17 14:32:10 +00:00
Yaroslav Vorobiov
5644fb1a7e satellite/accounting/nodetally: add ranged loop
Add node tally ranged loop observer and partial.
Add node tally randed observer to range loop peer.
Add config flag to select which loop to use for node tally.
Update satellite core to use segement/ranged loop based on a flag.
Duplicate existing node tally test but using ranged loop.

Change-Id: I6786f1a16933463fab5f79601bf438203a7a5f9e
2023-01-17 13:50:18 +01:00
Egon Elbre
90b7076d26 storagenode/pieces: fix log line
Change-Id: I8dba6b0f3d6af3140dfa503c8d6b33e6808d004f
2023-01-17 11:04:47 +02:00
Vitalii
079728f725 web/satellite: improved project level passphrase experience
Clicking continue in web toggles create project level passphrase which then redirects to project dashboard.
Added new create bucket modal.
Updated open bucket modal.
Updated project dashboard and buckets view to work correctly with no buckets state and no passphrase state.

Issue:
https://github.com/storj/storj/issues/5455

Change-Id: If6ddac7d3365854a02b2bb8898e4742e9d2c31c1
2023-01-17 08:15:06 +00:00
JT Olio
df9bbcecad blueprint: noise over tcp
Change-Id: I3871e540313100a7198d62a455c745f128ac51fb
2023-01-13 16:24:56 +00:00
NickolaiYurchenko
a80bbb9d16 web/satellite: navigation label fixed, add token address truncation changed
console error fixed;
https://github.com/storj/storj/issues/5462
https://github.com/storj/storj/issues/5469

Change-Id: I33e9f5ae09a166fdd8b21ef811fc078d14c10663
2023-01-13 15:42:02 +00:00
NickolaiYurchenko
86fb188be6 web/satellite: Setting.vue migrated to use composition api
added useLoading composable to get rid or repetative same code usage;

Change-Id: I3524efdb1919759046bbf11fe3510f8044adc828
2023-01-13 17:04:30 +02:00
NickolaiYurchenko
a7fc884a2c web/satellite: browser header buttons styles fixed
https://github.com/storj/storj/issues/5459

Change-Id: I75368aabd0faac3d6662b95a31bce4896e6a8341
2023-01-13 14:39:32 +02:00
Andrew Harding
c5b5695bca satellite/metabase/rangedloop: clean up observerstats init
Small cleanups of the observer stats init code:
1. Use sync.Once for race free addition to the monitoring chain
   (purely defensive)
2. Set the observer durations before adding to the monitoring chain on
   first use.
3. observerDurations slice does not need to be initialized to non-nil

Change-Id: I9ae8ec96debc7d52c4ee5d22762a89f21bb2e38c
2023-01-13 10:40:30 +00:00
Egon Elbre
dc6e752dff satellite/satellitedb/dbx: document project.dbx
Updates https://github.com/storj/team-delivery/issues/1

Change-Id: I28bafdc4af4149489439fb5ab6e5bbe1d8f5186b
2023-01-13 11:24:34 +02:00
NickolaiYurchenko
2e28e59a5e web/satellite: cant open bucket on name click bug fixed
https://github.com/storj/storj/issues/5467

Change-Id: I0c91167ddd1d01b15238dbebf10f49dc0cc6c340
2023-01-12 19:03:43 +00:00
Malcolm Bouzi
583ea51fb1 satellite/console: Prevent duplicate project names for the same user.
issue: https://github.com/storj/storj/issues/5247

Change-Id: Ie51f24dca5b9e5064cb06ae238055177bec5975a
2023-01-12 17:18:09 +00:00
Michal Niewrzal
0185bba90a cmd: cleanup segment verify/repair tools
* use the same DB application name for satellite and metabase
* use noop orders DB implementation to avoid storing allocated bandwidth
in DB

Change-Id: I20e88c694d38240fe1a20c45719e210cfb76402c
2023-01-12 15:27:07 +00:00
NickolaiYurchenko
e0f2087245 web/satellite: EncryptForm component migrated to use composition api
Change-Id: Iae41d7bd7b44f53c43b4bf2ab552b5f746915469
2023-01-12 14:35:49 +00:00
Michal Niewrzal
a2a9dafa33 satellite/orders: don't store allocated bandwidth in
bucket_bandwidth_rollups table

We have performance problems with updating bucket_bandwidth_rollups. To
improve situation we can stop storing allocated bandwidth in this table.
This should reduce large number of updates which are comming from
metainfo endpoints, repair workers and audit.

Next step will be to drop `allocated` column completely from
bucket_bandwidth_rollups.

Allocated GET bandwidth is all we need and we are keeping it in
bucket_bandwidth_rollups table.

Change-Id: Ifdd26a89ba8262acbca6d794a6c02883ad0c0c9b
2023-01-12 13:21:02 +00:00
Michal Niewrzal
2876d5f20b satellite/satellitedb: drop project_bandwidth_rollups table
We are not using this table.

Change-Id: I3e4d5d0c2d8522b319caf48fea460b1cc35d5630
2023-01-12 10:23:48 +00:00
Cameron
5da2544e62 satellite/console: support public id as generated api parameter
On generated console api endpoints allow either the project ID or the
public ID to be used as the ID parameter.

github issue: https://github.com/storj/storj/issues/5412

Change-Id: Ic9901ed273931a50ae12f20142a3c4938dfcc8c0
2023-01-11 22:03:35 +00:00
Erik van Velzen
ed910b6087
satellite/metabase/rangedloop: continue after error (#5430)
When an observer errors we still want to finish the other observers.

This changes store the error and continues the loop, skipping
the observer which errored out and setting the duration metric to -1.

When the error occurs in the process stage, it does continue the other
ranges of the same observer. It removes the observer entirely after the process
stage. To improve this would make it more complex due to race
conditions.

Closes https://github.com/storj/storj/issues/5389

Change-Id: I528432c491d4340817d6950f1200ee2b9e703309
2023-01-11 22:23:17 +01:00
Moby von Briesen
3d1007ae18 private/apigen: Change order of operations in api generation
Move the IsAuthenticated check until after initial parameter
parsing/validation. IsAuthenticated will be more expensive than
parsing/validation, so we should fail before auth if possible.

Change-Id: I96a020892eabcb750e8ec9ecc1d8b7d9bf8bf573
2023-01-11 16:54:53 +00:00
Erik van Velzen
2d863759b0 satellite/metabase/rangedloop: add AsOfSystemTime
Add option AsOfSystemTime to segment provider to make it equivalent to
the old segment loop.

There's no comment on what it does because it's pretty complex and
makes no sense, but we can improve it later.

closes https://github.com/storj/storj/issues/5434

Change-Id: I8f09b03803e681e2fd41008c5dba67804b0f37a1
2023-01-11 16:22:18 +00:00
prerna-parashar
6e9567cb25
satellite/analytics: Track Project Details Updates (#5451) 2023-01-11 21:25:41 +05:30
Clement Sam
b5d0021fb6 cmd/storagenode-updater: restart storagenode after update on BSD unix derivatives
On BSD, the storagenode-updater falls back to renaming the storagenode without
doing anything to restart the service.
Like the approach we have for linux, this change finds the process ID of the
storagenode using pgrep and sends an interrupt signal to the process.

Closes https://github.com/storj/storj/issues/5333

Change-Id: Icced90ea3e831528804784c2170a3b8b14952e8c
2023-01-11 12:38:26 +00:00
Michal Niewrzal
282aaf8945 satellite/metabase: fix GetStreamPieceCountByNodeID full table scan
Previous version of SQL query was causing full table scan.

Output of EXPLAIN:
---
distribution: local
vectorized: true

• lookup join
│ table: segments@segments_pkey
│ equality: (?column?) = (stream_id)
│ pred: remote_alias_pieces IS NOT NULL
│
└── • union
    │
    ├── • values
    │     size: 1 column, 1 row
    │
    └── • scan
          missing stats
          table: segment_copies@segment_copies_pkey
          spans: [/'\xff135285155378d980b8c49148cef3ca' - /'\xff135285155378d980b8c49148cef3ca']
---

Change-Id: I708d1df204ac2d33cefe80b23594442b193424d2
2023-01-10 23:35:22 +00:00