Commit Graph

5616 Commits

Author SHA1 Message Date
Vitalii Shpital
bab43af6ce web/satellite: added Upgrade to Paid Tier banner
Added new info banner to show user their used and total storage values with a button to upgrade to Paid Tier with auto limit increase

Change-Id: I827818dcb5179358df246218a47feb61bc1a1bac
2021-07-01 15:05:03 +00:00
NickolaiYurchenko
1230cde317 web/multinode: disk space usage and stat charts
Change-Id: Ib2b1d6f2b78d57bd43ae1152921d98a1d6c31243
2021-07-01 14:21:08 +00:00
Michał Niewrzał
b900f6b4f9 satellite/repair/checker: move checker to segment loop
Change-Id: I04b25e4fa14c822c9524586c25bde89db2a6cad9
2021-07-01 13:51:56 +00:00
NickolaiYurchenko
8686267e06 web/multinode: storage api, service and store module created
Change-Id: Ieb3dbcd9c967388315f5203598ff56a848477476
2021-07-01 12:32:09 +00:00
NickolaiYurchenko
55754df110 web/multinode: bandwidth related unit tests
Change-Id: If8da681e15db9e6f9f333bacdeab7670c48f2dfb
2021-07-01 12:32:04 +00:00
NickolaiYurchenko
cece8e4110 web/multinode: bandwidth charts added
bandwidth/egress/ingress chart and api/vuex connection

Change-Id: I16ba2bb82854a1d198384b3b8e6ffc4e58d8bb91
2021-07-01 12:31:58 +00:00
NickolaiYurchenko
cc5de4288b web/multinode: bandwidth api and store module created
Change-Id: I5a280f836d73ca9c34bab7dd5371bf1dce5a3974
2021-07-01 12:31:53 +00:00
Michał Niewrzał
0ca7583282 satellite/accounting: add total for bytes and segments to tallies
We want to calculate bucket tally only from iterating objects.
Object currently has an info about totals for bytes and segments.
We need to adjust tallies to keep those totals. Older entries will
be untouched and code will use totals only if available. Change
is adding columns for totals to bucket_storage_tally table and
is adding general handling for them.

Next step is to start using total columns instead of inline/remote.
This will be done with next change.

Change-Id: I37fed1b327789efcf1d0570318aee3045db17fad
2021-07-01 08:52:32 +00:00
TungHoang
e1379bea0f
storagenode/piecestore: allow rejecting slow clients
Estimate speed of the uploads by calculating the time a client has been in a non-congested state. Storage node is uncongested when it has used up over 80% of its maximum concurrent requests capacity.

Currently it's disabled by default.
2021-07-01 09:59:04 +03:00
Michał Niewrzał
4335b21332 satellite/metabase: add TotalEncryptedSize to LoopObjectEntry
We gonna need TotalEncryptedSize for buckets tally migrated to objects
iterator from metaloop.

Change-Id: I64774fdbda47abdaf5fbd370bd015b14bac9afdf
2021-06-30 16:43:31 +00:00
Michał Niewrzał
d53aacc058 satellite/repair: migrate to new repair_queue table
We want to use StreamID/Position to identify injured
segment. As it is hard to alter existing injuredsegments
table we are adding a new table that will replace existing
one. Old table will be dropped later.

Change-Id: I0d3b06522645013178b6678c19378ebafe485c49
2021-06-30 17:12:24 +02:00
Moby von Briesen
0ec3867ec0 satellitedb: add paid_tier column to users table
So that we can easily see whether a user is in the paid tier without
querying for payment methods.

Change-Id: I122566ddd0953203f852741fa12c71795bc1ec5c
2021-06-30 15:00:38 +00:00
Michał Niewrzał
aa7fd8c9cd satellite/satellitedb: fix ProjectAccounting project bandwidth calculation
Period end was calculated
incorrectly as it was still in current month but
should be the first day of next month.

Change-Id: I37451d29a9b901b69e6c3c401b333c58b3376d61
2021-06-30 14:54:12 +02:00
JT Olio
1c3688f1d6 storage/filestore: include satellite id in open_file_in_trash metrics
Change-Id: Ib7aa15f84c3acd9eedc4778bd2cde3072d0bf64b
2021-06-29 23:25:25 +00:00
JT Olio
cb18dc77fc satellite/stripecoinpayments: version is the wrong name
this service exists to do currency conversions, which is
the best I can assume that this was meant to be named.

Change-Id: Ia2416f5475749e8bfe8d05bf491649576f6d77bf
2021-06-29 15:31:11 +00:00
Moby von Briesen
d999a963ca satellite/console/.../consoleapi: Standardize serveJSONError
This change removes all the separate implementations for
`apiservice.serveJSONError()` and defines one for every service to use
in `consoleapi/common.go`.

Change-Id: Iabf184e5cba69a98eb25936ce11ebd07f02c8ff3
2021-06-29 09:40:07 -04:00
Michał Niewrzał
371517d93b satellite/accounting: fix as of system interval value for some tests
Live accounting has now as of system interval config but it was set incorrectly for some tests.

Change-Id: I6397d858c50e129bb8f9cd1a972047582ee9d59d
2021-06-29 10:16:48 +00:00
Moby von Briesen
4e95d27033 web,satellite: Remove paywall-related functionality
Because of our free/paid tier plan, we do not need a paywall anymore. We
have not used it in a while, but still have leftover code laying around.

Change-Id: Iaea8c39faf042a2f7a6b837727bb135c8bdf2907
2021-06-29 02:47:48 +02:00
Michał Niewrzał
4267a958d4 satellite/satellitedb: use AS OF SYSTEM TIME for GetProjectBandwidth query
Adding AS OF SYSTEM TIME to query that is calculating project bandiwdth.
As an addition method for setting interval is added as test doesn't
work well with default interval.

Change-Id: Id1e15be4f6afff13b9dc2b7f595e2edb6de28db9
2021-06-28 15:19:47 +00:00
Michał Niewrzał
70e6cdfd06 satellite/audit: move to segmentloop
Change-Id: I10e63a1e4b6b62f5cd3098f5922ad3de1ec5af51
2021-06-28 11:32:00 +00:00
Michał Niewrzał
8ce619706b satellite/audit: migrate to new segment_pending_audit table
Currently, pending audit is finding segment by segment location
(path) because we want to move audit to segmentloop and we will
have only StreamID and Position we need to add columns for those
fields. Altering existing table can cause issues while
migration and deployment. Cleaner choise is to make new table.
This change contains migration with new segment_pending_audit
table that will replace pending_audits table and adjustments
to use new table in the code.

Table pending_audits will be dropped with next release.

Change-Id: Id507e29c152da594bac1fd812c78d7ecf45ec51f
2021-06-28 13:19:49 +02:00
Fadila Khadar
a89c0763a0 satellite/gracefulexit: create table graceful_exit_segment_transfer_queue
table graceful_exit_segment_transfer_queue will be used to replace graceful_exit_transfer_queue. Currently, it uses the path of a segment to keep track of pieces to be transferred. As we want to use the segment metainfo loop, we will need to record stream_id and position of the segment instead of relying on object path.

This change also add a uses_segment_transfer_queue column to the graceful_exit_progress table to be able to know if a transfer has been initiated while using the old table.

Change-Id: Iafb1e8e65ba124e20de4a9ff76da181c3222de7e
2021-06-28 12:55:57 +02:00
Vitalii Shpital
d30fd77652 satellite/console: new endpoint to get total usage and limits for all the projects user owns
Added new endpoint and service method to return total usage and limits for all the projects that user owns.
It is needed for new paid tier UI

Change-Id: Ic5b67ca7b275ec4930d976a007168235c0500b70
2021-06-25 21:22:59 +00:00
Egon Elbre
4031336cbd private/lifecycle: monitor unexpected shutdowns
Change-Id: I9af3a95a1b60c1572cd57bb5e9539d60aa0337bb
2021-06-25 19:26:23 +03:00
Egon Elbre
3c8e7e679a mod: bump storj.io/private and monkit
This gives us nicer output for time.Time formatting when looking at
ongoing database queries.

Change-Id: I00c1ae4d162f537bd69fc2c8132c6507c5ee98cb
2021-06-25 16:57:00 +03:00
Michał Niewrzał
1f5fbbd24a satellite/metabase: while deleting bucket objects delete pieces in batches
It's possible that single object will have a lot of segments and
at the moment we are trying to collect all pieces at once and
send to storage nodes information about deletion. Such
approach my lead to using extensive amount of memory. This
change is handling this problem by calling DeletePieces
function multiple times with only part of pieces to delete for
a single call.

Change-Id: Ie1e66cd9d86d130eb89a61cf6e23f38b8cb8859e
2021-06-25 13:27:00 +00:00
Stefan Benten
a59232bb1d satellite: return directly to avoid shadowed err variable
Change-Id: I40046b37ba47f5038e8e0dd303b3ecb279441259
2021-06-24 20:52:13 +02:00
Cameron Ayer
8c124c6fa4 satellite/{reputation,overlay,satellitedb}: create reputation service, DB, add overlay method UpdateReputation
Define service and DB interface for storing node reputation data
and updating the overlay cache.
Add overlay service and DB method UpdateReputation.
See https://github.com/storj/storj/pull/4144

Change-Id: Iedd8bd3274457d26c595919303d55327c1464b8c
2021-06-24 16:19:15 +00:00
Qweder93
cff3da7647 multinode/operators: using old api instead of new until nodes updated.
Change-Id: I210e300f11d03d705ff98e95ab0d9debad59c9d6
2021-06-24 18:37:04 +03:00
nadimhq
e37df44910
Create a New User Business Account Test (#4148)
UItests: added new user (business) account test
2021-06-23 11:49:34 -04:00
Jeremy Wharton
07f8cff08c satellite/satellitedb: Add reputation table
The reputation table duplicates the reputation information in the
nodes table. It will be used for implementing the reputation
service.

Change-Id: I36c0318e8fa5f535e9d527df95b22a4f9eb365d4
2021-06-23 13:26:57 +00:00
igor gaidaienko
b1201df82c satellite/console: add more tests for uplink access permission
Change-Id: Icb93501df70767b36da68ee5b8ffa98ea67d22c0
2021-06-23 11:58:29 +00:00
Qweder93
b9d7874dc2 multinode: old drpc api returned
drpc api with old names added to prevent breaking
backwards compatibility.

Change-Id: I1b8a41f2c1e0bd11ac83c86f0b1bfbfc1f07378f
2021-06-22 23:02:47 +00:00
Jeff Wendling
8a6efa1f58 satellite/orders: query for node first before upsert/replace
the very common case is that the node api version is
indeed at least the requested version, so query for
that first to avoid write traffic.

Change-Id: Ib047d93078205bc07fee75d1f635503b792307f0
2021-06-22 15:16:12 -04:00
andriikotko
70efbd9837
jenkins UI fix (#4146)
* deleted curly bracket

* deleted stage Verification

* added close bracket for pipeline

* bracket deleted again

* new bracket added

* make look according to good practices
2021-06-22 21:04:27 +03:00
Vitalii Shpital
d9741491a1 web/satellite: added cancel upload warning popup
Added warning popup for when file upload is in progress.
It says that leaving objects page will cancel upload

Change-Id: If49c1ddc898f2e6dbdc86e5a00d604a55b628720
2021-06-22 17:12:04 +03:00
Vitalii Shpital
7d3b5b932e web/satellite: replace old fonts with new versions
Replaced old .ttf Inter fonts with new version fonts.
For all modern browsers that use woff2 we save ~700KBs

Change-Id: I937c6e850298ad4bdd1f6c7279c6552c897c1ac4
2021-06-22 13:47:52 +00:00
Egon Elbre
59e3b586e7 satellite/{gracefulexit,overlay}: enable as of system time queries
Change-Id: I2af5eb0e8a51fca7893ce07b78b5633be71dfef8
2021-06-22 11:50:50 +00:00
Michał Niewrzał
b582c974c3 satellite: remove irreparabledb leftovers from code
Change-Id: Iabceea2733d6e0d3ddb26c235ef26ae132a44fc2
2021-06-22 11:26:17 +00:00
nadimhq
f16bb4d198
Create a New User Personal Account Test (#4141)
* added signup personal user test & added testDefault:true to OpenRegistrationEnabled in service.go

* added copyright

* fixed import ordering

* fixed comment formatting and gofmt-ed with -s

* gofmt-ed with -s and -w

* fixed fragile elements

* fixed one more fragile element

* fixed nesting

* removed unnecessary timeout

* fixed imports
2021-06-22 07:06:40 -04:00
Michał Niewrzał
4e645059be satellite: rename Endpoint2 to Endpoint
Change-Id: I91faaa532d5a24f033b9865a89b91bc6821588be
2021-06-22 09:25:05 +00:00
Jeremy Wharton
8a070e7c25 satellite/overlay: Ignore unnecessary check-ins
This prevents the database from being contacted unnecessarily,
reducing load.

Change-Id: Ib2420f68a20636ec35eb3dd3df8e02bd5341b419
2021-06-22 09:00:41 +00:00
Michał Niewrzał
7815e647de cmd: add metabase-expiresat-migration tool
Expires_at column was added to segments table and
we need to migrate this value for existing segments
from corresponding objects. This standalone tool
will read all objects and if expires_at is set then
will send update query for this object segments.

Updates will be send in batches and in parallel.

Change-Id: I1fddf0af8cde0f560582d25c6d0e07a00b58e534
2021-06-22 07:42:49 +00:00
Maximillian von Briesen
036687b58a
docs/blueprints: Create coupon codes blueprint (#4024)
Change-Id: If7b364d1c6db389feaff275d95acf8d9c8e08c81
2021-06-21 19:24:01 -04:00
Egon Elbre
ca64e55281 satellite/gc: remove skip first
We used this to reduce initial load on the core to avoid OOM. However,
this is not a problem anymore with garbage collection running
separately.

Change-Id: Ifd62c822a74974bc21a5913199334469a4bc0130
2021-06-21 18:30:38 +00:00
JT Olio
8fec48f310 storagenode: use storj.io/dcs-satellites now
Change-Id: I4c791b37778757c77872f1c25744efa848a5fed3
2021-06-21 15:11:30 +00:00
Egon Elbre
5f901c8066 satellite/metabase/metaloop: add as of system interval to stats
Table stats query should use AS OF SYSTEM INTERVAL in the query to optimize
the behavior.

Change-Id: Ie14f5441062d3d264bb3c626081e77d6edc7b724
2021-06-17 23:43:41 +03:00
Egon Elbre
9640cc2c06 satellite/metabase/segmentloop: verify processed count
This adds verification for the processed count and before and after
segment/objects table counts.

This adds new flag:

  metainfo.segment-loop.suspicious-processed-ratio: 0.03

This defaults to 3%, which at 100M segments is 3M segments.

Change-Id: I5ee03e913ddc4e67e94010ced126a2a9ea51f41b
2021-06-17 19:46:55 +03:00
Egon Elbre
341033dda1 satellite/metabase/metaloop: verify processed count
This adds verification for the processed count and before and after
segment/objects table counts.

This adds new flag:

  metainfo.loop.suspicious-processed-ratio: 0.03

This defaults to 3%, which at 100M objects is 3M objects.

Change-Id: Ife5522ecc97bcc5a55667f36868a0f1fc8e4c561
2021-06-17 15:23:53 +03:00
Michał Niewrzał
a93e47514a satellite: remove irreparabledb
This is part of metaloop refactoring. We plan to remove
irreparable at some point but there was not time for it.
Now instead refatoring it for segmentloop its just easier
to drop it.

Later we still need to drop table with migration step.

Change-Id: I270e77f119273d39a1ecdcf5e1c37a5662a29ab4
2021-06-17 07:20:15 +00:00