Commit Graph

5598 Commits

Author SHA1 Message Date
Michał Niewrzał
4267a958d4 satellite/satellitedb: use AS OF SYSTEM TIME for GetProjectBandwidth query
Adding AS OF SYSTEM TIME to query that is calculating project bandiwdth.
As an addition method for setting interval is added as test doesn't
work well with default interval.

Change-Id: Id1e15be4f6afff13b9dc2b7f595e2edb6de28db9
2021-06-28 15:19:47 +00:00
Michał Niewrzał
70e6cdfd06 satellite/audit: move to segmentloop
Change-Id: I10e63a1e4b6b62f5cd3098f5922ad3de1ec5af51
2021-06-28 11:32:00 +00:00
Michał Niewrzał
8ce619706b satellite/audit: migrate to new segment_pending_audit table
Currently, pending audit is finding segment by segment location
(path) because we want to move audit to segmentloop and we will
have only StreamID and Position we need to add columns for those
fields. Altering existing table can cause issues while
migration and deployment. Cleaner choise is to make new table.
This change contains migration with new segment_pending_audit
table that will replace pending_audits table and adjustments
to use new table in the code.

Table pending_audits will be dropped with next release.

Change-Id: Id507e29c152da594bac1fd812c78d7ecf45ec51f
2021-06-28 13:19:49 +02:00
Fadila Khadar
a89c0763a0 satellite/gracefulexit: create table graceful_exit_segment_transfer_queue
table graceful_exit_segment_transfer_queue will be used to replace graceful_exit_transfer_queue. Currently, it uses the path of a segment to keep track of pieces to be transferred. As we want to use the segment metainfo loop, we will need to record stream_id and position of the segment instead of relying on object path.

This change also add a uses_segment_transfer_queue column to the graceful_exit_progress table to be able to know if a transfer has been initiated while using the old table.

Change-Id: Iafb1e8e65ba124e20de4a9ff76da181c3222de7e
2021-06-28 12:55:57 +02:00
Vitalii Shpital
d30fd77652 satellite/console: new endpoint to get total usage and limits for all the projects user owns
Added new endpoint and service method to return total usage and limits for all the projects that user owns.
It is needed for new paid tier UI

Change-Id: Ic5b67ca7b275ec4930d976a007168235c0500b70
2021-06-25 21:22:59 +00:00
Egon Elbre
4031336cbd private/lifecycle: monitor unexpected shutdowns
Change-Id: I9af3a95a1b60c1572cd57bb5e9539d60aa0337bb
2021-06-25 19:26:23 +03:00
Egon Elbre
3c8e7e679a mod: bump storj.io/private and monkit
This gives us nicer output for time.Time formatting when looking at
ongoing database queries.

Change-Id: I00c1ae4d162f537bd69fc2c8132c6507c5ee98cb
2021-06-25 16:57:00 +03:00
Michał Niewrzał
1f5fbbd24a satellite/metabase: while deleting bucket objects delete pieces in batches
It's possible that single object will have a lot of segments and
at the moment we are trying to collect all pieces at once and
send to storage nodes information about deletion. Such
approach my lead to using extensive amount of memory. This
change is handling this problem by calling DeletePieces
function multiple times with only part of pieces to delete for
a single call.

Change-Id: Ie1e66cd9d86d130eb89a61cf6e23f38b8cb8859e
2021-06-25 13:27:00 +00:00
Stefan Benten
a59232bb1d satellite: return directly to avoid shadowed err variable
Change-Id: I40046b37ba47f5038e8e0dd303b3ecb279441259
2021-06-24 20:52:13 +02:00
Cameron Ayer
8c124c6fa4 satellite/{reputation,overlay,satellitedb}: create reputation service, DB, add overlay method UpdateReputation
Define service and DB interface for storing node reputation data
and updating the overlay cache.
Add overlay service and DB method UpdateReputation.
See https://github.com/storj/storj/pull/4144

Change-Id: Iedd8bd3274457d26c595919303d55327c1464b8c
2021-06-24 16:19:15 +00:00
Qweder93
cff3da7647 multinode/operators: using old api instead of new until nodes updated.
Change-Id: I210e300f11d03d705ff98e95ab0d9debad59c9d6
2021-06-24 18:37:04 +03:00
nadimhq
e37df44910
Create a New User Business Account Test (#4148)
UItests: added new user (business) account test
2021-06-23 11:49:34 -04:00
Jeremy Wharton
07f8cff08c satellite/satellitedb: Add reputation table
The reputation table duplicates the reputation information in the
nodes table. It will be used for implementing the reputation
service.

Change-Id: I36c0318e8fa5f535e9d527df95b22a4f9eb365d4
2021-06-23 13:26:57 +00:00
igor gaidaienko
b1201df82c satellite/console: add more tests for uplink access permission
Change-Id: Icb93501df70767b36da68ee5b8ffa98ea67d22c0
2021-06-23 11:58:29 +00:00
Qweder93
b9d7874dc2 multinode: old drpc api returned
drpc api with old names added to prevent breaking
backwards compatibility.

Change-Id: I1b8a41f2c1e0bd11ac83c86f0b1bfbfc1f07378f
2021-06-22 23:02:47 +00:00
Jeff Wendling
8a6efa1f58 satellite/orders: query for node first before upsert/replace
the very common case is that the node api version is
indeed at least the requested version, so query for
that first to avoid write traffic.

Change-Id: Ib047d93078205bc07fee75d1f635503b792307f0
2021-06-22 15:16:12 -04:00
andriikotko
70efbd9837
jenkins UI fix (#4146)
* deleted curly bracket

* deleted stage Verification

* added close bracket for pipeline

* bracket deleted again

* new bracket added

* make look according to good practices
2021-06-22 21:04:27 +03:00
Vitalii Shpital
d9741491a1 web/satellite: added cancel upload warning popup
Added warning popup for when file upload is in progress.
It says that leaving objects page will cancel upload

Change-Id: If49c1ddc898f2e6dbdc86e5a00d604a55b628720
2021-06-22 17:12:04 +03:00
Vitalii Shpital
7d3b5b932e web/satellite: replace old fonts with new versions
Replaced old .ttf Inter fonts with new version fonts.
For all modern browsers that use woff2 we save ~700KBs

Change-Id: I937c6e850298ad4bdd1f6c7279c6552c897c1ac4
2021-06-22 13:47:52 +00:00
Egon Elbre
59e3b586e7 satellite/{gracefulexit,overlay}: enable as of system time queries
Change-Id: I2af5eb0e8a51fca7893ce07b78b5633be71dfef8
2021-06-22 11:50:50 +00:00
Michał Niewrzał
b582c974c3 satellite: remove irreparabledb leftovers from code
Change-Id: Iabceea2733d6e0d3ddb26c235ef26ae132a44fc2
2021-06-22 11:26:17 +00:00
nadimhq
f16bb4d198
Create a New User Personal Account Test (#4141)
* added signup personal user test & added testDefault:true to OpenRegistrationEnabled in service.go

* added copyright

* fixed import ordering

* fixed comment formatting and gofmt-ed with -s

* gofmt-ed with -s and -w

* fixed fragile elements

* fixed one more fragile element

* fixed nesting

* removed unnecessary timeout

* fixed imports
2021-06-22 07:06:40 -04:00
Michał Niewrzał
4e645059be satellite: rename Endpoint2 to Endpoint
Change-Id: I91faaa532d5a24f033b9865a89b91bc6821588be
2021-06-22 09:25:05 +00:00
Jeremy Wharton
8a070e7c25 satellite/overlay: Ignore unnecessary check-ins
This prevents the database from being contacted unnecessarily,
reducing load.

Change-Id: Ib2420f68a20636ec35eb3dd3df8e02bd5341b419
2021-06-22 09:00:41 +00:00
Michał Niewrzał
7815e647de cmd: add metabase-expiresat-migration tool
Expires_at column was added to segments table and
we need to migrate this value for existing segments
from corresponding objects. This standalone tool
will read all objects and if expires_at is set then
will send update query for this object segments.

Updates will be send in batches and in parallel.

Change-Id: I1fddf0af8cde0f560582d25c6d0e07a00b58e534
2021-06-22 07:42:49 +00:00
Maximillian von Briesen
036687b58a
docs/blueprints: Create coupon codes blueprint (#4024)
Change-Id: If7b364d1c6db389feaff275d95acf8d9c8e08c81
2021-06-21 19:24:01 -04:00
Egon Elbre
ca64e55281 satellite/gc: remove skip first
We used this to reduce initial load on the core to avoid OOM. However,
this is not a problem anymore with garbage collection running
separately.

Change-Id: Ifd62c822a74974bc21a5913199334469a4bc0130
2021-06-21 18:30:38 +00:00
JT Olio
8fec48f310 storagenode: use storj.io/dcs-satellites now
Change-Id: I4c791b37778757c77872f1c25744efa848a5fed3
2021-06-21 15:11:30 +00:00
Egon Elbre
5f901c8066 satellite/metabase/metaloop: add as of system interval to stats
Table stats query should use AS OF SYSTEM INTERVAL in the query to optimize
the behavior.

Change-Id: Ie14f5441062d3d264bb3c626081e77d6edc7b724
2021-06-17 23:43:41 +03:00
Egon Elbre
9640cc2c06 satellite/metabase/segmentloop: verify processed count
This adds verification for the processed count and before and after
segment/objects table counts.

This adds new flag:

  metainfo.segment-loop.suspicious-processed-ratio: 0.03

This defaults to 3%, which at 100M segments is 3M segments.

Change-Id: I5ee03e913ddc4e67e94010ced126a2a9ea51f41b
2021-06-17 19:46:55 +03:00
Egon Elbre
341033dda1 satellite/metabase/metaloop: verify processed count
This adds verification for the processed count and before and after
segment/objects table counts.

This adds new flag:

  metainfo.loop.suspicious-processed-ratio: 0.03

This defaults to 3%, which at 100M objects is 3M objects.

Change-Id: Ife5522ecc97bcc5a55667f36868a0f1fc8e4c561
2021-06-17 15:23:53 +03:00
Michał Niewrzał
a93e47514a satellite: remove irreparabledb
This is part of metaloop refactoring. We plan to remove
irreparable at some point but there was not time for it.
Now instead refatoring it for segmentloop its just easier
to drop it.

Later we still need to drop table with migration step.

Change-Id: I270e77f119273d39a1ecdcf5e1c37a5662a29ab4
2021-06-17 07:20:15 +00:00
Egon Elbre
0df4a27bf7 satellite/metabase: add method to get table statistics
In loop we need to start verifying it's correctness. This allows to
gather these stats.

Change-Id: I146fb50e2b3658b6f3c2682cdc1983e6abd73c29
2021-06-16 18:34:55 +03:00
Egon Elbre
2ebbc9757f mod: bump storj.io/private
This makes it possible to see queries in mon/ps output.

Change-Id: I0e8b55bd9dbc3d23bc3cc41a71aeeb64d32ce46f
2021-06-15 22:20:23 +03:00
Egon Elbre
9b2607d6ba satellite: remove garbage collection option from core
We don't run it anywhere in this configuration, so it's not worthwhile
to keep it that way.

Change-Id: I88afb8bb3eb3843801b15454408f10d1353596cb
2021-06-15 21:07:02 +03:00
Egon Elbre
08c0025332 satellite: remove contact service from core
It shouldn't be needed in core.

Change-Id: I9150ee2c43d57dcb5382531c80b079306ce443ca
2021-06-15 21:07:02 +03:00
crawter
a89e20895d web/multinode/operator: wallets features type changed to support null
Change-Id: I4e5a3aed9488747d10cd795e81d52e689fd08be0
2021-06-15 19:54:23 +03:00
crawter
d426d4f00d multinode/storage: disk space of concrete node and total disk space implemented
Change-Id: I50a96d3e9a77615e79b826a592c0a07c247e0520
2021-06-15 18:46:54 +03:00
Michał Niewrzał
1a974c327d satellite/metabase: set Segment.ExpiresAt while getting segment
Change-Id: I6504efd19c3516ffcf97ea3923e7f52e9832613b
2021-06-15 13:57:51 +00:00
JT Olio
6949dc0bac satellite/metaloop: missing monitoring on observers
Change-Id: I630fbb0448c8d08b426486b3e49abfbca03332a6
2021-06-15 13:39:13 +00:00
Jeff Wendling
ce87652a8c cmd/uplinkng: rm
this implements the rm command which has to add
a Remove method to the fileystem interface and
implement it for local, remote and test filesystems.

Change-Id: Id41add28f01938893530aae0b4b73c8954e9b715
2021-06-14 15:23:41 -04:00
Jeff Wendling
173d1e638c cmd/uplinkng: meta get
Change-Id: I6cd7e3fe408b42d257c35a6c715016f13559643f
2021-06-14 15:23:41 -04:00
Jeff Wendling
1323e1a666 cmd/uplinkng: rb
just implements the remove bucket command

Change-Id: I096a3c0c4a5fd1c7b018784d361221d1a1f5461a
2021-06-14 15:23:41 -04:00
Jeff Wendling
7fae5654ff cmd/uplinkng: access save: prompt for access
this adds a helper method to prompt for a line of
input using the clingy context to the global flag
state that errors if interactive mode is disabled.

Change-Id: Ie113c8920dfa4719e85cc24f11401d91b32812f9
2021-06-14 15:23:41 -04:00
Jeff Wendling
d73287f043 cmd/uplinkng: tests for cp
this adds some stuff to ultest so that the set of
files created by a test can be inspected after so
that we can write some tests for the cp command
to observe that it does what it is supposed to do.

Change-Id: I98b8fb214058140dfbb117baa7acea6a2cc340e1
2021-06-14 15:23:41 -04:00
Jeff Wendling
98be54b9a3 cmd/uplinkng: refactor into some focused packages
the directory was starting to get pretty large and
it was making it hard to pick concise names for
types and variables. this moves the location
stuff into a cmd/uplinkng/ulloc package, the
filesystem stuff into a cmd/uplinkng/ulfs package,
and the testing stuff into a cmd/uplinkng/ultest
package.

this should make the remaining stuff in cmd/uplinkng
only the business logic of how to implement the
commands, rather than also including a bunch of
helper utilities and scaffolding.

Change-Id: Id0901625ebfff9b1cf2dae52366aceb3b6c8f5b6
2021-06-14 15:23:41 -04:00
Jeff Wendling
b24ea2ead5 cmd/uplinkng: test framework and ls tests
this adds a test framework with fake implementations of a
filesystem so that unit tests can be written asserting
the output of different command invocations as well as
the effects they have on a hypothetical filesystem and
storj network.

it also implements the mb command lol

Change-Id: I134c7ea6bf34f46192956c274a96cb5df7632ac0
2021-06-14 15:23:41 -04:00
Egon Elbre
f1a9b45599 mod: update quic for Go 1.17rc1
Change-Id: I4218b695089189d5efac1101559a83079fb812cd
2021-06-14 10:14:36 +03:00
Yingrong Zhao
81dd7b2f37 storagenode/internalpb: fix protobuf generation for newer protobuf
version

Change-Id: I24cc5568b5f5c36f8c63062f79a4cb8d1a8f164e
2021-06-11 17:20:42 +00:00
Michał Niewrzał
9a113da361 satellite/metabase: expose expires_at with loops
Loops are using custom structs to provide
segments while iterating/looping so we need
to expose new field there too.

Change-Id: I12c8f4a01afeac171bf638d278253999fa90a8cb
2021-06-11 16:51:37 +00:00