Commit Graph

6036 Commits

Author SHA1 Message Date
Jeremy Wharton
07f8cff08c satellite/satellitedb: Add reputation table
The reputation table duplicates the reputation information in the
nodes table. It will be used for implementing the reputation
service.

Change-Id: I36c0318e8fa5f535e9d527df95b22a4f9eb365d4
2021-06-23 13:26:57 +00:00
igor gaidaienko
b1201df82c satellite/console: add more tests for uplink access permission
Change-Id: Icb93501df70767b36da68ee5b8ffa98ea67d22c0
2021-06-23 11:58:29 +00:00
Qweder93
b9d7874dc2 multinode: old drpc api returned
drpc api with old names added to prevent breaking
backwards compatibility.

Change-Id: I1b8a41f2c1e0bd11ac83c86f0b1bfbfc1f07378f
2021-06-22 23:02:47 +00:00
Jeff Wendling
8a6efa1f58 satellite/orders: query for node first before upsert/replace
the very common case is that the node api version is
indeed at least the requested version, so query for
that first to avoid write traffic.

Change-Id: Ib047d93078205bc07fee75d1f635503b792307f0
2021-06-22 15:16:12 -04:00
andriikotko
70efbd9837
jenkins UI fix (#4146)
* deleted curly bracket

* deleted stage Verification

* added close bracket for pipeline

* bracket deleted again

* new bracket added

* make look according to good practices
2021-06-22 21:04:27 +03:00
Vitalii Shpital
d9741491a1 web/satellite: added cancel upload warning popup
Added warning popup for when file upload is in progress.
It says that leaving objects page will cancel upload

Change-Id: If49c1ddc898f2e6dbdc86e5a00d604a55b628720
2021-06-22 17:12:04 +03:00
Vitalii Shpital
7d3b5b932e web/satellite: replace old fonts with new versions
Replaced old .ttf Inter fonts with new version fonts.
For all modern browsers that use woff2 we save ~700KBs

Change-Id: I937c6e850298ad4bdd1f6c7279c6552c897c1ac4
2021-06-22 13:47:52 +00:00
Egon Elbre
59e3b586e7 satellite/{gracefulexit,overlay}: enable as of system time queries
Change-Id: I2af5eb0e8a51fca7893ce07b78b5633be71dfef8
2021-06-22 11:50:50 +00:00
Michał Niewrzał
b582c974c3 satellite: remove irreparabledb leftovers from code
Change-Id: Iabceea2733d6e0d3ddb26c235ef26ae132a44fc2
2021-06-22 11:26:17 +00:00
nadimhq
f16bb4d198
Create a New User Personal Account Test (#4141)
* added signup personal user test & added testDefault:true to OpenRegistrationEnabled in service.go

* added copyright

* fixed import ordering

* fixed comment formatting and gofmt-ed with -s

* gofmt-ed with -s and -w

* fixed fragile elements

* fixed one more fragile element

* fixed nesting

* removed unnecessary timeout

* fixed imports
2021-06-22 07:06:40 -04:00
Michał Niewrzał
4e645059be satellite: rename Endpoint2 to Endpoint
Change-Id: I91faaa532d5a24f033b9865a89b91bc6821588be
2021-06-22 09:25:05 +00:00
Jeremy Wharton
8a070e7c25 satellite/overlay: Ignore unnecessary check-ins
This prevents the database from being contacted unnecessarily,
reducing load.

Change-Id: Ib2420f68a20636ec35eb3dd3df8e02bd5341b419
2021-06-22 09:00:41 +00:00
Michał Niewrzał
7815e647de cmd: add metabase-expiresat-migration tool
Expires_at column was added to segments table and
we need to migrate this value for existing segments
from corresponding objects. This standalone tool
will read all objects and if expires_at is set then
will send update query for this object segments.

Updates will be send in batches and in parallel.

Change-Id: I1fddf0af8cde0f560582d25c6d0e07a00b58e534
2021-06-22 07:42:49 +00:00
Maximillian von Briesen
036687b58a
docs/blueprints: Create coupon codes blueprint (#4024)
Change-Id: If7b364d1c6db389feaff275d95acf8d9c8e08c81
2021-06-21 19:24:01 -04:00
Egon Elbre
ca64e55281 satellite/gc: remove skip first
We used this to reduce initial load on the core to avoid OOM. However,
this is not a problem anymore with garbage collection running
separately.

Change-Id: Ifd62c822a74974bc21a5913199334469a4bc0130
2021-06-21 18:30:38 +00:00
JT Olio
8fec48f310 storagenode: use storj.io/dcs-satellites now
Change-Id: I4c791b37778757c77872f1c25744efa848a5fed3
2021-06-21 15:11:30 +00:00
Egon Elbre
5f901c8066 satellite/metabase/metaloop: add as of system interval to stats
Table stats query should use AS OF SYSTEM INTERVAL in the query to optimize
the behavior.

Change-Id: Ie14f5441062d3d264bb3c626081e77d6edc7b724
2021-06-17 23:43:41 +03:00
Egon Elbre
9640cc2c06 satellite/metabase/segmentloop: verify processed count
This adds verification for the processed count and before and after
segment/objects table counts.

This adds new flag:

  metainfo.segment-loop.suspicious-processed-ratio: 0.03

This defaults to 3%, which at 100M segments is 3M segments.

Change-Id: I5ee03e913ddc4e67e94010ced126a2a9ea51f41b
2021-06-17 19:46:55 +03:00
Egon Elbre
341033dda1 satellite/metabase/metaloop: verify processed count
This adds verification for the processed count and before and after
segment/objects table counts.

This adds new flag:

  metainfo.loop.suspicious-processed-ratio: 0.03

This defaults to 3%, which at 100M objects is 3M objects.

Change-Id: Ife5522ecc97bcc5a55667f36868a0f1fc8e4c561
2021-06-17 15:23:53 +03:00
Michał Niewrzał
a93e47514a satellite: remove irreparabledb
This is part of metaloop refactoring. We plan to remove
irreparable at some point but there was not time for it.
Now instead refatoring it for segmentloop its just easier
to drop it.

Later we still need to drop table with migration step.

Change-Id: I270e77f119273d39a1ecdcf5e1c37a5662a29ab4
2021-06-17 07:20:15 +00:00
Egon Elbre
0df4a27bf7 satellite/metabase: add method to get table statistics
In loop we need to start verifying it's correctness. This allows to
gather these stats.

Change-Id: I146fb50e2b3658b6f3c2682cdc1983e6abd73c29
2021-06-16 18:34:55 +03:00
Egon Elbre
2ebbc9757f mod: bump storj.io/private
This makes it possible to see queries in mon/ps output.

Change-Id: I0e8b55bd9dbc3d23bc3cc41a71aeeb64d32ce46f
2021-06-15 22:20:23 +03:00
Egon Elbre
9b2607d6ba satellite: remove garbage collection option from core
We don't run it anywhere in this configuration, so it's not worthwhile
to keep it that way.

Change-Id: I88afb8bb3eb3843801b15454408f10d1353596cb
2021-06-15 21:07:02 +03:00
Egon Elbre
08c0025332 satellite: remove contact service from core
It shouldn't be needed in core.

Change-Id: I9150ee2c43d57dcb5382531c80b079306ce443ca
2021-06-15 21:07:02 +03:00
crawter
a89e20895d web/multinode/operator: wallets features type changed to support null
Change-Id: I4e5a3aed9488747d10cd795e81d52e689fd08be0
2021-06-15 19:54:23 +03:00
crawter
d426d4f00d multinode/storage: disk space of concrete node and total disk space implemented
Change-Id: I50a96d3e9a77615e79b826a592c0a07c247e0520
2021-06-15 18:46:54 +03:00
Michał Niewrzał
1a974c327d satellite/metabase: set Segment.ExpiresAt while getting segment
Change-Id: I6504efd19c3516ffcf97ea3923e7f52e9832613b
2021-06-15 13:57:51 +00:00
JT Olio
6949dc0bac satellite/metaloop: missing monitoring on observers
Change-Id: I630fbb0448c8d08b426486b3e49abfbca03332a6
2021-06-15 13:39:13 +00:00
Jeff Wendling
ce87652a8c cmd/uplinkng: rm
this implements the rm command which has to add
a Remove method to the fileystem interface and
implement it for local, remote and test filesystems.

Change-Id: Id41add28f01938893530aae0b4b73c8954e9b715
2021-06-14 15:23:41 -04:00
Jeff Wendling
173d1e638c cmd/uplinkng: meta get
Change-Id: I6cd7e3fe408b42d257c35a6c715016f13559643f
2021-06-14 15:23:41 -04:00
Jeff Wendling
1323e1a666 cmd/uplinkng: rb
just implements the remove bucket command

Change-Id: I096a3c0c4a5fd1c7b018784d361221d1a1f5461a
2021-06-14 15:23:41 -04:00
Jeff Wendling
7fae5654ff cmd/uplinkng: access save: prompt for access
this adds a helper method to prompt for a line of
input using the clingy context to the global flag
state that errors if interactive mode is disabled.

Change-Id: Ie113c8920dfa4719e85cc24f11401d91b32812f9
2021-06-14 15:23:41 -04:00
Jeff Wendling
d73287f043 cmd/uplinkng: tests for cp
this adds some stuff to ultest so that the set of
files created by a test can be inspected after so
that we can write some tests for the cp command
to observe that it does what it is supposed to do.

Change-Id: I98b8fb214058140dfbb117baa7acea6a2cc340e1
2021-06-14 15:23:41 -04:00
Jeff Wendling
98be54b9a3 cmd/uplinkng: refactor into some focused packages
the directory was starting to get pretty large and
it was making it hard to pick concise names for
types and variables. this moves the location
stuff into a cmd/uplinkng/ulloc package, the
filesystem stuff into a cmd/uplinkng/ulfs package,
and the testing stuff into a cmd/uplinkng/ultest
package.

this should make the remaining stuff in cmd/uplinkng
only the business logic of how to implement the
commands, rather than also including a bunch of
helper utilities and scaffolding.

Change-Id: Id0901625ebfff9b1cf2dae52366aceb3b6c8f5b6
2021-06-14 15:23:41 -04:00
Jeff Wendling
b24ea2ead5 cmd/uplinkng: test framework and ls tests
this adds a test framework with fake implementations of a
filesystem so that unit tests can be written asserting
the output of different command invocations as well as
the effects they have on a hypothetical filesystem and
storj network.

it also implements the mb command lol

Change-Id: I134c7ea6bf34f46192956c274a96cb5df7632ac0
2021-06-14 15:23:41 -04:00
Egon Elbre
f1a9b45599 mod: update quic for Go 1.17rc1
Change-Id: I4218b695089189d5efac1101559a83079fb812cd
2021-06-14 10:14:36 +03:00
Yingrong Zhao
81dd7b2f37 storagenode/internalpb: fix protobuf generation for newer protobuf
version

Change-Id: I24cc5568b5f5c36f8c63062f79a4cb8d1a8f164e
2021-06-11 17:20:42 +00:00
Michał Niewrzał
9a113da361 satellite/metabase: expose expires_at with loops
Loops are using custom structs to provide
segments while iterating/looping so we need
to expose new field there too.

Change-Id: I12c8f4a01afeac171bf638d278253999fa90a8cb
2021-06-11 16:51:37 +00:00
Qweder93
5d70b6abef multinode/bandwidth: added monthly bandwidth summaries
montly bandwidth summaries for single/all nodes, single/all satellites added.

Change-Id: Ic384886c10622df74a4bd0645e2d7f2a85477644
2021-06-11 16:31:59 +00:00
Egon Elbre
f3a52d1da5 satellite/metabase/segmentloop: limit max interval
Ensure that we don't query too far in the history, which slows things
down.

Change-Id: Ia77aa522f7f4c5d43629d51bb9a51a49fab6fa14
2021-06-10 17:06:33 +00:00
Michał Niewrzał
bc79f01aaa satellite/metabase: set expires_at while committing segment
We added expires_at column to segments table and now
we need to populate this column while committing segment.
We still need to migrate existing segments with
separate tool.

Change-Id: Ibac8c63d97201dd98cc2cb9db385f4cb73bc3f7e
2021-06-10 16:32:28 +00:00
Yaroslav Vorobiov
157d3d980e multinode/console: storage usage and total storage usage
Change-Id: I4970275daf4a4a9c5d02aea6a205891869dd4eff
2021-06-10 16:01:41 +00:00
Vitalii Shpital
ed28fa3ff9 web/satellite: added loaders across all the UI. Removed most of the requests from initial load
Added loader spinners across all of the UI to be visible while data is being fetched.
Removed most of the requests from the initial load of the satellite dashboard.
Removed useless requests after creating of new projects.
This should make user's experience much more better since load time of the app is much lower than it was before.

Change-Id: Ib0941ad4eee6b3caf781d132062b55cb17703fe7
2021-06-10 15:16:52 +00:00
Jeremy Wharton
c27da95742
Update index.html to be consistent with deployed code (#4140)
This change is part of an ongoing effort to remove code from our branding repository.

Co-authored-by: Stefan Benten <mail@stefan-benten.de>
2021-06-10 09:28:57 -05:00
Yaroslav Vorobiov
b23a782c6b multinode/console: held amount summary use satellite address
Change-Id: I7816a263579831eaaeb808e3cdd49b3b7b0a07b8
2021-06-10 15:55:43 +03:00
Jeff Wendling
d674bc9c52 satellite/audit: include failing segment info in logs
Change-Id: I972fe19a2479f48bccc8a87a282467345a9dc1ec
2021-06-10 13:47:22 +03:00
Jeff Wendling
944bceabcd satellite/audit: fix reservoir sampling bias
Change-Id: Icc522fd86538b8182a1b7d42c1588c32a257acaf
2021-06-10 13:47:22 +03:00
Egon Elbre
1cd13adc1c ci: clearer initial build
We can run a lot of the code building and downloading steps in parallel.
We were using Makefile, however, that doesn't integrate with Jenkins,
which means, when there's a failure, you have no clue where.

This separates the concurrent build steps and makes it clear what we are
running concurrently.

Change-Id: I1c1dccd09dab67d2ef428a58d5eb9e004b11a74c
2021-06-09 12:30:31 +00:00
andriikotko
072dd24696
integration/ui: add UI integration tests
Introduce a new `integration/ui` package for integration tests.

`integration/ui/uitest` contains a helper package for running browser tests.
2021-06-09 14:39:38 +03:00
crawter
962f81433f web/multinode: operator entity extended with nodeID and undistributed amount
Change-Id: I683e53bc4c669728caae3521597a5fd2d6372921
2021-06-09 08:51:15 +00:00