Commit Graph

5575 Commits

Author SHA1 Message Date
Egon Elbre
e7cf369cd8 satellite/metabase: use pgxutil.Conn helper
Clean up underlying pgx.Conn extraction with helper.

Change-Id: Ie43f663a6036c439e675df54fccd3605c4d8d69e
2021-05-25 05:56:35 +00:00
Jeff Wendling
e460dc51f7 cmd/uplinkng: implement object listing
Change-Id: Ib5f6964a0c42718913a680529bb66c6f475aeac9
2021-05-24 23:02:19 +00:00
Egon Elbre
cdcc67207c satellite/satellitedb: fix nil panic in UpdateCheckIn
Change-Id: If6ae2c3d9b7c269b0a9d652e68854091f668b5ec
2021-05-25 00:30:36 +03:00
Qweder93
79172777bd mnd/payouts: estimations replaced with expectations
Added expectations endpoint (estimations and distributed), added
coalesce to db query, so in case of empty payouts db 0 will be returned instead of error.

Change-Id: I535f14ef097876448d8949bc302895b25da2b6e7
2021-05-24 18:11:30 +00:00
Qweder93
83e82eb473 mnd/payouts: estimated payouts per satellite changed to per node
Change-Id: I231e95ece1f52accb17839a00bfc1588b6e2a658
2021-05-24 19:47:57 +03:00
Egon Elbre
515f9b54ff mod: bump monkit
Change-Id: I9be05b91cd9171ebc7d053cd087d6761150d44b6
2021-05-24 16:20:00 +00:00
Qweder93
f0dd7f3739 mnd/payouts: undistributed payouts added
Change-Id: Id61ac58fceca13b9a9e12a7e03c3d9624cce3fea
2021-05-24 18:20:08 +03:00
Vitalii Shpital
2896a2736c web/satellite: remove time caveats for object browser AGs
WHAT:
removed notBefore and notAfter caveats from object browser AGs

WHY:
caused a bug when user's clock is not synced correctly

Change-Id: I1af10418f1668003a09cba313eaed3b0764eabdc
2021-05-24 14:50:17 +00:00
NickolaiYurchenko
f2842a27e2 web/multinode: by node payouts page markup
Change-Id: I4d760a9965d28afc37f4cfa971d95254bfcb1d7e
2021-05-24 17:28:25 +03:00
Michał Niewrzał
c9d4674859 cmd: remove metabase-createdat-migration tool
We finished migration for all satellites and we can now remove this
code.

Change-Id: If92091ba94b51c319528dc95845042c9dfd6d074
2021-05-24 09:37:40 +02:00
Yaroslav Vorobiov
f0d60a6ec2 Jenkins: add multinode UI stage
Change-Id: I77ed791ff9dbb5b046450ab625c1763b34691888
2021-05-21 15:49:31 +03:00
Yaroslav Vorobiov
1ff2e1a474 web/multinode: update test snapshots
Change-Id: I8a7adeba1b48c754dc50e68c479f89903cd98486
2021-05-21 15:46:46 +03:00
Egon Elbre
2d3c40fd71 cmd/internal/asset: delete unused code
Change-Id: I44970a4f5ac42dc3a297263cd64150e94c40f62a
2021-05-20 18:30:19 +03:00
Vitalii Shpital
a9fcbf7284 web/satellite: bump object browser version
WHAT:
use new object browser version

Change-Id: Ieb473b8f4b1beff81a5c2661f55e8e3f946daa2e
2021-05-19 16:01:15 +00:00
Jeff Wendling
a1bf9ab6de cmd/uplinkng: initial commit with skeleton
Change-Id: I764618cc60c46882955e9b08b72b3c162aa4929f
2021-05-18 10:04:44 -04:00
Egon Elbre
10372afbe4 ci: fix lint errors
Change-Id: Ib5893440807811f77175ccd347aa3f8ca9cccbdf
2021-05-17 13:37:31 +00:00
JT Olio
1852773e3e satellite/contact: rate limit node checkins
Change-Id: Ied386a2350aa073de46443e5259b56d49ec61dbf
2021-05-17 08:15:04 +00:00
Egon Elbre
8f15f975a2 satellite/overlay: improve contended update checkin
Improve UpdateCheckIn on a contended row:

  name                             old time/op  new time/op delta
  UpdateCheckInContended-100x-32   2.29s ±55%   0.17s ±61%  -92.45%  (p=0.008 n=5+5)

Change-Id: I053ab9f1cff136c306e5fb57f5e355cdc0269a8c
2021-05-16 20:41:12 +03:00
Yaroslav Vorobiov
b426248936 scripts/tests: fix missing multinode binary
Change-Id: I288d06c9eaa814ddca1e4aaacd86c5277aba86cc
2021-05-14 22:47:05 +00:00
Qweder93
f2812d76cd multinode/payouts: satellite period/allTime summaries added
payout summaries for specific satellite period/allTime added

Change-Id: I144138304f01f23d5c4b10931988eaaced656aaf
2021-05-14 18:49:31 +00:00
Cameron Ayer
dfe85beac2 satellite/satellitedb/dbx: fix duplicated lint line
Change-Id: Ia6677417ace208689eb6a814f2e93448fe57047c
2021-05-14 13:48:51 -04:00
Cameron Ayer
be87c80fe1 satellite/satellitedb: drop columns total_uptime_count and uptime_success_count
These columns on the nodes table are not referenced anywhere and are
ready to be dropped

Change-Id: I3f3652f635108419e93ced12b0ac1e58bba27a32
2021-05-14 11:49:12 -04:00
Cameron Ayer
3ea7aa2c7a satellite/repair/repairer: log piece hash verification failures
Piece hash verification failures during repair download are considered
audit failures, but we are not logging these occurrences. Now we log
them.

Change-Id: If456cebcfda6af7a659be3d1fc74448e681fb653
2021-05-14 15:03:15 +00:00
NickolaiYurchenko
e0f3166343 web/multinode: store and payouts components tests added
WHAT: unit tests for vue components and vuex store and environment for it

WHY: to cover frontend part with tests

Change-Id: I2aeaadb200176d4ba0a1406068304785e95d92cd
2021-05-14 12:45:10 +00:00
NickolaiYurchenko
2ccbb32e14 web/multinode: payouts page markup
WHAT: total payouts table, period calendar, details area and payout history area markup
also this component fumctionally connected to api and vuex store

Change-Id: Id3abc87bc7545aa3fc0a7ef2e480a8ed73974b76
2021-05-14 12:06:16 +00:00
igor gaidaienko
6ee2210297 satellite/console: add test for time based permission
Add test with NotBefore and NotAfter restricted permission to verify that we don't have an access to bucket

Change-Id: I7ec98a5b02c0098ee7ec81034278398f4435f1cf
2021-05-14 11:39:12 +00:00
NickolaiYurchenko
ce075a1d53 web/multinode: payout api
WHAT: multinode payout types, api and store

WHY: preparation of logic for mnd payouts page implementation

Change-Id: I4f2ea78056eab84c482853ef7a6c4cab4fb4c04f
2021-05-14 14:00:48 +03:00
Yaroslav Vorobiov
c08ca361d8 cmd/storj-sim: add multinode process
Change-Id: I7e6c8db045daa87c5d89f49e58fc82453d4024c6
2021-05-14 01:40:13 +03:00
Yaroslav Vorobiov
ea7fbdf843 multinode/multinodedb: add db migration
Change-Id: Ied1a7f3b951a25ab7a8edc25902c0479d2d08e87
2021-05-13 15:44:08 +00:00
Egon Elbre
910eec8eee satellite/metainfo: remove MetabaseDB interface
Currently the interface is not useful. When we need to vary the
implementation for testing purposes we can introduce a local interface
for the service/chore that needs it, rather than using the large api.

Unfortunately, this requires adding a cleanup callback for tests, there
might be a better solution to this problem.

Change-Id: I079fe4dbe297b0ae08c10081a1cea4dfbc277682
2021-05-13 13:22:14 +00:00
igor gaidaienko
d32ae0459b Revert "storagenode: docker image autoupdate binaries"
This reverts commit 244c488b10.
2021-05-13 14:21:06 +03:00
Yingrong Zhao
59f443e71a storagenode/contact: add authentication for PingNode endpoint
Currently, if a node has untrusted a satellite, the satellite can still
successfully ping the node. If a node decide to untrust a satellite, the
satellite should also mark it as conact failed

Change-Id: Idf80fa00d9849205533dd3e5b3b775b5b9686705
2021-05-12 18:15:32 +00:00
Qweder93
19561698ba multinode/payouts: all satellites summaries added.
payout summaries for specific/all periods added.

Change-Id: I92087edec548c0418a0f543d643e59f5c7df9621
2021-05-12 18:43:47 +03:00
Michał Niewrzał
547a6e9930 satellite/metabase: add DeletePart method
Method will delete all segments for given part.

Change-Id: I11bdfdf30176749aee03cea0d1d2fa467f7cebea
2021-05-11 22:45:43 +02:00
Malcolm Bouzi
9ccebfa7fd web/satellite: create select input component to capture employee count in registration
The current input dropdown solution allowed users to input their own text for employee count.
We want them to be limited to 3 choices, with a default value if none are chosen

Change-Id: I0fb2ab58fc756967efeac5f67e0f7001ef19c66f
2021-05-11 14:41:26 +00:00
Egon Elbre
2ae80690cb satellite/metabase/metabasetest: remove STORJ_TEST_DATABASES
Initially metabase was developed separately and it was useful to have a
separate environment flag for tests, however, it's more convenient to
use the same as rest of the testsuite.

Change-Id: Ia4d79be27ce5911cbae68d57cdf0b30f63459444
2021-05-11 13:31:01 +00:00
Qweder93
a11698f370 multinode/payouts: estimated payouts added
estimated payouts for specific/all satellites added.

Change-Id: I2530c9f1775593588e2a8f6c087ce6b4f9e354c4
2021-05-11 11:33:32 +00:00
Egon Elbre
0858c3797a satellite/{metabase,satellitedb}: deduplicate AS OF SYSTEM TIME code
Currently we were duplicating code for AS OF SYSTEM TIME in several
places. This replaces the code with using a method on
dbutil.Implementation.

As a consequence it's more useful to use a shorter name for
implementation - 'impl' should be sufficiently clear in the context.

Similarly, using AsOfSystemInterval and AsOfSystemTime to distinguish
between the two modes is useful and slightly shorter without causing
confusion.

Change-Id: Idefe55528efa758b6176591017b6572a8d443e3d
2021-05-11 12:40:36 +03:00
Michał Niewrzał
033006403f satellite/metainfo: fix setting object ZombieDeletionDeadline
So far we were setting ZombieDeletionDeadline alwasy as nil and because of that DB default was never set. This change adds separate query for inserting object if deadline is not set.

Change-Id: I3d6a16570e7c74b5304e13edad8c7adcd021340c
2021-05-11 07:26:42 +00:00
Ivan Fraixedes
7fb86617fc satellite/satellitedb: Use CRDB AS OF SYSTEM & batch for GE
Use the 'AS OF SYSTEM TIME' Cockroach DB clause for the Graceful Exit
(a.k.a GE) queries that count the delete the GE queue items of nodes
which have already exited the network.

Split the subquery used for deleting all the transfer queue items of
nodes which has exited when CRDB is used and batch the queries because
CRDB struggles when executing in a single query unlike Postgres.

The new test which has been added to this commit to verify the CRDB
batch logic for deleting all the transfer queue items of the exited
nodes has raised that the Enqueue method has to run in baches when CRDB
is used otherwise CRDB has return the error "driver: bad connection"
when a big a amount of items are passed to be enqueued. This error
didn't happen with the current test implementation it was with an
initial one that it was creating a big amount of exited nodes and
transfer queue items for those nodes.

Change-Id: I6a099cdbc515a240596bc93141fea3182c2e50a9
2021-05-07 13:09:19 -04:00
Egon Elbre
6161436d8b satellite/metabase/metabasetest: sort results
Deletion results are not guaranteed to be sorted, hence we need to sort
them before comparing.

Change-Id: I6b571dd812d4256327c6c64fff74c780c62b2948
2021-05-07 12:36:02 +03:00
Egon Elbre
6e6051b172 satellite/metabase: drop alias migration code
We have migrated all of the satellites and we shouldn't keep dead-code
around.

Change-Id: I539d6766cfafa2f278ff7767ceb2d39f6777ace3
2021-05-07 12:13:28 +03:00
Egon Elbre
2af7e4ef26 satellite/metabase/metaloop: use database time
The system and database time may drift. We should use database time for
absolute "as of system time" to ensure that it's not newer than the
current database time. When the "as of system time" is in the future,
then the query will fail.

Change-Id: I5423f6aaad966ca03a76b5ff805bfba932e44a51
2021-05-07 09:08:04 +00:00
Michał Niewrzał
60ff87a7d7 design doc: server-side move
Change-Id: Ic6ae180c7a2288bbf463adcf4ca371b9a9222ae1
2021-05-07 07:48:48 +00:00
Egon Elbre
205ea621cc .github: remove noisy dependabot
It seems to bump on any minor release bump and consuming Jenkins
resources. This way of reviewing dependency updates is rather slow.
A more effective way is to review all things that can be updated
at the same time.

Change-Id: I0a66fd04ab20cae40b049bff9c373ca908a49de0
2021-05-07 06:52:53 +00:00
Ivan Fraixedes
38c41d0943 satellite/admin: Fix typo in README
Fix a typo in the README of the Satellite Admin API.

Change-Id: I682a8e2310082b5fc14e69347f0796cc0b0fc0f8
2021-05-06 19:18:12 +00:00
JT Olio
a89a2b4b43 satellite/billing: make stripe invoice generation work with multiregion satellites
multiregion satellites have complex database connection strings
largely due to using a different backend for the repair queue than
cockroach.

billing stuff didn't work right with this.

Change-Id: Ie8759a8c47e71347c3a190abfc9d53945d7b8855
2021-05-06 11:51:03 -06:00
Egon Elbre
69b149a66f mod: bump uplink
uplink stopped using zap, hence some of the private methods needed to be
changed.

Change-Id: Iac1fae45a40cd3f1649b9f672bf8c250344986d5
2021-05-06 14:48:36 +00:00
Ivan Fraixedes
2f8969ae35
satellite/admin: Add a ToC to README
Add an auto-generated table of content to the README for easy to find
and browser the documentation of the available API end points.

Change-Id: Id94d904cefd30449234224072ddc50a181aaba04
2021-05-06 07:25:15 +02:00
Ivan Fraixedes
1504303363
satellite/admin: Add clarifications in README
Add some clarifications to the README of the satellite admin API.

Change-Id: I2f56c8778872a098442ecd601c2b8dcfe1ac19a2
2021-05-06 06:49:11 +02:00