Commit Graph

5549 Commits

Author SHA1 Message Date
Fadila Khadar
5af898fcc9 satellite/metabase: add migration step for index on pending objects
Add an index on (project_id, bucket_name) for objects with pending status

Change-Id: Ibd3c949a99e49e8e6bb884564331ec16f9184942
2021-04-28 09:54:10 +00:00
Qweder93
be7928d4bd crashcollect: removed redundant structure
Change-Id: Ied65a057baf4cbbe8a47685b0c52f3900363d4bd
2021-04-28 00:35:39 +03:00
Vitalii Shpital
917e4b81d3 web/satellite: regular header reworked
WHAT:
removed unnecessary templates splitting
fixed resources dropdown margins
fixes close navigation button responsiveness
aligned logo
added loader for projects dropdown and removed redundant request on dropdown closing
added satellite name to header's right side

WHY:
bug fixes/better user experience

Change-Id: I7ed30f882a1f5484efbf3e0e21ee5d637ef83bf2
2021-04-27 16:40:48 +00:00
Qweder93
a5c1e4b4a5 crashcollect: process for receiving panics
Added crashcollect server for tracking panics
on parent process and saving them in file.

Change-Id: I7926f9a16594227a3262e05d216199b7c2857385
2021-04-27 18:57:47 +03:00
Michał Niewrzał
02460fcc4c satellite/metainfo/expireddeletion: change chore interval to 24h
We want run deletion query more often to avoid long running DB operations.

Change-Id: I72c63bb8ec05cc3ab2e7ab63479ba6cdf830386a
2021-04-27 12:30:23 +00:00
Vitalii Shpital
606a501441 satellite/console: add tardigradeshare.io domain to media-src CSP whitelist
WHAT:
added tardigradeshare.io domain to media-src CSP whitelist

WHY:
we still support it

Change-Id: Id02daa0712cc84f5cb7edaafb21a143859531f26
2021-04-27 11:17:50 +00:00
Vitalii Shpital
f2e06ce882 web/satellite: added skip button for the onboarding flow
WHAT:
added skip button to onboardling flow

WHY:
better user experience

Change-Id: I80953de701e13730019efac38595116c9ce94689
2021-04-27 10:54:29 +00:00
Vitalii Shpital
64c82507c5 web/satellite: make project dashboard delay message more visible
WHAT:
replace info icon with a regular message

WHY:
better user experience

Change-Id: I7edaf1d139f824752eb3c96bcfbba7e5ef2aa09f
2021-04-27 10:54:25 +00:00
Vitalii Shpital
a84b8367dc web/satellite: changed Token to API Key label for CLI step of AG flow
WHAT:
label changed from Token to API Key

WHY:
to remove user's confusion

Change-Id: If30384bb3cf6b32ae7fede0a53af176c73965090
2021-04-27 10:35:01 +00:00
Malcolm Bouzi
c1fdf1ad71 satellite/satellitedb: add have_sales_contact column on users
For business accounts we need to track the sales contact.
It will be a question to business accounts during onboarding.

Change-Id: I8d101ce1b52091478dfb0ddd875e1cc717d765d3
2021-04-26 13:37:26 +00:00
Michał Niewrzał
7944df20d6 storj: use multipart API
Change-Id: I10b401434e3e77468d12ecd225b41689568fd197
2021-04-26 13:15:09 +00:00
Yaroslav Vorobiov
aa27a8f1dd private/crashreportpb: add protobuf for crashreport and crashcollect
Change-Id: I17da796030f63471ae0900142413523e8e12895d
2021-04-26 12:46:03 +00:00
Egon Elbre
948529a600 satellite/metabase/metabasetest: move test utils to a separate package
Change-Id: Iaa607996478497d5c97c6a6973d099b315b3abf0
2021-04-26 13:12:58 +03:00
Yingrong Zhao
307886ffe8 storagenode/piecestore: fix error handling in TestDownload
If a read error is returned, we want to preserve it instead of
overriding it with close error

Change-Id: Ie253a453c7b6b62598b89dffd7635b8266357074
2021-04-23 18:43:45 +00:00
Malcolm Bouzi
31c1137860 web/satellite: add login path
path will be used for a new link

Change-Id: Idb14ea5bd152cf169575b7223fc429eb44d7049e
2021-04-23 15:01:27 +00:00
Yingrong Zhao
08e1ce8930 cmd/connecttest: display more helpful error message and fix tls config
Change-Id: Ibedd643458ee8c84fd4668a23449d99eec43ba73
2021-04-23 14:27:34 +00:00
Egon Elbre
7802ab714f pkg/,private/: merge with private package
Initially there were pkg and private packages, however for all practical
purposes there's no significant difference between them. It's clearer to
have a single private package - and when we do get a specific
abstraction that needs to be reused, we can move it to storj.io/common
or storj.io/private.

Change-Id: Ibc2036e67f312f5d63cb4a97f5a92e38ae413aa5
2021-04-23 16:37:28 +03:00
Egon Elbre
c641ddcb54 pkg/lrucache: rename package
cache is really common variable and type name and we have already used
the package name alias in multiple places.

Change-Id: I6435785b7549b541d533de59ec94557b9bd11e04
2021-04-23 16:37:28 +03:00
Egon Elbre
4d37378129 storage/{cockroachkv,postgreskv}: delete unused code
Change-Id: I11da924508d207d8816bda206d28261931d9cd4e
2021-04-23 16:37:28 +03:00
Egon Elbre
8c62788b24 satellite/metabase/metaloop: stop timer properly
There's a rare chance that `Stop` returns false, however doesn't have
time triggered. Use a non-blocking drain to remove the token.

Change-Id: I1ae18a197424017f0ca76656602709a029b56bfd
2021-04-23 16:35:21 +03:00
Vitalii Shpital
d32515fa5d satellite/console: add storjshare.io domain to media-src CSP whitelist
WHAT:
whitelist .storjshare.io domain for media-src CSP

WHY:
to enable video preview for linksharing

Change-Id: Ib673602d31ca116e7ce1cee0eba17099a55d7dbc
2021-04-23 16:11:41 +03:00
Egon Elbre
a2e20c93ae private/dbutil: use dbutil and tagsql from storj.io/private
Initially we duplicated the code to avoid large scale changes to
the packages. Now we are past metainfo refactor we can remove the
duplication.

Change-Id: I9d0b2756cc6e2a2f4d576afa408a15273a7e1cef
2021-04-23 14:36:52 +03:00
Egon Elbre
dd5eb2616b satellite/metrics: join for monitoring
We don't want metrics to trigger the metaloop on it's own.

Change-Id: I648272a0e3043173b0a377697db1f172712bdda5
2021-04-23 12:21:48 +03:00
Egon Elbre
c1fbecb96b satellite/metabase/metaloop: add Monitor
We need some chores to join without triggering the loop.

For example it's fine to run metrics, only when something else is
running.

Change-Id: I9d8bd16f59c28c540c8d72971bc4e233a8660c02
2021-04-23 12:20:22 +03:00
Egon Elbre
fff21b330d cmd/metabase-verify: tool for verifying metabase state
Currently the tool verifies:
* validity of plain_offset
* whether plain_size is smaller than encrypted_size

Change-Id: I9ec4fb5ead3356a196392c26ca377fcdb367138e
2021-04-23 07:50:36 +00:00
Egon Elbre
6307875203 monkit: fix monkit lock
Previously check-monitoring ignored ScopeNamed.

Change-Id: I3d0d472e722cb19a65b471a914f5124cd224fc34
2021-04-23 06:36:40 +00:00
paul cannon
dcb73e46ca cmd/connect-test: add connect-test tool
This tool attempts to make TCP+TLS+DRPC and QUIC+DRPC connections to a
specified IP:port. If successful with either, it shows the node ID of
the opposite end of the connection.

This should be useful for SNOs who want to check that their packet
forwarding is working or that their node is listening for both
connection types.

Change-Id: I935dff037dd7f1106941a35567f6445230259d1a
2021-04-22 16:06:45 +00:00
Cameron Ayer
a0c5da6643 satellite/satellitedb: in stray nodes DQ, don't DQ nodes where last_contact_success = '0001-01-01 00:00:00+00'
When nodes check in for the very first time, if the satellite can't ping
them back, they are inserted into the nodes table with
last_contact_success of '0001-01-01 00:00:00+00'. If the stray nodes
chore runs before the node can fix their problem, they are DQd.

Solution: when DQing stray nodes, dont DQ where last_contact_success =
'0001-01-01 00:00:00+00'::timestamptz

Change-Id: I477a02d5ef85b2c930ed6b7d99a4d1995169bca8
2021-04-22 10:13:13 -04:00
Egon Elbre
4c9ed64f75 satellite/metabase/metaloop: move loop under metabase
Currently the loop handling is heavily related to the metabase rather
than metainfo.

metainfo over time has become related to the "public API" for accessing
the metabase data.

Currently updates monkit.lock, because monkit monitoring does not handle
ScopeNamed correctly. Needs a followup change to monitoring check.

Change-Id: Ie50519991d718dfb872ec9a0176a82e732c97584
2021-04-22 12:58:09 +03:00
Egon Elbre
ebc981291a satellite/accounting: use correct error in tests
rpcstatus codes are not part of the public uplink API
and the tests should not use them.

Change-Id: I9c96c557bf2dac5d6fff200fb217378524fd14ff
2021-04-22 07:25:42 +00:00
Malcolm Bouzi
52c3d7d642 web/satellite: add disabled state to onboarding overview RClone link
Change-Id: I4cef2d0772967f47e9fa063494de9f8903eb4f29
2021-04-21 19:34:55 +00:00
Yingrong Zhao
43f5888052 satellite/contact: only test PingBack failure case
The original test caused the testplanet to timeout due to contact
service stuck in a loop. We can change it to only test the failure case for
PingBack method instead of closing the TCP port on storagenodes.

Change-Id: Ic96aee637b39ae95050c6902c2bf9ca51fb586c3
2021-04-21 14:20:36 -04:00
Yaroslav Vorobiov
244c488b10 storagenode: docker image autoupdate binaries
Get storagenode and storagenode-updater binaries during
run of the container to not to release new docker image
on each new version of the storagenode binary.

Change-Id: Ic0eb4a9c18a98598dfd9b96c1d352c7399496fd2
2021-04-21 17:56:20 +00:00
Ethan
a9c9f080bb satellite/metaloop: Use AS OF SYSTEM TIME when querying the objects table
(*loopIterator).doNextQuery is not using AS OF SYSTEM TIME because the value is 0

Change-Id: I3c671c1b2aa32bd6379b5cdea6e16246b4def9fd
2021-04-21 17:18:52 +00:00
Ivan Fraixedes
2537bbf543
satellite/gracefulexit: Try avoiding randomly test failure
The test function fails randomly in the CI when runs with CRDB. There
isn't currently an explanation why the expectation of number of nodes
which exited 4 minutes ago reports 4 nodes rather than 5 and the only
clue that we have now to see if it gets remedied is to give 2 minutes
rather than 1 to the node that exited close to the time passed function
which makes the test to randomly fail.

Change-Id: I3a731e3eb7f19caebdf29713150727f2cf3e0e0a
2021-04-21 17:40:07 +02:00
Yingrong Zhao
8b2e18c910 pkg/quic: ignore successful session closed error
Change-Id: I205bccaae02df818de7d8f2e713ec295ef754088
2021-04-21 15:05:13 +00:00
Egon Elbre
267506bb20 satellite/metabase: move package one level higher
metabase has become a central concept and it's more suitable for it to
be directly nested under satellite rather than being part of metainfo.

metainfo is going to be the "endpoint" logic for handling requests.

Change-Id: I53770d6761ac1e9a1283b5aa68f471b21e784198
2021-04-21 15:54:22 +03:00
Kaloyan Raev
2ee3030275 all: remove code related to PointerDB
Change-Id: I6675c9597f87019020f6233b83ab2f1119d2bc46
2021-04-21 12:35:31 +00:00
Kaloyan Raev
785adfb849 go.mod: bump to uplink v1.4.6
Change-Id: I855d8a8860fa126756930fe09073ddb259bc99ac
2021-04-21 12:30:50 +03:00
Yingrong Zhao
a3c437a7bf satellite/contact,storagenode/contact: try ping back to nodes through
QUIC

We want to encourage storagenodes to open their udp port. This PR
changes contact service in satellite to try to connect to nodes through
QUIC. If satellite can't reach nodes through quic, it will send an error
message back to nodes. On the nodes side, it will always log out error
message from check in if the error message is not empty.
Whether satellite can reach nodes through quic has no affect on nodes'
uptime check.

Change-Id: I5ebf80f921c4a6504997d83c8bd45226da9d3703
2021-04-20 19:25:37 +00:00
Vitalii Shpital
f6ec7f9bfc web/satellite: remove paywall steps from onboarding flow
WHAT:
removed paywall related steps from onboarding flow
fixed back button bugs

WHY:
bug fixing

Change-Id: I150127fe9c2b3b134d0cb7ee9581d8fdd415a591
2021-04-20 17:38:50 +00:00
Vitalii Shpital
f8aaf5c5f3 web/satellite: bump object browser version with part size fix
WHAT:
this version includes a fix for part size to be 64MB

WHY:
post-launch fix

Change-Id: Ie50adddef0340e679604995bab5aaa8317496728
2021-04-20 16:26:34 +00:00
littleskunk
a6b3a565df jenkins/build: fix file extension for msi package 2021-04-20 17:27:44 +03:00
Vitalii Shpital
546e2d2e81 web/satellite: remove details button on info bar for Dashboard page
WHAT:
hide details button if user is already on project dashboard page

WHY:
confusing user

Change-Id: Ib363e0e1a040ca529de7ec8f1b82ce54943890c9
2021-04-20 09:42:45 +00:00
JT Olio
ebe5813964 satellite/emails: update mailing address
Change-Id: Iadcb7908a63b18ffb59e6b81632c1124dd6ad498
2021-04-19 17:40:09 +00:00
Ivan Fraixedes
5642a7a8dc docker-compose: Delete deprecated file
Delete the docker-compose file which was sitting in the root of the
repository because we don't use it anymore.
Remove the Makefile targets which were call docker-compose and
referencing the file.

Change-Id: I6060fc7f06cd4b612ecb5d13ff80d67cfbc4e438
2021-04-19 17:12:59 +00:00
prerna-parashar
613a95530b
satellite/analytics: Add analytics for "path selected" in onboarding step (#4086) 2021-04-19 09:44:25 -07:00
Egon Elbre
75ca01e381 cmd/uplink: fix importing access in Windows
Currently os.Create was leaving a file open causing atomic write file to
fail with access denied.

Also add a specific test for importing an access.

Change-Id: Id188bc480e795849ec7fdc72b1fc86433d76c47a
2021-04-19 15:11:39 +03:00
Egon Elbre
6a805b2891 satellite/metainfo/metabase: use pgx.Batch to delete expired segments
Change-Id: I51eeaadeeef0ca2faabac4b38158278aea177eb8
2021-04-19 11:19:29 +03:00
Egon Elbre
b2be1f1629 satellite/metainfo/metabase: fix delete expiration
The cursor was not being used in the batch deletion.

The stream ID was not being used while deleting, which could in rare
circumstaces delete a newly uploaded object.

Use the stream id in deletion, rather than passing that information from
one query to another.

Change-Id: I03271c6e72747e345dfb0bb70989f29e835efd8e
2021-04-19 11:19:29 +03:00