Commit Graph

5383 Commits

Author SHA1 Message Date
Egon Elbre
7802ab714f pkg/,private/: merge with private package
Initially there were pkg and private packages, however for all practical
purposes there's no significant difference between them. It's clearer to
have a single private package - and when we do get a specific
abstraction that needs to be reused, we can move it to storj.io/common
or storj.io/private.

Change-Id: Ibc2036e67f312f5d63cb4a97f5a92e38ae413aa5
2021-04-23 16:37:28 +03:00
Egon Elbre
c641ddcb54 pkg/lrucache: rename package
cache is really common variable and type name and we have already used
the package name alias in multiple places.

Change-Id: I6435785b7549b541d533de59ec94557b9bd11e04
2021-04-23 16:37:28 +03:00
Egon Elbre
4d37378129 storage/{cockroachkv,postgreskv}: delete unused code
Change-Id: I11da924508d207d8816bda206d28261931d9cd4e
2021-04-23 16:37:28 +03:00
Egon Elbre
8c62788b24 satellite/metabase/metaloop: stop timer properly
There's a rare chance that `Stop` returns false, however doesn't have
time triggered. Use a non-blocking drain to remove the token.

Change-Id: I1ae18a197424017f0ca76656602709a029b56bfd
2021-04-23 16:35:21 +03:00
Vitalii Shpital
d32515fa5d satellite/console: add storjshare.io domain to media-src CSP whitelist
WHAT:
whitelist .storjshare.io domain for media-src CSP

WHY:
to enable video preview for linksharing

Change-Id: Ib673602d31ca116e7ce1cee0eba17099a55d7dbc
2021-04-23 16:11:41 +03:00
Egon Elbre
a2e20c93ae private/dbutil: use dbutil and tagsql from storj.io/private
Initially we duplicated the code to avoid large scale changes to
the packages. Now we are past metainfo refactor we can remove the
duplication.

Change-Id: I9d0b2756cc6e2a2f4d576afa408a15273a7e1cef
2021-04-23 14:36:52 +03:00
Egon Elbre
dd5eb2616b satellite/metrics: join for monitoring
We don't want metrics to trigger the metaloop on it's own.

Change-Id: I648272a0e3043173b0a377697db1f172712bdda5
2021-04-23 12:21:48 +03:00
Egon Elbre
c1fbecb96b satellite/metabase/metaloop: add Monitor
We need some chores to join without triggering the loop.

For example it's fine to run metrics, only when something else is
running.

Change-Id: I9d8bd16f59c28c540c8d72971bc4e233a8660c02
2021-04-23 12:20:22 +03:00
Egon Elbre
fff21b330d cmd/metabase-verify: tool for verifying metabase state
Currently the tool verifies:
* validity of plain_offset
* whether plain_size is smaller than encrypted_size

Change-Id: I9ec4fb5ead3356a196392c26ca377fcdb367138e
2021-04-23 07:50:36 +00:00
Egon Elbre
6307875203 monkit: fix monkit lock
Previously check-monitoring ignored ScopeNamed.

Change-Id: I3d0d472e722cb19a65b471a914f5124cd224fc34
2021-04-23 06:36:40 +00:00
paul cannon
dcb73e46ca cmd/connect-test: add connect-test tool
This tool attempts to make TCP+TLS+DRPC and QUIC+DRPC connections to a
specified IP:port. If successful with either, it shows the node ID of
the opposite end of the connection.

This should be useful for SNOs who want to check that their packet
forwarding is working or that their node is listening for both
connection types.

Change-Id: I935dff037dd7f1106941a35567f6445230259d1a
2021-04-22 16:06:45 +00:00
Cameron Ayer
a0c5da6643 satellite/satellitedb: in stray nodes DQ, don't DQ nodes where last_contact_success = '0001-01-01 00:00:00+00'
When nodes check in for the very first time, if the satellite can't ping
them back, they are inserted into the nodes table with
last_contact_success of '0001-01-01 00:00:00+00'. If the stray nodes
chore runs before the node can fix their problem, they are DQd.

Solution: when DQing stray nodes, dont DQ where last_contact_success =
'0001-01-01 00:00:00+00'::timestamptz

Change-Id: I477a02d5ef85b2c930ed6b7d99a4d1995169bca8
2021-04-22 10:13:13 -04:00
Egon Elbre
4c9ed64f75 satellite/metabase/metaloop: move loop under metabase
Currently the loop handling is heavily related to the metabase rather
than metainfo.

metainfo over time has become related to the "public API" for accessing
the metabase data.

Currently updates monkit.lock, because monkit monitoring does not handle
ScopeNamed correctly. Needs a followup change to monitoring check.

Change-Id: Ie50519991d718dfb872ec9a0176a82e732c97584
2021-04-22 12:58:09 +03:00
Egon Elbre
ebc981291a satellite/accounting: use correct error in tests
rpcstatus codes are not part of the public uplink API
and the tests should not use them.

Change-Id: I9c96c557bf2dac5d6fff200fb217378524fd14ff
2021-04-22 07:25:42 +00:00
Malcolm Bouzi
52c3d7d642 web/satellite: add disabled state to onboarding overview RClone link
Change-Id: I4cef2d0772967f47e9fa063494de9f8903eb4f29
2021-04-21 19:34:55 +00:00
Yingrong Zhao
43f5888052 satellite/contact: only test PingBack failure case
The original test caused the testplanet to timeout due to contact
service stuck in a loop. We can change it to only test the failure case for
PingBack method instead of closing the TCP port on storagenodes.

Change-Id: Ic96aee637b39ae95050c6902c2bf9ca51fb586c3
2021-04-21 14:20:36 -04:00
Yaroslav Vorobiov
244c488b10 storagenode: docker image autoupdate binaries
Get storagenode and storagenode-updater binaries during
run of the container to not to release new docker image
on each new version of the storagenode binary.

Change-Id: Ic0eb4a9c18a98598dfd9b96c1d352c7399496fd2
2021-04-21 17:56:20 +00:00
Ethan
a9c9f080bb satellite/metaloop: Use AS OF SYSTEM TIME when querying the objects table
(*loopIterator).doNextQuery is not using AS OF SYSTEM TIME because the value is 0

Change-Id: I3c671c1b2aa32bd6379b5cdea6e16246b4def9fd
2021-04-21 17:18:52 +00:00
Ivan Fraixedes
2537bbf543
satellite/gracefulexit: Try avoiding randomly test failure
The test function fails randomly in the CI when runs with CRDB. There
isn't currently an explanation why the expectation of number of nodes
which exited 4 minutes ago reports 4 nodes rather than 5 and the only
clue that we have now to see if it gets remedied is to give 2 minutes
rather than 1 to the node that exited close to the time passed function
which makes the test to randomly fail.

Change-Id: I3a731e3eb7f19caebdf29713150727f2cf3e0e0a
2021-04-21 17:40:07 +02:00
Yingrong Zhao
8b2e18c910 pkg/quic: ignore successful session closed error
Change-Id: I205bccaae02df818de7d8f2e713ec295ef754088
2021-04-21 15:05:13 +00:00
Egon Elbre
267506bb20 satellite/metabase: move package one level higher
metabase has become a central concept and it's more suitable for it to
be directly nested under satellite rather than being part of metainfo.

metainfo is going to be the "endpoint" logic for handling requests.

Change-Id: I53770d6761ac1e9a1283b5aa68f471b21e784198
2021-04-21 15:54:22 +03:00
Kaloyan Raev
2ee3030275 all: remove code related to PointerDB
Change-Id: I6675c9597f87019020f6233b83ab2f1119d2bc46
2021-04-21 12:35:31 +00:00
Kaloyan Raev
785adfb849 go.mod: bump to uplink v1.4.6
Change-Id: I855d8a8860fa126756930fe09073ddb259bc99ac
2021-04-21 12:30:50 +03:00
Yingrong Zhao
a3c437a7bf satellite/contact,storagenode/contact: try ping back to nodes through
QUIC

We want to encourage storagenodes to open their udp port. This PR
changes contact service in satellite to try to connect to nodes through
QUIC. If satellite can't reach nodes through quic, it will send an error
message back to nodes. On the nodes side, it will always log out error
message from check in if the error message is not empty.
Whether satellite can reach nodes through quic has no affect on nodes'
uptime check.

Change-Id: I5ebf80f921c4a6504997d83c8bd45226da9d3703
2021-04-20 19:25:37 +00:00
Vitalii Shpital
f6ec7f9bfc web/satellite: remove paywall steps from onboarding flow
WHAT:
removed paywall related steps from onboarding flow
fixed back button bugs

WHY:
bug fixing

Change-Id: I150127fe9c2b3b134d0cb7ee9581d8fdd415a591
2021-04-20 17:38:50 +00:00
Vitalii Shpital
f8aaf5c5f3 web/satellite: bump object browser version with part size fix
WHAT:
this version includes a fix for part size to be 64MB

WHY:
post-launch fix

Change-Id: Ie50adddef0340e679604995bab5aaa8317496728
2021-04-20 16:26:34 +00:00
littleskunk
a6b3a565df jenkins/build: fix file extension for msi package 2021-04-20 17:27:44 +03:00
Vitalii Shpital
546e2d2e81 web/satellite: remove details button on info bar for Dashboard page
WHAT:
hide details button if user is already on project dashboard page

WHY:
confusing user

Change-Id: Ib363e0e1a040ca529de7ec8f1b82ce54943890c9
2021-04-20 09:42:45 +00:00
JT Olio
ebe5813964 satellite/emails: update mailing address
Change-Id: Iadcb7908a63b18ffb59e6b81632c1124dd6ad498
2021-04-19 17:40:09 +00:00
Ivan Fraixedes
5642a7a8dc docker-compose: Delete deprecated file
Delete the docker-compose file which was sitting in the root of the
repository because we don't use it anymore.
Remove the Makefile targets which were call docker-compose and
referencing the file.

Change-Id: I6060fc7f06cd4b612ecb5d13ff80d67cfbc4e438
2021-04-19 17:12:59 +00:00
prerna-parashar
613a95530b
satellite/analytics: Add analytics for "path selected" in onboarding step (#4086) 2021-04-19 09:44:25 -07:00
Egon Elbre
75ca01e381 cmd/uplink: fix importing access in Windows
Currently os.Create was leaving a file open causing atomic write file to
fail with access denied.

Also add a specific test for importing an access.

Change-Id: Id188bc480e795849ec7fdc72b1fc86433d76c47a
2021-04-19 15:11:39 +03:00
Egon Elbre
6a805b2891 satellite/metainfo/metabase: use pgx.Batch to delete expired segments
Change-Id: I51eeaadeeef0ca2faabac4b38158278aea177eb8
2021-04-19 11:19:29 +03:00
Egon Elbre
b2be1f1629 satellite/metainfo/metabase: fix delete expiration
The cursor was not being used in the batch deletion.

The stream ID was not being used while deleting, which could in rare
circumstaces delete a newly uploaded object.

Use the stream id in deletion, rather than passing that information from
one query to another.

Change-Id: I03271c6e72747e345dfb0bb70989f29e835efd8e
2021-04-19 11:19:29 +03:00
Vitalii Shpital
9cd17fd804 web/satellite: request passphrasse on project change. Small styling fixes
WHAT:
rerequest passphrase when user selects another project
small styling fixes

WHY:
bakeoff

Change-Id: I59da49b3efff1c1a60fbbdc622ae497a1326b347
2021-04-16 17:47:28 +03:00
Vitalii Shpital
b57819f590 satellite: take pricing from the config instead of hardcoding
WHAT:
take pricing from config instead of hardcoding

WHY:
bakeoff

Change-Id: Id8209f0905a9105c1f5796165e279acf31563c65
2021-04-16 07:59:31 +00:00
Moby von Briesen
f9a6fbea50 web/satellite: Replace Tardigrade logos with Storj DCS
This is not exhaustive, but it covers a lot of them.

Change-Id: Ia68682b79cc426c525227dbf7cee017e537be974
2021-04-16 00:31:54 +00:00
Malcolm Bouzi
f380bda814 web/satellite: add paths for register and login pages to allow for travel between both
Change-Id: Ic3a891c790bad1a851609fd6203f271dbca45762
2021-04-15 18:03:08 -04:00
Malcolm Bouzi
dfe4d09a89 web/satellite: Update login logo to DSC branding
Change-Id: I32ae076c4343c57c23478a8e80104870156c9d33
2021-04-15 20:53:44 +00:00
JT Olio
35d8a840d3 storj/storj: more domain changes
Change-Id: I643c38bdae6dc26c9346147d80a83ae9dde2eeae
2021-04-15 20:51:43 +00:00
Vitalii Shpital
d7488924cc web/satellite: bump file browser component version
WHAT:
bump file browser component version

WHY:
bakeoff

Change-Id: I857a130735feca438ca4bd44684e8408d47856bb
2021-04-15 23:22:22 +03:00
Vitalii Shpital
1215500c83 web/satellite: replace spaces with %20 in linksharing url
WHAT:
replace empty spaces with %20 in linksharing url

WHY:
bakeoff

Change-Id: I57e3cb6cde9986d32880942538fd1516790f8c8f
2021-04-15 19:46:01 +03:00
Vitalii Shpital
63c3dd2b2d web/satellite: add bucket name restriction info to popup
WHAT:
added info about restrictions for bucket name

WHY:
bakeoff

Change-Id: I4e931ed529c7ac36dad1131cffc1eb39b9ce528b
2021-04-15 15:42:47 +00:00
Vitalii Shpital
68f67e5356 web/satellite: trim passphrase for file browser flow
WHAT:
trim value on enter passphrase step
warning copy changed

WHY:
bakeoff

Change-Id: I907a885e0d323cb4c8e3804845a936278ac29234
2021-04-15 18:15:42 +03:00
Vitalii Shpital
29e55d69b0 web/satellite: overview page copy and routing changed
WHAT:
changed a copy of overview page to Upload in Browser
buttons' routes changed

WHY:
bakeoff

Change-Id: I5fb5ff41014b48db1492a24e01548d78bf7257a8
2021-04-15 18:10:14 +03:00
Vitalii Shpital
bd36a41a9e web/satellite/file_browser: open bucket on creation
WHAT:
open bucket after creation
back button for uploads page
fixed input's width
added client side bucket name validation

WHY:
bakeoff

Change-Id: I82b96d4180e4a80c01bf888adae5c08d0af15bec
2021-04-14 21:53:29 +00:00
JT Olio
3b09d6c308 storj/storj: update support request links
Change-Id: I7592df7fe4630508784b6cc92b751d3b567e911e
2021-04-14 20:52:45 +00:00
JT Olio
f3c8cac1ee storj/storj: more domain changes
Change-Id: I91ae3bc36be38c27269f9e6e678fa3616b8e4d53
2021-04-14 20:49:12 +00:00
Moby von Briesen
569942c04d web/satellite/src/api: Absorb analytics errors
On more privacy-sensitive browsers, attempts to notify the satellite
about client-side events might be blocked, because the API endpoint on
the satellite contains the word "analytics".

We want to respect the privacy of these types of visitors, so rather
than changing the name of the endpoint to something else, this change
catches errors that are caused by attempting to use the analytics API,
and logs them to the console without interrupting the user's experience
of the website. These errors do not affect any essential behavior, so it
is okay if the user is not aware of them.

Change-Id: I6f49e0abcd64fd69802e4efa71a8d307cc5a3aee
2021-04-14 20:27:41 +00:00
Moby von Briesen
031206e453 web/satellite/tests: Fix snapshots
Fixes snapshot errors caused by 034519a12 (whoops)

Change-Id: I094e3fae270c95ad2fcb500327ab867f3fb1c94b
2021-04-14 21:33:27 +02:00