Commit Graph

493 Commits

Author SHA1 Message Date
Stefan Benten
8c1a149de2 build: bump private dependency
Enables proper version printing.

Change-Id: I37e12d6823f86f9275ac9f64abf324003019bd18
2021-10-29 12:25:33 +13:00
Egon Elbre
1409288c49 mod: bump semantic version to go 1.17
This enables module graph pruning and lazy module loading.

Change-Id: I5d63a686d4f4eda93b23576169bb03847292f8c7
2021-10-26 11:58:13 +00:00
Stefan Benten
df6cf60ffe
go.mod: update to minimum supported go version (#4239) 2021-10-22 21:12:13 +02:00
Egon Elbre
56f35e9ea9 mod: bump private repo
Contains a fix to ignore cockroach DROP error while testing.

Change-Id: I74b02240f570c30a66f6f0b938ce036392c05e06
2021-10-22 15:02:48 +03:00
Clement Sam
f82b696858 mod: bump common dependency
Change-Id: I24ba0ef8d0ca25a34e28de1064a2f1213787ccc5
2021-10-21 16:33:48 +00:00
Egon Elbre
bdadc4b460 private/testplanet: add stack traces related to the planet
It's useful to see only the goroutines that are related to the current
testplanet.

Change-Id: Id77c62dec42c09d44c523b019f7bdc940905fcf1
2021-10-20 23:12:07 +03:00
Michał Niewrzał
b8e8110ca3 mod: bump common and uplink dependency
Contains also adjustments for latest storj/common changes.

Change-Id: I11f70d466b3d9785ee93ac424629b2b868889d02
2021-10-12 19:13:49 +00:00
Michał Niewrzał
1fdb0eaa5b Revert "satellite/metabase: use storj.Nonce instead []byte"
This change introduce problems with server side move so
let's revert it for now. Problem was found when latest
version of storj/storj was used in uplink tests.

This reverts commit 1ef06fae99.

Change-Id: I4d4fad5d1ea04ba15ff9d7bd765f7e078e9187c2
2021-10-12 15:39:54 +02:00
Michał Niewrzał
1ef06fae99 satellite/metabase: use storj.Nonce instead []byte
We were using mixed types for nonce fields. Protobuf
have storj.Nonce, metabase have []byte. This change
is a refactoring to have everywere its possible only
storj.Nonce.

Change-Id: Id54bd8481f30c721cdaf3df79206d25e7cfdab55
2021-10-11 16:13:34 +00:00
Michał Niewrzał
f837551d76 storj: bump uplink to v1.7.0
Change-Id: Ic5ffc704ffbcf7dc890fe0554a8e23f1a777ecbd
2021-10-08 08:04:57 +00:00
Egon Elbre
6511bb91fb private/testplanet: support writing monitoring spans
This allows to set `STORJ_TEST_MONKIT` to either
`svg` or `json` to write individual testplanet test
traces to disk.

It also allows to specify an absolute directory:

  STORJ_TEST_MONKIT=svg,dir:/abs/dir/path

It requires an absolute path, because from the context of
tests, there's no easy way to find the folder where tests
were called.

Change-Id: I6fe008a4d4237d221cf5a5bede798b46399ee197
2021-09-30 09:03:35 +03:00
Michał Niewrzał
0209bc6ef7 cmd/uplink: add mv command
New command for cli to move object to different
location.

uplink mv sj://bucket/your-object sj://bucket/moved-object

Change-Id: I85a4961aa59f250819954e78f20363ac3c570938
2021-09-29 19:10:58 +00:00
Egon Elbre
6d3fd33ca3 satellite/metabase/segmentloop: start immediately on manual trigger
Currently loops wait for the coalesce duration for TriggerWait.
Let's skip the coalesce when we trigger it manually.

Change-Id: If5bacd4e263d233f1f3ea41b989922d2ed5a48d4
2021-09-28 18:32:03 +00:00
Egon Elbre
6e660cecdd Jenkinsfile: test cross-compile and bump deps
Change-Id: I47091de2e80bd96fcada616f75e1db07a59fb0c1
2021-09-16 18:59:31 +03:00
Yingrong Zhao
f0b73b4f45 mod: bump drpc, uplink and common
Updates:
- drpc to v0.0.25
- uplink to latest
- common to latest

This will fix the crash on UDP network temporary error for windows and a
rare deadlock during shutdown

Change-Id: I7e9c4ace22665c5471f19f9b8bf63aec4d68ceb3
2021-09-16 07:42:32 +00:00
crawter
07311cdd6f satellite/metainfo: begin move object added
First from two methods needed to perform server-side move. It gets
metadata key and nonce and all segments key and nonces and returns
all of that to uplink.

Change-Id: Ied2c79559e77d3f63091c4d61948f2d6a2147d67
2021-09-14 13:18:21 +00:00
Yaroslav Vorobiov
ee4361fe0d satellite/audit: fix segment stripes length calculation
GetRandomStripe function to randomly select a segment stripe to
audit was using `segment.EncryptedSize/segment.Redundancy.StripeSize()`.
Since integer divsion truncates it leads to skipping last stripe if
its size is less than stripe size. Use `Redundancy.StripeCount` to
get correct stripe count.

Change-Id: Ida09e035be30a21219ab3e1aedd66af8be707d1b
2021-09-01 13:25:20 +03:00
Egon Elbre
a14bfdd51c mod: bump common and downgrade sqlite3
sqlite3 accidentally published v2 tags, however, they should not be
used.

Change-Id: I475c4c84f4d3f50bac5471c42a7e995d49eb382a
2021-08-27 00:29:43 +03:00
Michał Niewrzał
211a630982 cmd/uplink: add parallelism flag for single object download
Adds support for new uplink method DownloadObjectAt which
gives ability to download single object in parallel.

Change-Id: I8388653429992b0d24c383d17d7e90904203fe77
2021-08-26 13:13:38 +00:00
storj-antonio
8c8a3e217f testsuite/ui: move from integration
Move ui tests to testsuite/ and introduce a new go.mod.
This prevents cyclic references when building edge binaries for testing.

Change-Id: I613fdc21ef9eaffba2052f6ad973bfa18c7c86bd
2021-08-20 16:33:53 +00:00
Artur M. Wolff
2cd68bf4fb private/lrucache: import from common
Change-Id: Ia1f43d0440fef21122b071b05da59b4cf2689d6c
2021-08-16 10:04:32 +00:00
Jeremy Wharton
51ebc564d9 web/satellite,satellite/console: Overhaul password reset
Updates the password reset page to use the new theme.
Adds new endpoint '/api/v0/auth/reset-password'
for password reset.

Additionally, updates the link-clicking mail simulator to only
click links with a specified attribute. Otherwise, the password reset
cancellation link would be clicked before the password reset link
could be accessed, rendering testing impossible.

Change-Id: I8fde74ef7ad980880a7bf6558e3b9ed31509a393
2021-08-12 17:40:53 +00:00
Yingrong Zhao
077ec96d94 private/server: use quic implementation from storj.io/common
Change-Id: I820cf6444a3ddccee0d7c647dc84c80b2752068c
2021-08-10 13:32:21 +00:00
Michał Niewrzał
f87ad240ed satellite/metainfo: add DeletePart method
We need a way to delete whole part. This especially
needed for uplink multipart API to do cleanup after
aborted or failed part upload.

Test will be added when uplink part will be merged.

Change-Id: I9ba69a49e1adcdce0f42dd3a76f938fcf931155a
2021-08-05 14:43:30 +00:00
Qweder93
5bb3836312 satellite/metainfo: objectEntryToProtoListItem fills metadata on demand
Added includeMetadata parameter which represents if metadata should be included in response
by default true, in case of new uplink version - ObjectIncludes will be used instead.

Change-Id: I2f8d3b4cc354cd655f8093bbbebe0e3c2ae14e6f
2021-08-03 13:41:11 +00:00
Moby von Briesen
149f6f2626 satellite/payments: Implement coupon codes
Full path: satellite/{payments,console},web/satellite

* Adds the ability to apply coupon codes from the billing page in the
satellite UI.
* Flag for coupon code UI is split into two flags - one for the billing
page and one for the signup page. This commit implements the first, but
not the second.
* Update the Stripe dependency to v72, which is necessary to
use Stripe's promo code functionality.

Change-Id: I19d9815c48205932bef68d87d5cb0b000498fa70
2021-07-26 17:15:55 +00:00
Jeremy Wharton
dae6ed7d03 satellite/console: Implement MFA backend
Added MFA passcode and recovery code field for token requests.
Added endpoints for MFA-related activity: enabling MFA,
disabling MFA, generating a new MFA secret key, and
generating new MFA recovery codes.

Change-Id: Ia1443f05d3a2fecaa7f170f56d73c7a4e9b69ad5
2021-07-26 16:37:05 +00:00
Kaloyan Raev
a767aed591
satellite/{metabase,metainfo}: require StreamID for UpdateObjectMetadata
This avoids corrupting objects if reuploads and metadata updates happen
concurrently.
2021-07-08 17:50:37 +03:00
Kaloyan Raev
73b9223758 satellite/metainfo: implement UpdateObjectMetadata
Change-Id: I7d791bb84b73299d395b587074e721c14e4be31c
2021-07-07 12:57:32 +00:00
Jeff Wendling
f474bb6179 cmd/uplinkng: update for breaking clingy changes
clingy changed some and this is just fixes for that

Change-Id: I729aed6329fe0988fcb9b4407f16966a753b3204
2021-07-06 17:26:51 -04:00
Egon Elbre
3c8e7e679a mod: bump storj.io/private and monkit
This gives us nicer output for time.Time formatting when looking at
ongoing database queries.

Change-Id: I00c1ae4d162f537bd69fc2c8132c6507c5ee98cb
2021-06-25 16:57:00 +03:00
Egon Elbre
2ebbc9757f mod: bump storj.io/private
This makes it possible to see queries in mon/ps output.

Change-Id: I0e8b55bd9dbc3d23bc3cc41a71aeeb64d32ce46f
2021-06-15 22:20:23 +03:00
Egon Elbre
f1a9b45599 mod: update quic for Go 1.17rc1
Change-Id: I4218b695089189d5efac1101559a83079fb812cd
2021-06-14 10:14:36 +03:00
andriikotko
072dd24696
integration/ui: add UI integration tests
Introduce a new `integration/ui` package for integration tests.

`integration/ui/uitest` contains a helper package for running browser tests.
2021-06-09 14:39:38 +03:00
JT Olio
da9ca0c650 testplanet/satellite: reduce the number of places default values need to be configured
Satellites set their configuration values to default values using
cfgstruct, however, it turns out our tests don't test these values
at all! Instead, they have a completely separate definition system
that is easy to forget about.

As is to be expected, these values have drifted, and it appears
in a few cases test planet is testing unreasonable values that we
won't see in production, or perhaps worse, features enabled in
production were missed and weren't enabled in testplanet.

This change makes it so all values are configured the same,
systematic way, so it's easy to see when test values are different
than dev values or release values, and it's less hard to forget
to enable features in testplanet.

In terms of reviewing, this change should be actually fairly
easy to review, considering private/testplanet/satellite.go keeps
the current config system and the new one and confirms that they
result in identical configurations, so you can be certain that
nothing was missed and the config is all correct.
You can also check the config lock to see what actual config
values changed.

Change-Id: I6715d0794887f577e21742afcf56fd2b9d12170e
2021-06-01 22:14:17 +00:00
Egon Elbre
926bcd0f83 mod: bump private
Latest version contains a fix for pgxutil.Conn.

Change-Id: I200b1c11a1e536a57a7384c1a747a7216f8cd9ff
2021-05-25 15:28:16 +03:00
Egon Elbre
e7cf369cd8 satellite/metabase: use pgxutil.Conn helper
Clean up underlying pgx.Conn extraction with helper.

Change-Id: Ie43f663a6036c439e675df54fccd3605c4d8d69e
2021-05-25 05:56:35 +00:00
Egon Elbre
515f9b54ff mod: bump monkit
Change-Id: I9be05b91cd9171ebc7d053cd087d6761150d44b6
2021-05-24 16:20:00 +00:00
Jeff Wendling
a1bf9ab6de cmd/uplinkng: initial commit with skeleton
Change-Id: I764618cc60c46882955e9b08b72b3c162aa4929f
2021-05-18 10:04:44 -04:00
Qweder93
a11698f370 multinode/payouts: estimated payouts added
estimated payouts for specific/all satellites added.

Change-Id: I2530c9f1775593588e2a8f6c087ce6b4f9e354c4
2021-05-11 11:33:32 +00:00
Egon Elbre
0858c3797a satellite/{metabase,satellitedb}: deduplicate AS OF SYSTEM TIME code
Currently we were duplicating code for AS OF SYSTEM TIME in several
places. This replaces the code with using a method on
dbutil.Implementation.

As a consequence it's more useful to use a shorter name for
implementation - 'impl' should be sufficiently clear in the context.

Similarly, using AsOfSystemInterval and AsOfSystemTime to distinguish
between the two modes is useful and slightly shorter without causing
confusion.

Change-Id: Idefe55528efa758b6176591017b6572a8d443e3d
2021-05-11 12:40:36 +03:00
Egon Elbre
69b149a66f mod: bump uplink
uplink stopped using zap, hence some of the private methods needed to be
changed.

Change-Id: Iac1fae45a40cd3f1649b9f672bf8c250344986d5
2021-05-06 14:48:36 +00:00
Egon Elbre
a8533042a3 mod: bump uplink
Renamed a private package in uplink and some of the tests need to be
updated.

Change-Id: I8735fc16c2066ecba669618f0c1dadf68273704d
2021-04-30 16:53:54 +03:00
Yingrong Zhao
65b17df58e bump storj.io/monkit-jaeger to latest
This will include the fix to flag errored functions in the jaeger traces

Change-Id: I1187ca0e2a9a390f5a1f3d8353edea84ea76ac91
2021-04-29 13:47:21 -04:00
Ivan Fraixedes
5f29a20937 satellite/satellitedb: Add missing indexes
The DBs of our production satellites have some indexes that we didn't
have in the migrations because at that time we weren't able to add them
because our migration test was not able to deal with Cockroach indexes
with the STORING clause.
We have recently modified the storj.io/private/dbutil/pgutil package to
support the CRDB STRORING clause, so we are adding the missing indexes
to our migrations for being able to have them if we have to recover a DB
from scratch or we deploy a new DB satellite.

Change-Id: I686ff84e5b4c02d9615f50fa531261363affefb8
2021-04-29 15:34:46 +00:00
Michał Niewrzał
7944df20d6 storj: use multipart API
Change-Id: I10b401434e3e77468d12ecd225b41689568fd197
2021-04-26 13:15:09 +00:00
Egon Elbre
4d37378129 storage/{cockroachkv,postgreskv}: delete unused code
Change-Id: I11da924508d207d8816bda206d28261931d9cd4e
2021-04-23 16:37:28 +03:00
Egon Elbre
a2e20c93ae private/dbutil: use dbutil and tagsql from storj.io/private
Initially we duplicated the code to avoid large scale changes to
the packages. Now we are past metainfo refactor we can remove the
duplication.

Change-Id: I9d0b2756cc6e2a2f4d576afa408a15273a7e1cef
2021-04-23 14:36:52 +03:00
Egon Elbre
ebc981291a satellite/accounting: use correct error in tests
rpcstatus codes are not part of the public uplink API
and the tests should not use them.

Change-Id: I9c96c557bf2dac5d6fff200fb217378524fd14ff
2021-04-22 07:25:42 +00:00
Kaloyan Raev
785adfb849 go.mod: bump to uplink v1.4.6
Change-Id: I855d8a8860fa126756930fe09073ddb259bc99ac
2021-04-21 12:30:50 +03:00
Egon Elbre
75ca01e381 cmd/uplink: fix importing access in Windows
Currently os.Create was leaving a file open causing atomic write file to
fail with access denied.

Also add a specific test for importing an access.

Change-Id: Id188bc480e795849ec7fdc72b1fc86433d76c47a
2021-04-19 15:11:39 +03:00
JT Olio
037c7799b5 storj/storj: bump storj/common
This only pulls in https://review.dev.storj.io/c/storj/common/+/4474

Change-Id: I69dde0d66cd899ef4c91bc74a3c481ff2bf36a50
2021-04-12 07:36:15 -06:00
Egon Elbre
8f13dcacb9 mod: bump uplink
This starts using uplink with fixes:

* range downloading a migrated object
* downloading a zero length range

Change-Id: I4a95d4cdfe86ef44959dd81921ccad28ba7d7af5
2021-04-09 15:43:53 +00:00
Egon Elbre
372bb1140e mod: bump uplink and private
Change-Id: I12d540c47a47a9b16e8a9fa7e97cafbaf215e12f
2021-04-08 14:02:19 +00:00
Egon Elbre
47be501188 satellite/metainfo: add DownloadObject endpoint
Add endpoint for getting object information, list segments in a range
and download the first segment in the range.

Change-Id: I056d697ae87c9aa34e7deccba8713902db260457
2021-04-06 19:47:36 +03:00
Egon Elbre
17817d84e9 satellite/metainfo: add PlainOffset to SegmentDownloadResponse
Change-Id: Ib68f2fd0ad86556f443a01a514243247dc7989a3
2021-04-06 15:21:00 +03:00
Yingrong Zhao
f3df411b41 bump storj.io/common to latest
Change-Id: I1396008c6b7cae246653694126f1a54716a8bfa4
2021-04-05 19:07:02 -04:00
Yingrong Zhao
e0c3a7a47d bump quic-go to latest
Change-Id: I2716a6670e2eb0b226aa0a3510b0c1ec8e927ef6
2021-04-05 09:28:17 -04:00
Yingrong Zhao
7dfb89a871 bump quic-go to latest release
Change-Id: Ic5e93f7a6ffe0bb46fc8f8bf1ce310eac4aa1182
2021-03-31 13:28:41 +00:00
Kaloyan Raev
035c393da0 satellite: update tests to pass etag.Reader to multipart.PutObjectPart
Change-Id: Ibe99357945ae7a91f5b5d4f87b83d425c9fa84a5
2021-03-29 13:18:11 +00:00
JT Olio
2ddbaf1eb5 satellite/wasm: support restricting full access grants to paths
Change-Id: Id6d4fa41db068d32e7c0d542d9d8805fba927fc6
2021-03-26 19:49:21 +00:00
Moby von Briesen
3db52491ec satellite/analytics: Add analytics service to satellite
* Set up basic structure of new service.
* Implement a basic analytics track event for user creation.

Change-Id: Ica8c785540b1ef9d848404af307a22f21d33c6aa
2021-03-25 13:28:51 -04:00
Kaloyan Raev
1156d8b4ed mod: update drpc to v0.0.20
Change-Id: I0ac6ff9bebe5b16d4fb63b30242c87c899e7d358
2021-03-24 19:06:56 +02:00
Egon Elbre
54c2ace483 mod: update drpc to v0.0.19
Change-Id: Ia2e0e0a371368700c596ce89e24748ea4b5c016a
2021-03-23 17:13:28 +02:00
Michał Niewrzał
237782813b Merge remote-tracking branch 'origin/multipart-upload'
Change-Id: If6c5a450b238adab55d1e0dea67d01e5f5768a9f
2021-03-23 09:44:49 +01:00
Ivan Fraixedes
4c1098e571 Redis: Update Redis package to last major version
Update the Redis dependency to use the last major production version.
The last version accepts a context parameter in all the network methods
so it allows us to pass it through them.

Change-Id: I34121b2ec3c2728602115c724933ad24c9e6e4fd
2021-03-18 14:19:49 +00:00
Jeff Wendling
09adfc1205 mod: bump monkit to v3.0.10
Change-Id: Ieed0b4161b158edd7c8e7065f200675cfdfdebf8
2021-03-18 13:17:57 +00:00
Michał Niewrzał
fa083a7f05 Merge remote-tracking branch 'origin/main' into multipart-upload
Change-Id: Ib5ce5965b77b81c254d08c27ab30c7eccefbd4c6
2021-03-17 15:37:17 +01:00
Isaac Hess
1de6315b83 metainfo: Stub DownloadObject metainfo endpoint
This is the stub "not implemented" endpoint for the new DownloadObject
RPC.

Change-Id: Ifa724e20d89d839f2b6968d6283bc66a3bab8781
2021-03-15 10:52:57 -06:00
Yingrong Zhao
a36d854d8e bump monkit-jaeger to latest
Change-Id: I8c20769e667afd3e22ebd10e2e045d2e15fad7f7
2021-03-15 14:42:03 +00:00
Kaloyan Raev
6e661da0a0 satellite/metainfo/metabase: read created_at from DB
All SQL queries for reading segments are updated to read the created_at
column where appropriate.

Change-Id: Icd7c7672fa71e992673078598b28229bb898c728
2021-03-15 11:37:14 +00:00
Michał Niewrzał
67e26aafcd Merge remote-tracking branch 'origin/main' into multipart-upload
Change-Id: I9b183323cb470185be22f7c648bb76917d2e6fca
2021-03-10 08:53:38 +01:00
Egon Elbre
580eecf0bb mod: bump monkit to v3.0.9
Change-Id: I6b2e6276c4d83f06eab31481fc3826f8d1307dc4
2021-03-09 18:37:03 +02:00
Michał Niewrzał
dd9ad09301 satellite/metainfo: return RS value with DownloadSegment response
Until now we where using single RS per object but it turns out that we
need to be able to support RS per segment. We need to give uplink such information while downloading.

As an addition we are using RedundancySchemePerSegment flag for GetObject request to detect if
we should try to get RS from segment for this request response.

Change-Id: I209dad324496ff59b521b11d2343da61dcdbe7f5
2021-02-17 12:33:48 +00:00
Michał Niewrzał
f1169ad7bd satellite/metainfo: return RS value with DownloadSegment response
Until now we where using single RS per object but it turns out that we
need to be able to support RS per segment. We need to give uplink such information while downloading.

Change-Id: I6565b7c08962b3a1429f6079e7c2023a0a7c8b72
2021-02-17 12:39:53 +01:00
Michał Niewrzał
12402eb729 Merge remote-tracking branch 'origin/main' into multipart-upload
Change-Id: I38adf8218c1415c7ea1910f8bd6bed13544b0f03
2021-02-17 08:50:38 +01:00
Michał Niewrzał
8685c6c9cd private/testplanet: use better method in test
In ec client in uplink we have two methods Put and PutSingleResult. Logic is the same but result for PutSingleResult is combined into single var. We would like to remove unused Put method but to do this it needs to be replaced in test.

Change-Id: Ia65eff3ecc9e68d5b3180b557ea82fa31d3c969c
2021-02-16 13:24:01 +00:00
Jeff Wendling
fe5b6e1725 sat/console/consolewasm: restrict enc access to api key
Change-Id: Ie263ffc7343e59ee9a90996b259bb04258a78f29
2021-02-11 17:52:32 -05:00
Michał Niewrzał
908a96ae30 Merge remote-tracking branch 'origin/main' into multipart-upload
Change-Id: I075aaff42ca3f5dc538356cedfccd5939c75e791
2021-02-11 11:48:23 +01:00
Michał Niewrzał
dc2bec9f89 satellite/metainfo: collect uplink versions with monkit
This change adds metrics for uplink version and method which was used by this version.

Change-Id: I05fc425a024805cfcd6d8add810a2a0749405f4f
2021-02-10 17:47:13 +01:00
Rafael Gomes
d9d6edde71 go.mod: bump monkit-jaeger to replace dashes by underscore
Change-Id: I51fd920901ff559e9def795e9183bbd6d06e4045
2021-02-09 15:24:00 -03:00
Caleb Case
f011a5e8cc
Use WithNonce when creating caveats
This ensures the caveats are unique even when they contain the same
permissions and will result in unique macaroons. This is important to
ensure revocation doesn't impact more macaroons than intended.

Change-Id: I6354edd0119f2d85eaf580f2d1926a3de9151b88
2021-02-04 13:46:32 -05:00
Egon Elbre
615586a471 {satellite,storagnode}/internalpb: bump gogo to v1.3.2
Change-Id: I585a3197630b9b675a4b540cf25612e1ed39b599
2021-02-04 11:09:13 +00:00
Michał Niewrzał
9a60011774 Merge remote-tracking branch 'origin/main' into multipart-upload
Change-Id: Ia90f29be432e207c4125f7f955c912978eabe59a
2021-02-04 09:38:08 +01:00
Yingrong Zhao
21dfd99883 bump storj.io/private to latest
Change-Id: If4f08cd8cd8cd7cb09cd7d7051d966b08fe96c2e
2021-02-04 00:54:28 +00:00
Ivan Fraixedes
9c9f481469 satellite/orders: Remove deprecated endpoint
Remove the orders Settlement endpoint because it isn't used and it was
already always returning an error.

Change-Id: I81486fbe7044a1444182173bc0693698ee7cfe7e
2021-02-03 23:47:07 +00:00
Kaloyan Raev
6f3d0c4ad5 Merge remote-tracking branch 'origin/main' into multipart-upload
Conflicts:
	go.mod
	go.sum
	satellite/repair/repair_test.go
	satellite/repair/repairer/segments.go

Change-Id: Ie51a56878bee84ad9f2d31135f984881a882e906
2021-02-02 19:19:04 +02:00
Egon Elbre
0bbe2f1d61 satellite/metainfo: add unimplemented ListPendingObjectStreams
Change-Id: I6c0fd240ce5be82c1f464470a6f147289b1cdf9d
2021-02-02 16:51:36 +02:00
Natalie Villasana
91bd4191dd satellite/accounting: add rollup archiver chore
The rollup archiver chore moves bucket bandwidth rollups and
storagenode rollups that are older than a given duration
to two new archive tables.

Change-Id: I1626a3742ad4271bc744fbcefa6355a29d49c6a5
2021-02-01 09:29:54 -05:00
Kaloyan Raev
d0612199f0 Merge remote-tracking branch 'origin/main' into multipart-upload
Conflicts:
	go.mod
	go.sum
	satellite/metainfo/config.go
	satellite/metainfo/metainfo_test.go

Change-Id: I95cf3c1d020a7918795b5eec63f36112fdb86749
2021-02-01 14:32:12 +02:00
Egon Elbre
ee1f67bb18 go.mod: bump quic to fix Go 1.16rc1
Change-Id: Ief21af5302c36a91d19755e58a2e36fae61cd15b
2021-02-01 13:32:56 +02:00
Egon Elbre
a97b5c8552 go.mod: bump quic to master, such it compiles with Go 1.16
Change-Id: I4af35cd561955094a34fe3c96b0c532f325ac505
2021-01-28 11:13:02 +02:00
Michał Niewrzał
50bea3ab1a satellite/metainfo: adjust tests to changes in uplink
One of uplink method changed its signature and we need to fix test on satellite side.

Change-Id: Ib89ea6aa25c57bac11bc3e0e9c2c89a4b9debd7c
2021-01-26 08:54:35 +00:00
Qweder93
c139cbd76b storagenode/payouts: fix CurrentMonthExpectations timezone handling. Estimations based on node's join date.
On servers with non-UTC it would have calculated a different month boundary.
If node joined in current month calculations will be related on amount of days node've been working.

Change-Id: Ie572b197f50c6cdff5a044a53dfb5b9138f82f24
2021-01-25 19:03:30 +02:00
Egon Elbre
a700a1bdab satellite/metainfo/metabase: add benchmark
Change-Id: I4289aac252f9c951548444aa5a82e007e9f47806
2021-01-25 14:59:24 +00:00
Yingrong Zhao
ae03a0540c pkg/quic: add quic implementation
Due to the issues with the licensing and go version requirement by the
quic-go library, we can not introduce the library into libuplink at the
moment. Therefore, we decided to put our quic integration code into
storj/storj. It will unblock us from rolling out quic to storagenodes.

Change-Id: If9d765da10c45947a4e3a3a11e4679bab69bcf08
2021-01-21 21:28:30 +00:00
Kaloyan Raev
c24ada7114 Merge remote-tracking branch 'origin/main' into multipart-upload
Conflicts:
	go.mod
	go.sum

Change-Id: Icf7c029e9d800e5f6a9fdd208c36f28e05468690
2021-01-20 17:35:57 +02:00
Kaloyan Raev
ea48322dd3 satellite/metainfo: use deterministic signing for satStreamID
So we can have stable UploadID for multipart uploads.

Change-Id: Iac6780394c8cc0f96c0b9c4b850b92ed3627a9b0
2021-01-19 15:56:29 +00:00
Qweder93
6ba8f6c8a9 storanode, satellite: payout renamed to payouts, expected estimation payouts added, console api for audits reworked
Change-Id: I4aa5e99bffaa87d0a800a429a4c83aa498ad4b7b
2021-01-18 10:56:03 +00:00
Kaloyan Raev
424d2787eb go.mod: bump deps to uplink v1.4.5
Change-Id: Ic5a4efc78bb9c140a4104fd96c514477e225cba4
2021-01-14 17:51:29 +02:00
Kaloyan Raev
6dff40f5c5 Merge remote-tracking branch 'origin/main' into multipart-upload
Conflicts:
	go.mod
	go.sum
	satellite/metainfo/metainfo.go

Change-Id: Ib5c49f3c911c58319855a171f9ce73657da976d9
2021-01-14 14:33:59 +02:00
Egon Elbre
d11c2b709e go.mod: bump storj.io/common
* Add missing endpoints.
* Fix deprecated packages and funcs.

Change-Id: I756090c46a4d15eabf6d413a593cdc64c5809bc7
2021-01-13 14:51:08 +00:00
Michał Niewrzał
ec88d21a3c Merge 'main' branch.
Change-Id: I6e8162d1a6caf75e89c9f9c9f9522730aebf83ae
2021-01-11 10:26:58 +01:00
Kaloyan Raev
1e44be2900 go.mod: bump deps to uplink v1.4.4
Change-Id: I203259f0f324284ac2991e2cabf4bd471cd8ad5d
2021-01-06 13:07:17 +00:00
Michał Niewrzał
ad3e3a38c5 Merge 'main' branch
Change-Id: Ia0db1b1f9ef3e0671d3f2208881b0abc3064e200
2021-01-04 12:13:45 +01:00
Moby von Briesen
edbee53888 satellite,storagenode: Pass audit history over GetStats endpoint
Full prefix: satellite/{overlay,nodestats},storagenode/{reputation,nodestats}

Allow the storagenode to receive its audit history data from the
satellite via the satellite's GetStats endpoint.

The storagenode does not save this data for use in the API yet.

Change-Id: I9488f4d7a4ccb4ccf8336b8e4aeb3e5beee54979
2020-12-30 19:13:26 +00:00
JT Olio
f787aa4647 go.mod: bump uplink to v1.4.3
Change-Id: Ia6ff4f3a84897aaebe7aef7680f17b94b59cb9de
2020-12-23 11:18:26 +00:00
Michal Niewrzal
57f374af24 Merge 'master' branch
Change-Id: Idf6b10ea7ca94e4d232e6a3b6a38ef2e646ba197
2020-12-15 08:26:53 +01:00
Qweder93
c630037f34 multinodepb: diskspace, reputation and status added
Change-Id: I470fa8b59ce7f00f2fbedbd0c0878fb5fff0590c
2020-12-14 17:01:16 +00:00
Michal Niewrzal
b3acc1101a Merge 'master' branch
Change-Id: Iee99400c7095770e61cde94b3b2c8eb0ddec463d
2020-12-10 15:42:52 +01:00
Egon Elbre
588dff6d78 Upgrade to uplink v1.4.2
Change-Id: Ib1f5e70c76710d9c49a39347e4ed95bd4671efab
2020-12-10 15:47:11 +02:00
Kaloyan Raev
3269401515 deps: update to latest uplink from multipart-upload branch
This resolves an issue in Uplink CLI with listing size 0 for files
uploaded with multipart upload.

Change-Id: I80e0b11a96f87ed6a87eb5301034c08dbc09e8aa
2020-12-07 14:54:51 +00:00
Michal Niewrzal
218bbeaffa Merge 'master' branch
Change-Id: Ica5c25607a951076dd9f77e35e308062f71ce3f0
2020-12-07 15:05:52 +01:00
Caleb Case
c1bdb88d3c
Upgrade to uplink v1.4.1
Change-Id: Ie823f9b72d41f3da8e929d96a870fffc2e7ec12b
2020-12-04 11:38:19 -05:00
Caleb Case
6ba8dd58e6 Update to uplink v1.4.0
Change-Id: Ia01fec012031ceb2867b80a37e6a6c2a7d179cca
2020-12-02 13:44:32 +00:00
Michal Niewrzal
5a7bc9657d Merge 'master' branch
Change-Id: If583132a821274dc4b78cf5f72b853ba8460c619
2020-11-30 12:57:22 +01:00
Michal Niewrzal
21602e0494 satellite/metainfo: enable commented test
Test was commented to make uplink refactoring possible. Now we can bring
back this test.

Change-Id: I0511b76073efaafed8aac97f8e845dcec93dd059
2020-11-30 10:49:23 +01:00
Michal Niewrzal
cfb45a785c satellite/metainfo/metabase: add TotalPlainSize to objects table
We need to know how large is object in general and we shouldn't query
segments for that.

Change-Id: Ie316e1ec38bfc39065e6046b79b55d58f3ce9d53
2020-11-25 14:35:28 +01:00
Michal Niewrzal
46102c1942 satellite/metainfo: change ListSegments required permission to
Read/Download

We should treat listing segments as a part of download process, not
listing.

Change-Id: Iae72c97c77976c427f668a96486af613d58da3b9
2020-11-24 17:50:33 +01:00
Michal Niewrzal
5be4d413b0 satellite/metainfo: set PlainSize for inline segment
Plain size is required by metabase implementation.

Change-Id: I6739744bff4ce485e0066f2a49acdf38560d31fb
2020-11-20 13:54:14 +01:00
Kaloyan Raev
05fe497a6f satellite/metainfo: add stream id when listing pending objects
Change-Id: Idcdffd70b82c0fe4f79635fc193604b66101ef76
2020-11-19 14:22:09 +02:00
Michal Niewrzal
9bdc8ebf75 satellite/metainfo: adjust GetObject to handle
EncryptedMetadataEncryptedKey

Change-Id: If159fdcc9f37e9449a7d8459a277366d6d6b5802
2020-11-17 16:38:10 +00:00
Michal Niewrzal
cd7d911b26 satellite/metainfo: implement ListSegments with metabase
Change-Id: I78fb2400fed0bb01c5e05bd5bbcddef5e4a3bdf0
2020-11-16 08:33:35 +00:00
Michal Niewrzal
3fe16f4003 satellite/metainfo: upload/download with metabase
This change is adjusting metainfo endpoint to use metabase for uploading
and downloading remote objects. Inline segments will be added later.

Change-Id: I109d45bf644cd48096c47361043ebd8dfeaea0f3
2020-11-11 12:13:52 +00:00
Michal Niewrzal
7dde184cb5 Merge 'master' branch
Change-Id: I6070089128a150a4dd501bbc62a1f8b394aa643e
2020-11-10 11:58:59 +00:00
Egon Elbre
b892a00143 mod: bump dependencies and reenable test
We shouldn't have any EOF issues with recent drpc fix, let's reenable
and see whether it's still flaky.

Change-Id: I0de312bcb087c7f70ec9d3281d73d86f971845d5
2020-11-10 10:32:21 +00:00
jessicagreben
c4c29e370a wasm: add webassembly code for creating access grant in console web UI
Change-Id: I3c6d9afc660f3d959d6138db84341e9460b877a1
2020-11-04 12:08:30 -08:00
Egon Elbre
716068a1e0 Merge branch 'master'.
Change-Id: Ic14325edc291573582dce0cea3e04991a820b48b
2020-11-02 13:02:01 +02:00
Egon Elbre
9674ce40e6 mod: bump dependencies
Change-Id: I61beaf9793609f7614478cfa604b2721d9010b52
2020-10-30 16:17:55 +02:00
Egon Elbre
995900e02f Merge branch 'master' into multipart-upload
Change-Id: I8717f9f7cc54f3934402a98f21129e16841b25bd
2020-10-30 11:48:05 +02:00
Michal Niewrzal
8f26f66da0 internalpb: move satellite specific protobuf types storj/storj
We have some types that are only valid for satellite usage. Such types
are SatStreamID and SatSegmentID. This change moves those types to
storj/storj and adds basic infrastructure for generating code.

Change-Id: I1e643844f947ce06b13e51ff16b7e671267cea64
2020-10-30 08:49:16 +00:00
Qweder93
624255e8ba storagenode/secret: db tests added, small renaming fixes added
Change-Id: I7eae1a9a64c20a39c97e81fa741cfc9b9e1e615a
2020-10-29 14:23:04 +02:00
Michal Niewrzal
809eb14ac2 satellite/metainfo/metabase: move metainfo PoC into storj repo
Change-Id: I39356d8bc7305b4a8ea0c1fb5603010ad72a68b9
2020-10-29 09:44:55 +01:00
Caleb Case
be84616e69
Update to uplink v1.3.1
Change-Id: Iae1e6a63c00d07fee9047638ddc9a414780ce0e9
2020-10-14 09:39:29 -04:00
Egon Elbre
2f4bb114d4 go.mod: bump common to remove sha256-simd
Change-Id: I77aeb3b4bea87e8d5bb83b05e86b61a80a695f2a
2020-10-14 12:47:37 +03:00
Caleb Case
cf1748158a
Bump Dependencies
Change-Id: I4c8a4438e74379a490a19f1f88ea9dac7715dbbd
2020-10-09 09:33:49 -04:00
Yingrong Zhao
f13084983d bump common to latest
Change-Id: I55ca40a7b2c58d70225da07b7a62d5ad60451881
2020-10-01 15:39:09 -04:00
Yingrong Zhao
c085a17a52 bump common and uplink to latest
Change-Id: I717f0214dd9973acd51b7732c5d64587f610c805
2020-10-01 15:38:58 +00:00
Egon Elbre
139ff3427b go.mod: bump dependencies
Change-Id: I2b1bf014a60ef6ddc4b1b546f58393c34ffe22fc
2020-09-30 11:28:37 +03:00
Egon Elbre
c23a8e3b81 go.mod: update pgx to v4.9.0
Fix query to use TextArray instead of VarcharArray.
Fix queries to use the correct type.

Change-Id: Ibb7e55adba277d05778118d81ca697470e72c374
2020-09-29 19:03:08 +00:00
Kaloyan Raev
b409b53f7f cmd/satellite: command for verifying piece hashes
Jira: https://storjlabs.atlassian.net/browse/PG-69

There are a number of segments with piece_hashes_verified = false in
their metadata) on US-Central-1, Europe-West-1, and Asia-East-1
satellites. Most probably, this happened due to a bug we had in the
past. We want to verify them before executing the main migration to
metabase. This would simplify the main migration to metabase with one
less issue to think about.

Change-Id: I8831af1a254c560d45bb87d7104e49abd8242236
2020-09-29 10:58:24 +00:00
Egon Elbre
888bfaae4b cmd/satellite: only add google profiler to satellite
Previously uplink, storagenode etc. included google cloud profiler,
however they don't need it.

Change-Id: Ibc95cb03d667a3844672eecd49fa455a6acc3866
2020-09-25 18:56:59 +03:00
JT Olio
1a094fd6da storj: bump storj.io/private
Change-Id: Ia2927010332bb2e1d140603299341a8549093233
2020-09-10 16:21:12 -06:00
Egon Elbre
b45cad5eed bump common to latest
This makes tests pass on Go 1.15.

Change-Id: I8407f16dcfc0b306aa685da30dcfedea6d3b515d
2020-09-05 10:51:58 +03:00
Yingrong Zhao
cccd62fff9 bump uplink version to v1.3.0
Change-Id: I9e7833713ca00f19bfb662eeaf61ef57bdc3bbbd
2020-09-04 09:57:35 -04:00
Yingrong Zhao
af773ec8a6 cmd/uplink: use DeleteBucketWithObjects for bucket force deletion
This PR updates `uplink rb --force` command to use the new libuplink API
`DeleteBucketWithObjects`.
It also updates `DeleteBucket` endpoint to return a specific error
message when a given bucket has concurrent writes while being deleted.

Change-Id: Ic9593d55b0c27b26cd8966dd1bc8cd1e02a6666e
2020-09-02 16:39:20 +00:00
Qweder93
36d752e92d storagenode/reputation: offline_under_review_at added
Change-Id: Ia7ec79b2d6f20fe29de0c36223f9485380d2845c
2020-09-02 18:48:28 +03:00
Qweder93
7d9897b7af storagenode/nodestats: online_score added
Change-Id: I84b50a6cace306e5f10d53a2073fe8810d4d2960
2020-09-02 17:45:01 +03:00
Isaac Hess
ba1a113e2e satellite/metainfo: Add test for GetObjectIPs
This PR adds the integration test for GetObjectIPs, testing all repos.

Change-Id: Id4d4f19c427a2e320b2a82efa68150fa7f5f86fb
2020-08-31 10:27:56 -06:00
Cameron Ayer
3e343b683b cmd/segment-reaper: add metrics for zombie segments count
Change-Id: I106c6795946283165ba3de8465e5898346da1a3f
2020-08-26 18:42:59 +00:00