Commit Graph

404 Commits

Author SHA1 Message Date
Qweder93
b6625cadea satellite/metainfo: uncomment tests, renamed EncryptedPath into EncryptedObjectKey
moved tests back and renamed params that were renamed on uplink side

Change-Id: I4a9592bd955f2217c5809c235c394a4e78d337ea
2022-01-11 17:18:18 +02:00
dlamarmorgan
ab37b65cfc satellite/{accounting,orders,satellitedb}: group bucket bandwidth rollups by time window
Batching of the order submissions can lead to combining the allocated
traffic totals for two completely different time windows, resulting
in incorrect customer accounting. This change will group the batched
order submissions by projectID as well as time window, leading to
distinct updates of a buckets bandwidth rollup based on the hour
window in which the order was created.

Change-Id: Ifb4d67923eec8a533b9758379914f17ff7abea32
2022-01-05 20:24:48 +00:00
Jeff Wendling
4426470173 cmd/uplinkng: rename parallelism flags, small fixes
this makes the flags match rclone nomenclature

fixes test-uplinkng to use the temporary config dir
instead of the machine default, and clean up some.

bumps clingy so that the command errors when an unknown
command is specified.

also fixes some printfs in share to use clingy stdout.
it still does some external actions that should be
passed through a ulext.External for mocking, but
that's ok for now.

Change-Id: Icc231e7e26393541c312396fec907b640b97718e
2021-12-22 13:13:30 +00:00
Michał Niewrzał
db7985ca90 satellite/metainfo: set EncryptedMetadataEncryptedKey while listing objects
At some point we missed to add metadata key to list objects
response. Because of that uplink is taking key from pb.SteamMeta.
We need to clean it up.

Tests will be added on uplink side.

https://github.com/storj/uplink/issues/71

Change-Id: I3328e2f1b86bca15aeaf89f8e59cdca3c8e97742
2021-12-21 13:09:11 +00:00
Jeff Wendling
9615bd191f cmd/uplinkng: update clingy for better boolean support
this allows commands like

    uplinkng cp -r sj://foo sj://bar

to work correctly, rather than complain that sj://foo is
not a boolean.

Change-Id: I003e47aabb85566bc2b454851cf55043b17ee7ea
2021-12-14 09:41:38 +00:00
Michał Niewrzał
ace90fd844 mod: bump clingy version
Latest version have a fix for:
https://github.com/storj/storj/issues/4257

Change-Id: I0e3a67fe632cd5c209b24e1ee3080563cafbb626
2021-12-08 11:10:25 +01:00
Kaloyan Raev
f773bb80c8 mod: bump common to fetch latest placement type changes
Change-Id: I3d0813f05622e706c9be1a578b5e4d4159d16dfc
2021-11-16 12:42:25 +00:00
Mya
bf51c286d9
satellite/geoip: update node check-in to associate a country code
Resolves https://github.com/storj/storj/issues/4247

Change-Id: Idfd71bf1795d48ca3c686066bbdb95b9c6594f00
2021-11-10 16:44:41 +01:00
Márton Elek
9bdcc415bc satellite/nodeselection: add geofencing constraints to the node selection criteria
Closes https://github.com/storj/storj/issues/4242
Change-Id: Ieda59a4f37c673e4e81abb4c89c09daf3199bbc7
2021-11-08 17:04:31 +00:00
dlamarmorgan
1dd537953e satellite/metainfo: strip the uplink version from the UserAgent and bound its size
The UserAgent should be stored as is, with the exception of removing the trailing version from any libuplink user agents

Change-Id: If17ef2fc4b59480a3477300f2585a07d64cc2bf4
2021-11-04 10:09:54 -07:00
Jeff Wendling
4d023f773f cmd/storagenode-updater: update to correct version
Currently if the nodes is below the minimum version it will immediately
update to the suggested version, regardless if its eligible from the seed
or not. This change corrects the behaviour to update to the minimum
version only and then properly respect/wait for the rollout to include it.

Updates based on logic here: https://review.dev.storj.io/c/storj/private/+/6187

Change-Id: Ic6c91c48ae9b8a116378b2573fbfca7e7bd5cc32
2021-10-29 20:38:30 +00:00
Stefan Benten
8c1a149de2 build: bump private dependency
Enables proper version printing.

Change-Id: I37e12d6823f86f9275ac9f64abf324003019bd18
2021-10-29 12:25:33 +13:00
Egon Elbre
1409288c49 mod: bump semantic version to go 1.17
This enables module graph pruning and lazy module loading.

Change-Id: I5d63a686d4f4eda93b23576169bb03847292f8c7
2021-10-26 11:58:13 +00:00
Stefan Benten
df6cf60ffe
go.mod: update to minimum supported go version (#4239) 2021-10-22 21:12:13 +02:00
Egon Elbre
56f35e9ea9 mod: bump private repo
Contains a fix to ignore cockroach DROP error while testing.

Change-Id: I74b02240f570c30a66f6f0b938ce036392c05e06
2021-10-22 15:02:48 +03:00
Clement Sam
f82b696858 mod: bump common dependency
Change-Id: I24ba0ef8d0ca25a34e28de1064a2f1213787ccc5
2021-10-21 16:33:48 +00:00
Egon Elbre
bdadc4b460 private/testplanet: add stack traces related to the planet
It's useful to see only the goroutines that are related to the current
testplanet.

Change-Id: Id77c62dec42c09d44c523b019f7bdc940905fcf1
2021-10-20 23:12:07 +03:00
Michał Niewrzał
b8e8110ca3 mod: bump common and uplink dependency
Contains also adjustments for latest storj/common changes.

Change-Id: I11f70d466b3d9785ee93ac424629b2b868889d02
2021-10-12 19:13:49 +00:00
Michał Niewrzał
1fdb0eaa5b Revert "satellite/metabase: use storj.Nonce instead []byte"
This change introduce problems with server side move so
let's revert it for now. Problem was found when latest
version of storj/storj was used in uplink tests.

This reverts commit 1ef06fae99.

Change-Id: I4d4fad5d1ea04ba15ff9d7bd765f7e078e9187c2
2021-10-12 15:39:54 +02:00
Michał Niewrzał
1ef06fae99 satellite/metabase: use storj.Nonce instead []byte
We were using mixed types for nonce fields. Protobuf
have storj.Nonce, metabase have []byte. This change
is a refactoring to have everywere its possible only
storj.Nonce.

Change-Id: Id54bd8481f30c721cdaf3df79206d25e7cfdab55
2021-10-11 16:13:34 +00:00
Michał Niewrzał
f837551d76 storj: bump uplink to v1.7.0
Change-Id: Ic5ffc704ffbcf7dc890fe0554a8e23f1a777ecbd
2021-10-08 08:04:57 +00:00
Egon Elbre
6511bb91fb private/testplanet: support writing monitoring spans
This allows to set `STORJ_TEST_MONKIT` to either
`svg` or `json` to write individual testplanet test
traces to disk.

It also allows to specify an absolute directory:

  STORJ_TEST_MONKIT=svg,dir:/abs/dir/path

It requires an absolute path, because from the context of
tests, there's no easy way to find the folder where tests
were called.

Change-Id: I6fe008a4d4237d221cf5a5bede798b46399ee197
2021-09-30 09:03:35 +03:00
Michał Niewrzał
0209bc6ef7 cmd/uplink: add mv command
New command for cli to move object to different
location.

uplink mv sj://bucket/your-object sj://bucket/moved-object

Change-Id: I85a4961aa59f250819954e78f20363ac3c570938
2021-09-29 19:10:58 +00:00
Egon Elbre
6d3fd33ca3 satellite/metabase/segmentloop: start immediately on manual trigger
Currently loops wait for the coalesce duration for TriggerWait.
Let's skip the coalesce when we trigger it manually.

Change-Id: If5bacd4e263d233f1f3ea41b989922d2ed5a48d4
2021-09-28 18:32:03 +00:00
Egon Elbre
6e660cecdd Jenkinsfile: test cross-compile and bump deps
Change-Id: I47091de2e80bd96fcada616f75e1db07a59fb0c1
2021-09-16 18:59:31 +03:00
Yingrong Zhao
f0b73b4f45 mod: bump drpc, uplink and common
Updates:
- drpc to v0.0.25
- uplink to latest
- common to latest

This will fix the crash on UDP network temporary error for windows and a
rare deadlock during shutdown

Change-Id: I7e9c4ace22665c5471f19f9b8bf63aec4d68ceb3
2021-09-16 07:42:32 +00:00
crawter
07311cdd6f satellite/metainfo: begin move object added
First from two methods needed to perform server-side move. It gets
metadata key and nonce and all segments key and nonces and returns
all of that to uplink.

Change-Id: Ied2c79559e77d3f63091c4d61948f2d6a2147d67
2021-09-14 13:18:21 +00:00
Yaroslav Vorobiov
ee4361fe0d satellite/audit: fix segment stripes length calculation
GetRandomStripe function to randomly select a segment stripe to
audit was using `segment.EncryptedSize/segment.Redundancy.StripeSize()`.
Since integer divsion truncates it leads to skipping last stripe if
its size is less than stripe size. Use `Redundancy.StripeCount` to
get correct stripe count.

Change-Id: Ida09e035be30a21219ab3e1aedd66af8be707d1b
2021-09-01 13:25:20 +03:00
Egon Elbre
a14bfdd51c mod: bump common and downgrade sqlite3
sqlite3 accidentally published v2 tags, however, they should not be
used.

Change-Id: I475c4c84f4d3f50bac5471c42a7e995d49eb382a
2021-08-27 00:29:43 +03:00
Michał Niewrzał
211a630982 cmd/uplink: add parallelism flag for single object download
Adds support for new uplink method DownloadObjectAt which
gives ability to download single object in parallel.

Change-Id: I8388653429992b0d24c383d17d7e90904203fe77
2021-08-26 13:13:38 +00:00
storj-antonio
8c8a3e217f testsuite/ui: move from integration
Move ui tests to testsuite/ and introduce a new go.mod.
This prevents cyclic references when building edge binaries for testing.

Change-Id: I613fdc21ef9eaffba2052f6ad973bfa18c7c86bd
2021-08-20 16:33:53 +00:00
Artur M. Wolff
2cd68bf4fb private/lrucache: import from common
Change-Id: Ia1f43d0440fef21122b071b05da59b4cf2689d6c
2021-08-16 10:04:32 +00:00
Jeremy Wharton
51ebc564d9 web/satellite,satellite/console: Overhaul password reset
Updates the password reset page to use the new theme.
Adds new endpoint '/api/v0/auth/reset-password'
for password reset.

Additionally, updates the link-clicking mail simulator to only
click links with a specified attribute. Otherwise, the password reset
cancellation link would be clicked before the password reset link
could be accessed, rendering testing impossible.

Change-Id: I8fde74ef7ad980880a7bf6558e3b9ed31509a393
2021-08-12 17:40:53 +00:00
Yingrong Zhao
077ec96d94 private/server: use quic implementation from storj.io/common
Change-Id: I820cf6444a3ddccee0d7c647dc84c80b2752068c
2021-08-10 13:32:21 +00:00
Michał Niewrzał
f87ad240ed satellite/metainfo: add DeletePart method
We need a way to delete whole part. This especially
needed for uplink multipart API to do cleanup after
aborted or failed part upload.

Test will be added when uplink part will be merged.

Change-Id: I9ba69a49e1adcdce0f42dd3a76f938fcf931155a
2021-08-05 14:43:30 +00:00
Qweder93
5bb3836312 satellite/metainfo: objectEntryToProtoListItem fills metadata on demand
Added includeMetadata parameter which represents if metadata should be included in response
by default true, in case of new uplink version - ObjectIncludes will be used instead.

Change-Id: I2f8d3b4cc354cd655f8093bbbebe0e3c2ae14e6f
2021-08-03 13:41:11 +00:00
Moby von Briesen
149f6f2626 satellite/payments: Implement coupon codes
Full path: satellite/{payments,console},web/satellite

* Adds the ability to apply coupon codes from the billing page in the
satellite UI.
* Flag for coupon code UI is split into two flags - one for the billing
page and one for the signup page. This commit implements the first, but
not the second.
* Update the Stripe dependency to v72, which is necessary to
use Stripe's promo code functionality.

Change-Id: I19d9815c48205932bef68d87d5cb0b000498fa70
2021-07-26 17:15:55 +00:00
Jeremy Wharton
dae6ed7d03 satellite/console: Implement MFA backend
Added MFA passcode and recovery code field for token requests.
Added endpoints for MFA-related activity: enabling MFA,
disabling MFA, generating a new MFA secret key, and
generating new MFA recovery codes.

Change-Id: Ia1443f05d3a2fecaa7f170f56d73c7a4e9b69ad5
2021-07-26 16:37:05 +00:00
Kaloyan Raev
a767aed591
satellite/{metabase,metainfo}: require StreamID for UpdateObjectMetadata
This avoids corrupting objects if reuploads and metadata updates happen
concurrently.
2021-07-08 17:50:37 +03:00
Kaloyan Raev
73b9223758 satellite/metainfo: implement UpdateObjectMetadata
Change-Id: I7d791bb84b73299d395b587074e721c14e4be31c
2021-07-07 12:57:32 +00:00
Jeff Wendling
f474bb6179 cmd/uplinkng: update for breaking clingy changes
clingy changed some and this is just fixes for that

Change-Id: I729aed6329fe0988fcb9b4407f16966a753b3204
2021-07-06 17:26:51 -04:00
Egon Elbre
3c8e7e679a mod: bump storj.io/private and monkit
This gives us nicer output for time.Time formatting when looking at
ongoing database queries.

Change-Id: I00c1ae4d162f537bd69fc2c8132c6507c5ee98cb
2021-06-25 16:57:00 +03:00
Egon Elbre
2ebbc9757f mod: bump storj.io/private
This makes it possible to see queries in mon/ps output.

Change-Id: I0e8b55bd9dbc3d23bc3cc41a71aeeb64d32ce46f
2021-06-15 22:20:23 +03:00
Egon Elbre
f1a9b45599 mod: update quic for Go 1.17rc1
Change-Id: I4218b695089189d5efac1101559a83079fb812cd
2021-06-14 10:14:36 +03:00
andriikotko
072dd24696
integration/ui: add UI integration tests
Introduce a new `integration/ui` package for integration tests.

`integration/ui/uitest` contains a helper package for running browser tests.
2021-06-09 14:39:38 +03:00
JT Olio
da9ca0c650 testplanet/satellite: reduce the number of places default values need to be configured
Satellites set their configuration values to default values using
cfgstruct, however, it turns out our tests don't test these values
at all! Instead, they have a completely separate definition system
that is easy to forget about.

As is to be expected, these values have drifted, and it appears
in a few cases test planet is testing unreasonable values that we
won't see in production, or perhaps worse, features enabled in
production were missed and weren't enabled in testplanet.

This change makes it so all values are configured the same,
systematic way, so it's easy to see when test values are different
than dev values or release values, and it's less hard to forget
to enable features in testplanet.

In terms of reviewing, this change should be actually fairly
easy to review, considering private/testplanet/satellite.go keeps
the current config system and the new one and confirms that they
result in identical configurations, so you can be certain that
nothing was missed and the config is all correct.
You can also check the config lock to see what actual config
values changed.

Change-Id: I6715d0794887f577e21742afcf56fd2b9d12170e
2021-06-01 22:14:17 +00:00
Egon Elbre
926bcd0f83 mod: bump private
Latest version contains a fix for pgxutil.Conn.

Change-Id: I200b1c11a1e536a57a7384c1a747a7216f8cd9ff
2021-05-25 15:28:16 +03:00
Egon Elbre
e7cf369cd8 satellite/metabase: use pgxutil.Conn helper
Clean up underlying pgx.Conn extraction with helper.

Change-Id: Ie43f663a6036c439e675df54fccd3605c4d8d69e
2021-05-25 05:56:35 +00:00
Egon Elbre
515f9b54ff mod: bump monkit
Change-Id: I9be05b91cd9171ebc7d053cd087d6761150d44b6
2021-05-24 16:20:00 +00:00
Jeff Wendling
a1bf9ab6de cmd/uplinkng: initial commit with skeleton
Change-Id: I764618cc60c46882955e9b08b72b3c162aa4929f
2021-05-18 10:04:44 -04:00