Commit Graph

1053 Commits

Author SHA1 Message Date
Ethan Adams
27c6c6aeae
ci: Golangci lint v1.43.0 changes (#4307)
Co-authored-by: Stefan Benten <mail@stefan-benten.de>
2021-12-09 20:49:48 +01:00
Qweder93
b5194762f2 cmd/uplinkng: share command added
In uplink we have command uplink share and we need
to port it to uplinkng to have command with same functionality.

Command could be executed with parameter or without.
without any flags we share as readonly.

Change-Id: I973b11d00da237358834acf5a863ebab37e684cc
2021-12-02 13:02:14 +00:00
crawter
eec5ad4f5d cmd/uplinkng: access inspect command added
In uplink we have command uplink access inspect and we need
a command with the same functionality for uplinkng.

Command could be executed with parameter or without.
without parameter - we should show default access.
If parameter exists - it could be access name or value.
If access name or value is wrong - we show error.
F.e.
uplinkng access inspect
uplinkng access inspect accessName
uplinkng access inspect accessValue
https://storjlabs.atlassian.net/browse/PG-318

Change-Id: I85fd961283850feb8684db2d126441f6b9bf0270
2021-11-30 12:45:04 +00:00
Michał Niewrzał
8f07abafe0 cmd/uplinkng/ultest: make possible to test access commands
Test external implementation doesn't support OpenAccess
method. This makes imposible to test output of commands like
inspect or share. This implements only basic functionality.

Change-Id: I127ef0bb45a01634bd5265ed80840f8095c72794
2021-11-22 10:43:02 +00:00
Michał Niewrzał
24cf7e8ea6 cmd/uplinkng: add mv command
Add ability to move files and objects.

Change-Id: I4929da730984c06aa578678b1d8c8e9b4aceade8
2021-11-22 09:07:24 +00:00
Mya
814e3126fa satellite/buckets: add new buckets service
The main motivation is to wrap the bucket DB and metainfo DB, so we
could check if a bucket is empty before applying geofencing config.

Change-Id: I8bac21555e01d51a663fb557bc1acfc8106bc2e1
2021-11-16 12:36:17 +02:00
Egon Elbre
4a530ccffd cmd/storagenode: simplify windows service loop
Remove some testing code that is not necessary for the service.

Change-Id: Idd09d62bf022d6e66943983f98642fc3c9aa72f7
2021-11-05 15:56:18 +00:00
Egon Elbre
e721081354 cmd/storagenode: fix service initialization code
func init() code isn't that well defined and reordering of them
could cause problems when starting the whole process from it.

Change-Id: I4088a0db156ece15354877011a481f6f91c9b332
2021-11-05 01:09:41 +02:00
Clement Sam
16a334020f cmd/uplinkng: add ranged download
This change adds the ability to download byte ranges
to uplinkng.

Extended the uplinkng Filesystem interface with Stat
method and an OpenOptions struct as parameter for the
Open method.

Also added a few tests for the ranged download

Change-Id: I89a7276a75c51a4b22d7a450f15b3eb18ba838d4
2021-11-03 19:51:25 +00:00
Egon Elbre
6a69c66477 cmd/storj-sim: support printing node url
Add support for node url-s with PEER_URL to help configuring
authservice and gateway.

   storj-sim network env SATELLITE_0_URL

Change-Id: I8b595b398007730662f100a4e6b3529cc0a7512a
2021-11-03 16:11:41 +02:00
Jeff Wendling
bd2448bc4d cmd/uplinkng: expand version information
This includes the information that used to exist in uplink
as well as all of the module versions.

Change-Id: Ie6d6fa12da9c04b50611e5885e05ab0c24a2ec0d
2021-11-03 09:26:56 +00:00
Egon Elbre
431f55d53b ci: cross-compile storagenode-updater
Change-Id: I1db757b64e61f185ba710bf669f2e62441b9ab71
2021-11-01 11:21:06 +00:00
Jeff Wendling
4d023f773f cmd/storagenode-updater: update to correct version
Currently if the nodes is below the minimum version it will immediately
update to the suggested version, regardless if its eligible from the seed
or not. This change corrects the behaviour to update to the minimum
version only and then properly respect/wait for the rollout to include it.

Updates based on logic here: https://review.dev.storj.io/c/storj/private/+/6187

Change-Id: Ic6c91c48ae9b8a116378b2573fbfca7e7bd5cc32
2021-10-29 20:38:30 +00:00
Jeff Wendling
f94d8a200c cmd/uplinkng: mkdir config file folder
If the directory doesn't exist, then the first run in a non-migration
setting will error because it cannot create the config file. This
change creates the directory.

Change-Id: I439159c00047e9ae20e139318dad5a047c853253
2021-10-29 11:07:55 +00:00
mya
9749f9756c
cmd/uplinkng: port expanded option to show additional metadata (#4229) 2021-10-28 11:51:33 -05:00
Yingrong Zhao
d441c8da15 satellite: use segment count for billing
Change-Id: Iafaf41cb6cde4cb2abdac3f75ce0fead838cb758
2021-10-27 19:21:50 +00:00
paul cannon
1951450c57 cmd/satellite: add register-lost-segments command
Usage: from a host for the affected satellite, issue this command
and supply the number of segments that have been permanently lost. For
example, to indicate 2 segments permanently lost:

    satellite register-lost-segments 2

If the unthinkable happens and this is necessary to use, it is presumed
that we will also remove the non-recoverable segments from the metainfo
db, so they will not continue to appear as 'temporarily unavailable' to
the repair checker.

The influxQL query for this will probably look something like:

    SELECT sum(total) AS "sum_total"
    FROM "v3_stats_new"."autogen"."lost-segments"
    WHERE time > :dashboardTime: AND "scope" = 'segment-durability'
    GROUP BY time(:interval:), "application", "instance"
    FILL(null)

Or use the Flux language in order to get the benefit of the
cumulativeSum function:

    from(bucket: "v3_stats_new/autogen")
      |> range(start: dashboardTime)
      |> filter(fn: (r) => r._measurement == "lost-segments"
                           and r.scope == "segment-durability"
                           and r._field == "total")
      |> group(columns: ["application", "instance"])
      |> cumulativeSum()

Change-Id: I73d364937705aa815af7520ab79c00bc2aea09f6
2021-10-26 15:03:52 +00:00
Clement Sam
85b49bb27c cmd/uplink: add ranged download to uplink cli
Change-Id: Ib274df024a8ffc5db2d5c99f8f363efa3b43723f
2021-10-25 13:13:33 +00:00
Qweder93
50baefa10e satellite/metabase: limit maximum number of parts and size
Multipart upload limits added. Last part has no size limit.
Max number of parts: 10000, min part size: 5 MiB

Change-Id: Ic2262ce25f989b34d92f662bde720d4c4d0dc93d
2021-10-22 10:16:37 +00:00
crawter
9ed826eabb cmd/uplink/mv: moving files between folders
https://storjlabs.atlassian.net/browse/PG-305

we should extend method move of cmd/uplink/cmd/mv.go
if both parameters end with slash -
i should list all files and call move method in loop

parameters:
uplink mv sj://bucket/a/prefix/ sj://new-bucket/a/new-prefix/

Change-Id: Ic24c2af83153ea60ec74393e65736af094877151
2021-10-20 18:16:08 +00:00
Jeff Wendling
987cb6ab11 cmd/uplinkng: allow removing pending objects
Change-Id: I12457e7d6fb28492ed4b6c5816f78aa7820fed6b
2021-10-20 12:25:25 -04:00
dlamarmorgan
4bbf667ad1 satellite/{satellitedb,attribution,console}: value attribution changes that add userAgent field to buckets table and all tables that have partner_id
Change-Id: I36a13bb651b86bfc14fe5a0a2258f719e6cd2b48
2021-10-18 13:56:19 +00:00
Egon Elbre
c54bcd63a1 Makefile: bump node to v16
Change-Id: I0587168f373230710c7b50b9aefd34718f4a5b3e
2021-10-14 12:46:28 +00:00
Jeremy Wharton
3b751a35c5 satellite/{payments,satellitedb}: Remove custom coupon implementation
Removes database tables and functionality related to our custom
coupon implementation because it has been superseded by the Stripe
coupon and promo code system. Requires implementations of the
payments Invoices interface to return coupon usages along with
invoices.

Change-Id: Iac52d2ff64afca8cc4dbb2d1f20e6ad4b39ddfde
2021-10-11 19:47:00 +00:00
Michał Niewrzał
0209bc6ef7 cmd/uplink: add mv command
New command for cli to move object to different
location.

uplink mv sj://bucket/your-object sj://bucket/moved-object

Change-Id: I85a4961aa59f250819954e78f20363ac3c570938
2021-09-29 19:10:58 +00:00
Michał Niewrzał
5b66136312 cmd/uplinkng: fix mb command
Make bucket command was using full location
specified in command line instead only bucket name.

As an addition change contains basic integration tests
with storj-sim.

Change-Id: Ie3b5283468b7fbde0b1333f01dc4fc2a2952e1a1
2021-09-29 15:22:03 +00:00
Yaroslav Vorobiov
469ae72c19 satellite/repair: update audit records during repair
Change-Id: I788b2096968f043601aba6502a2e4e784f1f02a0
2021-09-24 00:48:13 +00:00
Egon Elbre
b2d724962c cmd/storagenode-updater: avoid depending on the storagenode code
Change-Id: If81927aee43ccdd678a0e42f91632f85e9fdefab
2021-09-16 11:44:55 +00:00
Egon Elbre
1aec831d98 satellite/audit,storage: increase sleep delay in TestMaxVerifyCount
Currently TextMaxVerifyCount flakes in some tests, try increasing the
sleep time to ensure that things are slow enough to trigger the error
condition.

Also pass ctx to all the funcs so we can handle sleep better.

Change-Id: I605b6ea8b14a0a66d81a605ce3251f57a1669c00
2021-09-10 15:30:37 +00:00
Michał Niewrzał
03d638bbb3 cmd/uplink: add parallel upload
Use multipart upload to upload single object in parts in
parallel. Its using parallelism flag added earlier.

Change-Id: I45b531a5db43c86f0112a5e3bb4a83bc1d65650f
2021-09-07 15:12:09 +00:00
Jeremy Wharton
eb11899acb cmd/satellite: Remove billing 'check-paid-tier' command
Now that the command has been run on all production satellites
(US1, AP1, EU1), we should not need it again.

Change-Id: I25a4ffb03a7172445d90a04ec539be36c4eb2c8e
2021-08-26 22:02:16 +00:00
Jeff Wendling
58bd85cbf4 uplinkng: some windows test fixes
Change-Id: Idbe66028ac77d45eff9ea1926e6e8bd66491ea19
2021-08-26 17:52:13 +00:00
Michał Niewrzał
211a630982 cmd/uplink: add parallelism flag for single object download
Adds support for new uplink method DownloadObjectAt which
gives ability to download single object in parallel.

Change-Id: I8388653429992b0d24c383d17d7e90904203fe77
2021-08-26 13:13:38 +00:00
Jeff Wendling
2782e000ac cmd/uplinkng: parallel rm
Change-Id: I4f911ce1f384f13812986a34b96a94ded70cfb4e
2021-08-19 17:48:53 +00:00
Jeff Wendling
d57583dfd4 cmd/uplinkng: initial setup
Change-Id: If4df3ec8b3b554f5228d43e97503eb8a87525b23
2021-08-19 17:48:49 +00:00
Jeff Wendling
7eaa7490a1 cmd/uplinkng: secret prompting
Change-Id: I1b407b59559281c2242b55ebcf835d465db38e1d
2021-08-19 17:48:45 +00:00
Moby von Briesen
ef9a5210a4 satellite/payments: Add command to ensure free tier coupons
This command is intended to be run as part of invoice generation - it
iterates over Stripe customers, and applies the free tier coupon to any
customer who doesn't already have a coupon.

This way, we can ensure that all customers have at least the free tier
coupon before and after invoice generation, in case a different coupon
has expired.

Change-Id: I33a4aff9174049f9e051de53ef65298ca65ed688
2021-08-18 13:48:44 +00:00
Artur M. Wolff
2cd68bf4fb private/lrucache: import from common
Change-Id: Ia1f43d0440fef21122b071b05da59b4cf2689d6c
2021-08-16 10:04:32 +00:00
Jeff Wendling
59b8422318 cmd/uplinkng: implement revoke and normalize some language
this makes the distinction between an "access name" and
an "access value" and talks about which is expected for
commands. most are "access name or value".

Change-Id: I43c0043a17d37e89ab5f87388ae9e890a8b59958
2021-08-11 02:30:13 +00:00
Jeff Wendling
08d860570b cmd/uplinkng: parallelsm and a ton of fixes
this was just supposed to add parallel uploads/downloads
and it does do that, but i then found a bunch of bugs
with respect to path handling that i thought i had under
control. oops.

so this adds a ton of tests and tries to make the logic
in ulloc to be more consistent. almost all of the actual
file handling bits and knowledge happens in cmd_cp now
where it should belong.

additionally, the s3 command has the behavior that if your
bucket has the file s3://bucket/file, then executing
s3 ls s3://bucket/fi returns nothing. this change makes
uplinkng match that behavior even if i don't personally
like it.

a big portion of the weirdness is the concept introduced
that i've named "directoryish", which intends to capture
the behavior that if a user copies a file to that location
then the base name of the source should be appended on
rather than a direct copy. this concept is entirely a
based on the string value and not the actual filesystem
state. hence, the cp command is responsible for checking
if local paths are actually a directory, and adding a
trailing slash if necessary to make them "directoryish".
additionally, the empty key for a bucket and the empty
string for local paths are considered "directoryish".

Change-Id: I9120d18616fd813b29ff81beed4f5993caa99fb6
2021-08-11 02:30:06 +00:00
Jeff Wendling
e33f8d7170 cmd/uplinkng: access creation/restriction and review fixes
Change-Id: I649ae3615363685c28c39d1efb6a65fcad507f46
2021-08-11 02:29:49 +00:00
Yingrong Zhao
077ec96d94 private/server: use quic implementation from storj.io/common
Change-Id: I820cf6444a3ddccee0d7c647dc84c80b2752068c
2021-08-10 13:32:21 +00:00
Clement Sam
1f353f3231 segment/{metabase,repair}: change segment created_at column to not accept nulls
This change adds a NOT NULL constraint to the created_at column in the segment table.
All occurrences of CreatedAt as a pointer are changed to non pointer version (metabase, segment loop, etc)

Change-Id: I3efd476ebd1edd3327b69c9223d9edc800e1cc52
2021-08-06 08:16:28 +00:00
Michał Niewrzał
55d7bcc59b satellite/metabase/segmentloop: don't shutdown satellite on loop error
We made decision to avoid satellite shutdown when segment loop
will return error. Loop still can reeturn error but it will be logged
and we will make monitoring/alert around that error.

Change-Id: I6aa8e284406edf644a09d6b1fe00c3155c5430c9
2021-07-30 06:49:10 +00:00
Yingrong Zhao
f8914ccce0 satellite/{repair, overlay}: use reputation store in repair
Change-Id: I48db9e68f48239d48621ccc77d33618ecb83ce1a
2021-07-28 13:22:05 -04:00
Moby von Briesen
8f072bdeee cmd/satellite: Skip non-existing users in paid tier conversion
There are some users in our QA satellite which are no longer in Stripe,
and there are some users in Stripe which are not on our QA satellite.
This change allows us to test the paid tier conversion script in QA
despite these problems.

Change-Id: If94c9e882327841d1fd294d75fd302e6a7feee41
2021-07-27 12:53:58 -04:00
Michał Niewrzał
420d2f6275 metabase-orphaned-segments: avoid processing recently created elements
To be sure we are comparing the same set of objects
and segments lets ignore objects and segments created
after processing was started.
Segmets without objects cannot be created in normal
way so we will have them only if we broke something in
the past.

Change-Id: I96c07caf9e5091775d4dc8dfc0fef2b08b87957c
2021-07-26 15:50:20 +00:00
Michał Niewrzał
b12d29935a satellite/metabase: remove metaloop package
We moved everything to segment loop so we can now
remove metaloop from code.

Change-Id: I9bd8d2349e5638d7cdad50f2f313f9bd89a8165c
2021-07-22 13:00:45 +00:00
Michał Niewrzał
237360e765 metabase-orphaned-segments: fix args binding for delete command
As an additon I clarify log message a little bit.

Change-Id: Id2487cc218e3dd62ad5a05b56e8b1d5d070676a7
2021-07-22 11:00:37 +02:00
Michał Niewrzał
f73fcee892 cmd: add tool for orphaned segments
This tool has two commands to execute. One is to 'report'
orphaned segments. Second to 'delete' orphaned
segments.

To find such segments tool is first finding all unique
segment stream ids. As a next step its removing from this
list stream ids of existing objects. What is left is a list of
orphaned segments.

Change-Id: I4a0ae3ad0b10a8d16572bfd22ac92cfa15ca19b3
2021-07-21 10:10:58 +00:00