Commit Graph

248 Commits

Author SHA1 Message Date
Qweder93
2a7b20e8e4 cmd/uplink: integrate server-side copy with uplink cp command
Resolves https://github.com/storj/storj/issues/4486

Change-Id: I42ac2ad2e1a05df4a83606f1990b639f08791403
2022-03-31 09:25:29 +00:00
Egon Elbre
e5972d8920 all: fix linting errors
strings.Title is deprecated in Go 1.18, replace it with
golang.org/x/text/cases.

Change-Id: I6185b97d37309dbe4a6715f794383ab259a0658b
2022-03-30 18:45:23 +03:00
Erik van Velzen
85fa78eae7 cmd/uplink: supporty expires in copy
When copying an object from cli you can now set the expiry.
It uses the same datetime format as restricting access grants.

Closes https://github.com/storj/storj/issues/4595

Change-Id: Icab73a64a9589817d6bc6d702b765b166ca1350d
2022-03-07 02:43:51 +01:00
Erik van Velzen
b4e42ceb23 cmd/uplinkng: fix linkshare bugs
Fix various bugs by remove superflous steps and letting libuplink
build the url.

$ uplinkng share --not-after +1h --url sj://mybucket/myprefix/

before: https://link.us1.storjshare.io/s/jxcvcme2xkb44xsec235xb2ccmbq//myprefix//
after: https://link.us1.storjshare.io/s/jxcvcme2xkb44xsec235xb2ccmbq/mybucket/myprefix/

$ uplinkng share --not-after +1h --url $(printf "sj://waterbear/aa\x03bb")

before: https://link.us1.storjshare.io/s/jwbcxjiv4ept7t2g3qmxfb73ljga//aabb
after: https://link.us1.storjshare.io/s/jvod7pixige62yrp5tubn4ct47yq/waterbear/aa%03bb

$ uplinkng share --not-after +1h --url

before: no url generated, no error
after: error "Need at least a bucket to create a working linkshare URL"
Change-Id: Ibca6847f8fc5d5fad9ee4642d11d3081d75d77ca
2022-03-03 16:00:10 +00:00
Jeff Wendling
e2e5882c86 cmd/uplink: fix migration for some old configs
some old configs had a value like

	access: <data>

in the yaml. this would end up causing migration to
create a json file where it had no access values and
a default name of the data. that's not what the command
expects to operate on, so now we fix that during
migration and add a little mini migration for any
users that may have hit it.

Change-Id: I4c98ca5d09d043fe9338738ef6b4f930f933892c
2022-02-16 21:13:52 +00:00
Erik van Velzen
0796653b07 cmd/uplinkng: registeraccess via libuplink
Change the implementation of register and share so that it uses the
uplink method to contact the Auth Service. The network protocol switches
from HTTP to DRPC.

Closes https://github.com/storj/storj/issues/4324

Change-Id: Ib8fdb1665c6385bb39a546ba46a8df43a136df9c
2022-02-11 11:36:45 +00:00
Jeff Wendling
9061dd309f cmd/uplinkng: become cmd/uplink
Change-Id: If426c32219d32044d715ab6dfa9718807f32cb9f
2022-02-09 17:02:21 +00:00
Egon Elbre
64c8de6ea5 mod: use vendored base58
Change-Id: I5aa29515928848c862500330218cc094618638d7
2022-01-31 15:54:33 +02:00
Erik van Velzen
4f96a85642 cmd/uplink/share: register access via DRPC
Convert registering access with the edge services
from the HTTP protocol to DRPC protocol

Change-Id: Iba88dd0758c26f613cf501be9a20ead07d122d0b
2022-01-13 13:24:05 +00:00
Michał Niewrzał
75acd6109d cmd/{uplink,uplinkng}: set user agent for uplink binaries
We want to monitor traffic and tools that are used
to interact with our network so we need to append
its user agent.
The same user agent is appended for current uplink
and uplinkng as eventually we will remove first.

Change-Id: I116080d6c2c6c85d591771facf01356de02a9392
2021-12-14 13:00:01 +00:00
Qweder93
b5194762f2 cmd/uplinkng: share command added
In uplink we have command uplink share and we need
to port it to uplinkng to have command with same functionality.

Command could be executed with parameter or without.
without any flags we share as readonly.

Change-Id: I973b11d00da237358834acf5a863ebab37e684cc
2021-12-02 13:02:14 +00:00
crawter
eec5ad4f5d cmd/uplinkng: access inspect command added
In uplink we have command uplink access inspect and we need
a command with the same functionality for uplinkng.

Command could be executed with parameter or without.
without parameter - we should show default access.
If parameter exists - it could be access name or value.
If access name or value is wrong - we show error.
F.e.
uplinkng access inspect
uplinkng access inspect accessName
uplinkng access inspect accessValue
https://storjlabs.atlassian.net/browse/PG-318

Change-Id: I85fd961283850feb8684db2d126441f6b9bf0270
2021-11-30 12:45:04 +00:00
Clement Sam
85b49bb27c cmd/uplink: add ranged download to uplink cli
Change-Id: Ib274df024a8ffc5db2d5c99f8f363efa3b43723f
2021-10-25 13:13:33 +00:00
crawter
9ed826eabb cmd/uplink/mv: moving files between folders
https://storjlabs.atlassian.net/browse/PG-305

we should extend method move of cmd/uplink/cmd/mv.go
if both parameters end with slash -
i should list all files and call move method in loop

parameters:
uplink mv sj://bucket/a/prefix/ sj://new-bucket/a/new-prefix/

Change-Id: Ic24c2af83153ea60ec74393e65736af094877151
2021-10-20 18:16:08 +00:00
Michał Niewrzał
0209bc6ef7 cmd/uplink: add mv command
New command for cli to move object to different
location.

uplink mv sj://bucket/your-object sj://bucket/moved-object

Change-Id: I85a4961aa59f250819954e78f20363ac3c570938
2021-09-29 19:10:58 +00:00
Michał Niewrzał
03d638bbb3 cmd/uplink: add parallel upload
Use multipart upload to upload single object in parts in
parallel. Its using parallelism flag added earlier.

Change-Id: I45b531a5db43c86f0112a5e3bb4a83bc1d65650f
2021-09-07 15:12:09 +00:00
Michał Niewrzał
211a630982 cmd/uplink: add parallelism flag for single object download
Adds support for new uplink method DownloadObjectAt which
gives ability to download single object in parallel.

Change-Id: I8388653429992b0d24c383d17d7e90904203fe77
2021-08-26 13:13:38 +00:00
Yingrong Zhao
077ec96d94 private/server: use quic implementation from storj.io/common
Change-Id: I820cf6444a3ddccee0d7c647dc84c80b2752068c
2021-08-10 13:32:21 +00:00
Bill Thorp
18c3252025 cmd/uplink: recommend notAfter expiration caveat be set
The auth service has no way to remove access grant registrations that
lack expiration dates.  We want to encourage people to set them, so as
to slow the rate at which the auth service DB fills up.

Change-Id: I1ccf629cd995dc184d2d90333166eab34d34ae07
2021-07-14 17:05:28 +00:00
Jeff Wendling
46a3242ed4 cmd/uplinkng: cp
Change-Id: I9c251028d9f72572f4d42815de31d44517cd5525
2021-06-02 14:35:27 +00:00
paul cannon
8675cd178c cmd/uplink: add flags to 'put' subcommand
This enables use of the --progress, --expires, and --metadata flags with
'uplink put'. These flags work similarly to their counterparts in the
'uplink cp' command.

Small difference from cp: "progress" is on by default in 'uplink cp',
but (for backwards compatibility) off by default in 'uplink put'.

Why: Requested by @Toyoo on the forum:
https://forum.storj.io/t/explicit-data-expiration-questions/13854/2

..and it's an easy addition to make.

Change-Id: Id20aedd3a280db43e4883338f92f6beec7a400de
2021-06-01 19:16:22 +00:00
paul cannon
fbd11ba4e9 cmd/uplink: add expanded format for 'uplink ls'
This expanded format shows expiration times for objects and how much
custom metadata each object has.

This commit also organizes output formatting code together in one
section for simpler adjustments in the future.

Change-Id: Ica041c8a1de6ee73c104a0554c5c259e447536c4
2021-06-01 18:50:27 +00:00
Egon Elbre
10372afbe4 ci: fix lint errors
Change-Id: Ib5893440807811f77175ccd347aa3f8ca9cccbdf
2021-05-17 13:37:31 +00:00
Michał Niewrzał
7944df20d6 storj: use multipart API
Change-Id: I10b401434e3e77468d12ecd225b41689568fd197
2021-04-26 13:15:09 +00:00
Egon Elbre
7802ab714f pkg/,private/: merge with private package
Initially there were pkg and private packages, however for all practical
purposes there's no significant difference between them. It's clearer to
have a single private package - and when we do get a specific
abstraction that needs to be reused, we can move it to storj.io/common
or storj.io/private.

Change-Id: Ibc2036e67f312f5d63cb4a97f5a92e38ae413aa5
2021-04-23 16:37:28 +03:00
Egon Elbre
75ca01e381 cmd/uplink: fix importing access in Windows
Currently os.Create was leaving a file open causing atomic write file to
fail with access denied.

Also add a specific test for importing an access.

Change-Id: Id188bc480e795849ec7fdc72b1fc86433d76c47a
2021-04-19 15:11:39 +03:00
JT Olio
f3c8cac1ee storj/storj: more domain changes
Change-Id: I91ae3bc36be38c27269f9e6e678fa3616b8e4d53
2021-04-14 20:49:12 +00:00
JT Olio
afcc55fb47 storj/storj: some domain changes we can make now
Change-Id: I5622edea9dfcad788cac65e1754b331b9fed6154
2021-04-13 17:12:06 +00:00
JT Olio
2ddbaf1eb5 satellite/wasm: support restricting full access grants to paths
Change-Id: Id6d4fa41db068d32e7c0d542d9d8805fba927fc6
2021-03-26 19:49:21 +00:00
Michał Niewrzał
237782813b Merge remote-tracking branch 'origin/multipart-upload'
Change-Id: If6c5a450b238adab55d1e0dea67d01e5f5768a9f
2021-03-23 09:44:49 +01:00
Yingrong Zhao
bf4fdc7717 cmd/uplink: add --client.enable-quic flag
Change-Id: I19e81ecb98688fd0a36174fd06c363e9c40b3866
2021-03-19 01:24:16 +00:00
Fadila Khadar
cf0c88c679 cmd/uplink: fix no result when listing single file
uplink ls sj://bucket/file.ext gives a result if file.ext exists.

Change-Id: Ife741926acacf510e4e098009ce1724386c7522f
2021-02-26 11:43:02 +00:00
Fadila Khadar
8bbabe57db cmd/uplink: allow deletion of pending objects
Basic command for deleting a pending object by object key.

Change-Id: I47f68acf79cb0568a42a65b6d8195ca309b185a7
2021-02-25 14:23:56 +00:00
Fadila Khadar
e96ed289e5 cmd/uplink: add --pending for listing pending objects
Change-Id: I19c1965117c386f63e7b1c71ba70402d312329f0
2021-02-25 13:50:10 +00:00
JT Olio
e23f423107 uplink: add api key back to inspect
Change-Id: I9aa46f3910a76a81a399597d7512056b218e55af
2021-02-19 14:46:35 +00:00
JT Olio
1ac17cfac5 uplink: link.tardigradeshare.io URLs should have an /s/ right after the domain
should be
https://link.tardigradeshare.io/s/<access>/<bucket>/<path>

legacy URLs have the /s/ missing but a redirect is issued

Change-Id: Ic2a3dc092ff68d7706fd888a9fbfc27716877c6e
2021-02-09 14:38:22 -07:00
Caleb Case
22bc69ad6d
cmd/uplink: Add more details to access inspect
Change-Id: Ia7e600c214d6097b48b9c868532b8fbe21b0035c
2021-02-02 10:25:26 -05:00
JT Olio
6e2450ecf7 uplink cli: make uplink share --register output access keys again
this used to work prior to https://review.dev.storj.io/c/storj/storj/+/3466

Change-Id: I9f02410f0e3768f0df0c5309c3107609fcbf7742
2021-02-01 15:50:02 +00:00
Caleb Case
d2148edcdb
cmd/uplink/cmd/setup.go: Access Grant vs API Key error messaging
Provide a clearer error message to users who confuse the API Key with
the Access Grant and suggest the right command to them.

Change-Id: If73ae8cde140b68a19f4cfc3f59bb88a3b74c9c1
2021-01-26 14:09:59 -05:00
Isaac Hess
1e0d57bf01 uplink: Add timeout to RegisterAccess
We add a timeout for the http client used to register the access with
the auth service. We have a hard-coded common default for now.

Change-Id: I50207ad83c9221b7cb61f39310e24b140b95673b
2021-01-19 20:14:58 +00:00
Bill Thorp
c0bc2123f5 uplink/cmd: fix uplink access register ability to use saved accesses
Jens noticed that 'uplink access register' wasn't working with named
accesses.

This was because GetNamedAccess was harcoded to use inspectCfg, which
in the case of 'uplink access register' wasn't being bound to the
config file.

Change-Id: I49403b45af28ad33408cfc5ec6545a395f0f080d
2021-01-06 15:08:51 +00:00
Moby von Briesen
641f01ed42 cmd/uplink: Actually overwrite accesses in uplink import
Previously, we were trying to overwrite accesses, which is a nested map
in the uplink config, by calling viper.MergeWithConfig with a nested
map. While this works for keys that don't exist already, it does not
overwrite already existing keys. In order to do that, we need to call
MergeWithConfig with "accesses.<accessname> -> value" rather than using
"accesses -> <accessname> -> value".

Change-Id: I74d7a9decf2078cdf2ff440eaf24821e30474b53
2020-12-30 15:25:53 +00:00
Bill Thorp
f65a1e8c1c uplink: use output formats for 'uplink access register' instead of writing to AWS file
The current uplink access register method has the ability to write to AWS credential files.
This is caused issues with repeat usage in recent AWS CLI code, and there was concern that
it was an unstable solution.  This version instead specifies various output formats "env"
and "aws". "env" formats the text so that it can be used with 'export'.  "Aws" generates
"aws configure" commands to persist the credentials to the AWS credential files as the
previous version could.

Example usages:

Setting ephemeral evironment variables in bash:
export $(uplink access register $(storj-sim network env GATEWAY_0_ACCESS)
--auth-service http://localhost:8000 --format env)

Setting persistant configs via AWS CLI in bash:
source <(uplink access register $(storj-sim network env GATEWAY_0_ACCESS)
--auth-service http://localhost:8000 --format aws --aws-profile storjsim)

Change-Id: I5d78d6462a3537780af3717a298bb2bebf9c2799
2020-12-30 13:34:01 +00:00
Jennifer Johnson
7cccbdb766 cmd/uplink/share: add -- to flags referenced in help statements
Change-Id: I86cd08d51c5306effec14f338f37c53c2743d6b2
2020-12-22 17:03:00 +00:00
Bill Thorp
1b0424cad6 uplink/cmd: Export a RegisterAccess method.
Gateway-MT requires integration tests, which would be aided by having an
exported RegisterAccess() method in uplink/cmd.

To support this change, a little of the Uplink cmd logic was shifted around
and a method was made public.  I also normalized finding the access
between accessInspect and accessRegister.

Change-Id: I29369296521c2cc179e27233f5451b95f46109d8
2020-12-22 14:52:40 +00:00
Stefan Benten
7eab859030
cmd: ensure proper arch is used for docker container 2020-12-20 09:26:23 +02:00
Egon Elbre
12055e7864 all: minor cleanups
Change-Id: I4248dbe36a62a223b06135254b32851485a2eec1
2020-12-16 10:47:46 +00:00
Jennifer Johnson
adb2c83e09 cmd/uplink: adds register, url, and dns flags to uplink share
and replaces access grant with access

uplink share <path> --> creates access grant

uplink share --register <path> --> registers access grant

uplink share --url <path> --> creates URL, implies register and public

uplink share --dns <hostname> <path> --> creates dns info, implies register and public

Change-Id: I7930c4973a602d3d721ec6f77170f90957dad8c0
2020-12-14 20:51:44 -05:00
Stefan Benten
3c77825b19 cmd: move ca-certificates build step to be arch independent
It turns out, that running a docker image build for specific
arches is not possible from amd64 (eg. installing ca-certificates).

Change-Id: I8b8f002b7e532fb4a0c6542d5b573c294c501068
2020-12-08 21:49:41 +01:00
Stefan Benten
a71c908c1e
cmd: add ca-certificates to Docker images (#3986) 2020-12-08 01:38:33 +01:00