With this change users can use the uplink cli in
scripts (ie. bash) more easily, since the output
can be switched to an easier processable json format.
It keeps the default of tabbed output.
Change-Id: I37e2c55f75c2250c3119fd8df8b66a766ff9096b
When ctx is cancelled limiter won't start a new goroutine.
The code didn't immediately return an error in that case.
The dst.Commit(ctx) would fail anyways due to a cancelled ctx.
However, we can make the behavior clearer by returning immediately.
Change-Id: I65df7ca85de55813f3200a50db2eaaa7a297ba2c
It was possible for the a previous write / part to fail or be aborted
and the next part write still happened. This causes a data ordering
corruption.
The whole write to parallel stdout fails, so there shouldn't be
confusion with regards to the output acceptability. However, it would
be clearer, if we avoided writing out-of-order data... mainly to be
clear that we didn't corrupt the data, just that it's incomplete.
Change-Id: I97b0d14404f29e8615e7d29b10cbd61ccb861e40
Also ensure that abort is given at least 5 seconds to clear up any
pending uploads on cancellation.
Change-Id: I814aa407ee5783f2609a76b54de2879dcd5f89bb
If the cp command is executed with higher level of parallelism, it would
open more connections to storage nodes at the same time. Therefore, the
connection pool capacity should be expanded accordingly.
The pool capacity is set to 100 * parallelism.
Change-Id: Ia8b3ab6a99340d8cbb87a7b80c3354b2b21c1958
I don't think it should matter for correctness whether this matches the
segment size or not, so I think there is something else wrong. However,
making this change seems to eliminate the "corruption when ulimit -n is
too low" problem we're seeing right now.
Change-Id: I232fe0d0a371b86ddf902e8c2d4778e140b2f1fc
Attribution is attached to bucket usage, but that's more granular than
necessary for the attribution report. This change iterates over the
bucket attributions, parses the user agent, converts the first entry
to lower case, and uses that as the key to a map which holds the
attribution totals for each unique user agent.
Change-Id: Ib2962ba0f57daa8a7298f11fcb1ac44a8bb97875
Now that we have both the storagenode and updater processes running
in a single docker container, we need a way to know which log entry
is logged by any of the processes.
This change includes a Process field in the log entries.
Resolves https://github.com/storj/storj/issues/4648
Change-Id: I167b9ab65728a41136d264b5fe2c41bb64ed1785
Before, the VA query was summing the total and dividing by the number of
rows. This gives the average bytes stored per hour, but we charge for
usage with byte-hours. Why not do value attribution the same way?
To do that, we don't divide by the number of rows. We also have object
and segment fees so return segment-hours and object-hours too.
Change-Id: I1f18b7e1b2bae1d3fae1ca3b93bfc24db5b9b0e6
We've had a lot of issues with alpine and currently there's a broken
network issue on alpine for users running on RPI arm32 architechture
which requires a workaround before docker is able to sync time between
the host and the container: https://wiki.alpinelinux.org/wiki/Release_Notes_for_Alpine_3.13.0\#time64_requirements.
Since we're switching the base image of the storagenode to debian,
it's best to switch the base image of all our docker images to
debian as well for consistency; less drift across them and keeps
the push target consistent.
Change-Id: If3adf7a57dc59f19ef2221b892f340d919798fc5
In the migration to migrate the corresponding name of the partner id to
user agent, part of the requirement was to migrate the partner id
itself if there was no partner name associated. This turned out to not
be so good. When we parse the user_agent column later, it is returning an
error if the user agent is one of these UUIDs.
Change-Id: I776ea458b82e1f99345005e5ba73d92264297bec
We are switching from alpine to debian due to a network issue
introduced in alpine 3.13+ which fails to verify certificates
due to not all armhf boards meet the time64 requirement:
https://wiki.alpinelinux.org/wiki/Release_Notes_for_Alpine_3.13.0\#time64_requirements
Also, Debian does not have official imagess for arm32v6 architecture
so we are building with arm32v5 arch in the Makefile.
Change-Id: I3660c3f64b7c2b342dd4ccb876af5f4e3036ea9d
When there is an error fetching a piece, the reader might be present or
it might not, depending on how far the fetch operation got. The
fetch-pieces code did not handle the "reader-not-present" case. Now it
should.
Change-Id: I263657d544d0ab8ba5d307a34ffc76bbf56835d0
Updating the version of the base image for the storagenode docker image.
Also fixes the non-root permission issue to /app directory
Change-Id: I8b55a1e3062f55ce6fc52e126ec1a18bfa24e669
This change fixes the following issues:
wget: Alpine docker image by default uses the builtin BusyBox wget which is not capable of handling SSL traffic via proxy unlike the GNU wget. We have to replace BusyBox wget with GNU wget.
updater failing to restart the node: supervisorctl pointing to wrong config file. We remove the default configuration file and point supervisorctl to custom config in systemctl
updates https://github.com/storj/storj/issues/4489
Change-Id: I24a7f18377ba723bbc377bb5d25aaa14f37021b1
Add ability to limit updates in migrations.
To make sure things are looking okay in the migration, we can run it
with a limit of something like 10 or 30. We can look at the output of
the migrated columns to see if they are correct. This should have no
effect on subsequently running the full migration.
Change-Id: I2c74879c8909c7938f994e1bd972d19325bc01f0
This change fixes the `sed: can't create temp file '/etc/supervisor/supervisord.confXXXXXX': Permission denied` issue when editing the supervisord.conf file during runtime as a non-root user.
While editing the config file, Sed creates a temporary file, saves the result and then finally mv the original file with the temporary one. So we need to set the permission for the /etc/supervisor where the temporary file is created.
Change-Id: Ic9c147a9cf0a6ef94adf702e33054edce1828806
The supervisord.conf file is edited to set the args for the storagenode and storagenode-updater binaries at runtime. This change moves the config file to the base image so we can set the permission to allow non-root users edit the config file.
Non-root user permission is also needed for the /app directory so we can install/update the binaries when run as a non-root user.
Updates https://github.com/storj/storj/issues/4489
Change-Id: If7a51a00ea171253e41923501174a43393f4638c
When copying an object from cli you can now set the expiry.
It uses the same datetime format as restricting access grants.
Closes https://github.com/storj/storj/issues/4595
Change-Id: Icab73a64a9589817d6bc6d702b765b166ca1350d
Having the storagenode and storagenode-updater processes in one container
requires a process manager to properly handle the individual processes.
Using a process manager like supervisord requires that you package
supervisord and it configuration in the image, along with the storagenode
and storagenode-updater binaries.
Installing supervisord requires that we run apk to install it and its
dependencies at build time which makes it difficult to build multi-platoform
images; executing apk forces a requirement of the build system to run
foreign architechtures.
This change adds a dockerfile which will be used to build the base image
for the storagenode and has supervisord packaged. The base image will be
built manually using docker buildx, with QEMU binfmt support.
Updates https://github.com/storj/storj/issues/4489
Change-Id: I33f8f01398a7207bca08d8a4a43f4ed56b6a2473
We would like to disable in production those parts of code
which are now mixed with new server-side copy logic.
Change-Id: Iff50682bc9545207330f58dd19b5eee53d404d7f
The text has been expanded a bit to clarify that it is necessary to create identity files with an example before using the Docker image.
Changed the <identity-dir> placeholder to <multinode-identity-dir> so no one confuses them with the storagenode identity files.
Changed the <storage-dir> placeholder to <multinode-config-dir> so no one confuses them with the storagenode 'config' folder.
fixed#4547
Closes#4547
We do not build an docker image for the multinode dashboard,
which makes monitoring for docker-focused environments harder.
This adds the basic image and ties it into CI/CD.
Change-Id: I14c01a7f1f0019f6f5c1b8fd75dc424fc362b18d
Currently the metainfo/metabase DB connections are missing the proper
application_name in order to differentiate and filter queries on the DB
side for analytics.
Without it, it is very time-consuming to correlate processes and their load.
This change adds the "check" on DB connection init and passes the fallbacks
in all places to catch connection strings, that do not set it.
Change-Id: Iea5cea8658bc63778ff89038e5c1c352bf482cfd
The "satellite fetch-pieces" command allows a satellite operator to
fetch as many pieces of a segment as possible, along with their
original order limits and hashes as provided by the storage nodes. The
fetched pieces and associated info will be stored on in a specified
folder as they are, rather than being RS-decoded or decrypted.
It is hoped that this will allow easier debugging of certain one-off
problems we've observed in the wild.
Change-Id: I42ae0e9ef0023538e42473a9be5a2460a3ac0f3a
some old configs had a value like
access: <data>
in the yaml. this would end up causing migration to
create a json file where it had no access values and
a default name of the data. that's not what the command
expects to operate on, so now we fix that during
migration and add a little mini migration for any
users that may have hit it.
Change-Id: I4c98ca5d09d043fe9338738ef6b4f930f933892c
In addition to upgrading the storj.io/common library, this change
moves off the TCPConnector in favor of the HybridConnector per
the deprecation warning.
Change-Id: I7e7e1e7568e8b95e4a99ad9caa158a799e68e1e3
Change the implementation of register and share so that it uses the
uplink method to contact the Auth Service. The network protocol switches
from HTTP to DRPC.
Closes https://github.com/storj/storj/issues/4324
Change-Id: Ib8fdb1665c6385bb39a546ba46a8df43a136df9c
Through `docker run storjlabs/storagenode:latest --help` we have always
made available around 100 command-line arguments.
However if you now pass such an argument it will be passed to
storagenode-update and it may no longer be recognized. This will cause
the storagenode not to start.
This was introduced in
https://review.dev.storj.io/c/storj/storj/+/5426
This change restores previous functionality.
Change-Id: I06823283ff82ffda12aee48c4d83717bddfbfdac
Change the order of when the storage node setup node loads the identity
for avoiding to write anything in the disk in the case that there is an
error loading the identity.
This bug was reported by @onionjake Github username's and the specific
changes to make.
Closes#4387#4396
Change-Id: I360fff3c23b160c9e055203d3526d749edfd9129
Get storagenode and storagenode-updater binaries during
run of the container to not to release new docker image
on each new version of the storagenode binary.
Fixes https://github.com/storj/storj/issues/4176
Change-Id: I994c4942136a2cc7298eb0346238689eb406ae5b
Use satellite.DB method TestingMigrateToLatest instead of
MigrateToLatest. TestingMigrateToLatest is much faster.
Also, run package tests in parallel.
Change-Id: I18bc0926dcfb80ace30d0b401e64ed919bfb966f
Value attribution codes were converted into UUIDs and stored
in the users, projects, api_keys, bucket_metainfos, and
value_attributions tables in the partner_id column. This
migration will lookup the appropriate partner name associated
with each of these UUIDs, and store the partner name directly
in the user_agent column within each table. If no corresponding
partner name exists for a partner_id, the partner_id value will
be stored instead.
Add migration for users table with tests.
Change-Id: I61254d9b81c474e76bcfc1c8cd863697c6ef44b6
Users signing up through a url containing a promo code will have that code applied to their stripe account instead of the free tier coupon.
Change-Id: I071041b0934648ef3f5bdb05b6ec97c400f89ae4
Currently the address being used is most of the time just :28967, which is not the correct address to reach the node from the public on.
This change uses the designated contact external address value that contains the set and preferred way to reach the node.
Change-Id: I99e979c2541043755b81e65c36c4289bfa3f60f3
this makes the flags match rclone nomenclature
fixes test-uplinkng to use the temporary config dir
instead of the machine default, and clean up some.
bumps clingy so that the command errors when an unknown
command is specified.
also fixes some printfs in share to use clingy stdout.
it still does some external actions that should be
passed through a ulext.External for mocking, but
that's ok for now.
Change-Id: Icc231e7e26393541c312396fec907b640b97718e
The Add method on the multinode DB interface accepts
a set of parameters which are already fields in the nodes.Node struct
excluding the name field.
When adding a new node, you're forced to call UpdateName() method
after calling the Add() method in order to save a node and update
the name.
This change allows passing the nodes.Node entity which includes
the name field. With this, a new node can be added together with
the name without invoking the UpdateName() method.
Change-Id: I281ec628dffaade35d6db4479a84f39636200072
The info command prints the details of the storagenode
to stdout.
It returns the storagenode info in JSON format
if --json flag is specified which can be piped
to the multinode add command.
Change-Id: I0163db8e02c4ec7346bfa69274d1772669357c6c
This change adds an add command to the multinode CLI.
The add command takes a json <file> as argument.
If dash (-) is specified, it reads data from stdin.
The <file> specified can be json file containing array of
nodes data or a single node object.
Change-Id: I44d68486dc9aea0bd0311a40e84d3262a0303aef
We want to monitor traffic and tools that are used
to interact with our network so we need to append
its user agent.
The same user agent is appended for current uplink
and uplinkng as eventually we will remove first.
Change-Id: I116080d6c2c6c85d591771facf01356de02a9392
this allows commands like
uplinkng cp -r sj://foo sj://bar
to work correctly, rather than complain that sj://foo is
not a boolean.
Change-Id: I003e47aabb85566bc2b454851cf55043b17ee7ea
In uplink we have command uplink share and we need
to port it to uplinkng to have command with same functionality.
Command could be executed with parameter or without.
without any flags we share as readonly.
Change-Id: I973b11d00da237358834acf5a863ebab37e684cc
In uplink we have command uplink access inspect and we need
a command with the same functionality for uplinkng.
Command could be executed with parameter or without.
without parameter - we should show default access.
If parameter exists - it could be access name or value.
If access name or value is wrong - we show error.
F.e.
uplinkng access inspect
uplinkng access inspect accessName
uplinkng access inspect accessValue
https://storjlabs.atlassian.net/browse/PG-318
Change-Id: I85fd961283850feb8684db2d126441f6b9bf0270
Test external implementation doesn't support OpenAccess
method. This makes imposible to test output of commands like
inspect or share. This implements only basic functionality.
Change-Id: I127ef0bb45a01634bd5265ed80840f8095c72794
The main motivation is to wrap the bucket DB and metainfo DB, so we
could check if a bucket is empty before applying geofencing config.
Change-Id: I8bac21555e01d51a663fb557bc1acfc8106bc2e1
func init() code isn't that well defined and reordering of them
could cause problems when starting the whole process from it.
Change-Id: I4088a0db156ece15354877011a481f6f91c9b332
This change adds the ability to download byte ranges
to uplinkng.
Extended the uplinkng Filesystem interface with Stat
method and an OpenOptions struct as parameter for the
Open method.
Also added a few tests for the ranged download
Change-Id: I89a7276a75c51a4b22d7a450f15b3eb18ba838d4
Add support for node url-s with PEER_URL to help configuring
authservice and gateway.
storj-sim network env SATELLITE_0_URL
Change-Id: I8b595b398007730662f100a4e6b3529cc0a7512a
Currently if the nodes is below the minimum version it will immediately
update to the suggested version, regardless if its eligible from the seed
or not. This change corrects the behaviour to update to the minimum
version only and then properly respect/wait for the rollout to include it.
Updates based on logic here: https://review.dev.storj.io/c/storj/private/+/6187
Change-Id: Ic6c91c48ae9b8a116378b2573fbfca7e7bd5cc32
If the directory doesn't exist, then the first run in a non-migration
setting will error because it cannot create the config file. This
change creates the directory.
Change-Id: I439159c00047e9ae20e139318dad5a047c853253
Usage: from a host for the affected satellite, issue this command
and supply the number of segments that have been permanently lost. For
example, to indicate 2 segments permanently lost:
satellite register-lost-segments 2
If the unthinkable happens and this is necessary to use, it is presumed
that we will also remove the non-recoverable segments from the metainfo
db, so they will not continue to appear as 'temporarily unavailable' to
the repair checker.
The influxQL query for this will probably look something like:
SELECT sum(total) AS "sum_total"
FROM "v3_stats_new"."autogen"."lost-segments"
WHERE time > :dashboardTime: AND "scope" = 'segment-durability'
GROUP BY time(:interval:), "application", "instance"
FILL(null)
Or use the Flux language in order to get the benefit of the
cumulativeSum function:
from(bucket: "v3_stats_new/autogen")
|> range(start: dashboardTime)
|> filter(fn: (r) => r._measurement == "lost-segments"
and r.scope == "segment-durability"
and r._field == "total")
|> group(columns: ["application", "instance"])
|> cumulativeSum()
Change-Id: I73d364937705aa815af7520ab79c00bc2aea09f6
Multipart upload limits added. Last part has no size limit.
Max number of parts: 10000, min part size: 5 MiB
Change-Id: Ic2262ce25f989b34d92f662bde720d4c4d0dc93d
https://storjlabs.atlassian.net/browse/PG-305
we should extend method move of cmd/uplink/cmd/mv.go
if both parameters end with slash -
i should list all files and call move method in loop
parameters:
uplink mv sj://bucket/a/prefix/ sj://new-bucket/a/new-prefix/
Change-Id: Ic24c2af83153ea60ec74393e65736af094877151
