The "satellite fetch-pieces" command allows a satellite operator to
fetch as many pieces of a segment as possible, along with their
original order limits and hashes as provided by the storage nodes. The
fetched pieces and associated info will be stored on in a specified
folder as they are, rather than being RS-decoded or decrypted.
It is hoped that this will allow easier debugging of certain one-off
problems we've observed in the wild.
Change-Id: I42ae0e9ef0023538e42473a9be5a2460a3ac0f3a
Part of server-side copy implementation.
Creates the methods BeginCopyObject and FinishCopyObject in satellite/metabase/copy_object.go.
This commit makes it possible to copy objects and their segments in metabase.
https://github.com/storj/team-metainfo/issues/79
Change-Id: Icc64afce16e5e0e15b83c43cff36797bb9bd39bc
All code on known satellites at this moment in time should know how to
populate and use the new numeric columns on the
stripecoinpayments_tx_conversion_rates and coinpayments_transactions
tables in the satellite db. However, there are still gob-encoded
big.Float values in the database from before these columns existed. To
get rid of those values, so that we can excise the gob-decoding code
from the relevant sections, however, we need something to read the gob
bytestrings and convert them to numeric values, a few at a time, until
they're all gone.
To accomplish that, this change adds two chores to be run in the
satellite core process- one for the coinpayments_transactions table, and
one for the stripecoinpayments_tx_conversion_rates table. They should
run relatively infrequently, so that we do not impose any undue load on
processing resources or the db.
Both of these chores work without using explicit sql transactions, but
should still be concurrent-safe, since they work by way of
compare-and-swap type operations.
If the satellite core process needs to be restarted, both of these
chores will start scanning for migrateable rows from the beginning of
the id space again. This is not ideal, but shouldn't be a problem (as
far as I can tell, there are only a few thousand rows at most in either
of these tables on any production satellite).
Change-Id: I733b7cd96760d506a1cf52735f598c6c3aa19735
the new uplink command expects to be able to migrate
and so we need to specify the --legacy-config-dir
flag sometimes as well. but unfortunately, the old
uplink will error if it gets a flag it doesn't
understand, so we have to set it as an env var.
but we can't use the env var for the --config-dir
flag all the time because the old uplink doesn't
look for that env var.
Change-Id: I019315192c0e6c348814527794342d823a5f9ec3
some old configs had a value like
access: <data>
in the yaml. this would end up causing migration to
create a json file where it had no access values and
a default name of the data. that's not what the command
expects to operate on, so now we fix that during
migration and add a little mini migration for any
users that may have hit it.
Change-Id: I4c98ca5d09d043fe9338738ef6b4f930f933892c
In addition to upgrading the storj.io/common library, this change
moves off the TCPConnector in favor of the HybridConnector per
the deprecation warning.
Change-Id: I7e7e1e7568e8b95e4a99ad9caa158a799e68e1e3
Tests refactoring to reuse testplanet instance between some
of tests. This should decrease resources needed to run those
tests.
Change-Id: I98f3041ec23085d3903b19acd339904973319ec1
* This commit stubs out the test plan for fuzz testing.
* pushing small change before rebasing.
* This commit implements a rough draft of our security tests, most importantly fuzz tests.
* Moving testplan up a directory and renaming to indicate executive summary.
* Cleaning out fuzzing tests from cmd/uplink,
* updates to tools list
added InactivityTimerEnabled flag to enable/disable feature
added InactivityTimerDelay to configure delay time in seconds
default timer set up to 10 minutes
reset dom events: keypress, mouseover, mousedown, touchmove
Change-Id: Idb66067c2902b2cdbe1a972225319c8abff97927
download performed without adding link to address bar
added warning to not share link with external people
Change-Id: Ifa79582c51c77c000342c1e03e752dd157912975
For some reason very small interval for chore is causing
very long execution time for Pause() method and that was
the reason why a simple test was slow.
Change-Id: Iebfdea16e31f9865493de3e1bd7d69c6f9116fc0
This patch fixes the link of "Transaction" on the storagenode web
console when polygon is used.
Tested with storj/storj-up based cluster and generating test payments.
Change-Id: Ia88954e055c6f41b92d77052afffebe633da549c
The Update method of the usersDB takes a console.User as an argument.
To update the columns in the DB, we have to migrate the fields from the
console.Users struct into a special dbx struct. If one of these fields
is left empty, then the zero value of that field's type will be used to
update the respective column.
In most cases where the users.Update method is called, the entire
console.User is apparently retrieved first, fields are updated, then it
is passed to users.Update. This is not the case for
service.UpdateAccount. Because these fields are not populated in the
user struct in UpdateAccount before it is passed into users.Update,
their respective columns in the database are overwritten with zero:
ProjectLimit, ProjectStorageLimit, ProjectBandwidthLimit,
ProjectSegmentLimit, PaidTier, MfaEnabled, MfaRecoveryCodes, MfaSecretKey
Solution: Do what is done in other places which call users.Update. Take
the console.User from the auth context, update the relevant fields on
that, then pass that in.
Change-Id: I3cbd560e8ea5397e5c27711fb40bb3907d987028
Change the implementation of register and share so that it uses the
uplink method to contact the Auth Service. The network protocol switches
from HTTP to DRPC.
Closes https://github.com/storj/storj/issues/4324
Change-Id: Ib8fdb1665c6385bb39a546ba46a8df43a136df9c
segment_copies table has (stream_id, ancestor_stream_id) as primary key,
but it should have only stream_id as primary as a stream_id can only have one ancestor_stream_id.
https://github.com/storj/team-metainfo/issues/84
Change-Id: I0218295e86e449c9dfefa5a21319f3083bf80afd
A user-agent string can contain multiple "products", in the case of
Gateway-MT at least this includes the HTTP client's full user agent.
This means that "other" is often logged even when we know the Storj
product, and sometimes logged more than once per call to "collect".
This makes sure that "other" is only logged if a product isn't
identified, and only logged once.
Change-Id: I8536f7eb32877e36fec97dab7b8d477ccb10f92e
For a thorough explanation of the overall transition, see the message on
commit c053bdbd70.
This change will rename the columns containing gob-encoded big.Floats
and add new columns which will contain the equivalent data in a more
sql-friendly format.
The change should *not* break already-running satellite processes,
because all functionality touching these tables has already been taught
to work with these new columns if it sees any "undefined column" errors.
Change-Id: I229324376533e383c5d05064b8aedad149cf825b
transfer-sh will be set as of https://github.com/dutchcoders/transfer.sh/pull/467.
filezilla needs to be verified and duplicati is set per info from @TopperDEL
comet and orbiter are added as preparation.
Change-Id: I44d730a7b3ba1969068e48c2477b478831799cd1
Part of server-side copy implementation.
Adds new table to metabase for keeping references between
copied segments and its ancestors.
Fixes https://github.com/storj/team-metainfo/issues/84
Change-Id: I436d4b533a3d951bcd752e222e3b721cee7f0844
This change adds some more checks to the deletion process for projects and
users, since we ran into a race condition during invoicing, where projects
have been deleted before the invoicing was finished, leading to missing
references.
This PR changes the logic to block user deletion if we are in exactly that period,
while also allowing the deletion of projects/users on free tier during the month.
Change-Id: Ic0735205e6633762fb7e3c2fa13e744cdfa5ec32
Finished implementing queries for both bandwidth and storage using pgx.Batch.
Fixed CSP styling issue.
Change-Id: I5f9e10abe8096be3115b4e1f6ed3b13f1e7232df
Through `docker run storjlabs/storagenode:latest --help` we have always
made available around 100 command-line arguments.
However if you now pass such an argument it will be passed to
storagenode-update and it may no longer be recognized. This will cause
the storagenode not to start.
This was introduced in
https://review.dev.storj.io/c/storj/storj/+/5426
This change restores previous functionality.
Change-Id: I06823283ff82ffda12aee48c4d83717bddfbfdac
Change the order of when the storage node setup node loads the identity
for avoiding to write anything in the disk in the case that there is an
error loading the identity.
This bug was reported by @onionjake Github username's and the specific
changes to make.
Closes#4387#4396
Change-Id: I360fff3c23b160c9e055203d3526d749edfd9129
There is a sev-2 issue to add more browser caching.
In this PR I made object map and object preview to be fetched by signed request with non-public credentials using AWS SignatureV4 package.
Change-Id: Ib5013fa6d6af3faa97eed5168c11a13f9629cd87
Before this change we were returning full DB error message.
That can be very confusing for end user. This change is translating
error message into more user frindly version and fixes also DRPC
error status code.
Fixes https://github.com/storj/team-metainfo/issues/76
Change-Id: I29b06ab4ba50a0d14db7a822a2906d95d65ab524
Get storagenode and storagenode-updater binaries during
run of the container to not to release new docker image
on each new version of the storagenode binary.
Fixes https://github.com/storj/storj/issues/4176
Change-Id: I994c4942136a2cc7298eb0346238689eb406ae5b
We don't have different version of object than 1 so at the moment
this method is not needed. Also using GetObjectExactVersion
should be slightly more performant.
Change-Id: I78235d8ae22594cc1d6345dabcc915f41cd7797b
We already split main code base, now we need to split test
to reflect new files structure (bucket/object/segment/other).
Fixes https://github.com/storj/team-metainfo/issues/12
Change-Id: Ica1054c4fc7df764483b03f204b4beba094df8e1
We have here two migrations in fact. One is for existing users,
we need to check if its paying user (paid_tier) and set 1M for
them and 150K for others.
Second migration is to set limits for projects depends on owner.
If owner is a paying user (paid_tier) then project should have
1M limit, otherwise it should be 150K. In this case to make
migration faster initially projects table segment_limit was set
to 1M by default. With migration we are selecting all paying
users and we are setting 150K limit for all projects which owners
are not in paying users set.
Initially we had a concern if that query wil lbe quick enough to
be executed during deployment but after investigation CRDB
team confirms that this should take seconds for out DBs.
Fixes https://github.com/storj/team-metainfo/issues/70
Change-Id: I8be06e9f949b68b993e043cc15525e8483bf49ea
The existing implementation doesn't send proper rfc1341 compatible mails
according to https://www.w3.org/Protocols/rfc1341/7_2_Multipart.html as
the closing `--boundary_id--` is missing from our mails. (see 7.2.1 from
the standard).
Mailservers which are not very flexible, deny the mails.
Example mail log: (see the
missing boundary delimiter at the end).
```
Subject: Activate your email
From: "Storj DCS - EU1" <noreply@eu1.storj.io>
To: "..." <...>
...
--26d7220f6f1c9f6fb47b535319eb15dce513bb6e1d941a0efddf25e96712
Content-Type: text/html; charset=UTF-8
....
</body>
</html>
.
smtp: DATA error {"msg_id":"6d82c9e3","reason":"unexpected EOF"}
smtp: 554 5.0.0 Internal server error (msg ID = 6d82c9e3)
```
This patch moves all the Multipart writing to a function to make sure
that the `wr.Close()` (which writes out the last part) is executed
BEFORE `body.Bytes()` (defer added it AFTER `body.Bytes()` was calculated)
Change-Id: I8f18fc81b1857b646470eab32e73d6cbdc50d2ad
Use satellite.DB method TestingMigrateToLatest instead of
MigrateToLatest. TestingMigrateToLatest is much faster.
Also, run package tests in parallel.
Change-Id: I18bc0926dcfb80ace30d0b401e64ed919bfb966f
