This change makes the authentication middleware reject any requests
that are not properly authenticated to prevent them from being
passed into endpoint-specific handlers.
Change-Id: I1f6b74f68fc7354e47fb825a128bad968129f420
Update the billing table to use generated IDs, to include the source and status fields, and to add metadata jsonb field that can be used for fields specific to a particular type of billing transaction. Additionally a new table was added to keep track of user balances
Change-Id: Ieb3a63aafd8fe21fc3386bafd43d52081b7d2838
satellite/{payments/billing,satellitedb}: refactor billing DB
Update the billing table to use generated IDs, to include the source and status fields, and to add metadata jsonb field that can be used for fields specific to a particular type of billing transaction. Additionally a new table was added to keep track of user balances
Change-Id: Ieb3a63aafd8fe21fc3386bafd43d52081b7d2838
metabasetest package utils can be used by both tests and benchmarks
if we will use interface TestingT from require package. This change
adjusts metabasettest.CreateObject method
Change-Id: I3c138e2ef9873b804ab5b3402804efa409397a9f
`len(m.attached)` was used without locking.
Also, use gofumpt to make whitespace usage more consistent.
Change-Id: Ifa9deedc8451f0c54e84d6ac3c2bdc1807688989
When a someone tries to create an account with an email that is already
associated with a verified account, send them an email with options to
sign in, create an account on another satellite, or reset password.
Change-Id: I844144d88b7356bd7064c4840c9441347a5368b0
Adding an interval_end_time column to the accounting_rollups table
to keep the last interval_end_time for each daily storage tallies.
Updates https://github.com/storj/storj/issues/4178
Change-Id: If7a8210c5e9fe2fc9df84b137a8b6e3db2471c58
removed segment limit validation and checks in metainfo endpoint and accounting/projectusage
since feature is live and has always has segment limitation now
Resolves: https://github.com/storj/storj/issues/4470
Change-Id: I8cf87cbbc40ac61262f9f05e52573d3ae6410611
With pointerdb listing objects operation was optimized to skip
objects from prefixes for non recursive listing. This change it
adopting this optimiaztion from old code.
Main change is to drop current page results if we detect a prefix
that needs to be skipped and jump with next listing query after this
prefix by setting cursor to "some-prefix(byte('/')+1) which is
effectively "some-prefix0".
Benchmark:
name old time/op new time/op delta
NonRecursiveListing/Postgres/listing_no_prefix-8 960µs ±11% 257µs ±12% -73.19% (p=0.008 n=5+5)
NonRecursiveListing/Postgres/listing_with_prefix-8 945µs ±11% 671µs ±12% -28.97% (p=0.008 n=5+5)
NonRecursiveListing/Cockroach/listing_no_prefix-8 4.31ms ± 8% 1.19ms ± 7% -72.44% (p=0.008 n=5+5)
NonRecursiveListing/Cockroach/listing_with_prefix-8 4.97ms ± 8% 3.35ms ±15% -32.67% (p=0.008 n=5+5)
Fixes https://github.com/storj/team-metainfo/issues/115
Change-Id: Iafdf3600d058abbaf441f792d32a7fc17cc08696
Previously there was no realtime administration of the storage usage
during copies. Now there is.
Closes https://github.com/storj/storj/issues/4719
Change-Id: I0d536bf551d16208116c3aceac89ed590ec473bf
When signing up, a user can opt in to having sales contact them. This
change alters the way this flag is passed to Hubspot and Segment.
Hubspot sends a form submission request to create the user, followed by
a "custom behavioral event" with some additional user info.
Segment sends an "Identify" call followed by a "create user" event.
This change moves "have_sales_contact" to the form submission for
Hubspot, and the Identify call for Segment.
This simplifies the process of applying this field to a contact/user in
Segment and Hubspot.
Change-Id: I5e6871b3926a76fb24f97fb2d835a26720275072
When a user's bandwidth/storage limits are manually set to exceed the
paid tier defaults, attempting to update their project via the satellite
UI (e.g. to change the name/description) would result in an error.
This change modifies the limit checks for updating a project to remove
this issue.
https://github.com/storj/storj/issues/4892
Change-Id: I48853a3289b0ac51587f268a18c1b25743123fcf
Piece deletion service was using KnownReliable method from
overlaycache to get nodes addresses to send delete request.
KnownReliable was always hitting DB because this method was
not using cache. This change is using new DownloadSelectionCache
to avoid direct DB calls.
Change is not perfect because DownloadSelectionCache is not as
precise as KnownReliable method and can select few more nodes
to which we will send delete request but difference should be
small and we can improve it later.
Updates https://github.com/storj/storj/issues/4959
Change-Id: I4c3d91089a18ac35ebcb469a56536c33f76e44ea
Currently we have a significant number of tallies that need to be
deleted together. Add a limit (by default 10k) to how many will
be deleted at the same time.
Change-Id: If530383f19b4d3bb83ed5fe956610a2e52f130a1
An object copy/move is done by 2 DRPC calls. It's possible a new object was uploaded ore moved to the source location between these calls. For copy, in that case the segments end up with the wrong keys. This change adds an explicit check for that by comparing the streamId supplied by the user with the streamId in the database.
Fixes https://github.com/storj/storj/issues/4930
Change-Id: Id600456ce78fb4069b93644828a0b3eb85e23e16
We need to provide the ability to see bucket attribution on the gateway side
so customers can validate if bucket is attributed to them. Extendet metainfo.ListBuckets
request with UserAgent.
Fixes https://github.com/storj/storj/issues/4965
Change-Id: I5624874a7faa14cda06183ad44013e9ebb385b63
Trace all the requests that the HTTP API endpoints receive.
We want to trace them with Monkit because we want to break them down by
request type and response code for seeing if they succeeded or failed.
Also log them with DEBUG level with the IP client.
Change-Id: Ia7b013351c788f131e775818f27091f3014ea861
We retry a GET_REPAIR operation in one case, and one case only (as far
as I can determine): when we are trying to connect to a node using its
last known working IP and port combination rather than its supplied
hostname, and we think the operation failed the first time because of a
Dial failure.
However, logs collected from storage node operators along with logs
collected from satellites are strongly indicating that we are retrying
GET_REPAIR operations in some cases even when we succeeded in connecting
to the node the first time. This results in the node complaining loudly
about being given a duplicate order limit (as it should), whereupon the
satellite counts that as an unknown error and potentially penalizes the
node.
See discussion at
https://forum.storj.io/t/get-repair-error-used-serial-already-exists-in-store/17922/36
.
Investigation into this problem has revealed that
`!piecestore.CloseError.Has(err)` may not be the best way of determining
whether a problem occurred during Dial. In fact, it is probably
downright Wrong. Handling of errors on a stream is somewhat complicated,
but it would appear that there are several paths by which an RPC error
originating on the remote side might show up during the Close() call,
and would thus be labeled as a "CloseError".
This change creates a new error class, repairer.ErrDialFailed, with
which we will now wrap errors that _really definitely_ occurred during
a Dial call. We will use this class to determine whether or not to retry
a GET_REPAIR operation. The error will still also be wrapped with
whatever wrapper classes it used to be wrapped with, so the potential
for breakage here should be minimal.
Refs: https://github.com/storj/storj/issues/4687
Change-Id: Ifdd3deadc8258f34cf3fbc42aff393fa545794eb
Added new email html template.
It is sent when user tries to reset password with unknown or unverified account.
Made a couple of minor config changes.
Issue: https://github.com/storj/storj/issues/4913
Change-Id: I730f48b3478e302d1e38e1f8a27c75f66a8ba6fd
Some nodes were added to the nodes table due to a bug in quic
based storagenode contact code. This is a tool to clean up these nodes.
Delete with batch-size 1k seems to take ~400ms on local CockroachDB.
Change-Id: Ic0c1180528c27952e19c431fc9cc327292a10a5f
Add payments method to payments to DepositWallets interface.
Exposes payments retrieval API for a particular wallet to
other systems such as console billing API.
Change-Id: Ifcb3a35514aab50be00f6360007954980b5d8b38
Use DownloadSelectionCache to avoid querying database for every
download.
This change only addresses downloads from users. The download selection
cache is not currently used for audit and repair.
Change-Id: I96a49e121dac0b4204f97592a63131edabd73fb5
Adding go.mod into node_modules is not sufficient, because npm install
wipes them quite often out. Similarly, when running npm install locally
it will remove it, causing the git state to be dirty.
Rather than having them committed, add them after running npm install.
Change-Id: Iaf21a9c6e198dc31fe50345ec5dee85b44617176
This just cleanup change to unblock libuplink to reorganize types
which are aliases to storj types.
Change-Id: Id3edf13f1b0aef52d7606d545aa7a6594cf8d13f
Add go.mod to node_modules folder, that way Go compiler doesn't
need to scan the node_module directories for any Go code.
Change-Id: I747909416490c847d6b4bfa3438fea66660fcd53
It seems the tests relied on time.Now(), which might cause some
discrepancies in calculations. Use a fixed time.Now() rather than
recalculating.
As a sidefix, remove "Test" prefix from t.Run. These are unnecessary.
Change-Id: I1de903fcf0fcf46fc8e3acf2463e17239b8e3cc6
The MinDownloadTimeout 950ms and delay of 1s were quiet close, possibly
causing flaky behavior in TestVerifierSlowDownload.
Change-Id: I4f6c1554a118b21427357642abe39986fd0af38d
Classify errors related to invalid tokens for activating user accounts
for returning 400 status code rather than 500 status code.
Don't log all the errors with "error" level, only the ones related to
internal server errors and the rest log them with "debug" level because
they pollute the production satellite errors with errors that are
misguiding.
Change-Id: Id2bd737edba8550ce08965b51b8bf2540bd13ca4
Previously copying an object to it's ancestor location (copy of copy)
broke the object and all copies.
This fixes this by calling the existing delete method rather than a
custom one when there is an existing object at the copy destination.
The check for existing object at destination has been moved to an
earlier point in FinishCopy.
metabase.DeleteObject exposes a transaction parameter so that it can be
reused within metabase.
Closes https://github.com/storj/storj/issues/4707
Uplink test at https://review.dev.storj.io/c/storj/uplink/+/7557
Change-Id: I418fc3337fa9f30146ccc1db456af168ae41c326
- instead of closing over the outer err variable, potentially
overwriting some errors or something, declare local variables.
- double check that we got the number of rows we expected to get
and error otherwise. this prevents a possible source of inserting
bogus rows into the database.
Change-Id: I30662be2727afe0a90e4215a182fedc2648d1169
Part of the delete query cause a full table scan of segment_copies. This
slowed down the system. This change should have the same semantics but
improved performance.
Part of https://github.com/storj/storj/issues/4898
Change-Id: I4afe23df05467eafc9c91591f47a7251a0f3dd31
