This change introduce problems with server side move so
let's revert it for now. Problem was found when latest
version of storj/storj was used in uplink tests.
This reverts commit 1ef06fae99.
Change-Id: I4d4fad5d1ea04ba15ff9d7bd765f7e078e9187c2
Removes database tables and functionality related to our custom
coupon implementation because it has been superseded by the Stripe
coupon and promo code system. Requires implementations of the
payments Invoices interface to return coupon usages along with
invoices.
Change-Id: Iac52d2ff64afca8cc4dbb2d1f20e6ad4b39ddfde
We were using mixed types for nonce fields. Protobuf
have storj.Nonce, metabase have []byte. This change
is a refactoring to have everywere its possible only
storj.Nonce.
Change-Id: Id54bd8481f30c721cdaf3df79206d25e7cfdab55
Cockroach doesn't like concurrent database creation, add a limiter to
avoid overloading the database.
Also limit the cockroach testing to the last 10 migrations.
Change-Id: Ie73c516ac2d362b251f605049e51eb25888432bd
We are not using the benchmark results for anything, they are mostly
there to ensure that we don't break the benchmarks. So we can disable
CockroachDB for them.
Similarly add short versions of other tests.
Also try to precompile test/benchmark code.
Change-Id: I60b501789f70c289af68c37a052778dc75ae2b69
* Add test cases to verify that all the endpoint that target a specific
entity respond 404 status code when the entity isn't found.
* Fix the endpoints that target a specific entity which responded a 500
status code response when the entity didn't exist to respond with 404
status code.
Additionally:
* Simplify some tests using an existing test helper function.
* Rename test functions to start with the entity name (e.g. Project,
User, etc.) for easing to run a set of test with the `-run` Go test
flag.
Change-Id: I82aad92e429207b72932ad4b79c08db6b486a19a
Populate the egress_dead column for taking into account allocated bandwidth that can be removed because orders have been sent by the storage nodes. The bandwidth not used in these orders can be allocated again.
Change-Id: I78c333a03945cd7330aec052edd3562ec671118e
It's safer to create new connection pool for piece deletion
only if dialer have no existing pool assigned.
Change-Id: I26661683ab7c0198587905478057c01c8f533a7e
We should be using object naming insted of path.
This is one place where we can easiliy change it.
To regenerate protbuf I had to remove gogo.proto.
Most probably it was confilicting with gogo.proto
from common/pb.
Change-Id: Ia5972f77994765c8f26bf1c3dc8205d2eadd70fa
We want to enable connection pool for piece deletion to avoid
doing multiple SSL hanshakes to SN while massive deletion process.
Change-Id: Ic917e4eda304ee16a286926ef046fe9e38bf38ca
We were manually converting ObjectKey fields to []byte to use it with
SQL query but we can just implement Value method to convert it automatically.
Change-Id: I6d346f4b59718e1e8ef37cd9f95e613b864b42cd
Update repair tests to check if audit score increases for nodes
that successfully send pieces during successfull and failed repairs.
Change-Id: Ie6abbde6155ab4697d209366c9fa497e731756e9
Drop table graceful_exit_transfer_queue which is not used anymore (replaced by graceful_exit_segment_transfer_queue).
Change-Id: Ie254fe9a54fb0784e350a439ce7a9bc99a3a58b5
We would like to verify if zombie object/segment works fine.
We need some metric for that. Figuring out number of deleted
objects is harder so let's for that later.
Change-Id: Ic99e2ce93256130b7c51f514824fddc009655075
Validates the new user data, through `Validate` method, on "create user"
endpoint earlier and remove specific checks of data that is already
validated by the `Validate` method.
Change-Id: I6b8f91c0b706f0c8316173c491e25c4858172640
A previous commit added a helper function for sending JSON data back to
the client.
This commit makes use of it for homogenizing the current implementation.
It also renames the existing helper message to send JSON errors to
starts with "send" because the new helper starts with it and they
helpers are clearer with their name starting with it.
Change-Id: I53ee0b4ca33d677a8ccd366c9ba6d73f4f472247
Fix the Admin API endpoints when it response with a client error
response code.
The most of the client response error send a JSON body but the
`Content-Type` header wasn't set to the appropriated value. This commit
fixes them and it adds assertions to the tests to very the
`Content-Type` header.
Updates the README to briefly document about the format of the client
errors endpoints responses.
Change-Id: Ifaf2122def801701211438ce241046be1adc0e8c
Move tests which where related with the `admin` package to the `console`
one because it's where the interface for the methods that the tests
check is defined.
Change-Id: I1bbccfe0badd989ed146a260af8d207d47737ef1
Migration tests are very heavy on database schema changes, which may
cause delays and retries. Separate out the migration tests and ensure
that they do not run concurrently on the same database.
Change-Id: I35b17525f18fd923546ce1fcc12d805c95073b6b
Why: big.Float is not an ideal type for dealing with monetary amounts,
because no matter how high the precision, some non-integer decimal
values can not be represented exactly in base-2 floating point. Also,
storing gob-encoded big.Float values in the database makes it very hard
to use those values in meaningful queries, making it difficult to do
any sort of analysis on billing.
Now that we have amounts represented using monetary.Amount, we can
simply store them in the database using integers (as given by the
.BaseUnits() method on monetary.Amount).
We should move toward storing the currency along with any monetary
amount, wherever we are storing amounts, because satellites might want
to deal with currencies other than STORJ and USD. Even better, it
becomes much clearer what currency each monetary value is _supposed_ to
be in (I had to dig through code to find that out for our current
monetary columns).
Deployment
----------
Getting rid of the big.Float columns will take multiple deployment
steps. There does not seem to be any way to make the change in a way
that lets existing queries continue to work on CockroachDB (it could be
done with rules and triggers and a stored procedure that knows how to
gob-decode big.Float objects, but CockroachDB doesn't have rules _or_
triggers _or_ stored procedures). Instead, in this first step, we make
no changes to the database schema, but add code that knows how to deal
with the planned changes to the schema when they are made in a future
"step 2" deployment. All functions that deal with the
coinbase_transactions table have been taught to recognize the "undefined
column" error, and when it is seen, to call a separate "transition shim"
function to accomplish the task. Once all the services are running this
code, and the step 2 deployment makes breaking changes to the schema,
any services that are still running and connected to the database will
keep working correctly because of the fallback code included here. The
step 2 deployment can be made without these transition shims included,
because it will apply the database schema changes before any of its code
runs.
Step 1:
No schema changes; just include code that recognizes the
"undefined column" error when dealing with the
coinbase_transactions or stripecoinpayments_tx_conversion_rates
tables, and if found, assumes that the column changes from Step
2 have already been made.
Step 2:
In coinbase_transactions:
* change the names of the 'amount' and 'received' columns to
'amount_gob' and 'received_gob' respectively
* add new 'amount_numeric' and 'received_numeric' columns with
INT8 type.
In stripecoinpayments_tx_conversion_rates:
* change the name of the 'rate' column to 'rate_gob'
* add new 'rate_numeric' column with NUMERIC(8, 8) type
Code reading from either of these tables must query both the X_gob
and X_numeric columns. If X_numeric is not null, its value should
be used; otherwise, the gob-encoded big.Float in X_gob should be
used. A chore might be included in this step that transitions values
from X_gob to X_numeric a few rows at a time.
Step 3:
Once all prod satellites have no values left in the _gob columns, we
can drop those columns and add NOT NULL constraints to the _numeric
columns.
Change-Id: Id6db304b404e6fde44f5a8c23cdaeeaaa2324f20
Why: big.Float is not an ideal type for dealing with monetary amounts,
because no matter how high the precision, some non-integer decimal
values can not be represented exactly in base-2 floating point. Also,
storing gob-encoded big.Float values in the database makes it very hard
to use those values in meaningful queries, making it difficult to do
any sort of analysis on billing.
For better accuracy, then, we can just represent monetary values as
integers (in whatever base units are appropriate for the currency). For
example, STORJ tokens or Bitcoins can not be split into pieces smaller
than 10^-8, so we can store amounts of STORJ or BTC with precision
simply by moving the decimal point 8 digits to the right. For USD values
(assuming we don't want to deal with fractional cents), we can move the
decimal point 2 digits to the right.
To make it easier and less error-prone to deal with the math involved, I
introduce here a new type, monetary.Amount, instances of which have an
associated value _and_ a currency.
Change-Id: I03395d52f0e2473cf301361f6033722b54640265
This PR utilize the new burst limit column from projects table to allow
control on the limit for request per seconds and token bucket size
When no burst limit is explicitly set, rate limit is applied to both so
we don't limit how quickly request can be made in a second.
Change-Id: I883235c60c5d6416aeadd1c80ed2ebd193aa4d9f
Currently loops wait for the coalesce duration for TriggerWait.
Let's skip the coalesce when we trigger it manually.
Change-Id: If5bacd4e263d233f1f3ea41b989922d2ed5a48d4
In order to limit the amount of overall requests a user can issue in a
time span, we need to have the ability to define such limit separate
from per second request rate.
This PR adds a new column on the projects table to store the burst limit
per project.
Change-Id: I7efc2ccdda4579252347cc6878cf846b85146dc7
Metrics observer is joining loop only for monitoring
and because of that it waits until non-monitoring
observer will join loop.
That was causing some delays in tests. To fix that
we are triggering non-monitoring observer manually.
Change-Id: Id93d59c9fd8d637dac0beea82ff2c40247d94b10
server-side move extended with moving between buckets, for this reason
we change bucket name for object in db.
Change-Id: Ie21bcccc170e6ff14dcd8053fdb86fdf6d8438a0
Some processing inside storagenodes is async compared to uplink upload
and download, hence we need to explicitly wait for storagenodes to
finish their pending work before flushing orders to the satellite.
Hopefully this fixes TestAttributionReport flakiness.
Change-Id: I77c651ab6471ae094b5c21d1ab3860c96cb0d039
Not all errors from RunOnce can be retried. The context can be cancelled
with several different errors, e.g. timeout. Ensure we stop the loop
when context has errored, because none of the queries will succeed
when it has failed.
Change-Id: If3ff11f11a6f43c0d67633be1cfaf23e3e9e55f3
Second method needed to perform server-side move. It updates
metadata key and nonce and all segments key and nonces.
Change-Id: Ia43b26622a13048269f0ae9e1524b345db112adb
Since we are sharing the reporting logic between repair and audit. We
need to remove metric reporting logic in reporter.
Change-Id: Ib87295ab19079329e7438327d785a7f5c21d3b21
In the situation where the flushes take longer than the incoming
rate of writes, the RollupsWriteCache will take every connection
in the database pool and use them forever. Instead of doing that
and taking down satellite availability, bound the number of flush
operations that it will perform and drop incoming writes earlier
to keep memory usage constant.
Adds monitoring events for if any flushes or updates are lost.
Change-Id: I81b169b73501ee9b999f4b03d1e79645fc56f167
First from two methods needed to perform server-side move. It gets
metadata key and nonce and all segments key and nonces and returns
all of that to uplink.
Change-Id: Ied2c79559e77d3f63091c4d61948f2d6a2147d67
Remove the logic associated to the old transfer queue.
A new transfer queue (gracefulexit_segment_transfer_queue) has been created for migration to segmentLoop.
Transfers from the old queue were not moved to the new queue.
Instead, it was still used for nodes which have initiated graceful exit before migration.
There is no such node left, so we can remove all this logic.
In a next step, we will drop the table.
Change-Id: I3aa9bc29b76065d34b57a73f6e9c9d0297587f54
Currently TextMaxVerifyCount flakes in some tests, try increasing the
sleep time to ensure that things are slow enough to trigger the error
condition.
Also pass ctx to all the funcs so we can handle sleep better.
Change-Id: I605b6ea8b14a0a66d81a605ce3251f57a1669c00
