Commit Graph

4710 Commits

Author SHA1 Message Date
Egon Elbre
2f4bb114d4 go.mod: bump common to remove sha256-simd
Change-Id: I77aeb3b4bea87e8d5bb83b05e86b61a80a695f2a
2020-10-14 12:47:37 +03:00
Egon Elbre
f06ce1ef01 release: remove binary stripping
We've had bizarre crashes for satellite and it's difficult to debug
stripped binaries. The only binary where stripping is useful, is uplink
cli. This change will increase uplink by 5MB.

Change-Id: I4d1dfd36452063c22e8471d99eec97f6de6167b8
2020-10-14 10:06:20 +03:00
Kaloyan Raev
830817ec0d cmd/storj-sim: run gateway without --access flag
This makes it possible to remove of this obsolete flag from the
multi-tenant gateway.

As a consequence, displaying the GATEWAY_0_ACCESS env var will always
require a running storj-sim. Until now, it was required only the first
time. Then the value was stored in the 'access' config. But this is now
not possible anymore.

The changes in StripeMock are required to fix failures in integration
tests. StripeMock is in-memory and its data does not survive restarts of
storj-sim. The second and following starts of storj-sim had invalid
state of StripeMock, which failed requests that were required to
populate the GATEWAY_0_ACCESS env var. The changes in StripeMock makes
it repopulate the Stripe customers from the database.

Change-Id: I981a208172b76577f12ecdaae485f5ae4ea269bc
2020-10-13 14:45:04 +00:00
Egon Elbre
cf2dd76db7 cmd/satellite: proper log usage
log.Fatal immediately terminates the program without running any defers.
We should properly close all the services and databases.

Change-Id: I5e959cef3eafedeacb3a2062e3da47e8d04e8e75
2020-10-13 16:56:35 +03:00
Egon Elbre
2268cc1df3 all: fix linter complaints
Change-Id: Ia01404dbb6bdd19a146fa10ff7302e08f87a8c95
2020-10-13 15:59:01 +03:00
Egon Elbre
0bdb952269 all: use keyed special comment
Change-Id: I57f6af053382c638026b64c5ff77b169bd3c6c8b
2020-10-13 15:13:41 +03:00
littleskunk
3ff8467878
satellite/projectlimit: Update limit increase link (#3950)
Co-authored-by: Stefan Benten <mail@stefan-benten.de>
2020-10-13 12:46:22 +02:00
Stefan Benten
1d3b728766 satellite/{console/payments/satellitedb}: add validation for deletion of account and project
The same was that our Admin API handles project and account deletions currently, we would like
to have the same checks on the user-facing API. This PR adds the same checks to the console service.
General more applicable checks have been moved directly into the payments service.

In addition it adds the BucketsDB to the console DB, to have easier access and avoiding import cycles with
the metainfo package.

A small cleanup around our unnecessary monkit imports made it in as well.

Change-Id: I8769b01c2271c1687fbd2269a738a41764216e51
2020-10-13 07:55:26 +00:00
Brandon Iglesias
50756cb434
Adding dominickmarino to the CLA bot list (#3952) 2020-10-12 23:41:59 +02:00
Jeff Wendling
4cbd4d52a9 satellite/orders: only hold the orders semaphore during database calls
holding it during node i/o means slow nodes can hold up order
processing for everyone else. this dramatically increases
the amount of tiem spent handling orders.

Change-Id: Iec999b7ed0817c921a0fd039097a75bdd3c70ea2
2020-10-10 15:40:50 -04:00
Jeff Wendling
0f0faf0a9f satellite/orders: do a better job limiting concurrent requests
Doing it at the ProcessOrders level was insufficient: the endpoints
make multiple database calls. It was a misguided attempt to only
have one spot enter the semaphore. By putting it in the endpoint
we can not only be sure that the concurrency is correctly limited
but it can be configurable easily.

Change-Id: I937149dd077adf9eb87fce52a1a17dc0afe96f64
2020-10-09 16:27:15 -04:00
Caleb Case
cf1748158a
Bump Dependencies
Change-Id: I4c8a4438e74379a490a19f1f88ea9dac7715dbbd
2020-10-09 09:33:49 -04:00
Stefan Benten
7161506b68 Makefile: handle msi packages correctly
With the current Makefile, both the msi and exe files get combined into one zip file.
This is not the expected behavior, which this change fixes.
It strips off only .exe from filenames going forward and leaves every other extension intact.

Change-Id: If8132b1427eec7a9e5ebd7ac6b8b3e9d12524080
2020-10-09 13:15:35 +02:00
Stefan Benten
c1ca470e7e storagenode/orders: fix import and cleanup go.mod and go.sum
Accidentally we imported the wrong monkit package with a previous
commit and made our go.mod and go.sum file unclean.
This should fix it.

Change-Id: I4c3c8b696f59cfd06dc2d5436bb7aea2805936ce
2020-10-09 00:04:57 +02:00
Jeff Wendling
7c303208ff satellite/satellitedb: emergency temporary order processing semaphore
we have thundering herds of order submissions that take all of the
database connections causing temporary periodic outages. limit
the amount of concurrent order processing to 2.

Change-Id: If3f86cdbd21085a4414c2ff17d9ef6d8839a6c2b
2020-10-08 19:16:47 +00:00
Stefan Benten
ad8da61dac cmd/satellite: Remove curl from Dockerfile
Sadly the build process with this command is very, very flaky and often fails pulling down curl via apk.
As we currently do not need it anyway, it is safe to remove.

Change-Id: I8a396c560d61a7fe6324560152a68c07c6b31638
2020-10-08 20:59:05 +02:00
Moby von Briesen
3209effeb6 storagenode/orders: Increase order sending interval from 5m to 1h
Since storage nodes check to see if any order files can be sent every 5
minutes, every storage node attempts to send orders to the satellite
within 5 minutes of each hour since this is when the files become
"available" to send. It is placing a lot of load on our satellite and
storage nodes are not being paid out properly due to timeouts during
order sending due to the increased satellite load.

Change-Id: I44d991b5884b8c11e8a3856d39aee8323f086b51
2020-10-08 12:51:21 -04:00
Yaroslav Vorobiov
e598876d79 cmd/storagenode-updater: trimm \n suffix on receiving service pid from systemctl
Change-Id: I92aac195522e46b712f05beb47d7472c2a1b4d6c
2020-10-08 15:20:20 +00:00
NickolaiYurchenko
a1488e53a0 web/storagenode: online score added
Change-Id: I8c0fd1332354063941d62891ba79ca895074b6c8
2020-10-08 13:02:59 +00:00
Stefan Benten
b3cf12f567 satellite/console: Add more validation for console requests
Adds membership checks for the following calls:
- GetProject

Add ownership checks for the following calls:
- DeleteProject

It also disables the API endpoint to delete a project.

Furthermore it adds tests for the console service.

Change-Id: I1ffc8dcb44746a74ad06a7dbd064a29c57c25272
2020-10-07 15:33:28 +00:00
Kaloyan Raev
e7f2ec7ddf satellite/audit: fix sanity check for verify-piece-hashes command
The VerifyPieceHashes method has a sanity check for the number pieces to
be removed from the pointer after the audit for verifying the piece
hashes.

This sanity check failed when we executed the command on the production
satellites because the Verify command removes Fails and PendingAudits
nodes from the audit report if piece_hashes_verified = false.

A new temporary UsedToVerifyPieceHashes flag is added to
audits.Verifier. It is set to true only by the verify-piece-hashes
command. If the flag is true then the Verify method will always include
Fails and PendingAudits nodes in the report.

Test case is added to cover this use case.

Change-Id: I2c7cb6b12029d52b2fc565365eee0826c3de6ee8
2020-10-07 17:17:48 +03:00
Kaloyan Raev
4280142b24 satellite/console: remove unnecessary Error.Wrap
Change-Id: If851ccce7932cbf72c2fff3b51f4f9f2ea07c124
2020-10-07 09:22:41 +00:00
Moby von Briesen
fbf2c0b242 storagenode/orders: Refactor orders store
Abstract details of writing and reading data to/from orders files so
that adding V1 and future maintenance are easier.

Change-Id: I85f4a91761293de1a782e197bc9e09db228933c9
2020-10-06 15:28:07 -04:00
Kaloyan Raev
bd177bff03 cmd/storj-sim: cleanup gateway setup
Remove usage of --non-interactive flag. It is not provided (and
necessary) by the multitenant S3 gateway anymore.

ACCESS_KEY and SECRET_KEY env vars are not provided anymore as they are
not generated by the multitenant S3 gateway.

Change-Id: I3ecfb92110e31ae13977de3899dad273daae6c1e
2020-10-06 14:22:47 +00:00
Stefan Benten
14a2050b8d pkg/auth: move package to consoleauth
To avoid further name collisions, the very broad named package gets moved into
the consoleauth package where its also mainly being used.

Change-Id: Ie563c9700adbf0553baca2b7b8ba4a1d9c29d144
2020-10-06 14:15:07 +02:00
Stefan Benten
44bd65795b satellite/console: ensure only project members can remove other project members
Change-Id: I815eb85f37631aaa65b5dc4cafa6851f241ca0f0
2020-10-06 11:03:12 +00:00
Stefan Benten
9deea2ffe2 satellite/console: disable account deletion via API
Change-Id: Ia8e43284c90fb2b833eb601e2c8f701cb5a4d9c0
2020-10-06 13:01:46 +02:00
Stefan Benten
338ed90876 Makefile: simplify binaries-upload
This change makes the loop to create the zip files a little easier (especially to read).

In addition, it also omits the .exe part for windows zip files, to ensure we can properly
use the versioncontrol URL for both windows and unix rollouts.

Change-Id: I0cbabb2ca61eb3346d97b2071411885383e870f3
2020-10-06 10:35:04 +02:00
Stefan Benten
0aaad88a44 satellite/{admin, console}: add test for projectLimit increase and update README
This change adds the capabilities to adjust the users project limit via the Admin API.
Adds a test for the new added function of the API and updates the existing tests.
It renames the json field on the user struct to be more consistent.

Change-Id: I9018acd80dae0af68d1d50526f20987132c654f3
2020-10-05 11:54:37 +00:00
NickolaiYurchenko
f27cee91b0 web/storagenode: 404 page rework
WHAT: new desist and test for 404 error page

WHY: 404 error page was hidden behind infinite loading screen
also, there were no dark theme deging for it

Change-Id: If9a1f5dd2321d9fc4a294637b8d5c041947cc9d1
2020-10-05 09:29:19 +00:00
Cameron Ayer
b39a99bae6 satellite/{overlay,satellitedb}: always show node's real online score
Previously if a node did not have audit history data for each of the
windows over the tracking period, we would give them the benefit of
the doubt and set their score to 1. This was to prevent nodes from
being suspended right out the gate. We need a minimum amount of data
to evaluate them.

However, a node who is actually failing at being online will have no
idea until they have received enough audits and we suspend them.

Instead, we will always use their real score, but use a flag to determine
whether they are eligible for suspension/dq.

Change-Id: I382218f12e8770f95d4bcddcf101ef348940cadf
2020-10-02 12:28:11 -04:00
NickolaiYurchenko
0d7ae8b182 web/storagenode: held history table: disposed subtracted from held
Change-Id: I11f3b019b3d1464abc932c8c7138b62f61924813
2020-10-02 15:35:06 +00:00
Qweder93
664b8f6821 storagenode/payout: estimation payout values switched from int64 to float64 to avoid incorrect rounding.
float64 values rounding to 2nd sign after dot.

Change-Id: Ice49f6a0944231ea6adb3343545bf1a62ff6dbc1
2020-10-02 11:33:43 +00:00
Yingrong Zhao
f13084983d bump common to latest
Change-Id: I55ca40a7b2c58d70225da07b7a62d5ad60451881
2020-10-01 15:39:09 -04:00
NickolaiYurchenko
406281fc13 web/storagenode: changed restriction to not show calendar in payuot history table [SG-516]
Change-Id: Ia1706c084f222e078b65ff34bb7d88a9d62c4ce2
2020-10-01 16:10:14 +00:00
Yingrong Zhao
c085a17a52 bump common and uplink to latest
Change-Id: I717f0214dd9973acd51b7732c5d64587f610c805
2020-10-01 15:38:58 +00:00
crawter
fa39274ffd multinode/multinodedb: add dbx support
We want to use dbx ORM in our codebase to have consistency with satellite codebase

Change-Id: I0646c9ef05db54a7b8e92c543ff8517bada2c5b0
2020-10-01 14:48:59 +00:00
Yaroslav Vorobiov
c4d5625965 cmd/storagenode-updater: fix service build tag, for platforms other than linux and windows
Change-Id: I7ed76b80a1df0ef53f2dd6726a602cc3bd4df9de
2020-10-01 15:32:28 +03:00
NickolaiYurchenko
3a7eb10c71 web: tests fix
Change-Id: I73cfb86fd214afb02be6a9e96e0ae5ffb92fd214
2020-10-01 13:06:28 +03:00
Egon Elbre
4e8d53c8fb private/dbutil/pgutil: ensure storagenode doesn't depend on pgx
pgx is a large dependency and there's no need to include it in
storagenode binary.

Change-Id: I49c304c6420733d5f095d7edb35d32811210e41a
2020-09-30 14:28:47 +00:00
Qweder93
245986d528 negative space calculations fix removed
Change-Id: I342c61856fce6d02dc99fd27fd3d563540f22b64
2020-09-30 14:08:24 +00:00
VitaliiShpital
80eeaf85db web/satellite: sign up link updated
WHAT:
sign up link in add user popup updated to redirect to correct page

WHY:
bug fix

Change-Id: I607d0b7d71bc6cc73b304d9ff8b8d4aca7549a45
2020-09-30 16:36:46 +03:00
Yaroslav
a43afa7ea0 make: fix storagenode-updater binary target
Adds -tags=service when building storagenode-updater which adds
interaction with Windows SVC and Linux systemd to the binary.

Change-Id: I21b4b3e8718348a30ec0453b07135ff3de033a5a
2020-09-30 11:53:11 +00:00
Yaroslav Vorobiov
a840cb71e7 storagenode: check db version before run
Change-Id: I912f63fd62f2bff10341346c28dfb92fcd683806
2020-09-30 10:58:09 +00:00
Egon Elbre
139ff3427b go.mod: bump dependencies
Change-Id: I2b1bf014a60ef6ddc4b1b546f58393c34ffe22fc
2020-09-30 11:28:37 +03:00
Moby von Briesen
000b1e6011 satellite/admin/project_test.go: Update TestCheckUsageLastMonthUnappliedInvoice
Test is failing because the mock date used in it is September 2020.
Update to 2030.

Change-Id: I4733487d0589dbeb0d2f3dd2abd88f767d59d0ca
2020-09-30 00:10:02 -04:00
Cameron Ayer
c2525ba2b5 satellite/{repair,satellitedb}: clean up healthy segments from repair queue at end of checker iteration
Repair workers prioritize the most unhealthy segments. This has the consequence that when we
finally begin to reach the end of the queue, a good portion of the remaining segments are
healthy again as their nodes have come back online. This makes it appear that there are more
injured segments than there actually are.

solution:
Any time the checker observes an injured segment it inserts it into the repair queue or
updates it if it already exists. Therefore, we can determine which segments are no longer
injured if they were not inserted or updated by the last checker iteration. To do this we
add a new column to the injured segments table, updated_at, which is set to the current time
when a segment is inserted or updated. At the end of the checker iteration, we can delete any
items where updated_at < checker start.

Change-Id: I76a98487a4a845fab2fbc677638a732a95057a94
2020-09-29 20:38:22 +00:00
Egon Elbre
c23a8e3b81 go.mod: update pgx to v4.9.0
Fix query to use TextArray instead of VarcharArray.
Fix queries to use the correct type.

Change-Id: Ibb7e55adba277d05778118d81ca697470e72c374
2020-09-29 19:03:08 +00:00
Michal Niewrzal
cd2a5484f3 storagenode/console: ignore untrusted satellite while returning
dashboard data and calculating satellites data

Change-Id: I71d596891477e0839863e007689b6e2e6e420a22
2020-09-29 18:27:49 +00:00
crawter
081fdcc835 multinode: initial infrastructure
Change-Id: Ibe31d555c144915acdcc26915deb68d6c4c3fbe9
2020-09-29 17:58:17 +03:00