Commit Graph

4589 Commits

Author SHA1 Message Date
Kaloyan Raev
4280142b24 satellite/console: remove unnecessary Error.Wrap
Change-Id: If851ccce7932cbf72c2fff3b51f4f9f2ea07c124
2020-10-07 09:22:41 +00:00
Moby von Briesen
fbf2c0b242 storagenode/orders: Refactor orders store
Abstract details of writing and reading data to/from orders files so
that adding V1 and future maintenance are easier.

Change-Id: I85f4a91761293de1a782e197bc9e09db228933c9
2020-10-06 15:28:07 -04:00
Kaloyan Raev
bd177bff03 cmd/storj-sim: cleanup gateway setup
Remove usage of --non-interactive flag. It is not provided (and
necessary) by the multitenant S3 gateway anymore.

ACCESS_KEY and SECRET_KEY env vars are not provided anymore as they are
not generated by the multitenant S3 gateway.

Change-Id: I3ecfb92110e31ae13977de3899dad273daae6c1e
2020-10-06 14:22:47 +00:00
Stefan Benten
14a2050b8d pkg/auth: move package to consoleauth
To avoid further name collisions, the very broad named package gets moved into
the consoleauth package where its also mainly being used.

Change-Id: Ie563c9700adbf0553baca2b7b8ba4a1d9c29d144
2020-10-06 14:15:07 +02:00
Stefan Benten
44bd65795b satellite/console: ensure only project members can remove other project members
Change-Id: I815eb85f37631aaa65b5dc4cafa6851f241ca0f0
2020-10-06 11:03:12 +00:00
Stefan Benten
9deea2ffe2 satellite/console: disable account deletion via API
Change-Id: Ia8e43284c90fb2b833eb601e2c8f701cb5a4d9c0
2020-10-06 13:01:46 +02:00
Stefan Benten
338ed90876 Makefile: simplify binaries-upload
This change makes the loop to create the zip files a little easier (especially to read).

In addition, it also omits the .exe part for windows zip files, to ensure we can properly
use the versioncontrol URL for both windows and unix rollouts.

Change-Id: I0cbabb2ca61eb3346d97b2071411885383e870f3
2020-10-06 10:35:04 +02:00
Stefan Benten
0aaad88a44 satellite/{admin, console}: add test for projectLimit increase and update README
This change adds the capabilities to adjust the users project limit via the Admin API.
Adds a test for the new added function of the API and updates the existing tests.
It renames the json field on the user struct to be more consistent.

Change-Id: I9018acd80dae0af68d1d50526f20987132c654f3
2020-10-05 11:54:37 +00:00
NickolaiYurchenko
f27cee91b0 web/storagenode: 404 page rework
WHAT: new desist and test for 404 error page

WHY: 404 error page was hidden behind infinite loading screen
also, there were no dark theme deging for it

Change-Id: If9a1f5dd2321d9fc4a294637b8d5c041947cc9d1
2020-10-05 09:29:19 +00:00
Cameron Ayer
b39a99bae6 satellite/{overlay,satellitedb}: always show node's real online score
Previously if a node did not have audit history data for each of the
windows over the tracking period, we would give them the benefit of
the doubt and set their score to 1. This was to prevent nodes from
being suspended right out the gate. We need a minimum amount of data
to evaluate them.

However, a node who is actually failing at being online will have no
idea until they have received enough audits and we suspend them.

Instead, we will always use their real score, but use a flag to determine
whether they are eligible for suspension/dq.

Change-Id: I382218f12e8770f95d4bcddcf101ef348940cadf
2020-10-02 12:28:11 -04:00
NickolaiYurchenko
0d7ae8b182 web/storagenode: held history table: disposed subtracted from held
Change-Id: I11f3b019b3d1464abc932c8c7138b62f61924813
2020-10-02 15:35:06 +00:00
Qweder93
664b8f6821 storagenode/payout: estimation payout values switched from int64 to float64 to avoid incorrect rounding.
float64 values rounding to 2nd sign after dot.

Change-Id: Ice49f6a0944231ea6adb3343545bf1a62ff6dbc1
2020-10-02 11:33:43 +00:00
Yingrong Zhao
f13084983d bump common to latest
Change-Id: I55ca40a7b2c58d70225da07b7a62d5ad60451881
2020-10-01 15:39:09 -04:00
NickolaiYurchenko
406281fc13 web/storagenode: changed restriction to not show calendar in payuot history table [SG-516]
Change-Id: Ia1706c084f222e078b65ff34bb7d88a9d62c4ce2
2020-10-01 16:10:14 +00:00
Yingrong Zhao
c085a17a52 bump common and uplink to latest
Change-Id: I717f0214dd9973acd51b7732c5d64587f610c805
2020-10-01 15:38:58 +00:00
crawter
fa39274ffd multinode/multinodedb: add dbx support
We want to use dbx ORM in our codebase to have consistency with satellite codebase

Change-Id: I0646c9ef05db54a7b8e92c543ff8517bada2c5b0
2020-10-01 14:48:59 +00:00
Yaroslav Vorobiov
c4d5625965 cmd/storagenode-updater: fix service build tag, for platforms other than linux and windows
Change-Id: I7ed76b80a1df0ef53f2dd6726a602cc3bd4df9de
2020-10-01 15:32:28 +03:00
NickolaiYurchenko
3a7eb10c71 web: tests fix
Change-Id: I73cfb86fd214afb02be6a9e96e0ae5ffb92fd214
2020-10-01 13:06:28 +03:00
Egon Elbre
4e8d53c8fb private/dbutil/pgutil: ensure storagenode doesn't depend on pgx
pgx is a large dependency and there's no need to include it in
storagenode binary.

Change-Id: I49c304c6420733d5f095d7edb35d32811210e41a
2020-09-30 14:28:47 +00:00
Qweder93
245986d528 negative space calculations fix removed
Change-Id: I342c61856fce6d02dc99fd27fd3d563540f22b64
2020-09-30 14:08:24 +00:00
VitaliiShpital
80eeaf85db web/satellite: sign up link updated
WHAT:
sign up link in add user popup updated to redirect to correct page

WHY:
bug fix

Change-Id: I607d0b7d71bc6cc73b304d9ff8b8d4aca7549a45
2020-09-30 16:36:46 +03:00
Yaroslav
a43afa7ea0 make: fix storagenode-updater binary target
Adds -tags=service when building storagenode-updater which adds
interaction with Windows SVC and Linux systemd to the binary.

Change-Id: I21b4b3e8718348a30ec0453b07135ff3de033a5a
2020-09-30 11:53:11 +00:00
Yaroslav Vorobiov
a840cb71e7 storagenode: check db version before run
Change-Id: I912f63fd62f2bff10341346c28dfb92fcd683806
2020-09-30 10:58:09 +00:00
Egon Elbre
139ff3427b go.mod: bump dependencies
Change-Id: I2b1bf014a60ef6ddc4b1b546f58393c34ffe22fc
2020-09-30 11:28:37 +03:00
Moby von Briesen
000b1e6011 satellite/admin/project_test.go: Update TestCheckUsageLastMonthUnappliedInvoice
Test is failing because the mock date used in it is September 2020.
Update to 2030.

Change-Id: I4733487d0589dbeb0d2f3dd2abd88f767d59d0ca
2020-09-30 00:10:02 -04:00
Cameron Ayer
c2525ba2b5 satellite/{repair,satellitedb}: clean up healthy segments from repair queue at end of checker iteration
Repair workers prioritize the most unhealthy segments. This has the consequence that when we
finally begin to reach the end of the queue, a good portion of the remaining segments are
healthy again as their nodes have come back online. This makes it appear that there are more
injured segments than there actually are.

solution:
Any time the checker observes an injured segment it inserts it into the repair queue or
updates it if it already exists. Therefore, we can determine which segments are no longer
injured if they were not inserted or updated by the last checker iteration. To do this we
add a new column to the injured segments table, updated_at, which is set to the current time
when a segment is inserted or updated. At the end of the checker iteration, we can delete any
items where updated_at < checker start.

Change-Id: I76a98487a4a845fab2fbc677638a732a95057a94
2020-09-29 20:38:22 +00:00
Egon Elbre
c23a8e3b81 go.mod: update pgx to v4.9.0
Fix query to use TextArray instead of VarcharArray.
Fix queries to use the correct type.

Change-Id: Ibb7e55adba277d05778118d81ca697470e72c374
2020-09-29 19:03:08 +00:00
Michal Niewrzal
cd2a5484f3 storagenode/console: ignore untrusted satellite while returning
dashboard data and calculating satellites data

Change-Id: I71d596891477e0839863e007689b6e2e6e420a22
2020-09-29 18:27:49 +00:00
crawter
081fdcc835 multinode: initial infrastructure
Change-Id: Ibe31d555c144915acdcc26915deb68d6c4c3fbe9
2020-09-29 17:58:17 +03:00
Brandon Iglesias
8783c2024a
satellite/rewards: add partner videocoin (#3946) 2020-09-29 14:30:46 +02:00
VitaliiShpital
8499323ac5 web/satellite: project summary added to dashboard page
WHAT:
new project summary (details) section added to project dashboard page.
v1 shows team size, API keys amount, buckets amount, estimated charges for selected project for current month.

WHY:
redesign. Better user experience?

Change-Id: I31204a3e68db49486bad1e1a0eedd238eba6b84e
2020-09-29 11:16:50 +00:00
Kaloyan Raev
b409b53f7f cmd/satellite: command for verifying piece hashes
Jira: https://storjlabs.atlassian.net/browse/PG-69

There are a number of segments with piece_hashes_verified = false in
their metadata) on US-Central-1, Europe-West-1, and Asia-East-1
satellites. Most probably, this happened due to a bug we had in the
past. We want to verify them before executing the main migration to
metabase. This would simplify the main migration to metabase with one
less issue to think about.

Change-Id: I8831af1a254c560d45bb87d7104e49abd8242236
2020-09-29 10:58:24 +00:00
Egon Elbre
d508c4c985 scripts: remove duplicate check config lock
We have configlock_test for checking changes so the separate script and
makefile target are not needed.

Change-Id: I2bbc1c21ad849c9b7ec8bba43c0e11e94e04f6a6
2020-09-29 10:03:34 +00:00
Egon Elbre
2d27bc8787 satellite/satellitedb: separate cockroach for migration tests
Currently Cockroach migration test is the most heavy with regards to
schema changes. This causes other tests to time out. This adds an
alternate cockroach instance that is used for migration tests.

Change-Id: I01fe9313527ff002f0bb0914dd52c3645b8eaf6d
2020-09-29 09:31:33 +00:00
Stefan Benten
79eb682f9c satellite/console: allow coupons to be a valid payment option
Currently a user is only able to create a project if either
a STORJ deposit or CC was added to his account. With this change, an existing
coupon is also valid to let the user proceed.

Change-Id: I7be8d2d9ec58a15c50755b3fe33af04d2fd64ea2
2020-09-28 21:24:04 +00:00
Yaroslav Vorobiov
8786e55a78 storagenode/storagenodedb: allow existing dbs on setup
Allow existing storagenode dbs on setup to be able to reinstall
the node with existing data.

Change-Id: Ib42ab585432e61dfecc10640b6cd755ce83f0c46
2020-09-28 16:31:48 +03:00
Jessica Grebenschikov
4a2c66fa06 satellite/accounting: add cache for getting project storage and bw limits
This PR adds the following items:
1) an in-memory read-only cache thats stores project limit info for projectIDs

This cache is stored in-memory since this is expected to be a small amount of data. In this implementation we are only storing in the cache projects that have been accessed. Currently for the largest Satellite (eu-west) there is about 4500 total projects. So storing the storage limit (int64) and the bandwidth limit (int64), this would end up being about 200kb (including the 32 byte project ID) if all 4500 projectIDs were in the cache. So this all fits in memory for the time being. At some point it may not as usage grows, but that seems years out.

The cache is a read only cache. When requests come in to upload/download a file, we will read from the cache what the current limits are for that project. If the cache does not contain the projectID, it will get the info from the database (satellitedb project table), then add it to the cache.

The only time the values in the cache are modified is when either a) the project ID is not in the cache, or b) the item in the cache has expired (default 10mins), then the data gets refreshed out of the database. This occurs by default every 10 mins. This means that if we update the usage limits in the database, that change might not show up in the cache for 10 mins which mean it will not be reflected to limit end users uploading/downloading files for that time period..

Change-Id: I3fd7056cf963676009834fcbcf9c4a0922ca4a8f
2020-09-25 16:28:49 +00:00
Egon Elbre
888bfaae4b cmd/satellite: only add google profiler to satellite
Previously uplink, storagenode etc. included google cloud profiler,
however they don't need it.

Change-Id: Ibc95cb03d667a3844672eecd49fa455a6acc3866
2020-09-25 18:56:59 +03:00
René Smeekes
ff57b30bac
Raise warning to ensure early notification (#3945) 2020-09-24 17:53:38 +03:00
VitaliiShpital
e9ffffcaad web/satellite: nav bar colour fixes
WHAT:
navigation side bar colours changed.

WHY:
to correspond design.

Change-Id: I851cb0f801e632cf54e7d6b373494d782412a045
2020-09-24 14:43:35 +03:00
VitaliiShpital
ca1f0cfe4d web/satellite: dropdowns' working logic moved to local store
WHAT:
working logic of dropdowns moved to local store. Removed ability to open more than one dropdown. Dropdowns' colors changed to satisfy needed contrast ratio

WHY:
better user experience

Change-Id: Ia7a5683a3544fcb6bdd8a05d1fd6a12755b76caf
2020-09-24 11:18:13 +00:00
nerdatwork
870abd8676
storagenode/pieces: tidying trash log 2020-09-24 11:55:06 +03:00
Stefan Benten
9d0d0ad728 satellite/console: enable multiple projects all users
Change-Id: I42cc9f48cac387e1a67d21c1dd394f28cc5ff399
2020-09-23 16:18:28 +00:00
Stefan Benten
38108828ac satellite/satellitedb: enable multiple projects existing users
Change-Id: I2ef77182d5464d72574698c8abfbbfdbda3f5a9e
2020-09-23 18:17:38 +02:00
Stefan Benten
5f6fccc6e8 satellite/satellitedb: makes limits nullable change backwards compatible
Our current endpoints bail on us, if the column data is null. Thus we need
to take the intermediate step and set the default to a fixed value and
reset those with the following release.

It sets the default column value to our current config values of 50GB
for storage and bandwidth and 100 buckets, while still enabling the field to be nullable.
All 0 values are migrated to be the default as well to ensure they can
keep using their projects, as with the original change, 0 actually means 0.

Change-Id: I797be80ce2d2105091599dc1b3fc76f74336b66b
2020-09-23 17:54:42 +02:00
VitaliiShpital
8614d49e66 web/satellite: account dropdown reworked
WHAT:
account dropdown reworked to show user's full name and logout button

WHY:
GUI redesign

Change-Id: I129071ddcca5a05ebffb73db535240dfa498586e
2020-09-23 11:04:51 +00:00
Monty Anderson
7710e083d4
cmd/uplink: update CLI format (#3773) 2020-09-23 12:54:00 +02:00
Moby von Briesen
8287e3a32d storagenode/orders/store.go: combine writeLimit/writeOrder operations
Combine store.writeLimit and store.writeOrder into
store.writeLimitAndOrder, which only requires a single call to
file.Write(). This simplifies code, but it also reduces the likelihood
of multiple calls to Write() increasing the likelihood of file
corruption.

Also combine the corresponding readLimit/readOrder functions for
consistency.

Change-Id: I62ed406fa2c02708465a678d18293f510f666440
2020-09-22 17:53:12 +00:00
VitaliiShpital
1e5641aa55 web/satellite: unnecessary console error on no paywall user's login removed
WHAT:
in case when user was in no paywall cohourt, unnecessary error message appeared in console after login.
Error removed.

WHY:
better user experience.

Change-Id: Ieafd756e3dc0d6a10bf0058b52ebfaf38d02e8fd
2020-09-22 15:34:43 +00:00
nerdatwork
54dd430048
storagenode/pieces: fix typo for satellite id and piece id 2020-09-22 08:19:12 +03:00