Commit Graph

4610 Commits

Author SHA1 Message Date
Brandon Iglesias
8783c2024a
satellite/rewards: add partner videocoin (#3946) 2020-09-29 14:30:46 +02:00
VitaliiShpital
8499323ac5 web/satellite: project summary added to dashboard page
WHAT:
new project summary (details) section added to project dashboard page.
v1 shows team size, API keys amount, buckets amount, estimated charges for selected project for current month.

WHY:
redesign. Better user experience?

Change-Id: I31204a3e68db49486bad1e1a0eedd238eba6b84e
2020-09-29 11:16:50 +00:00
Kaloyan Raev
b409b53f7f cmd/satellite: command for verifying piece hashes
Jira: https://storjlabs.atlassian.net/browse/PG-69

There are a number of segments with piece_hashes_verified = false in
their metadata) on US-Central-1, Europe-West-1, and Asia-East-1
satellites. Most probably, this happened due to a bug we had in the
past. We want to verify them before executing the main migration to
metabase. This would simplify the main migration to metabase with one
less issue to think about.

Change-Id: I8831af1a254c560d45bb87d7104e49abd8242236
2020-09-29 10:58:24 +00:00
Egon Elbre
d508c4c985 scripts: remove duplicate check config lock
We have configlock_test for checking changes so the separate script and
makefile target are not needed.

Change-Id: I2bbc1c21ad849c9b7ec8bba43c0e11e94e04f6a6
2020-09-29 10:03:34 +00:00
Egon Elbre
2d27bc8787 satellite/satellitedb: separate cockroach for migration tests
Currently Cockroach migration test is the most heavy with regards to
schema changes. This causes other tests to time out. This adds an
alternate cockroach instance that is used for migration tests.

Change-Id: I01fe9313527ff002f0bb0914dd52c3645b8eaf6d
2020-09-29 09:31:33 +00:00
Stefan Benten
79eb682f9c satellite/console: allow coupons to be a valid payment option
Currently a user is only able to create a project if either
a STORJ deposit or CC was added to his account. With this change, an existing
coupon is also valid to let the user proceed.

Change-Id: I7be8d2d9ec58a15c50755b3fe33af04d2fd64ea2
2020-09-28 21:24:04 +00:00
Yaroslav Vorobiov
8786e55a78 storagenode/storagenodedb: allow existing dbs on setup
Allow existing storagenode dbs on setup to be able to reinstall
the node with existing data.

Change-Id: Ib42ab585432e61dfecc10640b6cd755ce83f0c46
2020-09-28 16:31:48 +03:00
Jessica Grebenschikov
4a2c66fa06 satellite/accounting: add cache for getting project storage and bw limits
This PR adds the following items:
1) an in-memory read-only cache thats stores project limit info for projectIDs

This cache is stored in-memory since this is expected to be a small amount of data. In this implementation we are only storing in the cache projects that have been accessed. Currently for the largest Satellite (eu-west) there is about 4500 total projects. So storing the storage limit (int64) and the bandwidth limit (int64), this would end up being about 200kb (including the 32 byte project ID) if all 4500 projectIDs were in the cache. So this all fits in memory for the time being. At some point it may not as usage grows, but that seems years out.

The cache is a read only cache. When requests come in to upload/download a file, we will read from the cache what the current limits are for that project. If the cache does not contain the projectID, it will get the info from the database (satellitedb project table), then add it to the cache.

The only time the values in the cache are modified is when either a) the project ID is not in the cache, or b) the item in the cache has expired (default 10mins), then the data gets refreshed out of the database. This occurs by default every 10 mins. This means that if we update the usage limits in the database, that change might not show up in the cache for 10 mins which mean it will not be reflected to limit end users uploading/downloading files for that time period..

Change-Id: I3fd7056cf963676009834fcbcf9c4a0922ca4a8f
2020-09-25 16:28:49 +00:00
Egon Elbre
888bfaae4b cmd/satellite: only add google profiler to satellite
Previously uplink, storagenode etc. included google cloud profiler,
however they don't need it.

Change-Id: Ibc95cb03d667a3844672eecd49fa455a6acc3866
2020-09-25 18:56:59 +03:00
René Smeekes
ff57b30bac
Raise warning to ensure early notification (#3945) 2020-09-24 17:53:38 +03:00
VitaliiShpital
e9ffffcaad web/satellite: nav bar colour fixes
WHAT:
navigation side bar colours changed.

WHY:
to correspond design.

Change-Id: I851cb0f801e632cf54e7d6b373494d782412a045
2020-09-24 14:43:35 +03:00
VitaliiShpital
ca1f0cfe4d web/satellite: dropdowns' working logic moved to local store
WHAT:
working logic of dropdowns moved to local store. Removed ability to open more than one dropdown. Dropdowns' colors changed to satisfy needed contrast ratio

WHY:
better user experience

Change-Id: Ia7a5683a3544fcb6bdd8a05d1fd6a12755b76caf
2020-09-24 11:18:13 +00:00
nerdatwork
870abd8676
storagenode/pieces: tidying trash log 2020-09-24 11:55:06 +03:00
Stefan Benten
9d0d0ad728 satellite/console: enable multiple projects all users
Change-Id: I42cc9f48cac387e1a67d21c1dd394f28cc5ff399
2020-09-23 16:18:28 +00:00
Stefan Benten
38108828ac satellite/satellitedb: enable multiple projects existing users
Change-Id: I2ef77182d5464d72574698c8abfbbfdbda3f5a9e
2020-09-23 18:17:38 +02:00
Stefan Benten
5f6fccc6e8 satellite/satellitedb: makes limits nullable change backwards compatible
Our current endpoints bail on us, if the column data is null. Thus we need
to take the intermediate step and set the default to a fixed value and
reset those with the following release.

It sets the default column value to our current config values of 50GB
for storage and bandwidth and 100 buckets, while still enabling the field to be nullable.
All 0 values are migrated to be the default as well to ensure they can
keep using their projects, as with the original change, 0 actually means 0.

Change-Id: I797be80ce2d2105091599dc1b3fc76f74336b66b
2020-09-23 17:54:42 +02:00
VitaliiShpital
8614d49e66 web/satellite: account dropdown reworked
WHAT:
account dropdown reworked to show user's full name and logout button

WHY:
GUI redesign

Change-Id: I129071ddcca5a05ebffb73db535240dfa498586e
2020-09-23 11:04:51 +00:00
Monty Anderson
7710e083d4
cmd/uplink: update CLI format (#3773) 2020-09-23 12:54:00 +02:00
Moby von Briesen
8287e3a32d storagenode/orders/store.go: combine writeLimit/writeOrder operations
Combine store.writeLimit and store.writeOrder into
store.writeLimitAndOrder, which only requires a single call to
file.Write(). This simplifies code, but it also reduces the likelihood
of multiple calls to Write() increasing the likelihood of file
corruption.

Also combine the corresponding readLimit/readOrder functions for
consistency.

Change-Id: I62ed406fa2c02708465a678d18293f510f666440
2020-09-22 17:53:12 +00:00
VitaliiShpital
1e5641aa55 web/satellite: unnecessary console error on no paywall user's login removed
WHAT:
in case when user was in no paywall cohourt, unnecessary error message appeared in console after login.
Error removed.

WHY:
better user experience.

Change-Id: Ieafd756e3dc0d6a10bf0058b52ebfaf38d02e8fd
2020-09-22 15:34:43 +00:00
nerdatwork
54dd430048
storagenode/pieces: fix typo for satellite id and piece id 2020-09-22 08:19:12 +03:00
Stefan Benten
2f648fd981 satellite: make limits be nullable
Currently we have no way to actually set one
of the following limits to 0 (meaning not usable):

- maxBuckets
- usageLimit
- bandwidthLimit

With having the field nullable,
NULL corresponds to the global default,
0 now actually 0 and
a set value determines a custom limit.

Change-Id: I92bb77529dcbd0881ae8368921be9d246eb0919e
2020-09-21 19:34:19 +00:00
Kaloyan Raev
34613e4588 cmd/satellite: command for fixing old-style objects
Jira: https://storjlabs.atlassian.net/browse/PG-67

There are a number of old-style objects (without the number of segments
in their metadata) on US-Central-1, Europe-West-1, and Asia-East-1
satellites. We want to migrate their metadata to contain the number of
segments before executing the main migration to metabase. This would
simplify the main migration to metabase with one less issue to think
about.

Change-Id: I42497ae0375b5eb972aab08c700048b9a93bb18f
2020-09-21 14:46:19 +00:00
VitaliiShpital
2bdbe2124a web/satellite: stored selected project feature
WHAT:
lastly selected project is now stored in browser's local storage and is being selected automatically on data load (login/refresh)

WHY:
better user experience for multiproject state

Change-Id: Idadbb75c8391017ee1845415bec282d9e33309eb
2020-09-18 15:44:18 +03:00
nerdatwork
96ec44ff1b
storagenode/pieces: make log more legible 2020-09-18 15:10:13 +03:00
NickolaiYurchenko
92a336cb5a web/storagenode: held returned block added
WHAT: held information block on payout page

WHY: to show held and held disposed amounts

Change-Id: I34b8f17993f93d7fdbc65021d0a088c8a5490f8d
2020-09-17 12:09:27 +03:00
Matt Robinson
84b6b91ee1
build: Go 1.15.2 2020-09-17 07:57:20 +03:00
JT Olio
f46161cf20 consoleweb: log index template failures
Change-Id: I286ded309fed6198f1c450a2a31df36b6a015551
2020-09-16 16:26:14 +00:00
VitaliiShpital
2668ec818e web/satellite: added ability to edit project name
WHAT:
added edit project dropdown to navigation side bar.
edit project details page implemented.
added ability to edit project name.
project details section removed from project dashboard.

WHY:
enable users to change their project name.

Change-Id: I36b6214ffe7adf4a12a1a09530ff1212e926aafe
2020-09-16 18:46:59 +03:00
Qweder93
8182fdad0b storagenode: heldamount renamed to payouts, renamed some methods and structs to more meaningful names. grouped estimated payout with pathouts
satellite: heldamount renamed to SNOpayouts.

Change-Id: I244b4d2454e0621f4b8e22d3c0d3e602c0bbcb02
2020-09-16 14:57:35 +00:00
VitaliiShpital
3bbf08917c web/satellite: node packages updated
WHAT:
node packages updated to be up to date

Change-Id: Ic5a44c548562f16b23da59d3c39c4634572f7b35
2020-09-16 12:37:21 +00:00
Michal Niewrzal
d199daa907 satellite/metainfo: replace local constants with metabase values
Change-Id: Ie855c10dd259464658952186251b7d3210eb49ce
2020-09-16 13:25:14 +02:00
NickolaiYurchenko
9a627b22e8 web/storagenode: frontend refactoring
WHAT:
separation of domain and presentation entities

WHY:
to have ability to easily reuse logic

Change-Id: I48f7d1831c217dec999ff1b678034bb2c10cfbaa
2020-09-16 09:14:12 +00:00
Moby von Briesen
7db5794c16 storagenode/orders/store: Do not lock order enqueues for entire duration of ListUnsentBySatellite
We only need to lock aquire mutexes inside ListUnsentBySatellite when we
want to determine whether a file has an active enqueue in progress.
On some nodes, ListUnsentBySatellite can take a particularly long time, having
undesired side-effects, so if we can minimize locking time, those nodes
will be better off.

Also, lock archive mu during ListUnsentBySatellite so files cannot be
archived and listed at the same time.

Change-Id: Ieb7e2a759c20c724a74dd8315728c873ccab14a3
2020-09-15 15:15:30 +00:00
VitaliiShpital
7d5e0259f6 satellite/projects: initial update project name functionality implemented
WHAT:
added functionality for user to update project name. Logic only, without actual GUI updates.

WHY:
better user experience

Change-Id: I1e38e33ba827b0bdf2c89e29de24e4e87edb474a
2020-09-15 12:21:56 +03:00
Cameron Ayer
e7c34a053d satellite/satellitedb: add column and index "updated_at" to injuredsegments
Change-Id: I59e9bb2077885f09e17795375fe98ed31bd83d54
2020-09-14 12:53:04 -04:00
Qweder93
528aa76ae6 storagenode/payouts: payoutHistoryMonthly surge reworked, empty receipt now won't return error
Change-Id: If99f8aec102550cd30e5906f986a4417903100be
2020-09-14 18:19:17 +03:00
paul cannon
5b9fd4e50a cmd/storj-sim: fix prefix writer infinite loop bug
Fix case where line is larger than maxline and writer ended up in an
infinite loop trying to buffer more data.

Change-Id: I243da738b331279d6bf27255778b5798e7f37f95
2020-09-11 12:33:04 +00:00
Moby von Briesen
789b07e226 storagenode/orders/store.go: Do not return error from ListUnsentBySatellite when order files are corrupted.
If we see an UnexpectedEOF error when attempting to read orders, return
the orders we have been able to read successfully and do not return an
error. This behavior ensures that the storagenode orders service
attempts to archive corrupted files and does not retry them repeatedly
and get stuck.

Change-Id: I0d00d1e174f968af6e99ca861eddad190f1339e2
2020-09-10 23:36:05 +00:00
JT Olio
1a094fd6da storj: bump storj.io/private
Change-Id: Ia2927010332bb2e1d140603299341a8549093233
2020-09-10 16:21:12 -06:00
VitaliiShpital
df5a6ebe32 web/satellite: navigation sidebar and dashboard header reworked
WHAT:
Navigation sidebar reworked.
Project dropdown moved back to header, resources and settings sections moved to header as dropdowns.
Header reworked to match new design.

WHY:
GUI redesign

Change-Id: I73e1269326d3dbfba2e44d848dba4a415e64e539
2020-09-10 10:38:00 +00:00
Qweder93
ac29d80495 storagenode: heldamount GetPaystub refactored, estimationPayouts logic separated form console to separate service, storagenodeapi tests fixed.
Change-Id: I902823ef40a62861ce32799e9fb7a67a1e14710d
2020-09-09 15:31:16 +00:00
Jeff Wendling
1fecaed7df satellite/orders: don't version check old endpoint
nodes are submitting using both the legacy and windowed endpoints
and thus having their legacy submissions rejected.

it is legal to use both the legacy and windowed endpoints
in phase1 since they use the same backend. the legacy endpoint
is disabled in phase2 and phase3.

therefore, if we wait an order expiration period (2 days) after
we determine enough nodes have started using the windowed
endpoint, we can be sure that any orders they did have to
submit with the legacy endpoint will have expired.

Change-Id: I4418a881bf8bb9377efaef4c651e6103a5dc6ed0
2020-09-09 10:23:48 -04:00
stefanbenten
2cf12b9409 satellite/admin: use correct period end time to get results
Change-Id: Ib0e76f96e61b39c98f0f5f683b39d622d235c4d8
2020-09-09 09:01:42 +00:00
Michal Niewrzal
27a9d14e2a satellite/repair: use metabase.SegmentKey type in repair package
Another change which is a part of refactoring to replace path parameter
(string/[]byte) with key paramter (metabase.SegmentKey)

Change-Id: I617878442442e5d59bbe5c995f913c3c93c16928
2020-09-08 19:35:20 +00:00
Michal Niewrzal
c753d17e8f satellite/metainfo/objectdeletion: replace
objectdeletion.ObjectIdentifier with metabase.ObjectLocation

Another change to use metabase.ObjectLocation across satellite codebase
to avoid duplication and provide better type safety.

Change-Id: I82cb52b94a9107ed3144255a6ef4ad9f3fc1ca63
2020-09-08 21:09:34 +02:00
VitaliiShpital
c4d6f472fc web/satellite: notification bar for reaching projects count limit
WHAT:
notification bar added to project dashboard page. It is shown when projects count limit is reached.
Create project button is removed after creating last available project

WHY:
inform user that their projects count limit was reached

Change-Id: If0d67148003be40cc9eb4d8b25cc17f8204008d4
2020-09-08 15:48:27 +00:00
Egon Elbre
9fd97fa973 ci: add check-monkit
Change-Id: I124d013ffb434fdea8ec315face090de4e572ebf
2020-09-08 17:20:27 +03:00
JT Olio
249f98f4c1 audit logging: log important user activity
Change-Id: Ic423dab4b6c513fa44a48cb6fe2903c70fe916fb
2020-09-07 15:26:34 +00:00
Matt Robinson
9996aeca8d
build: Go 1.15 (#3927) 2020-09-07 16:01:29 +03:00