Commit Graph

5551 Commits

Author SHA1 Message Date
Egon Elbre
f1a9b45599 mod: update quic for Go 1.17rc1
Change-Id: I4218b695089189d5efac1101559a83079fb812cd
2021-06-14 10:14:36 +03:00
Yingrong Zhao
81dd7b2f37 storagenode/internalpb: fix protobuf generation for newer protobuf
version

Change-Id: I24cc5568b5f5c36f8c63062f79a4cb8d1a8f164e
2021-06-11 17:20:42 +00:00
Michał Niewrzał
9a113da361 satellite/metabase: expose expires_at with loops
Loops are using custom structs to provide
segments while iterating/looping so we need
to expose new field there too.

Change-Id: I12c8f4a01afeac171bf638d278253999fa90a8cb
2021-06-11 16:51:37 +00:00
Qweder93
5d70b6abef multinode/bandwidth: added monthly bandwidth summaries
montly bandwidth summaries for single/all nodes, single/all satellites added.

Change-Id: Ic384886c10622df74a4bd0645e2d7f2a85477644
2021-06-11 16:31:59 +00:00
Egon Elbre
f3a52d1da5 satellite/metabase/segmentloop: limit max interval
Ensure that we don't query too far in the history, which slows things
down.

Change-Id: Ia77aa522f7f4c5d43629d51bb9a51a49fab6fa14
2021-06-10 17:06:33 +00:00
Michał Niewrzał
bc79f01aaa satellite/metabase: set expires_at while committing segment
We added expires_at column to segments table and now
we need to populate this column while committing segment.
We still need to migrate existing segments with
separate tool.

Change-Id: Ibac8c63d97201dd98cc2cb9db385f4cb73bc3f7e
2021-06-10 16:32:28 +00:00
Yaroslav Vorobiov
157d3d980e multinode/console: storage usage and total storage usage
Change-Id: I4970275daf4a4a9c5d02aea6a205891869dd4eff
2021-06-10 16:01:41 +00:00
Vitalii Shpital
ed28fa3ff9 web/satellite: added loaders across all the UI. Removed most of the requests from initial load
Added loader spinners across all of the UI to be visible while data is being fetched.
Removed most of the requests from the initial load of the satellite dashboard.
Removed useless requests after creating of new projects.
This should make user's experience much more better since load time of the app is much lower than it was before.

Change-Id: Ib0941ad4eee6b3caf781d132062b55cb17703fe7
2021-06-10 15:16:52 +00:00
Jeremy Wharton
c27da95742
Update index.html to be consistent with deployed code (#4140)
This change is part of an ongoing effort to remove code from our branding repository.

Co-authored-by: Stefan Benten <mail@stefan-benten.de>
2021-06-10 09:28:57 -05:00
Yaroslav Vorobiov
b23a782c6b multinode/console: held amount summary use satellite address
Change-Id: I7816a263579831eaaeb808e3cdd49b3b7b0a07b8
2021-06-10 15:55:43 +03:00
Jeff Wendling
d674bc9c52 satellite/audit: include failing segment info in logs
Change-Id: I972fe19a2479f48bccc8a87a282467345a9dc1ec
2021-06-10 13:47:22 +03:00
Jeff Wendling
944bceabcd satellite/audit: fix reservoir sampling bias
Change-Id: Icc522fd86538b8182a1b7d42c1588c32a257acaf
2021-06-10 13:47:22 +03:00
Egon Elbre
1cd13adc1c ci: clearer initial build
We can run a lot of the code building and downloading steps in parallel.
We were using Makefile, however, that doesn't integrate with Jenkins,
which means, when there's a failure, you have no clue where.

This separates the concurrent build steps and makes it clear what we are
running concurrently.

Change-Id: I1c1dccd09dab67d2ef428a58d5eb9e004b11a74c
2021-06-09 12:30:31 +00:00
andriikotko
072dd24696
integration/ui: add UI integration tests
Introduce a new `integration/ui` package for integration tests.

`integration/ui/uitest` contains a helper package for running browser tests.
2021-06-09 14:39:38 +03:00
crawter
962f81433f web/multinode: operator entity extended with nodeID and undistributed amount
Change-Id: I683e53bc4c669728caae3521597a5fd2d6372921
2021-06-09 08:51:15 +00:00
crawter
b445e7d397 multinode/operators: operator entity extended with nodeID and undistributed amount
Change-Id: I423fe79b0aefe09d0b0ddab490d7a39e2b3f3249
2021-06-08 20:01:09 +03:00
crawter
be16f4b68b web/multinode: wallets logic
Change-Id: Ib9b469cb968978acb784252022e94d619f88e7a0
2021-06-08 16:39:11 +00:00
NickolaiYurchenko
c0b170149d web/multinode: wallets page markup
Change-Id: I01cb973196d4be8d5cc959a3129c238236736538
2021-06-08 18:47:43 +03:00
Fadila Khadar
21731ff8d0 satellite/metabase: add expires_at column to segments
Change-Id: If798198d87edb68cda8859ce2277fdfbd6f8380b
2021-06-07 19:11:20 +00:00
Egon Elbre
347f5f87e0 satellite/metabase/metaloop: limit as of system time
Currently we did not limit the "as of system time" for iterating over
objects table. Using just an interval would cause problems with the
tests. That could be overcome skipping that interval for tests
altogether, however, we should probably test those more to ensure that
GC stays working as intended.

This is a safer code, however, maybe not as straigthforward as it could
be.

Change-Id: I374f77783b2af42bb6da846735ceea20a7ce5e60
2021-06-07 13:01:06 +00:00
Fadila Khadar
3de9655b68 satellite/redis: used bandwidth key depends on day
The redis key associated to bandwidth usage depended on the current month.
This change makes it depends also on the day, so that we update bandwidth usage
daily to take into account changes associated to expired but not used allocated bandwidth.

It also add a test to to check that the allocated but not settled bandwidth is not counted after 2 days.

Change-Id: Iee9dbe3517cc3b9825438360b276a07a43dfbc64
2021-06-05 17:14:52 +00:00
Yaroslav Vorobiov
c9cfb5ed0c storagenode/retain: add more verbose monkit monitoring
Change-Id: Ibb9804268751b4b1842eb729bc510dba83e9b28b
2021-06-04 20:20:11 +00:00
Egon Elbre
e6fe9d209e satellite/metabase: capture iterator errors
It was possible for the iterator to silently ignore scanning, nextQuery
and close errors.

Change-Id: I7e44674d9eae53267a3ed649b7657d932743bf73
2021-06-04 22:46:30 +03:00
Egon Elbre
5044337440 satellite/metabase: add monitoring for objects and segments
Currently it's difficult to gather how many objects and segments are
being inserted. Adding separate monitoring counters make this easier.

Change-Id: I986cd82f03e99d2aa6fc76028255ee1090d1b294
2021-06-04 20:01:10 +03:00
Moby von Briesen
b17d684f40 web/satellite: Update login area to use latest production code
This is another step in removing code from our branding repository. The
code between our three prod satellites was almost identical - this
change just uses the html/scss for the prod satellites, and generalizes
a small portion of the html so that the satellite selection dropdown is
based on configuration rather than hardcoded.

There are significant differences in the code for non-prod satellites
like saltlake, eu-north, and us2. We can address this in a future
change.

Change-Id: I72a79f5b60670abb6340c62132b35ba3884d0750
2021-06-03 16:59:31 -04:00
Egon Elbre
cbf5939984 satellite/metabase/metaloop: fix objectsIterated metric
Change-Id: Id4f39e7524d1764cb729ab40895af32666c00204
2021-06-03 20:13:36 +03:00
Fadila Khadar
1d405f45ea satellite/orders: add egress_dead to project_bandwidth_daily_rollups
Migration step for adding a 'egress_dead' column to the project_bandwidth_daily_rollups.
It will be used to track bandwidth allocation that won't be consumned
as the corresponding order has already been processed and has a settled
bandwidth amount lower than the order limit (allocated bandwidth).

Change-Id: Ic07592e69292ae2076e69f6038bb0e0fae79b271
2021-06-03 16:05:56 +00:00
NickolaiYurchenko
8c0a0f019c web/multinode: payouts by node
vuex store types added, markup and logic connected, unused components hidden, tests added

Change-Id: I42c573be3d05fc13521033b089f3f2d3126c36f8
2021-06-03 15:04:59 +00:00
Egon Elbre
556b80e06b web/: ignore node_modules while linting css
For some reason in CI environment, stylelint started analyzing code in
node_modules.

Change-Id: I4fdac8e84a320d05506a01de753d19db57536bc4
2021-06-03 16:18:16 +03:00
JT Olio
9c46490f6f satellite/accounting/projectbwcleanup: more realistic test value
Change-Id: I2c2dd51f6f73e4e5f24268fe9e9fdb6d25e17ba0
2021-06-03 12:25:22 +00:00
NickolaiYurchenko
49662e98c2 web/multinode: node selection component added
node selection dropdown component, fixed minor bugs, bandwidth route added

Change-Id: I87fd1bf6c6a59d895b14524ab7c0fa5d80955cd5
2021-06-02 19:48:16 +03:00
Egon Elbre
4469d229f8 satellite/metabase/{meta,segment}loop: avoid passing config
Currently the iterate is being called in only one location so there's no
benefit in passing them as arguments over using the receiver.

Change-Id: I433a5d8b795b1bcc1f1e9320d87b10820cf537f1
2021-06-02 15:55:59 +00:00
Jeff Wendling
46a3242ed4 cmd/uplinkng: cp
Change-Id: I9c251028d9f72572f4d42815de31d44517cd5525
2021-06-02 14:35:27 +00:00
Malcolm Bouzi
136af8e630 web, satellite: allow registering business accounts to ask for contact from sales team
Full prefix: web/satellite, satellite/{console, analytics, satellitedb}

- checkbox added to register view - business tab
- user being saved with new column
- add sales contact choice to Segment calls
- ui fix added to employee count dropdown

Change-Id: Ib976872463b88874ea9714db635d58c79cdbe3a1
2021-06-02 13:10:38 +00:00
Egon Elbre
2cf10a7bf4 satellite/metabase/{segment,meta}loop: avoid no observers error
In a rare case it's possible to start the loop iteration without
observers. The most likely case is that the observer is cancelled and
the coalesce timer trigger asynchronously, although being stopped.

Nevertheless, all the observers may also exit during the iteration, in
either case it should not result in an error.

If there's a probem with the observers, then they can report their own
error as they see fit.

Change-Id: Ie423fec41e6295be05536a4b7b0b6623ffebf2fb
2021-06-02 11:57:59 +00:00
Vitalii Shpital
a04c96b1bf web/satellite: replace initial cloud loader with a new one
Replaced loading spinner of the loading state after login.
This is the first step of unifying of all the spinners across UI

Change-Id: I00bffb0c7113803ffcda580888f5ba6f425b16d4
2021-06-02 09:03:08 +00:00
Vitalii Shpital
c26d52a538 web/satellite: remove duplicated default project creation code
Moved creation of default project to vuex store to dispatch it in different places.
It removes duplicated code fragments

Change-Id: Ic5e52776727e681797bbd76342b78480812552d1
2021-06-02 08:43:44 +00:00
JT Olio
9e2d837473 testplanet: remove brittle config
Change-Id: I032c6a53bf32d1fa3a3e4cd8f46bc38d2e38d73c
2021-06-02 07:14:55 +00:00
Michał Niewrzał
d987990c15 satellite/satellitedb: removing usage of project_bandwidth_rollups table
We are not using this table so make no sense to put data there.
This change removes only code that is using this table. Before next
release we need to drop table with migration step.

Change-Id: I80f400aa778c717e70324bd00da502b7032c9d9f
2021-06-02 05:58:38 +00:00
JT Olio
da9ca0c650 testplanet/satellite: reduce the number of places default values need to be configured
Satellites set their configuration values to default values using
cfgstruct, however, it turns out our tests don't test these values
at all! Instead, they have a completely separate definition system
that is easy to forget about.

As is to be expected, these values have drifted, and it appears
in a few cases test planet is testing unreasonable values that we
won't see in production, or perhaps worse, features enabled in
production were missed and weren't enabled in testplanet.

This change makes it so all values are configured the same,
systematic way, so it's easy to see when test values are different
than dev values or release values, and it's less hard to forget
to enable features in testplanet.

In terms of reviewing, this change should be actually fairly
easy to review, considering private/testplanet/satellite.go keeps
the current config system and the new one and confirms that they
result in identical configurations, so you can be certain that
nothing was missed and the config is all correct.
You can also check the config lock to see what actual config
values changed.

Change-Id: I6715d0794887f577e21742afcf56fd2b9d12170e
2021-06-01 22:14:17 +00:00
prerna-parashar
75bf2ad586
satellite/analytics: Add analytics for "link shared" in objects view (#4130) 2021-06-01 14:44:37 -07:00
Cameron Ayer
53322bb0a7 satellite/{audit,satellitedb}: release nodes from containment in Reverify rather than (Batch)UpdateStats
Until now, whenever audits were recorded we would try to delete
the node from containment just in case it exists. Since we now
want to treat segment repair downloads as audits, this would
erroneously remove nodes from containment, as repair does not go
through a Reverify step. With this changeset, (Batch)UpdateStats
will not remove nodes from containment. The Reverify method will
remove all necessary nodes from containment.

Change-Id: Iabc9496293076dccba32ddfa028e92580b26167f
2021-06-01 21:02:44 +00:00
Michał Niewrzał
e76cbc9bd5 satellite/gc: move GC to segments loop
This change is refactor to move GC from metainfo loop
(objects/segments)  to segments loop.

Change-Id: I21f1ff7cb0b6f98c41aa8930447b8d9bea227975
2021-06-01 20:36:02 +00:00
JT Olio
b07a39bfea satellite: log check in success node id
This is so we can see what's going on if we get a weird node DoS thing again

Change-Id: I5a14c95277562e496fcefb6d368068a6ec1dbc9f
2021-06-01 19:35:24 +00:00
paul cannon
8675cd178c cmd/uplink: add flags to 'put' subcommand
This enables use of the --progress, --expires, and --metadata flags with
'uplink put'. These flags work similarly to their counterparts in the
'uplink cp' command.

Small difference from cp: "progress" is on by default in 'uplink cp',
but (for backwards compatibility) off by default in 'uplink put'.

Why: Requested by @Toyoo on the forum:
https://forum.storj.io/t/explicit-data-expiration-questions/13854/2

..and it's an easy addition to make.

Change-Id: Id20aedd3a280db43e4883338f92f6beec7a400de
2021-06-01 19:16:22 +00:00
paul cannon
fbd11ba4e9 cmd/uplink: add expanded format for 'uplink ls'
This expanded format shows expiration times for objects and how much
custom metadata each object has.

This commit also organizes output formatting code together in one
section for simpler adjustments in the future.

Change-Id: Ica041c8a1de6ee73c104a0554c5c259e447536c4
2021-06-01 18:50:27 +00:00
Qweder93
d8c11a79b9 multinode/payouts: paystub extended with disposed
added Disposed to paystub struct to count NetTotal value.

Change-Id: Iaae1f98a69b82166ba5594e8589c7d2a540bd07f
2021-06-01 16:28:20 +00:00
crawter
4af0037a67 multinode/console: operators controller added
Change-Id: I170371baec3c6996bd2af3c332620bd6fee3ed63
2021-06-01 19:01:14 +03:00
crawter
0b3ee4bda7 multinode/operators: domain entities and service created
Change-Id: I146229df0a279b95ce4d3ffec658509ccd6ff59a
2021-06-01 16:51:48 +03:00
Michał Niewrzał
053e58b683 satellite/metabase: add segmentloop service
We want to move some of current metainfo loop observers to
segment loop. This change adds new service, similar to metainfo
loop but which is iterating only over segments.

Change-Id: I67f7f461781723a4476e2b83377f31736d7c4870
2021-06-01 11:15:07 +00:00