Commit Graph

5860 Commits

Author SHA1 Message Date
Michał Niewrzał
1a974c327d satellite/metabase: set Segment.ExpiresAt while getting segment
Change-Id: I6504efd19c3516ffcf97ea3923e7f52e9832613b
2021-06-15 13:57:51 +00:00
JT Olio
6949dc0bac satellite/metaloop: missing monitoring on observers
Change-Id: I630fbb0448c8d08b426486b3e49abfbca03332a6
2021-06-15 13:39:13 +00:00
Jeff Wendling
ce87652a8c cmd/uplinkng: rm
this implements the rm command which has to add
a Remove method to the fileystem interface and
implement it for local, remote and test filesystems.

Change-Id: Id41add28f01938893530aae0b4b73c8954e9b715
2021-06-14 15:23:41 -04:00
Jeff Wendling
173d1e638c cmd/uplinkng: meta get
Change-Id: I6cd7e3fe408b42d257c35a6c715016f13559643f
2021-06-14 15:23:41 -04:00
Jeff Wendling
1323e1a666 cmd/uplinkng: rb
just implements the remove bucket command

Change-Id: I096a3c0c4a5fd1c7b018784d361221d1a1f5461a
2021-06-14 15:23:41 -04:00
Jeff Wendling
7fae5654ff cmd/uplinkng: access save: prompt for access
this adds a helper method to prompt for a line of
input using the clingy context to the global flag
state that errors if interactive mode is disabled.

Change-Id: Ie113c8920dfa4719e85cc24f11401d91b32812f9
2021-06-14 15:23:41 -04:00
Jeff Wendling
d73287f043 cmd/uplinkng: tests for cp
this adds some stuff to ultest so that the set of
files created by a test can be inspected after so
that we can write some tests for the cp command
to observe that it does what it is supposed to do.

Change-Id: I98b8fb214058140dfbb117baa7acea6a2cc340e1
2021-06-14 15:23:41 -04:00
Jeff Wendling
98be54b9a3 cmd/uplinkng: refactor into some focused packages
the directory was starting to get pretty large and
it was making it hard to pick concise names for
types and variables. this moves the location
stuff into a cmd/uplinkng/ulloc package, the
filesystem stuff into a cmd/uplinkng/ulfs package,
and the testing stuff into a cmd/uplinkng/ultest
package.

this should make the remaining stuff in cmd/uplinkng
only the business logic of how to implement the
commands, rather than also including a bunch of
helper utilities and scaffolding.

Change-Id: Id0901625ebfff9b1cf2dae52366aceb3b6c8f5b6
2021-06-14 15:23:41 -04:00
Jeff Wendling
b24ea2ead5 cmd/uplinkng: test framework and ls tests
this adds a test framework with fake implementations of a
filesystem so that unit tests can be written asserting
the output of different command invocations as well as
the effects they have on a hypothetical filesystem and
storj network.

it also implements the mb command lol

Change-Id: I134c7ea6bf34f46192956c274a96cb5df7632ac0
2021-06-14 15:23:41 -04:00
Egon Elbre
f1a9b45599 mod: update quic for Go 1.17rc1
Change-Id: I4218b695089189d5efac1101559a83079fb812cd
2021-06-14 10:14:36 +03:00
Yingrong Zhao
81dd7b2f37 storagenode/internalpb: fix protobuf generation for newer protobuf
version

Change-Id: I24cc5568b5f5c36f8c63062f79a4cb8d1a8f164e
2021-06-11 17:20:42 +00:00
Michał Niewrzał
9a113da361 satellite/metabase: expose expires_at with loops
Loops are using custom structs to provide
segments while iterating/looping so we need
to expose new field there too.

Change-Id: I12c8f4a01afeac171bf638d278253999fa90a8cb
2021-06-11 16:51:37 +00:00
Qweder93
5d70b6abef multinode/bandwidth: added monthly bandwidth summaries
montly bandwidth summaries for single/all nodes, single/all satellites added.

Change-Id: Ic384886c10622df74a4bd0645e2d7f2a85477644
2021-06-11 16:31:59 +00:00
Egon Elbre
f3a52d1da5 satellite/metabase/segmentloop: limit max interval
Ensure that we don't query too far in the history, which slows things
down.

Change-Id: Ia77aa522f7f4c5d43629d51bb9a51a49fab6fa14
2021-06-10 17:06:33 +00:00
Michał Niewrzał
bc79f01aaa satellite/metabase: set expires_at while committing segment
We added expires_at column to segments table and now
we need to populate this column while committing segment.
We still need to migrate existing segments with
separate tool.

Change-Id: Ibac8c63d97201dd98cc2cb9db385f4cb73bc3f7e
2021-06-10 16:32:28 +00:00
Yaroslav Vorobiov
157d3d980e multinode/console: storage usage and total storage usage
Change-Id: I4970275daf4a4a9c5d02aea6a205891869dd4eff
2021-06-10 16:01:41 +00:00
Vitalii Shpital
ed28fa3ff9 web/satellite: added loaders across all the UI. Removed most of the requests from initial load
Added loader spinners across all of the UI to be visible while data is being fetched.
Removed most of the requests from the initial load of the satellite dashboard.
Removed useless requests after creating of new projects.
This should make user's experience much more better since load time of the app is much lower than it was before.

Change-Id: Ib0941ad4eee6b3caf781d132062b55cb17703fe7
2021-06-10 15:16:52 +00:00
Jeremy Wharton
c27da95742
Update index.html to be consistent with deployed code (#4140)
This change is part of an ongoing effort to remove code from our branding repository.

Co-authored-by: Stefan Benten <mail@stefan-benten.de>
2021-06-10 09:28:57 -05:00
Yaroslav Vorobiov
b23a782c6b multinode/console: held amount summary use satellite address
Change-Id: I7816a263579831eaaeb808e3cdd49b3b7b0a07b8
2021-06-10 15:55:43 +03:00
Jeff Wendling
d674bc9c52 satellite/audit: include failing segment info in logs
Change-Id: I972fe19a2479f48bccc8a87a282467345a9dc1ec
2021-06-10 13:47:22 +03:00
Jeff Wendling
944bceabcd satellite/audit: fix reservoir sampling bias
Change-Id: Icc522fd86538b8182a1b7d42c1588c32a257acaf
2021-06-10 13:47:22 +03:00
Egon Elbre
1cd13adc1c ci: clearer initial build
We can run a lot of the code building and downloading steps in parallel.
We were using Makefile, however, that doesn't integrate with Jenkins,
which means, when there's a failure, you have no clue where.

This separates the concurrent build steps and makes it clear what we are
running concurrently.

Change-Id: I1c1dccd09dab67d2ef428a58d5eb9e004b11a74c
2021-06-09 12:30:31 +00:00
andriikotko
072dd24696
integration/ui: add UI integration tests
Introduce a new `integration/ui` package for integration tests.

`integration/ui/uitest` contains a helper package for running browser tests.
2021-06-09 14:39:38 +03:00
crawter
962f81433f web/multinode: operator entity extended with nodeID and undistributed amount
Change-Id: I683e53bc4c669728caae3521597a5fd2d6372921
2021-06-09 08:51:15 +00:00
crawter
b445e7d397 multinode/operators: operator entity extended with nodeID and undistributed amount
Change-Id: I423fe79b0aefe09d0b0ddab490d7a39e2b3f3249
2021-06-08 20:01:09 +03:00
crawter
be16f4b68b web/multinode: wallets logic
Change-Id: Ib9b469cb968978acb784252022e94d619f88e7a0
2021-06-08 16:39:11 +00:00
NickolaiYurchenko
c0b170149d web/multinode: wallets page markup
Change-Id: I01cb973196d4be8d5cc959a3129c238236736538
2021-06-08 18:47:43 +03:00
Fadila Khadar
21731ff8d0 satellite/metabase: add expires_at column to segments
Change-Id: If798198d87edb68cda8859ce2277fdfbd6f8380b
2021-06-07 19:11:20 +00:00
Egon Elbre
347f5f87e0 satellite/metabase/metaloop: limit as of system time
Currently we did not limit the "as of system time" for iterating over
objects table. Using just an interval would cause problems with the
tests. That could be overcome skipping that interval for tests
altogether, however, we should probably test those more to ensure that
GC stays working as intended.

This is a safer code, however, maybe not as straigthforward as it could
be.

Change-Id: I374f77783b2af42bb6da846735ceea20a7ce5e60
2021-06-07 13:01:06 +00:00
Fadila Khadar
3de9655b68 satellite/redis: used bandwidth key depends on day
The redis key associated to bandwidth usage depended on the current month.
This change makes it depends also on the day, so that we update bandwidth usage
daily to take into account changes associated to expired but not used allocated bandwidth.

It also add a test to to check that the allocated but not settled bandwidth is not counted after 2 days.

Change-Id: Iee9dbe3517cc3b9825438360b276a07a43dfbc64
2021-06-05 17:14:52 +00:00
Yaroslav Vorobiov
c9cfb5ed0c storagenode/retain: add more verbose monkit monitoring
Change-Id: Ibb9804268751b4b1842eb729bc510dba83e9b28b
2021-06-04 20:20:11 +00:00
Egon Elbre
e6fe9d209e satellite/metabase: capture iterator errors
It was possible for the iterator to silently ignore scanning, nextQuery
and close errors.

Change-Id: I7e44674d9eae53267a3ed649b7657d932743bf73
2021-06-04 22:46:30 +03:00
Egon Elbre
5044337440 satellite/metabase: add monitoring for objects and segments
Currently it's difficult to gather how many objects and segments are
being inserted. Adding separate monitoring counters make this easier.

Change-Id: I986cd82f03e99d2aa6fc76028255ee1090d1b294
2021-06-04 20:01:10 +03:00
Moby von Briesen
b17d684f40 web/satellite: Update login area to use latest production code
This is another step in removing code from our branding repository. The
code between our three prod satellites was almost identical - this
change just uses the html/scss for the prod satellites, and generalizes
a small portion of the html so that the satellite selection dropdown is
based on configuration rather than hardcoded.

There are significant differences in the code for non-prod satellites
like saltlake, eu-north, and us2. We can address this in a future
change.

Change-Id: I72a79f5b60670abb6340c62132b35ba3884d0750
2021-06-03 16:59:31 -04:00
Egon Elbre
cbf5939984 satellite/metabase/metaloop: fix objectsIterated metric
Change-Id: Id4f39e7524d1764cb729ab40895af32666c00204
2021-06-03 20:13:36 +03:00
Fadila Khadar
1d405f45ea satellite/orders: add egress_dead to project_bandwidth_daily_rollups
Migration step for adding a 'egress_dead' column to the project_bandwidth_daily_rollups.
It will be used to track bandwidth allocation that won't be consumned
as the corresponding order has already been processed and has a settled
bandwidth amount lower than the order limit (allocated bandwidth).

Change-Id: Ic07592e69292ae2076e69f6038bb0e0fae79b271
2021-06-03 16:05:56 +00:00
NickolaiYurchenko
8c0a0f019c web/multinode: payouts by node
vuex store types added, markup and logic connected, unused components hidden, tests added

Change-Id: I42c573be3d05fc13521033b089f3f2d3126c36f8
2021-06-03 15:04:59 +00:00
Egon Elbre
556b80e06b web/: ignore node_modules while linting css
For some reason in CI environment, stylelint started analyzing code in
node_modules.

Change-Id: I4fdac8e84a320d05506a01de753d19db57536bc4
2021-06-03 16:18:16 +03:00
JT Olio
9c46490f6f satellite/accounting/projectbwcleanup: more realistic test value
Change-Id: I2c2dd51f6f73e4e5f24268fe9e9fdb6d25e17ba0
2021-06-03 12:25:22 +00:00
NickolaiYurchenko
49662e98c2 web/multinode: node selection component added
node selection dropdown component, fixed minor bugs, bandwidth route added

Change-Id: I87fd1bf6c6a59d895b14524ab7c0fa5d80955cd5
2021-06-02 19:48:16 +03:00
Egon Elbre
4469d229f8 satellite/metabase/{meta,segment}loop: avoid passing config
Currently the iterate is being called in only one location so there's no
benefit in passing them as arguments over using the receiver.

Change-Id: I433a5d8b795b1bcc1f1e9320d87b10820cf537f1
2021-06-02 15:55:59 +00:00
Jeff Wendling
46a3242ed4 cmd/uplinkng: cp
Change-Id: I9c251028d9f72572f4d42815de31d44517cd5525
2021-06-02 14:35:27 +00:00
Malcolm Bouzi
136af8e630 web, satellite: allow registering business accounts to ask for contact from sales team
Full prefix: web/satellite, satellite/{console, analytics, satellitedb}

- checkbox added to register view - business tab
- user being saved with new column
- add sales contact choice to Segment calls
- ui fix added to employee count dropdown

Change-Id: Ib976872463b88874ea9714db635d58c79cdbe3a1
2021-06-02 13:10:38 +00:00
Egon Elbre
2cf10a7bf4 satellite/metabase/{segment,meta}loop: avoid no observers error
In a rare case it's possible to start the loop iteration without
observers. The most likely case is that the observer is cancelled and
the coalesce timer trigger asynchronously, although being stopped.

Nevertheless, all the observers may also exit during the iteration, in
either case it should not result in an error.

If there's a probem with the observers, then they can report their own
error as they see fit.

Change-Id: Ie423fec41e6295be05536a4b7b0b6623ffebf2fb
2021-06-02 11:57:59 +00:00
Vitalii Shpital
a04c96b1bf web/satellite: replace initial cloud loader with a new one
Replaced loading spinner of the loading state after login.
This is the first step of unifying of all the spinners across UI

Change-Id: I00bffb0c7113803ffcda580888f5ba6f425b16d4
2021-06-02 09:03:08 +00:00
Vitalii Shpital
c26d52a538 web/satellite: remove duplicated default project creation code
Moved creation of default project to vuex store to dispatch it in different places.
It removes duplicated code fragments

Change-Id: Ic5e52776727e681797bbd76342b78480812552d1
2021-06-02 08:43:44 +00:00
JT Olio
9e2d837473 testplanet: remove brittle config
Change-Id: I032c6a53bf32d1fa3a3e4cd8f46bc38d2e38d73c
2021-06-02 07:14:55 +00:00
Michał Niewrzał
d987990c15 satellite/satellitedb: removing usage of project_bandwidth_rollups table
We are not using this table so make no sense to put data there.
This change removes only code that is using this table. Before next
release we need to drop table with migration step.

Change-Id: I80f400aa778c717e70324bd00da502b7032c9d9f
2021-06-02 05:58:38 +00:00
JT Olio
da9ca0c650 testplanet/satellite: reduce the number of places default values need to be configured
Satellites set their configuration values to default values using
cfgstruct, however, it turns out our tests don't test these values
at all! Instead, they have a completely separate definition system
that is easy to forget about.

As is to be expected, these values have drifted, and it appears
in a few cases test planet is testing unreasonable values that we
won't see in production, or perhaps worse, features enabled in
production were missed and weren't enabled in testplanet.

This change makes it so all values are configured the same,
systematic way, so it's easy to see when test values are different
than dev values or release values, and it's less hard to forget
to enable features in testplanet.

In terms of reviewing, this change should be actually fairly
easy to review, considering private/testplanet/satellite.go keeps
the current config system and the new one and confirms that they
result in identical configurations, so you can be certain that
nothing was missed and the config is all correct.
You can also check the config lock to see what actual config
values changed.

Change-Id: I6715d0794887f577e21742afcf56fd2b9d12170e
2021-06-01 22:14:17 +00:00
prerna-parashar
75bf2ad586
satellite/analytics: Add analytics for "link shared" in objects view (#4130) 2021-06-01 14:44:37 -07:00