Fixed daily usage query returning single bucket usage.
We sum up bucket usages now.
Also fixed https://github.com/storj/storj/issues/4559.
Change-Id: I2eb6299f1ef500d68150879195011b6fbb5f37ed
The admin UI assets aren't inside of the `web` directory they are
directly in the `satellite` one.
The invalid path provoked that storj-sim generated a satellite
configuration with an invalid path to the Admin UI static assets
provoking that it didn't load the UI by default.
Change-Id: I49fb289377f51634057173690fbd8cf863ca9a9d
TRUNCATE requires table recreation which involves 'online schema change' with crdb.
(with psql it might be fater than DROP, that was the motivation of the original change)
`online schema change` is an async operation with crdb and it's eventually very slow therefore we try to avoid it.
This·reverts·commit·15bed0ed0e81d54fe4ffac9928bdf648f5e06ec6.¬
Change-Id: I93e1ab470962be77e3458d74c8787442c9d7bee0
This patch suggests small improvements for the previously committed (very nice <3) unit test helper
* Use non-default ports for postgres + cockroach. This would help to run same services for ad-hoc unit test (for IDE development for example)
* Use the same flags for cockroach what we use for unit tests (mem store supposed to make tests faster)
* Use `docker-compose down -v` for removing all networks and annynmous volumes
* Remove connection limit (I have seen related problems in my jenkins runs)
* Omit unused databases to avoid getting failures (eg. omit psql when crdb is used)
* Do not drop cockroach database (full database will be dropped, anyway...): it also makes testing faster
Change-Id: Iadc04f8617a2825ea4f10072bd023a9c86883680
Main issue was that when one part copy failed while being inside
goroutine (limiter) and another part was still collecting src/dst parts
it was possible to drop errors from failed part copy. It was possible
bacause on fail context was canceled and if we were still getting
part src/dst then it was returning error immediately and error
group with errors from goroutine was ignored.
Change-Id: I75c6799eba358741629795f2971c7a964cb2c9ce
Fixed main content width to be appropriate.
In case of very long access grant name (100+ symbols) main content gets overlapped with window border. That's an unhandled edge case.
Change-Id: I42ef709edc8454f054baa52231c605afadc471e2
Add the admin API endpoint for disabling an user's multifacdtor
authentication to the satellite admin UI.
Remove a couple of commented code lines too.
Change-Id: Iaee7efe7a3d4d38bdd6541311447a9726806f0f1
We have a couple of support tickets so far that require us to
disable the mfa on accounts. Since we currently had no other
way than doing a SQL War Crime, it makes sense to add it to the
admin API.
Change-Id: Ib16735c1961380b04345a3495d4eebee5fa0bc41
Currently we have a bug in which we would require that a project of
a paid tier user needs to be two months unused before we can delete it.
This change fixes it and reduces it back to the normal next billing cycle.
Change-Id: I28610b6c45c68943fd4f2621233bccc06cab28a0
An older change plummed the full console config as subconfig of
the admin api configuration in. This bloated the generated satellite
configuration unnecessarily while also allow for confusion/mistakes.
Change-Id: Icf49cc1f147711e37e85f6eac1143fab8ddf1659
when deleting an object that has been copied multiple times, we look for an ancestor_stream_id by taking the min of all copies stream_id.
This change simplifies this process by picking any stream_id as a new ancestor by using 'distinct on'.
Fixes https://github.com/storj/storj/issues/4745
Change-Id: Iffb519b82d2ae2ed73af48fa0e86f87384e0158f
Few improvements were made to how we are handling errors
while doing parallel upload/download for single object:
* unhide error under 'context canceled' which was shown in most of
cases
* add part number to error message
* don't try to commit if any error occurs while operation
* combine errors into more readable form, example:
---
failed to download part 3: uplink: eestream: failed to download stripe 0:
error retrieving piece 00: ecclient: piecestore: rpc: tcp connector failed: rpc: dial tcp 97.119.158.36:28967: i/o timeout
...
error retrieving piece 89: ecclient: piecestore: rpc: tcp connector failed: rpc: dial tcp 161.129.152.194:28967: i/o timeout
failed to download part 1: uplink: eestream: failed to download stripe 0:
error retrieving piece 01: io: read/write on closed pipe
...
error retrieving piece 97: io: read/write on closed pipe
failed to download part 2: uplink: eestream: failed to download stripe 0:
error retrieving piece 00: io: read/write on closed pipe
...
error retrieving piece 01: ecclient: piecestore: rpc: tcp connector failed: rpc: dial tcp 180.183.132.234:28967: operation was canceled
error retrieving piece 96: io: read/write on closed pipe
main.(*cmdCp).parallelCopy:418
main.(*cmdCp).copyFile:262
main.(*cmdCp).Execute:156
main.(*external).Wrap:123
github.com/zeebo/clingy.(*Environment).dispatchDesc:126
github.com/zeebo/clingy.(*Environment).dispatch:53
github.com/zeebo/clingy.Environment.Run:34
main.main:26
runtime.main:250
---
Change-Id: I9bb70b3f754567761fa8d17bef8ef59b0709e33b
Reworked "add payment method" and "create project prompt" modals to use new common VModal component.
Change-Id: I73039e5d30077e35f05501a48f39142b799da875
`os/exec.Cmd.CombineOutput` runs the command, hence, it cannot be used
after calling the `Run` method.
Because `CombineOutput` already runs the command, we can use it directly
instead of `Run`.
Without this change if the command returns an non-zero code we get an
error because of the command already started and we don't get the
output.
Example removing a copyright notice from one file and running the linter
(only showing the affected printed line)
Without this fix
2022/05/26 15:48:40 [/storj check-copyright] error exec: already started
With this fix
2022/05/26 16:22:40 [/storj check-copyright] error missing copyright certificate/doc.go: %!w(<nil>)
NOTE the `%!w(<nil)` is a bug in the check-copyright linter.
Change-Id: I40b64842028399b92a8982bfb143e1f87f92467b
Add billing DB to the satellite. This DB will hold all transactions on the users account and can be used to compute the users current usable account balance.
Change-Id: I056416efc169e5e5e30c9f30cd8bc766b7bc8073
Implemented new service method for generating API keys.
Implemented new endpoint.
Improved multiple endpoint groups handling.
Change-Id: Iba26fbf9123707b5b4c2d5e8c5a35d507404f24a
We are not using this method and most probably we
won't need to list objects with all statuses at once.
Removing for now.
Change-Id: I7aa0468c5f635ee2fb1fe51db382595c6343dd9c
- parallel deletion of 50 objects and their 50 copies (one copy per object)
This test is skipped because it's creating deadlocks that are not automatically retried on postgres
- parallel deletion of 1 object and its 50 copies.
Fixes https://github.com/storj/storj/issues/4745
Change-Id: Id7a28251c06bb12b5edcc88721f60bf7a4bc0492
Our linting process depends heavily on custom tools and linting
configuration. To help improve this process for running on local
developer machines, we can run the various tasks in our existing CI
container.
Change-Id: I60407686ce9233cc4f16e3724c5e8d44367aa200
This change attempts to run tests as close to CI as possible. It
introduces a docker-compose file that deploys some supporting
services. While this change makes it possible to run tests locally,
it does not address the core time it takes to run tests end to end.
make test/postgres 1417.87s user 263.38s system 291% cpu 9:37.73 total
Resolves https://github.com/storj/dev-enablement/issues/10
Change-Id: I4d9adc125992c1b1c133e8fe108384fa3c66ff7a
testplanet executes cockroach and postgress tests parallel, therefore using http.DefaultClient is safe only as long as we don't modify it.
TestActivationRouting modifies it (client.CheckRedirect=...), therefore it should use a local version instead of the default one.
Problem reported by a jenkins build:
```
==================
WARNING: DATA RACE
Write at 0x000003486af0 by goroutine 143:
storj.io/storj/satellite/console/consoleweb_test.TestActivationRouting.func1()
/home/jenkins/workspace/storj-testing-experiments/satellite/console/consoleweb/server_test.go:66 +0x378
storj.io/storj/private/testplanet.Run.func1.1()
...
Previous read at 0x000003486af0 by goroutine 104:
net/http.(*Client).checkRedirect()
/usr/local/go/src/net/http/client.go:494 +0xd73
net/http.(*Client).do()
/usr/local/go/src/net/http/client.go:691 +0xd31
net/http.(*Client).Do()
/usr/local/go/src/net/http/client.go:593 +0x204
storj.io/storj/satellite/console/consoleweb_test.TestActivationRouting.func1.1()
/home/jenkins/workspace/storj-testing-experiments/satellite/console/consoleweb/server_test.go:48 +0x1e5
storj.io/storj/satellite/console/consoleweb_test.TestActivationRouting.func1()
/home/jenkins/workspace/storj-testing-experiments/satellite/console/consoleweb/server_test.go:74 +0x49d
storj.io/storj/private/testplanet.Run.func1.1()
...
```
Change-Id: I73319a5a593e067b906ec1fda70a44ca1e5a49a2
This has been a cause of some confusion, even though the fields are
labeled as being copies of config values.
Having them be under a field explicitly named "Config" makes this
clearer, plus, allows the values to be passed in simply as a copy
of the Config struct from the satellite, rather than copying the fields
individually (which can be error-prone, particularly as the AuditCount
field in UpdateRequest is apparently not the same thing as the
AuditCount field in reputation.Config).
Refs: https://github.com/storj/storj/issues/4601
Change-Id: I386953347b71068596618616934aa28e3245cdc1
Add storjscan wallets DB to the satellite. For now this DB is a one to one mapping of the users account to a storjscan wallet that can be used by the account holder to make payments on their Storj account.
relates to https://github.com/storj/storj/issues/4347
Change-Id: I6e65b15817b90ceb75641244f9bf173c3b4228a7
The two protobuf types are identical except that one is in our common/pb
package, and the other is in internalpb. Since the type is public
already, and there is no difference in the internal one, it seems better
to use the public one for all satellite needs.
There is also another type which is essentially identical, but which is
not a protobuf type, also called "AuditHistory". It looks like we don't
ever actually need to have a separate type from the protobuf one.
This change makes us use "storj/common/pb".AuditHistory for all of our
AuditHistory needs.
Refs: https://github.com/storj/storj/issues/4601
Change-Id: If845fde21bb31c801db6d67ffc9a146d1617b991
logo redirects to homepage on login, signup, forgot password, reset
password, and activate account pages
Change-Id: I992aeae197004d620addd8d515cae1c1ca80a778
github issue: https://github.com/storj/storj/issues/4289
also change other notifications in CardDialog.vue to use $notify plugin
Change-Id: Ic38c39cce8ade720a8fcd9061e916978168575b8
Created and styled Access Grant Flow modal. Added open and close functionality
Added proper text for access grant create buttons
Added logic to access grants selection type
Added permissions check box logic and new input field for buckets.
Added tooltips and hover logic for the tooltips
Added acknowledgement functionality to encrypt step
Added conditional logic to encrypt access button and the acknowledgement box
Added a prop that shows the scrollbar for the bucket selector
Added name validation - needs proper error message
Implemented download utility and simplified true/false method
Co-authored-by: cl-mitch <mitch.george@compozelabs.com>
Tests are intermittently fail with similar error:
```
--- FAIL: TestDeletePendingObject/Cockroach (15.85s)
test.go:380:
Error Trace: test.go:380
delete_test.go:221
Error: Should be zero, but was metabase.DeleteObjectResult{
Objects: []metabase.Object{
{
ObjectStream: {ProjectID: {0x0f, 0x40, 0x70, 0x41, ...}, BucketName: "txxywyg4", ObjectKey: "\xbb+$\x17\x80\xc6\xcaC\xa3\xdb\xc3z*\xa8\xbe\xaf", Version: 1, ...},
- CreatedAt: s"2022-05-20 14:40:15.995376773 +0200 CEST",
+ CreatedAt: s"2022-05-20 14:40:21.04949 +0200 CEST",
ExpiresAt: nil,
Status: 1,
... // 9 identical fields
},
},
Segments: {{RootPieceID: {0x01, 0x00, 0x00, 0x00, ...}, Pieces: {{...}}}, {RootPieceID: {0x01, 0x00, 0x00, 0x00, ...}, Pieces: {{...}}}},
}
Test: TestDeletePendingObject/Cockroach/with_segments
--- FAIL: TestDeletePendingObject/Cockroach/with_segments (0.68s)
```
Looks like we shouldn't have an assumption that all tests can be finished in 5 seconds, especially not in highly parallel environment.
These tests use `time.Now` at the beginning and compare the time saved in the database (usually filled by the database).
The difference shouldn't be higher than 20 seconds (before this commit: 5 seconds) which assumes that the records are saved in this timeframe...
Change-Id: Ia6f52897d13f88c6857c05d728bf8e72ab863c9b
old bucket creation flow removed
new flow added
name and passphrase splitted into separate views
demo bucket will not be created automatically
bucket creation progress bar added
Change-Id: I2a1d7d77c3038caaafb3c06bdb0ac5dd1ad17599
This functionality will be needed in both packages, so here we move it
into the more general reputation-code package and export it for use in
satellitedb.
This also removes the related UpdateAuditHistory() signature from the
reputation DB interface, since it doesn't have anything to do with the
db. It doesn't need to be a method, either.
Finally, this changes the test for addAudit to be a plain test function
instead of using testplanet.Run(). It didn't need a whole testplanet
setup or any databases.
Refs: https://github.com/storj/storj/issues/4601
Change-Id: I90f6a909e5404f03ad776b95cfa2f248308c57c1
Styled Access Grant Table according to new flow UX
Added search functionality to My Access Table.
Separated search styles into 2 components
Co-authored-by: Moby von Briesen <mobyvb@gmail.com>
Co-authored-by: hovex023 <97616907+hovex023@users.noreply.github.com>
Recently we applied this optimization to metrics observer and time
used by its method dropped from 12m to 3m for us1 (220m segments).
It looks that it make sense to apply the same code to all observers.
Change-Id: I05898aaacbd9bcdf21babc7be9955da1db57bdf2
If TestCommitInlineSegment tests are taking longer time
then zombieDeadline created at the beginning of test
can be too far in the past. Creating zombieDeadline for each case
should avoid flakines.
Change-Id: Ieb011e8e470f6f1c32cf9365c8ae819317de6738
Márton found out that DROP DATABASE is rather slow on CRDB, and it makes
a significant impact when running the whole testsuite. In sum of test
times it's ~2.5h compared to ~2h. And the end-to-end ~20m to ~16m.
This adds a new flag STORJ_TEST_COCKROACH_NODROP for enabling this
behavior in the CI environment.
Fixes https://github.com/storj/dev-enablement/issues/6
Change-Id: I5a6616c32dc6596a96ba3d203f409368307d7438
We can use PieceIDDeriver in all places where we are deriving id from
the same id multiple times. We have serveral such places: gc, segment
deletion, segment validation, order limit creation. Using it should
save some resources.
Change-Id: I24668d516c0f7cea4aec6470614067734149501d
