Commit Graph

5045 Commits

Author SHA1 Message Date
Yaroslav Vorobiov
1b4bfbb9d2 multinode/console: nodes addition and removal
Change-Id: I60c685953a8d0e24f78b1414c34a28d4b87863b0
2020-11-12 20:26:08 +02:00
crawter
e6967720cd cmd/multinode: create schema command added, run command bug fixed
Change-Id: Ief76fc4a878441e5f112bd79810c66e8d85d7acb
2020-11-12 18:00:18 +00:00
NickolaiYurchenko
f8d3a977fa web/storagenode: PayoutPeriodCalendar.vue unit tests
Change-Id: I6a41611e28993577eb72426b941cf272ae8da46f
2020-11-12 18:50:16 +02:00
Jessica Grebenschikov
226e13e616 satellite/cosole: add tests for wasm access code
Change-Id: I78f71b2f0bef03b6e87cd7d79ccaef5f45393b55
2020-11-12 08:03:36 -08:00
NickolaiYurchenko
259f4ebcf1 web/storagenode: EstimationArea.vue unit tests
Change-Id: I5b7606f3deb0b3b8cccf6ec1026d06f3558fd808
2020-11-12 13:21:10 +00:00
Egon Elbre
8182d8a726 satellite/metainfo/metabase: use txutil.WithTx in delete
Change-Id: I75965c4dcf57479ace2367f5d3069d785628e86a
2020-11-12 14:38:36 +02:00
Egon Elbre
bc460cd62d satellite/metainfo/metabase: use txutil.WithTx in commit
Change-Id: I84347291179c7042e22bda703a1c9bd80a1513a0
2020-11-12 14:38:33 +02:00
Michal Niewrzal
9d25b3a7d3 satellite/metainfo/metabase: add ListSegments request
Change-Id: Ie777a0f791685beab3973e76b10a9ce76398e7d7
2020-11-12 11:46:17 +01:00
Yingrong Zhao
54c5d564a1 scripts/tests/testversions: fix older uplink setup
This PR does follwing changes:
    1. Change oldest uplink version in the test to v0.35.3
        When the test is first created, we decided to support uplink
        version starting from v0.17.1, however with many API changes,
        older uplinks are not usable with latest version of the network
        anymore. One of the reasons being older uplinks uses deprecated
        endpoint. Therefore, we will change the oldest uplink version to
        the one that's using only new endpoints.
    2. Disable tls certificate verification in uplink
    3. Use storj-sim version control server instead of production one
    4. Skip uplink version v1.3.x due to bug in that release

Change-Id: I926a6bb9829cb7181ee752437cdcb67e59197fe0
2020-11-11 17:00:01 -05:00
paul cannon
3e56403599 satellite/repair: add a repair health function
This will be used to rank segments in need of repair for attention by
the repair workers.

Change-Id: I5b70650cec933696b4c6d73bb7efb97e3efdf24a
2020-11-11 18:48:51 +00:00
Michal Niewrzal
fd707392e7 satellite/metainfo: set RedundancyScheme for begin segment
response

Multipart upload needs it for uploading parts.

Change-Id: I34886a9dabd17ecb7d55d4c6e19914a1c821f638
2020-11-11 17:25:52 +00:00
Jeff Wendling
31533ed1a1 satellite/console/wasm: remove storj.io/uplink deependency
Change-Id: Iee95389e4ba24618e31aff7be44d05377b2e2419
2020-11-11 16:51:14 +00:00
Malcolm Bouzi
592d0bd6bc
web/satellite: access grant routing (#3966) 2020-11-11 18:41:46 +02:00
Cameron Ayer
5a337c48ec {cmd,private,storagenode}: create storage dir verification during setup
Previously, we created a new file to use for directory verification
every time the storage node starts. This is not helpful if the storage node
points to the wrong directory when restarting. Now we will only create the file
on setup. Now the file should be created only once and will be verified at
runtime.

Change-Id: Id529f681469138d368e5ea3c63159befe62b1a5b
2020-11-11 11:01:36 -05:00
crawter
4ce00c7caa cmd/multinode: run and setup commands added
Change-Id: If7b39c392a9a5617315cefaeafffddab845cf071
2020-11-11 14:48:16 +00:00
VitaliiShpital
5e0106f1fe web/satellite: web worker for wasm
WHAT:
web worker for compiling and instantiation of web assembly module

WHY:
Currently webassembly requires unsafe-eval, however we don't want to
add it to main site due to CSP. The workaround for this is to instantiate
wasm code inside a web worker.

Change-Id: I0c3c9cafa3a0c344761cf6dd86bf96248f1103ca
2020-11-11 16:24:06 +02:00
Kaloyan Raev
9dcd0936d7 satellite/metainfo: MakeInlineSegment to use metabase
Change-Id: Ib32c1d9e91e3f13b4e44abefb2b5d0352b64bd79
2020-11-11 15:24:21 +02:00
Cameron Ayer
07acf0e574 cmd/storagenode: add docker env variable to toggle running setup
Previously, we ran setup if no config file was found in the expected dir.
However, there may be situations where a previously set up node's files
may be unreachable. In this case, we would prefer to exit with an error
rather than assume this node needs to be initialized.

The solution here is to add a new env variable to call the setup command.
If SETUP == true, the node will setup, but not run. If SETUP != true,
the node will run and not setup.

If a previously set up node runs with SETUP, it will return an error.
If a node runs without an initial SETUP, it will return an error.

Change-Id: Id2c796ec3d43f2add5e5f34fb777a563eae59f2f
2020-11-11 13:11:19 +00:00
Cameron Ayer
da9f1f0611 satellite/repair: add monkit counter for segments below minimum required
The current monkit reporting for "remote_segments_lost" is not usable for
triggering alerts, as it has reported no data. To allow alerting, two new
metrics "checker_segments_below_min_req" and "repairer_segments_below_min_req"
will increment by zero on each segment unless it is below the minimum
required piece count. The two metrics report what is found by the checker
and the repairer respectively.

Change-Id: I98a68bb189eaf68a833d25cf5db9e68df535b9d7
2020-11-11 12:48:23 +00:00
Michal Niewrzal
3fe16f4003 satellite/metainfo: upload/download with metabase
This change is adjusting metainfo endpoint to use metabase for uploading
and downloading remote objects. Inline segments will be added later.

Change-Id: I109d45bf644cd48096c47361043ebd8dfeaea0f3
2020-11-11 12:13:52 +00:00
Egon Elbre
2ff7925e65 ci: set GOTRACEBACK=all
Currently when there's a timeout or panic, the culprit goroutines might
not be printed. Set traceback to all, which prints all user created
goroutines.

Change-Id: I29f87812d2a60f671b3eb172499e24cf70d990b5
2020-11-11 13:57:45 +02:00
Isaac Hess
d4d43f02b5 cmd/internal/wizard: select satellite
This change forward-proofs the satellite selection on uplink setup.
Currently we have hard-coded response values, so if we add satellites in
the future we have to remember to update the switch statement below. In
this change it should work for any number of future satellites.

Change-Id: I3250fe2154dbeb4820efadf49780b20c4b7a3408
2020-11-10 15:26:05 -07:00
Isaac Hess
e32466696a cmd/uplink: add access register command
This command allows you to register your access with a separate service
(hosted gateway).

Change-Id: I67200cd4553fd41e9b1a56cca7ccc6553136a76a
2020-11-10 15:25:57 -07:00
Yingrong Zhao
8fd841b910 scripts/tests/testversions: fix installation during setup
This PR fixes below issues:
1. remove concurrent installation for various versions
    We were doing this to decrease the amount of execution time the versions test.
    However, it's returning incorrect exit code when there's an
    installation failure.
    Right now, we are only installing two versions of `storj-sim` and
    the rest are only doing uplink cli installation. The performance of
    this test should be hugely impacted by the setup step now.
2. only remove release settings instead of deleting the entire file
    uplink CLI is referrencing `private/version` package. Therefore, we
    cannot delete it
3. add back `GATEWAY_0_API_KEY` in storj-sim
    In order to set up older version of uplink cli, we need access to
    the gate way api key.

Change-Id: Ia3c37c197bd007b6e1f7c2bd71adde42181d46f0
2020-11-10 20:38:49 +00:00
Yingrong Zhao
2ce3170bb4 satellite/console/wasm: expose method to add caveats in the browser
This PR does the following three things:
    1. Defines a high-level interface for this wasm package
        - All return value from this package will be wrapped with an
          result object that contains a value field and an error field
    2. Exposes two new functions to allow users to add permissions for a
       given API key
        - newPermission()
        - setAPIKeyPermission()
    3. Adds API documentation for the newly added API functions

Change-Id: Id995189702b369bba18fa344bef4ddfb0f3f1f44
2020-11-10 20:10:53 +00:00
Stefan Benten
5f840aae6e
build: use go v1.15.4 (#3968) 2020-11-10 17:45:38 +01:00
Fadila Khadar
2bd239bb75 satellite/metainfo/metabase: tests commit inline segment
Change-Id: Ifad461041c7b9d8c78fc9f998a4e708853087b02
2020-11-10 15:00:35 +00:00
NickolaiYurchenko
dbaf8b6c0d web/storagenode: sno types/store/api refactoring
WHAT:
node store, api, types refactoring;
service creation;
minor payout store changes;

Change-Id: I5c52740730db49bc16393b07858354a2914846fb
2020-11-10 15:55:27 +02:00
Michal Niewrzal
48f1f7132e satellite/metainfo: set RS defaults
While resolving conflicts with `master` I missed this change which is
needed e.g. to run storj-sim.

Change-Id: I56a548ed92b978510526c26c81af03051acfde2f
2020-11-10 13:57:27 +01:00
Brandon Iglesias
3ba52b25a9
satellite/rewards: update partners to include MAXN 2020-11-10 14:08:32 +02:00
Michal Niewrzal
7dde184cb5 Merge 'master' branch
Change-Id: I6070089128a150a4dd501bbc62a1f8b394aa643e
2020-11-10 11:58:59 +00:00
Michal Niewrzal
92f9251074 satellite/metainfo/metabase: add GetSegmentByOffset request
Change-Id: Iea2ce439ec1f7285e447d590c1297294204edc2e
2020-11-10 11:33:34 +00:00
Egon Elbre
b892a00143 mod: bump dependencies and reenable test
We shouldn't have any EOF issues with recent drpc fix, let's reenable
and see whether it's still flaky.

Change-Id: I0de312bcb087c7f70ec9d3281d73d86f971845d5
2020-11-10 10:32:21 +00:00
Kaloyan Raev
3ed4183e52 satellite/metainfo: delete object to use metabase
Change-Id: I2ab63a719fdbc1f8a7fbb4ad73d51a2d2dcfadc6
2020-11-10 09:55:23 +00:00
Egon Elbre
a5e05a63ba
.clabot: clarify the cla message (#3964)
It was not obvious that the CLA needs to be manually reviewed before cla-bot approves the message.
2020-11-10 10:51:48 +02:00
Moby von Briesen
db6bc6503d satellite/metainfo: Update metainfo RS config to more easily support multiple RS schemes.
Make metainfo.RSConfig a valid pflag config value. This allows us to
configure the RSConfig as a string like k/m/o/n-shareSize, which makes
having multiple supported RS schemes easier in the future.

RS-related config values that are no longer needed have been removed
(MinTotalThreshold, MaxTotalThreshold, MaxBufferMem, Verify).

Change-Id: I0178ae467dcf4375c504e7202f31443d627c15e1
2020-11-09 22:16:13 +00:00
Hector Fernandez
dc5a5df7f5
chore: fix typos in the documentation (#3959) 2020-11-09 22:00:34 +02:00
crawter
e6dd3ecaa7 multinode/database: members repository created
Change-Id: I429791636f667a19c383a2a0c524a2068cf2812f
2020-11-09 16:23:19 +00:00
Michal Niewrzal
dbf9f121ed ci: mark failures as unstable
Change-Id: I3cfa07a35cb789f409d95d4778c07ca3355c0f3c
2020-11-09 16:48:38 +01:00
Qweder93
8dc10e32ad stefan benten satellited added to historical payout data
Change-Id: I1177b2d2ef10d514f7d401e29891fa7dd964e9ac
2020-11-09 15:43:41 +00:00
Egon Elbre
184e1ffa9d satellite/metainfo/metabase: commit object without proofs
Change-Id: I788111b9e5bdc961be9e731f8203e52ca9da9dca
2020-11-09 15:18:07 +00:00
Cameron Ayer
d63b7658e8 satellite/repair: fix lastSeenSegmentKey bug in IrreparableProcess
A change was made to use a metabase.SegmentKey (a byte slice alias)
as the last seen item to iterate through the irreparable DB in a
for loop. However, this SegmentKey was not initialized, thus it was
nil. This caused the DB query to return nothing, and healthy segments
could not be cleaned out of the irreparable DB.

Change-Id: Idb30d6fef6113a30a27158d548f62c7443e65a81
2020-11-09 14:48:15 +00:00
NickolaiYurchenko
1e356f1c5f web/storagenode: held progress label changed 15 month -> 16 month
Change-Id: I08efbf6ce64b5ca562346e74cdf50014612e27e9
2020-11-09 15:26:27 +02:00
NickolaiYurchenko
3b388c21cf web/storagenode: notifications domain and app types separations
WHAT:
notifications store, api, types refactoring;
service creation;
notifications module store tests creation;

WHY:
to separate domain and app types;
to be able to reuse code more easier;

Change-Id: I01c6584fc41bbf73e0b6f84501cc66bbebd50ace
2020-11-09 11:51:38 +00:00
Egon Elbre
074784e1b4
.clabot: add hectorj2f (#3963) 2020-11-09 13:44:27 +02:00
Egon Elbre
754838f5d0 ci: convert test failures to unstable
Change-Id: Ib0252c666c086379bd6e2a3a47011479fe3988ed
2020-11-09 11:18:12 +00:00
Egon Elbre
e1f37ece08 private/lifecycle: warn on slow service shutdown
Adds a warning when service takes over 15s to shutdown.

Change-Id: I44307b4b7560ac2978f62a623894a4af4f5a7402
2020-11-06 15:01:54 +00:00
Egon Elbre
cbc1922590 private/dbutil/pgtest: use round robin to pick databases
Currently we were picking databases randomly for testing,
however a round-robin picking might have more predictable
behavior and cause less cockroach timeouts.

Change-Id: I74ac0d5b38c89452d3c46d3811330e46e7449514
2020-11-06 12:55:55 +00:00
VitaliiShpital
f8c3848c78 satellite/console: change user's email endpoint/feature
WHAT:
change user's email endpoint and appropriate service method was implemented

WHY:
make it possible to change user's email for temporary filezilla account

Change-Id: Ieea41bf49819a42b5f433e8dfaeec24c6d5ddc9f
2020-11-06 11:54:07 +00:00
Egon Elbre
60bb34a096 private/testblobs: fix data race in BadDB
The database is accessed concurrently and modifications need to be synchronized.

Change-Id: I72a91ae2eac55d48a15aa7b0af8966aa3b038021
2020-11-06 11:56:46 +02:00