Commit Graph

4905 Commits

Author SHA1 Message Date
VitaliiShpital
5e0106f1fe web/satellite: web worker for wasm
WHAT:
web worker for compiling and instantiation of web assembly module

WHY:
Currently webassembly requires unsafe-eval, however we don't want to
add it to main site due to CSP. The workaround for this is to instantiate
wasm code inside a web worker.

Change-Id: I0c3c9cafa3a0c344761cf6dd86bf96248f1103ca
2020-11-11 16:24:06 +02:00
Cameron Ayer
07acf0e574 cmd/storagenode: add docker env variable to toggle running setup
Previously, we ran setup if no config file was found in the expected dir.
However, there may be situations where a previously set up node's files
may be unreachable. In this case, we would prefer to exit with an error
rather than assume this node needs to be initialized.

The solution here is to add a new env variable to call the setup command.
If SETUP == true, the node will setup, but not run. If SETUP != true,
the node will run and not setup.

If a previously set up node runs with SETUP, it will return an error.
If a node runs without an initial SETUP, it will return an error.

Change-Id: Id2c796ec3d43f2add5e5f34fb777a563eae59f2f
2020-11-11 13:11:19 +00:00
Cameron Ayer
da9f1f0611 satellite/repair: add monkit counter for segments below minimum required
The current monkit reporting for "remote_segments_lost" is not usable for
triggering alerts, as it has reported no data. To allow alerting, two new
metrics "checker_segments_below_min_req" and "repairer_segments_below_min_req"
will increment by zero on each segment unless it is below the minimum
required piece count. The two metrics report what is found by the checker
and the repairer respectively.

Change-Id: I98a68bb189eaf68a833d25cf5db9e68df535b9d7
2020-11-11 12:48:23 +00:00
Egon Elbre
2ff7925e65 ci: set GOTRACEBACK=all
Currently when there's a timeout or panic, the culprit goroutines might
not be printed. Set traceback to all, which prints all user created
goroutines.

Change-Id: I29f87812d2a60f671b3eb172499e24cf70d990b5
2020-11-11 13:57:45 +02:00
Isaac Hess
d4d43f02b5 cmd/internal/wizard: select satellite
This change forward-proofs the satellite selection on uplink setup.
Currently we have hard-coded response values, so if we add satellites in
the future we have to remember to update the switch statement below. In
this change it should work for any number of future satellites.

Change-Id: I3250fe2154dbeb4820efadf49780b20c4b7a3408
2020-11-10 15:26:05 -07:00
Isaac Hess
e32466696a cmd/uplink: add access register command
This command allows you to register your access with a separate service
(hosted gateway).

Change-Id: I67200cd4553fd41e9b1a56cca7ccc6553136a76a
2020-11-10 15:25:57 -07:00
Yingrong Zhao
8fd841b910 scripts/tests/testversions: fix installation during setup
This PR fixes below issues:
1. remove concurrent installation for various versions
    We were doing this to decrease the amount of execution time the versions test.
    However, it's returning incorrect exit code when there's an
    installation failure.
    Right now, we are only installing two versions of `storj-sim` and
    the rest are only doing uplink cli installation. The performance of
    this test should be hugely impacted by the setup step now.
2. only remove release settings instead of deleting the entire file
    uplink CLI is referrencing `private/version` package. Therefore, we
    cannot delete it
3. add back `GATEWAY_0_API_KEY` in storj-sim
    In order to set up older version of uplink cli, we need access to
    the gate way api key.

Change-Id: Ia3c37c197bd007b6e1f7c2bd71adde42181d46f0
2020-11-10 20:38:49 +00:00
Yingrong Zhao
2ce3170bb4 satellite/console/wasm: expose method to add caveats in the browser
This PR does the following three things:
    1. Defines a high-level interface for this wasm package
        - All return value from this package will be wrapped with an
          result object that contains a value field and an error field
    2. Exposes two new functions to allow users to add permissions for a
       given API key
        - newPermission()
        - setAPIKeyPermission()
    3. Adds API documentation for the newly added API functions

Change-Id: Id995189702b369bba18fa344bef4ddfb0f3f1f44
2020-11-10 20:10:53 +00:00
Stefan Benten
5f840aae6e
build: use go v1.15.4 (#3968) 2020-11-10 17:45:38 +01:00
NickolaiYurchenko
dbaf8b6c0d web/storagenode: sno types/store/api refactoring
WHAT:
node store, api, types refactoring;
service creation;
minor payout store changes;

Change-Id: I5c52740730db49bc16393b07858354a2914846fb
2020-11-10 15:55:27 +02:00
Brandon Iglesias
3ba52b25a9
satellite/rewards: update partners to include MAXN 2020-11-10 14:08:32 +02:00
Egon Elbre
b892a00143 mod: bump dependencies and reenable test
We shouldn't have any EOF issues with recent drpc fix, let's reenable
and see whether it's still flaky.

Change-Id: I0de312bcb087c7f70ec9d3281d73d86f971845d5
2020-11-10 10:32:21 +00:00
Egon Elbre
a5e05a63ba
.clabot: clarify the cla message (#3964)
It was not obvious that the CLA needs to be manually reviewed before cla-bot approves the message.
2020-11-10 10:51:48 +02:00
Moby von Briesen
db6bc6503d satellite/metainfo: Update metainfo RS config to more easily support multiple RS schemes.
Make metainfo.RSConfig a valid pflag config value. This allows us to
configure the RSConfig as a string like k/m/o/n-shareSize, which makes
having multiple supported RS schemes easier in the future.

RS-related config values that are no longer needed have been removed
(MinTotalThreshold, MaxTotalThreshold, MaxBufferMem, Verify).

Change-Id: I0178ae467dcf4375c504e7202f31443d627c15e1
2020-11-09 22:16:13 +00:00
Hector Fernandez
dc5a5df7f5
chore: fix typos in the documentation (#3959) 2020-11-09 22:00:34 +02:00
crawter
e6dd3ecaa7 multinode/database: members repository created
Change-Id: I429791636f667a19c383a2a0c524a2068cf2812f
2020-11-09 16:23:19 +00:00
Qweder93
8dc10e32ad stefan benten satellited added to historical payout data
Change-Id: I1177b2d2ef10d514f7d401e29891fa7dd964e9ac
2020-11-09 15:43:41 +00:00
Cameron Ayer
d63b7658e8 satellite/repair: fix lastSeenSegmentKey bug in IrreparableProcess
A change was made to use a metabase.SegmentKey (a byte slice alias)
as the last seen item to iterate through the irreparable DB in a
for loop. However, this SegmentKey was not initialized, thus it was
nil. This caused the DB query to return nothing, and healthy segments
could not be cleaned out of the irreparable DB.

Change-Id: Idb30d6fef6113a30a27158d548f62c7443e65a81
2020-11-09 14:48:15 +00:00
NickolaiYurchenko
1e356f1c5f web/storagenode: held progress label changed 15 month -> 16 month
Change-Id: I08efbf6ce64b5ca562346e74cdf50014612e27e9
2020-11-09 15:26:27 +02:00
NickolaiYurchenko
3b388c21cf web/storagenode: notifications domain and app types separations
WHAT:
notifications store, api, types refactoring;
service creation;
notifications module store tests creation;

WHY:
to separate domain and app types;
to be able to reuse code more easier;

Change-Id: I01c6584fc41bbf73e0b6f84501cc66bbebd50ace
2020-11-09 11:51:38 +00:00
Egon Elbre
074784e1b4
.clabot: add hectorj2f (#3963) 2020-11-09 13:44:27 +02:00
Egon Elbre
e1f37ece08 private/lifecycle: warn on slow service shutdown
Adds a warning when service takes over 15s to shutdown.

Change-Id: I44307b4b7560ac2978f62a623894a4af4f5a7402
2020-11-06 15:01:54 +00:00
Egon Elbre
cbc1922590 private/dbutil/pgtest: use round robin to pick databases
Currently we were picking databases randomly for testing,
however a round-robin picking might have more predictable
behavior and cause less cockroach timeouts.

Change-Id: I74ac0d5b38c89452d3c46d3811330e46e7449514
2020-11-06 12:55:55 +00:00
VitaliiShpital
f8c3848c78 satellite/console: change user's email endpoint/feature
WHAT:
change user's email endpoint and appropriate service method was implemented

WHY:
make it possible to change user's email for temporary filezilla account

Change-Id: Ieea41bf49819a42b5f433e8dfaeec24c6d5ddc9f
2020-11-06 11:54:07 +00:00
Egon Elbre
60bb34a096 private/testblobs: fix data race in BadDB
The database is accessed concurrently and modifications need to be synchronized.

Change-Id: I72a91ae2eac55d48a15aa7b0af8966aa3b038021
2020-11-06 11:56:46 +02:00
jessicagreben
c4c29e370a wasm: add webassembly code for creating access grant in console web UI
Change-Id: I3c6d9afc660f3d959d6138db84341e9460b877a1
2020-11-04 12:08:30 -08:00
Ivan Fraixedes
2dffaebc6f satellite/accounting: Fix and enhance code doc comments
Fix and enhance the source code documentation comments for the
satellite/accounting packaged.

Change-Id: I965742cf378e8b6b80d18bc84a4ff76e9af1e8b7
2020-11-04 09:50:48 +00:00
paul cannon
8616fc146d satellite/orders: send IPs for graceful exit
Storage nodes undergoing Graceful Exit have up to now been receiving
hostnames for all other storage nodes they need to contact when
transferring pieces. This adds up to a lot of DNS lookups, which
apparently overwhelm some home routers. There does not seem to be any
need for us to send hostnames for graceful exit as opposed to IP
addresses; we already use IP addresses (as given by the last_ip_port
column in the nodes table) for all the GET and PUT orders we send out.

This change causes IP addresses to be used instead.

I started trying to construct a test to ensure that the behavior
changed, but it was rabbit-holing, so I've begun to feel that maybe this
change doesn't require one; it is a very simple change, and very much of
the same nature as what we already do for IPs in CreateGetOrderLimits
and CreatePutOrderLimits (and others).

Change-Id: Ib2b5ffe7a9310e9cdbe7464450cc7c934fa229a1
2020-11-04 00:17:20 +00:00
Egon Elbre
c55c23f81f private/testplanet: add STORJ_TESTPLANET_ABSTIME
Allow setting STORJ_TESTPLANET_ABSTIME=1 to use absolute time in
testplanet logs.

Change-Id: I4df5dfc1fc055d9726aed65242ab71338550e671
2020-11-03 15:44:18 +02:00
Cameron Ayer
dc67ce74c9 satellite: remove IsUp field from overlay.UpdateRequest
With the new overlay.AuditOutcome type for offline audits, the
IsUp field is redundant. If AuditOutcome != AuditOffline, then
the node is online.

In addition to removing the field itself, other changes needed
to be made regarding the relationship between 'uptime' and 'audits'.
Previously, uptime and audit outcome were completely separated. For
example, it was possible to update a node's stats to give it a
successful/failed/unknown audit while simultaneously indicating that
the node was offline by setting IsUp to false. This is no longer possible
under this changeset. Some test which did this have been changed slightly
in order to pass.

Also add new benchmarks for UpdateStats and BatchUpdateStats with different
audit outcomes.

Change-Id: I998892d615850b1f138dc62f9b050f720ea0926b
2020-11-02 15:34:17 -05:00
Egon Elbre
0c23b12038 private/testplanet: use relative time logging
Instead of printing RFC3339 timestamp, we'll print relative time
since the creation of the testplanet.

Before:

    logger.go:130: 2020-11-02T14:54:53.864+0200 DEBUG   versioncontrol   addr= 127.0.0.1:30904

After:

    log.go:54: 00:00.002        DEBUG   versioncontrol   addr= 127.0.0.1:30945

Change-Id: Ifa423f9d54d4e7c583d9290fe36a791d28166f8f
2020-11-02 17:53:18 +00:00
Egon Elbre
7183dca6cb all: fix defers in loop
defer should not be called in a loop.

Change-Id: Ifa5a25a56402814b974bcdfb0c2fce56df8e7e59
2020-11-02 15:06:38 +02:00
Egon Elbre
fd8e697ab2 {satellite,storagenode}/internalpb: use specific package name
Ensure we don't register types with the same name into protobuf.

Change-Id: I53d025863fff8c91a067ca5819befa87eb5e35bb
2020-10-30 17:31:08 +02:00
Egon Elbre
9674ce40e6 mod: bump dependencies
Change-Id: I61beaf9793609f7614478cfa604b2721d9010b52
2020-10-30 16:17:55 +02:00
Michal Niewrzal
0205f0d807 satellite/metainfo: fix usage of types from internalpb
After moving SatStreamID and SatSegmentID from common I missed changing
some methods in metainfo endpoint. This change is a fix for that.

Change-Id: I34e121fce47371ee4cfd92cce03809520b68859f
2020-10-30 16:03:45 +02:00
Egon Elbre
77c4f99fa0 satellite/internalpb: move delegated_repair.proto
Change-Id: If4f37c52b151e09cf35d2145b463ef1e9ab529ae
2020-10-30 15:31:32 +02:00
Egon Elbre
11338e9beb satellite/internalpb: move audithistory.pb
Change-Id: I8eee84d49ed90459168ddaf04ae57f790c2a22c4
2020-10-30 15:30:11 +02:00
Egon Elbre
1903b15474 storagenode/internalpb: move gracefulexit.proto
Change-Id: Ia3614846ed49a39c8f39331516d16d45a695240b
2020-10-30 15:24:56 +02:00
Egon Elbre
cda67a659a storagenode/internalpb: move inspector.proto
Change-Id: I951379c3b2ff00d1bc09d6a49c026a7e723432d6
2020-10-30 14:51:26 +02:00
Egon Elbre
7ce372c686 satellite/internalpb: add inspectors
Change-Id: Ib688e43d05135c0c31ae95df533f1e4535ea396a
2020-10-30 13:28:17 +02:00
Egon Elbre
004e610d0f satellite/internalpb: move datarepair.pb to internal
Change-Id: If901d9ff4e5ee6715b963eeeb46513a602a44b3d
2020-10-30 13:28:14 +02:00
Michal Niewrzal
8f26f66da0 internalpb: move satellite specific protobuf types storj/storj
We have some types that are only valid for satellite usage. Such types
are SatStreamID and SatSegmentID. This change moves those types to
storj/storj and adds basic infrastructure for generating code.

Change-Id: I1e643844f947ce06b13e51ff16b7e671267cea64
2020-10-30 08:49:16 +00:00
Qweder93
f5ba8b8009 storagenode/suspensions: added offline-suspension notificatio chore + tests
Change-Id: I2521cd2e7d08a1dd379e717a554a026c7508c18f
2020-10-29 19:44:22 +02:00
Egon Elbre
e0dca4042d all: add pprof labels for debugger
By using pprof.Labels debugger is able to show service/peer names in
goroutine names.

Change-Id: I5f55253470f7cc7e556f8e8b87f746394e41675f
2020-10-29 15:10:07 +00:00
Qweder93
624255e8ba storagenode/secret: db tests added, small renaming fixes added
Change-Id: I7eae1a9a64c20a39c97e81fa741cfc9b9e1e615a
2020-10-29 14:23:04 +02:00
Egon Elbre
caefde6b32 private/{dbutil,tagsql}: pass ctx to database opening
Database opening usually dial and hence we should pass ctx to them.

Change-Id: Iaa2875981570d83e65be3710f841cf30349f807b
2020-10-29 10:51:29 +00:00
Egon Elbre
e3985799a1 storage/{cockroachkv,postgreskv}: add ctx to opening
Database opening usually dial and hence we should pass ctx to them.

Change-Id: Iecf41241aaa94d54506cbc80b0e53449848d8819
2020-10-29 10:49:08 +00:00
Egon Elbre
89ce1fe626 storagenode/storagenodedb: add ctx to OpenNew and OpenExisting
Database opening usually dial and hence we should pass ctx to them.

Change-Id: I9160ae95829f22f347bd525904898a47279a7427
2020-10-29 09:52:37 +02:00
Egon Elbre
096445bc1c certificate/authorization: add ctx to OpenDB
Database opening usually dial and hence we should pass ctx to them.

Change-Id: I1362783568f66383c46f07be7549327bb1aaa39e
2020-10-29 09:46:23 +02:00
Egon Elbre
d0beaa4a87 pkg/revocation: pass ctx into opening the database
Opening a databases requires ctx, this is first step to passing ctx
to the appropriate level.

Change-Id: I12700f39a320206d8a2a4e054452319f8585b44b
2020-10-29 07:15:36 +00:00