Commit Graph

6751 Commits

Author SHA1 Message Date
dlamarmorgan
c8b72c8f71 satellite/{payments/billing,satellitedb}: refactor billing DB
Update the billing table to use generated IDs, to include the source and status fields, and to add metadata jsonb field that can be used for fields specific to a particular type of billing transaction. Additionally a new table was added to keep track of user balances

Change-Id: Ieb3a63aafd8fe21fc3386bafd43d52081b7d2838

satellite/{payments/billing,satellitedb}: refactor billing DB

Update the billing table to use generated IDs, to include the source and status fields, and to add metadata jsonb field that can be used for fields specific to a particular type of billing transaction. Additionally a new table was added to keep track of user balances

Change-Id: Ieb3a63aafd8fe21fc3386bafd43d52081b7d2838
2022-08-03 17:52:32 +00:00
Cameron
2a38740efa satellite/satellitedb: don't select public_id in ListByOwnerID because it will be NULL for old projects
Change-Id: I062fa617b622a55d9482206d3a2938cf8a080f2d
2022-08-03 11:37:12 -04:00
Michal Niewrzal
5458cbdad8 satellite/metabase/metabasetest: use interface for tests and benchmarks
metabasetest package utils can be used by both tests and benchmarks
if we will use interface TestingT from require package. This change
adjusts metabasettest.CreateObject method

Change-Id: I3c138e2ef9873b804ab5b3402804efa409397a9f
2022-08-03 13:08:06 +00:00
Márton Elek
bed2af3e87 web: use fixed version from eslint-storj
eslint-plugin-storj is uploaded to the main npm registry.

While we have no clear proof that it causes the random freeze of npm install, it can be fater anyway as fixed version can be cached safely.

Change-Id: Ie74d93c7c4863e46121a9ab6c4e6b787fa3899b3
2022-08-03 12:32:01 +00:00
Márton Elek
2a3c821bd2 build: turn off parallel Verification for premerge/public
Our tests depend on timing (timeouts / sleeps). Therefore we must not overcommit the CPU with highly parallel execution as builds become flaky.

go test can do parallel execution (that's fine as it doesn't overcommit CPU, if only one job is running), but we shouldn't execute different type of
integration tests at the same time.

This commit changes only public (PR + main build) and pre-merge (one build per patch), which are less build time sensitive.

Change-Id: Ib9aaabd301cfa9cd3515a8aa4dd8ae82f58eb811
2022-08-03 11:34:58 +00:00
Egon Elbre
bc9ab8ee5e satellite/audit,storagenode/gracefulexit: fixes to limiter
Ensure we don't rely on limiter to wait multiple times.

Change-Id: I75d48420236216d4c2fc6fa99293f51f80cd9c33
2022-08-03 10:24:16 +03:00
Egon Elbre
1cb3cbaecf cmd/storj-sim: ignore context canceled and flush output
It's possible that content was not being flushed from processes.

For now, ignore other process failures under storj-sim network test.
Once we get other processes stable, we can repropagate the error.

Change-Id: I01ed572d7c779ab6451124f1e24e3d1168b3ea79
2022-08-02 20:48:46 +03:00
Qweder93
98b8c7be06 storagenode/console: consoleAPI EstimatedPayout flacky test fixed
In rare cases time frame between creating time.Now() variable and calling
service method that receives it (after API method call) was big enough to distort
current month expectations and make test fail with last digit difference in 1

Change-Id: Ib811492d62f6598a5c40a09de6a87bffeaa0a78e
2022-08-01 15:13:15 +00:00
Egon Elbre
d2f4ea1f24 satellite/payments/stripecoinpayments: fix mock data-race
`len(m.attached)` was used without locking.

Also, use gofumpt to make whitespace usage more consistent.

Change-Id: Ifa9deedc8451f0c54e84d6ac3c2bdc1807688989
2022-08-01 14:30:39 +00:00
Moby von Briesen
78fa0c11a3 satellite/analytics: Add user agent to TrackCreateUser
Send UserAgent (as "signup_partner") to Segment and Hubspot.

Change-Id: I3313eb902e87b429e7fa8e18afd397a63a3ed07d
2022-07-29 17:27:24 -04:00
Cameron
b4ea1bac42 satellite/console/consoleweb: send email when account already exists
When a someone tries to create an account with an email that is already
associated with a verified account, send them an email with options to
sign in, create an account on another satellite, or reset password.

Change-Id: I844144d88b7356bd7064c4840c9441347a5368b0
2022-07-28 15:29:16 +00:00
wilfredasomani
0eb2cc4511 web/satellite: update table component for project member page
This changes the project member page's table to match the new table design,
And updates the common table component fit this page's use case.

For: https://github.com/storj/storj/issues/4993

Change-Id: Ic7e2c4ceade9bbc98793760927a9243914f454d0
2022-07-28 12:57:05 +00:00
Yaroslav Vorobiov
2ba06aca87 satellitedb: storjscandb return payment with UTC time
Return UTC cached payment timestamp on converting from dbx
payment.

Change-Id: I693802d5ac3b786f9973e8299a95215a2e975982
2022-07-27 20:05:37 +00:00
cl-mitch
24b6a801c2
web/satellite: Add Token card to new Payment Methods flow (#4975)
Added the Add Token and Token Balance cards to the new Payment Methods flow.

Link to associated issue: https://github.com/storj/storj/issues/4632
2022-07-27 14:38:15 -05:00
Clement Sam
d1d15cbf4f satellite: return interval_end_time in DailyStorageUsage endpoint
Updates https://github.com/storj/storj/issues/4178

Change-Id: If0c6f66c58a258300bbf563eef56caba4f1dc626
2022-07-27 18:24:27 +00:00
Clement Sam
f2f046af7a satellite/satellitedb: add interval_end_time column to accounting_rollups table
Adding an interval_end_time column to the accounting_rollups table
to keep the last interval_end_time for each daily storage tallies.

Updates https://github.com/storj/storj/issues/4178

Change-Id: If7a8210c5e9fe2fc9df84b137a8b6e3db2471c58
2022-07-27 17:53:27 +00:00
wilfredasomani
477f28fdab web/satellite: update upgrade account modal with support info
Coinpayments is not working properly; This change replaces the storj deposit form
on the upgrade account modal with a message to contact support.

Related to 1bd609c00

Change-Id: I02eeb6dd721320099a3c523c538dac597c4e8914
2022-07-27 17:05:15 +00:00
Jeff Wendling
f343b856dd go.mod: bump common/private/monkit/monkit-jaeger
this includes all of the fixes identified for end to end
tracing to finally work.

Change-Id: Ia433a2cf9130dc43b6810379af32f5c10f9d3888
2022-07-27 16:17:00 +00:00
Qweder93
2b24632610 {satellite/metainfo, satellite/accounting}: remove ValidateSegmentLimit config value and check
removed segment limit validation and checks in metainfo endpoint and accounting/projectusage
since feature is live and has always has segment limitation now

Resolves: https://github.com/storj/storj/issues/4470

Change-Id: I8cf87cbbc40ac61262f9f05e52573d3ae6410611
2022-07-27 15:36:53 +00:00
Yaroslav Vorobiov
7194973b56 satellite/payments/storjscan: add usd value to cached payments
Add USD value to storjscan cached payment struct to store and
retrieve payments from payments DB.

Change-Id: Ic0b2b4a198e4ea8230bc9a01865b429343f52767
2022-07-27 14:55:45 +00:00
Jeff Wendling
75f19cb100 cmd/uplink: run the collector goroutine
without this goroutine, spans will not be collected or sent
from the uplink.

Change-Id: Ibbf2ab79dee7447a1d8ab62005e102e670114bf2
2022-07-27 14:38:30 +00:00
Michał Niewrzał
7390f389cd satellite/metabase: non recursive listing optimization
With pointerdb listing objects operation was optimized to skip
objects from prefixes for non recursive listing. This change it
adopting this optimiaztion from old code.

Main change is to drop current page results if we detect a prefix
that needs to be skipped and jump with next listing query after this
prefix by setting cursor to "some-prefix(byte('/')+1) which is
effectively "some-prefix0".

Benchmark:
name                                                 old time/op  new time/op  delta
NonRecursiveListing/Postgres/listing_no_prefix-8      960µs ±11%   257µs ±12%  -73.19%  (p=0.008 n=5+5)
NonRecursiveListing/Postgres/listing_with_prefix-8    945µs ±11%   671µs ±12%  -28.97%  (p=0.008 n=5+5)
NonRecursiveListing/Cockroach/listing_no_prefix-8    4.31ms ± 8%  1.19ms ± 7%  -72.44%  (p=0.008 n=5+5)
NonRecursiveListing/Cockroach/listing_with_prefix-8  4.97ms ± 8%  3.35ms ±15%  -32.67%  (p=0.008 n=5+5)

Fixes https://github.com/storj/team-metainfo/issues/115

Change-Id: Iafdf3600d058abbaf441f792d32a7fc17cc08696
2022-07-27 11:26:35 +00:00
wilfredasomani
1bd609c00b web/satellite: update storj deposit form with support info
Coinpayments is not working properly; This change replaces the storj deposit form
with a message to contact support.

see: https://github.com/storj/storj-private/issues/43

Change-Id: If3792d05282c102c1d5ca0664bd8c70ad103f61c
2022-07-26 18:26:49 +00:00
JT Olio
452c243d57 cmd/uplink: support sending traces
Change-Id: I10ef49a24a5b6821b17eb5d206aacfe7e0f54e9e
2022-07-26 11:39:30 -04:00
Jeff Wendling
567986e388 cmd/uplink: minimal tracing support
this allows one to specify a trace id and cause any remote
spans to be sent up to wherever. it doesn't collect any
local traces.

Change-Id: Ia87e294bb276d966f9f3dbfbaf6e7916b1ec7af9
2022-07-26 11:39:30 -04:00
Erik van Velzen
b5fc04af89 satellite/metainfo: usage limits for copy
Previously there was no realtime administration of the storage usage
during copies. Now there is.

Closes https://github.com/storj/storj/issues/4719

Change-Id: I0d536bf551d16208116c3aceac89ed590ec473bf
2022-07-25 12:35:27 +02:00
cl-mitch
6df867bb3d
web/satellite: Added credit cards to new payment methods flow (#4978)
* tabs and routing implemented
* added new method card
* token payment card complete - need to discuss responsiveness on this
* began table implementation
* latest stripe integration changes

Co-authored-by: hovex023 <97616907+hovex023@users.noreply.github.com>
Co-authored-by: Andrew Hove <andrew.hove@concordusa.com>
Co-authored-by: Maximillian von Briesen <mobyvb@gmail.com>
2022-07-23 07:31:09 -05:00
Moby von Briesen
c96ff2fdef satellite/analytics: Track have_sales_contact differently
When signing up, a user can opt in to having sales contact them. This
change alters the way this flag is passed to Hubspot and Segment.

Hubspot sends a form submission request to create the user, followed by
a "custom behavioral event" with some additional user info.

Segment sends an "Identify" call followed by a "create user" event.

This change moves "have_sales_contact" to the form submission for
Hubspot, and the Identify call for Segment.

This simplifies the process of applying this field to a contact/user in
Segment and Hubspot.

Change-Id: I5e6871b3926a76fb24f97fb2d835a26720275072
2022-07-22 19:22:05 +00:00
Moby von Briesen
75544aa435 web/satellite: Add router-view back to old billing page
When implementing the new billing screen, we mistakenly removed the
router-view component from the old billing screen. There was also an
issue where the button would add an extra /billing in the path, so this
change modifies the way the router navigates to the add-coupon route.

Change-Id: Iad3e9aea8c91dbf3bb13022e0995b24ade26af0d
2022-07-22 17:03:01 +00:00
prerna-parashar
ed2ee8443f
satellite/analytics: Modified property update in HubSpot (#4998)
satellite/analytics: Update lifecyclestage = "other" in segment identify calls and createUser Api in Hubspot
2022-07-22 09:59:53 -07:00
Moby von Briesen
f1048e16ce web/satellite: Update partnerStorjLogo svg
The old version has a weird artifact in Firefox which draws a diagonal
line from the logomark across the logo text.

Change-Id: I0172c908c3fd0ea3ac363b8196488a98b09afb67
2022-07-21 17:24:40 -04:00
Cameron
4815cfc09b satellite/{console,satellitedb}: add PublicID to Project, db method GetByPublicID
github issue: https://github.com/storj/storj/issues/4861

Change-Id: Ia83635c0de751a77cd5a49d641da19ed76132c46
2022-07-21 19:49:26 +00:00
Cameron
9cd91c6df8 web/satellite: add modal for creating a project
github issue: https://github.com/storj/storj/issues/4372

Change-Id: I9b8cf64aa76f2d75c740ee8a0158f0382f4a1918
2022-07-21 17:52:42 +00:00
wilfredasomani
23e5ccb9c9 web/satellite: Fix styling on registration page
On small screens the top of the registration page is cut off. This change updates the styles to fix that

Change-Id: I73420cb30c5e34acf7aa6312c054655db410f5e2
2022-07-21 16:36:38 +00:00
Lizzy Thomson
c237468ac9 satellite/console: allow user to update project when limits are above paid defaults
When a user's bandwidth/storage limits are manually set to exceed the
paid tier defaults, attempting to update their project via the satellite
UI (e.g. to change the name/description) would result in an error.
This change modifies the limit checks for updating a project to remove
this issue.

https://github.com/storj/storj/issues/4892

Change-Id: I48853a3289b0ac51587f268a18c1b25743123fcf
2022-07-21 08:40:18 -06:00
Michał Niewrzał
d72f9525d4 satellite/metainfo/piecedeletion: use nodes cache
Piece deletion service was using KnownReliable method from
overlaycache to get nodes addresses to send delete request.
KnownReliable was always hitting DB because this method was
not using cache. This change is using new DownloadSelectionCache
to avoid direct DB calls.

Change is not perfect because DownloadSelectionCache is not as
precise as KnownReliable method and can select few more nodes
to which we will send delete request but difference should be
small and we can improve it later.

Updates https://github.com/storj/storj/issues/4959

Change-Id: I4c3d91089a18ac35ebcb469a56536c33f76e44ea
2022-07-21 00:43:24 +00:00
NickolaiYurchenko
ba009b846f web/satellite: register page made configurable
we want sign up page to be more configurable since we have to add partner branding to it.
added ability to add logo, different title and decpription and even html after common markup.
v-html rule disabled and sanitize plusin added to avoid xss injections.
tsconfig changed to be able to load modules from json files.

Change-Id: I9ff2cf6e5cf977b8c41ca48953b491bd361f26a1
2022-07-20 18:57:47 +03:00
Egon Elbre
82fede2132 satellite/satellitedb: batch delete storage node tallies
Currently we have a significant number of tallies that need to be
deleted together. Add a limit (by default 10k) to how many will
be deleted at the same time.

Change-Id: If530383f19b4d3bb83ed5fe956610a2e52f130a1
2022-07-20 15:42:35 +03:00
cl-mitch
a9de5ce6c3
web/satellite: Add PaymentMethodsTab to new billing flow (#4973)
Add PaymentMethodsTab.vue with stubbed components. Code will be added to the stubbed components in future PR.
2022-07-19 13:17:06 -05:00
Vitalii
b3f909051a web/satellite: fix for dev build
Fixed dev build to do only one build at a time.
Basically, it will do only one legacy type of build and ignore modern build.

Issue:
https://github.com/storj/storj/issues/4908

Change-Id: Ia90deac1f8bf4f609ec776a3580ab0dc8f22a84e
2022-07-19 19:03:04 +03:00
Vitalii
08120d1445 web/satellite: pass query param on satellite switch in signup form
If "partner" query param is set on signup screen we pass it on satellite switch.

Issue: https://github.com/storj/storj/issues/4955

Change-Id: I59039d6ae660d8790390e59650de0ce1e3bba1b4
2022-07-19 13:42:18 +00:00
Erik van Velzen
87e1528104 satellite/metabase: error if source object changed
An object copy/move is done by 2 DRPC calls. It's possible a new object was uploaded ore moved to the source location between these calls. For copy, in that case the segments end up with the wrong keys. This change adds an explicit check for that by comparing the streamId supplied by the user with the streamId in the database.

Fixes https://github.com/storj/storj/issues/4930

Change-Id: Id600456ce78fb4069b93644828a0b3eb85e23e16
2022-07-19 13:27:55 +02:00
Márton Elek
95d85c640c ci: use persisted cache files during verify phase
.

Change-Id: If8af41fd4e3df1d04fef50d9fa53d2038c7d6cc4
2022-07-19 09:50:09 +00:00
Qweder93
cf12802a12 satellite/metainfo: ListBuckets extended with value attribution assigned to bucket
We need to provide the ability to see bucket attribution on the gateway side
so customers can validate if bucket is attributed to them. Extendet metainfo.ListBuckets
request with UserAgent.

Fixes https://github.com/storj/storj/issues/4965

Change-Id: I5624874a7faa14cda06183ad44013e9ebb385b63
2022-07-18 18:23:50 +00:00
NickolaiYurchenko
0317c819dc web/satellite: logout added after password change
functionality moved from reset password where no cookie and causes error to proper place
added redirect with 2 sec delay after success

Change-Id: I9cfe7e88bed7cfaf04fc7dc7fde58702b0040c5c
2022-07-18 17:48:05 +03:00
Ivan Fraixedes
7aaab3c4c4 satellite/console/consoleweb: Trace HTTP endpoints
Trace all the requests that the HTTP API endpoints receive.

We want to trace them with Monkit because we want to break them down by
request type and response code for seeing if they succeeded or failed.

Also log them with DEBUG level with the IP client.

Change-Id: Ia7b013351c788f131e775818f27091f3014ea861
2022-07-18 13:27:50 +00:00
NickolaiYurchenko
d571b04ac8 web/satellite: new dashboard limit visual component added
also added vue filter to transform number of bytes into readable label

Change-Id: I416cf477fb94059c0f2ceffef755aa7fceacc352
2022-07-18 07:26:30 +00:00
NickolaiYurchenko
1f66bb3931 web/satellite: access grant empty validation added
after I copied and checked an acknowledgment i was able to delete the passphrase and create the access grant
now if I do this "copied" and "acknowledge" statuses if passphrase is changed

Change-Id: I57199a476c802eb3c44e4dddc43ef40fcedcee2f
2022-07-18 06:54:15 +00:00
paul cannon
726c95160b satellite/repair: avoid retrying GET_REPAIR incorrectly
We retry a GET_REPAIR operation in one case, and one case only (as far
as I can determine): when we are trying to connect to a node using its
last known working IP and port combination rather than its supplied
hostname, and we think the operation failed the first time because of a
Dial failure.

However, logs collected from storage node operators along with logs
collected from satellites are strongly indicating that we are retrying
GET_REPAIR operations in some cases even when we succeeded in connecting
to the node the first time. This results in the node complaining loudly
about being given a duplicate order limit (as it should), whereupon the
satellite counts that as an unknown error and potentially penalizes the
node.

See discussion at
https://forum.storj.io/t/get-repair-error-used-serial-already-exists-in-store/17922/36
.

Investigation into this problem has revealed that
`!piecestore.CloseError.Has(err)` may not be the best way of determining
whether a problem occurred during Dial. In fact, it is probably
downright Wrong. Handling of errors on a stream is somewhat complicated,
but it would appear that there are several paths by which an RPC error
originating on the remote side might show up during the Close() call,
and would thus be labeled as a "CloseError".

This change creates a new error class, repairer.ErrDialFailed, with
which we will now wrap errors that _really definitely_ occurred during
a Dial call. We will use this class to determine whether or not to retry
a GET_REPAIR operation. The error will still also be wrapped with
whatever wrapper classes it used to be wrapped with, so the potential
for breakage here should be minimal.

Refs: https://github.com/storj/storj/issues/4687
Change-Id: Ifdd3deadc8258f34cf3fbc42aff393fa545794eb
2022-07-18 05:11:56 +00:00
paul cannon
2f20bbf4d8 satellite/reputation: add a reputation write cache
This should lower the amount of database load coming from
reputation updates.

Change-Id: Iaacfb81480075261da77c5cc93e08b24f69f8949
2022-07-14 21:40:16 +00:00