Add some code to generate a basic markdown file documenting a generated
API. Generate this document for the API in
satellite/console/consoleweb/consoleapi/gen.
The documentation is not completely correct, as it may include some
values in the request body that are not actually usable by the
requester. This can be fixed by making sure all types used within the
generated API are properly annotated with `json` tags.
Issue: https://github.com/storj/storj-private/issues/244
Change-Id: I57b259967fb0db8f548b6598a10c825da15ba723
Project member invitations may now be requested through GraphQL
queries. This is necessary for the satellite frontend to display
invitations in the Team page.
References #5855
Change-Id: Ibc8526ba768fd82c1b1890201004ef0f066df2fc
The console service method responsible for removing members from a
project has been extended to remove project member invitations as well.
This will allow invitations to be deleted through the satellite
frontend.
References #5855
Change-Id: I90ca042cc6fb9a75fcd9b391e317caabb1c828f2
A method has been implemented that allows for paged searching through
project members and project member invitations. In the future, this
will be used to display invitations in the Team page of the satellite
frontend.
References #5855
Change-Id: I0937c425f60f1318e55202bf30b44a33ff695414
This change adds analytics events for accepting and rejecting project
member invitations.
References #5855
Change-Id: I2101eb711312a683ecdb4b7d0c6d7cc3ae09c440
An index has been added on the project_id column of the project_members
satellite database table so that we can retrieve members of a project
without performing a full table scan.
References #5855
Change-Id: I1cc30686f836c8fd1aa319247ce857a2392e7a52
We missed to set placement as a part of selection request. It can case
uploading repaired data out of specified placement.
I will provide test as a separate change.
Change-Id: I4efe67f2d5f545a1d70e831e5d297f0977a4eed1
Built side Vuetify subproject inside web/satellite with limited functinality.
For now it has navigation side bar, simple project dashboard and team page (where you can list/add team members).
Issue:
https://github.com/storj/storj/issues/5854
Change-Id: I9ff3e80b8ace1dc31de6a788174c5ffc19f050f8
* optimize SQL for zombie objects deletion query by reducing some direct
selects to segments table
* set AOST for expired/zombie object deletion (was 0 since now)
https://github.com/storj/storj/issues/5881
Change-Id: I50482151d056a86fe0e31678a463f413d410759d
For now we will use bucket placement to determine if we should exclude
some node IPs from metainfo.GetObjectIPs results. Bucket placement is
retrieved directly from DB in parallel to metabase
GetStreamPieceCountByNodeID request.
GetObjectIPs is not heavily used so additional request to DB shouldn't
be a problem for now.
https://github.com/storj/storj/issues/5950
Change-Id: Idf58b1cfbcd1afff5f23868ba2f71ce239f42439
This change adds new email templates for project invites, one for
existing users, one for new users. It changes the project invite code
to use the new template for existing users.
Issue: https://github.com/storj/storj/issues/5860
Change-Id: Ic7b14a677277ea6c25ee527d03f709474fc05f83
We no longer use registration information in ways that could be
exploited by malicious agents, so filtering special characters is not
necessary and has been removed.
Resolves storj-private#133
Change-Id: I3eb4803c71ccb307b38f0288fe2af5eec70f8309
We were reusing a slice to save on allocations, but it turns out the
function using it was being called in multiple goroutines at the same
time.
This is definitely a problem with repairer/segments.go. I'm not 100%
sure if it also is a problem with checker/observer.go, but I'm making
the change there as well to be on the safe side for now.
Repair workers only ran with this bug on testing satellites, and it
looks like the worst that could have happened was that we repaired
pieces off of well-behaved, non-clumped, in-placement nodes by mistake.
Change-Id: I33c112b05941b63d066caab6a34a543840c6b85d
API endpoints and associated methods have been implemented to allow
users to accept or decline their pending project member invitations
through the satellite frontend.
References #5855
Change-Id: Ic23721c64a65e741dc1015838e617fd1af5c8ca4
this change adds code to CreateGetOrderLimits to filter
out any nodes that are not in the placement specified
by the segment. notably, it does not change the audit
or repair order limits. the list segments code had to be
changed to include getting the placement field from the
database.
Change-Id: Ice3e42a327811bb20928c619a72ed94e0c1464ac
Checker when qualifying segment for repair is now looking at pieces
location and if they are outisde segment placement puts them into
repair queue.
Fixes https://github.com/storj/storj/issues/5895
Change-Id: If0d941b30ad94c5ef02fb1a03c7f3d04a2df25c7
Segment repairer should take into account segment 'placement' field
and remove or repair pieces from nodes that are outside this placement.
In case when after considering pieces out of placement we are still above
repair threshold we are only updating segment pieces to remove
problematic pieces. Otherwise we are doing regular repair.
https://github.com/storj/storj/issues/5896
Change-Id: I72b652aff2e6b20be3ac6dbfb1d32c2840ce3d59
It currently is possible to create a violation with regards to
the uniqueness of the user account emails that is used for the
login.
When an update via the admin API is made, it currently is possible
to set the accounts email to an already occupied email address.
This will result in very flacky login behaviour, as well as creating
a lot of other related issues.
This small change adds a check to ensure the email is not attached to
any account.
Change-Id: I167be673082d59ef32cafe41047fce9f5ae534d0
This change limits the length of user input fields like search, email,
username. It also limits the receivable size of request payloads.
This is to prevent potential DDoS attacks resulting from receiving
large payloads.
Improvements are also made to the accounts page and register success
pages to display long names/emails better.
Issue: https://github.com/storj/storj-private/issues/201
Change-Id: I5d36eb83609b3605335a8d150b275c89deaf3b43
Added new gallery view for object browser.
It is behind new feature flag.
TODO: add options dropdown and modals
Issue:
https://github.com/storj/storj/issues/5824
Change-Id: I21829c599cd904b833eaf429690c66c3da306a0f
This change prevents the redirect to all projects dashboard when no
project is selected (if all projects dash is enabled).
Since a previously selected project id is saved in local storage, it is
used to store it's associated project in memory.
This change also makes a small change to a test that ignores potential
failures.
Issue: https://github.com/storj/storj/issues/5920
Change-Id: Ie758893dfb655893520c642fb47b934cd59f177e
There were some dbx-generated queries on the old satellite version that
still reference this column, so we need to add it back until the next
version.
Change-Id: I78b19336d9ca0384936d6cc11f5c50e579b4f2ab
Add a config flag (default false) to hide the new limit cards (e.g.
segment, storage, bandwidth limits) from the UI. We need to investigate
some queries the egress card is using before enabling these everywhere.
Change-Id: I762e7d9e6a0a4315f1520e688b2bad32b100e5a0
We would like to verify if nodes matches specific placement e.g. to
validate segment pieces are correctly geofenced.
https://github.com/storj/storj/issues/5896
Change-Id: I842767dccc121a3c60224f677ab55e5dc150c76e
This change includes STORJ bonuses to the list of transactions returned
by the /wallet/payments endpoint.
Issue: https://github.com/storj/storj/issues/5755
Change-Id: Icc95c2cb9dd9fc5ee7a373e68c1cf8a991e1aa58
Methods SelectAllStorageNodesUpload and SelectAllStorageNodesDownload
are not returning full info with overlay.SelectedNode because its
missing CountryCode.
Change-Id: Ie3cb396bf28d7ec4c6ab8927e5bb560236036aa6
This allows scripted automation to get more details of the
API key such as project ID, and paid tier status.
Updates https://github.com/storj/gateway-mt/issues/321
Change-Id: I8a835752d4fd67382aca804b8c93e63de6c9a846
Tallies are now created for buckets with no objects. Previously, the
bucket tally collector skipped empty buckets since it created tallies
only using information from the objects table. Methods that used
bucket tallies when calculating usage costs would return incorrect
results because of this.
Change-Id: I0b37fe7159a11cc02a51562000dad9258555d9f9
The last code referencing these columns was removed as of satellite
release v1.79, so it is safe to remove them now.
Resolves https://github.com/storj/storj/issues/5432
Change-Id: I2e9d641b2511a61e0b9482ef0f4955a73c290709
This change removes the use of the objects loop for calculating bucket
tallies. It has been superseded by a custom query.
Change-Id: I9ea4633006c9af3ea14d7de40871639e7b687c22
During billing, before invoice creation, check if users are part of a
package plan. If so, and if the package plan is expired, remove unused
credit from the user's balance. If the user has credit in addition to
the package credit, send an analytics event to notify someone to handle
the credit removal manually.
Change-Id: Iad71d791f67c9733f9d9e42f962c64b2780264cc
this is a very old tool built in the very early days
of v3, when we didn't know how the network would be
used. this tool anticipated being able to query remote
nodes for internal state. we don't do that. i don't
think anyone uses this.
Change-Id: Ie1ded3ecbedb09313f2d6fc721039e0f15e4ee85
rather than only logging the last_nets we see in clumpedPieces, this
will run through all the last_nets and log any that have more than one
node. This should have the same outcome, except the counts will be 1
higher (because FindClumpedPieces won't include the first node found in
a clumped network, and this will).
This should be quite a bit faster.
Change-Id: I6a7b2fd387e98963d5295c9ecfde80f2e1ee3b7a
We were using the UploadSelectionCache previously, which does _not_ have
all nodes, or even all online nodes, in it. So all nodes with less than
MinimumVersion, or with less than MinimumDiskSpace, or nodes suspended
for unknown audit errors, or nodes that have started graceful exit, were
all missing, and ended up having empty last_nets. Even with all that,
I'm kind of surprised how many nodes this involved, but using the upload
selection cache was definitely wrong.
This change uses the download selection cache instead, which excludes
nodes only when they are disqualified, gracefully exited (completely),
or offline.
Change-Id: Iaa07c988aa29c1eb05796ac48a6f19d69f5826c1
It seems that the "what pieces are clumped" code does not work right, so
this logic is causing repair overload or other repair failures.
Hide it behind a flag while we figure out what is going on, so that
repair can still work in the meantime.
Change-Id: If83ef7895cba870353a67ab13573193d92fff80b
* Update defaults for gateway credentials URL and linksharing URL to use
storjsatelliteshare.io instead of storjshare.io
* Add new config for "public linksharing URL" and set it to
link.storjshare.io
* Use "private" linksharing URL for actions within the object browser
* Use "public" linksharing URL for sharing files externally
Resolves https://github.com/storj/storj/issues/5805
Change-Id: I2c8fbd04141755b4751dcf4d054253a7ff8d6cf3
Clumped segments (segments with multiple pieces on the same subnet) may
need repair, but the clumped pieces are considered retrievable and we
don't need to call such segments irreparable.
We do want to know where they're coming from, though, if we can, because
we are seeing more than expected.
Change-Id: I41863b243f4bb007ef8929191a3fde1562565ef9
The project member invitations table has been modified to contain a
column for the ID of the user who sent the invitation. This ID is
required for us to return information about the inviter to the
satellite frontend.
References #5855
Change-Id: I928d987a8db2340f731ca65ce30173d4f90a9837
The query for GetNodesNetworkInOrder is causing far too much load on the
database. Since it is not critical that the repair checker have
perfectly up-to-date node network information, we can use a cache
instead.
Change-Id: I07ad45bfdeb46529da093941a06c2da8a00ce878
87d0789691 replaces offset usage with cursors.
But to continue the interation from a specific cursor, we need to iterate over ordered records.
(at least this is what I understood based on the failing tests)
87d0789691
Change-Id: Ic4da3a7c5f03386dd4c373c05102f05871900a3a
We will remove segments loop soon so we need first to move
Segment definition to rangedloop package.
https://github.com/storj/storj/issues/5237
Change-Id: Ibe6aad316ffb7073cc4de166f1f17b87aac07363
This change updates the replacer in satellite/satellitedb/dbx/gen/main.go
to work with an updated dbx.
Change-Id: I08e89d6d27e6f1d435416105fe5f622009add7ad
* Don't use rpcstatus.Unknown as an indicator of dial failure; instead,
GetShare now indicates with a per-share field where a failure happened
(DialFailure, RequestFailure, NoFailure). Use that information in
Verify() to determine how to treat the source node.
* Add a test that replaces a storage node with a black hole, so that
connections there will always time out. Make sure we handle that case
correctly.
Refs: https://github.com/storj/storj/issues/5632
Change-Id: I513a53520fb48b7187d4c4d7e14e01e2cfc0a721
Stripe invoice project records while listing are causing full table scan
because of OFFSET caluse. This change is refactoring query to list using
cursor.
Change-Id: I6b73b9b2815173d7ef02cf615408778476eb3b7b
We have method which is getting projects owned by specific user but it's
causing full table scan because we don't have index on owner_id column.
Change-Id: Icb71c9ac5b73104a52241ed8ba126c995c10811f
The string check previously used to check for constraint errors is now
replaced with dbx.IsConstraintError check.
Change-Id: I553ccd69e3c02b6b54441bd9f929b85a155eaf00
Fix an error that can occur when processing multiple invoices for the same user in a single invoice cycle when the user is paying with Storj tokens.
Change-Id: I54af8c7dde1965d994f687fdfc4e4b5ef4deeb2d
w.Header().Set needs to be called before WriteHeader,
because WriteHeader sends all the headers and calls to
Set won't have any effect afterwards.
Change-Id: Ia6b1c5e2cd54201a6c3980d63de04a0095b2db9a
The console DB cleanup chore has been extended to remove old project
member invitation records.
Resolves#5816
Change-Id: Id0a748e40f5acf03b9b903265c653b072846ba19
There is still a reference to partner_id in a query, which we cannot
move until dropping the "not null" constraint for it. This change adds
that migration.
Related to https://github.com/storj/storj/issues/5432
Change-Id: I98802a6e1bd59f3d9214de3db6688d9daf664a70
A chore responsible for purging data from the console DB has been
implemented. Currently, it removes old records for unverified user
accounts. We plan to extend this functionality to include expired
project member invitations in the future.
Resolves#5790
References #5816
Change-Id: I1f3ef62fc96c10a42a383804b3b1d2846d7813f7
This change makes the error thrown when adding an existing member to a
project readable.
Issue: https://github.com/storj/storj/issues/5840
Change-Id: I4269495f9b7b09c77fbf1af1fc605e5c95bd7cbf
This change adds the user's passphrase prompt setting to the
/account/settings endpoints.
Issue: https://github.com/storj/storj/issues/5616
Change-Id: I48d470d49e82096fd090b74da323b279e342546e
Allow a longer encrypted key length to reduce 'key length is too big'
errors in gateway-mt. Gateway is enforcing an unencrypted key length
of 1024 bytes but when encrypted some keys are exceeding the current
limit.
updates https://github.com/storj/gateway-mt/issues/335
Change-Id: I861a2313e558c9f7d39569d21c7a3d429c83575c
Ensure that the value of "pricing packages enabled" flag on frontend is
the same as what is configured on the backend.
Change-Id: Id78771800a4973ebd3ad4e22f1953f6f71c75dd4
The "object not found" included an additional prefix "metabase:", which
broke uplink error message detection.
Ideally, changing an internal error message shouldn't break the uplink.
Change-Id: I5ce7789cc11742d3435af1ec555bc96927f1bedc
Current observer used with ranged loop is using massive amount of
memory because each range is generating separate set of bloom filters.
Each bloom filter can be up to 2MB of memory. That's a lot.
This change is initial change to reduce used memory by sharing bloom
filters between ranges and just synchronize access to them. This
implementation is rather simple and even naive but maybe it will be
enough without doing something more complex.
https://github.com/storj/storj/issues/5803
Change-Id: Ie62d19276aa9023076b1c97f712b788bce963cbe
Remove the command to apply free tier coupons from the generate invoices
command. Applying free tier coupons should instead be done outside the
normal time window for invoice generation to save time during the
invoicing cycle.
Change-Id: If8fecc558411d5a6fff9d5689143d72f3b709e55
This is refactor/cleanup change before I will start working on adding
separate GC observer with optimized memory consumption.
https://github.com/storj/storj/issues/5803
Change-Id: I854cb3797802a32942c25f2765dbb72be88bacbd
This change immplements methods for interacting with the project member
invitations table.
Resolves#5766
Change-Id: I0090c50f9fde5bcdae4ebdaa72cdcaa84d341b54
A table for storing pending project member invitations has been added.
This table is required to satisfy our new project invitation UX revamp.
References #5766
Change-Id: I6f948de66ed5b4dc81532564958ff7f48533cad2
Add a new billing command that will convert stripe customer balances into invoice items so that the charges can be processed normally by the invoicing workflow.
Change-Id: Iaa8350e7aca80a0f14e94eb8ef8b7d6ce0b5b3b8
Fixed nil pointer dereference panic.
Updated naming conventions so that PUT request and GET response bodies are the same (bandwidth, storage and segment).
Allowed usage of notations like 150GB, 2TB for storage and bandwidth limits.
Updated tests.
Issue:
https://github.com/storj/storj/issues/5674
Change-Id: I7ac27c00721a9b4bf507afa34cb05c4475a809ad
This change adds more tests to the autofreeze chore and the freeze
service according to the testplan linked in the issue below.
Issue: https://github.com/storj/storj/issues/5738
Change-Id: Ib2afaa283961b2e7ef6fb6e5613ee083ac7d79eb
This field is deprecated in favor of UserAgent; Removing these
references is the final step necessary before dropping the columns from
the database.
https: //github.com/storj/storj/issues/5432
Change-Id: I3a6619170dcf382f82dc8eddb73b6547eaf636f0
