Commit Graph

13 Commits

Author SHA1 Message Date
Márton Elek
98921f9faa satellite/overlay: fix placement selection config parsing
When we do `satellite run api --placement '...'`, the placement rules are not parsed well.

The problem is based on `viper.AllSettings()`, and the main logic is sg. like this (from a new unit test):

```
		r := ConfigurablePlacementRule{}
		err := r.Set(p)
		require.NoError(t, err)
		serialized := r.String()

		r2 := ConfigurablePlacementRule{}
		err = r2.Set(serialized)
		require.NoError(t, err)

		require.Equal(t, p, r2.String())
```

All settings evaluates the placement rules in `ConfigurablePlacementRules` and stores the string representation.

The problem is that we don't have proper `String()` implementation (it prints out the structs instead of the original definition.

There are two main solutions for this problem:

 1. We can fix the `String()`. When we parse a placement rule, the `String()` method should print out the original definition
 2. We can switch to use pure string as configuration parameter, and parse the rules only when required.

I feel that 1 is error prone, we can do it (and in this patch I added a lot of `String()` implementations, but it's hard to be sure that our `String()` logic is inline with the parsing logic.

Therefore I decided to make the configuration value of the placements a string (or a wrapper around string).

That's the main reason why this patch seems to be big, as I updated all the usages.

But the main part is in beginning of the `placement.go` (configuration parsing is not a pflag.Value implementation any more, but a separated step).

And `filter.go`, (a few more String implementation for filters.

https://github.com/storj/storj/issues/6248

Change-Id: I47c762d3514342b76a2e85683b1c891502a0756a
2023-09-21 14:31:41 +00:00
Márton Elek
f4fe983b1e satellite/{placement,nodeselection}: introduce empty() and notEmpty() for tag value selection
It helps to implement rules like `tag("nodeid","select",notEmpty())

Change-Id: If7a4532eacc0e4e670ffe81d504aab9d5b34302f
2023-09-14 19:30:29 +00:00
Márton Elek
5c12a3406d satellite/nodeselection: improve annotation composability
We would like to make it easier to accept multiple annotations.

Examples:
```
country("GB") && annotation(...)
annotated(annotated(X,...),...)
```

Change-Id: I92e622e8b985b314dadddf83b17976c245eb2069
2023-08-28 09:27:04 +00:00
Márton Elek
b218002752 satellite/overlay/placement: improve placement configurability with &&, placement, region and ! support in country
This patch makes it easier to configure existing placement rules only with string.

 1. placement(n) rule can be used to reuse earlier definitions
 2 .&& can be used in addition to all(n1,n2)
 3. country(c) accepts exclusions (like '!RU'), regions ('EU','EEA'), all and none

See the 'full example' unit test, which uses all of these, in a realistic example.

https://github.com/storj/storj/issues/6126

Change-Id: Ica76f016ebd002eb7ea8103d4258bacd6a6d77bf
2023-08-17 16:12:53 +00:00
Márton Elek
da08117fcd satellite/~placement: do not ignore placement check for placement=0
There are cases when we would like to override the default placement=0 rule.

For example when we would like to exclude tagged nodes from the selection (by default).

Therefore we couldn't use a shortcut any more, we should always check the placement rules, even if we use placement=0.

TODO: we need to update common, and rename `EveryCountry` to `DefaultPlacement`, just to avoid confusion.

https://github.com/storj/storj/issues/6126

Change-Id: Iba6c655bd623e04351ea7ff91fd741785dc193e4
2023-08-16 07:06:56 +00:00
Márton Elek
c08792f066 satellite/overlay: implement an exclude filter for placement configuration
https://github.com/storj/storj/issues/6126

Change-Id: I05215b5d46bec958001cc020edf1fa97b00d3299
2023-08-15 17:29:29 +00:00
Márton Elek
0e17b1018c satellite/{nodeselection,overlay}: support annotations on node filters
Change-Id: I844d8a25042750aae189175842113e2f052d5b17
2023-08-15 16:49:57 +00:00
Márton Elek
6f002f4220
satellite/overlay: NR placement should exclude nodes without geofencing information
https://github.com/storj/storj-private/issues/378

Change-Id: If2af02083496e5a8eefe27beabb406388ee50644
2023-07-31 09:55:54 +02:00
Egon Elbre
465941b345 satellite/{nodeselection,overlay}: use location.Set
location.Set is faster for comparisons.

Updates #6028

Change-Id: I764eb5cafc507f908e4168b16a7994cc7721ce4d
2023-07-11 17:16:30 +00:00
Egon Elbre
9370bc4580 satellite/{nodeselection,overlay}: bump common and fix some potential issues
* Handle failed country code conversion.
* Avoid potential issues with a data-race due to shared slice.

Updates #6028

Change-Id: If7beef2619abd084e1f4109de2d323f834a6090a
2023-07-11 11:13:41 +00:00
Márton Elek
97a89c3476 satellite: switch to use nodefilters instead of old placement.AllowedCountry
placement.AllowedCountry is the old way to specify placement, with the new approach we can use a more generic (dynamic method), which can check full node information instead of just the country code.

The 90% of this patch is just search and replace:

 * we need to use NodeFilters instead of placement.AllowedCountry
 * which means, we need an initialized PlacementRules available everywhere
 * which means we need to configure the placement rules

The remaining 10% is the placement.go, where we introduced a new type of configuration (lightweight expression language) to define any kind of placement without code change.

Change-Id: Ie644b0b1840871b0e6bbcf80c6b50a947503d7df
2023-07-07 16:55:45 +00:00
Márton Elek
70cdca5d3c
satellite: move satellite/nodeselection/uploadselection => satellite/nodeselection
All the files in uploadselection are (in fact) related to generic node selection, and used not only for upload,
but for download, repair, etc...

Change-Id: Ie4098318a6f8f0bbf672d432761e87047d3762ab
2023-07-07 10:32:03 +02:00
Márton Elek
ddf1f1c340 satellite/{nodeselection,overlay}: NodeFilters for dynamic placement implementations
Change-Id: Ica3a7b535fa6736cd8fb12066e615b70e1fa65d6
2023-07-06 12:08:01 +00:00