| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-10-17 | |||
| 12:31:44 | pas-ha[m] | boot | |
| 12:31:54 | sean-k-mooney | disableing computes is expictly for makign it imporssible to schdule a new worklaod | |
| 12:32:02 | sean-k-mooney | so https://bugs.launchpad.net/nova/+bug/2076089 is not valid | |
| 12:32:13 | pas-ha[m] | but it was working before 😕 | |
| 12:32:18 | sean-k-mooney | the disables feature is descided to explcliy block that | |
| 12:32:23 | pas-ha[m] | and is a normal part of node onboarding | |
| 12:32:30 | sean-k-mooney | you can start existing vms on a disbale compute | |
| 12:32:51 | pas-ha[m] | how does one test a node w/o exposing it to the whole cloud? | |
| 12:33:24 | sean-k-mooney | in general you dont but in practice va a host aggreate and something like the tenant filter | |
| 12:33:25 | pas-ha[m] | short of adding a new az of course... that users still can see and attempt to boot on | |
| 12:33:53 | sean-k-mooney | you and do it in a way that users wont see using a number of methods | |
| 12:33:57 | pas-ha[m] | yeah... sad ☹️ though still can migrate to a disabled host using old API | |
| 12:34:14 | sean-k-mooney | you can use a private flavor and required trait | |
| 12:34:24 | sean-k-mooney | the tenat isolation filter or some other methods | |
| 12:34:55 | sean-k-mooney | pas-ha[m]: basiclaly set https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.enable_new_services to false | |
| 12:34:57 | pas-ha[m] | with required trait that does not prevent from other flavors seeing it as available | |
| 12:35:30 | sean-k-mooney | that will cause it to be enroled as disabeld then you can put it into a seperate aggret that only your test proejct can use and test it after you enable it | |
| 12:37:53 | sean-k-mooney | pas-ha[m]: the basic workflow is enrole new nodes as disabled, put them in a restircted test aggreate, enable them, test them and finally remove them for the aggrate | |
| 12:38:44 | sean-k-mooney | pas-ha[m]: https://docs.openstack.org/nova/latest/admin/aggregates.html#tenant-isolation-with-placement is what i generally recommend fo this | |
| 12:39:40 | sean-k-mooney | but there are other approchs that work too | |
| 12:40:12 | sean-k-mooney | i prefer doing it with tenant isolation so that you can use the actual flavors/images you intent do use and just delete the placment aggreate when your done | |
| 12:41:01 | sean-k-mooney | pas-ha[m]: you cant migrate via the old api by the way, the way disabel works will filter out the host so --foce wont work | |
| 12:41:24 | pas-ha[m] | nope,it does work, tested on Antelope | |
| 12:41:49 | sean-k-mooney | it should not because the placment prefilter cant be turned off | |
| 12:42:01 | pas-ha[m] | ``` | |
| 12:42:03 | sean-k-mooney | if it works its a bug | |
| 12:42:13 | pas-ha[m] | * import openstack... (full message at <https://matrix.org/oftc/media/v1/media/download/AUsfx3faFN1prujTkMr2mB2XbtUHmg98qQVs5w-ulEaFy1UTw3_HViUOe21mF6pw3ijesukW_RFxPoO-qfRuuFpCeS4pkHBQAG1hdHJpeC5vcmcvWEJQRlZGcGRCc0JuWWFiZmdocmlKaGlw>) | |
| 12:43:02 | sean-k-mooney | pas-ha[m] https://github.com/openstack/nova/blob/master/nova/scheduler/request_filter.py#L242-L255 | |
| 12:43:02 | pas-ha[m] | API ref says the same btw. if you use older that 2.30 api, the scheduler does not apply when migrating to a specific host | |
| 12:43:29 | sean-k-mooney | pas-ha[m]: every request we make to placmenet unconditon fobids host with the COMPUTE_STAUS_DISBALED trait | |
| 12:44:12 | pas-ha[m] | afaiu in older api scheduler is not involved at all I guess... | |
| 12:44:28 | pas-ha[m] | https://docs.openstack.org/api-ref/compute/#live-migrate-server-os-migratelive-action | |
| 12:44:29 | sean-k-mooney | it is but it does very minimal checks | |
| 12:44:29 | pas-ha[m] | Prior to microversion 2.30, specifying a host will bypass validation by the scheduler, which could result in failures to actually migrate the instance to the specified host, or over-subscription of the host. | |
| 12:44:40 | sean-k-mooney | but the palcment one i did nto think was skiped | |
| 12:44:54 | sean-k-mooney | basically it does nto run any fitler or weighers | |
| 12:45:00 | sean-k-mooney | but it does confirm the host extist | |
| 12:45:28 | sean-k-mooney | its possibel we have a bug and we are early outing before check with placment | |
| 12:49:21 | pas-ha[m] | this is also also something that was working before the AZFilter was removed (we had to re-instate it back in downstream 😕 ) | |
| 12:49:21 | pas-ha[m] | sean-k-mooney: and, changing to other scheduling problems, what would you think on https://bugs.launchpad.net/nova/+bug/2078669 (and https://review.opendev.org/c/openstack/nova/+/927744)? | |
| 12:50:29 | pas-ha[m] | how can I boot an instance 'not in any of those AZs that I see' ? | |
| 12:52:27 | sean-k-mooney | thats a diffent issue and strictly speakign also not supproted | |
| 12:53:02 | sean-k-mooney | pas-ha[m]: see the red warining box in https://docs.openstack.org/nova/latest/admin/availability-zones.html | |
| 12:54:20 | sean-k-mooney | i am more open to supproting the defautl nova az but bauzas is stongly against it. we have docuemented taht you should not ever request the nova az effectivly since AZs were first added | |
| 12:57:42 | bauzas | sean-k-mooney: hmmm, what is the question ? | |
| 12:57:49 | pas-ha[m] | yeah, I know that doc. Still, this is in admin part of docs, and users rarely are aware of it, and they do see 'nova' as in az list... and again, if I have hosts in az 'foo' and hosts in no explicit az ('nova'), previously it was possible to boot "not in foo" == boot in 'nova' az. Now its not possible any more 😕 | |
| 12:58:28 | pas-ha[m] | btw, the api ref says that regular users should not see 'nova' az in az list, but I am pretty sure that's still the case even on master | |
| 12:58:48 | pas-ha[m] | bauzas: https://bugs.launchpad.net/nova/+bug/2078669 and the patch to that | |
| 12:59:07 | sean-k-mooney | horizon adds it to the lsit if no azs are retruned i belive | |
| 12:59:28 | pas-ha[m] | openstack cli does show it too | |
| 13:00:30 | bauzas | pas-ha[m]: tbc, the concern is that by default 'nova' isn't an AZ | |
| 13:01:06 | bauzas | if you want, you can change the default value for instances so there you could create a specific aggregate with a 'nova' AZ | |
| 13:01:34 | bauzas | but if you set an instance with '-az nova' then we'll pin into it | |
| 13:03:17 | pas-ha[m] | I know, but it is represented as such to the users still. And the user sees az 'expensive' and az 'nova', user thinks 'i definitely do not need 'expensive', and uses 'nova'. Before, with AZFilter being available, the instances landed where user intended. Now they are not. | |
| 13:05:18 | sean-k-mooney | ubuntu@devstack-iso:~$ openstack --os-cloud=devstack --debug availability zone list --compute 2>&1| nc termbin.com 9999 | |
| 13:05:20 | sean-k-mooney | https://termbin.com/b8q9 | |
| 13:05:26 | sean-k-mooney | bauzas: we do have a bug ^ | |
| 13:06:38 | pas-ha[m] | I proposed a patch that would make az 'nova' (or whatever default az is) be treated as 'not any host that belongs to any explicit az' (if there's no actual explicit az nova of course). IMO that should be exaclty what the user intended. | |
| 13:06:53 | sean-k-mooney | pas-ha[m]: that would break things | |
| 13:07:31 | pas-ha[m] | this was effectively working like that until AZFilter was removed recently | |
| 13:07:43 | pas-ha[m] | (in bobcat or caracal) | |
| 13:07:43 | sean-k-mooney | the problem with that is we do not allow moving host betwene azs | |
| 13:08:15 | sean-k-mooney | if they have instnace on them that is | |
| 13:08:23 | pas-ha[m] | yeah, but I am not forcing any AZ here. I am only forcing placement to discard any other candidates | |
| 13:09:04 | sean-k-mooney | you cant do that effiectnly without pasically passing all az aggreatd in a not | |
| 13:09:11 | sean-k-mooney | which does not scale | |
| 13:09:19 | sean-k-mooney | and will hit the max url limit in large clouds | |
| 13:09:44 | sean-k-mooney | the only way to make this work woudl be to group all hosts not in an az into a seperate aggreate | |
| 13:09:58 | sean-k-mooney | and then explcitly request that | |
| 13:10:28 | pas-ha[m] | yeah, that's an interesting thought.. about the url length at least | |
| 13:10:52 | bauzas | not sure I understand the problem sorry | |
| 13:11:04 | bauzas | the fact that we return 'nova' per the user AZ list ? | |
| 13:11:06 | sean-k-mooney | bauzas: as a non admin if you do an az list you get nova | |
| 13:11:16 | bauzas | if so, we should *not* return it, I agree | |
| 13:11:22 | sean-k-mooney | which mean it looks valid to and end suer | |
| 13:11:33 | sean-k-mooney | because they cant no what the defautl az is | |
| 13:11:42 | sean-k-mooney | *default internal az is | |
| 13:12:03 | sean-k-mooney | actuilly no i was right the first time https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.default_availability_zone | |
| 13:12:32 | bauzas | okay, then I agree: we shouldn't return the default AZ in that list | |
| 13:12:35 | bauzas | https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.default_availability_zone | |
| 13:12:48 | sean-k-mooney | you can actully create a real az callded nova and change that to something else | |
| 13:13:03 | sean-k-mooney | the defautl az can be returned if its an actual az | |
| 13:13:07 | bauzas | for defaulting to pin a specific AZ, this is https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.default_schedule_zone | |
| 13:13:28 | bauzas | we could even make default_az to None, I don't see why we need to have 'nova' | |
| 13:13:37 | sean-k-mooney | i was just thinking about that | |
| 13:13:41 | bauzas | but we could and should discuss this at the PTG | |
| 13:13:55 | sean-k-mooney | ya i dont know of the top of my head anything that would break | |
| 13:14:27 | sean-k-mooney | it would stop instance reporting an az if they are on a host that are not part of any az | |
| 13:14:34 | sean-k-mooney | but that is more correct then "nova" | |
| 13:14:36 | pas-ha[m] | as I said, api ref already promises that non-admin should not see it | |
| 13:14:37 | pas-ha[m] | https://docs.openstack.org/api-ref/compute/#create-server | |
| 13:14:37 | pas-ha[m] | > The default availability zone is named nova. This AZ is only shown when listing the availability zones as an admin. | |
| 13:15:02 | sean-k-mooney | pas-ha[m] reight beu twe have a bug because that is not what the api is doing | |
| 13:16:22 | sean-k-mooney | http://192.168.50.7:80 "GET /compute/v2.1/os-availability-zone HTTP/1.1" 200 97 | |
| 13:16:24 | sean-k-mooney | RESP: [200] Connection: close Content-Length: 97 Content-Type: application/json Date: Thu, 17 Oct 2024 13:15:54 GMT OpenStack-API-Version: compute 2.1 Server: Apache/2.4.58 (Ubuntu) Vary: OpenStack-API-Version,X-OpenStack-Nova-API-Version X-OpenStack-Nova-API-Version: 2.1 x-compute-request-id: req-17af8a83-82f0-45ce-81fd-c6d54a0c0ad9 x-openstack-request-id: | |
| 13:16:26 | sean-k-mooney | req-17af8a83-82f0-45ce-81fd-c6d54a0c0ad9 | |
| 13:16:28 | sean-k-mooney | RESP BODY: {"availabilityZoneInfo": [{"zoneName": "nova", "zoneState": {"available": true}, "hosts": null}]} | |
| 13:16:56 | sean-k-mooney | ill quickly checkc but im 99% shure i dont have any hostaggrate defiend on this devstack | |
| 13:17:24 | pas-ha[m] | And I am afraid something would break if we start doing that. This would effectively force operators to always put hosts into some explicit AZ, because users who are used to pick the AZ from the list will just use one of those the list presents them, making hosts not in any AZ underused | |
| 13:18:30 | sean-k-mooney | well az are optional | |