| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-08-26 | |||
| 11:23:40 | alex_xu | sean-k-mooney: if forget turn the fallback off, the scheduler will chocie the host with shared_cpu_set? | |
| 11:24:11 | alex_xu | oh, it is not, since host.pcpu is empty | |
| 11:28:41 | sean-k-mooney | so with the fall back enabled on a cloudl with all new config options | |
| 11:28:52 | sean-k-mooney | and smt enabled | |
| 11:29:33 | sean-k-mooney | when we do the initall query the hyperthreading forbiden trait will cause the PCPU queury to retrun no hosts since they all have smt enabled | |
| 11:29:41 | alex_xu | sean-k-mooney: it is new deployed host, with dedicated and shared cpu set, that allows the mixed instance. But the fallback still enabled, then in the case of there is no more pCPU, then scheduler fallback to VCPU. Then host with shared cpu set allow that scheduling? | |
| 11:29:53 | sean-k-mooney | then the fallback with vcpus will return allocation agains tthe vcpu inventories | |
| 11:30:23 | sean-k-mooney | alex_xu: yes basicly | |
| 11:31:30 | sean-k-mooney | so what happens is we end up claiming vCPUs in placement and then pin the vm to the cpu_dedicated_set cpus | |
| 11:31:42 | alex_xu | if that is the case, a shared thread instance also can be scheduled to that host, right? | |
| 11:31:52 | sean-k-mooney | yes | |
| 11:32:03 | alex_xu | emm...that is still wrong | |
| 11:32:33 | sean-k-mooney | well shared instance and pinned can mix on the same host as of train | |
| 11:32:36 | sean-k-mooney | that is ok | |
| 11:32:41 | sean-k-mooney | but the allocation are wrong | |
| 11:32:44 | alex_xu | so we should ask the user turn off the fallback before upgrading to V? | |
| 11:33:05 | sean-k-mooney | we shoudl remove it when we removed vcpu_pin_set | |
| 11:33:31 | sean-k-mooney | with https://review.opendev.org/#/c/744021/3 its safe to leave it on | |
| 11:33:50 | sean-k-mooney | as placement will still do the wrong thing but the numa toplogy filter or the hardware module on teh compute host will block it | |
| 11:33:51 | alex_xu | but it doesn't fix the case for shared thread instance, right? | |
| 11:34:03 | sean-k-mooney | that is not broken | |
| 11:34:27 | sean-k-mooney | if you have cpu_policy=shared then it will request vcpus | |
| 11:34:40 | noonedeadpunk | hey everyone! I was wondering if it's possible to prohibit users to use ephemeral storage? I was trying to look through the policies but the only thing I found and was related is to allow use zero disk flavors. While it allows to use cinder volumes, it doesn't prohibits ephemeral ones... | |
| 11:34:41 | sean-k-mooney | the fallback only applies to cpu_policy=dedicated | |
| 11:35:05 | sean-k-mooney | and cpu_thread_policy only applies to cpu_polcy=dedicated | |
| 11:35:18 | alex_xu | yes | |
| 11:35:24 | noonedeadpunk | Eventually while I was looking I saw other ppl was also wondering about that opportuninty | |
| 11:35:26 | sean-k-mooney | noonedeadpunk: yes you can | |
| 11:35:28 | sean-k-mooney | via config | |
| 11:35:41 | alex_xu | I mean the case cpu_thread_policy is shared and cpu_policy is dedicated | |
| 11:35:59 | sean-k-mooney | oh in that case we dont claim vcpus at all | |
| 11:36:04 | sean-k-mooney | well we will | |
| 11:36:13 | sean-k-mooney | because of the bug | |
| 11:36:41 | sean-k-mooney | alex_xu: the vm is still being pinned to the cpu_dedicated_set | |
| 11:37:00 | sean-k-mooney | and its emulator treads will stll be pinned to the cpu_shared_set | |
| 11:37:13 | sean-k-mooney | we just wont claim PCPUs in plament | |
| 11:37:29 | sean-k-mooney | noonedeadpunk: https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.max_local_block_devices | |
| 11:37:44 | sean-k-mooney | noonedeadpunk: you can set that to 0 | |
| 11:37:44 | alex_xu | wait, the filter will stop that, since there is no enough pcpu in hostnuma.pcpu | |
| 11:38:00 | sean-k-mooney | alex_xu: no it wont | |
| 11:38:15 | noonedeadpunk | sean-k-mooney: oh, thanks! I was thinking about https://docs.openstack.org/nova/ussuri/configuration/config.html#libvirt.images_type setting to noop or smth but saw that's not an option | |
| 11:38:23 | alex_xu | oh...I see now, due to hyper thread | |
| 11:38:28 | sean-k-mooney | alex_xu: it will only stop it if thos pcpus are claimed | |
| 11:39:01 | sean-k-mooney | so if the host is more or less empty it will spawn if its getting full then yes it can reject it | |
| 11:39:08 | sean-k-mooney | but its not always going to | |
| 11:40:03 | sean-k-mooney | noonedeadpunk: are you trybing to force all vms to be boot from volume | |
| 11:40:11 | noonedeadpunk | yep | |
| 11:40:28 | sean-k-mooney | ah i see ya so that config option is the way to go | |
| 11:40:36 | noonedeadpunk | tbh ideal scenario would be to force all except some selected group | |
| 11:40:45 | sean-k-mooney | but just be aware we consider swap to be a local disk too | |
| 11:41:01 | sean-k-mooney | noonedeadpunk: this is a per host option by the way | |
| 11:41:12 | sean-k-mooney | so you can set it on a subset of hosts | |
| 11:41:26 | sean-k-mooney | and ideally use an aggreate to group them for scheduling | |
| 11:41:29 | noonedeadpunk | Oh, so it's effective on compute, not on api/scheduler hosts? | |
| 11:41:37 | sean-k-mooney | noonedeadpunk: correct | |
| 11:41:39 | noonedeadpunk | Ok, good to know! | |
| 11:41:49 | noonedeadpunk | thanks for the help and sorry for disturbing:) | |
| 11:42:09 | noonedeadpunk | really wasn't able to find this specific option with code words I was thinking about) | |
| 11:42:15 | sean-k-mooney | noonedeadpunk: what release are you on | |
| 11:42:16 | alex_xu | sean-k-mooney: ah, i see now. The host has enough pcpu for that instance, but the instance forbidden the hyperthread, that is why the placement reject, then fallback to VCPU. And then filter begin to filter based pcpu again, then the scheduling success | |
| 11:42:31 | sean-k-mooney | alex_xu: exactly | |
| 11:42:43 | alex_xu | finally i see that | |
| 11:42:50 | alex_xu | sean-k-mooney: thanks | |
| 11:42:53 | sean-k-mooney | its an edgecase to and edgecase but customers hit it imeditely | |
| 11:43:06 | noonedeadpunk | sean-k-mooney: I think train mostly | |
| 11:43:28 | noonedeadpunk | (maybe some U) | |
| 11:43:29 | sean-k-mooney | ok so i think trian has isolated aggreates support one sec whil i get the link | |
| 11:43:51 | sean-k-mooney | https://docs.openstack.org/nova/latest/reference/isolate-aggregates.html | |
| 11:44:38 | sean-k-mooney | what you can do is add a custom CUSTOM_BFV_ONLY trait to a subset of host and add it as a required trait in your BFV flavors | |
| 11:45:18 | sean-k-mooney | if you configure the aggreate to requrie it then it will block non BFV flavor from landing on those hosts | |
| 11:45:29 | noonedeadpunk | is scheduler.enable_isolated_aggregate_filtering option also compute specific? | |
| 11:45:31 | sean-k-mooney | then on those host you can also set the config option if you want too | |
| 11:45:39 | noonedeadpunk | (I think not) | |
| 11:45:42 | sean-k-mooney | noonedeadpunk: that is cloud wide | |
| 11:46:01 | sean-k-mooney | but this allows you to use traits and host aggrate to dynamically force certen behavior | |
| 11:46:20 | noonedeadpunk | yeah, that's pretty handy, thanks! | |
| 11:46:28 | sean-k-mooney | in this case it will prevent vms without the trait form landing on host in a given aggreate | |
| 11:46:49 | sean-k-mooney | noonedeadpunk: this would allow you to get teh same effect without needing to modify config options | |
| 11:46:51 | noonedeadpunk | (I would probably still prefer this to be configurable by policy though tbh, but whatever) | |
| 11:47:11 | sean-k-mooney | in what way | |
| 11:47:40 | sean-k-mooney | have a specific bfv_only flag of some kind? per host or cloud wide? | |
| 11:48:25 | sean-k-mooney | noonedeadpunk: im trying to understand if there is a gap that we might need to address beyond having a topic doc on how to use the exisitng feature to do this | |
| 11:48:53 | noonedeadpunk | So yeah, we have `os_compute_api:servers:create:zero_disk_flavor` so maybe option like `os_compute_api:servers:create:local_drive` which by default allowed for everyone | |
| 11:49:27 | noonedeadpunk | but this can be overriden and local_drive can be created only to specific groups of users | |
| 11:49:28 | sean-k-mooney | ah to block flavor creation with root, ephmeral or swap | |
| 11:49:59 | openstackgerrit | Merged openstack/nova stable/ussuri: libvirt: Do not reference VIR_ERR_DEVICE_MISSING when libvirt is < v4.1.0 https://review.opendev.org/747357 | |
| 11:50:00 | sean-k-mooney | the only real issue i see with that is root ephmerla and swap might not be local | |
| 11:50:05 | openstackgerrit | Merged openstack/nova master: Set different VirtualDevice.key https://review.opendev.org/713565 | |
| 11:50:12 | sean-k-mooney | e.g. if you are using ceph for storage | |
| 11:50:25 | noonedeadpunk | And eventually currently I am.... | |
| 11:50:49 | sean-k-mooney | but we proably could add a policy flag for each filed gmann is that a reasonable thing to do? | |
| 11:51:53 | sean-k-mooney | gmann: question i have is it resonable to add a policy flag to restirct creating flavor with non 0 root, swap or ephemeral storage | |
| 11:51:56 | noonedeadpunk | so, if I have ceph backend configured, than max_local_block_devices is not really applicable? | |
| 11:52:12 | sean-k-mooney | noonedeadpunk: more or less yes | |
| 11:52:31 | sean-k-mooney | i think it techniall is still enforced but its not really local | |
| 11:52:48 | noonedeadpunk | ah, ok, then this should work | |
| 11:53:17 | sean-k-mooney | im not sure if the point where we check that option know what storage is configured | |
| 11:53:25 | sean-k-mooney | we might have a skip for the rbd backend | |
| 11:54:29 | sean-k-mooney | noonedeadpunk: storage is one area i understand but avoid so the details are not something i rember and have to lookup each time | |
| 11:55:10 | bauzas | whoops | |