Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-08-26
11:09:10 stephenfin This patch should be backported to Ussuri and Train
11:09:26 sean-k-mooney yes
11:10:14 stephenfin Is there something upgrade specific I'm missing, perhaps?
11:10:14 sean-k-mooney alex_xu: it was an edgecase we forgot to block, we taught it would be blocked by the numa toplogy filter but the check was missing
11:10:32 alex_xu if the host with legacy config is running isolated thread instance, but the 744021 stop the isolated thread instance? (I'm still try to ramp up my mind)
11:10:53 sean-k-mooney alex_xu: if its using legacy configs it will allow it to boot
11:11:01 sean-k-mooney alex_xu: it only changes the behavior for new configs
11:11:43 alex_xu oh, right
11:12:14 sean-k-mooney alex_xu: we dont allow the shared and dedicated set to overlap so the only time they contian the same values is if we are using the legacy config options
11:12:51 sean-k-mooney so and host_cell.pcpuset != host_cell.cpuset
11:13:05 sean-k-mooney is an indirect check for is the host using new config options
11:13:08 lyarwood elod: sorry missed that https://review.opendev.org/#/c/747357/ wasn't included
11:13:11 sean-k-mooney that will also work in the schduler
11:13:14 lyarwood elod: can we wait for that and then release?
11:13:23 sean-k-mooney since we cant actully check the config directly there
11:14:33 jsuchome lyarwood: gibi stephenfin https://review.opendev.org/#/c/574301/ finally green again, any further changes necessary?
11:15:50 elod lyarwood: oh, of course, sorry, I forgot to check what else on the queue :S
11:16:30 openstackgerrit Merged openstack/nova master: Provider Config File: Functions to merge provider configs to provider tree https://review.opendev.org/676522
11:17:00 lyarwood jsuchome: https://review.opendev.org/#/c/746904/ - I just need to respin this after lunch and I think we should be good to go
11:17:43 alex_xu sean-k-mooney: but with new config, why we meet the case fallback to query VCPU?
11:18:44 alex_xu fallback query should find the host with legacy config
11:19:53 sean-k-mooney alex_xu: not if all host are using the new configs and you forget to trun off the fallback
11:20:17 sean-k-mooney the fallback can select the vcpu on host with the new config options
11:20:33 sean-k-mooney there is noting in the fallback query to prevent that
11:20:52 sean-k-mooney we were relying on the numa toplogy filter to handel that case and this check was missing
11:21:44 sean-k-mooney alex_xu: for what its worth i was 99% certin the numa toplogy filter would block this untill i repoduced the issue locally
11:23:40 alex_xu sean-k-mooney: if forget turn the fallback off, the scheduler will chocie the host with shared_cpu_set?
11:24:11 alex_xu oh, it is not, since host.pcpu is empty
11:28:41 sean-k-mooney so with the fall back enabled on a cloudl with all new config options
11:28:52 sean-k-mooney and smt enabled
11:29:33 sean-k-mooney when we do the initall query the hyperthreading forbiden trait will cause the PCPU queury to retrun no hosts since they all have smt enabled
11:29:41 alex_xu sean-k-mooney: it is new deployed host, with dedicated and shared cpu set, that allows the mixed instance. But the fallback still enabled, then in the case of there is no more pCPU, then scheduler fallback to VCPU. Then host with shared cpu set allow that scheduling?
11:29:53 sean-k-mooney then the fallback with vcpus will return allocation agains tthe vcpu inventories
11:30:23 sean-k-mooney alex_xu: yes basicly
11:31:30 sean-k-mooney so what happens is we end up claiming vCPUs in placement and then pin the vm to the cpu_dedicated_set cpus
11:31:42 alex_xu if that is the case, a shared thread instance also can be scheduled to that host, right?
11:31:52 sean-k-mooney yes
11:32:03 alex_xu emm...that is still wrong
11:32:33 sean-k-mooney well shared instance and pinned can mix on the same host as of train
11:32:36 sean-k-mooney that is ok
11:32:41 sean-k-mooney but the allocation are wrong
11:32:44 alex_xu so we should ask the user turn off the fallback before upgrading to V?
11:33:05 sean-k-mooney we shoudl remove it when we removed vcpu_pin_set
11:33:31 sean-k-mooney with https://review.opendev.org/#/c/744021/3 its safe to leave it on
11:33:50 sean-k-mooney as placement will still do the wrong thing but the numa toplogy filter or the hardware module on teh compute host will block it
11:33:51 alex_xu but it doesn't fix the case for shared thread instance, right?
11:34:03 sean-k-mooney that is not broken
11:34:27 sean-k-mooney if you have cpu_policy=shared then it will request vcpus
11:34:40 noonedeadpunk hey everyone! I was wondering if it's possible to prohibit users to use ephemeral storage? I was trying to look through the policies but the only thing I found and was related is to allow use zero disk flavors. While it allows to use cinder volumes, it doesn't prohibits ephemeral ones...
11:34:41 sean-k-mooney the fallback only applies to cpu_policy=dedicated
11:35:05 sean-k-mooney and cpu_thread_policy only applies to cpu_polcy=dedicated
11:35:18 alex_xu yes
11:35:24 noonedeadpunk Eventually while I was looking I saw other ppl was also wondering about that opportuninty
11:35:26 sean-k-mooney noonedeadpunk: yes you can
11:35:28 sean-k-mooney via config
11:35:41 alex_xu I mean the case cpu_thread_policy is shared and cpu_policy is dedicated
11:35:59 sean-k-mooney oh in that case we dont claim vcpus at all
11:36:04 sean-k-mooney well we will
11:36:13 sean-k-mooney because of the bug
11:36:41 sean-k-mooney alex_xu: the vm is still being pinned to the cpu_dedicated_set
11:37:00 sean-k-mooney and its emulator treads will stll be pinned to the cpu_shared_set
11:37:13 sean-k-mooney we just wont claim PCPUs in plament
11:37:29 sean-k-mooney noonedeadpunk: https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.max_local_block_devices
11:37:44 sean-k-mooney noonedeadpunk: you can set that to 0
11:37:44 alex_xu wait, the filter will stop that, since there is no enough pcpu in hostnuma.pcpu
11:38:00 sean-k-mooney alex_xu: no it wont
11:38:15 noonedeadpunk sean-k-mooney: oh, thanks! I was thinking about https://docs.openstack.org/nova/ussuri/configuration/config.html#libvirt.images_type setting to noop or smth but saw that's not an option
11:38:23 alex_xu oh...I see now, due to hyper thread
11:38:28 sean-k-mooney alex_xu: it will only stop it if thos pcpus are claimed
11:39:01 sean-k-mooney so if the host is more or less empty it will spawn if its getting full then yes it can reject it
11:39:08 sean-k-mooney but its not always going to
11:40:03 sean-k-mooney noonedeadpunk: are you trybing to force all vms to be boot from volume
11:40:11 noonedeadpunk yep
11:40:28 sean-k-mooney ah i see ya so that config option is the way to go
11:40:36 noonedeadpunk tbh ideal scenario would be to force all except some selected group
11:40:45 sean-k-mooney but just be aware we consider swap to be a local disk too
11:41:01 sean-k-mooney noonedeadpunk: this is a per host option by the way
11:41:12 sean-k-mooney so you can set it on a subset of hosts
11:41:26 sean-k-mooney and ideally use an aggreate to group them for scheduling
11:41:29 noonedeadpunk Oh, so it's effective on compute, not on api/scheduler hosts?
11:41:37 sean-k-mooney noonedeadpunk: correct
11:41:39 noonedeadpunk Ok, good to know!
11:41:49 noonedeadpunk thanks for the help and sorry for disturbing:)
11:42:09 noonedeadpunk really wasn't able to find this specific option with code words I was thinking about)
11:42:15 sean-k-mooney noonedeadpunk: what release are you on
11:42:16 alex_xu sean-k-mooney: ah, i see now. The host has enough pcpu for that instance, but the instance forbidden the hyperthread, that is why the placement reject, then fallback to VCPU. And then filter begin to filter based pcpu again, then the scheduling success
11:42:31 sean-k-mooney alex_xu: exactly
11:42:43 alex_xu finally i see that
11:42:50 alex_xu sean-k-mooney: thanks
11:42:53 sean-k-mooney its an edgecase to and edgecase but customers hit it imeditely
11:43:06 noonedeadpunk sean-k-mooney: I think train mostly
11:43:28 noonedeadpunk (maybe some U)
11:43:29 sean-k-mooney ok so i think trian has isolated aggreates support one sec whil i get the link
11:43:51 sean-k-mooney https://docs.openstack.org/nova/latest/reference/isolate-aggregates.html
11:44:38 sean-k-mooney what you can do is add a custom CUSTOM_BFV_ONLY trait to a subset of host and add it as a required trait in your BFV flavors
11:45:18 sean-k-mooney if you configure the aggreate to requrie it then it will block non BFV flavor from landing on those hosts
11:45:29 noonedeadpunk is scheduler.enable_isolated_aggregate_filtering option also compute specific?
11:45:31 sean-k-mooney then on those host you can also set the config option if you want too
11:45:39 noonedeadpunk (I think not)
11:45:42 sean-k-mooney noonedeadpunk: that is cloud wide

Earlier   Later