| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-08-26 | |||
| 11:05:47 | sean-k-mooney | ill get the path on sec | |
| 11:06:20 | alex_xu | stephenfin: sean-k-mooney ,I'm thinking of 744021 may stop the upgrade from U to V if the operate transft from legacy to new config in that time | |
| 11:06:53 | sean-k-mooney | rmart04: cat /sys/bus/node/devices/node*/meminfo | |
| 11:07:31 | alex_xu | if we force that legacy config change finished in U, then it should be fine | |
| 11:08:09 | sean-k-mooney | alex_xu: its there because we had customer upgraind into an invlaid config | |
| 11:08:13 | stephenfin | alex_xu: I don't understand the issue | |
| 11:08:41 | sean-k-mooney | they had vms with the old configuration where they were using isolate but the new config options | |
| 11:08:44 | stephenfin | A user should never get into a situation where they have new-style config but a pinned instance is consuming VCPU instead of PCPU | |
| 11:08:48 | sean-k-mooney | so they were using vcpus instead of pcpus | |
| 11:09:02 | sean-k-mooney | even thogh the host reported pcpus | |
| 11:09:10 | stephenfin | This patch should be backported to Ussuri and Train | |
| 11:09:26 | sean-k-mooney | yes | |
| 11:10:14 | stephenfin | Is there something upgrade specific I'm missing, perhaps? | |
| 11:10:14 | sean-k-mooney | alex_xu: it was an edgecase we forgot to block, we taught it would be blocked by the numa toplogy filter but the check was missing | |
| 11:10:32 | alex_xu | if the host with legacy config is running isolated thread instance, but the 744021 stop the isolated thread instance? (I'm still try to ramp up my mind) | |
| 11:10:53 | sean-k-mooney | alex_xu: if its using legacy configs it will allow it to boot | |
| 11:11:01 | sean-k-mooney | alex_xu: it only changes the behavior for new configs | |
| 11:11:43 | alex_xu | oh, right | |
| 11:12:14 | sean-k-mooney | alex_xu: we dont allow the shared and dedicated set to overlap so the only time they contian the same values is if we are using the legacy config options | |
| 11:12:51 | sean-k-mooney | so and host_cell.pcpuset != host_cell.cpuset | |
| 11:13:05 | sean-k-mooney | is an indirect check for is the host using new config options | |
| 11:13:08 | lyarwood | elod: sorry missed that https://review.opendev.org/#/c/747357/ wasn't included | |
| 11:13:11 | sean-k-mooney | that will also work in the schduler | |
| 11:13:14 | lyarwood | elod: can we wait for that and then release? | |
| 11:13:23 | sean-k-mooney | since we cant actully check the config directly there | |
| 11:14:33 | jsuchome | lyarwood: gibi stephenfin https://review.opendev.org/#/c/574301/ finally green again, any further changes necessary? | |
| 11:15:50 | elod | lyarwood: oh, of course, sorry, I forgot to check what else on the queue :S | |
| 11:16:30 | openstackgerrit | Merged openstack/nova master: Provider Config File: Functions to merge provider configs to provider tree https://review.opendev.org/676522 | |
| 11:17:00 | lyarwood | jsuchome: https://review.opendev.org/#/c/746904/ - I just need to respin this after lunch and I think we should be good to go | |
| 11:17:43 | alex_xu | sean-k-mooney: but with new config, why we meet the case fallback to query VCPU? | |
| 11:18:44 | alex_xu | fallback query should find the host with legacy config | |
| 11:19:53 | sean-k-mooney | alex_xu: not if all host are using the new configs and you forget to trun off the fallback | |
| 11:20:17 | sean-k-mooney | the fallback can select the vcpu on host with the new config options | |
| 11:20:33 | sean-k-mooney | there is noting in the fallback query to prevent that | |
| 11:20:52 | sean-k-mooney | we were relying on the numa toplogy filter to handel that case and this check was missing | |
| 11:21:44 | sean-k-mooney | alex_xu: for what its worth i was 99% certin the numa toplogy filter would block this untill i repoduced the issue locally | |
| 11:23:40 | alex_xu | sean-k-mooney: if forget turn the fallback off, the scheduler will chocie the host with shared_cpu_set? | |
| 11:24:11 | alex_xu | oh, it is not, since host.pcpu is empty | |
| 11:28:41 | sean-k-mooney | so with the fall back enabled on a cloudl with all new config options | |
| 11:28:52 | sean-k-mooney | and smt enabled | |
| 11:29:33 | sean-k-mooney | when we do the initall query the hyperthreading forbiden trait will cause the PCPU queury to retrun no hosts since they all have smt enabled | |
| 11:29:41 | alex_xu | sean-k-mooney: it is new deployed host, with dedicated and shared cpu set, that allows the mixed instance. But the fallback still enabled, then in the case of there is no more pCPU, then scheduler fallback to VCPU. Then host with shared cpu set allow that scheduling? | |
| 11:29:53 | sean-k-mooney | then the fallback with vcpus will return allocation agains tthe vcpu inventories | |
| 11:30:23 | sean-k-mooney | alex_xu: yes basicly | |
| 11:31:30 | sean-k-mooney | so what happens is we end up claiming vCPUs in placement and then pin the vm to the cpu_dedicated_set cpus | |
| 11:31:42 | alex_xu | if that is the case, a shared thread instance also can be scheduled to that host, right? | |
| 11:31:52 | sean-k-mooney | yes | |
| 11:32:03 | alex_xu | emm...that is still wrong | |
| 11:32:33 | sean-k-mooney | well shared instance and pinned can mix on the same host as of train | |
| 11:32:36 | sean-k-mooney | that is ok | |
| 11:32:41 | sean-k-mooney | but the allocation are wrong | |
| 11:32:44 | alex_xu | so we should ask the user turn off the fallback before upgrading to V? | |
| 11:33:05 | sean-k-mooney | we shoudl remove it when we removed vcpu_pin_set | |
| 11:33:31 | sean-k-mooney | with https://review.opendev.org/#/c/744021/3 its safe to leave it on | |
| 11:33:50 | sean-k-mooney | as placement will still do the wrong thing but the numa toplogy filter or the hardware module on teh compute host will block it | |
| 11:33:51 | alex_xu | but it doesn't fix the case for shared thread instance, right? | |
| 11:34:03 | sean-k-mooney | that is not broken | |
| 11:34:27 | sean-k-mooney | if you have cpu_policy=shared then it will request vcpus | |
| 11:34:40 | noonedeadpunk | hey everyone! I was wondering if it's possible to prohibit users to use ephemeral storage? I was trying to look through the policies but the only thing I found and was related is to allow use zero disk flavors. While it allows to use cinder volumes, it doesn't prohibits ephemeral ones... | |
| 11:34:41 | sean-k-mooney | the fallback only applies to cpu_policy=dedicated | |
| 11:35:05 | sean-k-mooney | and cpu_thread_policy only applies to cpu_polcy=dedicated | |
| 11:35:18 | alex_xu | yes | |
| 11:35:24 | noonedeadpunk | Eventually while I was looking I saw other ppl was also wondering about that opportuninty | |
| 11:35:26 | sean-k-mooney | noonedeadpunk: yes you can | |
| 11:35:28 | sean-k-mooney | via config | |
| 11:35:41 | alex_xu | I mean the case cpu_thread_policy is shared and cpu_policy is dedicated | |
| 11:35:59 | sean-k-mooney | oh in that case we dont claim vcpus at all | |
| 11:36:04 | sean-k-mooney | well we will | |
| 11:36:13 | sean-k-mooney | because of the bug | |
| 11:36:41 | sean-k-mooney | alex_xu: the vm is still being pinned to the cpu_dedicated_set | |
| 11:37:00 | sean-k-mooney | and its emulator treads will stll be pinned to the cpu_shared_set | |
| 11:37:13 | sean-k-mooney | we just wont claim PCPUs in plament | |
| 11:37:29 | sean-k-mooney | noonedeadpunk: https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.max_local_block_devices | |
| 11:37:44 | sean-k-mooney | noonedeadpunk: you can set that to 0 | |
| 11:37:44 | alex_xu | wait, the filter will stop that, since there is no enough pcpu in hostnuma.pcpu | |
| 11:38:00 | sean-k-mooney | alex_xu: no it wont | |
| 11:38:15 | noonedeadpunk | sean-k-mooney: oh, thanks! I was thinking about https://docs.openstack.org/nova/ussuri/configuration/config.html#libvirt.images_type setting to noop or smth but saw that's not an option | |
| 11:38:23 | alex_xu | oh...I see now, due to hyper thread | |
| 11:38:28 | sean-k-mooney | alex_xu: it will only stop it if thos pcpus are claimed | |
| 11:39:01 | sean-k-mooney | so if the host is more or less empty it will spawn if its getting full then yes it can reject it | |
| 11:39:08 | sean-k-mooney | but its not always going to | |
| 11:40:03 | sean-k-mooney | noonedeadpunk: are you trybing to force all vms to be boot from volume | |
| 11:40:11 | noonedeadpunk | yep | |
| 11:40:28 | sean-k-mooney | ah i see ya so that config option is the way to go | |
| 11:40:36 | noonedeadpunk | tbh ideal scenario would be to force all except some selected group | |
| 11:40:45 | sean-k-mooney | but just be aware we consider swap to be a local disk too | |
| 11:41:01 | sean-k-mooney | noonedeadpunk: this is a per host option by the way | |
| 11:41:12 | sean-k-mooney | so you can set it on a subset of hosts | |
| 11:41:26 | sean-k-mooney | and ideally use an aggreate to group them for scheduling | |
| 11:41:29 | noonedeadpunk | Oh, so it's effective on compute, not on api/scheduler hosts? | |
| 11:41:37 | sean-k-mooney | noonedeadpunk: correct | |
| 11:41:39 | noonedeadpunk | Ok, good to know! | |
| 11:41:49 | noonedeadpunk | thanks for the help and sorry for disturbing:) | |
| 11:42:09 | noonedeadpunk | really wasn't able to find this specific option with code words I was thinking about) | |
| 11:42:15 | sean-k-mooney | noonedeadpunk: what release are you on | |
| 11:42:16 | alex_xu | sean-k-mooney: ah, i see now. The host has enough pcpu for that instance, but the instance forbidden the hyperthread, that is why the placement reject, then fallback to VCPU. And then filter begin to filter based pcpu again, then the scheduling success | |
| 11:42:31 | sean-k-mooney | alex_xu: exactly | |
| 11:42:43 | alex_xu | finally i see that | |
| 11:42:50 | alex_xu | sean-k-mooney: thanks | |
| 11:42:53 | sean-k-mooney | its an edgecase to and edgecase but customers hit it imeditely | |