| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-05-09 | |||
| 14:33:25 | efried | tssurya: Okay. Because we talked about enabling some placement-isms to help with this in Train itself, yah? | |
| 14:33:43 | tssurya | efried: yep, that would be a spec | |
| 14:33:53 | tssurya | I'll get onto that too soon-ish | |
| 14:33:54 | mriedem | in_tree for strict affinity but that's not backportable | |
| 14:33:58 | mriedem | it's also racy | |
| 14:34:19 | ganso | mriedem: so, this problem is likely caused by a bug that got fixed, correct? or was the periodic task added in ocata? | |
| 14:34:22 | efried | cool, just making sure I had the big picture | |
| 14:34:26 | tssurya | mriedem: no no ^ that fix is just making limits None if affinity/anti is requested | |
| 14:34:41 | tssurya | like we did earlier on for force_hosts/nodes | |
| 14:34:49 | tssurya | only that needs to be backported | |
| 14:34:49 | mriedem | ganso: the periodic was always around, | |
| 14:34:58 | mriedem | the stats thing i fixed was a regression in ocata so shouldn't affect you on mitaka | |
| 14:36:32 | mriedem | ganso: for that num_instances thing, you'd likely need to add some debug logging to that update_available_resource flow to see what it's doing, | |
| 14:36:58 | mriedem | but it should 0 out num_instances in the stats at the start of the run, and then add instances to the stats as it processes instances for that host/node | |
| 14:37:31 | mriedem | ganso: does https://developer.openstack.org/api-ref/compute/#list-hypervisor-servers also show 4 servers on that hypervisor? | |
| 14:38:16 | efried | tssurya: We have in_tree in Stein though... | |
| 14:38:51 | efried | ...so you could make a backportable patch that adds in_tree for positive affinity and removes the limit for anti-affinity. | |
| 14:39:01 | efried | ...and then if you want to backport further, use this one. | |
| 14:39:03 | ganso | mriedem: hmmm is that equivalent of "openstack hypervisor list" ? | |
| 14:39:09 | efried | tssurya: is that the plan? | |
| 14:39:35 | mriedem | efried: tssurya: it should be a 2 part series at least, | |
| 14:39:37 | ganso | mriedem: I have confirmed that the database has the value "2", so the database is wrong | |
| 14:39:41 | tssurya | efried: oh didn't realise in_tree was there in Stein since it was used only in train | |
| 14:39:52 | tssurya | but what you said makes sense | |
| 14:39:56 | mriedem | tssurya: but the bug goes back further than that | |
| 14:39:57 | efried | tssurya: I *think* it's in stein, sec... | |
| 14:40:06 | mriedem | in_tree usage in nova is new in train | |
| 14:40:08 | mriedem | from tetsuro | |
| 14:40:35 | tssurya | mriedem: yup I think the bug is from rocky, and it would be good for us to have it in rocky | |
| 14:41:02 | mriedem | it probably goes back further than that | |
| 14:41:12 | mriedem | my guess is pike | |
| 14:41:39 | efried | tssurya: Confirmed, in_tree is in Stein. | |
| 14:41:43 | efried | So you could do this in three parts. | |
| 14:41:51 | mriedem | efried: in_tree is in stein in placement right? | |
| 14:41:57 | efried | yes --^ | |
| 14:42:00 | mriedem | but the nova object usage of it is not | |
| 14:42:00 | efried | 1.31 | |
| 14:42:07 | efried | does that matter? | |
| 14:42:14 | mriedem | if you're using versioned objects it does yes | |
| 14:42:24 | efried | versioned objects | |
| 14:42:34 | efried | you mean the RequestSpec? | |
| 14:42:39 | mriedem | RequestGroup | |
| 14:42:40 | mriedem | but yes | |
| 14:42:52 | mriedem | https://review.opendev.org/#/q/topic:bug/1777591+(status:open+OR+status:merged) | |
| 14:43:29 | mriedem | it's really 2 changes, one to backport which just disables the limit if the server is in a group | |
| 14:43:34 | efried | groan, okay. | |
| 14:43:43 | mriedem | and another to be smarter about in_tree if it's in a strict affinity policy | |
| 14:43:49 | mriedem | but noting that ^ is racy | |
| 14:44:34 | tssurya | mriedem, efired: ok so we stick to what we discussed in the ptg right ? | |
| 14:44:49 | tssurya | the new in_tree affinity thing stays in train then | |
| 14:45:12 | tssurya | mriedem: yea you might be right, I can trace it back to at least queens | |
| 14:45:16 | mriedem | max_placement_results was new in queens so i guess it goes to queens | |
| 14:45:33 | mriedem | ganso: no not hypervisor list, sec | |
| 14:45:40 | efried | tssurya: That's not my favorite misspelling of my handle. | |
| 14:46:02 | mriedem | she's channeling her inner trump | |
| 14:46:04 | tssurya | efried: yikes, I didn't even realised I missplled it :D | |
| 14:46:21 | mriedem | tssurya is a big fan, some are saying the biggest fan | |
| 14:46:27 | mriedem | folks | |
| 14:46:37 | efried | anyway, +2 on this patch, thanks for explaining | |
| 14:46:37 | tssurya | mriedem: :P | |
| 14:47:17 | tssurya | thanks efried, mriedem | |
| 14:47:20 | mriedem | ganso: more like this https://docs.openstack.org/python-novaclient/latest/cli/nova.html#nova-hypervisor-servers | |
| 14:47:40 | mriedem | ganso: i'm wondering if ^ will show 4 for that node while running_vms shows 2 | |
| 14:47:46 | ganso | mriedem: oh I see! | |
| 14:47:52 | ganso | mriedem: will run it and report back! | |
| 14:47:53 | mriedem | if ^ shows 2 servers as well, then you have some issue | |
| 14:48:07 | mriedem | like, the db might say 2 of those guests you think are on that node actually aren't | |
| 14:48:12 | mriedem | e.g. failed live migration or something | |
| 14:50:08 | ganso | mriedem: it does list 4 VMs! =) | |
| 14:51:00 | mriedem | ok so yeah the stats stuff is busted somehow, but you'd likely need to add logging statements to the code to debug it | |
| 14:51:55 | ganso | mriedem: ok, so... I am curious on how I could reset the value, and double check if it is being recalculated incorrectly, or if it is not being recalculated at all | |
| 14:52:42 | ganso | mriedem: like, if I restart the service (nova-cpu I'd assume), it would reset to 0 and recalculate? Does it help if I set the value to 0 in the database and then restart the service? | |
| 14:53:13 | mriedem | the periodic should set it every time | |
| 14:53:26 | mriedem | so you could set it to 0 in the db if you want and wait for the periodic to run, or restart the service | |
| 14:54:21 | ganso | mriedem: ok, will try that, thank you! | |
| 14:54:43 | openstackgerrit | Merged openstack/nova stable/rocky: Use migration_status during volume migrating and retyping https://review.opendev.org/657577 | |
| 14:59:05 | mriedem | tssurya: comments in that change, but it's not just the group hint | |
| 14:59:14 | ganso | mriedem: sorry, one more question. I am trying to track the name of the periodic task, I haven't seen periodic_task decorators or where they are registered in resource_tracker.py. But it seems the periodic task is either "update_available_resource" or another method that invokes this one. Could you please confirm? | |
| 14:59:15 | mriedem | we also have same_host/different_host hints/filters | |
| 14:59:32 | mriedem | ganso: it's the ComputeManager.update_available_resource method | |
| 14:59:43 | mriedem | called on start of the compute service and in a periodic | |
| 14:59:44 | ganso | mriedem: oh it starts in the ComputeManager, thank you! | |
| 14:59:57 | tssurya | mriedem: oh hmm looking | |
| 15:01:18 | mriedem | https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#samehostfilter | |
| 15:01:41 | mnaser | so it looks like the kernel that enables nested by default (4.19) also added a gate that blocks live migrations if nested virt is enabled p | |
| 15:01:43 | jangutter | Did the ptg etherpad finally die? | |
| 15:02:10 | mnaser | ERROR nova.virt.libvirt.driver [-] [instance: 738eb188-1545-4126-b05b-54d384e55f73] Live Migration failure: internal error: unable to execute QEMU command 'migrate': Nested VMX virtualization does not support live migration yet: libvirtError: internal error: unable to execute QEMU command 'migrate': Nested VMX virtualization does not support live migration yet | |
| 15:02:29 | mnaser | so by default vmx is enabled, so every instance is not live migratable... | |
| 15:02:32 | tssurya | mriedem: I have never used that filter before, but since I am fixing it for the other two filters, I will update the patch for this one as well | |
| 15:02:54 | tssurya | and https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#differenthostfilter like you have said | |
| 15:03:21 | mriedem | i really wish we could functionally test this code to recreate the bug but i don't think there is an easy way to do that | |
| 15:03:35 | tssurya | jangutter: I have been trying to access it too :( not sure if someone has a backup | |
| 15:04:38 | tssurya | mriedem: you mean the bug I am working on ? | |
| 15:11:14 | mriedem | tssurya: yes | |
| 15:11:36 | mriedem | but there isn't really a good way to do that with functional tests that don't involve stubbing placement responses, which defeats the purpose of a functional test | |
| 15:14:45 | mnaser | I’m trying to figure out how I can make both nested virt available as an opt in | |
| 15:15:09 | mnaser | But also without making complicated host aggregates and flavour explosion | |
| 15:18:29 | mriedem | required trait on vmx? | |
| 15:20:44 | mriedem | libvirt computes with vmx enabled would report the HW_CPU_X86_VMX trait right? | |
| 15:20:59 | mriedem | and you could have flavors and/or images which require that trait so the user can opt into nested virt that way | |