| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-05-09 | |||
| 14:43:49 | mriedem | but noting that ^ is racy | |
| 14:44:34 | tssurya | mriedem, efired: ok so we stick to what we discussed in the ptg right ? | |
| 14:44:49 | tssurya | the new in_tree affinity thing stays in train then | |
| 14:45:12 | tssurya | mriedem: yea you might be right, I can trace it back to at least queens | |
| 14:45:16 | mriedem | max_placement_results was new in queens so i guess it goes to queens | |
| 14:45:33 | mriedem | ganso: no not hypervisor list, sec | |
| 14:45:40 | efried | tssurya: That's not my favorite misspelling of my handle. | |
| 14:46:02 | mriedem | she's channeling her inner trump | |
| 14:46:04 | tssurya | efried: yikes, I didn't even realised I missplled it :D | |
| 14:46:21 | mriedem | tssurya is a big fan, some are saying the biggest fan | |
| 14:46:27 | mriedem | folks | |
| 14:46:37 | efried | anyway, +2 on this patch, thanks for explaining | |
| 14:46:37 | tssurya | mriedem: :P | |
| 14:47:17 | tssurya | thanks efried, mriedem | |
| 14:47:20 | mriedem | ganso: more like this https://docs.openstack.org/python-novaclient/latest/cli/nova.html#nova-hypervisor-servers | |
| 14:47:40 | mriedem | ganso: i'm wondering if ^ will show 4 for that node while running_vms shows 2 | |
| 14:47:46 | ganso | mriedem: oh I see! | |
| 14:47:52 | ganso | mriedem: will run it and report back! | |
| 14:47:53 | mriedem | if ^ shows 2 servers as well, then you have some issue | |
| 14:48:07 | mriedem | like, the db might say 2 of those guests you think are on that node actually aren't | |
| 14:48:12 | mriedem | e.g. failed live migration or something | |
| 14:50:08 | ganso | mriedem: it does list 4 VMs! =) | |
| 14:51:00 | mriedem | ok so yeah the stats stuff is busted somehow, but you'd likely need to add logging statements to the code to debug it | |
| 14:51:55 | ganso | mriedem: ok, so... I am curious on how I could reset the value, and double check if it is being recalculated incorrectly, or if it is not being recalculated at all | |
| 14:52:42 | ganso | mriedem: like, if I restart the service (nova-cpu I'd assume), it would reset to 0 and recalculate? Does it help if I set the value to 0 in the database and then restart the service? | |
| 14:53:13 | mriedem | the periodic should set it every time | |
| 14:53:26 | mriedem | so you could set it to 0 in the db if you want and wait for the periodic to run, or restart the service | |
| 14:54:21 | ganso | mriedem: ok, will try that, thank you! | |
| 14:54:43 | openstackgerrit | Merged openstack/nova stable/rocky: Use migration_status during volume migrating and retyping https://review.opendev.org/657577 | |
| 14:59:05 | mriedem | tssurya: comments in that change, but it's not just the group hint | |
| 14:59:14 | ganso | mriedem: sorry, one more question. I am trying to track the name of the periodic task, I haven't seen periodic_task decorators or where they are registered in resource_tracker.py. But it seems the periodic task is either "update_available_resource" or another method that invokes this one. Could you please confirm? | |
| 14:59:15 | mriedem | we also have same_host/different_host hints/filters | |
| 14:59:32 | mriedem | ganso: it's the ComputeManager.update_available_resource method | |
| 14:59:43 | mriedem | called on start of the compute service and in a periodic | |
| 14:59:44 | ganso | mriedem: oh it starts in the ComputeManager, thank you! | |
| 14:59:57 | tssurya | mriedem: oh hmm looking | |
| 15:01:18 | mriedem | https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#samehostfilter | |
| 15:01:41 | mnaser | so it looks like the kernel that enables nested by default (4.19) also added a gate that blocks live migrations if nested virt is enabled p | |
| 15:01:43 | jangutter | Did the ptg etherpad finally die? | |
| 15:02:10 | mnaser | ERROR nova.virt.libvirt.driver [-] [instance: 738eb188-1545-4126-b05b-54d384e55f73] Live Migration failure: internal error: unable to execute QEMU command 'migrate': Nested VMX virtualization does not support live migration yet: libvirtError: internal error: unable to execute QEMU command 'migrate': Nested VMX virtualization does not support live migration yet | |
| 15:02:29 | mnaser | so by default vmx is enabled, so every instance is not live migratable... | |
| 15:02:32 | tssurya | mriedem: I have never used that filter before, but since I am fixing it for the other two filters, I will update the patch for this one as well | |
| 15:02:54 | tssurya | and https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#differenthostfilter like you have said | |
| 15:03:21 | mriedem | i really wish we could functionally test this code to recreate the bug but i don't think there is an easy way to do that | |
| 15:03:35 | tssurya | jangutter: I have been trying to access it too :( not sure if someone has a backup | |
| 15:04:38 | tssurya | mriedem: you mean the bug I am working on ? | |
| 15:11:14 | mriedem | tssurya: yes | |
| 15:11:36 | mriedem | but there isn't really a good way to do that with functional tests that don't involve stubbing placement responses, which defeats the purpose of a functional test | |
| 15:14:45 | mnaser | I’m trying to figure out how I can make both nested virt available as an opt in | |
| 15:15:09 | mnaser | But also without making complicated host aggregates and flavour explosion | |
| 15:18:29 | mriedem | required trait on vmx? | |
| 15:20:44 | mriedem | libvirt computes with vmx enabled would report the HW_CPU_X86_VMX trait right? | |
| 15:20:59 | mriedem | and you could have flavors and/or images which require that trait so the user can opt into nested virt that way | |
| 15:30:50 | mriedem | live migration is going to be hairy though right? because you could have instances land on those hosts that don't want nested virt and then you can't live migrate them? | |
| 15:31:13 | mriedem | not sure if "Live Migration failure: internal error: unable to execute QEMU command 'migrate': Nested VMX virtualization does not support live migration yet: libvirtError: internal error: unable to execute QEMU command 'migrate': Nested VMX virtualization does not support live migration yet" is a problem if vmx is on the source, dest or both | |
| 15:31:28 | mnaser | right, but then the idea is to make it opt-in and have a note that says "you won't be live migrated if you enable nested virtualize" | |
| 15:31:32 | mnaser | virt* | |
| 15:31:38 | mriedem | my point is, | |
| 15:31:45 | mriedem | the people that aren't opting in could still land on those hosts | |
| 15:31:52 | mriedem | i.e. this is the exclusion problem in the scheduler | |
| 15:31:59 | mriedem | for which forbidden aggregates are going to be used | |
| 15:32:16 | mriedem | https://specs.openstack.org/openstack/nova-specs/specs/train/approved/placement-req-filter-forbidden-aggregates.html | |
| 15:33:00 | mriedem | i'm not sure how you get around this without aggregates | |
| 15:34:59 | mnaser | hmm, so its not necessarily possible to "request" a custom cpu flag for a specific instance | |
| 15:36:55 | openstackgerrit | Rodrigo Barbieri proposed openstack/nova stable/queens: [DEBUG] Add functional confirm_migration_error test https://review.opendev.org/658136 | |
| 15:37:12 | openstackgerrit | Dan Smith proposed openstack/nova master: Update the contributor doc for macos https://review.opendev.org/658137 | |
| 15:37:21 | dansmith | trivial contributor doc update ^ | |
| 15:43:58 | bnemec | efried: Would you be able to add the nova-specific freeze dates to https://releases.openstack.org/train/schedule.html ? | |
| 15:44:03 | bnemec | The keystone patch is a good example: https://review.opendev.org/#/c/653544/ | |
| 15:44:22 | bnemec | It came up last week that people were surprised by some Nova freeze dates so it would be good to get them on the common schedule. | |
| 15:44:38 | efried | bnemec: can do. I think melwitt conveniently created the schedule a month or two ago in the wiki, please hold. | |
| 15:44:54 | efried | bnemec: https://wiki.openstack.org/wiki/Nova/Train_Release_Schedule | |
| 15:44:55 | mriedem | mnaser: the user would request via the flavor/image which would have the required cpu flag as a trait | |
| 15:45:31 | bnemec | efried: Cool, thanks. | |
| 15:46:00 | bnemec | I guess the main one would be spec freeze then. | |
| 15:46:42 | mnaser | mriedem: would it be appended to the 'existing' cpu flags or it would have to match a hypervisor that has those cpu flags (i.e I would have a bunch of nested-virt enabled computes, and those that aren't) | |
| 15:48:07 | mriedem | mnaser: my understanding from https://specs.openstack.org/openstack/nova-specs/specs/rocky/implemented/report-cpu-features-as-traits.html is that the compute reports the cpu features as traits, and you can configure each compute with additional flags using https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.cpu_model_extra_flags | |
| 15:48:48 | mriedem | i think cpu_model_extra_flags get reported as traits as well, but i would have to dig into the code to confirm | |
| 15:49:32 | mnaser | but if I add `cpu_model_extra_flags` that would then make every instance get nested virt | |
| 15:49:39 | mnaser | (on that host) | |
| 15:49:49 | mriedem | oh i see, you want per-instance | |
| 15:50:01 | mnaser | yeah, that way I don't have to maintain a whole bunch of host aggrgates | |
| 15:50:06 | mriedem | well, geez, that sounds like starlingx craziness bro! | |
| 15:50:28 | mnaser | is that something they do / looking to do ? | |
| 15:50:36 | mriedem | i want to say yes | |
| 15:51:19 | mriedem | i thought somewhat related to https://review.opendev.org/#/c/637834/ but not really what you want | |
| 15:51:29 | mnaser | it looks like the workflow of google is per-image based -- https://cloud.google.com/compute/docs/instances/enable-nested-virtualization-vm-instances | |
| 15:53:08 | mriedem | yar, i know we have things like hw_vif_model, hw_pointer_model, hw_scsi_model but no hw_cpu_model | |
| 15:54:30 | mnaser | yeah I guess the weird thing is `hw_cpu_model` actually ties into physical infra instead of virtual only | |
| 15:54:32 | mriedem | not sure where vcpu_model comes from on the instance | |
| 15:54:56 | mnaser | i.e. `hw_cpu_model` depends on the physical CPU having a certain capability so it might not always work | |
| 15:55:10 | mnaser | whereas the rest will probably always work as they are virtualized things | |
| 15:55:36 | mriedem | https://review.opendev.org/#/c/148138/ | |
| 15:57:40 | mriedem | allowing the user to specify the cpu features they want sounds like something kashyap might be aware of if someone is working on it | |
| 15:58:51 | mnaser | because right now I think the assumption is: if a host advertises support for a specific trait, it will always boot vms with that cpu flag | |
| 15:59:33 | mriedem | mnaser: ah here you go, starlingx did have a spec for it at one point https://review.opendev.org/#/c/168982/ (liberty) | |
| 15:59:47 | mriedem | thank my handy dandy starlingx diffs spreadsheet | |
| 16:00:02 | mnaser | where as I'm trying to have it be: if a host advertises support for a specific trait, it can be an opt-in one, so all vms boot without it, unless it's requested explicitly | |
| 16:01:42 | mnaser | yeah.. this seems to pretty much cover the majority of the use case, but huge scope | |
| 16:01:54 | mriedem | last week starlingx claimed to be down to 11 critical patches outside of nova and when i asked about that they said their were all for numa live migration, so if they have support for the user specifying cpu model in the flavor/image, i'm not sure what they are doing now | |