| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-18 | |||
| 16:18:02 | mnaser | then we start with https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L10158-L10166 | |
| 16:18:03 | mriedem | stephenfin: artom said i can't trust functional tests for numa stuff so... | |
| 16:18:11 | mnaser | sean-k-mooney: yeah i think i might try that next | |
| 16:18:22 | mnaser | i think the config docs aren't up to date in that case | |
| 16:18:28 | mnaser | unset/none != host-model | |
| 16:18:34 | sean-k-mooney | mnaser: that code path is ment to be equavlent but i think its actullly not | |
| 16:18:42 | stephenfin | it's not NUMA stuff | |
| 16:18:45 | mriedem | HA | |
| 16:18:46 | stephenfin | https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_ebb/682111/1/check/nova-tox-functional/ebbc041/testr_results.html.gz | |
| 16:19:04 | stephenfin | Also the reason that patch is WIP (the test doesn't belong in that file) | |
| 16:19:11 | sean-k-mooney | mnaser: we would bail out here normally https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L10152-L10156 | |
| 16:19:47 | mnaser | except if its not set, which the config docs say: If ``virt_type="kvm|qemu"``, it will default to ``host-model``, otherwise it will default to ``none``. | |
| 16:19:47 | stephenfin | but yeah, with that we see the exact issue I was seeing locally | |
| 16:20:05 | mnaser | https://github.com/openstack/nova/blob/stable/stein/nova/conf/libvirt.py#L539-L540 | |
| 16:20:16 | stephenfin | I also fixed the other spotted the thing kashyap pointed out earlier | |
| 16:20:20 | mnaser | which meant either the config docs should be updated, or the correct behaviour should be done | |
| 16:21:02 | sean-k-mooney | ya but i think on mater the new supprot for multiple custom cpu models has changed things | |
| 16:21:05 | mnaser | what it seems to be doing is actaully grabbing qemu64 (base) and then looping over all the features of baseline cpu and adding them as extra features | |
| 16:21:15 | sean-k-mooney | mnaser: it used to be the same | |
| 16:21:31 | mnaser | this is a stein deployment so maybe we just forgot to document the config change | |
| 16:22:09 | sean-k-mooney | the change i was refering to only landed on master recently but we might have acidentally changed the behavior in rocky or stine | |
| 16:22:14 | sean-k-mooney | and not upstaed the config | |
| 16:22:41 | mnaser | ok yeah no this code isnt there in master | |
| 16:22:48 | sean-k-mooney | this is what we used to do https://github.com/openstack/nova/blob/stable/stein/nova/virt/libvirt/driver.py#L3912-L3931 | |
| 16:23:20 | mnaser | i wonder if we ended up running newer than stein in some messed up wrong way :X | |
| 16:23:51 | sean-k-mooney | mnaser: you would have to be running master form last week too have that new code i think | |
| 16:23:57 | mnaser | ok then def not | |
| 16:24:18 | sean-k-mooney | mnaser: did you pull the wheels form pypi/tarballs.openstack.org by mistake? | |
| 16:24:53 | mnaser | no i just double checked and its the right stuff | |
| 16:25:51 | sean-k-mooney | oh you were allso looking at the traits code not the xml generation code | |
| 16:26:11 | mnaser | ah gotcha | |
| 16:26:29 | sean-k-mooney | the config is still the same https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L4150-L4181 | |
| 16:26:59 | openstackgerrit | melanie witt proposed openstack/nova stable/stein: Add reno about nova-api eventlet monkey-patching and rabbitmq https://review.opendev.org/662095 | |
| 16:27:16 | openstackgerrit | Merged openstack/nova stable/stein: neutron: refactor nw info cache refresh out of associate_floating_ip https://review.opendev.org/682181 | |
| 16:27:23 | openstackgerrit | Merged openstack/nova stable/stein: Trap and log errors from _update_inst_info_cache_for_disassociated_fip https://review.opendev.org/682182 | |
| 16:27:32 | openstackgerrit | Merged openstack/nova stable/stein: Find instance in another cell during floating IP re-association https://review.opendev.org/682183 | |
| 16:27:39 | openstackgerrit | Merged openstack/nova stable/stein: Log notifications if assertion in _test_live_migration_force_complete fails https://review.opendev.org/681743 | |
| 16:28:41 | mriedem | stephenfin: you mean https://review.opendev.org/#/c/682267/ right? | |
| 16:29:03 | stephenfin | yup | |
| 16:29:04 | sean-k-mooney | mnaser: those feature flags still look wrong to me | |
| 16:29:05 | mriedem | i definitely think we need some functional testing for the libvirt driver that sets cpu extra flags b/c that's 2 regressions when using that config option in the last week or so | |
| 16:29:22 | mriedem | and configuring cpu extra flags seems pretty important for red hat customers... | |
| 16:29:24 | sean-k-mooney | mriedem: it imples we are getting to the extra_flag section or libvirt added them | |
| 16:29:25 | stephenfin | Agreed. I want to flesh out that functional test of mine and use that | |
| 16:29:36 | stephenfin | Just need to figure out where to put it now | |
| 16:29:37 | sean-k-mooney | mriedem: soory that was for mnaser | |
| 16:29:46 | mnaser | sean-k-mooney: yeah it doesn't look right | |
| 16:29:54 | mnaser | but i dont know if virsh dumpxml does some magic | |
| 16:29:59 | mnaser | and doesnt actaully print out what it got from nova | |
| 16:30:00 | mriedem | stephenfin: maybe just a new simple libvirt functoinal test that isn't in the test_numa module or whatever | |
| 16:30:01 | sean-k-mooney | mnaser: was that config form the nova compute log or libvirt via virsh | |
| 16:30:04 | mnaser | no virsh | |
| 16:30:10 | mnaser | so i havent blamed nova yet | |
| 16:30:10 | sean-k-mooney | virsh does not | |
| 16:30:17 | sean-k-mooney | libvirt transforms the xml we pass in | |
| 16:30:27 | mnaser | yeah so this might be transformed | |
| 16:30:28 | sean-k-mooney | e.g. it does not actually use the xml we provide. | |
| 16:30:33 | stephenfin | yeah, I'd say so. I just need to check how much of the mocking I need to carry across | |
| 16:30:33 | mnaser | though i wonder why it wouldn't keep host-model there | |
| 16:31:20 | mnaser | hmm i remember virsh dumpxml used to have a "dumpable" version | |
| 16:31:36 | sean-k-mooney | mnaser: ya i dont know either. this is where i delegate to danpb or kashyap as i read C but dont really want to have to find where that happens | |
| 16:31:49 | sean-k-mooney | mnaser: its not a virsh issue | |
| 16:31:59 | mnaser | ok fair nuff | |
| 16:32:12 | sean-k-mooney | wehn we create teh xml and define the domain libvirt parses it and save an updated version to disk and in memory | |
| 16:32:49 | sean-k-mooney | virsh talks to libvirt over a unix socket and virsh dumpxml give you the rendered version | |
| 16:33:11 | mnaser | oooh | |
| 16:33:13 | sean-k-mooney | libvirt when it ingest the xml fills in a bunch of things like guest pci address that we dont set | |
| 16:33:15 | mnaser | wait so i can do cat on the local disk then | |
| 16:33:43 | sean-k-mooney | there is a copy saved to disk somewhere but you could also just check the nova compute log | |
| 16:33:47 | sean-k-mooney | the xml is saved in it | |
| 16:33:55 | mnaser | yeah not running $world with DEBUG on | |
| 16:33:59 | mnaser | so that'd have to be a little exercise :p | |
| 16:34:12 | mnaser | its only logged when running under debug afaik | |
| 16:34:21 | sean-k-mooney | yes it is | |
| 16:34:40 | mnaser | time to google for the millionth time "how to run an openstack instance on a specific node" | |
| 16:34:51 | sean-k-mooney | anyway i would check the qemu instance log and see what the error is | |
| 16:35:00 | openstackgerrit | Merged openstack/nova-specs master: Update spec: filtering of alloc candidates by forbidden aggregates https://review.opendev.org/675384 | |
| 16:35:15 | sean-k-mooney | cat /var/log/libvirt/qemu/instanace... | |
| 16:35:33 | mnaser | yeah but i tihnk its a layer before that because the qemu process doesnt have the flag | |
| 16:35:34 | mnaser | "-cpu IvyBridge-IBRS,ss=on,pcid=on,hypervisor=on,arat=on,tsc_adjust=on,md-clear=on,stibp=on,ssbd=on,xsaveopt=on,pdpe1gb=on" | |
| 16:35:54 | sean-k-mooney | which flag is missing? | |
| 16:36:04 | sean-k-mooney | i though the vm was crashing? | |
| 16:36:06 | mnaser | vmx (and the fact that things are manually added) | |
| 16:36:13 | mnaser | nope, no crashes at all, just literally a flag missing | |
| 16:36:23 | mnaser | virsh capabilities shows vmx | |
| 16:36:32 | mnaser | but booting instances without defining cpu_model does not | |
| 16:36:40 | sean-k-mooney | right but did you turn nested virt on in the host kernel | |
| 16:36:42 | mnaser | does not boot it with vmx then | |
| 16:36:46 | mnaser | yep | |
| 16:37:05 | sean-k-mooney | ok i normally use host-passthough with nested virt so the flag is there | |
| 16:37:08 | mnaser | `virsh capabilities` wouldn't report vmx there if it wasnt on either afaik | |
| 16:37:11 | mnaser | yeah, im trying to avoid that one | |
| 16:37:17 | sean-k-mooney | the model your using might just not included it | |
| 16:37:21 | sean-k-mooney | you could jsut add it | |
| 16:37:26 | mnaser | cat /sys/module/kvm_intel/parameters/nested => Y | |
| 16:37:29 | sean-k-mooney | with the config | |
| 16:37:53 | mnaser | sean-k-mooney: it actually does though -- http://paste.openstack.org/show/777431/ | |
| 16:37:54 | mnaser | thats the thing | |
| 16:38:09 | mnaser | i could add it manually but im trying to see if its something we can improve upstream :> | |
| 16:38:10 | sean-k-mooney | the host does | |