Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-09-18
16:04:30 artom Makes it kinda hard to RCA :)
16:04:31 sean-k-mooney mriedem: its from https://review.opendev.org/#/c/674072/ we have a test backport for the customer to try to see if that actully works for them
16:05:15 sean-k-mooney i was wrigitn the spec for this but im fixing the config drive test currently
16:06:55 sean-k-mooney mriedem: we need to backport https://review.opendev.org/#/c/624444/ first and artom noticed that require did not have test coverage
16:07:23 artom Not in func tests, anyways
16:07:40 sean-k-mooney right there are unit tests
16:07:42 efried dansmith: The isolated aggs spec update lgtm https://review.opendev.org/#/c/675384/ want me to fast approve?
16:07:50 mriedem unit tests are generally insufficient for any of these kinds of features
16:08:02 artom mriedem, the same could be said for func tests ;)
16:08:02 mriedem at this point in nova, unit tests are generally insufficient for just about anything
16:08:18 mriedem artom: for low level hardware stuff sure
16:08:21 mriedem i wouldn't see that in general
16:08:34 artom mriedem, I mean low level hardware stuff, like PCI or NUMA
16:09:05 mriedem so the broken feature that landed in queens is just getting rolled out to the customer that wanted it now?
16:09:19 mriedem 18 months later...right on time
16:09:26 sean-k-mooney yes
16:09:32 mriedem and queens is your current LTS?
16:09:51 artom Long live the queen, indeed
16:09:59 sean-k-mooney newton will be end of line in decmeber and queens is our next LTS release downstream
16:10:17 sean-k-mooney so people are upgrading
16:11:13 mnaser this maybe more of a libvirt question, but i'll ask.. `cpu_mode` defaults to `host-model` in nova when using kvm, but the libvirt definition seems to be `<cpu mode='custom' match='exact' check='full'> ... </cpu>` (this is what i gather from virsh dumpxml)
16:11:29 mriedem mnaser: meet kashyap
16:11:38 mriedem <3
16:11:48 mnaser reading the nova code, it seems that nova is doing the right thing(tm)
16:12:16 sean-k-mooney mnaser: libvirt is also ment to default to host model if we say nothing for what its worth
16:12:33 mnaser right, im just checkin an issue where nested virt is available, virsh capabilities shows vmx
16:12:36 mnaser but vms dont spawn with it
16:12:39 sean-k-mooney but i dont think it always did which is why nova does.
16:12:57 efried I'm going to be taking most of the rest of the week off (family in town). Will check in periodically. Email if you need to reach me. o/
16:13:01 mnaser so i can add it to the list of extra flags (or try and solve it upstream :])
16:13:19 sean-k-mooney do you have a warning about svm not abliable in teh qemu instance log
16:13:31 mriedem mnaser: there are a bunch of recently updated docs here as well https://docs.openstack.org/nova/stein/admin/configuration/hypervisor-kvm.html#specify-the-cpu-model-of-kvm-guests
16:13:33 mriedem if that helps
16:13:52 mnaser ya i went over those, i can totally do passthrough or the other options, but right now my combination should just work ideally
16:14:10 mnaser man it would be nice if we split otu some of the libvirt driver code
16:14:13 mnaser :p
16:14:13 mnaser so github search indexes it
16:14:16 sean-k-mooney qemu does not always detect the host model correctly and somethime expose amd version of vmx (svm) if you use host model
16:14:30 sean-k-mooney and nested virt is enabled
16:14:50 mnaser yeah but the libvirt definition i see here clearly says use a custom set of stuff forbidding callback too
16:15:13 mnaser with a bunch of "requires" too
16:15:30 sean-k-mooney can you paste the full xml somewhere
16:15:32 mnaser i dont know if virsh dumpxml shows the thing that virsh determines or the thing that it was fed
16:15:48 sean-k-mooney i dont think we should be listing a lot of features in the xml
16:15:52 mnaser http://paste.openstack.org/show/777429/
16:16:35 sean-k-mooney do you have the extra cpu flags config set out of interest
16:16:48 mnaser https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L10171-L10174
16:16:50 mnaser nope none are set
16:16:51 mnaser but that..
16:16:52 mriedem stephenfin: https://review.opendev.org/#/c/681938/ now depends on your fix so once that passes i'll review your fix
16:17:05 mnaser that doesnt look right
16:17:29 stephenfin mriedem: You saw https://review.opendev.org/#/c/682111/, I assume?
16:17:42 mnaser https://github.com/openstack/nova/blob/master/nova/virt/libvirt/utils.py#L533-L539
16:17:42 mriedem i didn't dig into it
16:17:46 mnaser so tohat gives it back qemu64
16:17:57 sean-k-mooney mnaser: can you set cpu-mode="host-model" explcitly?
16:18:02 mnaser then we start with https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L10158-L10166
16:18:03 mriedem stephenfin: artom said i can't trust functional tests for numa stuff so...
16:18:11 mnaser sean-k-mooney: yeah i think i might try that next
16:18:22 mnaser i think the config docs aren't up to date in that case
16:18:28 mnaser unset/none != host-model
16:18:34 sean-k-mooney mnaser: that code path is ment to be equavlent but i think its actullly not
16:18:42 stephenfin it's not NUMA stuff
16:18:45 mriedem HA
16:18:46 stephenfin https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_ebb/682111/1/check/nova-tox-functional/ebbc041/testr_results.html.gz
16:19:04 stephenfin Also the reason that patch is WIP (the test doesn't belong in that file)
16:19:11 sean-k-mooney mnaser: we would bail out here normally https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L10152-L10156
16:19:47 stephenfin but yeah, with that we see the exact issue I was seeing locally
16:19:47 mnaser except if its not set, which the config docs say: If ``virt_type="kvm|qemu"``, it will default to ``host-model``, otherwise it will default to ``none``.
16:20:05 mnaser https://github.com/openstack/nova/blob/stable/stein/nova/conf/libvirt.py#L539-L540
16:20:16 stephenfin I also fixed the other spotted the thing kashyap pointed out earlier
16:20:20 mnaser which meant either the config docs should be updated, or the correct behaviour should be done
16:21:02 sean-k-mooney ya but i think on mater the new supprot for multiple custom cpu models has changed things
16:21:05 mnaser what it seems to be doing is actaully grabbing qemu64 (base) and then looping over all the features of baseline cpu and adding them as extra features
16:21:15 sean-k-mooney mnaser: it used to be the same
16:21:31 mnaser this is a stein deployment so maybe we just forgot to document the config change
16:22:09 sean-k-mooney the change i was refering to only landed on master recently but we might have acidentally changed the behavior in rocky or stine
16:22:14 sean-k-mooney and not upstaed the config
16:22:41 mnaser ok yeah no this code isnt there in master
16:22:48 sean-k-mooney this is what we used to do https://github.com/openstack/nova/blob/stable/stein/nova/virt/libvirt/driver.py#L3912-L3931
16:23:20 mnaser i wonder if we ended up running newer than stein in some messed up wrong way :X
16:23:51 sean-k-mooney mnaser: you would have to be running master form last week too have that new code i think
16:23:57 mnaser ok then def not
16:24:18 sean-k-mooney mnaser: did you pull the wheels form pypi/tarballs.openstack.org by mistake?
16:24:53 mnaser no i just double checked and its the right stuff
16:25:51 sean-k-mooney oh you were allso looking at the traits code not the xml generation code
16:26:11 mnaser ah gotcha
16:26:29 sean-k-mooney the config is still the same https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L4150-L4181
16:26:59 openstackgerrit melanie witt proposed openstack/nova stable/stein: Add reno about nova-api eventlet monkey-patching and rabbitmq https://review.opendev.org/662095
16:27:16 openstackgerrit Merged openstack/nova stable/stein: neutron: refactor nw info cache refresh out of associate_floating_ip https://review.opendev.org/682181
16:27:23 openstackgerrit Merged openstack/nova stable/stein: Trap and log errors from _update_inst_info_cache_for_disassociated_fip https://review.opendev.org/682182
16:27:32 openstackgerrit Merged openstack/nova stable/stein: Find instance in another cell during floating IP re-association https://review.opendev.org/682183
16:27:39 openstackgerrit Merged openstack/nova stable/stein: Log notifications if assertion in _test_live_migration_force_complete fails https://review.opendev.org/681743
16:28:41 mriedem stephenfin: you mean https://review.opendev.org/#/c/682267/ right?
16:29:03 stephenfin yup
16:29:04 sean-k-mooney mnaser: those feature flags still look wrong to me
16:29:05 mriedem i definitely think we need some functional testing for the libvirt driver that sets cpu extra flags b/c that's 2 regressions when using that config option in the last week or so
16:29:22 mriedem and configuring cpu extra flags seems pretty important for red hat customers...
16:29:24 sean-k-mooney mriedem: it imples we are getting to the extra_flag section or libvirt added them
16:29:25 stephenfin Agreed. I want to flesh out that functional test of mine and use that

Earlier   Later