Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-09-18
15:58:25 openstack bug 1835822 in OpenStack Compute (nova) "vms loose acess to config drive with CONF.force_config_drive=True after hard reboot" [Medium,In progress] https://launchpad.net/bugs/1835822 - Assigned to Matt Riedemann (mriedem)
15:58:36 sean-k-mooney rebase to kick it down the queue
15:58:39 kashyap mriedem: Thanks for updating
16:00:19 openstackgerrit Merged openstack/nova master: Remove an unused file and a related description https://review.opendev.org/681955
16:01:59 openstackgerrit Shilpa Devharakar proposed openstack/nova-specs master: Update spec: filtering of alloc candidates by forbidden aggregates https://review.opendev.org/675384
16:02:09 openstackgerrit Artom Lifshitz proposed openstack/nova master: Add func test for 'required' PCI NUMA policy https://review.opendev.org/682941
16:02:45 artom sean-k-mooney, stephenfin ^^ Dunno if it's that useful given that we test the 'legacy' policy which essentially does the same, but since I noticed it was missing while doing downstream reviews...
16:03:24 sean-k-mooney the legacy policy works differently if you have no device numa info on the host
16:03:44 mriedem artom: is that from you investigating that numa live migration testing thread in the ML?
16:04:06 artom mriedem, nope, from reviewing downstream backports
16:04:19 artom mriedem, he still hasn't posted logs from that ML post, btw
16:04:30 artom Makes it kinda hard to RCA :)
16:04:31 sean-k-mooney mriedem: its from https://review.opendev.org/#/c/674072/ we have a test backport for the customer to try to see if that actully works for them
16:05:15 sean-k-mooney i was wrigitn the spec for this but im fixing the config drive test currently
16:06:55 sean-k-mooney mriedem: we need to backport https://review.opendev.org/#/c/624444/ first and artom noticed that require did not have test coverage
16:07:23 artom Not in func tests, anyways
16:07:40 sean-k-mooney right there are unit tests
16:07:42 efried dansmith: The isolated aggs spec update lgtm https://review.opendev.org/#/c/675384/ want me to fast approve?
16:07:50 mriedem unit tests are generally insufficient for any of these kinds of features
16:08:02 artom mriedem, the same could be said for func tests ;)
16:08:02 mriedem at this point in nova, unit tests are generally insufficient for just about anything
16:08:18 mriedem artom: for low level hardware stuff sure
16:08:21 mriedem i wouldn't see that in general
16:08:34 artom mriedem, I mean low level hardware stuff, like PCI or NUMA
16:09:05 mriedem so the broken feature that landed in queens is just getting rolled out to the customer that wanted it now?
16:09:19 mriedem 18 months later...right on time
16:09:26 sean-k-mooney yes
16:09:32 mriedem and queens is your current LTS?
16:09:51 artom Long live the queen, indeed
16:09:59 sean-k-mooney newton will be end of line in decmeber and queens is our next LTS release downstream
16:10:17 sean-k-mooney so people are upgrading
16:11:13 mnaser this maybe more of a libvirt question, but i'll ask.. `cpu_mode` defaults to `host-model` in nova when using kvm, but the libvirt definition seems to be `<cpu mode='custom' match='exact' check='full'> ... </cpu>` (this is what i gather from virsh dumpxml)
16:11:29 mriedem mnaser: meet kashyap
16:11:38 mriedem <3
16:11:48 mnaser reading the nova code, it seems that nova is doing the right thing(tm)
16:12:16 sean-k-mooney mnaser: libvirt is also ment to default to host model if we say nothing for what its worth
16:12:33 mnaser right, im just checkin an issue where nested virt is available, virsh capabilities shows vmx
16:12:36 mnaser but vms dont spawn with it
16:12:39 sean-k-mooney but i dont think it always did which is why nova does.
16:12:57 efried I'm going to be taking most of the rest of the week off (family in town). Will check in periodically. Email if you need to reach me. o/
16:13:01 mnaser so i can add it to the list of extra flags (or try and solve it upstream :])
16:13:19 sean-k-mooney do you have a warning about svm not abliable in teh qemu instance log
16:13:31 mriedem mnaser: there are a bunch of recently updated docs here as well https://docs.openstack.org/nova/stein/admin/configuration/hypervisor-kvm.html#specify-the-cpu-model-of-kvm-guests
16:13:33 mriedem if that helps
16:13:52 mnaser ya i went over those, i can totally do passthrough or the other options, but right now my combination should just work ideally
16:14:10 mnaser man it would be nice if we split otu some of the libvirt driver code
16:14:13 mnaser :p
16:14:13 mnaser so github search indexes it
16:14:16 sean-k-mooney qemu does not always detect the host model correctly and somethime expose amd version of vmx (svm) if you use host model
16:14:30 sean-k-mooney and nested virt is enabled
16:14:50 mnaser yeah but the libvirt definition i see here clearly says use a custom set of stuff forbidding callback too
16:15:13 mnaser with a bunch of "requires" too
16:15:30 sean-k-mooney can you paste the full xml somewhere
16:15:32 mnaser i dont know if virsh dumpxml shows the thing that virsh determines or the thing that it was fed
16:15:48 sean-k-mooney i dont think we should be listing a lot of features in the xml
16:15:52 mnaser http://paste.openstack.org/show/777429/
16:16:35 sean-k-mooney do you have the extra cpu flags config set out of interest
16:16:48 mnaser https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L10171-L10174
16:16:50 mnaser nope none are set
16:16:51 mnaser but that..
16:16:52 mriedem stephenfin: https://review.opendev.org/#/c/681938/ now depends on your fix so once that passes i'll review your fix
16:17:05 mnaser that doesnt look right
16:17:29 stephenfin mriedem: You saw https://review.opendev.org/#/c/682111/, I assume?
16:17:42 mnaser https://github.com/openstack/nova/blob/master/nova/virt/libvirt/utils.py#L533-L539
16:17:42 mriedem i didn't dig into it
16:17:46 mnaser so tohat gives it back qemu64
16:17:57 sean-k-mooney mnaser: can you set cpu-mode="host-model" explcitly?
16:18:02 mnaser then we start with https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L10158-L10166
16:18:03 mriedem stephenfin: artom said i can't trust functional tests for numa stuff so...
16:18:11 mnaser sean-k-mooney: yeah i think i might try that next
16:18:22 mnaser i think the config docs aren't up to date in that case
16:18:28 mnaser unset/none != host-model
16:18:34 sean-k-mooney mnaser: that code path is ment to be equavlent but i think its actullly not
16:18:42 stephenfin it's not NUMA stuff
16:18:45 mriedem HA
16:18:46 stephenfin https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_ebb/682111/1/check/nova-tox-functional/ebbc041/testr_results.html.gz
16:19:04 stephenfin Also the reason that patch is WIP (the test doesn't belong in that file)
16:19:11 sean-k-mooney mnaser: we would bail out here normally https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L10152-L10156
16:19:47 stephenfin but yeah, with that we see the exact issue I was seeing locally
16:19:47 mnaser except if its not set, which the config docs say: If ``virt_type="kvm|qemu"``, it will default to ``host-model``, otherwise it will default to ``none``.
16:20:05 mnaser https://github.com/openstack/nova/blob/stable/stein/nova/conf/libvirt.py#L539-L540
16:20:16 stephenfin I also fixed the other spotted the thing kashyap pointed out earlier
16:20:20 mnaser which meant either the config docs should be updated, or the correct behaviour should be done
16:21:02 sean-k-mooney ya but i think on mater the new supprot for multiple custom cpu models has changed things
16:21:05 mnaser what it seems to be doing is actaully grabbing qemu64 (base) and then looping over all the features of baseline cpu and adding them as extra features
16:21:15 sean-k-mooney mnaser: it used to be the same
16:21:31 mnaser this is a stein deployment so maybe we just forgot to document the config change
16:22:09 sean-k-mooney the change i was refering to only landed on master recently but we might have acidentally changed the behavior in rocky or stine
16:22:14 sean-k-mooney and not upstaed the config
16:22:41 mnaser ok yeah no this code isnt there in master
16:22:48 sean-k-mooney this is what we used to do https://github.com/openstack/nova/blob/stable/stein/nova/virt/libvirt/driver.py#L3912-L3931
16:23:20 mnaser i wonder if we ended up running newer than stein in some messed up wrong way :X
16:23:51 sean-k-mooney mnaser: you would have to be running master form last week too have that new code i think
16:23:57 mnaser ok then def not
16:24:18 sean-k-mooney mnaser: did you pull the wheels form pypi/tarballs.openstack.org by mistake?
16:24:53 mnaser no i just double checked and its the right stuff
16:25:51 sean-k-mooney oh you were allso looking at the traits code not the xml generation code
16:26:11 mnaser ah gotcha
16:26:29 sean-k-mooney the config is still the same https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L4150-L4181
16:26:59 openstackgerrit melanie witt proposed openstack/nova stable/stein: Add reno about nova-api eventlet monkey-patching and rabbitmq https://review.opendev.org/662095

Earlier   Later