| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-18 | |||
| 16:27:23 | openstackgerrit | Merged openstack/nova stable/stein: Trap and log errors from _update_inst_info_cache_for_disassociated_fip https://review.opendev.org/682182 | |
| 16:27:32 | openstackgerrit | Merged openstack/nova stable/stein: Find instance in another cell during floating IP re-association https://review.opendev.org/682183 | |
| 16:27:39 | openstackgerrit | Merged openstack/nova stable/stein: Log notifications if assertion in _test_live_migration_force_complete fails https://review.opendev.org/681743 | |
| 16:28:41 | mriedem | stephenfin: you mean https://review.opendev.org/#/c/682267/ right? | |
| 16:29:03 | stephenfin | yup | |
| 16:29:04 | sean-k-mooney | mnaser: those feature flags still look wrong to me | |
| 16:29:05 | mriedem | i definitely think we need some functional testing for the libvirt driver that sets cpu extra flags b/c that's 2 regressions when using that config option in the last week or so | |
| 16:29:22 | mriedem | and configuring cpu extra flags seems pretty important for red hat customers... | |
| 16:29:24 | sean-k-mooney | mriedem: it imples we are getting to the extra_flag section or libvirt added them | |
| 16:29:25 | stephenfin | Agreed. I want to flesh out that functional test of mine and use that | |
| 16:29:36 | stephenfin | Just need to figure out where to put it now | |
| 16:29:37 | sean-k-mooney | mriedem: soory that was for mnaser | |
| 16:29:46 | mnaser | sean-k-mooney: yeah it doesn't look right | |
| 16:29:54 | mnaser | but i dont know if virsh dumpxml does some magic | |
| 16:29:59 | mnaser | and doesnt actaully print out what it got from nova | |
| 16:30:00 | mriedem | stephenfin: maybe just a new simple libvirt functoinal test that isn't in the test_numa module or whatever | |
| 16:30:01 | sean-k-mooney | mnaser: was that config form the nova compute log or libvirt via virsh | |
| 16:30:04 | mnaser | no virsh | |
| 16:30:10 | mnaser | so i havent blamed nova yet | |
| 16:30:10 | sean-k-mooney | virsh does not | |
| 16:30:17 | sean-k-mooney | libvirt transforms the xml we pass in | |
| 16:30:27 | mnaser | yeah so this might be transformed | |
| 16:30:28 | sean-k-mooney | e.g. it does not actually use the xml we provide. | |
| 16:30:33 | stephenfin | yeah, I'd say so. I just need to check how much of the mocking I need to carry across | |
| 16:30:33 | mnaser | though i wonder why it wouldn't keep host-model there | |
| 16:31:20 | mnaser | hmm i remember virsh dumpxml used to have a "dumpable" version | |
| 16:31:36 | sean-k-mooney | mnaser: ya i dont know either. this is where i delegate to danpb or kashyap as i read C but dont really want to have to find where that happens | |
| 16:31:49 | sean-k-mooney | mnaser: its not a virsh issue | |
| 16:31:59 | mnaser | ok fair nuff | |
| 16:32:12 | sean-k-mooney | wehn we create teh xml and define the domain libvirt parses it and save an updated version to disk and in memory | |
| 16:32:49 | sean-k-mooney | virsh talks to libvirt over a unix socket and virsh dumpxml give you the rendered version | |
| 16:33:11 | mnaser | oooh | |
| 16:33:13 | sean-k-mooney | libvirt when it ingest the xml fills in a bunch of things like guest pci address that we dont set | |
| 16:33:15 | mnaser | wait so i can do cat on the local disk then | |
| 16:33:43 | sean-k-mooney | there is a copy saved to disk somewhere but you could also just check the nova compute log | |
| 16:33:47 | sean-k-mooney | the xml is saved in it | |
| 16:33:55 | mnaser | yeah not running $world with DEBUG on | |
| 16:33:59 | mnaser | so that'd have to be a little exercise :p | |
| 16:34:12 | mnaser | its only logged when running under debug afaik | |
| 16:34:21 | sean-k-mooney | yes it is | |
| 16:34:40 | mnaser | time to google for the millionth time "how to run an openstack instance on a specific node" | |
| 16:34:51 | sean-k-mooney | anyway i would check the qemu instance log and see what the error is | |
| 16:35:00 | openstackgerrit | Merged openstack/nova-specs master: Update spec: filtering of alloc candidates by forbidden aggregates https://review.opendev.org/675384 | |
| 16:35:15 | sean-k-mooney | cat /var/log/libvirt/qemu/instanace... | |
| 16:35:33 | mnaser | yeah but i tihnk its a layer before that because the qemu process doesnt have the flag | |
| 16:35:34 | mnaser | "-cpu IvyBridge-IBRS,ss=on,pcid=on,hypervisor=on,arat=on,tsc_adjust=on,md-clear=on,stibp=on,ssbd=on,xsaveopt=on,pdpe1gb=on" | |
| 16:35:54 | sean-k-mooney | which flag is missing? | |
| 16:36:04 | sean-k-mooney | i though the vm was crashing? | |
| 16:36:06 | mnaser | vmx (and the fact that things are manually added) | |
| 16:36:13 | mnaser | nope, no crashes at all, just literally a flag missing | |
| 16:36:23 | mnaser | virsh capabilities shows vmx | |
| 16:36:32 | mnaser | but booting instances without defining cpu_model does not | |
| 16:36:40 | sean-k-mooney | right but did you turn nested virt on in the host kernel | |
| 16:36:42 | mnaser | does not boot it with vmx then | |
| 16:36:46 | mnaser | yep | |
| 16:37:05 | sean-k-mooney | ok i normally use host-passthough with nested virt so the flag is there | |
| 16:37:08 | mnaser | `virsh capabilities` wouldn't report vmx there if it wasnt on either afaik | |
| 16:37:11 | mnaser | yeah, im trying to avoid that one | |
| 16:37:17 | sean-k-mooney | the model your using might just not included it | |
| 16:37:21 | sean-k-mooney | you could jsut add it | |
| 16:37:26 | mnaser | cat /sys/module/kvm_intel/parameters/nested => Y | |
| 16:37:29 | sean-k-mooney | with the config | |
| 16:37:53 | mnaser | sean-k-mooney: it actually does though -- http://paste.openstack.org/show/777431/ | |
| 16:37:54 | mnaser | thats the thing | |
| 16:38:09 | mnaser | i could add it manually but im trying to see if its something we can improve upstream :> | |
| 16:38:10 | sean-k-mooney | the host does | |
| 16:38:29 | mnaser | isnt the output of virsh capabiltiies show the model used by libvirt? | |
| 16:38:35 | mnaser | in this case IvyBridge-IBRS which does include vmx | |
| 16:38:36 | sean-k-mooney | that does not mean the IvyBridge-IBRS has the flag | |
| 16:38:43 | mnaser | wait what | |
| 16:38:45 | mnaser | seriously | |
| 16:38:51 | sean-k-mooney | yes | |
| 16:39:07 | sean-k-mooney | there is an xml file you can check | |
| 16:39:10 | mnaser | i thought virsh capabilities was what would host-model be | |
| 16:39:17 | sean-k-mooney | no | |
| 16:39:20 | mnaser | cpu_map.xml | |
| 16:39:28 | sean-k-mooney | its litrally the host capablites unfiltered | |
| 16:39:57 | sean-k-mooney | the model it reporet is the closet on to your host cpu | |
| 16:40:04 | mnaser | well | |
| 16:40:07 | mnaser | that explains all my confusion | |
| 16:40:14 | mnaser | then i need to add the extra cpu flag | |
| 16:40:28 | mnaser | cpu_map.xml shows nothing actually has vmx flag | |
| 16:40:37 | mnaser | i could _swear_ that it used to be there before | |
| 16:40:51 | sean-k-mooney | where is cpu_map by the way i was looking for it locally | |
| 16:41:03 | mnaser | sean-k-mooney: /usr/share/libvirt/cpu_map.xml | |
| 16:41:08 | mnaser | (on centos at least) | |
| 16:41:14 | sean-k-mooney | ah user share i was looking in /var/lib | |
| 16:41:54 | mnaser | ok im really confused as to how it used to work before but i guess it did | |
| 16:42:20 | sean-k-mooney | it could be a libvirt change | |
| 16:42:53 | sean-k-mooney | or you were using host-passthough and did not no/notice/remember | |
| 16:43:20 | mnaser | no as much as host-passthrough is great im very anti that because live migrations | |
| 16:43:51 | sean-k-mooney | you can use it with live migrations | |
| 16:43:59 | sean-k-mooney | but you need to group things by hardware type | |
| 16:44:08 | mnaser | yeah it gets tricky for new hardware | |
| 16:44:09 | sean-k-mooney | by the way host-model is not relaly better | |
| 16:44:40 | sean-k-mooney | if you use host model and you move for ivybridge to haswell then reboot the vm it wont be abel to migrte back | |
| 16:44:51 | mnaser | yeah im perfectly ok with migrating from old to new but not new to old | |
| 16:44:58 | sean-k-mooney | if you really care about migration you should use custom and select a version across all nodes | |
| 16:44:58 | mnaser | it enables my use case of "upgrading things" | |
| 16:52:26 | stephenfin | mriedem: Can you cut an sqlalchemy-migrate release? | |