| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-04-24 | |||
| 18:33:46 | cdent | efried: even if that change isn't your problem, it may be the clue. are you in py2? | |
| 18:33:57 | mriedem | sean-k-mooney: so do you think this is safe https://review.openstack.org/#/c/551370/16/nova/virt/libvirt/migration.py@229 where it's basically taking the source vif and just overwriting whatever we got from the dest to get the new config xml? | |
| 18:34:02 | efried | cdent: Probably | |
| 18:34:21 | cdent | so if you've got a unicode there (for whatever random reason) | |
| 18:35:10 | efried | cdent: Our "official" fix is going to be using is_uuid_like. But that's going to require changes to pypowervm, which is going to need a requirements bump, which I'm not sure if we can swing in stable. Because we're going to have the same problem in nova (right esberglu?) | |
| 18:36:37 | cdent | I'm totally riffing at this point (because multitasking), but does six.text_type make a difference? | |
| 18:36:45 | sean-k-mooney | mriedem: my concern is how to you merge the two so that if i have a vm with 2 interfaces of the same type they dont swap places e.g. eth0 becomes eth1 and vice versa. | |
| 18:36:56 | efried | and pike | |
| 18:37:25 | esberglu | efried: That wasn't hitting queens | |
| 18:37:29 | esberglu | Only master | |
| 18:37:33 | efried | esberglu: See email - it is now. | |
| 18:37:47 | efried | and based on https://review.openstack.org/#/q/Ic6b6308fb1960ec40407e6efde30137b64543e72 it'll hit pike as soon as that percolates down. | |
| 18:37:52 | cdent | melwitt: okay, I've repeated fhe failures so can look more sensibly now | |
| 18:38:03 | sean-k-mooney | mriedem: i think you will have to loop over each interface element in the original xml and find the corresponding vif based on mac and then update the xml with the new atributes but maintain the order of the interfaces and maintianing the guest pci adress | |
| 18:39:34 | mriedem | sean-k-mooney: but wouldn't it we be weird to have interface xmls from the guest with certain source bridge and target dev values which are then unchanged on the dest host, but things like the vif type and vnic type could change? | |
| 18:39:42 | melwitt | thanks cdent | |
| 18:40:14 | melwitt | mriedem: pike https://review.openstack.org/#/c/562879 and ocata https://review.openstack.org/#/c/564044 release reviews for your perusal | |
| 18:40:15 | sean-k-mooney | mriedem: the bridge name might change on the dest as might hte vhost-user socekt path. | |
| 18:40:47 | sean-k-mooney | mriedem: centos use /run/openvswitch/... ubuntu uses /var/run/openvswitch/... | |
| 18:40:59 | mriedem | sean-k-mooney: but we don't have the bridge name in the migrate data object from the dest when we're munging the guest xml on the source | |
| 18:41:46 | sean-k-mooney | mriedem: you should have that as we have already created the binding on the dest host but not activated it | |
| 18:41:54 | sean-k-mooney | mriedem: it will be in the respocne from neutron | |
| 18:42:13 | mriedem | is that in the details or profile dict? | |
| 18:42:27 | sean-k-mooney | its in the vif binding_details | |
| 18:42:36 | sean-k-mooney | i belive the key is just bridge_name | |
| 18:43:05 | sean-k-mooney | mriedem: yep https://github.com/openstack/neutron-lib/blob/master/neutron_lib/api/definitions/portbindings.py#L53 | |
| 18:43:28 | cdent | melwitt: looks like a google summoer of code project totally revamped accept* handling in webob 1.8.*. still digging up details | |
| 18:43:41 | mriedem | is the vhostuser socket patch also in the vif binding details dict? | |
| 18:43:45 | mriedem | *path | |
| 18:43:53 | sean-k-mooney | yes | |
| 18:43:56 | mriedem | ok | |
| 18:44:11 | sean-k-mooney | get_vif_config(vif=vif) is the same fuction we use when spawning a vm right? | |
| 18:44:21 | mriedem | yes | |
| 18:44:26 | sean-k-mooney | if so it should be able to extract that info from the vif object | |
| 18:44:31 | sean-k-mooney | ok cool | |
| 18:46:35 | mriedem | sean-k-mooney: when you say "maintain the guest pci address" i don't see anything in the interface elements in the xml, e.g. http://logs.openstack.org/95/563995/1/check/neutron-tempest-linuxbridge/a3015a9/logs/screen-n-cpu.txt#_Apr_24_16_42_59_327342 | |
| 18:46:51 | mriedem | oh, is that only for sriov ports? | |
| 18:47:10 | mriedem | like vnic_type='direct'? | |
| 18:47:47 | sean-k-mooney | no the libvirt xml for the running vm is not the same as the one we hand to libvirt. it adds pci adress for each device itself | |
| 18:48:33 | sean-k-mooney | at least i think it does let me dump the xml from a running vm | |
| 18:51:07 | sean-k-mooney | mriedem: so this is a running vm xml http://paste.openstack.org/show/719855/ | |
| 18:51:29 | sean-k-mooney | the interfaces have an addtional <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> element added which is the guest virtual pci adresss | |
| 18:52:04 | sean-k-mooney | when we generate the xml fragment using get_vif_config we do not set that | |
| 18:52:26 | mriedem | right | |
| 18:52:29 | sean-k-mooney | so if you just merged the generated fagment with the exising interface and done touch the adress element it will be fine | |
| 18:52:33 | mriedem | we can parse it out of the running domain | |
| 18:52:42 | sean-k-mooney | yep | |
| 18:53:11 | mriedem | so here is another wrinkle wrt merging, | |
| 18:53:21 | mriedem | let's say i'm starting from a source host with ovs and have an interface element like this http://paste.openstack.org/show/719858/ | |
| 18:53:26 | mriedem | <source bridge="qbra188171c-ea"/> | |
| 18:53:37 | mriedem | then i live migrate to a dest host using vhostuser, | |
| 18:53:47 | mriedem | looking at a dpdk vhostuser CI logs, that is something like http://paste.openstack.org/show/719857/ | |
| 18:53:53 | mriedem | <source mode="server" path="/var/run/openvswitch/vhu88445a68-94" type="unix"/> | |
| 18:54:04 | mriedem | if we merge those, we have something like <source bridge="qbra188171c-ea" mode="server" path="/var/run/openvswitch/vhu88445a68-94" type="unix"/> | |
| 18:54:17 | mriedem | i would expect that to totally eff with libvirt/qemu | |
| 18:54:44 | mriedem | unless it's smart enough to only parse out attributes that it knows matter for the given interface type | |
| 18:54:57 | sean-k-mooney | right so if the vif types change we just want to copy the adress element other wise we replace it with the new vif | |
| 18:55:08 | sean-k-mooney | that might work in all cases actully | |
| 18:55:50 | mriedem | address or mac address? | |
| 18:55:57 | mriedem | get_vif_config doesn't give me the device address | |
| 18:56:19 | sean-k-mooney | the address element e.g. guest pci adress | |
| 18:57:08 | sean-k-mooney | so we jsut copy the <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> form the runing xml and add it ot what we get form get_vif_config | |
| 18:57:53 | sean-k-mooney | i think that will always be correct even if the vifs types are the same as i think that is the only thing we dont set | |
| 18:58:52 | mriedem | yeah ok | |
| 18:59:08 | mriedem | i knew there was a reason i've been putting off implementing this TODO | |
| 19:00:17 | sean-k-mooney | mriedem: ya its a bit of a pain but its doable. here is a xml from a running guest with vhost user for reference http://paste.openstack.org/show/719861/ | |
| 19:00:59 | sean-k-mooney | you should be able to use that to fake out the conversion from http://paste.openstack.org/show/719855/ which is kernel ovs | |
| 19:01:56 | sean-k-mooney | those are two completely different vms unfortunetly but atleast it has the interface definitions which is what you need | |
| 19:03:05 | sean-k-mooney | i have a 2 hour drive to my parents to do tonight so i have to run. ill be offline tomrow but feel free to ping me later in the week if you want any more input. | |
| 19:04:09 | mriedem | ack, thanks | |
| 19:06:54 | efried | edmondsw, esberglu, mriedem: https://bugs.launchpad.net/pypowervm/+bug/1766692 | |
| 19:06:54 | openstack | Launchpad bug 1766692 in pypowervm "instance.uuid no longer being a str breaks powervm scsi disconnect" [Undecided,New] | |
| 19:07:12 | efried | mriedem: What are the odds of backporting a pypowervm requirements bump to queens & pike? | |
| 19:08:33 | mriedem | not good | |
| 19:08:47 | mriedem | i don't understand why this is an issue on stable though | |
| 19:11:56 | openstackgerrit | Merged openstack/os-traits master: Add compute capabilities traits https://review.openstack.org/546713 | |
| 19:14:13 | edmondsw | mriedem because https://review.openstack.org/#/q/Ic6b6308fb1960ec40407e6efde30137b64543e72 was backported to pike and queens | |
| 19:15:10 | edmondsw | I suspect we'll have to do something like https://review.openstack.org/#/c/563314/1/nova_powervm/virt/powervm/vm.py@808 for pike and queens, as much as I hate that, to avoid a pypowervm requirements bump | |
| 19:15:26 | edmondsw | but if we could get a bump, we could avoid that | |
| 19:18:25 | mriedem | edmondsw: those aren't released in pike or queens | |
| 19:18:33 | mriedem | upper-constraints depends on released packages on pypi | |
| 19:19:22 | mriedem | are you guys sure that your CI running against stable isn't using upper-constraints from master? | |
| 19:19:55 | edmondsw | mriedem yes, we're sure | |
| 19:19:58 | edmondsw | but I'm not following you | |
| 19:20:15 | mriedem | your CI should not be pulling in those changes from stable | |
| 19:20:36 | edmondsw | I don't know that we've seen this in CI for stable... we just saw that it merged in stable | |
| 19:20:48 | edmondsw | we've seen this in CI for master | |
| 19:20:51 | mriedem | then you revert those changes before they are released | |
| 19:21:13 | mriedem | or, backport whatever the fix ends up being from master | |
| 19:21:52 | edmondsw | for master, we should just fix pypowervm and bump requirements | |
| 19:22:05 | edmondsw | hence the question of whether we can backport a requirements bump | |
| 19:22:23 | edmondsw | we could try to get the oslo change reverted | |
| 19:22:50 | efried | mriedem: We have seen this in queens, live, this morning. | |
| 19:22:59 | mriedem | efried: in CI? | |
| 19:23:03 | efried | no, live. | |
| 19:23:04 | mriedem | if so, link me to the failed patch | |
| 19:23:21 | mriedem | so someone isn't using upper-constraints in production | |
| 19:23:35 | efried | esberglu: Can you spin up a queens CI run without the pypowervm patch so we can confirm whether this fails there? | |
| 19:23:44 | efried | please and thank you | |
| 19:23:54 | esberglu | efried: Sure | |