Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-04-24
18:25:09 cdent efried: so "ValueError: invalid literal for int() with base 10: '4E27E1E6-6A24-4F0A-8E7B-2BBE7B4A28BA'" is caused by a conditional faililng somewhere that wasn't before? I'm unable to grep isinstance anywhere in the log?
18:25:59 efried cdent: It's caused by this: https://github.com/powervm/pypowervm/blob/master/pypowervm/utils/uuid.py#L50
18:26:21 efried That isinstance() fails, so we try to int() the UUID on L55, leading to the ValueError.
18:26:48 mriedem efried: have you looked at https://github.com/openstack/oslo.versionedobjects/compare/1.32.0...1.33.1 ?
18:27:04 mriedem https://github.com/openstack/oslo.versionedobjects/commit/b1d0b5d886afef8c08330bf3c2291e180aa1f534
18:27:09 efried cdent: or, shit, I guess it's possible the regex match could be failing. But when I str(instance.uuid) up the stack, it succeeds.
18:27:11 efried mriedem: looking...
18:27:14 sean-k-mooney mriedem: so if we can assume that the vm cannont have 2 intefaces with the same mac then i think that is likely the best way to approch. if we had a way to store the neutron port uuid in the xml that would help alot but i dont think we can do that with libvirt
18:27:50 efried mriedem: Bingo. When did we subsume that req?
18:27:53 efried in nova
18:27:57 efried in queens
18:28:00 efried cause I looked for that
18:28:06 mriedem efried: upper-constraints on friday for rocky
18:28:14 mriedem https://github.com/openstack/requirements/commit/87540884100650cfd1a67f05163a724906efb46f#diff-0bdd949ed8a7fdd4f95240bd951779c8
18:28:15 efried ahhhh, upper-constraints.
18:28:28 efried Yup, that'd do it. Thanks mriedem
18:28:37 efried I knew you would come through for me.
18:30:03 efried mriedem: That same thing must have gone into queens somehow.
18:30:32 mriedem or your CI is using the wrong upper-constraints?
18:30:41 mriedem your queens CI is likely pointing at master
18:31:25 mriedem although maybe not https://github.com/openstack/oslo.versionedobjects/commit/e918eb976fb5a6f9fa7b7644d5a10d383fcfcf21
18:31:38 mriedem https://review.openstack.org/#/q/Ic6b6308fb1960ec40407e6efde30137b64543e72
18:32:04 mriedem that's not released yet on stable though
18:33:46 cdent efried: even if that change isn't your problem, it may be the clue. are you in py2?
18:33:57 mriedem sean-k-mooney: so do you think this is safe https://review.openstack.org/#/c/551370/16/nova/virt/libvirt/migration.py@229 where it's basically taking the source vif and just overwriting whatever we got from the dest to get the new config xml?
18:34:02 efried cdent: Probably
18:34:21 cdent so if you've got a unicode there (for whatever random reason)
18:35:10 efried cdent: Our "official" fix is going to be using is_uuid_like. But that's going to require changes to pypowervm, which is going to need a requirements bump, which I'm not sure if we can swing in stable. Because we're going to have the same problem in nova (right esberglu?)
18:36:37 cdent I'm totally riffing at this point (because multitasking), but does six.text_type make a difference?
18:36:45 sean-k-mooney mriedem: my concern is how to you merge the two so that if i have a vm with 2 interfaces of the same type they dont swap places e.g. eth0 becomes eth1 and vice versa.
18:36:56 efried and pike
18:37:25 esberglu efried: That wasn't hitting queens
18:37:29 esberglu Only master
18:37:33 efried esberglu: See email - it is now.
18:37:47 efried and based on https://review.openstack.org/#/q/Ic6b6308fb1960ec40407e6efde30137b64543e72 it'll hit pike as soon as that percolates down.
18:37:52 cdent melwitt: okay, I've repeated fhe failures so can look more sensibly now
18:38:03 sean-k-mooney mriedem: i think you will have to loop over each interface element in the original xml and find the corresponding vif based on mac and then update the xml with the new atributes but maintain the order of the interfaces and maintianing the guest pci adress
18:39:34 mriedem sean-k-mooney: but wouldn't it we be weird to have interface xmls from the guest with certain source bridge and target dev values which are then unchanged on the dest host, but things like the vif type and vnic type could change?
18:39:42 melwitt thanks cdent
18:40:14 melwitt mriedem: pike https://review.openstack.org/#/c/562879 and ocata https://review.openstack.org/#/c/564044 release reviews for your perusal
18:40:15 sean-k-mooney mriedem: the bridge name might change on the dest as might hte vhost-user socekt path.
18:40:47 sean-k-mooney mriedem: centos use /run/openvswitch/... ubuntu uses /var/run/openvswitch/...
18:40:59 mriedem sean-k-mooney: but we don't have the bridge name in the migrate data object from the dest when we're munging the guest xml on the source
18:41:46 sean-k-mooney mriedem: you should have that as we have already created the binding on the dest host but not activated it
18:41:54 sean-k-mooney mriedem: it will be in the respocne from neutron
18:42:13 mriedem is that in the details or profile dict?
18:42:27 sean-k-mooney its in the vif binding_details
18:42:36 sean-k-mooney i belive the key is just bridge_name
18:43:05 sean-k-mooney mriedem: yep https://github.com/openstack/neutron-lib/blob/master/neutron_lib/api/definitions/portbindings.py#L53
18:43:28 cdent melwitt: looks like a google summoer of code project totally revamped accept* handling in webob 1.8.*. still digging up details
18:43:41 mriedem is the vhostuser socket patch also in the vif binding details dict?
18:43:45 mriedem *path
18:43:53 sean-k-mooney yes
18:43:56 mriedem ok
18:44:11 sean-k-mooney get_vif_config(vif=vif) is the same fuction we use when spawning a vm right?
18:44:21 mriedem yes
18:44:26 sean-k-mooney if so it should be able to extract that info from the vif object
18:44:31 sean-k-mooney ok cool
18:46:35 mriedem sean-k-mooney: when you say "maintain the guest pci address" i don't see anything in the interface elements in the xml, e.g. http://logs.openstack.org/95/563995/1/check/neutron-tempest-linuxbridge/a3015a9/logs/screen-n-cpu.txt#_Apr_24_16_42_59_327342
18:46:51 mriedem oh, is that only for sriov ports?
18:47:10 mriedem like vnic_type='direct'?
18:47:47 sean-k-mooney no the libvirt xml for the running vm is not the same as the one we hand to libvirt. it adds pci adress for each device itself
18:48:33 sean-k-mooney at least i think it does let me dump the xml from a running vm
18:51:07 sean-k-mooney mriedem: so this is a running vm xml http://paste.openstack.org/show/719855/
18:51:29 sean-k-mooney the interfaces have an addtional <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> element added which is the guest virtual pci adresss
18:52:04 sean-k-mooney when we generate the xml fragment using get_vif_config we do not set that
18:52:26 mriedem right
18:52:29 sean-k-mooney so if you just merged the generated fagment with the exising interface and done touch the adress element it will be fine
18:52:33 mriedem we can parse it out of the running domain
18:52:42 sean-k-mooney yep
18:53:11 mriedem so here is another wrinkle wrt merging,
18:53:21 mriedem let's say i'm starting from a source host with ovs and have an interface element like this http://paste.openstack.org/show/719858/
18:53:26 mriedem <source bridge="qbra188171c-ea"/>
18:53:37 mriedem then i live migrate to a dest host using vhostuser,
18:53:47 mriedem looking at a dpdk vhostuser CI logs, that is something like http://paste.openstack.org/show/719857/
18:53:53 mriedem <source mode="server" path="/var/run/openvswitch/vhu88445a68-94" type="unix"/>
18:54:04 mriedem if we merge those, we have something like <source bridge="qbra188171c-ea" mode="server" path="/var/run/openvswitch/vhu88445a68-94" type="unix"/>
18:54:17 mriedem i would expect that to totally eff with libvirt/qemu
18:54:44 mriedem unless it's smart enough to only parse out attributes that it knows matter for the given interface type
18:54:57 sean-k-mooney right so if the vif types change we just want to copy the adress element other wise we replace it with the new vif
18:55:08 sean-k-mooney that might work in all cases actully
18:55:50 mriedem address or mac address?
18:55:57 mriedem get_vif_config doesn't give me the device address
18:56:19 sean-k-mooney the address element e.g. guest pci adress
18:57:08 sean-k-mooney so we jsut copy the <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> form the runing xml and add it ot what we get form get_vif_config
18:57:53 sean-k-mooney i think that will always be correct even if the vifs types are the same as i think that is the only thing we dont set
18:58:52 mriedem yeah ok
18:59:08 mriedem i knew there was a reason i've been putting off implementing this TODO
19:00:17 sean-k-mooney mriedem: ya its a bit of a pain but its doable. here is a xml from a running guest with vhost user for reference http://paste.openstack.org/show/719861/
19:00:59 sean-k-mooney you should be able to use that to fake out the conversion from http://paste.openstack.org/show/719855/ which is kernel ovs
19:01:56 sean-k-mooney those are two completely different vms unfortunetly but atleast it has the interface definitions which is what you need
19:03:05 sean-k-mooney i have a 2 hour drive to my parents to do tonight so i have to run. ill be offline tomrow but feel free to ping me later in the week if you want any more input.
19:04:09 mriedem ack, thanks
19:06:54 efried edmondsw, esberglu, mriedem: https://bugs.launchpad.net/pypowervm/+bug/1766692
19:06:54 openstack Launchpad bug 1766692 in pypowervm "instance.uuid no longer being a str breaks powervm scsi disconnect" [Undecided,New]
19:07:12 efried mriedem: What are the odds of backporting a pypowervm requirements bump to queens & pike?
19:08:33 mriedem not good
19:08:47 mriedem i don't understand why this is an issue on stable though
19:11:56 openstackgerrit Merged openstack/os-traits master: Add compute capabilities traits https://review.openstack.org/546713
19:14:13 edmondsw mriedem because https://review.openstack.org/#/q/Ic6b6308fb1960ec40407e6efde30137b64543e72 was backported to pike and queens

Earlier   Later