Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-10-29
16:16:06 mriedem efried: i guess https://docs.openstack.org/nova/latest/reference/update-provider-tree.html is out of date given reshaper and the allocations kwarg
16:16:12 BlackDex Hello there. Something went wrong with a live-migrate. The instance is running on compute03 now, but openstack/horizon still thinks it is on compute01
16:16:19 BlackDex how can i fix this?
16:16:32 jaypipes BlackDex: please see /topic
16:16:50 BlackDex ah sorry
16:16:56 mriedem BlackDex: likely something failed in post-live migration and compute01 logs should tell you
16:17:08 finucannot gibi: I don't know if you found someone but I can chair it if not, sure
16:17:10 mriedem failed before updating the instance.host/node values to reflect compute03
16:18:50 jaypipes heh
16:21:48 lbragstad jaypipes sounds good - i'll be available this afternoon if you have questions/advice, just ping
16:23:26 gibi stephenfin: mriedem already took it but thanks for the offer
16:25:21 jaypipes lbragstad: will do, thx
16:32:56 openstackgerrit Matt Riedemann proposed openstack/nova master: Provide allocation_ratio/reserved amounts from update_provider_tree() https://review.openstack.org/613126
16:32:57 mriedem bauzas: efried: jaypipes: cdent: i think this is much more sane now ^
16:33:21 cdent mriedem: huzzah, on my list for today
16:36:32 mnaser do we have someone who's a libvirt expert around here?
16:36:43 mnaser i'm fairly certain i found a bug that breaks live migration because of libvirt
16:36:55 mnaser but i need a bit of guidance in nailing down the root cause :\
16:37:20 mnaser or heck, could be a nova bug if it's nova that generates the xml at the destination libvirt
16:37:32 mnaser but afaik that's not how it works
16:37:33 mdbooth_390 mnaser: Shoot. I might know something.
16:37:42 mdbooth_390 mnaser: Or kashyap might.
16:38:02 mnaser mdbooth_390: i have a live migration that is failing with "qemu-kvm: get_pci_config_device: Bad config data: i=0x10 read: 61 device: 1 cmask: ff wmask: c0 w1cmask:0"
16:38:34 mnaser upon doing research, it looks like that issue would come up if you change host_mtu, so i realized that the VMs that fail to migrate don't have host_mtu defined in the source, but for some reason, it's defined in the destination
16:39:00 mnaser so cat /var/log/libvirt/qemu/instance-being-migrated.log shows the new process being created (the one waiting for memory to be streamed in) with host_mtu=1500 defined
16:39:15 mnaser and then the migration fails because there's something something that becomes of a different size that causes that issue
16:39:48 efried mriedem: Re upt doc, yes, agree it should get updated. Guess that was missed when we did the docstrings.
16:40:11 mnaser so ideally: vms shouldn't have host_mtu in the qemu command, so the libvirt xml shouldn't have that... and i'm not sure how we're getting to that state
16:40:35 openstackgerrit Chris Friesen proposed openstack/nova-specs master: Add support for emulated virtual TPM https://review.openstack.org/571111
16:40:55 mdbooth_390 mnaser: Different libvirt versions source and dest?
16:40:59 mnaser mdbooth_390: yes
16:41:27 mnaser mdbooth_390: also, new instances booted in new libvirt contain host_mtu=1500
16:41:41 mnaser which means newly booted instances post-upgrade are live-migrateable, but pre-upgrade are not
16:41:52 mdbooth_390 mnaser: Any chance it's this: https://bugzilla.redhat.com/show_bug.cgi?id=1449346 ?
16:41:52 openstack bugzilla.redhat.com bug 1449346 in libvirt "Addition of host_mtu=XXXX changes PCI ioport size & addresses for virtio-net device" [Unspecified,Closed: errata] - Assigned to laine
16:42:18 mnaser mdbooth_390: i actaully saw this one, the revert is included in my local branch *however* the symptoms of the failure is exactly that (and that's what made me look for host_mtu)
16:42:33 mnaser so both my src and dest already have that patch
16:43:28 mnaser my question is: are xml files generated during a live migration by nova?
16:43:46 mdbooth_390 mnaser: No, they aren't.
16:44:05 mdbooth_390 They're tweaked by Nova, though.
16:44:08 mnaser ok so that eliminates nova possibly sending an mtu when it starts up
16:44:10 mnaser Aaaaah
16:44:24 mdbooth_390 But I don't think we set host_mtu anywhere
16:44:31 mnaser you do
16:44:34 mnaser one sec
16:44:44 mnaser mdbooth_390: https://github.com/openstack/nova/commit/f02b3800051234ecc14f3117d5987b1a8ef75877
16:44:45 cfriesen mnaser: we do generate xml files during live migration
16:45:02 mnaser so this ignores the fact that the source might *not* have host_mtu set
16:45:03 cfriesen the source generates the xml and passes it to libvirt as part of the migration
16:45:04 mdbooth_390 cfriesen: Nah, we do it post.
16:45:21 mdbooth_390 cfriesen: IIRC the xml sent to the dest is the running xml.
16:45:29 cfriesen mdbooth_390: we save the xml post, but we *generate* the xml and pass it to libvirt as part of the live migration call
16:45:36 mdbooth_390 cfriesen: So not generated by nova during the LM.
16:45:43 tomtom001 Hello, I ran into an issue in queens where if I shutdown CPU-A and try to evacuate the evacuation fails, then I delete the VM that I tried to evacuate. When I startup CPU-A nova-cmopute service fails to start because it can't find remote instance (the VM). What data is nova trying to use to determine what instances it needs to look for in it's startup process?
16:46:11 mnaser right, well if you look at the commit above, it does set the host_mtu stuff
16:46:19 mdbooth_390 cfriesen: Otherwise we'd lose all address info, for eg, and the lm would fail.
16:46:22 mnaser so.. could it be the culprit ?
16:46:41 mnaser added in rocky too and this is a rocky deployment
16:47:02 cfriesen mdbooth_390: https://github.com/openstack/nova/blob/master/nova/virt/libvirt/guest.py#L604 note the "destination_xml" arg
16:48:01 melwitt
16:48:35 mnaser oh
16:48:41 mdbooth_390 cfriesen: Look how it's generated
16:48:53 mdbooth_390 cfriesen: 'generated'
16:49:08 mdbooth_390 It's from the running guest
16:49:08 mnaser https://github.com/openstack/nova/blob/d87852ae6a1987b6faa3cb5851f9758b47ef4636/nova/virt/libvirt/driver.py#L7092-L7124
16:49:09 cfriesen mnaser: that commit looks suspicious to me
16:49:10 mnaser yes
16:49:11 mnaser but
16:49:17 mnaser it refreshes all the nic info
16:49:18 mnaser from that xml
16:49:59 mnaser https://github.com/openstack/nova/blob/f02b3800051234ecc14f3117d5987b1a8ef75877/nova/virt/libvirt/vif.py#L602-L624
16:50:17 mnaser https://github.com/openstack/nova/blob/f02b3800051234ecc14f3117d5987b1a8ef75877/nova/virt/libvirt/vif.py#L566-L598
16:50:44 cfriesen sean-k-mooney: possible problem with one of your commits: ^
16:50:47 mnaser https://github.com/openstack/nova/blob/9ae0cb67fabe09d6114680602f13b24da9dc88e2/nova/virt/libvirt/migration.py#L90
16:50:52 mnaser and that's how it breaks
16:51:28 mnaser we upgrade the vif config so if you're trying to migrate, host_mtu will be set in the destination which results in live migration failing
16:51:34 mdbooth_390 mnaser: So the guest was created before mtu was added, then nova was upgraded on the source, then we added mtu before lm?
16:51:44 mnaser mdbooth_390: yes
16:52:23 mnaser guest was long living, nova upgraded to rocky, live migration attempted, mtu is added before lm starts, lm fails at destination because host_mtu affects some internal kvm stuff that are beyond me :)
16:52:29 mdbooth_390 mnaser: Still sounds like a libvirt bug to me that this changes addressing, no?
16:52:44 cfriesen mdbooth_390: no, we're feeding the dest xml additional parameters that weren't there before
16:52:46 mnaser HOWEVER this also confirms the theory that instances started AFTER rocky would not be affected by this issue, i see it here
16:53:12 mnaser mdbooth_390: the libvirt bug there did the same behaviour we just started doing in nova, it started feeding host_mtu all the time and broke live migrations, so they reverted it
16:53:18 cfriesen in a live migration you need to create the dest guest with xml that is compatible with the source
16:53:20 mnaser we're doing the same mistake.. just one layer above :)
16:55:36 mdbooth_390 mnaser: I feel like I was at best a catalyst in that discussion. A figurative rubber duck :)
16:55:39 cfriesen I think we'll need to special-case this in the virt driver to ensure we don't add/remove the MTU setting over a live migration
16:56:17 cfriesen and we get to make fun of sean-k-mooney and mriedem for missing this scenario
16:56:21 mnaser well i needed to know about the nova-touching-xml part and then it made more sense :)
16:56:22 mnaser okay so
16:56:28 mnaser i'll file a bug
16:56:40 mnaser and i can hack on this because it's blocking a huge upgrade right now..
16:57:23 mnaser if anyone has suggestions.. maybe try to see if there's no mtu set and it's an lm, not set it ?
16:57:44 openstackgerrit Matt Riedemann proposed openstack/nova master: No longer call _normalize_inventory_from_cn_obj from upt flow https://review.openstack.org/613991
16:58:23 cfriesen I don't think we store the nova service version at the time an instance moved to a given compute node, so I think you might have to look at that instance xml before updating the vif-related xml stuff and ensure the mtu part stays the same
16:58:40 mriedem what did i miss? mixed version hypervisors with host mtu stuff on one of the 100 different vif type configurations in neutron? oops.
16:59:18 mdbooth_390 mriedem: Yeah, come on. It's so obvious!
17:04:31 cfriesen I updated the vTPM spec, I think it should address all the issues that were raised in the pervious version.
17:05:00 mnaser mriedem: not even mixed version hypervisor
17:05:12 mnaser i think this breaks live migrations of any vm created before pre-rocky to post-rocky

Earlier   Later