| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-06-08 | |||
| 17:44:21 | artom | kashyap would help here, but I think he's gone | |
| 17:44:46 | melwitt | I think it's something like it creates a transient domain on-the-fly and uses that for the migration? something like that? | |
| 17:44:47 | mriedem | reading ^ makes me think all of our guests are transient | |
| 17:44:47 | ihrachys | oh it's just a in-ram domain? | |
| 17:44:57 | mriedem | this isn't a migration though | |
| 17:45:01 | melwitt | oh | |
| 17:45:15 | mriedem | http://git.openstack.org/cgit/openstack/tempest/tree/tempest/scenario/test_network_basic_ops.py#n475 | |
| 17:46:34 | cfriesen | mriedem: I think we generally make the xml file and tell libvirt to make a guest from it, making them persistent | |
| 17:48:46 | artom | Wait, that's not a libvirt error | |
| 17:49:28 | melwitt | one thing to know is a detach from transient domain is optional, that is, the guest can ignore it | |
| 17:50:02 | artom | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/guest.py#L442-L449 | |
| 17:50:10 | cfriesen | melwitt: that sounds like a terrible idea. | |
| 17:50:51 | mriedem | artom: oh nice spot | |
| 17:50:56 | mriedem | we should be checking the persistent flag there | |
| 17:50:59 | ihrachys | ehm, why in the snippet do we raise unconditionally? | |
| 17:51:07 | mriedem | ihrachys: it's part of a retry loop | |
| 17:51:10 | mriedem | see the decorator | |
| 17:51:27 | artom | It retries until it stops raising? | |
| 17:51:27 | mriedem | it retries on that expected error until the device conf is gone or we timeout | |
| 17:51:31 | ihrachys | ah I see | |
| 17:51:41 | cfriesen | melwitt: mriedem: so under what circumstances would nova create a transient domain? | |
| 17:52:28 | cfriesen | looks like maybe while doing a snapshot | |
| 17:52:28 | melwitt | I don't think nova does it, I thought it was something with qemu but I can't remember what kashyap told me. maybe I'm totally off on that | |
| 17:53:23 | mriedem | we're not using os-vif http://logs.openstack.org/63/471763/2/check/gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial/dccf187/logs/screen-n-cpu.txt.gz?level=DEBUG#_Jun_08_16_56_19_922445 | |
| 17:53:31 | mriedem | but that probably doesn't matter | |
| 17:56:09 | artom | Hrmm, that could be a libvirt error though | |
| 17:56:25 | artom | If we've failed to detach and reach the max retry count | |
| 17:56:50 | mriedem | it's got to be something with linuxbridge and this tap vif type | |
| 17:56:54 | mriedem | not a problem for ovs | |
| 17:57:04 | mriedem | and a race, since it's not 100% fail | |
| 17:57:48 | openstackgerrit | Merged openstack/nova master: Use microversions for new style volume attachments https://review.openstack.org/469579 | |
| 17:58:16 | kfox1111 | so... whats the easiest way to list out all vms in all tenants that use a particular flavor? | |
| 17:58:18 | mriedem | Successfully detached device <nova.virt.libvirt.config.LibvirtConfigGuestInterface object at 0x7f86a0051650> from guest. Persistent? 1. Live? True {{(pid=24466) _try_detach_device /opt/stack/new/nova/nova/virt/libvirt/guest.py:407}} | |
| 17:58:31 | melwitt | yeah, initially I was thinking we must have reached the max retry count because the guest wasn't responding, but I'm not sure how to tell from the log, if we can look at a timestamp delta at all to see how long it took before the error was raised | |
| 17:58:45 | mriedem | kfox1111: nova list --all-tenants --flavor x ? | |
| 17:59:01 | ihrachys | mriedem, you think we may try to revert the tap switch patch for the time being? | |
| 17:59:05 | kfox1111 | doh. there it was, right in front of me. | |
| 17:59:08 | kfox1111 | thanks. :) | |
| 17:59:08 | ihrachys | will it possibly help | |
| 17:59:16 | mriedem | ihrachys: the neutron one? | |
| 17:59:26 | mriedem | ihrachys: we could see if that's the cause by rechecking that a few times | |
| 17:59:27 | ihrachys | at this point, we can't land much in neutron, and we have a line of gate fixes, so anything that could help.. | |
| 17:59:42 | ihrachys | ok lemme revert and see where it goes | |
| 17:59:55 | mriedem | i don't know enough about the low level networking stuff here to know why this would cause issues | |
| 17:59:59 | mriedem | i'm going to go libvirtd log diving | |
| 18:03:09 | mriedem | right before we detach in the driver, we get the network-vif-unplugged event from neutron | |
| 18:05:32 | mriedem | but we don't do anything with that | |
| 18:06:07 | mriedem | "Successfully detached device <nova.virt.libvirt.config.LibvirtConfigGuestInterface object at 0x7f86a0051650> from guest. Persistent? 1. Live? True" | |
| 18:06:19 | mriedem | so i guess the domain is persistent on the first detach_device call to the guest | |
| 18:08:35 | openstackgerrit | Merged openstack/nova master: Use 3.27 microversion when creating new style volume attachments https://review.openstack.org/469609 | |
| 18:08:52 | mriedem | hmm, could we be hitting one of these libvirt errors and raising DeviceNotFound? | |
| 18:09:09 | mriedem | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/guest.py#L409 | |
| 18:09:21 | melwitt | compute manager ignores DeviceNotFound though, just considers detach already done, right? | |
| 18:09:22 | openstackgerrit | Merged openstack/nova master: Skip microversion discovery check for update/delete volume attachments https://review.openstack.org/469668 | |
| 18:09:32 | mriedem | for volumes | |
| 18:09:58 | mriedem | the driver ignores DeviceNotFound during detach_interface | |
| 18:10:01 | mriedem | so that's probably not it | |
| 18:10:44 | mriedem | wtf, i don't think it's even retrying | |
| 18:11:03 | mriedem | so we detach the guest here http://logs.openstack.org/63/471763/2/check/gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial/dccf187/logs/screen-n-cpu.txt.gz?level=DEBUG#_Jun_08_16_56_25_266814 | |
| 18:11:07 | mriedem | s/guest/device | |
| 18:11:13 | mriedem | that's https://github.com/openstack/nova/blob/master/nova/virt/libvirt/guest.py#L405 | |
| 18:11:17 | mriedem | on the first attempt | |
| 18:11:19 | mriedem | where persistent=1 | |
| 18:11:32 | mriedem | then we start our retry loop http://logs.openstack.org/63/471763/2/check/gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial/dccf187/logs/screen-n-cpu.txt.gz?level=DEBUG#_Jun_08_16_56_25_267223 | |
| 18:11:55 | mriedem | then we try detach again http://logs.openstack.org/63/471763/2/check/gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial/dccf187/logs/screen-n-cpu.txt.gz?level=DEBUG#_Jun_08_16_56_30_278317 | |
| 18:11:59 | mriedem | this time persistent=False | |
| 18:12:06 | mriedem | because https://github.com/openstack/nova/blob/master/nova/virt/libvirt/guest.py#L445 | |
| 18:13:04 | mriedem | oh nvm it does retry | |
| 18:13:07 | ihrachys | it may be interesting that tap3e81295f-c1 never shows in l2 agent log since 08 16:56:20.783179 which is long before the detach. should we have seen the agent to e.g. get a detach event from neutron-server and untangling in from the integration bridge, or smth? I assume nova talks to neutron when detaching? | |
| 18:13:19 | mriedem | the retry decorator doesn't have the logging context for the request id | |
| 18:13:58 | mriedem | ihrachys: nova will only update the port to unbind it | |
| 18:14:11 | mriedem | otherwise nova just unplugs the vif and relies on neutron agent picking that up and processing it | |
| 18:14:20 | mriedem | and then neutron sends the network-vif-unplugged event | |
| 18:17:30 | mriedem | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/guest.py#L247-L256 | |
| 18:17:30 | mriedem | i wonder if we're not checking the right things here | |
| 18:18:29 | mriedem | this is our xml | |
| 18:18:30 | mriedem | http://paste.openstack.org/show/612041/ | |
| 18:18:35 | mriedem | from LibvirtConfigGuestInterface | |
| 18:19:56 | ihrachys | mriedem, what do you mean unplugging the VIF? from the domain? because with tap vif type, everything else is probably on neutron to do right? meaning, you request unbind and then we handle it all? | |
| 18:20:20 | ihrachys | kevinbenton, are you up already? | |
| 18:20:32 | mriedem | ihrachys: i mean this https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1450 | |
| 18:22:35 | ihrachys | which is empty for lb | |
| 18:23:51 | mriedem | ihrachys: no it's going here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/vif.py#L906 | |
| 18:24:19 | ihrachys | oh. I was looking at unplug in os-vif | |
| 18:24:24 | ihrachys | for lb plugin | |
| 18:24:45 | mriedem | yeah it's not using os-vif | |
| 18:24:57 | mriedem | because there is no os-vif tap conversion in nova yet | |
| 18:25:08 | mriedem | http://logs.openstack.org/63/471763/2/check/gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial/dccf187/logs/screen-n-cpu.txt.gz?level=DEBUG#_Jun_08_16_56_19_922445 | |
| 18:25:13 | mriedem | No conversion for VIF type tap yet | |
| 18:25:28 | mriedem | so nova does: sudo nova-rootwrap /etc/nova/rootwrap.conf ip link delete tap3e81295f-c1 | |
| 18:30:51 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: fix alternative_device_name for detaching interfaces https://review.openstack.org/472371 | |
| 18:31:10 | mriedem | ihrachys: i also noticed this comment which is probably from danpb https://github.com/openstack/nova/blob/master/nova/virt/libvirt/designer.py#L56 | |
| 18:31:21 | mriedem | but that's the method called to build the device config for this type of tap device | |
| 18:31:29 | ihrachys | btw I misinterpreted the logs before. I now see that neutron logs operations on the tap device at the same time nova starts the decorated func | |
| 18:32:39 | mriedem | this is all i see about it in the libvirt docs https://libvirt.org/formatdomain.html#elementsNICSEthernet | |
| 18:32:53 | mriedem | i assume it's discouraged because "Provides a means for the administrator to execute an arbitrary script to connect the guest's network to the LAN." | |
| 18:33:52 | mriedem | as far as i can tell though nova doesn't pass through any script path | |
| 18:34:05 | mriedem | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/designer.py#L69 | |
| 18:34:27 | mriedem | oh i think that means if not provided, it defaults to /etc/qemu-ifup | |
| 18:38:19 | mriedem | ihrachys: i'm going to push a change that catches the final exception, dumps a bunch of debug info to the logs, and then re-raises | |