| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-06-08 | |||
| 17:51:31 | ihrachys | ah I see | |
| 17:51:41 | cfriesen | melwitt: mriedem: so under what circumstances would nova create a transient domain? | |
| 17:52:28 | cfriesen | looks like maybe while doing a snapshot | |
| 17:52:28 | melwitt | I don't think nova does it, I thought it was something with qemu but I can't remember what kashyap told me. maybe I'm totally off on that | |
| 17:53:23 | mriedem | we're not using os-vif http://logs.openstack.org/63/471763/2/check/gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial/dccf187/logs/screen-n-cpu.txt.gz?level=DEBUG#_Jun_08_16_56_19_922445 | |
| 17:53:31 | mriedem | but that probably doesn't matter | |
| 17:56:09 | artom | Hrmm, that could be a libvirt error though | |
| 17:56:25 | artom | If we've failed to detach and reach the max retry count | |
| 17:56:50 | mriedem | it's got to be something with linuxbridge and this tap vif type | |
| 17:56:54 | mriedem | not a problem for ovs | |
| 17:57:04 | mriedem | and a race, since it's not 100% fail | |
| 17:57:48 | openstackgerrit | Merged openstack/nova master: Use microversions for new style volume attachments https://review.openstack.org/469579 | |
| 17:58:16 | kfox1111 | so... whats the easiest way to list out all vms in all tenants that use a particular flavor? | |
| 17:58:18 | mriedem | Successfully detached device <nova.virt.libvirt.config.LibvirtConfigGuestInterface object at 0x7f86a0051650> from guest. Persistent? 1. Live? True {{(pid=24466) _try_detach_device /opt/stack/new/nova/nova/virt/libvirt/guest.py:407}} | |
| 17:58:31 | melwitt | yeah, initially I was thinking we must have reached the max retry count because the guest wasn't responding, but I'm not sure how to tell from the log, if we can look at a timestamp delta at all to see how long it took before the error was raised | |
| 17:58:45 | mriedem | kfox1111: nova list --all-tenants --flavor x ? | |
| 17:59:01 | ihrachys | mriedem, you think we may try to revert the tap switch patch for the time being? | |
| 17:59:05 | kfox1111 | doh. there it was, right in front of me. | |
| 17:59:08 | kfox1111 | thanks. :) | |
| 17:59:08 | ihrachys | will it possibly help | |
| 17:59:16 | mriedem | ihrachys: the neutron one? | |
| 17:59:26 | mriedem | ihrachys: we could see if that's the cause by rechecking that a few times | |
| 17:59:27 | ihrachys | at this point, we can't land much in neutron, and we have a line of gate fixes, so anything that could help.. | |
| 17:59:42 | ihrachys | ok lemme revert and see where it goes | |
| 17:59:55 | mriedem | i don't know enough about the low level networking stuff here to know why this would cause issues | |
| 17:59:59 | mriedem | i'm going to go libvirtd log diving | |
| 18:03:09 | mriedem | right before we detach in the driver, we get the network-vif-unplugged event from neutron | |
| 18:05:32 | mriedem | but we don't do anything with that | |
| 18:06:07 | mriedem | "Successfully detached device <nova.virt.libvirt.config.LibvirtConfigGuestInterface object at 0x7f86a0051650> from guest. Persistent? 1. Live? True" | |
| 18:06:19 | mriedem | so i guess the domain is persistent on the first detach_device call to the guest | |
| 18:08:35 | openstackgerrit | Merged openstack/nova master: Use 3.27 microversion when creating new style volume attachments https://review.openstack.org/469609 | |
| 18:08:52 | mriedem | hmm, could we be hitting one of these libvirt errors and raising DeviceNotFound? | |
| 18:09:09 | mriedem | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/guest.py#L409 | |
| 18:09:21 | melwitt | compute manager ignores DeviceNotFound though, just considers detach already done, right? | |
| 18:09:22 | openstackgerrit | Merged openstack/nova master: Skip microversion discovery check for update/delete volume attachments https://review.openstack.org/469668 | |
| 18:09:32 | mriedem | for volumes | |
| 18:09:58 | mriedem | the driver ignores DeviceNotFound during detach_interface | |
| 18:10:01 | mriedem | so that's probably not it | |
| 18:10:44 | mriedem | wtf, i don't think it's even retrying | |
| 18:11:03 | mriedem | so we detach the guest here http://logs.openstack.org/63/471763/2/check/gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial/dccf187/logs/screen-n-cpu.txt.gz?level=DEBUG#_Jun_08_16_56_25_266814 | |
| 18:11:07 | mriedem | s/guest/device | |
| 18:11:13 | mriedem | that's https://github.com/openstack/nova/blob/master/nova/virt/libvirt/guest.py#L405 | |
| 18:11:17 | mriedem | on the first attempt | |
| 18:11:19 | mriedem | where persistent=1 | |
| 18:11:32 | mriedem | then we start our retry loop http://logs.openstack.org/63/471763/2/check/gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial/dccf187/logs/screen-n-cpu.txt.gz?level=DEBUG#_Jun_08_16_56_25_267223 | |
| 18:11:55 | mriedem | then we try detach again http://logs.openstack.org/63/471763/2/check/gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial/dccf187/logs/screen-n-cpu.txt.gz?level=DEBUG#_Jun_08_16_56_30_278317 | |
| 18:11:59 | mriedem | this time persistent=False | |
| 18:12:06 | mriedem | because https://github.com/openstack/nova/blob/master/nova/virt/libvirt/guest.py#L445 | |
| 18:13:04 | mriedem | oh nvm it does retry | |
| 18:13:07 | ihrachys | it may be interesting that tap3e81295f-c1 never shows in l2 agent log since 08 16:56:20.783179 which is long before the detach. should we have seen the agent to e.g. get a detach event from neutron-server and untangling in from the integration bridge, or smth? I assume nova talks to neutron when detaching? | |
| 18:13:19 | mriedem | the retry decorator doesn't have the logging context for the request id | |
| 18:13:58 | mriedem | ihrachys: nova will only update the port to unbind it | |
| 18:14:11 | mriedem | otherwise nova just unplugs the vif and relies on neutron agent picking that up and processing it | |
| 18:14:20 | mriedem | and then neutron sends the network-vif-unplugged event | |
| 18:17:30 | mriedem | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/guest.py#L247-L256 | |
| 18:17:30 | mriedem | i wonder if we're not checking the right things here | |
| 18:18:29 | mriedem | this is our xml | |
| 18:18:30 | mriedem | http://paste.openstack.org/show/612041/ | |
| 18:18:35 | mriedem | from LibvirtConfigGuestInterface | |
| 18:19:56 | ihrachys | mriedem, what do you mean unplugging the VIF? from the domain? because with tap vif type, everything else is probably on neutron to do right? meaning, you request unbind and then we handle it all? | |
| 18:20:20 | ihrachys | kevinbenton, are you up already? | |
| 18:20:32 | mriedem | ihrachys: i mean this https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1450 | |
| 18:22:35 | ihrachys | which is empty for lb | |
| 18:23:51 | mriedem | ihrachys: no it's going here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/vif.py#L906 | |
| 18:24:19 | ihrachys | oh. I was looking at unplug in os-vif | |
| 18:24:24 | ihrachys | for lb plugin | |
| 18:24:45 | mriedem | yeah it's not using os-vif | |
| 18:24:57 | mriedem | because there is no os-vif tap conversion in nova yet | |
| 18:25:08 | mriedem | http://logs.openstack.org/63/471763/2/check/gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial/dccf187/logs/screen-n-cpu.txt.gz?level=DEBUG#_Jun_08_16_56_19_922445 | |
| 18:25:13 | mriedem | No conversion for VIF type tap yet | |
| 18:25:28 | mriedem | so nova does: sudo nova-rootwrap /etc/nova/rootwrap.conf ip link delete tap3e81295f-c1 | |
| 18:30:51 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: fix alternative_device_name for detaching interfaces https://review.openstack.org/472371 | |
| 18:31:10 | mriedem | ihrachys: i also noticed this comment which is probably from danpb https://github.com/openstack/nova/blob/master/nova/virt/libvirt/designer.py#L56 | |
| 18:31:21 | mriedem | but that's the method called to build the device config for this type of tap device | |
| 18:31:29 | ihrachys | btw I misinterpreted the logs before. I now see that neutron logs operations on the tap device at the same time nova starts the decorated func | |
| 18:32:39 | mriedem | this is all i see about it in the libvirt docs https://libvirt.org/formatdomain.html#elementsNICSEthernet | |
| 18:32:53 | mriedem | i assume it's discouraged because "Provides a means for the administrator to execute an arbitrary script to connect the guest's network to the LAN." | |
| 18:33:52 | mriedem | as far as i can tell though nova doesn't pass through any script path | |
| 18:34:05 | mriedem | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/designer.py#L69 | |
| 18:34:27 | mriedem | oh i think that means if not provided, it defaults to /etc/qemu-ifup | |
| 18:38:19 | mriedem | ihrachys: i'm going to push a change that catches the final exception, dumps a bunch of debug info to the logs, and then re-raises | |
| 18:38:33 | mriedem | to see what's actually in the guest xml when we fail compared to the device xml we're trying to remove | |
| 18:54:22 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: dump debug info when interface detach times out https://review.openstack.org/472377 | |
| 18:54:24 | mriedem | ihrachys: i'll push a neutron change that depends on this ^ to debug, i see you pushed a revert | |
| 18:56:11 | openstackgerrit | Corey Bryant proposed openstack/nova master: Make nova-placement-api default config less strict https://review.openstack.org/472378 | |
| 18:59:22 | cfriesen | if instances have attached volumes, that information would be in the block_device_mapping table of the nova db, right? | |
| 19:00:33 | mriedem | cfriesen: yes | |
| 19:05:12 | openstack | Launchpad bug 1696834 in OpenStack Compute (nova) "Intermittent "KeyError: 'allocations'" in functional tests" [Medium,Confirmed] | |
| 19:05:12 | mriedem | we have an intermittent KeyError in functional tests https://bugs.launchpad.net/nova/+bug/1696834 | |
| 19:05:31 | melwitt | yeah, seems to hit pretty often too | |
| 19:05:42 | mriedem | 14 in 7 days | |
| 19:17:10 | mriedem | dansmith: melwitt: can one of you put this through so i don't have to work it into my hypervisors uuid api series? https://review.openstack.org/#/c/471970/ | |
| 19:17:15 | mriedem | i have to build on that one | |
| 19:17:17 | mriedem | to change the type | |
| 19:17:33 | dansmith | done | |
| 19:17:37 | openstackgerrit | Corey Bryant proposed openstack/nova master: Make nova-placement-api default config less strict https://review.openstack.org/472378 | |
| 19:17:39 | mriedem | <3 | |
| 19:18:27 | mriedem | fleet week failed again, | |
| 19:18:30 | mriedem | but something different this time http://logs.openstack.org/94/436094/33/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/bed0e6a/console.html#_2017-06-08_18_39_03_929168 | |
| 19:19:40 | dansmith | o.O | |