| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-06-08 | |||
| 17:33:26 | artom | Keep commits topical | |
| 17:36:04 | mriedem | i like my creams just like i like my commits | |
| 17:36:04 | ihrachys | seems like port detach issues still happens even with https://review.openstack.org/#/c/441204/ in | |
| 17:36:10 | ihrachys | mriedem, ^ | |
| 17:36:20 | ihrachys | example: http://logs.openstack.org/63/471763/2/check/gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial/dccf187/logs/screen-n-cpu.txt.gz?level=TRACE#_Jun_08_16_58_01_395989 | |
| 17:37:17 | mriedem | hmm, that's not fun | |
| 17:38:11 | mriedem | ihrachys: it's always this same test right http://logs.openstack.org/63/471763/2/check/gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial/dccf187/console.html#_2017-06-08_17_13_16_844927 ? | |
| 17:38:14 | mriedem | the hotpluc nic one? | |
| 17:38:19 | mriedem | and usually on the linuxbridge job? | |
| 17:38:25 | artom | mriedem, the word you're looking for is "ointments" - it's up there with "moist" as the most shudder-inducing word in the English language | |
| 17:38:32 | mriedem | yeah it's like 99% LB job | |
| 17:38:35 | ihrachys | I think yes | |
| 17:38:43 | mriedem | i wonder what's different about LB in thiscase | |
| 17:38:55 | mriedem | maybe the vif isn't unplugging how we'd expect before we detach from the guest? | |
| 17:39:13 | mriedem | artom: i think "topical cream" | |
| 17:39:16 | mriedem | not "topical ointment" | |
| 17:39:21 | ihrachys | afaik for linuxbridge case, neutron now manages the whole plugging cycle ? | |
| 17:39:28 | ihrachys | since late switch to tap device type | |
| 17:39:38 | mriedem | ihrachys: i don't know anything about that | |
| 17:39:47 | ihrachys | I mean https://review.openstack.org/#/c/447150/ | |
| 17:39:55 | artom | mriedem, because of you my search history is weird now | |
| 17:40:08 | mriedem | artom: search on your phone, not your work system | |
| 17:40:25 | mriedem | nothing weird about grown man pain reliefe | |
| 17:40:27 | mriedem | *relief | |
| 17:40:37 | mriedem | just ask shaq | |
| 17:41:37 | mriedem | should be in the libvirtd logs wouldn't it? | |
| 17:43:22 | mriedem | i also don't understand why libvirt considers this a transient domain | |
| 17:43:52 | artom | I *think* it does both? | |
| 17:43:52 | mriedem | https://wiki.libvirt.org/page/VM_lifecycle#Transient_guest_domains_vs_Persistent_guest_domains | |
| 17:43:54 | mriedem | me neither | |
| 17:44:11 | artom | As in, a running domain has the persistent part, and the transient part | |
| 17:44:21 | artom | kashyap would help here, but I think he's gone | |
| 17:44:46 | melwitt | I think it's something like it creates a transient domain on-the-fly and uses that for the migration? something like that? | |
| 17:44:47 | mriedem | reading ^ makes me think all of our guests are transient | |
| 17:44:47 | ihrachys | oh it's just a in-ram domain? | |
| 17:44:57 | mriedem | this isn't a migration though | |
| 17:45:01 | melwitt | oh | |
| 17:45:15 | mriedem | http://git.openstack.org/cgit/openstack/tempest/tree/tempest/scenario/test_network_basic_ops.py#n475 | |
| 17:46:34 | cfriesen | mriedem: I think we generally make the xml file and tell libvirt to make a guest from it, making them persistent | |
| 17:48:46 | artom | Wait, that's not a libvirt error | |
| 17:49:28 | melwitt | one thing to know is a detach from transient domain is optional, that is, the guest can ignore it | |
| 17:50:02 | artom | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/guest.py#L442-L449 | |
| 17:50:10 | cfriesen | melwitt: that sounds like a terrible idea. | |
| 17:50:51 | mriedem | artom: oh nice spot | |
| 17:50:56 | mriedem | we should be checking the persistent flag there | |
| 17:50:59 | ihrachys | ehm, why in the snippet do we raise unconditionally? | |
| 17:51:07 | mriedem | ihrachys: it's part of a retry loop | |
| 17:51:10 | mriedem | see the decorator | |
| 17:51:27 | artom | It retries until it stops raising? | |
| 17:51:27 | mriedem | it retries on that expected error until the device conf is gone or we timeout | |
| 17:51:31 | ihrachys | ah I see | |
| 17:51:41 | cfriesen | melwitt: mriedem: so under what circumstances would nova create a transient domain? | |
| 17:52:28 | cfriesen | looks like maybe while doing a snapshot | |
| 17:52:28 | melwitt | I don't think nova does it, I thought it was something with qemu but I can't remember what kashyap told me. maybe I'm totally off on that | |
| 17:53:23 | mriedem | we're not using os-vif http://logs.openstack.org/63/471763/2/check/gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial/dccf187/logs/screen-n-cpu.txt.gz?level=DEBUG#_Jun_08_16_56_19_922445 | |
| 17:53:31 | mriedem | but that probably doesn't matter | |
| 17:56:09 | artom | Hrmm, that could be a libvirt error though | |
| 17:56:25 | artom | If we've failed to detach and reach the max retry count | |
| 17:56:50 | mriedem | it's got to be something with linuxbridge and this tap vif type | |
| 17:56:54 | mriedem | not a problem for ovs | |
| 17:57:04 | mriedem | and a race, since it's not 100% fail | |
| 17:57:48 | openstackgerrit | Merged openstack/nova master: Use microversions for new style volume attachments https://review.openstack.org/469579 | |
| 17:58:16 | kfox1111 | so... whats the easiest way to list out all vms in all tenants that use a particular flavor? | |
| 17:58:18 | mriedem | Successfully detached device <nova.virt.libvirt.config.LibvirtConfigGuestInterface object at 0x7f86a0051650> from guest. Persistent? 1. Live? True {{(pid=24466) _try_detach_device /opt/stack/new/nova/nova/virt/libvirt/guest.py:407}} | |
| 17:58:31 | melwitt | yeah, initially I was thinking we must have reached the max retry count because the guest wasn't responding, but I'm not sure how to tell from the log, if we can look at a timestamp delta at all to see how long it took before the error was raised | |
| 17:58:45 | mriedem | kfox1111: nova list --all-tenants --flavor x ? | |
| 17:59:01 | ihrachys | mriedem, you think we may try to revert the tap switch patch for the time being? | |
| 17:59:05 | kfox1111 | doh. there it was, right in front of me. | |
| 17:59:08 | kfox1111 | thanks. :) | |
| 17:59:08 | ihrachys | will it possibly help | |
| 17:59:16 | mriedem | ihrachys: the neutron one? | |
| 17:59:26 | mriedem | ihrachys: we could see if that's the cause by rechecking that a few times | |
| 17:59:27 | ihrachys | at this point, we can't land much in neutron, and we have a line of gate fixes, so anything that could help.. | |
| 17:59:42 | ihrachys | ok lemme revert and see where it goes | |
| 17:59:55 | mriedem | i don't know enough about the low level networking stuff here to know why this would cause issues | |
| 17:59:59 | mriedem | i'm going to go libvirtd log diving | |
| 18:03:09 | mriedem | right before we detach in the driver, we get the network-vif-unplugged event from neutron | |
| 18:05:32 | mriedem | but we don't do anything with that | |
| 18:06:07 | mriedem | "Successfully detached device <nova.virt.libvirt.config.LibvirtConfigGuestInterface object at 0x7f86a0051650> from guest. Persistent? 1. Live? True" | |
| 18:06:19 | mriedem | so i guess the domain is persistent on the first detach_device call to the guest | |
| 18:08:35 | openstackgerrit | Merged openstack/nova master: Use 3.27 microversion when creating new style volume attachments https://review.openstack.org/469609 | |
| 18:08:52 | mriedem | hmm, could we be hitting one of these libvirt errors and raising DeviceNotFound? | |
| 18:09:09 | mriedem | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/guest.py#L409 | |
| 18:09:21 | melwitt | compute manager ignores DeviceNotFound though, just considers detach already done, right? | |
| 18:09:22 | openstackgerrit | Merged openstack/nova master: Skip microversion discovery check for update/delete volume attachments https://review.openstack.org/469668 | |
| 18:09:32 | mriedem | for volumes | |
| 18:09:58 | mriedem | the driver ignores DeviceNotFound during detach_interface | |
| 18:10:01 | mriedem | so that's probably not it | |
| 18:10:44 | mriedem | wtf, i don't think it's even retrying | |
| 18:11:03 | mriedem | so we detach the guest here http://logs.openstack.org/63/471763/2/check/gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial/dccf187/logs/screen-n-cpu.txt.gz?level=DEBUG#_Jun_08_16_56_25_266814 | |
| 18:11:07 | mriedem | s/guest/device | |
| 18:11:13 | mriedem | that's https://github.com/openstack/nova/blob/master/nova/virt/libvirt/guest.py#L405 | |
| 18:11:17 | mriedem | on the first attempt | |
| 18:11:19 | mriedem | where persistent=1 | |
| 18:11:32 | mriedem | then we start our retry loop http://logs.openstack.org/63/471763/2/check/gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial/dccf187/logs/screen-n-cpu.txt.gz?level=DEBUG#_Jun_08_16_56_25_267223 | |
| 18:11:55 | mriedem | then we try detach again http://logs.openstack.org/63/471763/2/check/gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial/dccf187/logs/screen-n-cpu.txt.gz?level=DEBUG#_Jun_08_16_56_30_278317 | |
| 18:11:59 | mriedem | this time persistent=False | |
| 18:12:06 | mriedem | because https://github.com/openstack/nova/blob/master/nova/virt/libvirt/guest.py#L445 | |
| 18:13:04 | mriedem | oh nvm it does retry | |
| 18:13:07 | ihrachys | it may be interesting that tap3e81295f-c1 never shows in l2 agent log since 08 16:56:20.783179 which is long before the detach. should we have seen the agent to e.g. get a detach event from neutron-server and untangling in from the integration bridge, or smth? I assume nova talks to neutron when detaching? | |