Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-06-08
17:38:32 mriedem yeah it's like 99% LB job
17:38:35 ihrachys I think yes
17:38:43 mriedem i wonder what's different about LB in thiscase
17:38:55 mriedem maybe the vif isn't unplugging how we'd expect before we detach from the guest?
17:39:13 mriedem artom: i think "topical cream"
17:39:16 mriedem not "topical ointment"
17:39:21 ihrachys afaik for linuxbridge case, neutron now manages the whole plugging cycle ?
17:39:28 ihrachys since late switch to tap device type
17:39:38 mriedem ihrachys: i don't know anything about that
17:39:47 ihrachys I mean https://review.openstack.org/#/c/447150/
17:39:55 artom mriedem, because of you my search history is weird now
17:40:08 mriedem artom: search on your phone, not your work system
17:40:25 mriedem nothing weird about grown man pain reliefe
17:40:27 mriedem *relief
17:40:37 mriedem just ask shaq
17:41:37 mriedem should be in the libvirtd logs wouldn't it?
17:43:22 mriedem i also don't understand why libvirt considers this a transient domain
17:43:52 artom I *think* it does both?
17:43:52 mriedem https://wiki.libvirt.org/page/VM_lifecycle#Transient_guest_domains_vs_Persistent_guest_domains
17:43:54 mriedem me neither
17:44:11 artom As in, a running domain has the persistent part, and the transient part
17:44:21 artom kashyap would help here, but I think he's gone
17:44:46 melwitt I think it's something like it creates a transient domain on-the-fly and uses that for the migration? something like that?
17:44:47 mriedem reading ^ makes me think all of our guests are transient
17:44:47 ihrachys oh it's just a in-ram domain?
17:44:57 mriedem this isn't a migration though
17:45:01 melwitt oh
17:45:15 mriedem http://git.openstack.org/cgit/openstack/tempest/tree/tempest/scenario/test_network_basic_ops.py#n475
17:46:34 cfriesen mriedem: I think we generally make the xml file and tell libvirt to make a guest from it, making them persistent
17:48:46 artom Wait, that's not a libvirt error
17:49:28 melwitt one thing to know is a detach from transient domain is optional, that is, the guest can ignore it
17:50:02 artom https://github.com/openstack/nova/blob/master/nova/virt/libvirt/guest.py#L442-L449
17:50:10 cfriesen melwitt: that sounds like a terrible idea.
17:50:51 mriedem artom: oh nice spot
17:50:56 mriedem we should be checking the persistent flag there
17:50:59 ihrachys ehm, why in the snippet do we raise unconditionally?
17:51:07 mriedem ihrachys: it's part of a retry loop
17:51:10 mriedem see the decorator
17:51:27 artom It retries until it stops raising?
17:51:27 mriedem it retries on that expected error until the device conf is gone or we timeout
17:51:31 ihrachys ah I see
17:51:41 cfriesen melwitt: mriedem: so under what circumstances would nova create a transient domain?
17:52:28 cfriesen looks like maybe while doing a snapshot
17:52:28 melwitt I don't think nova does it, I thought it was something with qemu but I can't remember what kashyap told me. maybe I'm totally off on that
17:53:23 mriedem we're not using os-vif http://logs.openstack.org/63/471763/2/check/gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial/dccf187/logs/screen-n-cpu.txt.gz?level=DEBUG#_Jun_08_16_56_19_922445
17:53:31 mriedem but that probably doesn't matter
17:56:09 artom Hrmm, that could be a libvirt error though
17:56:25 artom If we've failed to detach and reach the max retry count
17:56:50 mriedem it's got to be something with linuxbridge and this tap vif type
17:56:54 mriedem not a problem for ovs
17:57:04 mriedem and a race, since it's not 100% fail
17:57:48 openstackgerrit Merged openstack/nova master: Use microversions for new style volume attachments https://review.openstack.org/469579
17:58:16 kfox1111 so... whats the easiest way to list out all vms in all tenants that use a particular flavor?
17:58:18 mriedem Successfully detached device <nova.virt.libvirt.config.LibvirtConfigGuestInterface object at 0x7f86a0051650> from guest. Persistent? 1. Live? True {{(pid=24466) _try_detach_device /opt/stack/new/nova/nova/virt/libvirt/guest.py:407}}
17:58:31 melwitt yeah, initially I was thinking we must have reached the max retry count because the guest wasn't responding, but I'm not sure how to tell from the log, if we can look at a timestamp delta at all to see how long it took before the error was raised
17:58:45 mriedem kfox1111: nova list --all-tenants --flavor x ?
17:59:01 ihrachys mriedem, you think we may try to revert the tap switch patch for the time being?
17:59:05 kfox1111 doh. there it was, right in front of me.
17:59:08 kfox1111 thanks. :)
17:59:08 ihrachys will it possibly help
17:59:16 mriedem ihrachys: the neutron one?
17:59:26 mriedem ihrachys: we could see if that's the cause by rechecking that a few times
17:59:27 ihrachys at this point, we can't land much in neutron, and we have a line of gate fixes, so anything that could help..
17:59:42 ihrachys ok lemme revert and see where it goes
17:59:55 mriedem i don't know enough about the low level networking stuff here to know why this would cause issues
17:59:59 mriedem i'm going to go libvirtd log diving
18:03:09 mriedem right before we detach in the driver, we get the network-vif-unplugged event from neutron
18:05:32 mriedem but we don't do anything with that
18:06:07 mriedem "Successfully detached device <nova.virt.libvirt.config.LibvirtConfigGuestInterface object at 0x7f86a0051650> from guest. Persistent? 1. Live? True"
18:06:19 mriedem so i guess the domain is persistent on the first detach_device call to the guest
18:08:35 openstackgerrit Merged openstack/nova master: Use 3.27 microversion when creating new style volume attachments https://review.openstack.org/469609
18:08:52 mriedem hmm, could we be hitting one of these libvirt errors and raising DeviceNotFound?
18:09:09 mriedem https://github.com/openstack/nova/blob/master/nova/virt/libvirt/guest.py#L409
18:09:21 melwitt compute manager ignores DeviceNotFound though, just considers detach already done, right?
18:09:22 openstackgerrit Merged openstack/nova master: Skip microversion discovery check for update/delete volume attachments https://review.openstack.org/469668
18:09:32 mriedem for volumes
18:09:58 mriedem the driver ignores DeviceNotFound during detach_interface
18:10:01 mriedem so that's probably not it
18:10:44 mriedem wtf, i don't think it's even retrying
18:11:03 mriedem so we detach the guest here http://logs.openstack.org/63/471763/2/check/gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial/dccf187/logs/screen-n-cpu.txt.gz?level=DEBUG#_Jun_08_16_56_25_266814
18:11:07 mriedem s/guest/device
18:11:13 mriedem that's https://github.com/openstack/nova/blob/master/nova/virt/libvirt/guest.py#L405
18:11:17 mriedem on the first attempt
18:11:19 mriedem where persistent=1
18:11:32 mriedem then we start our retry loop http://logs.openstack.org/63/471763/2/check/gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial/dccf187/logs/screen-n-cpu.txt.gz?level=DEBUG#_Jun_08_16_56_25_267223
18:11:55 mriedem then we try detach again http://logs.openstack.org/63/471763/2/check/gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial/dccf187/logs/screen-n-cpu.txt.gz?level=DEBUG#_Jun_08_16_56_30_278317
18:11:59 mriedem this time persistent=False
18:12:06 mriedem because https://github.com/openstack/nova/blob/master/nova/virt/libvirt/guest.py#L445
18:13:04 mriedem oh nvm it does retry
18:13:07 ihrachys it may be interesting that tap3e81295f-c1 never shows in l2 agent log since 08 16:56:20.783179 which is long before the detach. should we have seen the agent to e.g. get a detach event from neutron-server and untangling in from the integration bridge, or smth? I assume nova talks to neutron when detaching?
18:13:19 mriedem the retry decorator doesn't have the logging context for the request id
18:13:58 mriedem ihrachys: nova will only update the port to unbind it
18:14:11 mriedem otherwise nova just unplugs the vif and relies on neutron agent picking that up and processing it
18:14:20 mriedem and then neutron sends the network-vif-unplugged event
18:17:30 mriedem https://github.com/openstack/nova/blob/master/nova/virt/libvirt/guest.py#L247-L256
18:17:30 mriedem i wonder if we're not checking the right things here
18:18:29 mriedem this is our xml
18:18:30 mriedem http://paste.openstack.org/show/612041/
18:18:35 mriedem from LibvirtConfigGuestInterface
18:19:56 ihrachys mriedem, what do you mean unplugging the VIF? from the domain? because with tap vif type, everything else is probably on neutron to do right? meaning, you request unbind and then we handle it all?

Earlier   Later