| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-06-08 | |||
| 18:09:09 | mriedem | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/guest.py#L409 | |
| 18:09:21 | melwitt | compute manager ignores DeviceNotFound though, just considers detach already done, right? | |
| 18:09:22 | openstackgerrit | Merged openstack/nova master: Skip microversion discovery check for update/delete volume attachments https://review.openstack.org/469668 | |
| 18:09:32 | mriedem | for volumes | |
| 18:09:58 | mriedem | the driver ignores DeviceNotFound during detach_interface | |
| 18:10:01 | mriedem | so that's probably not it | |
| 18:10:44 | mriedem | wtf, i don't think it's even retrying | |
| 18:11:03 | mriedem | so we detach the guest here http://logs.openstack.org/63/471763/2/check/gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial/dccf187/logs/screen-n-cpu.txt.gz?level=DEBUG#_Jun_08_16_56_25_266814 | |
| 18:11:07 | mriedem | s/guest/device | |
| 18:11:13 | mriedem | that's https://github.com/openstack/nova/blob/master/nova/virt/libvirt/guest.py#L405 | |
| 18:11:17 | mriedem | on the first attempt | |
| 18:11:19 | mriedem | where persistent=1 | |
| 18:11:32 | mriedem | then we start our retry loop http://logs.openstack.org/63/471763/2/check/gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial/dccf187/logs/screen-n-cpu.txt.gz?level=DEBUG#_Jun_08_16_56_25_267223 | |
| 18:11:55 | mriedem | then we try detach again http://logs.openstack.org/63/471763/2/check/gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial/dccf187/logs/screen-n-cpu.txt.gz?level=DEBUG#_Jun_08_16_56_30_278317 | |
| 18:11:59 | mriedem | this time persistent=False | |
| 18:12:06 | mriedem | because https://github.com/openstack/nova/blob/master/nova/virt/libvirt/guest.py#L445 | |
| 18:13:04 | mriedem | oh nvm it does retry | |
| 18:13:07 | ihrachys | it may be interesting that tap3e81295f-c1 never shows in l2 agent log since 08 16:56:20.783179 which is long before the detach. should we have seen the agent to e.g. get a detach event from neutron-server and untangling in from the integration bridge, or smth? I assume nova talks to neutron when detaching? | |
| 18:13:19 | mriedem | the retry decorator doesn't have the logging context for the request id | |
| 18:13:58 | mriedem | ihrachys: nova will only update the port to unbind it | |
| 18:14:11 | mriedem | otherwise nova just unplugs the vif and relies on neutron agent picking that up and processing it | |
| 18:14:20 | mriedem | and then neutron sends the network-vif-unplugged event | |
| 18:17:30 | mriedem | i wonder if we're not checking the right things here | |
| 18:17:30 | mriedem | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/guest.py#L247-L256 | |
| 18:18:29 | mriedem | this is our xml | |
| 18:18:30 | mriedem | http://paste.openstack.org/show/612041/ | |
| 18:18:35 | mriedem | from LibvirtConfigGuestInterface | |
| 18:19:56 | ihrachys | mriedem, what do you mean unplugging the VIF? from the domain? because with tap vif type, everything else is probably on neutron to do right? meaning, you request unbind and then we handle it all? | |
| 18:20:20 | ihrachys | kevinbenton, are you up already? | |
| 18:20:32 | mriedem | ihrachys: i mean this https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1450 | |
| 18:22:35 | ihrachys | which is empty for lb | |
| 18:23:51 | mriedem | ihrachys: no it's going here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/vif.py#L906 | |
| 18:24:19 | ihrachys | oh. I was looking at unplug in os-vif | |
| 18:24:24 | ihrachys | for lb plugin | |
| 18:24:45 | mriedem | yeah it's not using os-vif | |
| 18:24:57 | mriedem | because there is no os-vif tap conversion in nova yet | |
| 18:25:08 | mriedem | http://logs.openstack.org/63/471763/2/check/gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial/dccf187/logs/screen-n-cpu.txt.gz?level=DEBUG#_Jun_08_16_56_19_922445 | |
| 18:25:13 | mriedem | No conversion for VIF type tap yet | |
| 18:25:28 | mriedem | so nova does: sudo nova-rootwrap /etc/nova/rootwrap.conf ip link delete tap3e81295f-c1 | |
| 18:30:51 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: fix alternative_device_name for detaching interfaces https://review.openstack.org/472371 | |
| 18:31:10 | mriedem | ihrachys: i also noticed this comment which is probably from danpb https://github.com/openstack/nova/blob/master/nova/virt/libvirt/designer.py#L56 | |
| 18:31:21 | mriedem | but that's the method called to build the device config for this type of tap device | |
| 18:31:29 | ihrachys | btw I misinterpreted the logs before. I now see that neutron logs operations on the tap device at the same time nova starts the decorated func | |
| 18:32:39 | mriedem | this is all i see about it in the libvirt docs https://libvirt.org/formatdomain.html#elementsNICSEthernet | |
| 18:32:53 | mriedem | i assume it's discouraged because "Provides a means for the administrator to execute an arbitrary script to connect the guest's network to the LAN." | |
| 18:33:52 | mriedem | as far as i can tell though nova doesn't pass through any script path | |
| 18:34:05 | mriedem | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/designer.py#L69 | |
| 18:34:27 | mriedem | oh i think that means if not provided, it defaults to /etc/qemu-ifup | |
| 18:38:19 | mriedem | ihrachys: i'm going to push a change that catches the final exception, dumps a bunch of debug info to the logs, and then re-raises | |
| 18:38:33 | mriedem | to see what's actually in the guest xml when we fail compared to the device xml we're trying to remove | |
| 18:54:22 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: dump debug info when interface detach times out https://review.openstack.org/472377 | |
| 18:54:24 | mriedem | ihrachys: i'll push a neutron change that depends on this ^ to debug, i see you pushed a revert | |
| 18:56:11 | openstackgerrit | Corey Bryant proposed openstack/nova master: Make nova-placement-api default config less strict https://review.openstack.org/472378 | |
| 18:59:22 | cfriesen | if instances have attached volumes, that information would be in the block_device_mapping table of the nova db, right? | |
| 19:00:33 | mriedem | cfriesen: yes | |
| 19:05:12 | mriedem | we have an intermittent KeyError in functional tests https://bugs.launchpad.net/nova/+bug/1696834 | |
| 19:05:12 | openstack | Launchpad bug 1696834 in OpenStack Compute (nova) "Intermittent "KeyError: 'allocations'" in functional tests" [Medium,Confirmed] | |
| 19:05:31 | melwitt | yeah, seems to hit pretty often too | |
| 19:05:42 | mriedem | 14 in 7 days | |
| 19:17:10 | mriedem | dansmith: melwitt: can one of you put this through so i don't have to work it into my hypervisors uuid api series? https://review.openstack.org/#/c/471970/ | |
| 19:17:15 | mriedem | i have to build on that one | |
| 19:17:17 | mriedem | to change the type | |
| 19:17:33 | dansmith | done | |
| 19:17:37 | openstackgerrit | Corey Bryant proposed openstack/nova master: Make nova-placement-api default config less strict https://review.openstack.org/472378 | |
| 19:17:39 | mriedem | <3 | |
| 19:18:27 | mriedem | fleet week failed again, | |
| 19:18:30 | mriedem | but something different this time http://logs.openstack.org/94/436094/33/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/bed0e6a/console.html#_2017-06-08_18_39_03_929168 | |
| 19:19:40 | dansmith | o.O | |
| 19:21:10 | mordred | efried: https://review.openstack.org/#/q/project:openstack/keystoneauth+topic:version-discovery is updated from reviews if you wanna poke | |
| 19:21:25 | efried | mordred Roger wilco. | |
| 19:21:31 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Handle uuid in HostAPI.compute_node_get https://review.openstack.org/461532 | |
| 19:21:31 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Handle uuids in os-hypervisors API https://review.openstack.org/471899 | |
| 19:21:31 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Use uuid for id in Services API https://review.openstack.org/464280 | |
| 19:21:48 | mordred | efried: still missing interface lists and service type aliases - but we're getting real close | |
| 19:21:58 | efried | neat | |
| 19:22:59 | mriedem | dansmith: unrelated | |
| 19:23:12 | mriedem | the test is posting a password to the metadata server | |
| 19:23:13 | dansmith | good, because....... | |
| 19:23:18 | mriedem | and then checks to see it shows up | |
| 19:23:32 | mriedem | which seems super racy | |
| 19:23:51 | mriedem | that's totally async so tempest should probably be polling here | |
| 19:26:08 | mriedem | maybe i'm not sure, | |
| 19:26:15 | mriedem | it sets a password field in the instance sys meta | |
| 19:26:17 | mriedem | it's a POST, | |
| 19:26:24 | mriedem | but i'm not sure where the metadata service returns | |
| 19:26:34 | openstackgerrit | Merged openstack/nova master: api-ref: fix type for hypervisor_marker https://review.openstack.org/471970 | |
| 19:26:41 | kevinbenton | ihrachys: hey | |
| 19:28:07 | kevinbenton | ihrachys: looking at backlog. l2-agent never deletes a tap device, that's left up to nova | |
| 19:28:12 | kevinbenton | mriedem: ^^ | |
| 19:28:23 | kevinbenton | but it looks like you already found that | |
| 19:28:26 | kevinbenton | anything i can help with? | |
| 19:32:59 | mriedem | dansmith: i guess the metadata POST is synchronous | |
| 19:36:47 | mriedem | dansmith: aha http://logs.openstack.org/94/436094/33/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/bed0e6a/logs/screen-n-api.txt.gz#_Jun_08_18_36_06_549748 | |
| 19:36:53 | mriedem | the cache fs us again | |
| 19:37:25 | mriedem | we update inst.system_metadata, then tempest calls the rest api to get the password, and it looks like things are going through the metadata cache | |
| 19:37:35 | mriedem | not sure why we wouldn't update the cache when we update/set the password... | |
| 19:38:10 | dansmith | but that's still unrelated right? | |
| 19:38:11 | dansmith | but yeah, that metadata stuff needs fixin' | |
| 19:38:21 | mriedem | wtf, we're getting an InstanceNotFound | |
| 19:38:37 | mriedem | oooooo | |