| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-06-08 | |||
| 18:18:29 | mriedem | this is our xml | |
| 18:18:30 | mriedem | http://paste.openstack.org/show/612041/ | |
| 18:18:35 | mriedem | from LibvirtConfigGuestInterface | |
| 18:19:56 | ihrachys | mriedem, what do you mean unplugging the VIF? from the domain? because with tap vif type, everything else is probably on neutron to do right? meaning, you request unbind and then we handle it all? | |
| 18:20:20 | ihrachys | kevinbenton, are you up already? | |
| 18:20:32 | mriedem | ihrachys: i mean this https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1450 | |
| 18:22:35 | ihrachys | which is empty for lb | |
| 18:23:51 | mriedem | ihrachys: no it's going here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/vif.py#L906 | |
| 18:24:19 | ihrachys | oh. I was looking at unplug in os-vif | |
| 18:24:24 | ihrachys | for lb plugin | |
| 18:24:45 | mriedem | yeah it's not using os-vif | |
| 18:24:57 | mriedem | because there is no os-vif tap conversion in nova yet | |
| 18:25:08 | mriedem | http://logs.openstack.org/63/471763/2/check/gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial/dccf187/logs/screen-n-cpu.txt.gz?level=DEBUG#_Jun_08_16_56_19_922445 | |
| 18:25:13 | mriedem | No conversion for VIF type tap yet | |
| 18:25:28 | mriedem | so nova does: sudo nova-rootwrap /etc/nova/rootwrap.conf ip link delete tap3e81295f-c1 | |
| 18:30:51 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: fix alternative_device_name for detaching interfaces https://review.openstack.org/472371 | |
| 18:31:10 | mriedem | ihrachys: i also noticed this comment which is probably from danpb https://github.com/openstack/nova/blob/master/nova/virt/libvirt/designer.py#L56 | |
| 18:31:21 | mriedem | but that's the method called to build the device config for this type of tap device | |
| 18:31:29 | ihrachys | btw I misinterpreted the logs before. I now see that neutron logs operations on the tap device at the same time nova starts the decorated func | |
| 18:32:39 | mriedem | this is all i see about it in the libvirt docs https://libvirt.org/formatdomain.html#elementsNICSEthernet | |
| 18:32:53 | mriedem | i assume it's discouraged because "Provides a means for the administrator to execute an arbitrary script to connect the guest's network to the LAN." | |
| 18:33:52 | mriedem | as far as i can tell though nova doesn't pass through any script path | |
| 18:34:05 | mriedem | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/designer.py#L69 | |
| 18:34:27 | mriedem | oh i think that means if not provided, it defaults to /etc/qemu-ifup | |
| 18:38:19 | mriedem | ihrachys: i'm going to push a change that catches the final exception, dumps a bunch of debug info to the logs, and then re-raises | |
| 18:38:33 | mriedem | to see what's actually in the guest xml when we fail compared to the device xml we're trying to remove | |
| 18:54:22 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: dump debug info when interface detach times out https://review.openstack.org/472377 | |
| 18:54:24 | mriedem | ihrachys: i'll push a neutron change that depends on this ^ to debug, i see you pushed a revert | |
| 18:56:11 | openstackgerrit | Corey Bryant proposed openstack/nova master: Make nova-placement-api default config less strict https://review.openstack.org/472378 | |
| 18:59:22 | cfriesen | if instances have attached volumes, that information would be in the block_device_mapping table of the nova db, right? | |
| 19:00:33 | mriedem | cfriesen: yes | |
| 19:05:12 | openstack | Launchpad bug 1696834 in OpenStack Compute (nova) "Intermittent "KeyError: 'allocations'" in functional tests" [Medium,Confirmed] | |
| 19:05:12 | mriedem | we have an intermittent KeyError in functional tests https://bugs.launchpad.net/nova/+bug/1696834 | |
| 19:05:31 | melwitt | yeah, seems to hit pretty often too | |
| 19:05:42 | mriedem | 14 in 7 days | |
| 19:17:10 | mriedem | dansmith: melwitt: can one of you put this through so i don't have to work it into my hypervisors uuid api series? https://review.openstack.org/#/c/471970/ | |
| 19:17:15 | mriedem | i have to build on that one | |
| 19:17:17 | mriedem | to change the type | |
| 19:17:33 | dansmith | done | |
| 19:17:37 | openstackgerrit | Corey Bryant proposed openstack/nova master: Make nova-placement-api default config less strict https://review.openstack.org/472378 | |
| 19:17:39 | mriedem | <3 | |
| 19:18:27 | mriedem | fleet week failed again, | |
| 19:18:30 | mriedem | but something different this time http://logs.openstack.org/94/436094/33/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/bed0e6a/console.html#_2017-06-08_18_39_03_929168 | |
| 19:19:40 | dansmith | o.O | |
| 19:21:10 | mordred | efried: https://review.openstack.org/#/q/project:openstack/keystoneauth+topic:version-discovery is updated from reviews if you wanna poke | |
| 19:21:25 | efried | mordred Roger wilco. | |
| 19:21:31 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Use uuid for id in Services API https://review.openstack.org/464280 | |
| 19:21:31 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Handle uuids in os-hypervisors API https://review.openstack.org/471899 | |
| 19:21:31 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Handle uuid in HostAPI.compute_node_get https://review.openstack.org/461532 | |
| 19:21:48 | mordred | efried: still missing interface lists and service type aliases - but we're getting real close | |
| 19:21:58 | efried | neat | |
| 19:22:59 | mriedem | dansmith: unrelated | |
| 19:23:12 | mriedem | the test is posting a password to the metadata server | |
| 19:23:13 | dansmith | good, because....... | |
| 19:23:18 | mriedem | and then checks to see it shows up | |
| 19:23:32 | mriedem | which seems super racy | |
| 19:23:51 | mriedem | that's totally async so tempest should probably be polling here | |
| 19:26:08 | mriedem | maybe i'm not sure, | |
| 19:26:15 | mriedem | it sets a password field in the instance sys meta | |
| 19:26:17 | mriedem | it's a POST, | |
| 19:26:24 | mriedem | but i'm not sure where the metadata service returns | |
| 19:26:34 | openstackgerrit | Merged openstack/nova master: api-ref: fix type for hypervisor_marker https://review.openstack.org/471970 | |
| 19:26:41 | kevinbenton | ihrachys: hey | |
| 19:28:07 | kevinbenton | ihrachys: looking at backlog. l2-agent never deletes a tap device, that's left up to nova | |
| 19:28:12 | kevinbenton | mriedem: ^^ | |
| 19:28:23 | kevinbenton | but it looks like you already found that | |
| 19:28:26 | kevinbenton | anything i can help with? | |
| 19:32:59 | mriedem | dansmith: i guess the metadata POST is synchronous | |
| 19:36:47 | mriedem | dansmith: aha http://logs.openstack.org/94/436094/33/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/bed0e6a/logs/screen-n-api.txt.gz#_Jun_08_18_36_06_549748 | |
| 19:36:53 | mriedem | the cache fs us again | |
| 19:37:25 | mriedem | we update inst.system_metadata, then tempest calls the rest api to get the password, and it looks like things are going through the metadata cache | |
| 19:37:35 | mriedem | not sure why we wouldn't update the cache when we update/set the password... | |
| 19:38:10 | dansmith | but that's still unrelated right? | |
| 19:38:11 | dansmith | but yeah, that metadata stuff needs fixin' | |
| 19:38:21 | mriedem | wtf, we're getting an InstanceNotFound | |
| 19:38:37 | mriedem | oooooo | |
| 19:38:47 | mriedem | it is cells | |
| 19:38:49 | mriedem | we're not targeted | |
| 19:38:50 | mriedem | mofo | |
| 19:38:57 | mriedem | opening a bug | |
| 19:39:36 | dansmith | we shouldn't be targeted | |
| 19:39:47 | mriedem | actually the instance should be targeted when we look it up | |
| 19:40:03 | mriedem | i mean, we look up the mapping and then target the instance don't we? | |
| 19:40:14 | dansmith | metadata should be a cell thing so it shouldn't be using target_cell, is my point | |
| 19:40:28 | dansmith | maybe that's the problem | |
| 19:40:44 | dansmith | does metadata run as a separate process in devstack? | |
| 19:41:01 | mriedem | not here, it's part of nova-api | |
| 19:41:25 | openstackgerrit | Merged openstack/nova master: Add docstring for test_limit_check_project_and_user_zero_values https://review.openstack.org/472287 | |
| 19:42:50 | openstack | Launchpad bug 1696843 in OpenStack Compute (nova) "metadata: POST /openstack/2013-10-17/password fails with InstanceNotFound" [Undecided,New] | |
| 19:42:50 | mriedem | https://bugs.launchpad.net/nova/+bug/1696843 | |
| 19:43:44 | mriedem | kevinbenton: i've got a debug patch up to see if i can recreate in a neutron patch and get some dumps of the related domain xml | |
| 19:43:55 | dansmith | mriedem: well, that's going to be a problem then | |
| 19:44:02 | mriedem | and ihar has a patch up with a revert of the lb tap change to see if that is any different | |
| 19:45:01 | mriedem | dansmith: right here https://github.com/openstack/nova/blob/b535c7432d02366f11fad7d1b1a3e3a8b8ac5415/nova/api/metadata/password.py#L70 | |
| 19:45:09 | mriedem | why wouldn't we lookup the instance mapping via that uuid | |
| 19:45:16 | mriedem | and then target the get_by_uuid to the cell | |
| 19:45:19 | dansmith | mriedem: because metadata is in the cell and we can't do that | |
| 19:45:29 | mriedem | i thought it was global | |
| 19:45:43 | mriedem | actually i don't think i've ever heard of anyone ever talk about the metadata service wrt cells v2 | |
| 19:46:10 | dansmith | not sure why it would be global, | |