| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-06-08 | |||
| 18:20:20 | ihrachys | kevinbenton, are you up already? | |
| 18:20:32 | mriedem | ihrachys: i mean this https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1450 | |
| 18:22:35 | ihrachys | which is empty for lb | |
| 18:23:51 | mriedem | ihrachys: no it's going here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/vif.py#L906 | |
| 18:24:19 | ihrachys | oh. I was looking at unplug in os-vif | |
| 18:24:24 | ihrachys | for lb plugin | |
| 18:24:45 | mriedem | yeah it's not using os-vif | |
| 18:24:57 | mriedem | because there is no os-vif tap conversion in nova yet | |
| 18:25:08 | mriedem | http://logs.openstack.org/63/471763/2/check/gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial/dccf187/logs/screen-n-cpu.txt.gz?level=DEBUG#_Jun_08_16_56_19_922445 | |
| 18:25:13 | mriedem | No conversion for VIF type tap yet | |
| 18:25:28 | mriedem | so nova does: sudo nova-rootwrap /etc/nova/rootwrap.conf ip link delete tap3e81295f-c1 | |
| 18:30:51 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: fix alternative_device_name for detaching interfaces https://review.openstack.org/472371 | |
| 18:31:10 | mriedem | ihrachys: i also noticed this comment which is probably from danpb https://github.com/openstack/nova/blob/master/nova/virt/libvirt/designer.py#L56 | |
| 18:31:21 | mriedem | but that's the method called to build the device config for this type of tap device | |
| 18:31:29 | ihrachys | btw I misinterpreted the logs before. I now see that neutron logs operations on the tap device at the same time nova starts the decorated func | |
| 18:32:39 | mriedem | this is all i see about it in the libvirt docs https://libvirt.org/formatdomain.html#elementsNICSEthernet | |
| 18:32:53 | mriedem | i assume it's discouraged because "Provides a means for the administrator to execute an arbitrary script to connect the guest's network to the LAN." | |
| 18:33:52 | mriedem | as far as i can tell though nova doesn't pass through any script path | |
| 18:34:05 | mriedem | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/designer.py#L69 | |
| 18:34:27 | mriedem | oh i think that means if not provided, it defaults to /etc/qemu-ifup | |
| 18:38:19 | mriedem | ihrachys: i'm going to push a change that catches the final exception, dumps a bunch of debug info to the logs, and then re-raises | |
| 18:38:33 | mriedem | to see what's actually in the guest xml when we fail compared to the device xml we're trying to remove | |
| 18:54:22 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: dump debug info when interface detach times out https://review.openstack.org/472377 | |
| 18:54:24 | mriedem | ihrachys: i'll push a neutron change that depends on this ^ to debug, i see you pushed a revert | |
| 18:56:11 | openstackgerrit | Corey Bryant proposed openstack/nova master: Make nova-placement-api default config less strict https://review.openstack.org/472378 | |
| 18:59:22 | cfriesen | if instances have attached volumes, that information would be in the block_device_mapping table of the nova db, right? | |
| 19:00:33 | mriedem | cfriesen: yes | |
| 19:05:12 | mriedem | we have an intermittent KeyError in functional tests https://bugs.launchpad.net/nova/+bug/1696834 | |
| 19:05:12 | openstack | Launchpad bug 1696834 in OpenStack Compute (nova) "Intermittent "KeyError: 'allocations'" in functional tests" [Medium,Confirmed] | |
| 19:05:31 | melwitt | yeah, seems to hit pretty often too | |
| 19:05:42 | mriedem | 14 in 7 days | |
| 19:17:10 | mriedem | dansmith: melwitt: can one of you put this through so i don't have to work it into my hypervisors uuid api series? https://review.openstack.org/#/c/471970/ | |
| 19:17:15 | mriedem | i have to build on that one | |
| 19:17:17 | mriedem | to change the type | |
| 19:17:33 | dansmith | done | |
| 19:17:37 | openstackgerrit | Corey Bryant proposed openstack/nova master: Make nova-placement-api default config less strict https://review.openstack.org/472378 | |
| 19:17:39 | mriedem | <3 | |
| 19:18:27 | mriedem | fleet week failed again, | |
| 19:18:30 | mriedem | but something different this time http://logs.openstack.org/94/436094/33/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/bed0e6a/console.html#_2017-06-08_18_39_03_929168 | |
| 19:19:40 | dansmith | o.O | |
| 19:21:10 | mordred | efried: https://review.openstack.org/#/q/project:openstack/keystoneauth+topic:version-discovery is updated from reviews if you wanna poke | |
| 19:21:25 | efried | mordred Roger wilco. | |
| 19:21:31 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Handle uuid in HostAPI.compute_node_get https://review.openstack.org/461532 | |
| 19:21:31 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Handle uuids in os-hypervisors API https://review.openstack.org/471899 | |
| 19:21:31 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Use uuid for id in Services API https://review.openstack.org/464280 | |
| 19:21:48 | mordred | efried: still missing interface lists and service type aliases - but we're getting real close | |
| 19:21:58 | efried | neat | |
| 19:22:59 | mriedem | dansmith: unrelated | |
| 19:23:12 | mriedem | the test is posting a password to the metadata server | |
| 19:23:13 | dansmith | good, because....... | |
| 19:23:18 | mriedem | and then checks to see it shows up | |
| 19:23:32 | mriedem | which seems super racy | |
| 19:23:51 | mriedem | that's totally async so tempest should probably be polling here | |
| 19:26:08 | mriedem | maybe i'm not sure, | |
| 19:26:15 | mriedem | it sets a password field in the instance sys meta | |
| 19:26:17 | mriedem | it's a POST, | |
| 19:26:24 | mriedem | but i'm not sure where the metadata service returns | |
| 19:26:34 | openstackgerrit | Merged openstack/nova master: api-ref: fix type for hypervisor_marker https://review.openstack.org/471970 | |
| 19:26:41 | kevinbenton | ihrachys: hey | |
| 19:28:07 | kevinbenton | ihrachys: looking at backlog. l2-agent never deletes a tap device, that's left up to nova | |
| 19:28:12 | kevinbenton | mriedem: ^^ | |
| 19:28:23 | kevinbenton | but it looks like you already found that | |
| 19:28:26 | kevinbenton | anything i can help with? | |
| 19:32:59 | mriedem | dansmith: i guess the metadata POST is synchronous | |
| 19:36:47 | mriedem | dansmith: aha http://logs.openstack.org/94/436094/33/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/bed0e6a/logs/screen-n-api.txt.gz#_Jun_08_18_36_06_549748 | |
| 19:36:53 | mriedem | the cache fs us again | |
| 19:37:25 | mriedem | we update inst.system_metadata, then tempest calls the rest api to get the password, and it looks like things are going through the metadata cache | |
| 19:37:35 | mriedem | not sure why we wouldn't update the cache when we update/set the password... | |
| 19:38:10 | dansmith | but that's still unrelated right? | |
| 19:38:11 | dansmith | but yeah, that metadata stuff needs fixin' | |
| 19:38:21 | mriedem | wtf, we're getting an InstanceNotFound | |
| 19:38:37 | mriedem | oooooo | |
| 19:38:47 | mriedem | it is cells | |
| 19:38:49 | mriedem | we're not targeted | |
| 19:38:50 | mriedem | mofo | |
| 19:38:57 | mriedem | opening a bug | |
| 19:39:36 | dansmith | we shouldn't be targeted | |
| 19:39:47 | mriedem | actually the instance should be targeted when we look it up | |
| 19:40:03 | mriedem | i mean, we look up the mapping and then target the instance don't we? | |
| 19:40:14 | dansmith | metadata should be a cell thing so it shouldn't be using target_cell, is my point | |
| 19:40:28 | dansmith | maybe that's the problem | |
| 19:40:44 | dansmith | does metadata run as a separate process in devstack? | |
| 19:41:01 | mriedem | not here, it's part of nova-api | |
| 19:41:25 | openstackgerrit | Merged openstack/nova master: Add docstring for test_limit_check_project_and_user_zero_values https://review.openstack.org/472287 | |
| 19:42:50 | mriedem | https://bugs.launchpad.net/nova/+bug/1696843 | |
| 19:42:50 | openstack | Launchpad bug 1696843 in OpenStack Compute (nova) "metadata: POST /openstack/2013-10-17/password fails with InstanceNotFound" [Undecided,New] | |
| 19:43:44 | mriedem | kevinbenton: i've got a debug patch up to see if i can recreate in a neutron patch and get some dumps of the related domain xml | |
| 19:43:55 | dansmith | mriedem: well, that's going to be a problem then | |
| 19:44:02 | mriedem | and ihar has a patch up with a revert of the lb tap change to see if that is any different | |
| 19:45:01 | mriedem | dansmith: right here https://github.com/openstack/nova/blob/b535c7432d02366f11fad7d1b1a3e3a8b8ac5415/nova/api/metadata/password.py#L70 | |
| 19:45:09 | mriedem | why wouldn't we lookup the instance mapping via that uuid | |
| 19:45:16 | mriedem | and then target the get_by_uuid to the cell | |
| 19:45:19 | dansmith | mriedem: because metadata is in the cell and we can't do that | |
| 19:45:29 | mriedem | i thought it was global | |
| 19:45:43 | mriedem | actually i don't think i've ever heard of anyone ever talk about the metadata service wrt cells v2 | |
| 19:46:10 | dansmith | not sure why it would be global, | |
| 19:46:11 | dansmith | but also, | |
| 19:46:19 | dansmith | not sure from a neutron perspective | |
| 19:46:28 | dansmith | I always think of the world like nova net multihost | |
| 19:46:40 | dansmith | maybe we can ask kevinbenton since he's here | |