Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-02-02
15:38:01 openstack Launchpad bug 1323658 in neutron "Nova resize/restart results in guest ending up in inconsistent state with Neutron" [Medium,Fix released] - Assigned to Dariusz Smigiel (smigiel-dariusz)
15:39:44 belmoreira melwitt mriedem we upgraded to Ocata yesterday and I'm observing that "memory_mb_used" in "compute_nodes" tables is not correct. However, the resourcetracker log in the hypervisor is correct.
15:39:52 belmoreira Is there a known bug that you are aware about this issue?
15:40:16 melwitt mriedem: nice sleuthing
15:41:14 mriedem melwitt: so in the resize case, we don't wait for vif-plugged events on the destination host because the vifs are already plugged on the source host, and the source host vifs aren't unplugged until the user confirms the resize and we cleanup the guest from the source
15:41:16 melwitt ihar said that if os-vif doesn't rebind the port, neutron would probably not be aware of the unplug/plug. I'm not sure whether it rebinds it
15:44:13 sean-k-mooney mriedem: so when os-vif adds a port we set the neurtorn port uuid in the external_ids field and wehn it get the notification fo the port added the neutron agent looks up the port and does the binding
15:44:32 sean-k-mooney melwitt: os-vif should not be binding the port port binding and port plugging are 2 different things
15:44:50 sean-k-mooney melwitt: you cannont call os-vif wiht an unbound port
15:45:07 melwitt thanks sean-k-mooney. I really need to read a book about this stuff
15:45:24 sean-k-mooney melwitt: if you do you proably will wish you had not :P
15:45:30 melwitt heh, fair
15:45:45 mriedem sean-k-mooney: it sounds like this behavior is different based on if you're using linuxbridge or not
15:46:45 sean-k-mooney mriedem: kind of. the general patteren is the same but some of the details are different
15:46:55 Spazmotic melwitt, you happen to have time to talk?
15:47:12 melwitt Spazmotic: that sounds mysterious
15:47:16 sean-k-mooney mriedem: for ovs neuton can subscrible to notification form the ovsdb for port add/remove events
15:47:26 Spazmotic Lol nah nah I assure you, just a quick conversation about nova related activities
15:47:39 sean-k-mooney melwitt: for linuxbridge i belive they have to poll
15:47:47 sean-k-mooney * mriedem ^
15:47:47 Spazmotic If you're too busy it's no rush :)
15:47:54 melwitt Spazmotic: go ahead
15:48:25 mriedem sean-k-mooney: really just trying to figure out if/why vif plug events from neutron might be different in the case of a hard reboot if you're using LB or OVS
15:48:40 mriedem since the wait logic on the nova side would suck if we had to know it changes per network backend
15:49:18 openstackgerrit Eric Fried proposed openstack/nova master: Use update_provider_tree from resource tracker https://review.openstack.org/520246
15:49:18 openstackgerrit Eric Fried proposed openstack/nova master: Fix nits in update_provider_tree series https://review.openstack.org/531260
15:49:19 openstackgerrit Eric Fried proposed openstack/nova master: Move refresh time from report client to prov tree https://review.openstack.org/535517
15:49:19 openstackgerrit Eric Fried proposed openstack/nova master: Make generation optional in ProviderTree https://review.openstack.org/539324
15:50:08 sean-k-mooney mriedem: if you do a hard reboot we will actully terminate the qemu process and remove the tap and re add it
15:50:23 sean-k-mooney if you are polling you may miss that
15:50:50 sean-k-mooney with ovs you will get the notifcation from the ovsdb for the add and remove
15:52:19 sean-k-mooney mriedem: is this all in relation to https://bugs.launchpad.net/nova/+bug/1323658
15:52:20 openstack Launchpad bug 1323658 in neutron "Nova resize/restart results in guest ending up in inconsistent state with Neutron" [Medium,Fix released] - Assigned to Dariusz Smigiel (smigiel-dariusz)
15:52:36 mriedem sean-k-mooney: no, https://review.openstack.org/#/c/540168
15:57:16 bauwser damn, interesting convo about vif pluging and neutron agents
15:57:24 giblet figleaf: Locally it raises for me if I pull the oslo ovo patch into my env (Depends-On: https://review.openstack.org/540344)
15:57:38 giblet figleaf: I mean it raised before the rebase
15:57:55 giblet figleaf: let me try with the new patch
15:58:22 bauwser mriedem: yeah, I guess the crux of the problem is how linuxbridge notifies the events
15:58:34 bauwser since that bug report is all about it
15:59:07 bauwser I mean for Quantum
15:59:08 sean-k-mooney welll so i think we are conflating too things here
15:59:16 sean-k-mooney vif pluging and vif binding
15:59:27 bauwser vif binding is the allocation, right?
15:59:28 sean-k-mooney vif plugging is the act of adding the tap to the backend
15:59:40 bauwser vif plugging is the fact to attach the vif to the guest ?
16:00:03 sean-k-mooney vif binding id that ack of the neutron ml2 direver populating the port details with a vif_type and vif_binding_details
16:00:25 mriedem this might be closer to the issue https://review.openstack.org/#/c/540168/1/nova/virt/libvirt/driver.py@5372
16:00:38 mriedem for one, we're operating on a stale cache to determine which vifs to wait for
16:00:46 mriedem b/c we never wait for the vifs to be unplugged and refresh the nw info cache
16:01:02 mriedem based on that, it's likely better to yes just ignore the vif plugged event,
16:01:19 mriedem in the server create case, we have the async network info wrapper that is waiting for the nw_info cache updates during port binding,
16:01:24 mriedem and then the virt driver spawn is waiting on those
16:01:40 mriedem so the nw info cache is up to date in the server create case, but we have nothing like that in the case of hard reboot
16:02:26 melwitt yeah, IIUC we get the network_info fresh at the beginning of a reboot and then that's it
16:02:33 sean-k-mooney mriedem: well one thing im not clear on. when we do a hard reboot do we redo the neutron port binding or jsut reuse the old one
16:02:43 mriedem re-use the existing one
16:03:38 sean-k-mooney well the neutron event is sent when neutron finish wirieing up a prot for the first time after portbinding. i dont know if neutron will send it again if we dont rebind
16:04:06 mriedem it must, otherwise this would be 100% fail in the gate
16:04:09 mriedem what we have is a race right now
16:04:25 sean-k-mooney mriedem: oh thats good to know
16:05:37 mriedem melwitt: i think we should probably not copy the comment from finish_migration
16:06:11 mriedem i bet this is primarily an issue based on (1) which network backend you're using and (2) the fact we're operating on a stale nw info cache, which is different from how spawn() works and waits for vif plug events
16:06:20 bauwser mriedem: melwitt: good point, the nw_info is only accurate when we call hard_reboot, not when it comes to _create_domain_and_nw()
16:06:24 melwitt okay
16:06:26 mriedem and therefore we should assume the vifs are already plugged and we don't need to wait for them
16:06:43 mriedem if we wanted to be thorough,
16:07:00 mriedem we'd (1) wait for the vif-unplugged event, and then (2) refresh the nw-info cache before moving on to plug the vifs
16:07:08 bauwser right
16:07:27 mriedem i'm not advocating that we do that in this patch, since we have to backport this
16:07:34 mriedem melwitt: so if you wanted to just update the comments, i'd be cool with that
16:07:56 bauwser fair enough
16:07:56 melwitt I was scrutinizing the logs of a failed run yesterday
16:10:10 sean-k-mooney mriedem: when you say the vif are already plugged you mean from a neutron point of view correct. not that the vif is added to the backend which is done by os-vif
16:10:50 mriedem sean-k-mooney: i'm talking about this vifs_already_plugged kwarg in the libvirt driver code
16:11:16 openstackgerrit Eric Fried proposed openstack/nova master: New-style _set_inventory_for_provider https://review.openstack.org/537648
16:11:17 openstackgerrit Eric Fried proposed openstack/nova master: SchedulerReportClient.update_from_provider_tree https://review.openstack.org/533821
16:11:18 openstackgerrit Eric Fried proposed openstack/nova master: Use update_provider_tree from resource tracker https://review.openstack.org/520246
16:11:19 openstackgerrit Eric Fried proposed openstack/nova master: Fix nits in update_provider_tree series https://review.openstack.org/531260
16:11:19 openstackgerrit Eric Fried proposed openstack/nova master: Move refresh time from report client to prov tree https://review.openstack.org/535517
16:11:20 openstackgerrit Eric Fried proposed openstack/nova master: Make generation optional in ProviderTree https://review.openstack.org/539324
16:11:21 sean-k-mooney mriedem: yes but im trying to figure out if this opens a security hole
16:11:46 sean-k-mooney mriedem: if that arge is set to True we will not wait for neutron
16:12:00 bauwser sean-k-mooney: well, all other server actions do the same, y'kno<
16:12:05 bauwser know*
16:12:14 sean-k-mooney os-vif will add the port when we call plug
16:12:32 bauwser sean-k-mooney: we're only able to correctly wait for Neutron in the boot case only because we have a fresh network info
16:12:37 sean-k-mooney but we wait for neutron to make sure the firewal rules are active on the port before unpasing the vm
16:12:45 bauwser fixing that in other server action calls would require far more
16:13:04 bauwser we == os-vif ?
16:13:15 sean-k-mooney we == nova
16:13:29 melwitt mriedem: he's the run I was looking at yesterday. there are no network-vif-unplugged events for the instance for the failed test (the hard reboot) http://logs.openstack.org/42/525842/11/check/neutron-tempest-linuxbridge/2502b64/logs/screen-q-svc.txt.gz#_Jan_29_16_01_28_170964
16:13:30 sean-k-mooney os-vif does not talk to neutron ever
16:13:53 melwitt I linked to a network-changed but if you look further in the log there are no unplug events
16:14:14 melwitt so I don't expect things would work if we tried to wait for an unplug event because neutron never sends one
16:16:14 sean-k-mooney melwitt: correct but wen we delete the tap any iprules create by the neutron security group driver are removed. if neutron never sees the device is unplugged and plugged again i dont think it will ever recreate them
16:18:18 melwitt sean-k-mooney: os-vif unplug deletes the tap, is what you are saying?
16:18:22 mriedem giblet: thanks for reviewing the nova-multiattach job change, replies inline https://review.openstack.org/#/c/532689/
16:18:51 sean-k-mooney melwitt: actully qemu doing a hard reboot does but os-vif will also remove it form the linux bridge
16:19:06 sean-k-mooney or ovs bridge depending on your life choices

Earlier   Later