| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-06-24 | |||
| 14:06:57 | rouk | packet loss ceph client hangs for 30 seconds per retry or so, but if the backend is not responding for even over 1 second, somethings up. | |
| 14:07:27 | rouk | yeah, anything over 1 second is bad, and 20 seconds imo is an ERROR tier. | |
| 14:07:46 | sean-k-mooney | ya i think as a general enchancement haveing a config option to control a timout on ceph calls would likely be a good thing | |
| 14:08:14 | rouk | should i fill it out somewhere in particular? | |
| 14:09:05 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove comments about mirroring changes to nova/cells/messaging.py https://review.opendev.org/667107 | |
| 14:09:09 | sean-k-mooney | rouk: https://bugs.launchpad.net/nova/+filebug | |
| 14:10:07 | sean-k-mooney | rouk: if you just file a bug and state i asked you too then it wont get closed imediatly as a ceph issue :) | |
| 14:11:19 | sean-k-mooney | state something like "nova should timeout request to ceph, instead of hanging on network issues" and descibe the behavior you saw | |
| 14:11:51 | sean-k-mooney | mriedem: i dont know if we can fix it but at least we can sse if its sane or not and then close it :) | |
| 14:12:27 | mriedem | i was just joking | |
| 14:13:39 | rouk | sean-k-mooney: can do. | |
| 14:14:18 | sean-k-mooney | :) i was assuming that but i also need to go grab coffe so brb | |
| 14:14:20 | rouk | oh, launchpad, my ubuntuone cant use launchpad due to "oops" error, no response from support about it... guess ill make a new account or something. | |
| 14:18:59 | dansmith | mriedem: sean-k-mooney: so what's the deal? is there another patch to fix same host we're watching, or are we going to merge this, re-propose, etc? | |
| 14:19:25 | dansmith | I imagine neutron people want to know | |
| 14:21:17 | mriedem | dansmith: i said i think we should land the revert | |
| 14:21:28 | mriedem | the commit message could use a tweak to explain why the original is busted | |
| 14:21:34 | mriedem | but slaweq appears to be gone | |
| 14:21:44 | dansmith | mriedem: okay I saw you said you were "OK" with it, but didn't see that was the plan | |
| 14:21:51 | dansmith | so let's update it for him and approve? | |
| 14:21:57 | mriedem | yeah i can do that | |
| 14:22:15 | sean-k-mooney | dansmith: ya we are going to revert | |
| 14:23:11 | openstackgerrit | Slawek Kaplonski proposed openstack/nova master: Revert "Revert resize: wait for events according to hybrid plug" https://review.opendev.org/667035 | |
| 14:23:16 | sean-k-mooney | to be honest this give me more time to test this for the same host resize case too and the backport will be simpler since it will be 1 patch so it better in the log run | |
| 14:23:44 | dansmith | mriedem: ^ good? | |
| 14:24:18 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Revert "Revert resize: wait for events according to hybrid plug" https://review.opendev.org/667035 | |
| 14:24:23 | mriedem | d'oh | |
| 14:25:21 | mriedem | dansmith: +2 | |
| 14:26:00 | dansmith | okay cool | |
| 14:29:09 | slaweq | mriedem: hi, I just updated this commit message, sorry for the delay, I was on meeting | |
| 14:29:10 | mriedem | sean-k-mooney: i've marked the original bug as "triaged" rather than "fix released" so your patch can still close the original bug https://bugs.launchpad.net/nova/+bug/1832028 | |
| 14:29:11 | openstack | Launchpad bug 1832028 in OpenStack Compute (nova) stein "revert resize: vif-plugged external event sent too soon if Neutron is using OVS hybrid plug" [Medium,In progress] - Assigned to Lee Yarwood (lyarwood) | |
| 14:29:29 | mriedem | slaweq: yup me too :) the revert is approved | |
| 14:29:36 | slaweq | mriedem: thx a lot | |
| 14:29:49 | mriedem | slaweq: i'll just say again that the inconsistency of when neutron sends these events per vif type sucks | |
| 14:29:52 | mriedem | from a consumer pov | |
| 14:30:23 | slaweq | mriedem: yes, I know | |
| 14:30:49 | slaweq | we also found other issue with live-migration and e.g. networking-ovn recently but that will be reported separately later | |
| 14:31:08 | mriedem | i've heard (from the rh nova team) | |
| 14:31:17 | mriedem | which is why you changed to ovs hybrid plug and then found this issue | |
| 14:31:18 | sean-k-mooney | mriedem: ack, do i need to respond on the ml | |
| 14:32:01 | mriedem | sean-k-mooney: saying we approved the revert on the ML is probably good for people to hold their rechecks | |
| 14:33:06 | dansmith | promotion time? | |
| 14:33:30 | mriedem | i'm not the person to ask probably | |
| 14:33:44 | dansmith | sean-k-mooney: lol | |
| 14:33:48 | mriedem | it won't block nova of course | |
| 14:33:57 | mriedem | we're just blocked by the rate limiting | |
| 14:34:23 | sean-k-mooney | mriedem: on the upside since neutron are not using there gate currently nova should speed up :P | |
| 14:36:55 | zigo | Is there an openstackclient equivalent to nova server-migration-show? | |
| 14:37:08 | amorin | hey all | |
| 14:38:21 | sean-k-mooney | zigo: not that i see at a glance | |
| 14:38:41 | zigo | sean-k-mooney: Ok, so I'm not completely bind then! :) | |
| 14:39:06 | stephenfin | zigo: I don't think so. That was one of the gaps we identified in Denver (summit) | |
| 14:39:09 | sean-k-mooney | you are not :) we dont have full parity | |
| 14:39:36 | sean-k-mooney | zigo: initally osc did not have admin commnds so that is the area we have the most gaps today | |
| 14:39:46 | zigo | Right. | |
| 14:40:46 | zigo | If I put arp_responder = True, will this speed-up MAC learning for the network equipment connected to my compute? | |
| 14:41:07 | zigo | It usually takes dozens of seconds for it to learn what the new floating IP location is... | |
| 14:41:09 | sean-k-mooney | no | |
| 14:41:21 | zigo | sean-k-mooney: Is there some tricks? :) | |
| 14:42:04 | sean-k-mooney | arp_resopnder = true need the l2_population ml2 driver and does arp suppression so that openflow rules rewrite the arp request to an arp responce | |
| 14:42:22 | sean-k-mooney | so if you set that and the vswich has the flow installed the arp will never get to the tor | |
| 14:42:22 | amorin | I am currently facing a race condition in my OpenStack deployment. If I delete a port while the instance was booting (before nova plug the interface), then nova plug the interface and left the interface like this, after a while I have a lot of interface staying on the host | |
| 14:42:37 | zigo | Right. | |
| 14:43:09 | sean-k-mooney | so it will have the opisite effect in that learnign will only happen when you get an non arp packet | |
| 14:43:28 | rouk | sean-k-mooney: https://bugs.launchpad.net/nova/+bug/1834048 let me know if more is needed. | |
| 14:43:29 | openstack | Launchpad bug 1834048 in OpenStack Compute (nova) "Nova waits indefinitely on ceph client hangs due to network problems" [Undecided,New] | |
| 14:43:52 | zigo | sean-k-mooney: Ok, thanks. | |
| 14:44:10 | mriedem | amorin: like this? https://review.opendev.org/#/c/663382/ | |
| 14:44:27 | sean-k-mooney | rouk: cool thanks for filing ^ lyarwood you might be able to determing if there is soemthing we can do to fix the ceph issue | |
| 14:44:58 | sean-k-mooney | mriedem: hehe i was looking for that but it might help | |
| 14:45:23 | sean-k-mooney | mriedem: that is specifcaly for the case wehre you delete the instance after teh vm has started running but before we have updated the db | |
| 14:45:33 | sean-k-mooney | i belive there are other case were this can happen too | |
| 14:45:34 | mriedem | dansmith: regarding promoting that revert, slaweq could request it since i wouldn't be surprised if it takes all day to run the nova change through and even then will likely fail on some gate bug | |
| 14:45:44 | sean-k-mooney | that was the one i know was definetly broken | |
| 14:45:45 | amorin | mriedem, sean-k-mooney almost like this | |
| 14:46:09 | dansmith | mriedem: yeah that's what I was thinking.. looking at how many other things have -1s on them, this could take a while | |
| 14:46:17 | dansmith | slaweq: mriedem: is that neutron job in neutron's gate? | |
| 14:46:33 | mriedem | it's voting for neutron so i expect it's in neutron's gate queue yeah | |
| 14:46:34 | sean-k-mooney | dansmith: yes it is | |
| 14:47:02 | dansmith | so anything that has a recent check on it will head to the gate but fail every time, making the gate queue super slow too yeah? | |
| 14:47:05 | amorin | mriedem: something like this: | |
| 14:47:07 | amorin | http://paste.openstack.org/show/753313/ | |
| 14:47:11 | sean-k-mooney | amorin: right in your case you are deleteing the neutorn port | |
| 14:47:19 | amorin | yes, while booting the instance | |
| 14:48:00 | sean-k-mooney | well you should not actully get stale port in that case in ovs | |
| 14:48:11 | mriedem | amorin: so nova plugs the vif, starts waiting for the network-vif-plugged event, the port is then deleted and we don't get the event so we timeout and fail the server build | |
| 14:48:17 | sean-k-mooney | in you logg you have the vif unplug which should clean it up | |
| 14:48:38 | mriedem | sean-k-mooney: pretty sure we don't handle vif unplug and actually unplug | |
| 14:48:44 | mriedem | from out of band network-vif-deleted events | |
| 14:48:57 | sean-k-mooney | mriedem: we dont but we do on instance delete | |
| 14:49:17 | sean-k-mooney | so when we role back the spawn we should call unplug and clean up | |
| 14:49:29 | amorin | I dont have unplug call | |
| 14:49:35 | mriedem | isn't that stephen's patch? | |
| 14:49:53 | sean-k-mooney | no his is for a specifc case were we get an exception form the db | |
| 14:50:02 | sean-k-mooney | in that case we dont call cleanup | |
| 14:50:10 | sean-k-mooney | for the timeout case we do | |
| 14:50:21 | mriedem | https://review.opendev.org/#/c/663382/3/nova/compute/manager.py@2023 | |
| 14:50:52 | mriedem | stephen's patch doesn't have any specific cases in it from what i can tell | |
| 14:51:07 | sean-k-mooney | the downstream bug did | |