| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-06-24 | |||
| 13:58:56 | dansmith | rouk: ouch :) | |
| 13:59:31 | rouk | nova needs some logging of its check cycle in debug mode though, having to trace from rabbitmq heartbeats as the symptom is rough | |
| 13:59:50 | sean-k-mooney | rouk: im sure that was fun to debug | |
| 14:00:00 | sean-k-mooney | it explains the behavior you were seeing however | |
| 14:00:35 | rouk | 4 days of man hours tracing through from the rabbitmq server because im always told "its not the network". | |
| 14:01:00 | sean-k-mooney | im still slightly annoyed/upsent/concerned that the nova-compute agent basicaly hard locks when it cant connect to ceph to get the capsity fo the ceph rbd pool | |
| 14:01:09 | rouk | so, can i wishlist some debug messages for the check phases in nova's cycle? lol | |
| 14:01:54 | rouk | yeah, it would send the message to ceph, and the ceph client retries every 30 seconds or so, and random dice based on what primary connection the LAPC uses determines if it returns or not, and it hangs forever till it gets a response. | |
| 14:02:01 | sean-k-mooney | hehe if you can suggest what they should be we can proably add them | |
| 14:03:11 | rouk | issue was only found tcpdumping ceph, would see a 9k byte response out of ceph and the client wouldnt hear about it. | |
| 14:03:46 | rouk | well... im not aware of all the checks, would be good if it was even just a basic debug on what function it was checking, just to narrow down debug. | |
| 14:04:17 | rouk | right now the messages stop pretty high up, so the only thing to check is everything. | |
| 14:04:49 | sean-k-mooney | im guessing it was working just well enough for the tcp connection not to drop | |
| 14:05:10 | sean-k-mooney | if we got a disconnect that would proably have shown up in the logs | |
| 14:05:12 | rouk | any tcp/udp/icmp with RESPONSES under 1500 worked perfectly 100% of the time. | |
| 14:05:42 | sean-k-mooney | ya im not sure nova could figure that our however. | |
| 14:05:51 | sean-k-mooney | rouk: did you file a bug for this by the way? | |
| 14:06:02 | rouk | nope, since its my fault. | |
| 14:06:29 | rouk | if nova would give up on the ceph client after even 20 seconds it would be fine imo | |
| 14:06:30 | sean-k-mooney | well we proably should be making that call with a timeout and raising an error if we dont get the resopce | |
| 14:06:57 | rouk | packet loss ceph client hangs for 30 seconds per retry or so, but if the backend is not responding for even over 1 second, somethings up. | |
| 14:07:27 | rouk | yeah, anything over 1 second is bad, and 20 seconds imo is an ERROR tier. | |
| 14:07:46 | sean-k-mooney | ya i think as a general enchancement haveing a config option to control a timout on ceph calls would likely be a good thing | |
| 14:08:14 | rouk | should i fill it out somewhere in particular? | |
| 14:09:05 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove comments about mirroring changes to nova/cells/messaging.py https://review.opendev.org/667107 | |
| 14:09:09 | sean-k-mooney | rouk: https://bugs.launchpad.net/nova/+filebug | |
| 14:10:07 | sean-k-mooney | rouk: if you just file a bug and state i asked you too then it wont get closed imediatly as a ceph issue :) | |
| 14:11:19 | sean-k-mooney | state something like "nova should timeout request to ceph, instead of hanging on network issues" and descibe the behavior you saw | |
| 14:11:51 | sean-k-mooney | mriedem: i dont know if we can fix it but at least we can sse if its sane or not and then close it :) | |
| 14:12:27 | mriedem | i was just joking | |
| 14:13:39 | rouk | sean-k-mooney: can do. | |
| 14:14:18 | sean-k-mooney | :) i was assuming that but i also need to go grab coffe so brb | |
| 14:14:20 | rouk | oh, launchpad, my ubuntuone cant use launchpad due to "oops" error, no response from support about it... guess ill make a new account or something. | |
| 14:18:59 | dansmith | mriedem: sean-k-mooney: so what's the deal? is there another patch to fix same host we're watching, or are we going to merge this, re-propose, etc? | |
| 14:19:25 | dansmith | I imagine neutron people want to know | |
| 14:21:17 | mriedem | dansmith: i said i think we should land the revert | |
| 14:21:28 | mriedem | the commit message could use a tweak to explain why the original is busted | |
| 14:21:34 | mriedem | but slaweq appears to be gone | |
| 14:21:44 | dansmith | mriedem: okay I saw you said you were "OK" with it, but didn't see that was the plan | |
| 14:21:51 | dansmith | so let's update it for him and approve? | |
| 14:21:57 | mriedem | yeah i can do that | |
| 14:22:15 | sean-k-mooney | dansmith: ya we are going to revert | |
| 14:23:11 | openstackgerrit | Slawek Kaplonski proposed openstack/nova master: Revert "Revert resize: wait for events according to hybrid plug" https://review.opendev.org/667035 | |
| 14:23:16 | sean-k-mooney | to be honest this give me more time to test this for the same host resize case too and the backport will be simpler since it will be 1 patch so it better in the log run | |
| 14:23:44 | dansmith | mriedem: ^ good? | |
| 14:24:18 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Revert "Revert resize: wait for events according to hybrid plug" https://review.opendev.org/667035 | |
| 14:24:23 | mriedem | d'oh | |
| 14:25:21 | mriedem | dansmith: +2 | |
| 14:26:00 | dansmith | okay cool | |
| 14:29:09 | slaweq | mriedem: hi, I just updated this commit message, sorry for the delay, I was on meeting | |
| 14:29:10 | mriedem | sean-k-mooney: i've marked the original bug as "triaged" rather than "fix released" so your patch can still close the original bug https://bugs.launchpad.net/nova/+bug/1832028 | |
| 14:29:11 | openstack | Launchpad bug 1832028 in OpenStack Compute (nova) stein "revert resize: vif-plugged external event sent too soon if Neutron is using OVS hybrid plug" [Medium,In progress] - Assigned to Lee Yarwood (lyarwood) | |
| 14:29:29 | mriedem | slaweq: yup me too :) the revert is approved | |
| 14:29:36 | slaweq | mriedem: thx a lot | |
| 14:29:49 | mriedem | slaweq: i'll just say again that the inconsistency of when neutron sends these events per vif type sucks | |
| 14:29:52 | mriedem | from a consumer pov | |
| 14:30:23 | slaweq | mriedem: yes, I know | |
| 14:30:49 | slaweq | we also found other issue with live-migration and e.g. networking-ovn recently but that will be reported separately later | |
| 14:31:08 | mriedem | i've heard (from the rh nova team) | |
| 14:31:17 | mriedem | which is why you changed to ovs hybrid plug and then found this issue | |
| 14:31:18 | sean-k-mooney | mriedem: ack, do i need to respond on the ml | |
| 14:32:01 | mriedem | sean-k-mooney: saying we approved the revert on the ML is probably good for people to hold their rechecks | |
| 14:33:06 | dansmith | promotion time? | |
| 14:33:30 | mriedem | i'm not the person to ask probably | |
| 14:33:44 | dansmith | sean-k-mooney: lol | |
| 14:33:48 | mriedem | it won't block nova of course | |
| 14:33:57 | mriedem | we're just blocked by the rate limiting | |
| 14:34:23 | sean-k-mooney | mriedem: on the upside since neutron are not using there gate currently nova should speed up :P | |
| 14:36:55 | zigo | Is there an openstackclient equivalent to nova server-migration-show? | |
| 14:37:08 | amorin | hey all | |
| 14:38:21 | sean-k-mooney | zigo: not that i see at a glance | |
| 14:38:41 | zigo | sean-k-mooney: Ok, so I'm not completely bind then! :) | |
| 14:39:06 | stephenfin | zigo: I don't think so. That was one of the gaps we identified in Denver (summit) | |
| 14:39:09 | sean-k-mooney | you are not :) we dont have full parity | |
| 14:39:36 | sean-k-mooney | zigo: initally osc did not have admin commnds so that is the area we have the most gaps today | |
| 14:39:46 | zigo | Right. | |
| 14:40:46 | zigo | If I put arp_responder = True, will this speed-up MAC learning for the network equipment connected to my compute? | |
| 14:41:07 | zigo | It usually takes dozens of seconds for it to learn what the new floating IP location is... | |
| 14:41:09 | sean-k-mooney | no | |
| 14:41:21 | zigo | sean-k-mooney: Is there some tricks? :) | |
| 14:42:04 | sean-k-mooney | arp_resopnder = true need the l2_population ml2 driver and does arp suppression so that openflow rules rewrite the arp request to an arp responce | |
| 14:42:22 | amorin | I am currently facing a race condition in my OpenStack deployment. If I delete a port while the instance was booting (before nova plug the interface), then nova plug the interface and left the interface like this, after a while I have a lot of interface staying on the host | |
| 14:42:22 | sean-k-mooney | so if you set that and the vswich has the flow installed the arp will never get to the tor | |
| 14:42:37 | zigo | Right. | |
| 14:43:09 | sean-k-mooney | so it will have the opisite effect in that learnign will only happen when you get an non arp packet | |
| 14:43:28 | rouk | sean-k-mooney: https://bugs.launchpad.net/nova/+bug/1834048 let me know if more is needed. | |
| 14:43:29 | openstack | Launchpad bug 1834048 in OpenStack Compute (nova) "Nova waits indefinitely on ceph client hangs due to network problems" [Undecided,New] | |
| 14:43:52 | zigo | sean-k-mooney: Ok, thanks. | |
| 14:44:10 | mriedem | amorin: like this? https://review.opendev.org/#/c/663382/ | |
| 14:44:27 | sean-k-mooney | rouk: cool thanks for filing ^ lyarwood you might be able to determing if there is soemthing we can do to fix the ceph issue | |
| 14:44:58 | sean-k-mooney | mriedem: hehe i was looking for that but it might help | |
| 14:45:23 | sean-k-mooney | mriedem: that is specifcaly for the case wehre you delete the instance after teh vm has started running but before we have updated the db | |
| 14:45:33 | sean-k-mooney | i belive there are other case were this can happen too | |
| 14:45:34 | mriedem | dansmith: regarding promoting that revert, slaweq could request it since i wouldn't be surprised if it takes all day to run the nova change through and even then will likely fail on some gate bug | |
| 14:45:44 | sean-k-mooney | that was the one i know was definetly broken | |
| 14:45:45 | amorin | mriedem, sean-k-mooney almost like this | |
| 14:46:09 | dansmith | mriedem: yeah that's what I was thinking.. looking at how many other things have -1s on them, this could take a while | |
| 14:46:17 | dansmith | slaweq: mriedem: is that neutron job in neutron's gate? | |
| 14:46:33 | mriedem | it's voting for neutron so i expect it's in neutron's gate queue yeah | |
| 14:46:34 | sean-k-mooney | dansmith: yes it is | |