| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-06-24 | |||
| 13:25:16 | openstackgerrit | Merged openstack/os-traits master: add GOP, NONE and VIRTIO grapics models https://review.opendev.org/665830 | |
| 13:33:44 | openstackgerrit | Merged openstack/nova master: Remove redundant group host setup https://review.opendev.org/665954 | |
| 13:33:53 | openstackgerrit | Merged openstack/nova master: [FUP] fix backleveling unit test for video models https://review.opendev.org/666919 | |
| 13:34:03 | openstackgerrit | Merged openstack/nova master: Mention [cinder]/cross_az_attach in the AZ docs https://review.opendev.org/650456 | |
| 13:41:01 | mriedem | sean-k-mooney: so same host revert resize huh, who know neutron had a gate job for ovs hybrid plug... | |
| 13:41:12 | mriedem | we should probably throw that into nova's experimental queue to be run on-demand | |
| 13:41:31 | mriedem | s/should/could/ since that means someone would have to remember to run it | |
| 13:44:00 | sean-k-mooney | ya we should | |
| 13:45:45 | efried_pto | sean-k-mooney: see question above about libvirt-video-device-models? | |
| 13:46:50 | efried_pto | oh | |
| 13:47:18 | sean-k-mooney | sorry give me 5 mins | |
| 13:47:35 | efried | sure, no hurry. Meanwhile, I'll hammer you in -placement as well. | |
| 13:49:56 | mriedem | sean is also working on pushing a patch to try and fix the neutron gate, | |
| 13:50:04 | mriedem | unless we just go with this revert https://review.opendev.org/#/c/667035/2 | |
| 13:50:18 | mriedem | slaweq: looks like ^ makes the job pass now, i checked zuul on your neutron change that depends on the nova revrt | |
| 13:50:21 | mriedem | *Revert | |
| 13:51:26 | efried | that's way more important than my blueprint paperwork | |
| 13:51:41 | mriedem | slaweq: what i'd like to know is do we still get a network-vif-plugged event for ovs hybrid plug ports during same host revert resize at the time of plugging the vif (because apparently we don't when binding the port and the host doesn't change - which is different in behavior from how non-hybrid plug ovs ports work apparently) | |
| 13:51:58 | efried | (</ptl-hat> for pretty much any value of $that) | |
| 13:52:05 | sean-k-mooney | mriedem: actully we had a production outage of our internal cloud and i just spent the last 2 hours helping them fix it | |
| 13:52:32 | mriedem | sean-k-mooney: please oh please say it was because of this ovs hybrid plug patch :) | |
| 13:53:12 | sean-k-mooney | mriedem: so im about to start on a patch to fix the neutron issue but ill leave it up to you do we want to revert it and merge a fixed patch later or merge a fix and wait form me to write it | |
| 13:53:47 | sean-k-mooney | mriedem: no it was releated to messing up the neutorn policy.json that currpted the network info_cache and all vms nolonger had interfaces | |
| 13:54:14 | mriedem | sean-k-mooney: aha that sounds familiar (the same thing ovh fixed in stein) | |
| 13:54:24 | sean-k-mooney | yep | |
| 13:54:30 | mriedem | apparently screwing up your neutron policy and blasting the nova info cache is something a lot of people do... | |
| 13:54:30 | sean-k-mooney | we have not backported it to 13 yet | |
| 13:55:12 | sean-k-mooney | ya so i just figured out what db surgery is need and our devops folks are now doing it | |
| 13:55:33 | sean-k-mooney | so do we want to fast revert the event change | |
| 13:55:47 | mriedem | sean-k-mooney: for artom's patch i think we likely should land the revert and then work the same host thing into a combined patch | |
| 13:55:51 | mriedem | since you guys want to backport this | |
| 13:55:54 | sean-k-mooney | or will i write a patch quickly to skip waiting for event for same host resize | |
| 13:56:28 | mriedem | reverting gives time to sort out why same host resize events are different for this vif type than non-hybrid plug ovs - that inconsistency is really screwy | |
| 13:56:29 | sean-k-mooney | ok im fine with that i just didnt want to cause more revew burden for you/others | |
| 13:56:51 | mriedem | slaweq's revert patch's commit message likely needs work | |
| 13:57:10 | rouk | sean-k-mooney: i found the ceph issue and it fixed the nova issue, it was an MTU someone thought would be fun to drop to 1500 on a single switch of the 2 switches doing LACP to that host. | |
| 13:57:40 | rouk | so some ceph responses would just fail, cause 9k byte, and then nova would sit forever silently | |
| 13:58:56 | dansmith | rouk: ouch :) | |
| 13:59:31 | rouk | nova needs some logging of its check cycle in debug mode though, having to trace from rabbitmq heartbeats as the symptom is rough | |
| 13:59:50 | sean-k-mooney | rouk: im sure that was fun to debug | |
| 14:00:00 | sean-k-mooney | it explains the behavior you were seeing however | |
| 14:00:35 | rouk | 4 days of man hours tracing through from the rabbitmq server because im always told "its not the network". | |
| 14:01:00 | sean-k-mooney | im still slightly annoyed/upsent/concerned that the nova-compute agent basicaly hard locks when it cant connect to ceph to get the capsity fo the ceph rbd pool | |
| 14:01:09 | rouk | so, can i wishlist some debug messages for the check phases in nova's cycle? lol | |
| 14:01:54 | rouk | yeah, it would send the message to ceph, and the ceph client retries every 30 seconds or so, and random dice based on what primary connection the LAPC uses determines if it returns or not, and it hangs forever till it gets a response. | |
| 14:02:01 | sean-k-mooney | hehe if you can suggest what they should be we can proably add them | |
| 14:03:11 | rouk | issue was only found tcpdumping ceph, would see a 9k byte response out of ceph and the client wouldnt hear about it. | |
| 14:03:46 | rouk | well... im not aware of all the checks, would be good if it was even just a basic debug on what function it was checking, just to narrow down debug. | |
| 14:04:17 | rouk | right now the messages stop pretty high up, so the only thing to check is everything. | |
| 14:04:49 | sean-k-mooney | im guessing it was working just well enough for the tcp connection not to drop | |
| 14:05:10 | sean-k-mooney | if we got a disconnect that would proably have shown up in the logs | |
| 14:05:12 | rouk | any tcp/udp/icmp with RESPONSES under 1500 worked perfectly 100% of the time. | |
| 14:05:42 | sean-k-mooney | ya im not sure nova could figure that our however. | |
| 14:05:51 | sean-k-mooney | rouk: did you file a bug for this by the way? | |
| 14:06:02 | rouk | nope, since its my fault. | |
| 14:06:29 | rouk | if nova would give up on the ceph client after even 20 seconds it would be fine imo | |
| 14:06:30 | sean-k-mooney | well we proably should be making that call with a timeout and raising an error if we dont get the resopce | |
| 14:06:57 | rouk | packet loss ceph client hangs for 30 seconds per retry or so, but if the backend is not responding for even over 1 second, somethings up. | |
| 14:07:27 | rouk | yeah, anything over 1 second is bad, and 20 seconds imo is an ERROR tier. | |
| 14:07:46 | sean-k-mooney | ya i think as a general enchancement haveing a config option to control a timout on ceph calls would likely be a good thing | |
| 14:08:14 | rouk | should i fill it out somewhere in particular? | |
| 14:09:05 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove comments about mirroring changes to nova/cells/messaging.py https://review.opendev.org/667107 | |
| 14:09:09 | sean-k-mooney | rouk: https://bugs.launchpad.net/nova/+filebug | |
| 14:10:07 | sean-k-mooney | rouk: if you just file a bug and state i asked you too then it wont get closed imediatly as a ceph issue :) | |
| 14:11:19 | sean-k-mooney | state something like "nova should timeout request to ceph, instead of hanging on network issues" and descibe the behavior you saw | |
| 14:11:51 | sean-k-mooney | mriedem: i dont know if we can fix it but at least we can sse if its sane or not and then close it :) | |
| 14:12:27 | mriedem | i was just joking | |
| 14:13:39 | rouk | sean-k-mooney: can do. | |
| 14:14:18 | sean-k-mooney | :) i was assuming that but i also need to go grab coffe so brb | |
| 14:14:20 | rouk | oh, launchpad, my ubuntuone cant use launchpad due to "oops" error, no response from support about it... guess ill make a new account or something. | |
| 14:18:59 | dansmith | mriedem: sean-k-mooney: so what's the deal? is there another patch to fix same host we're watching, or are we going to merge this, re-propose, etc? | |
| 14:19:25 | dansmith | I imagine neutron people want to know | |
| 14:21:17 | mriedem | dansmith: i said i think we should land the revert | |
| 14:21:28 | mriedem | the commit message could use a tweak to explain why the original is busted | |
| 14:21:34 | mriedem | but slaweq appears to be gone | |
| 14:21:44 | dansmith | mriedem: okay I saw you said you were "OK" with it, but didn't see that was the plan | |
| 14:21:51 | dansmith | so let's update it for him and approve? | |
| 14:21:57 | mriedem | yeah i can do that | |
| 14:22:15 | sean-k-mooney | dansmith: ya we are going to revert | |
| 14:23:11 | openstackgerrit | Slawek Kaplonski proposed openstack/nova master: Revert "Revert resize: wait for events according to hybrid plug" https://review.opendev.org/667035 | |
| 14:23:16 | sean-k-mooney | to be honest this give me more time to test this for the same host resize case too and the backport will be simpler since it will be 1 patch so it better in the log run | |
| 14:23:44 | dansmith | mriedem: ^ good? | |
| 14:24:18 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Revert "Revert resize: wait for events according to hybrid plug" https://review.opendev.org/667035 | |
| 14:24:23 | mriedem | d'oh | |
| 14:25:21 | mriedem | dansmith: +2 | |
| 14:26:00 | dansmith | okay cool | |
| 14:29:09 | slaweq | mriedem: hi, I just updated this commit message, sorry for the delay, I was on meeting | |
| 14:29:10 | mriedem | sean-k-mooney: i've marked the original bug as "triaged" rather than "fix released" so your patch can still close the original bug https://bugs.launchpad.net/nova/+bug/1832028 | |
| 14:29:11 | openstack | Launchpad bug 1832028 in OpenStack Compute (nova) stein "revert resize: vif-plugged external event sent too soon if Neutron is using OVS hybrid plug" [Medium,In progress] - Assigned to Lee Yarwood (lyarwood) | |
| 14:29:29 | mriedem | slaweq: yup me too :) the revert is approved | |
| 14:29:36 | slaweq | mriedem: thx a lot | |
| 14:29:49 | mriedem | slaweq: i'll just say again that the inconsistency of when neutron sends these events per vif type sucks | |
| 14:29:52 | mriedem | from a consumer pov | |
| 14:30:23 | slaweq | mriedem: yes, I know | |
| 14:30:49 | slaweq | we also found other issue with live-migration and e.g. networking-ovn recently but that will be reported separately later | |
| 14:31:08 | mriedem | i've heard (from the rh nova team) | |
| 14:31:17 | mriedem | which is why you changed to ovs hybrid plug and then found this issue | |
| 14:31:18 | sean-k-mooney | mriedem: ack, do i need to respond on the ml | |
| 14:32:01 | mriedem | sean-k-mooney: saying we approved the revert on the ML is probably good for people to hold their rechecks | |
| 14:33:06 | dansmith | promotion time? | |