| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-06-05 | |||
| 18:48:06 | sean-k-mooney | well not quite | |
| 18:48:24 | artom | And resize is special in that it leaves in the instance paused on the source until confirm or revert | |
| 18:48:39 | dansmith | no, it's not paused on the source | |
| 18:48:39 | sean-k-mooney | some network backeds only send events on port bind and we dont bind the ports on hard reboot or rebuild | |
| 18:48:42 | dansmith | it's off | |
| 18:48:43 | artom | So.. how would that be applicable for evacuate, when the instance was never on the dest to begin with? | |
| 18:48:51 | artom | dansmith, sorry, off (aka destoryed) | |
| 18:49:28 | artom | sean-k-mooney, ah, so some backends might send the event as soon as we change the port binding to a host, regardless if it's wired or not? | |
| 18:49:44 | sean-k-mooney | artom: yes | |
| 18:49:51 | artom | I'd say that's a bug in those backends | |
| 18:50:00 | sean-k-mooney | it was by design | |
| 18:50:06 | artom | It shouldn't be sending the event if the thing isn't actually plugged/wired | |
| 18:50:39 | sean-k-mooney | not all networking backend can support that or at least could back when this was firt intoduced | |
| 18:50:58 | artom | That doesn't invalidate my argument :) | |
| 18:51:08 | sean-k-mooney | odl took 2 years to catch up and intoduce a way for odl to send events to neutron which neutron could send to nova | |
| 18:51:14 | artom | The idea was for Nova to be sure we have networking for instance by the time it spawns | |
| 18:51:17 | artom | So things like DHCP work | |
| 18:51:27 | artom | If we get the event before that's happened, that's not for Nova to fix | |
| 18:51:49 | sean-k-mooney | yep i had this argument with the odl folk back in 2014 | |
| 18:51:53 | artom | Because we'd be acting like networking is ready, when in reality it isn't | |
| 18:52:01 | artom | (yey, 5 years late) | |
| 18:52:39 | sean-k-mooney | for what its worth im pretty sure netorking-aci still does not send it when the port is wireded up an send it on bind | |
| 18:52:57 | sean-k-mooney | and ovn is not doing the right thing either in all cases | |
| 18:52:58 | artom | The Cisco thing? | |
| 18:53:02 | sean-k-mooney | yes | |
| 18:53:14 | artom | That's just one more argument for people not to buy Cisco ;) | |
| 18:53:36 | mriedem | "IIRC, mriedem (and maybe others) really want to keep the "can keep it on the source if it's not too late" behavior" | |
| 18:53:40 | mriedem | i have no opinion on that | |
| 18:53:42 | mriedem | that i'm aware of | |
| 18:54:14 | mriedem | i'm also assuming i don't want to cruft up an already complicated change to cover other cases in the same patch if we're doing backports | |
| 18:54:23 | mriedem | b/c there is a >0% chance this will introduce a regression somehow | |
| 18:54:27 | mriedem | b/c this is all very brittle sounding | |
| 18:54:30 | mriedem | and super duper latent | |
| 18:55:12 | mriedem | how would one even recreate this hybrid ovs resize thing in the gate? configure devstack for that backend (is that easy?) and then put a sleep in the nova code between migrate_instance_start(finish?) and the call to the virt driver? | |
| 18:55:16 | artom | Like osteoporosis | |
| 18:56:14 | sean-k-mooney | mriedem: we have patches to recreate. you configure ovs of iptables and then we can race on teh revert tests | |
| 18:56:39 | sean-k-mooney | *ovs to use the iptables firewall driver | |
| 18:56:53 | artom | https://review.opendev.org/#/c/660782/5 specifically | |
| 18:57:59 | sean-k-mooney | our downstream ci default to iptables firewall still on older releases so that is why we are seeing it downstream. upstream use the openvswtich contrack firewall driver by default | |
| 18:58:53 | mriedem | why does https://review.opendev.org/#/c/653498/ come after the fix in the series? | |
| 18:59:27 | artom | mriedem, because without hybrid plug it's actually irrelevant to the fix :/ | |
| 18:59:32 | artom | Well | |
| 18:59:39 | artom | I suppose it tests the non-hybrid-plug case | |
| 19:00:12 | mriedem | correct | |
| 19:00:30 | mriedem | you said in the ML it was the only job we run that is multi-node for the resize revert stuff right? | |
| 19:00:39 | mriedem | or would be | |
| 19:00:57 | artom | Yeah | |
| 19:01:00 | artom | I can move it below | |
| 19:02:16 | mriedem | you know, | |
| 19:02:23 | sean-k-mooney | artom: actuly i think you used to have it below and inverted it in a recent revision | |
| 19:02:28 | mriedem | test_server_connectivity_cold_migration_revert in the tempest-slow-py3 job would run resize revert in a multi-node job, | |
| 19:02:31 | mriedem | if you removed the skip on the test | |
| 19:02:38 | mriedem | and the test is skipped b/c of https://bugs.launchpad.net/nova/+bug/1788403 | |
| 19:02:40 | openstack | Launchpad bug 1788403 in OpenStack Compute (nova) "test_server_connectivity_cold_migration_revert randomly fails ssh check" [Medium,Fix released] - Assigned to Matt Riedemann (mriedem) | |
| 19:02:46 | mriedem | which sounds very familiar to your issue yeah? | |
| 19:03:13 | artom | It would, except again, upstream gate doesn't use hybrid plug, so it's a different root cause | |
| 19:03:19 | dansmith | mriedem: you emotionally scarred me with discussion of that opinion in the past, which is why I remember it | |
| 19:04:06 | dansmith | mriedem: and yeah, not saying fixing the other operations needs to be in this patch, but if we're undoing not only this wait, but also making previous ones (like the hard reboot) no longer accurate, I think fixing them all in the same way (re-using helpers if they apply, etc) is a good idea | |
| 19:04:06 | mriedem | dansmith: logs or it never happened! | |
| 19:04:21 | dansmith | mriedem: I can show you the scars. those' | |
| 19:04:24 | dansmith | re good enough yeah? | |
| 19:04:34 | mriedem | pictures or it didn't happen! | |
| 19:05:08 | artom | Faxes? Damn boomers | |
| 19:06:41 | dansmith | I know you didn't just confuse me with someone of baby boomer age | |
| 19:07:08 | sean-k-mooney | mriedem: https://bugs.launchpad.net/nova/+bug/1788403 is marked as fix releaed and i think you fixed it by intoducing the waits on revert correct so the current skip of test_server_connectivity_cold_migration_revert probly should have been removed a while ago | |
| 19:07:09 | openstack | Launchpad bug 1788403 in OpenStack Compute (nova) "test_server_connectivity_cold_migration_revert randomly fails ssh check" [Medium,Fix released] - Assigned to Matt Riedemann (mriedem) | |
| 19:21:19 | sean-k-mooney | alex_xu: yonglihe i think https://review.opendev.org/#/c/658716/ look fine. sorry for not getting to it yesterday | |
| 19:21:36 | mriedem | artom: dansmith: sean-k-mooney: i've dumped a bit in https://review.opendev.org/#/c/644881/ - i'm going to concentrate on something else now | |
| 19:21:39 | sean-k-mooney | now to go revew teh virtual perstent memory spec | |
| 19:22:07 | artom | mriedem, thank you for your dump :D | |
| 19:22:12 | mriedem | sean-k-mooney: yeah i was wondering why i didn't remove the skip on test_server_connectivity_cold_migration_revert | |
| 19:24:12 | sean-k-mooney | mriedem: is the skip in tempest or in nova out of interest? | |
| 19:24:21 | mriedem | tempest | |
| 19:24:27 | mriedem | i'll post a tempest change to remove the skip and see what happens | |
| 19:24:29 | sean-k-mooney | ah so its the skip decorator | |
| 19:25:19 | sean-k-mooney | i would expect it to pass since we default to the ovs firewall | |
| 19:25:57 | sean-k-mooney | i dont think neutorn has a multinode iptables job or any project currently so i doubt we will see failures | |
| 19:26:20 | sean-k-mooney | but i could be wrong about that | |
| 19:28:08 | mriedem | https://review.opendev.org/663405 | |
| 19:30:40 | sean-k-mooney | i need to go stack a multi node env so ill see if i can run that locally too | |
| 19:35:38 | sean-k-mooney | efried: haha you got to before i did | |
| 19:36:46 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Drop pre-cinder 3.44 version compatibility https://review.opendev.org/621061 | |
| 19:36:56 | sean-k-mooney | efried: i think we have so issues with those db unit test randomly failing | |
| 19:37:32 | sean-k-mooney | its not the first time i have seen them fail on py36 and pass on py27 and py37 | |
| 19:37:46 | efried | sean-k-mooney: what are we talking about? | |
| 19:38:03 | sean-k-mooney | the hide hypervior id patch | |
| 19:38:08 | efried | sean-k-mooney: https://bugs.launchpad.net/nova/+bug/1823251 ? | |
| 19:38:09 | openstack | Launchpad bug 1823251 in OpenStack Compute (nova) "Spike in TestNovaMigrationsMySQL.test_walk_versions/test_innodb_tables failures since April 1 2019 on limestone-regionone" [High,Confirmed] | |
| 19:38:39 | sean-k-mooney | and yes i was refering to the failures of those tests | |
| 19:38:39 | efried | sean-k-mooney: You'll be my hero if you can figure that one out. | |
| 19:39:12 | efried | probably even shiny nickel worthy | |
| 19:40:27 | sean-k-mooney | what strang is they appear to be failing only in the limestone-regionone | |
| 19:41:44 | artom | *family | |
| 19:41:58 | artom | Gonna have to finish addressing your feedback later tonight, mriedem | |
| 19:42:14 | artom | Moving to test_compute_mgr isn't as trivial as it first sounded :( | |
| 19:45:29 | sean-k-mooney | efried: looks like gibi is trying to fix it with https://review.opendev.org/#/c/662434/ | |
| 19:46:01 | efried | Oh, okay. | |
| 19:46:21 | efried | except it didn't work? | |
| 19:46:21 | sean-k-mooney | well its not actully a fix but just a cleanup to help find a fix | |
| 19:46:32 | efried | yeah | |