| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-06-05 | |||
| 18:50:58 | artom | That doesn't invalidate my argument :) | |
| 18:51:08 | sean-k-mooney | odl took 2 years to catch up and intoduce a way for odl to send events to neutron which neutron could send to nova | |
| 18:51:14 | artom | The idea was for Nova to be sure we have networking for instance by the time it spawns | |
| 18:51:17 | artom | So things like DHCP work | |
| 18:51:27 | artom | If we get the event before that's happened, that's not for Nova to fix | |
| 18:51:49 | sean-k-mooney | yep i had this argument with the odl folk back in 2014 | |
| 18:51:53 | artom | Because we'd be acting like networking is ready, when in reality it isn't | |
| 18:52:01 | artom | (yey, 5 years late) | |
| 18:52:39 | sean-k-mooney | for what its worth im pretty sure netorking-aci still does not send it when the port is wireded up an send it on bind | |
| 18:52:57 | sean-k-mooney | and ovn is not doing the right thing either in all cases | |
| 18:52:58 | artom | The Cisco thing? | |
| 18:53:02 | sean-k-mooney | yes | |
| 18:53:14 | artom | That's just one more argument for people not to buy Cisco ;) | |
| 18:53:36 | mriedem | "IIRC, mriedem (and maybe others) really want to keep the "can keep it on the source if it's not too late" behavior" | |
| 18:53:40 | mriedem | i have no opinion on that | |
| 18:53:42 | mriedem | that i'm aware of | |
| 18:54:14 | mriedem | i'm also assuming i don't want to cruft up an already complicated change to cover other cases in the same patch if we're doing backports | |
| 18:54:23 | mriedem | b/c there is a >0% chance this will introduce a regression somehow | |
| 18:54:27 | mriedem | b/c this is all very brittle sounding | |
| 18:54:30 | mriedem | and super duper latent | |
| 18:55:12 | mriedem | how would one even recreate this hybrid ovs resize thing in the gate? configure devstack for that backend (is that easy?) and then put a sleep in the nova code between migrate_instance_start(finish?) and the call to the virt driver? | |
| 18:55:16 | artom | Like osteoporosis | |
| 18:56:14 | sean-k-mooney | mriedem: we have patches to recreate. you configure ovs of iptables and then we can race on teh revert tests | |
| 18:56:39 | sean-k-mooney | *ovs to use the iptables firewall driver | |
| 18:56:53 | artom | https://review.opendev.org/#/c/660782/5 specifically | |
| 18:57:59 | sean-k-mooney | our downstream ci default to iptables firewall still on older releases so that is why we are seeing it downstream. upstream use the openvswtich contrack firewall driver by default | |
| 18:58:53 | mriedem | why does https://review.opendev.org/#/c/653498/ come after the fix in the series? | |
| 18:59:27 | artom | mriedem, because without hybrid plug it's actually irrelevant to the fix :/ | |
| 18:59:32 | artom | Well | |
| 18:59:39 | artom | I suppose it tests the non-hybrid-plug case | |
| 19:00:12 | mriedem | correct | |
| 19:00:30 | mriedem | you said in the ML it was the only job we run that is multi-node for the resize revert stuff right? | |
| 19:00:39 | mriedem | or would be | |
| 19:00:57 | artom | Yeah | |
| 19:01:00 | artom | I can move it below | |
| 19:02:16 | mriedem | you know, | |
| 19:02:23 | sean-k-mooney | artom: actuly i think you used to have it below and inverted it in a recent revision | |
| 19:02:28 | mriedem | test_server_connectivity_cold_migration_revert in the tempest-slow-py3 job would run resize revert in a multi-node job, | |
| 19:02:31 | mriedem | if you removed the skip on the test | |
| 19:02:38 | mriedem | and the test is skipped b/c of https://bugs.launchpad.net/nova/+bug/1788403 | |
| 19:02:40 | openstack | Launchpad bug 1788403 in OpenStack Compute (nova) "test_server_connectivity_cold_migration_revert randomly fails ssh check" [Medium,Fix released] - Assigned to Matt Riedemann (mriedem) | |
| 19:02:46 | mriedem | which sounds very familiar to your issue yeah? | |
| 19:03:13 | artom | It would, except again, upstream gate doesn't use hybrid plug, so it's a different root cause | |
| 19:03:19 | dansmith | mriedem: you emotionally scarred me with discussion of that opinion in the past, which is why I remember it | |
| 19:04:06 | mriedem | dansmith: logs or it never happened! | |
| 19:04:06 | dansmith | mriedem: and yeah, not saying fixing the other operations needs to be in this patch, but if we're undoing not only this wait, but also making previous ones (like the hard reboot) no longer accurate, I think fixing them all in the same way (re-using helpers if they apply, etc) is a good idea | |
| 19:04:21 | dansmith | mriedem: I can show you the scars. those' | |
| 19:04:24 | dansmith | re good enough yeah? | |
| 19:04:34 | mriedem | pictures or it didn't happen! | |
| 19:05:08 | artom | Faxes? Damn boomers | |
| 19:06:41 | dansmith | I know you didn't just confuse me with someone of baby boomer age | |
| 19:07:08 | sean-k-mooney | mriedem: https://bugs.launchpad.net/nova/+bug/1788403 is marked as fix releaed and i think you fixed it by intoducing the waits on revert correct so the current skip of test_server_connectivity_cold_migration_revert probly should have been removed a while ago | |
| 19:07:09 | openstack | Launchpad bug 1788403 in OpenStack Compute (nova) "test_server_connectivity_cold_migration_revert randomly fails ssh check" [Medium,Fix released] - Assigned to Matt Riedemann (mriedem) | |
| 19:21:19 | sean-k-mooney | alex_xu: yonglihe i think https://review.opendev.org/#/c/658716/ look fine. sorry for not getting to it yesterday | |
| 19:21:36 | mriedem | artom: dansmith: sean-k-mooney: i've dumped a bit in https://review.opendev.org/#/c/644881/ - i'm going to concentrate on something else now | |
| 19:21:39 | sean-k-mooney | now to go revew teh virtual perstent memory spec | |
| 19:22:07 | artom | mriedem, thank you for your dump :D | |
| 19:22:12 | mriedem | sean-k-mooney: yeah i was wondering why i didn't remove the skip on test_server_connectivity_cold_migration_revert | |
| 19:24:12 | sean-k-mooney | mriedem: is the skip in tempest or in nova out of interest? | |
| 19:24:21 | mriedem | tempest | |
| 19:24:27 | mriedem | i'll post a tempest change to remove the skip and see what happens | |
| 19:24:29 | sean-k-mooney | ah so its the skip decorator | |
| 19:25:19 | sean-k-mooney | i would expect it to pass since we default to the ovs firewall | |
| 19:25:57 | sean-k-mooney | i dont think neutorn has a multinode iptables job or any project currently so i doubt we will see failures | |
| 19:26:20 | sean-k-mooney | but i could be wrong about that | |
| 19:28:08 | mriedem | https://review.opendev.org/663405 | |
| 19:30:40 | sean-k-mooney | i need to go stack a multi node env so ill see if i can run that locally too | |
| 19:35:38 | sean-k-mooney | efried: haha you got to before i did | |
| 19:36:46 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Drop pre-cinder 3.44 version compatibility https://review.opendev.org/621061 | |
| 19:36:56 | sean-k-mooney | efried: i think we have so issues with those db unit test randomly failing | |
| 19:37:32 | sean-k-mooney | its not the first time i have seen them fail on py36 and pass on py27 and py37 | |
| 19:37:46 | efried | sean-k-mooney: what are we talking about? | |
| 19:38:03 | sean-k-mooney | the hide hypervior id patch | |
| 19:38:08 | efried | sean-k-mooney: https://bugs.launchpad.net/nova/+bug/1823251 ? | |
| 19:38:09 | openstack | Launchpad bug 1823251 in OpenStack Compute (nova) "Spike in TestNovaMigrationsMySQL.test_walk_versions/test_innodb_tables failures since April 1 2019 on limestone-regionone" [High,Confirmed] | |
| 19:38:39 | efried | sean-k-mooney: You'll be my hero if you can figure that one out. | |
| 19:38:39 | sean-k-mooney | and yes i was refering to the failures of those tests | |
| 19:39:12 | efried | probably even shiny nickel worthy | |
| 19:40:27 | sean-k-mooney | what strang is they appear to be failing only in the limestone-regionone | |
| 19:41:44 | artom | *family | |
| 19:41:58 | artom | Gonna have to finish addressing your feedback later tonight, mriedem | |
| 19:42:14 | artom | Moving to test_compute_mgr isn't as trivial as it first sounded :( | |
| 19:45:29 | sean-k-mooney | efried: looks like gibi is trying to fix it with https://review.opendev.org/#/c/662434/ | |
| 19:46:01 | efried | Oh, okay. | |
| 19:46:21 | sean-k-mooney | well its not actully a fix but just a cleanup to help find a fix | |
| 19:46:21 | efried | except it didn't work? | |
| 19:46:32 | efried | yeah | |
| 19:47:04 | sean-k-mooney | well its passing but i need to check if it ran on limestone or not | |
| 19:47:19 | efried | It failed once | |
| 19:47:35 | efried | http://logs.openstack.org/34/662434/1/check/openstack-tox-py27/3ead2fd/testr_results.html.gz | |
| 19:48:37 | sean-k-mooney | v2 ran on ovh so the fact v2 pass doesnot tell us much | |
| 19:49:12 | sean-k-mooney | the one that failed ran on limestone http://logs.openstack.org/34/662434/1/check/openstack-tox-py27/3ead2fd/job-output.txt.gz#_2019-05-31_11_45_41_625314 | |
| 19:50:41 | sean-k-mooney | in theory nodepool is building and uploading the same image to all providers so it should be the same but i guess we could check with the infra folks to confirm that | |
| 19:52:21 | sean-k-mooney | oh and thats interest so the failure also happens on py27 | |
| 20:13:23 | cfriesen | mriedem: I'm hitting something interesting in pike. when deleting a nova-compute service I'm still managing to hit the "Unable to delete resource provider X: Resource provider has allocations": error, even with your check to see if there are instances on the compute host. | |
| 20:13:52 | efried | did you check placement for allocations? | |
| 20:14:08 | cfriesen | mriedem: this leaves us with a stale resource_provider entry, whose UUID ends up out of sync with the nova compute_nodes table, which causes scheduling to fail when we add the node back with the same name | |
| 20:14:28 | cfriesen | efried: checking that now | |
| 20:14:52 | sean-k-mooney | cfriesen: there could be allocation related to stale migrations against that node | |
| 20:15:04 | efried | is this what nova manage heal allocations is for? | |