Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-06-05
18:58:53 mriedem why does https://review.opendev.org/#/c/653498/ come after the fix in the series?
18:59:27 artom mriedem, because without hybrid plug it's actually irrelevant to the fix :/
18:59:32 artom Well
18:59:39 artom I suppose it tests the non-hybrid-plug case
19:00:12 mriedem correct
19:00:30 mriedem you said in the ML it was the only job we run that is multi-node for the resize revert stuff right?
19:00:39 mriedem or would be
19:00:57 artom Yeah
19:01:00 artom I can move it below
19:02:16 mriedem you know,
19:02:23 sean-k-mooney artom: actuly i think you used to have it below and inverted it in a recent revision
19:02:28 mriedem test_server_connectivity_cold_migration_revert in the tempest-slow-py3 job would run resize revert in a multi-node job,
19:02:31 mriedem if you removed the skip on the test
19:02:38 mriedem and the test is skipped b/c of https://bugs.launchpad.net/nova/+bug/1788403
19:02:40 openstack Launchpad bug 1788403 in OpenStack Compute (nova) "test_server_connectivity_cold_migration_revert randomly fails ssh check" [Medium,Fix released] - Assigned to Matt Riedemann (mriedem)
19:02:46 mriedem which sounds very familiar to your issue yeah?
19:03:13 artom It would, except again, upstream gate doesn't use hybrid plug, so it's a different root cause
19:03:19 dansmith mriedem: you emotionally scarred me with discussion of that opinion in the past, which is why I remember it
19:04:06 dansmith mriedem: and yeah, not saying fixing the other operations needs to be in this patch, but if we're undoing not only this wait, but also making previous ones (like the hard reboot) no longer accurate, I think fixing them all in the same way (re-using helpers if they apply, etc) is a good idea
19:04:06 mriedem dansmith: logs or it never happened!
19:04:21 dansmith mriedem: I can show you the scars. those'
19:04:24 dansmith re good enough yeah?
19:04:34 mriedem pictures or it didn't happen!
19:05:08 artom Faxes? Damn boomers
19:06:41 dansmith I know you didn't just confuse me with someone of baby boomer age
19:07:08 sean-k-mooney mriedem: https://bugs.launchpad.net/nova/+bug/1788403 is marked as fix releaed and i think you fixed it by intoducing the waits on revert correct so the current skip of test_server_connectivity_cold_migration_revert probly should have been removed a while ago
19:07:09 openstack Launchpad bug 1788403 in OpenStack Compute (nova) "test_server_connectivity_cold_migration_revert randomly fails ssh check" [Medium,Fix released] - Assigned to Matt Riedemann (mriedem)
19:21:19 sean-k-mooney alex_xu: yonglihe i think https://review.opendev.org/#/c/658716/ look fine. sorry for not getting to it yesterday
19:21:36 mriedem artom: dansmith: sean-k-mooney: i've dumped a bit in https://review.opendev.org/#/c/644881/ - i'm going to concentrate on something else now
19:21:39 sean-k-mooney now to go revew teh virtual perstent memory spec
19:22:07 artom mriedem, thank you for your dump :D
19:22:12 mriedem sean-k-mooney: yeah i was wondering why i didn't remove the skip on test_server_connectivity_cold_migration_revert
19:24:12 sean-k-mooney mriedem: is the skip in tempest or in nova out of interest?
19:24:21 mriedem tempest
19:24:27 mriedem i'll post a tempest change to remove the skip and see what happens
19:24:29 sean-k-mooney ah so its the skip decorator
19:25:19 sean-k-mooney i would expect it to pass since we default to the ovs firewall
19:25:57 sean-k-mooney i dont think neutorn has a multinode iptables job or any project currently so i doubt we will see failures
19:26:20 sean-k-mooney but i could be wrong about that
19:28:08 mriedem https://review.opendev.org/663405
19:30:40 sean-k-mooney i need to go stack a multi node env so ill see if i can run that locally too
19:35:38 sean-k-mooney efried: haha you got to before i did
19:36:46 openstackgerrit Matt Riedemann proposed openstack/nova master: Drop pre-cinder 3.44 version compatibility https://review.opendev.org/621061
19:36:56 sean-k-mooney efried: i think we have so issues with those db unit test randomly failing
19:37:32 sean-k-mooney its not the first time i have seen them fail on py36 and pass on py27 and py37
19:37:46 efried sean-k-mooney: what are we talking about?
19:38:03 sean-k-mooney the hide hypervior id patch
19:38:08 efried sean-k-mooney: https://bugs.launchpad.net/nova/+bug/1823251 ?
19:38:09 openstack Launchpad bug 1823251 in OpenStack Compute (nova) "Spike in TestNovaMigrationsMySQL.test_walk_versions/test_innodb_tables failures since April 1 2019 on limestone-regionone" [High,Confirmed]
19:38:39 sean-k-mooney and yes i was refering to the failures of those tests
19:38:39 efried sean-k-mooney: You'll be my hero if you can figure that one out.
19:39:12 efried probably even shiny nickel worthy
19:40:27 sean-k-mooney what strang is they appear to be failing only in the limestone-regionone
19:41:44 artom *family
19:41:58 artom Gonna have to finish addressing your feedback later tonight, mriedem
19:42:14 artom Moving to test_compute_mgr isn't as trivial as it first sounded :(
19:45:29 sean-k-mooney efried: looks like gibi is trying to fix it with https://review.opendev.org/#/c/662434/
19:46:01 efried Oh, okay.
19:46:21 efried except it didn't work?
19:46:21 sean-k-mooney well its not actully a fix but just a cleanup to help find a fix
19:46:32 efried yeah
19:47:04 sean-k-mooney well its passing but i need to check if it ran on limestone or not
19:47:19 efried It failed once
19:47:35 efried http://logs.openstack.org/34/662434/1/check/openstack-tox-py27/3ead2fd/testr_results.html.gz
19:48:37 sean-k-mooney v2 ran on ovh so the fact v2 pass doesnot tell us much
19:49:12 sean-k-mooney the one that failed ran on limestone http://logs.openstack.org/34/662434/1/check/openstack-tox-py27/3ead2fd/job-output.txt.gz#_2019-05-31_11_45_41_625314
19:50:41 sean-k-mooney in theory nodepool is building and uploading the same image to all providers so it should be the same but i guess we could check with the infra folks to confirm that
19:52:21 sean-k-mooney oh and thats interest so the failure also happens on py27
20:13:23 cfriesen mriedem: I'm hitting something interesting in pike. when deleting a nova-compute service I'm still managing to hit the "Unable to delete resource provider X: Resource provider has allocations": error, even with your check to see if there are instances on the compute host.
20:13:52 efried did you check placement for allocations?
20:14:08 cfriesen mriedem: this leaves us with a stale resource_provider entry, whose UUID ends up out of sync with the nova compute_nodes table, which causes scheduling to fail when we add the node back with the same name
20:14:28 cfriesen efried: checking that now
20:14:52 sean-k-mooney cfriesen: there could be allocation related to stale migrations against that node
20:15:04 efried is this what nova manage heal allocations is for?
20:15:14 efried (however that's spelled)
20:16:25 sean-k-mooney im not sure
20:16:37 cfriesen efried: I see 14 allocations for the stale node for resource classes 0, 1, 2. (cpu/ram/disk, I think)
20:16:38 sean-k-mooney i dont think so but it may be a sideffect
20:17:04 efried cfriesen: Good to know placement wasn't lying.
20:17:10 sean-k-mooney cfriesen: do the allcoation consume ids match any instance or migrations
20:18:07 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Use SATA bus for cdrom devices when using Q35 machine type https://review.opendev.org/663011
20:18:08 openstackgerrit Lee Yarwood proposed openstack/nova master: DNM: Run tempest-full-py3 with q35 machine type https://review.opendev.org/662887
20:20:19 cfriesen sean-k-mooney: they match 6 instances that are currently running (they had been running on the node that I deleted)
20:21:07 mriedem cfriesen: kvm or ironic service?
20:21:16 cfriesen mriedem: kvm
20:21:54 mriedem heal_allocations is for adding missing allocations to a provider that has instances on it
20:22:09 sean-k-mooney cfriesen: ok so placement still thinks they are running on the wrong node
20:22:42 cfriesen this is coming from a customer test, but my way of reproducing is to have instances running, then power off the node uncleanly, and then try to delete the node while the automated tools are evacuating the instances that had been running on it.
20:23:11 mriedem as sean-k-mooney said, there are probably allocations from migration records
20:23:13 sean-k-mooney heal_allocations would add allcotion for the node the vms are now running on but does it also clean up allocation where the vms are nolonger running
20:23:17 mriedem b/c you're deleting during a resize or something
20:23:42 cfriesen sean-k-mooney: nope. there are allocations with the same consumer id but two differnet resource_provider_id
20:24:41 mriedem https://bugs.launchpad.net/nova/+bug/1793569
20:24:41 cfriesen mriedem: would there be allocations from the evacuate?
20:24:43 openstack Launchpad bug 1793569 in OpenStack Compute (nova) "Add placement audit commands" [Wishlist,Confirmed]
20:25:16 mriedem cfriesen: i don't think there would be migration consumer allocatoins during an evac,
20:25:26 mriedem but there could be allocations for the instances that were on that evacuated host
20:25:45 sean-k-mooney i dont think there is either but we should claim resouce on the destiation
20:25:47 mriedem when the old host is brought back, it should clean up those evacuated instance allocations on startup
20:25:47 cfriesen they would have all been live-migrated onto that host not too long before I killed the node

Earlier   Later