Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-06-05
19:07:08 sean-k-mooney mriedem: https://bugs.launchpad.net/nova/+bug/1788403 is marked as fix releaed and i think you fixed it by intoducing the waits on revert correct so the current skip of test_server_connectivity_cold_migration_revert probly should have been removed a while ago
19:07:09 openstack Launchpad bug 1788403 in OpenStack Compute (nova) "test_server_connectivity_cold_migration_revert randomly fails ssh check" [Medium,Fix released] - Assigned to Matt Riedemann (mriedem)
19:21:19 sean-k-mooney alex_xu: yonglihe i think https://review.opendev.org/#/c/658716/ look fine. sorry for not getting to it yesterday
19:21:36 mriedem artom: dansmith: sean-k-mooney: i've dumped a bit in https://review.opendev.org/#/c/644881/ - i'm going to concentrate on something else now
19:21:39 sean-k-mooney now to go revew teh virtual perstent memory spec
19:22:07 artom mriedem, thank you for your dump :D
19:22:12 mriedem sean-k-mooney: yeah i was wondering why i didn't remove the skip on test_server_connectivity_cold_migration_revert
19:24:12 sean-k-mooney mriedem: is the skip in tempest or in nova out of interest?
19:24:21 mriedem tempest
19:24:27 mriedem i'll post a tempest change to remove the skip and see what happens
19:24:29 sean-k-mooney ah so its the skip decorator
19:25:19 sean-k-mooney i would expect it to pass since we default to the ovs firewall
19:25:57 sean-k-mooney i dont think neutorn has a multinode iptables job or any project currently so i doubt we will see failures
19:26:20 sean-k-mooney but i could be wrong about that
19:28:08 mriedem https://review.opendev.org/663405
19:30:40 sean-k-mooney i need to go stack a multi node env so ill see if i can run that locally too
19:35:38 sean-k-mooney efried: haha you got to before i did
19:36:46 openstackgerrit Matt Riedemann proposed openstack/nova master: Drop pre-cinder 3.44 version compatibility https://review.opendev.org/621061
19:36:56 sean-k-mooney efried: i think we have so issues with those db unit test randomly failing
19:37:32 sean-k-mooney its not the first time i have seen them fail on py36 and pass on py27 and py37
19:37:46 efried sean-k-mooney: what are we talking about?
19:38:03 sean-k-mooney the hide hypervior id patch
19:38:08 efried sean-k-mooney: https://bugs.launchpad.net/nova/+bug/1823251 ?
19:38:09 openstack Launchpad bug 1823251 in OpenStack Compute (nova) "Spike in TestNovaMigrationsMySQL.test_walk_versions/test_innodb_tables failures since April 1 2019 on limestone-regionone" [High,Confirmed]
19:38:39 sean-k-mooney and yes i was refering to the failures of those tests
19:38:39 efried sean-k-mooney: You'll be my hero if you can figure that one out.
19:39:12 efried probably even shiny nickel worthy
19:40:27 sean-k-mooney what strang is they appear to be failing only in the limestone-regionone
19:41:44 artom *family
19:41:58 artom Gonna have to finish addressing your feedback later tonight, mriedem
19:42:14 artom Moving to test_compute_mgr isn't as trivial as it first sounded :(
19:45:29 sean-k-mooney efried: looks like gibi is trying to fix it with https://review.opendev.org/#/c/662434/
19:46:01 efried Oh, okay.
19:46:21 efried except it didn't work?
19:46:21 sean-k-mooney well its not actully a fix but just a cleanup to help find a fix
19:46:32 efried yeah
19:47:04 sean-k-mooney well its passing but i need to check if it ran on limestone or not
19:47:19 efried It failed once
19:47:35 efried http://logs.openstack.org/34/662434/1/check/openstack-tox-py27/3ead2fd/testr_results.html.gz
19:48:37 sean-k-mooney v2 ran on ovh so the fact v2 pass doesnot tell us much
19:49:12 sean-k-mooney the one that failed ran on limestone http://logs.openstack.org/34/662434/1/check/openstack-tox-py27/3ead2fd/job-output.txt.gz#_2019-05-31_11_45_41_625314
19:50:41 sean-k-mooney in theory nodepool is building and uploading the same image to all providers so it should be the same but i guess we could check with the infra folks to confirm that
19:52:21 sean-k-mooney oh and thats interest so the failure also happens on py27
20:13:23 cfriesen mriedem: I'm hitting something interesting in pike. when deleting a nova-compute service I'm still managing to hit the "Unable to delete resource provider X: Resource provider has allocations": error, even with your check to see if there are instances on the compute host.
20:13:52 efried did you check placement for allocations?
20:14:08 cfriesen mriedem: this leaves us with a stale resource_provider entry, whose UUID ends up out of sync with the nova compute_nodes table, which causes scheduling to fail when we add the node back with the same name
20:14:28 cfriesen efried: checking that now
20:14:52 sean-k-mooney cfriesen: there could be allocation related to stale migrations against that node
20:15:04 efried is this what nova manage heal allocations is for?
20:15:14 efried (however that's spelled)
20:16:25 sean-k-mooney im not sure
20:16:37 cfriesen efried: I see 14 allocations for the stale node for resource classes 0, 1, 2. (cpu/ram/disk, I think)
20:16:38 sean-k-mooney i dont think so but it may be a sideffect
20:17:04 efried cfriesen: Good to know placement wasn't lying.
20:17:10 sean-k-mooney cfriesen: do the allcoation consume ids match any instance or migrations
20:18:07 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Use SATA bus for cdrom devices when using Q35 machine type https://review.opendev.org/663011
20:18:08 openstackgerrit Lee Yarwood proposed openstack/nova master: DNM: Run tempest-full-py3 with q35 machine type https://review.opendev.org/662887
20:20:19 cfriesen sean-k-mooney: they match 6 instances that are currently running (they had been running on the node that I deleted)
20:21:07 mriedem cfriesen: kvm or ironic service?
20:21:16 cfriesen mriedem: kvm
20:21:54 mriedem heal_allocations is for adding missing allocations to a provider that has instances on it
20:22:09 sean-k-mooney cfriesen: ok so placement still thinks they are running on the wrong node
20:22:42 cfriesen this is coming from a customer test, but my way of reproducing is to have instances running, then power off the node uncleanly, and then try to delete the node while the automated tools are evacuating the instances that had been running on it.
20:23:11 mriedem as sean-k-mooney said, there are probably allocations from migration records
20:23:13 sean-k-mooney heal_allocations would add allcotion for the node the vms are now running on but does it also clean up allocation where the vms are nolonger running
20:23:17 mriedem b/c you're deleting during a resize or something
20:23:42 cfriesen sean-k-mooney: nope. there are allocations with the same consumer id but two differnet resource_provider_id
20:24:41 mriedem https://bugs.launchpad.net/nova/+bug/1793569
20:24:41 cfriesen mriedem: would there be allocations from the evacuate?
20:24:43 openstack Launchpad bug 1793569 in OpenStack Compute (nova) "Add placement audit commands" [Wishlist,Confirmed]
20:25:16 mriedem cfriesen: i don't think there would be migration consumer allocatoins during an evac,
20:25:26 mriedem but there could be allocations for the instances that were on that evacuated host
20:25:45 sean-k-mooney i dont think there is either but we should claim resouce on the destiation
20:25:47 mriedem when the old host is brought back, it should clean up those evacuated instance allocations on startup
20:25:47 cfriesen they would have all been live-migrated onto that host not too long before I killed the node
20:26:38 mriedem gibi recently had something merged for evacuate that sounds related https://review.opendev.org/#/c/512623/
20:28:10 sean-k-mooney oh ya i rember looking at this a few days ago but not having time to review
20:32:00 sean-k-mooney mriedem: was this the patch you were asking about performace of "x in set()" vs "x in dict()"
20:32:00 mriedem cfriesen: see my note on https://review.opendev.org/#/c/657070/2/nova/scheduler/client/report.py as well
20:32:12 mriedem sean-k-mooney: no
20:32:24 sean-k-mooney ok
20:32:36 mriedem anywya, we don't have migration consumers for evac
20:33:05 sean-k-mooney we dont but we do claim an allcoation candate for teh destination node
20:33:25 sean-k-mooney but i dont know if we free the allcoation of the source node in all cases
20:33:54 mriedem correct the scheduler will claim (or conductor if you're forcing the dest host) https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L923
20:34:03 mriedem the source host allocations should be cleaned up when the service is restarted
20:34:05 openstackgerrit John Garbutt proposed openstack/nova master: WIP: add functional test for admin_password https://review.opendev.org/663422
20:34:13 cfriesen when evacuating an instance, when are the allocations on the old node supposed to be cleaned up?
20:34:27 mriedem in here https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L620
20:34:34 mriedem on restart of the service ^
20:34:54 cfriesen okay, so if we never restart the service they'll just stay stale in the allocations table
20:35:00 mriedem right,
20:35:03 sean-k-mooney cfriesen: when you start up the compute agent on the souce node after fixing it meaning you wont be able to delete the RP until after that
20:35:05 mriedem but presumably that node is dead and fenced anyway
20:35:09 mriedem so you're not scheduling to it
20:35:36 sean-k-mooney mriedem: i think cfriesen was trying to remove teh RP before repovisoing the fixed node
20:35:36 cfriesen yeah...the problem is that those allocations prevent us from deleting the resource_provider on compute node deletion
20:36:02 sean-k-mooney cfriesen: why are you delete in the compute node recored out of interest
20:36:25 cfriesen customer is deleting the node. no idea why. testing functionality?
20:36:32 sean-k-mooney if you are replacing the node with another and reusing the hostname it should not be needed

Earlier   Later