| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-06-05 | |||
| 19:36:46 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Drop pre-cinder 3.44 version compatibility https://review.opendev.org/621061 | |
| 19:36:56 | sean-k-mooney | efried: i think we have so issues with those db unit test randomly failing | |
| 19:37:32 | sean-k-mooney | its not the first time i have seen them fail on py36 and pass on py27 and py37 | |
| 19:37:46 | efried | sean-k-mooney: what are we talking about? | |
| 19:38:03 | sean-k-mooney | the hide hypervior id patch | |
| 19:38:08 | efried | sean-k-mooney: https://bugs.launchpad.net/nova/+bug/1823251 ? | |
| 19:38:09 | openstack | Launchpad bug 1823251 in OpenStack Compute (nova) "Spike in TestNovaMigrationsMySQL.test_walk_versions/test_innodb_tables failures since April 1 2019 on limestone-regionone" [High,Confirmed] | |
| 19:38:39 | efried | sean-k-mooney: You'll be my hero if you can figure that one out. | |
| 19:38:39 | sean-k-mooney | and yes i was refering to the failures of those tests | |
| 19:39:12 | efried | probably even shiny nickel worthy | |
| 19:40:27 | sean-k-mooney | what strang is they appear to be failing only in the limestone-regionone | |
| 19:41:44 | artom | *family | |
| 19:41:58 | artom | Gonna have to finish addressing your feedback later tonight, mriedem | |
| 19:42:14 | artom | Moving to test_compute_mgr isn't as trivial as it first sounded :( | |
| 19:45:29 | sean-k-mooney | efried: looks like gibi is trying to fix it with https://review.opendev.org/#/c/662434/ | |
| 19:46:01 | efried | Oh, okay. | |
| 19:46:21 | sean-k-mooney | well its not actully a fix but just a cleanup to help find a fix | |
| 19:46:21 | efried | except it didn't work? | |
| 19:46:32 | efried | yeah | |
| 19:47:04 | sean-k-mooney | well its passing but i need to check if it ran on limestone or not | |
| 19:47:19 | efried | It failed once | |
| 19:47:35 | efried | http://logs.openstack.org/34/662434/1/check/openstack-tox-py27/3ead2fd/testr_results.html.gz | |
| 19:48:37 | sean-k-mooney | v2 ran on ovh so the fact v2 pass doesnot tell us much | |
| 19:49:12 | sean-k-mooney | the one that failed ran on limestone http://logs.openstack.org/34/662434/1/check/openstack-tox-py27/3ead2fd/job-output.txt.gz#_2019-05-31_11_45_41_625314 | |
| 19:50:41 | sean-k-mooney | in theory nodepool is building and uploading the same image to all providers so it should be the same but i guess we could check with the infra folks to confirm that | |
| 19:52:21 | sean-k-mooney | oh and thats interest so the failure also happens on py27 | |
| 20:13:23 | cfriesen | mriedem: I'm hitting something interesting in pike. when deleting a nova-compute service I'm still managing to hit the "Unable to delete resource provider X: Resource provider has allocations": error, even with your check to see if there are instances on the compute host. | |
| 20:13:52 | efried | did you check placement for allocations? | |
| 20:14:08 | cfriesen | mriedem: this leaves us with a stale resource_provider entry, whose UUID ends up out of sync with the nova compute_nodes table, which causes scheduling to fail when we add the node back with the same name | |
| 20:14:28 | cfriesen | efried: checking that now | |
| 20:14:52 | sean-k-mooney | cfriesen: there could be allocation related to stale migrations against that node | |
| 20:15:04 | efried | is this what nova manage heal allocations is for? | |
| 20:15:14 | efried | (however that's spelled) | |
| 20:16:25 | sean-k-mooney | im not sure | |
| 20:16:37 | cfriesen | efried: I see 14 allocations for the stale node for resource classes 0, 1, 2. (cpu/ram/disk, I think) | |
| 20:16:38 | sean-k-mooney | i dont think so but it may be a sideffect | |
| 20:17:04 | efried | cfriesen: Good to know placement wasn't lying. | |
| 20:17:10 | sean-k-mooney | cfriesen: do the allcoation consume ids match any instance or migrations | |
| 20:18:07 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Use SATA bus for cdrom devices when using Q35 machine type https://review.opendev.org/663011 | |
| 20:18:08 | openstackgerrit | Lee Yarwood proposed openstack/nova master: DNM: Run tempest-full-py3 with q35 machine type https://review.opendev.org/662887 | |
| 20:20:19 | cfriesen | sean-k-mooney: they match 6 instances that are currently running (they had been running on the node that I deleted) | |
| 20:21:07 | mriedem | cfriesen: kvm or ironic service? | |
| 20:21:16 | cfriesen | mriedem: kvm | |
| 20:21:54 | mriedem | heal_allocations is for adding missing allocations to a provider that has instances on it | |
| 20:22:09 | sean-k-mooney | cfriesen: ok so placement still thinks they are running on the wrong node | |
| 20:22:42 | cfriesen | this is coming from a customer test, but my way of reproducing is to have instances running, then power off the node uncleanly, and then try to delete the node while the automated tools are evacuating the instances that had been running on it. | |
| 20:23:11 | mriedem | as sean-k-mooney said, there are probably allocations from migration records | |
| 20:23:13 | sean-k-mooney | heal_allocations would add allcotion for the node the vms are now running on but does it also clean up allocation where the vms are nolonger running | |
| 20:23:17 | mriedem | b/c you're deleting during a resize or something | |
| 20:23:42 | cfriesen | sean-k-mooney: nope. there are allocations with the same consumer id but two differnet resource_provider_id | |
| 20:24:41 | cfriesen | mriedem: would there be allocations from the evacuate? | |
| 20:24:41 | mriedem | https://bugs.launchpad.net/nova/+bug/1793569 | |
| 20:24:43 | openstack | Launchpad bug 1793569 in OpenStack Compute (nova) "Add placement audit commands" [Wishlist,Confirmed] | |
| 20:25:16 | mriedem | cfriesen: i don't think there would be migration consumer allocatoins during an evac, | |
| 20:25:26 | mriedem | but there could be allocations for the instances that were on that evacuated host | |
| 20:25:45 | sean-k-mooney | i dont think there is either but we should claim resouce on the destiation | |
| 20:25:47 | cfriesen | they would have all been live-migrated onto that host not too long before I killed the node | |
| 20:25:47 | mriedem | when the old host is brought back, it should clean up those evacuated instance allocations on startup | |
| 20:26:38 | mriedem | gibi recently had something merged for evacuate that sounds related https://review.opendev.org/#/c/512623/ | |
| 20:28:10 | sean-k-mooney | oh ya i rember looking at this a few days ago but not having time to review | |
| 20:32:00 | mriedem | cfriesen: see my note on https://review.opendev.org/#/c/657070/2/nova/scheduler/client/report.py as well | |
| 20:32:00 | sean-k-mooney | mriedem: was this the patch you were asking about performace of "x in set()" vs "x in dict()" | |
| 20:32:12 | mriedem | sean-k-mooney: no | |
| 20:32:24 | sean-k-mooney | ok | |
| 20:32:36 | mriedem | anywya, we don't have migration consumers for evac | |
| 20:33:05 | sean-k-mooney | we dont but we do claim an allcoation candate for teh destination node | |
| 20:33:25 | sean-k-mooney | but i dont know if we free the allcoation of the source node in all cases | |
| 20:33:54 | mriedem | correct the scheduler will claim (or conductor if you're forcing the dest host) https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L923 | |
| 20:34:03 | mriedem | the source host allocations should be cleaned up when the service is restarted | |
| 20:34:05 | openstackgerrit | John Garbutt proposed openstack/nova master: WIP: add functional test for admin_password https://review.opendev.org/663422 | |
| 20:34:13 | cfriesen | when evacuating an instance, when are the allocations on the old node supposed to be cleaned up? | |
| 20:34:27 | mriedem | in here https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L620 | |
| 20:34:34 | mriedem | on restart of the service ^ | |
| 20:34:54 | cfriesen | okay, so if we never restart the service they'll just stay stale in the allocations table | |
| 20:35:00 | mriedem | right, | |
| 20:35:03 | sean-k-mooney | cfriesen: when you start up the compute agent on the souce node after fixing it meaning you wont be able to delete the RP until after that | |
| 20:35:05 | mriedem | but presumably that node is dead and fenced anyway | |
| 20:35:09 | mriedem | so you're not scheduling to it | |
| 20:35:36 | cfriesen | yeah...the problem is that those allocations prevent us from deleting the resource_provider on compute node deletion | |
| 20:35:36 | sean-k-mooney | mriedem: i think cfriesen was trying to remove teh RP before repovisoing the fixed node | |
| 20:36:02 | sean-k-mooney | cfriesen: why are you delete in the compute node recored out of interest | |
| 20:36:25 | cfriesen | customer is deleting the node. no idea why. testing functionality? | |
| 20:36:32 | sean-k-mooney | if you are replacing the node with another and reusing the hostname it should not be needed | |
| 20:36:40 | sean-k-mooney | ah ok | |
| 20:36:50 | sean-k-mooney | customer always break things | |
| 20:36:52 | cfriesen | when we add the new node back in with a different hostname it's all fine | |
| 20:37:28 | mriedem | the shitty thing i didn't realize until cdent pointed it out in an open review is that https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L2183 won't raise if deleting the provider fails, which means we'll continue to delete the compute service record | |
| 20:37:28 | cfriesen | if we reuse the same hostname we get a different UUID in the compute_node table, but we don't get a new entry in the resource_provider table, so the UUIDs don't match and we never schedule anything to the node | |
| 20:37:31 | sean-k-mooney | cfriesen: if you add it with the same host name it would be fine too just dont delete the compute node recored and it will fix everything up when the compute agent starts | |
| 20:37:58 | cfriesen | sean-k-mooney: the compute node record is deleted by running the "delete service" api | |
| 20:38:19 | sean-k-mooney | cfriesen: yes. which you dont need to do to replace teh server | |
| 20:38:48 | mriedem | cfriesen: https://bugs.launchpad.net/nova/+bug/1829479 could also be related to your issue | |
| 20:38:49 | openstack | Launchpad bug 1829479 in OpenStack Compute (nova) "The allocation table has residual records when instance is evacuated and the source physical node is removed" [Undecided,Incomplete] | |
| 20:38:53 | sean-k-mooney | and if you dont delete the compute service record you will get the same uuid when the new agent conencts | |
| 20:40:17 | cfriesen | sean-k-mooney: I suspect you're right as a workaround. but it *should* work in an ideal world. :) | |
| 20:40:51 | cfriesen | mriedem: yeah, that looks like what I'm seeing | |
| 20:41:35 | sean-k-mooney | cfriesen: we could "fix it" by having the compute node when it fails to find a compute node record by hostname first check if there is a placemetn RP by looking it up by hostname and then reuse the uuid of the RP which was the old compute node uuid | |
| 20:41:41 | openstackgerrit | John Garbutt proposed openstack/nova master: WIP: add functional test for admin_password https://review.opendev.org/663422 | |
| 20:42:53 | cfriesen | sean-k-mooney: would work but seems kind of kludgy. can we delete allocations when deleting a resource_provider? | |
| 20:43:35 | sean-k-mooney | we current dont allow you to delete inventoryes if they have allocatin intentionally | |