| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-11 | |||
| 18:56:52 | eandersson | Having an odd bug - a VM can't be rebuilt with a new image. Pretty sure placement is filtering out all the potential matches. | |
| 18:57:02 | eandersson | Rebuilding it with the same image works fine. | |
| 18:57:05 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Follow up for Ib50b6b02208f5bd2972de8a6f8f685c19745514c https://review.opendev.org/681490 | |
| 18:57:41 | mriedem | eandersson: which release? | |
| 18:57:46 | eandersson | Running Rocky | |
| 18:57:58 | eandersson | Very latest stable/rocky | |
| 18:58:26 | eandersson | I see that the scheduler gets a single result, but it is failing on NUMA Topology. | |
| 18:58:32 | mriedem | we shouldn't be calling placement for rebuild https://github.com/openstack/nova/blob/stable/rocky/nova/scheduler/manager.py#L126 | |
| 18:58:46 | eandersson | > 'NUMATopologyFilter: (start: 1, end: 0)' | |
| 18:58:51 | eandersson | Ah bad assumption | |
| 18:58:53 | mriedem | https://bugs.launchpad.net/nova/+bug/1763766 ? | |
| 18:58:54 | openstack | Launchpad bug 1763766 in OpenStack Compute (nova) "nova needs to disallow resource consumption changes on image rebuild" [Medium,Triaged] | |
| 18:59:14 | mriedem | or https://bugzilla.redhat.com/show_bug.cgi?id=1700412 | |
| 18:59:15 | openstack | bugzilla.redhat.com bug 1700412 in openstack-nova "[RFE] Rebuild with a different image fails due to NUMATopologyFilter" [High,New] - Assigned to nova-maint | |
| 18:59:37 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Skip querying resource request if no qos port https://review.opendev.org/681513 | |
| 18:59:46 | eandersson | Yep looks like it | |
| 18:59:52 | dansmith | yeah if the image is what causes it trouble, | |
| 19:00:07 | dansmith | it's likely the new numa topo filtering out the one host you're on | |
| 19:00:37 | mriedem | sean-k-mooney has to fix it and get that backported to queens | |
| 19:02:19 | artom | mriedem, it's not getting backported, but one of the things we discussed was making that RUN_ON_REBUILD flag we added way back when for the rebuild CVE configurable in some way | |
| 19:02:26 | artom | We kinda dreaded coming to upstream with that | |
| 19:02:49 | sean-k-mooney | dansmith: for the downstream issue yes its just the numa toploy filter filtering it out beacue tehre is no space for the rebuild | |
| 19:03:08 | dansmith | sean-k-mooney: I'm talking about eandersson's case | |
| 19:03:25 | mriedem | artom: we don't claim on rebuild so if the numa topo changes in the new image we won't claim for it properly, | |
| 19:03:33 | mriedem | so you could just be kicking the error down to compute | |
| 19:03:33 | sean-k-mooney | ill read scollback | |
| 19:03:43 | sean-k-mooney | mriedem: correct | |
| 19:03:54 | artom | mriedem, yeah, it wouldn't be a correct fix | |
| 19:04:04 | sean-k-mooney | we rebuild with the new image but old toplogy | |
| 19:04:23 | artom | More of a candy we throw our customers on, like, 10 to placate them until they an upgrade to whatever release the un-backportable fix lands in | |
| 19:04:23 | mriedem | sean-k-mooney: you mean in some downstream-only patch? | |
| 19:04:52 | sean-k-mooney | mriedem: no updstream if you trun off the numa toplogy filter to get passed it | |
| 19:05:03 | sean-k-mooney | when we rebuild but the toplogy change via the image | |
| 19:05:09 | mriedem | oh | |
| 19:05:14 | sean-k-mooney | we end up with the old numa toplogy and new image | |
| 19:05:29 | sean-k-mooney | so i want to put an upstream check to prevent that | |
| 19:05:38 | mriedem | ok so hear me out, | |
| 19:05:38 | sean-k-mooney | and backport a config option to opt out | |
| 19:05:44 | mriedem | the rebuild api takes new metadata for the server, | |
| 19:06:00 | dansmith | no | |
| 19:06:02 | dansmith | no | |
| 19:06:05 | mriedem | what if we have a skip_numa_topology_filter metadata that the user can pass in?! | |
| 19:06:09 | mriedem | and then the filter checks for that | |
| 19:06:11 | mriedem | and it's great | |
| 19:06:11 | dansmith | noooooooooo | |
| 19:06:13 | mriedem | problem solved | |
| 19:06:38 | mriedem | ok so the image takes random metadata, | |
| 19:06:39 | mriedem | so hear me out | |
| 19:06:57 | mriedem | os_skip_numa_filter_but_only_for_rebuild=true | |
| 19:07:04 | mriedem | we'll fix it in U | |
| 19:07:12 | dansmith | Well, officer, you should have heard what he was suggesting before I shot him... Hear me out... | |
| 19:07:23 | mriedem | that's justifiable homicide in some states | |
| 19:07:34 | dansmith | yeah | |
| 19:07:40 | artom | We fix it? No, U fix it | |
| 19:07:52 | sean-k-mooney | well the custromer has validated that tehy never us the image to change numa toplogy | |
| 19:07:53 | mriedem | get in line with bauzas and the bad jokes | |
| 19:08:03 | sean-k-mooney | so for them that skip on rebuild is valid | |
| 19:08:13 | sean-k-mooney | but i have aded https://bugs.launchpad.net/nova/+bug/1763766 to the downstrema bug | |
| 19:08:14 | openstack | Launchpad bug 1763766 in OpenStack Compute (nova) "nova needs to disallow resource consumption changes on image rebuild" [Medium,Triaged] | |
| 19:08:28 | sean-k-mooney | so what i am going to propose is i implemnt a check for that | |
| 19:08:44 | sean-k-mooney | and if and only if that passes we skip the numa toplogy filter | |
| 19:08:54 | sean-k-mooney | becasue not numa things changed | |
| 19:09:03 | sean-k-mooney | on rebuild only | |
| 19:09:54 | sean-k-mooney | and like the check we have to prevent numa migratio nwe can backport the saftey check for no resouce change on rebuild | |
| 19:10:17 | sean-k-mooney | and give peole a workaofund config option to disable it if they want to shot them selves in the foot | |
| 19:11:11 | sean-k-mooney | i have aslo said that if we dont fix it upstream im not fixing it downstream only | |
| 19:13:37 | sean-k-mooney | mriedem: dansmith does ^ sound fair. fixing https://bugs.launchpad.net/nova/+bug/1763766 is a prequisite for fixing https://bugs.launchpad.net/nova/+bug/1804502 safely | |
| 19:13:39 | openstack | Launchpad bug 1763766 in OpenStack Compute (nova) "nova needs to disallow resource consumption changes on image rebuild" [Medium,Triaged] | |
| 19:13:40 | openstack | Launchpad bug 1804502 in OpenStack Compute (nova) "Rebuild server with NUMATopologyFilter enabled fails (in some cases)" [Undecided,In progress] - Assigned to David Hill (david-hill-ubisoft) | |
| 19:14:03 | eandersson | Would it be possible to add an option to allow rebuilds to re-schedule the VM? | |
| 19:14:23 | eandersson | something like openstack server rebuild <id> --migrate/reschedule/bla | |
| 19:14:28 | sean-k-mooney | like evauate | |
| 19:14:46 | eandersson | I can't reproduce this as well | |
| 19:15:26 | sean-k-mooney | rebuild is ment to be posiblein virt driver that dont support migration | |
| 19:16:29 | sean-k-mooney | there is noting in principal that would disallow a driver form having rebiuld be to a different host like a cold migration but it is not how its currelty impelmeted | |
| 19:16:49 | sean-k-mooney | we would have to change the noop claim to a move claim if the inplcem rebuild did not succesed | |
| 19:17:29 | sean-k-mooney | or expose it at the api like you suggest and make the resuling to a different host user contolerale | |
| 19:18:52 | sean-k-mooney | ingoring numa for a second since image can have tratis we need to check with placmenet in anycase to see it the traits on the new image are valid on the current host | |
| 19:20:44 | sean-k-mooney | eandersson: what is the behavior you would like to see | |
| 19:21:11 | eandersson | Better error message at least :p | |
| 19:21:40 | eandersson | But not sure what the ideal would be. | |
| 19:21:50 | efried | stephenfin, sean-k-mooney, luyao, alex_xu: I'm +2 on everything vpmem except: | |
| 19:21:50 | efried | - https://review.opendev.org/#/c/678455/ where I'll proxy a vote from sean-k-mooney | |
| 19:21:50 | efried | - the bottom where I'll flip from -2 to +2 when we're ready | |
| 19:21:53 | eandersson | Maybe just documentation on how to avoid this on image builds. | |
| 19:22:02 | efried | I'm now going to sleep. | |
| 19:22:06 | eandersson | It would be nice as an admin to have a way to fix it at least. | |
| 19:22:33 | sean-k-mooney | eandersson: well you cant avoid it if the instance has a numa toplogy | |
| 19:22:44 | sean-k-mooney | you can work around it via cold/live migation | |
| 19:22:50 | sean-k-mooney | but that it | |
| 19:23:13 | eandersson | Maybe always force reschedule on rebuild when numa is a thing | |
| 19:23:37 | sean-k-mooney | rebuild are not allowed to change host today | |
| 19:23:42 | sean-k-mooney | so there is no rescudler | |
| 19:24:03 | sean-k-mooney | whats actully happening is the numa toplogy filter is treating the rebuild like a new isntance spawn | |
| 19:24:09 | sean-k-mooney | so you need double the resouces | |
| 19:24:26 | eandersson | oh so that is why some of them worked | |
| 19:24:32 | sean-k-mooney | yep | |
| 19:24:49 | sean-k-mooney | if the instance does not have a numa toplogy or there is enough space it works | |
| 19:25:35 | sean-k-mooney | we can actully now tell if its a rebuild in the filter code but we have not added special handeling for that in the numa toplogy filter | |
| 19:26:17 | sean-k-mooney | the correct thign to do is to check if the numa topldoy woudl change. reject the rebuild if its true or skip the numa toplogy filter if not | |