| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-11 | |||
| 17:50:23 | gibi_bus | mriedem: I will ping you when it is up | |
| 17:51:17 | mriedem | ok, then i'm going to get lunch | |
| 17:52:20 | gibi_bus | mriedem: have a nice one! | |
| 18:03:34 | efried | o/ dolpher :) | |
| 18:07:19 | dolpher | efried: Hello, seems the filter is not working, no one is triggering the CI job now | |
| 18:07:39 | dolpher | Hi donnyd :) | |
| 18:19:30 | donnyd | sean-k-mooney: I didn't hear back on the labels.. but i hope this https://review.opendev.org/#/c/634827/ isn't waiting on jobs for FN | |
| 18:19:58 | donnyd | I submitted a bit more for resources, kinda need it right now | |
| 18:20:06 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Support migrating SRIOV port with bandwidth https://review.opendev.org/676980 | |
| 18:20:11 | gibi_bus | mriedem: ^^ | |
| 18:21:20 | gibi_bus | bauzas: for tomorrow's coffee ^^ | |
| 18:22:03 | sean-k-mooney | donnyd: no but the jobs got canceld | |
| 18:22:27 | sean-k-mooney | the 4 build i had in flight got killed because of a rebase/fix of the patches | |
| 18:22:51 | sean-k-mooney | i can start them again and see what happens | |
| 18:24:33 | openstackgerrit | sean mooney proposed openstack/nova master: multi numa nfv testing job https://review.opendev.org/679656 | |
| 18:24:48 | openstackgerrit | sean mooney proposed openstack/nova master: [DNM] testing with new lable. https://review.opendev.org/680738 | |
| 18:25:44 | sean-k-mooney | donnyd: just rebased those via the gerrit ui that shoudl run with the old and new lable | |
| 18:26:04 | donnyd | sean-k-mooney: ok cool | |
| 18:39:00 | efried | dolpher: Can we kill the filter for now? | |
| 18:39:28 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Allow migrating server with port resource request https://review.opendev.org/671497 | |
| 18:40:52 | dolpher | https://github.com/intel/Intel-OpenStack-CI-jobs/pull/4 | |
| 18:41:26 | dolpher | efried: send the pull request to remove the filter, and I'll ping Matt to merge it. | |
| 18:41:34 | efried | okay | |
| 18:42:03 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Do not query allocations twice in finish_revert_resize https://review.opendev.org/678827 | |
| 18:44:36 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Allow resizing server with port resource request https://review.opendev.org/679019 | |
| 18:52:00 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Extract pf$N literals as constants from func test https://review.opendev.org/680991 | |
| 18:52:36 | openstackgerrit | Luyao Zhong proposed openstack/nova master: Retrieve the allocations early https://review.opendev.org/678450 | |
| 18:52:37 | openstackgerrit | Luyao Zhong proposed openstack/nova master: Claim resources in resource tracker https://review.opendev.org/678452 | |
| 18:52:37 | openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Enable driver discovering PMEM namespaces https://review.opendev.org/678453 | |
| 18:52:38 | openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: report VPMEM resources by provider tree https://review.opendev.org/678454 | |
| 18:52:38 | openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Support VM creation with vpmems and vpmems cleanup https://review.opendev.org/678455 | |
| 18:52:39 | openstackgerrit | Luyao Zhong proposed openstack/nova master: Parse vpmem related flavor extra spec https://review.opendev.org/678456 | |
| 18:52:39 | openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Enable driver configuring PMEM namespaces https://review.opendev.org/679640 | |
| 18:52:40 | openstackgerrit | Luyao Zhong proposed openstack/nova master: Add functional tests for virtual persistent memory https://review.opendev.org/678470 | |
| 18:52:40 | openstackgerrit | Luyao Zhong proposed openstack/nova master: objects: use all_things_equal from objects.base https://review.opendev.org/681397 | |
| 18:54:34 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Improve dest service level func tests https://review.opendev.org/680998 | |
| 18:56:52 | eandersson | Having an odd bug - a VM can't be rebuilt with a new image. Pretty sure placement is filtering out all the potential matches. | |
| 18:57:02 | eandersson | Rebuilding it with the same image works fine. | |
| 18:57:05 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Follow up for Ib50b6b02208f5bd2972de8a6f8f685c19745514c https://review.opendev.org/681490 | |
| 18:57:41 | mriedem | eandersson: which release? | |
| 18:57:46 | eandersson | Running Rocky | |
| 18:57:58 | eandersson | Very latest stable/rocky | |
| 18:58:26 | eandersson | I see that the scheduler gets a single result, but it is failing on NUMA Topology. | |
| 18:58:32 | mriedem | we shouldn't be calling placement for rebuild https://github.com/openstack/nova/blob/stable/rocky/nova/scheduler/manager.py#L126 | |
| 18:58:46 | eandersson | > 'NUMATopologyFilter: (start: 1, end: 0)' | |
| 18:58:51 | eandersson | Ah bad assumption | |
| 18:58:53 | mriedem | https://bugs.launchpad.net/nova/+bug/1763766 ? | |
| 18:58:54 | openstack | Launchpad bug 1763766 in OpenStack Compute (nova) "nova needs to disallow resource consumption changes on image rebuild" [Medium,Triaged] | |
| 18:59:14 | mriedem | or https://bugzilla.redhat.com/show_bug.cgi?id=1700412 | |
| 18:59:15 | openstack | bugzilla.redhat.com bug 1700412 in openstack-nova "[RFE] Rebuild with a different image fails due to NUMATopologyFilter" [High,New] - Assigned to nova-maint | |
| 18:59:37 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Skip querying resource request if no qos port https://review.opendev.org/681513 | |
| 18:59:46 | eandersson | Yep looks like it | |
| 18:59:52 | dansmith | yeah if the image is what causes it trouble, | |
| 19:00:07 | dansmith | it's likely the new numa topo filtering out the one host you're on | |
| 19:00:37 | mriedem | sean-k-mooney has to fix it and get that backported to queens | |
| 19:02:19 | artom | mriedem, it's not getting backported, but one of the things we discussed was making that RUN_ON_REBUILD flag we added way back when for the rebuild CVE configurable in some way | |
| 19:02:26 | artom | We kinda dreaded coming to upstream with that | |
| 19:02:49 | sean-k-mooney | dansmith: for the downstream issue yes its just the numa toploy filter filtering it out beacue tehre is no space for the rebuild | |
| 19:03:08 | dansmith | sean-k-mooney: I'm talking about eandersson's case | |
| 19:03:25 | mriedem | artom: we don't claim on rebuild so if the numa topo changes in the new image we won't claim for it properly, | |
| 19:03:33 | mriedem | so you could just be kicking the error down to compute | |
| 19:03:33 | sean-k-mooney | ill read scollback | |
| 19:03:43 | sean-k-mooney | mriedem: correct | |
| 19:03:54 | artom | mriedem, yeah, it wouldn't be a correct fix | |
| 19:04:04 | sean-k-mooney | we rebuild with the new image but old toplogy | |
| 19:04:23 | artom | More of a candy we throw our customers on, like, 10 to placate them until they an upgrade to whatever release the un-backportable fix lands in | |
| 19:04:23 | mriedem | sean-k-mooney: you mean in some downstream-only patch? | |
| 19:04:52 | sean-k-mooney | mriedem: no updstream if you trun off the numa toplogy filter to get passed it | |
| 19:05:03 | sean-k-mooney | when we rebuild but the toplogy change via the image | |
| 19:05:09 | mriedem | oh | |
| 19:05:14 | sean-k-mooney | we end up with the old numa toplogy and new image | |
| 19:05:29 | sean-k-mooney | so i want to put an upstream check to prevent that | |
| 19:05:38 | mriedem | ok so hear me out, | |
| 19:05:38 | sean-k-mooney | and backport a config option to opt out | |
| 19:05:44 | mriedem | the rebuild api takes new metadata for the server, | |
| 19:06:00 | dansmith | no | |
| 19:06:02 | dansmith | no | |
| 19:06:05 | mriedem | what if we have a skip_numa_topology_filter metadata that the user can pass in?! | |
| 19:06:09 | mriedem | and then the filter checks for that | |
| 19:06:11 | mriedem | and it's great | |
| 19:06:11 | dansmith | noooooooooo | |
| 19:06:13 | mriedem | problem solved | |
| 19:06:38 | mriedem | ok so the image takes random metadata, | |
| 19:06:39 | mriedem | so hear me out | |
| 19:06:57 | mriedem | os_skip_numa_filter_but_only_for_rebuild=true | |
| 19:07:04 | mriedem | we'll fix it in U | |
| 19:07:12 | dansmith | Well, officer, you should have heard what he was suggesting before I shot him... Hear me out... | |
| 19:07:23 | mriedem | that's justifiable homicide in some states | |
| 19:07:34 | dansmith | yeah | |
| 19:07:40 | artom | We fix it? No, U fix it | |
| 19:07:52 | sean-k-mooney | well the custromer has validated that tehy never us the image to change numa toplogy | |
| 19:07:53 | mriedem | get in line with bauzas and the bad jokes | |
| 19:08:03 | sean-k-mooney | so for them that skip on rebuild is valid | |
| 19:08:13 | sean-k-mooney | but i have aded https://bugs.launchpad.net/nova/+bug/1763766 to the downstrema bug | |
| 19:08:14 | openstack | Launchpad bug 1763766 in OpenStack Compute (nova) "nova needs to disallow resource consumption changes on image rebuild" [Medium,Triaged] | |
| 19:08:28 | sean-k-mooney | so what i am going to propose is i implemnt a check for that | |
| 19:08:44 | sean-k-mooney | and if and only if that passes we skip the numa toplogy filter | |
| 19:08:54 | sean-k-mooney | becasue not numa things changed | |
| 19:09:03 | sean-k-mooney | on rebuild only | |
| 19:09:54 | sean-k-mooney | and like the check we have to prevent numa migratio nwe can backport the saftey check for no resouce change on rebuild | |