| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-11 | |||
| 17:25:18 | sean-k-mooney | all the numa jobs just got killed | |
| 17:25:52 | artom | mriedem, dansmith ^^ | |
| 17:26:13 | efried | alex_xu: Isn't this the famous "doubled allocations" bug? | |
| 17:26:30 | alex_xu | efried: no, it isn't | |
| 17:27:07 | alex_xu | efried: when a host enable same host resize, and that host is the only host has available PCPU now. | |
| 17:27:22 | sean-k-mooney | alex_xu: cant ever ignore the source host for same host reszie or rebuild | |
| 17:27:40 | efried | yeah, I would think for same-host resize we would want to restrict to *only* the current host | |
| 17:27:53 | alex_xu | efried: we will try to allocate PCPU in placement, and yes, placement tell you the source host has PCPU, but later the scheduler will check ignore host, the source host is in the ignore host. so the scheduling failed. | |
| 17:27:53 | efried | ...having calculated the positive difference from our current allocations | |
| 17:28:06 | dansmith | efried: you don't know you're doing a same-host resize | |
| 17:28:19 | efried | oh, bummer | |
| 17:28:23 | alex_xu | sean-k-mooney: sorry, I mean the enable same host resize | |
| 17:28:27 | alex_xu | oops | |
| 17:28:35 | alex_xu | disalbe the same host resize | |
| 17:28:40 | sean-k-mooney | same host resize with pcpus is way more involved | |
| 17:28:47 | efried | so what we should really do is two GET /a_c calls | |
| 17:28:48 | sean-k-mooney | well not pcpu numa in general | |
| 17:29:24 | efried | and then combine the results. | |
| 17:29:24 | efried | one with in_tree=$cur_host with the diff'd set of resources | |
| 17:29:24 | efried | one with in_tree=!$cur_host with the full set of resources | |
| 17:29:56 | mriedem | umm, you're not talking about adding that into the series for train now are you? | |
| 17:30:05 | sean-k-mooney | that still has the double allocation proablem | |
| 17:30:12 | alex_xu | dansmith: I know the ignored_host from request spec, so I can check that, if the allocation_candidates return the host in the ignore_hosts, then fallback to VCPU | |
| 17:30:17 | efried | sean-k-mooney: how would it? | |
| 17:30:23 | mriedem | are you essentially trying to say, avoid the same host for a resize if the instance has PCPU resources? | |
| 17:30:35 | dansmith | alex_xu: you want to fall back to vcpu allocation if the only host that gets returned is the one you're on? | |
| 17:30:39 | dansmith | alex_xu: that's....crazy | |
| 17:30:54 | sean-k-mooney | the in_tree=$cur_host call would require double the resouces | |
| 17:31:01 | sean-k-mooney | the current resouce and the resize to resouces | |
| 17:31:07 | dansmith | alex_xu: or are you saying fall back to vcpu allocation but with the current host excluded? | |
| 17:31:26 | sean-k-mooney | anyway the numa toplogy filter requires double the resouces so its not a new thing | |
| 17:31:32 | efried | sean-k-mooney: no, that's why the in_tree=$cur_host is made with the *diff* of the current and new resources (only the positive ones) | |
| 17:31:45 | efried | (this is all theoretical btw) | |
| 17:31:52 | alex_xu | dansmith: fallback to vcpu if the only host get returned is the one you are on. After fallback vcpu, I won't get VCPu availalbe on the source host again | |
| 17:32:01 | sean-k-mooney | efried: i mention this on irc last week | |
| 17:32:06 | sean-k-mooney | in placment channel | |
| 17:32:23 | sean-k-mooney | efried: i have a customer request to fix inplemce rebuild with numa toplogy | |
| 17:32:37 | efried | sean-k-mooney: Yeah, I thought it sounded familiar. | |
| 17:32:39 | sean-k-mooney | so i or someone on my team needs to actully make this work in U | |
| 17:32:47 | cdent | i confirm that in_tree=! hasn't happened yet (sorry was away from computer for a few minutes doing network management) | |
| 17:32:53 | sean-k-mooney | and backport somthi esle to queens | |
| 17:33:01 | mriedem | sean-k-mooney: there is another nickel | |
| 17:33:18 | mriedem | sean-k-mooney: remember when you said you'd make sriov pci device live migration claims part of the actual claims code in U too/ | |
| 17:33:19 | mriedem | ? | |
| 17:33:32 | sean-k-mooney | mriedem: yes but this time we have the customer wanting use to ship them the backported code in october | |
| 17:33:34 | dansmith | U is going to be a big cycle.. HUGE. YUGE. | |
| 17:33:42 | dtroyer | efried, sean-k-mooney, alex_xu: status update: we've re-triggered the pmem job on Intel CI with the changes on PS22, should have some results in about 45 min… | |
| 17:33:52 | mriedem | sean-k-mooney: backported code for PCPU? | |
| 17:33:54 | efried | dansmith: no it's not. | |
| 17:33:58 | alex_xu | dtroyer: thanks | |
| 17:33:58 | mriedem | including all of the upgrade shit that entails? | |
| 17:33:59 | dtroyer | https://review.opendev.org/#/c/679640/22 | |
| 17:34:02 | sean-k-mooney | mriedem: no inplace numa vma rebuild | |
| 17:34:10 | efried | thanks dtroyer | |
| 17:34:24 | mriedem | inplace numa vma rebuild? | |
| 17:34:25 | efried | it appears that 'pleaselookagain' is the magic invocation for pmem CI | |
| 17:34:37 | mriedem | i'm lost - i don't know what that has to do with what we're talking about | |
| 17:34:47 | alex_xu | mriedem: https://review.opendev.org/#/c/681383/5/nova/scheduler/manager.py@155 | |
| 17:34:59 | alex_xu | mriedem: or you can see the PS2 comment I give | |
| 17:35:48 | sean-k-mooney | mriedem: https://bugzilla.redhat.com/show_bug.cgi?id=1700412 | |
| 17:35:49 | efried | dansmith: If the PTL has anything to say about it, we're going to accept only blueprints that were already approved or proposed-but-didn't-make-spec-freeze | |
| 17:35:50 | openstack | bugzilla.redhat.com bug 1700412 in openstack-nova "[RFE] Rebuild with a different image fails due to NUMATopologyFilter" [High,New] - Assigned to nova-maint | |
| 17:35:51 | mriedem | "if same-host resize is disabled and the only host available is the host you're on there are resource problems" is not a new issue | |
| 17:36:13 | mriedem | ^ is also a problem for servers in a strict affinity group | |
| 17:36:18 | mriedem | since you can't resize out of that | |
| 17:36:27 | dansmith | efried: good luck with that | |
| 17:36:46 | efried | you can be part of the solution | |
| 17:36:46 | sean-k-mooney | mriedem: no its not a new proablem at all | |
| 17:36:53 | mriedem | i think i tried that strategy back in like ocata | |
| 17:36:54 | dansmith | efried: don't want to | |
| 17:37:13 | alex_xu | mriedem: okay... | |
| 17:37:18 | dansmith | efried: we've approved things I don't think we should do almost every cycle, so...limiting to doing things we've "approved" is counterproductive, IMHO | |
| 17:37:21 | alex_xu | dansmith: yes, you can be :) | |
| 17:38:11 | mriedem | sean-k-mooney: so https://bugs.launchpad.net/nova/+bug/1763766 ? | |
| 17:38:12 | openstack | Launchpad bug 1763766 in OpenStack Compute (nova) "nova needs to disallow resource consumption changes on image rebuild" [Medium,Triaged] | |
| 17:38:24 | artom | mriedem, fwiw, the SRIOV live migration thing can be me | |
| 17:38:36 | alex_xu | dansmith: back to preivous, why I check ignore_host is crazy... | |
| 17:38:37 | sean-k-mooney | mriedem: its related | |
| 17:39:08 | dansmith | alex_xu: what I said is crazy is not what you are actually suggesting I think | |
| 17:39:10 | mriedem | artom: i've resigned that it's never going to happen so was just using it as an example of another "we'll do it in U" thing | |
| 17:39:36 | sean-k-mooney | for older release i was planing to check for that and allow skiping the numa topology filter if there were no changes in resouce usaage or numa toplogy on rebuild only. | |
| 17:39:36 | alex_xu | dansmith: ok...i'm safe | |
| 17:39:45 | artom | mriedem, I mean, I followed through on device tagging attachments, back in the day, didn't I? | |
| 17:39:52 | alex_xu | stephenfin: so...check that https://review.opendev.org/#/c/681383/5/nova/scheduler/manager.py@155, if that is solution for you | |
| 17:39:55 | artom | Though I did drop the ball in placement for osc live migration | |
| 17:40:20 | mriedem | artom: you had a business justification for tagged attachments right? | |
| 17:40:25 | artom | mriedem, - | |
| 17:40:27 | dansmith | artom: you gonna wait for the last three weeks of the U cycle for that one too? | |
| 17:40:28 | artom | Err, 0 | |
| 17:40:35 | mriedem | customers aren't asking that you work on sriov pci device claims architecture | |
| 17:40:41 | sean-k-mooney | mriedem: of corse we will only do dthat if we fix it upstream and block it upstream first | |
| 17:40:49 | artom | It was purely a "let's be good community citizens" thing | |
| 17:41:01 | mriedem | i'm not sure what "placement for osc live migration" is | |
| 17:41:10 | stephenfin | alex_xu: I'm not sure you get you | |
| 17:41:21 | artom | mriedem, err, I meant the osc live migration refactor | |
| 17:41:23 | mriedem | stephenfin: obligatory smiley face please | |
| 17:41:30 | mriedem | artom: i got that | |
| 17:41:34 | mriedem | and the --boot-from-volume sugar | |
| 17:41:40 | artom | I didn't even do the device tagging for it | |
| 17:41:46 | artom | Which we *did* have an ask for | |