| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-11 | |||
| 15:04:35 | spatel | Help!! - I want to evacuate compute nodes because it has hardware issue, but all running vm using local-disk ( NOT shared disk ) can i do evacuate? | |
| 15:04:35 | alex_xu | stephenfin: no, that isn't. The reason do implicit NUMA topology for instance just due to someone asking that usecase in the PTG | |
| 15:04:38 | mriedem | artom: i don't think that's what dan suggested | |
| 15:04:49 | mriedem | "Computes are too old to do the smart thing, workaround is enabled, doing that" | |
| 15:04:59 | dansmith | spatel: see topic | |
| 15:05:13 | alex_xu | stephenfin: the explicity NUMA topology for instance and without affinity is clear plan in the PTG | |
| 15:05:15 | mriedem | you adjust the warning to say that computes are too old to do the supported method or whatever | |
| 15:05:23 | dansmith | spatel: you might notice that we're a day before FF and there are multiple dev-related discussions going on in here | |
| 15:05:53 | spatel | dansmith: sorry about that, you guys continue. | |
| 15:05:53 | luyao | stephenfin,alex_xu: what is implicit NUMA affinity | |
| 15:05:58 | stephenfin | alex_xu: It's sounding like the implications of that plan were not properly understood though. Certainly I didn't grasp them | |
| 15:07:01 | alex_xu | luyao: I guess that is just stephenfin typing wrong word, actually mean implicity NUMA topology | |
| 15:07:19 | alex_xu | stephenfin: ^ is that right? or you pointed to something I may not get | |
| 15:07:26 | mriedem | artom: i left a suggestion on wording | |
| 15:07:28 | stephenfin | luyao: Implicit NUMA affinity is what we do for hugepages and CPU pinning. If you use either of those features, your guest will have a one-node NUMA topology and it will be pinned to a host NUMA node | |
| 15:07:45 | stephenfin | alex_xu: Yeah, an implicit NUMA topology | |
| 15:08:19 | luyao | stephenfin: I thought implicit Alex said is the guest numa not bind to host numa | |
| 15:08:24 | mriedem | stephenfin: oh i have a solution - drop the numa part but only support vpmems on ppc64 https://github.com/libvirt/libvirt/blob/master/src/qemu/qemu_domain.c#L11604-L11615 :) | |
| 15:08:32 | mriedem | power to the rescue | |
| 15:08:47 | stephenfin | fight the power | |
| 15:08:51 | luyao | stephenfin, alex_xu: so you are discuss different things ? | |
| 15:09:15 | mriedem | does tonyb have openshift working on power yet? | |
| 15:09:23 | stephenfin | luyao: no, the disagreement is on the lack of pinning | |
| 15:09:45 | artom | mriedem, https://review.opendev.org/#/c/640021/50/nova/conductor/tasks/live_migrate.py@200 | |
| 15:10:40 | stephenfin | the fact that for hugepages, the single guest NUMA cell is pinned to a host NUMA cell, but for VPMEM, the single guest NUMA cell floats across all host NUMA cells | |
| 15:10:48 | stephenfin | *for hugepages and CPU pinning | |
| 15:11:19 | alex_xu | stephenfin: yes, so you want the pinning, but is it ok without affinity for now? | |
| 15:11:25 | stephenfin | bauzas: Think you could hit these two too? https://review.opendev.org/#/c/681060/ https://review.opendev.org/#/c/681061/ | |
| 15:11:43 | stephenfin | alex_xu: I'll take that as a middle ground, yes | |
| 15:12:04 | bauzas | stephenfin: looking | |
| 15:12:15 | alex_xu | stephenfin: so the only I need is fall into this branch https://github.com/openstack/nova/blob/master/nova/virt/hardware.py#L1740, right | |
| 15:12:24 | stephenfin | yup | |
| 15:12:28 | alex_xu | cool | |
| 15:12:57 | mriedem | artom: replied | |
| 15:13:02 | mriedem | with anger | |
| 15:13:07 | luyao | stephenfin: Does it make sense to do the numa pinning now without affinity. | |
| 15:13:39 | sean-k-mooney | luyao: yes | |
| 15:13:47 | stephenfin | luyao: It's definitely sub-optimal, I agree, but it's better than breaking our long-standing one guest NUMA node == one host NUMA node policy | |
| 15:14:01 | sean-k-mooney | its not ideal but its not a bad thing | |
| 15:14:01 | artom | mriedem, ok, there's something I genuinely don't get then | |
| 15:14:09 | luyao | sean-k-mooney: Okay | |
| 15:14:14 | stephenfin | you'll get inferior performance but from a user perspective, things behavior as expected | |
| 15:14:20 | artom | Not trying to make you angry on purpose, for serious | |
| 15:14:32 | stephenfin | *behave | |
| 15:14:59 | artom | If the conductor's train, and the source is train, and the dest is train, but there are other stein computes in the cell | |
| 15:15:15 | artom | We'll get past the min sevice version check | |
| 15:15:39 | artom | get past = not return | |
| 15:15:49 | artom | Meaning we check the value of the workaround | |
| 15:15:54 | artom | If the workaround is disabled, that's that | |
| 15:16:22 | artom | If the workaround is enabled, in *this specific case of dest and source being new* | |
| 15:16:27 | artom | We'll get NUMA LM | |
| 15:17:47 | mriedem | artom: even if the source is new and you checked that, the dest - which we might not have yet if the user didn't force the dest from the api and bypass the scheduler - might not be new, and we won't know in that code with the warning log message because it comes before we ask the scheduler for a dest | |
| 15:18:15 | mriedem | so if we get lucky and the source host and selected dest host are new enough, yeah it might be fine | |
| 15:18:24 | mriedem | but, | |
| 15:18:26 | dansmith | mriedem: I don't think that's whathe's saying | |
| 15:18:29 | mriedem | it probably won't be because if you have old computes, | |
| 15:18:42 | mriedem | rpc is going to be pinned and we'll backlevel the new numa migrate_data object stuff | |
| 15:18:56 | dansmith | this ^ is what he's saying | |
| 15:19:14 | dansmith | so he just doesn't want to specifically say that the src or dst is too old.. right? | |
| 15:19:23 | mriedem | we never asked him to say that | |
| 15:19:35 | mriedem | we just said "say something about computes not being upgraded yet" | |
| 15:19:38 | dansmith | I don't think he claimed we did | |
| 15:19:46 | mriedem | who's on first? | |
| 15:19:55 | dansmith | I think he just said he's trying to wordsmith a suitably generic message right? | |
| 15:20:03 | mriedem | i provided one | |
| 15:20:16 | artom | Well there's the logging on L196 as well - it all fits together | |
| 15:20:32 | artom | Or I'm just overcomplicating everything? | |
| 15:20:50 | dansmith | I don't see any logging on L 196 | |
| 15:21:14 | artom | You asked for it :) | |
| 15:21:40 | mriedem | artom: i think dan just meant adjusting the msg object | |
| 15:21:40 | dansmith | you mean the logging that would be on L196 if you updated? | |
| 15:21:53 | mriedem | if the config is enabled, we log a warning | |
| 15:21:58 | mriedem | else we raise, but with the same message | |
| 15:22:21 | dansmith | mriedem: I was suggesting one more log message inside the service version check | |
| 15:23:10 | mriedem | ok i'm going to move onto reviewing gibi's stuff now | |
| 15:23:14 | artom | Haha | |
| 15:23:45 | dansmith | how about artom just fixes the test issue, adds logs like he thinks he needs, and then we can sort out the log minutia in a later patch if necessary? | |
| 15:24:20 | bauzas | stephenfin: good news, I now start looking at your reshaper change :) | |
| 15:24:22 | mriedem | that's fine | |
| 15:24:30 | stephenfin | huzzah! | |
| 15:26:15 | gibi_fly | mriedem: hit me with any questions | |
| 15:27:14 | artom | dansmith, btw, the compute RPC version is per-cell, right? So, if we're in a cell with mixed stein/train computes, none of the will be able to send 5.3? | |
| 15:27:46 | artom | There's no backporting like what's done for objects - if a node can send 5.3, then all others must be able to receive it? | |
| 15:27:48 | dansmith | artom: not really, but I definitely do not think that level of detail needs to be in there | |
| 15:27:49 | efried | sean-k-mooney: https://review.opendev.org/#/c/681474/2 appears not to have worked -- | |
| 15:27:49 | efried | http://52.27.155.124/74/681474/2/check/pmem-tempest-plugin-filtered/134b243/job-output.txt.gz @ 2019-09-11 14:17:14.914120 is still applying PS13 | |
| 15:28:05 | artom | dansmith, yeah, maybe I'm overthinking it | |
| 15:29:27 | efried | sean-k-mooney, dtroyer: I can't tell by looking at https://github.com/intel/Intel-OpenStack-CI-jobs/pull/1/commits/5d4a998a890fef3a2b0a46f73d26aa1deeb3cd8b -- is this merged? | |
| 15:29:29 | sean-k-mooney | efried in that case they are likely using the git plug not the gitub plugin | |
| 15:30:09 | sean-k-mooney | efried: the zuul git driver does not support pull request but works with any git repo via a timer | |
| 15:30:27 | sean-k-mooney | the gitub one require the zuul app to be installed in the github org | |
| 15:30:35 | sean-k-mooney | but once that is done pull requeist work | |
| 15:30:47 | sean-k-mooney | i suspect intel it dont allow the zuul app | |
| 15:30:49 | efried | sean-k-mooney, dtroyer: I wanted to just recheck one of the top vpmem patches to get new results | |
| 15:31:03 | efried | but no point unless that pr merged | |
| 15:32:04 | mriedem | gibi_fly: | |
| 15:32:05 | mriedem | https://review.opendev.org/#/c/676980/19 | |
| 15:32:11 | gibi_fly | mriedem: looking | |
| 15:33:22 | bauzas | I believe I can gibi_fly | |
| 15:33:32 | bauzas | (sorry) | |