| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-11-20 | |||
| 13:45:43 | sean-k-mooney | rahter then a change | |
| 13:45:49 | mnaser | sean-k-mooney: right but at least the guest will be aware there is possible latency | |
| 13:45:54 | sean-k-mooney | yep | |
| 13:45:55 | mnaser | yep, indeed, its not optimal, but its an improvement | |
| 13:46:46 | sean-k-mooney | so the reason i suggested disabling the multiple numa nodes in zen 1 was our pms insisting vnf dont understand numa | |
| 13:47:08 | sean-k-mooney | after 5+ years of eveyone explaiming what numa is to vnf vendors i dont really buy that | |
| 13:47:57 | mnaser | im wondering which would be better: NPS=4 + no NUMA per L3 + 1 numa node .. vs .. NPS=4 + NUMA per L3 + 2 numa nodes -- im thinking the latter will likely be much faster (or at least the VM will be 'smarter' at understanding the topology.. somewhat) | |
| 13:48:26 | mnaser | it might be unpredictable because the NUMA node might have a distance of 32, or 12, or 10 .. but at least it might know there is distance | |
| 13:48:40 | sean-k-mooney | the later i think would optimize the memory latency | |
| 13:50:21 | mnaser | im gathering some benchmark info | |
| 13:50:29 | mnaser | and then ill try the two numa node + l3 per numa | |
| 13:50:32 | sean-k-mooney | exposing NUMA per L3 should give the best performance it just requires your vms to have multiple numa nodes if they have more then 3-4 cores or >32-64G of ram | |
| 13:50:53 | mnaser | yep, most of the VMs will be 8 core / 8 gb | |
| 13:51:10 | mnaser | also FWIW those aren't dedicated cores | |
| 13:51:39 | sean-k-mooney | sure you just setting hw:numa_nodes=2 but not enabling pinning | |
| 13:51:48 | aarents | hi dansmith, can you confirm that the last update is fine for you ? https://review.opendev.org/#/c/670000 | |
| 13:51:57 | mnaser | yep, os that way im thinknig nova should do 4 cores per NUMA, 4gb in each numa node | |
| 13:52:03 | sean-k-mooney | if you do that you should also set hw:mem_page_size to either large or small | |
| 13:52:27 | sean-k-mooney | mnaser: yes it would | |
| 13:53:00 | sean-k-mooney | the reason for doing hw:mem_page_size by the way is you want to enable novas numa aware memory tracking | |
| 13:53:24 | sean-k-mooney | if your dont want to use hugepages set it to "small" | |
| 13:53:39 | sean-k-mooney | that will use 4k pages | |
| 13:53:46 | mnaser | sean-k-mooney: i think huge pages is soemthing id want, but i think transparent huge pages are enabled right now | |
| 13:54:26 | sean-k-mooney | right but you should still set hw:mem_page_size=small if you have a numa toplogy and are not using explcit hugepages | |
| 13:54:33 | sean-k-mooney | if you dont you can get OOM events | |
| 13:54:45 | mnaser | i think id probably want large anyways, host has 512G memory and plenty of reserved memoryt | |
| 13:55:12 | sean-k-mooney | ya if you dont allow memory oversubsription then there is no reason not to use hugepages | |
| 13:55:32 | mnaser | ya nope no oversubscription | |
| 13:58:52 | mnaser | epyc rome numbers https://www.irccloud.com/pastebin/3bgON0HX/ | |
| 13:59:18 | mnaser | intel (but much older hardware) https://www.irccloud.com/pastebin/6M7B4QLY/ | |
| 13:59:22 | mnaser | sean-k-mooney: ^ fyi, interesting to see latency numbers | |
| 14:01:05 | coreycb | kashyap: do you know if anything has landed for removing cpu flags, similar to cpu_model_extra_flags? | |
| 14:01:28 | kashyap | coreycb: Hi...afraid, no; but I've filed a Blueprint for it: | |
| 14:01:55 | kashyap | coreycb: https://blueprints.launchpad.net/nova/+spec/allow-disabling-cpu-flags | |
| 14:02:30 | coreycb | kashyap: thanks | |
| 14:03:40 | coreycb | kashyap: fyi this bug is why I'm asking: https://bugs.launchpad.net/bugs/1853200 | |
| 14:03:40 | openstack | Launchpad bug 1853200 in libvirt (Ubuntu) "cpu features hle and rtm disabled for security are present in /usr/share/libvirt/cpu_map.xml" [High,Confirmed] - Assigned to Ubuntu Security Team (ubuntu-security) | |
| 14:03:44 | kashyap | coreycb: I know :-( | |
| 14:03:51 | kashyap | And I guessed as much | |
| 14:04:35 | kashyap | coreycb: One (very valid) 'workaround' is that have QEMU add new "named CPU models" to remove the said flags. | |
| 14:05:00 | coreycb | kashyap: ok I'll mention that, thanks | |
| 14:05:43 | kashyap | coreycb: For that, upstream QEMU folks must add them...please file a QEMU "RFE" bug on launchpad for it | |
| 14:06:03 | sean-k-mooney | mnaser: did you enable cluster on die for the intel system out of interest | |
| 14:06:28 | coreycb | kashyap: thanks I'll pass the details on to cpaelzer, he's our qemu maintainer | |
| 14:06:39 | kashyap | (Nod) | |
| 14:06:50 | sean-k-mooney | mnaser: but yes the amd number seam to be much higher | |
| 14:07:17 | sean-k-mooney | mnaser: that is quite surprising to be honest | |
| 14:39:15 | efried | bauzas, gibi, stephenfin: would one of you please have a look at the nova/cyborg spec update https://review.opendev.org/#/c/684151/ and +A if appropriate? | |
| 14:39:35 | bauzas | efried: yeah, I can do | |
| 14:39:40 | efried | thanks bauzas | |
| 14:39:48 | stephenfin | I didn't review the original so I'll defer to others | |
| 14:39:55 | bauzas | I promised to do spec reviews, in particular following the ones we discussed at the PTG | |
| 14:39:58 | efried | After all, melissaml and Viere are +1. | |
| 14:45:43 | efried | dustinc: Did you pick up what I laid down yesterday about compute node conflicts for provider config? http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2019-11-19.log.html#t2019-11-19T22:19:41 | |
| 14:45:59 | aarents | Hi guys I need some review on this https://review.opendev.org/#/c/678016/ gibi already put +2, if you can have a look ? | |
| 14:46:55 | bauzas | aarents: /me clicks | |
| 14:47:24 | efried | dansmith: TL;DR: since, ironic notwithstanding, it would be insane for one compute node's $name to be the same as another's $uuid, we're just going to punt if there's a conflict across the whole $name+$uuid space. With that in play, it makes sense to treat $COMPUTE_NODE as a "default", which can be overridden by a specific $name/$uuid. And all that means we can detect conflicts immediately (on startup) and fail (to start up) if o | |
| 14:48:40 | dansmith | buffer overflow | |
| 14:50:10 | aarents | thks | |
| 14:51:21 | sean-k-mooney | efried: ill try and review the cyborg spec shortly | |
| 14:51:39 | efried | sean-k-mooney: thanks. It's just an update to the existing spec, nothing earth-shaking. | |
| 14:52:02 | dansmith | efried: your message was too long and was cut off after "fail to (start up)" | |
| 14:52:04 | sean-k-mooney | ya looks short | |
| 14:52:28 | efried | dansmith: oh, weird. "...fail (to start up) if one is detected." | |
| 14:52:33 | efried | that was all. | |
| 14:52:51 | efried | my buffer says I had another 25c | |
| 14:52:56 | efried | clearly I need to work on my TL;DRing. | |
| 14:53:11 | dansmith | efried: okay, so I think I was (and likely still am) missing some context yesterday and I was hurrying, | |
| 14:53:34 | dansmith | but the problem is that ironic nodes specified there may not exist as providers on startup, is that it? | |
| 14:53:53 | efried | that's pretty much the only scenario that makes us have to deal with this, yeah. | |
| 14:54:23 | dansmith | so we wanted to make that hard-fail if you provide something that doesn't exist there yeah? | |
| 14:54:28 | efried | no | |
| 14:54:48 | efried | generically the "problem" is that theoretically you can specify a config by $name and another by $uuid, but those are for the same node, but we wouldn't know that until that node "appeared". | |
| 14:55:04 | efried | In practice this shouldn't be possible because for ironic, $name == $uuid. | |
| 14:56:03 | dansmith | you mean if you had two entries in the list, one by name and one by uuid and they were in fact the same provider? | |
| 14:56:08 | efried | right | |
| 14:56:44 | dansmith | that seems like a really tiny detail to be concerned about.. did this come up in some testing or something? | |
| 14:56:53 | efried | more code inspection | |
| 14:57:06 | dansmith | remind me why we have the by-name option anyway? | |
| 14:57:10 | efried | ikr | |
| 14:57:27 | dansmith | uuid or the "self" option should really be all we need I would think | |
| 14:57:30 | efried | It was because you don't necessarily know the UUID yet in a green field. | |
| 14:57:44 | efried | but you want your tripleo to be able to lay down the config a priori. | |
| 14:57:46 | dansmith | in which case, non-ironic? in that case you use $COMPUTE_NODE yeah? | |
| 14:58:12 | efried | Yeah, I would think so. gibi, can we convince you we don't need identification by name? | |
| 14:58:23 | efried | (I think it was gibi who talked us into it) | |
| 14:58:26 | dansmith | either way, | |
| 14:58:46 | dansmith | I'm not sure we really need to care that much if someone puts a thing in there twice | |
| 14:59:12 | efried | right, you kinda f'ed up if that happens, but the code has to do *something* in that case. | |
| 14:59:21 | dansmith | *if* we can detect and log an error later that's helpful, but.. | |
| 14:59:34 | dansmith | your concern is just not being able to detect that at startup? | |
| 14:59:40 | efried | right. | |
| 14:59:51 | gibi | efried: I'm sort on context. Is it about provider config? | |
| 15:00:05 | efried | if we can detect it at startup, we can halt the service and force you to fix it. But we don't want to kill the service if it creeps in after we've already started running. | |
| 15:00:07 | efried | gibi: yes | |
| 15:00:16 | dansmith | efried: if it's hard/impossible to do at start, is clearly wrong config, and we can log an error later I think that's reasonable | |
| 15:00:29 | dansmith | efried: no, don't halt.. log an error periodically | |
| 15:00:36 | efried | dansmith: ack | |
| 15:00:41 | gibi | efried: so in case of a compute RP the name is known before the compute was ever started but the uuid only known after the compute creates the RP | |
| 15:00:44 | dansmith | we should never kill nova-compute after it's hit steady-state | |