| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-20 | |||
| 20:45:39 | efried | cfriesen: So what you can actually do here is express your resource requests as separate numbered groups. | |
| 20:45:57 | cfriesen | efried: sort of. currently if you have an instance numa_topology nova will strictly constrain your instance to a single host numa node. | |
| 20:46:10 | efried | cfriesen: It *sort of* requires knowing how many NUMA nodes are possible on a host. What's the max for that, anyway? 2? 4? | |
| 20:46:12 | cfriesen | efried: and if you don't have an instance numa_topology then nova will let you float over the whole compute node | |
| 20:46:35 | efried | cfriesen: I'm talking about in the idyllic future where we're doing all this with NRP and granular resource requests. | |
| 20:46:42 | cfriesen | efried: common hardware is typically 2, some is 4. exotic hardware has many more. | |
| 20:47:29 | efried | Okay, so if you care about strict affinity, you always ask for all your memory in one numbered request group. If you don't - if you want to allow spread - you ask for blocks in separate numbered request groups. | |
| 20:47:33 | cfriesen | efried: first we need to figure out how we want to handle host NUMA affinity for instances with no numa_topology. | |
| 20:48:32 | cfriesen | efried: because if we continue to allow it to consume VCPUs and 4KB pages from the whole compute node, then we can't track those per numa node | |
| 20:48:48 | efried | Bear with me, then | |
| 20:49:54 | efried | cfriesen: So let's say you want 4096 4K pages and 4 VCPUs. | |
| 20:50:18 | efried | cfriesen: If you want strict affinity, you say resources1=VCPU:4,MEMORY_4K_PAGE:4096 | |
| 20:50:33 | efried | cfriesen: But if you don't care about strict affinity... | |
| 20:50:49 | efried | let's say you know your cloud doesn't have any hosts that support more than 4 NUMA nodes. | |
| 20:52:03 | efried | Your flavor that doesn't need strict affinity could say resources1=MEMORY_4K_PAGE:1024&resources2=MEMORY_4K_PAGE:1024&resources3=MEMORY_4K_PAGE:1024&resources4=MEMORY_4K_PAGE:1024&resources5=VCPU:1&resources6=VCPU:1&resources7=VCPU:1&resources8=VCPU:1 | |
| 20:52:25 | efried | Now, the results you get back can still include permutations where all of those resources come from the same RP. | |
| 20:52:35 | efried | And if you're using a weigher, you can choose those first. | |
| 20:52:49 | efried | But it'll also allow for permutations where the resources come from different RPs. | |
| 20:52:53 | cfriesen | efried: the problem arises when we actually start up the qemu process. The memory is actually consumed on whatever numa node happens to request it...and now nova/placement no longer knows how much memory is available on each host numa node. | |
| 20:52:53 | sean-k-mooney | efried: there is nothing in the api preventing you form creating a vm with 4 virtual numa nodes on a host with on 2 phyical numanodes | |
| 20:53:25 | efried | cfriesen: That would be a libvirt fix. | |
| 20:53:29 | efried | sean-k-mooney: Cool, that's useful. | |
| 20:53:53 | sean-k-mooney | efried: the libvirt dirver does not allow that but the only guarentee that the api provides in this case is that if i request a vm with 1 numa node it will not span 2+ host numa nodes | |
| 20:54:00 | cfriesen | efried: it's not a libvirt problem, it's how linux works. If you haven't overridden the numa affinity, then by default memory allocations occur on the numa node that you're on. | |
| 20:54:28 | sean-k-mooney | cfriesen: not quite | |
| 20:54:30 | efried | cfriesen: Then I don't understand how we ever wind up with memory coming from disparate NUMA nodes. | |
| 20:54:55 | sean-k-mooney | the linux kernel prefers to but numactl makes the desision and it may allocated form a remote numa node in some cases | |
| 20:55:30 | sean-k-mooney | efried: a vm without hugepages can have its memory provided by any numa node | |
| 20:55:47 | efried | But it's all provided by the same NUMA node? | |
| 20:55:47 | cfriesen | efried: if there is no instance numa topology, then nova doesn't specify any affinity. this means the qemu threads are free to float across the whole compute node. when they do a memory allocation they will by default be allocated memory from the numa node they're currently running on. | |
| 20:55:52 | cfriesen | efried: no | |
| 20:55:56 | sean-k-mooney | efried: infact today we do not have a facility to enforce that it comes form the same numa node in nova | |
| 20:56:25 | cfriesen | we explicitly let the host decide....we did for a while restrict it to a single numa node, then removed that for increased density | |
| 20:56:32 | cfriesen | (we being nova) | |
| 20:56:44 | efried | And just so I'm clear, you want that kind of VM to be able to run on the same node as the strictly-affinitized one? | |
| 20:56:52 | efried | s/node/host/ | |
| 20:57:00 | sean-k-mooney | cfriesen: did that ever land upstream. i dont think we ever did a release with that behavior for 4k pages | |
| 20:57:28 | sean-k-mooney | efried: yes i dont see why not | |
| 20:57:42 | efried | Well, y'all seem to be splainin why not. | |
| 20:57:46 | sean-k-mooney | efried: the only way to request numa affined memory in openstack today is via hugepages | |
| 20:57:51 | cfriesen | efried: same compute node, yes. there's a spec under review right now to support shared and dedicated vcpus on the same compute node, and we already support 4KB and 2MB page backing on the same compute node | |
| 20:58:25 | cfriesen | it's just that right now the resource tracking is kind of messed up for 4KB pages | |
| 20:59:01 | sean-k-mooney | cfriesen: yes 4k pages are not tracked in the numa topology blob so we cant numa afine them | |
| 20:59:38 | sean-k-mooney | cfriesen: you can request 4k pages specifically but the is special case code the skips the numa suff for them | |
| 20:59:58 | cfriesen | sean-k-mooney: aren't they part of mempages? | |
| 21:00:09 | openstackgerrit | Merged openstack/nova master: Always pass 'NUMACell.siblings' to _pack_instance_onto_cores' https://review.openstack.org/537364 | |
| 21:00:28 | sean-k-mooney | mempages? | |
| 21:00:38 | cfriesen | NUMACell.mempages | |
| 21:01:04 | sean-k-mooney | you can set hw:mem_page_size=4k but i dont think they are stored in NUMACell.mempages | |
| 21:01:13 | sean-k-mooney | i guess i can check the db one seck | |
| 21:04:22 | sean-k-mooney | that is what the numatopology blob looks like http://paste.openstack.org/show/706682/ | |
| 21:05:05 | sean-k-mooney | cfriesen: so yes "nova_object.data": {"used": 0, "total": 2043576, "reserved": 0, "size_kb": 4} they are there | |
| 21:06:22 | sean-k-mooney | cfriesen: they are actully tracked per numa node but we cant tie teh per numa node values back to the host memory_mb value simply today | |
| 21:06:55 | cfriesen | I think we can, for ones that are strictly pinned. you just subtract the same value from both | |
| 21:07:13 | cfriesen | but for floating ones we don't know the actual per-host-numa-node consumption | |
| 21:07:19 | sean-k-mooney | i mean technically host memory_mb is the sum of all the cells 4k pages but im not sure if they will always agreee | |
| 21:08:49 | sean-k-mooney | cfriesen: the reall issue is the host reserved memory option. that is host wide and we jsut subtract it form the memory_mb value in the code but not sure how to translate that to per numa reserved values | |
| 21:09:10 | mriedem | efried: melwitt: i reckon an upgrade release note will be in order for this endpoint not found thing with the legacy notification payload image_ref_url being an image id rather than a url | |
| 21:09:19 | sean-k-mooney | cfriesen: that said we really need to deprecate it and replace it with a per numa version at some point | |
| 21:09:24 | cfriesen | sean-k-mooney: agreed | |
| 21:09:53 | efried | mriedem: Yeah, sounds like a plan. Though I expect it to be a while before folks quit using api_servers IRL. | |
| 21:10:03 | sean-k-mooney | anyway its 9 so im going to go home and have dinner o/ | |
| 21:10:08 | cfriesen | later | |
| 21:10:11 | efried | ...which (I think) makes the issue moot. | |
| 21:10:11 | mriedem | efried: same | |
| 21:10:26 | efried | This bug came out of a devstack? | |
| 21:10:38 | efried | Cause that's the one place I know for sure we got rid of api_servers. | |
| 21:10:44 | mriedem | yes it did | |
| 21:10:55 | mriedem | "This on devstack with commit id: 5d2add74534719c5670b29152964a60e8f23b42b" | |
| 21:11:22 | melwitt | yeah, release note is always helpful. I do wonder if the field is nullable and if sending nothing would be better than breaking the contract, but meh, not sure | |
| 21:11:26 | efried | But this is good - flushing out these bugs/corner cases in devstack before they hit the proverbial fan in production. | |
| 21:11:59 | mriedem | melwitt: nullable is also breaking the contract a bit | |
| 21:12:13 | mriedem | fwiw, this is already dumb if you're using bfv, | |
| 21:12:21 | mriedem | because instance.image_ref is '' for bfv | |
| 21:12:32 | mriedem | so we're sending an image ref url with no image id in it | |
| 21:12:42 | melwitt | heh, okay | |
| 21:12:46 | cfriesen | sean-k-mooney: prior to 1231c469d (circa 2014) we did pin instances without a numa_topology to a single host numa node | |
| 21:13:32 | melwitt | efried: yeah, we need to check when this broke so we know how far back to backport. and also if it's in ocata then people are probably hitting this in production | |
| 21:14:21 | efried | melwitt: https://review.openstack.org/#/c/490057/ | |
| 21:14:23 | melwitt | it's maybe not "as noticeable" since it's likely only this sync power states periodic that results in notifications maybe | |
| 21:14:54 | melwitt | efried: okay, cool. | |
| 21:15:21 | melwitt | so only need to backport to queens | |
| 21:15:27 | efried | melwitt: And here's the change that removed api_servers from devstack: https://review.openstack.org/#/c/490031/ | |
| 21:16:12 | efried | melwitt: Well, I'll be a little surprised if that's the only place we're using a tokenless auth context to look up a glance endpoint. | |
| 21:16:20 | efried | given the scope of 490057 | |
| 21:19:43 | melwitt | efried: yeah ... probably. but since it's so recent, I think you're right this is being caught a lot sooner than most people upgrade to queens. I was worried it was going to go back to ocata or something crazy. I know you linked the patch earlier but I didn't notice it was pretty recent | |
| 21:22:14 | melwitt | mriedem: what I was thinking with the nullable thing, would be in the worst case if someone had automation parsing image_ref_url and we make it not a url, but empty string isn't one either. not sure if they're equally bad. just thinking out loud | |
| 21:22:40 | mriedem | if someone blindly takes the image_ref_url and makes a GET curl request with it, it's going to blow up for all volume-backed instances | |
| 21:23:12 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Handle EndpointNotFound when building image_ref_url in notifications https://review.openstack.org/554703 | |
| 21:23:14 | mriedem | i think in the long ago, you could also create servers by passing an image URL to the imageRef parameter | |
| 21:23:15 | melwitt | yeah. I was thinking before that as far as the format of it. probably overthinking it | |
| 21:23:16 | mriedem | and nova would parse it | |
| 21:29:15 | mriedem | multiattach is fixed with the queens UCA | |
| 21:29:19 | mriedem | https://review.openstack.org/#/c/554667/ | |
| 21:29:30 | mriedem | that's the only thing blocking us from using the queens UCA in devstack now | |
| 21:30:08 | imacdonn | ooh | |
| 21:31:01 | imacdonn | is there any actually need to use the UCA? ZFSSA CI seems to be doing OK with out .. but that's cinder | |
| 21:31:32 | mriedem | well, a few things, | |
| 21:31:37 | mriedem | we get newer libvirt and qemu, | |
| 21:31:51 | mriedem | testing rocky against pike UCA seems weird, we should at least use queens if it's there | |