| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-20 | |||
| 20:42:38 | dansmith | edleafe: okay | |
| 20:42:46 | mriedem | sean-k-mooney: not really | |
| 20:42:46 | efried | cfriesen: s/either compute node/either NUMA node/ ? | |
| 20:43:00 | cfriesen | efried: whoops,yes | |
| 20:43:55 | cfriesen | efried: we try to account for memory fairly tightly on our compute nodes, and had to make 4KB pages use "preferred" rather than "strict" numa mempolicy because we were hitting the oom killer | |
| 20:44:11 | efried | cfriesen: You could have your memory expressed in terms of resource class MEMORY_4K_PAGE. | |
| 20:44:20 | efried | cfriesen: But I guess that's not the issue you're stuck on. | |
| 20:44:51 | efried | cfriesen: You're concerned about being allowed to get those pages from separate NUMA nodes in cases where a) NUMA affinity is not strictly required; and b) you can't get 'em all from one. | |
| 20:45:39 | efried | cfriesen: So what you can actually do here is express your resource requests as separate numbered groups. | |
| 20:45:57 | cfriesen | efried: sort of. currently if you have an instance numa_topology nova will strictly constrain your instance to a single host numa node. | |
| 20:46:10 | efried | cfriesen: It *sort of* requires knowing how many NUMA nodes are possible on a host. What's the max for that, anyway? 2? 4? | |
| 20:46:12 | cfriesen | efried: and if you don't have an instance numa_topology then nova will let you float over the whole compute node | |
| 20:46:35 | efried | cfriesen: I'm talking about in the idyllic future where we're doing all this with NRP and granular resource requests. | |
| 20:46:42 | cfriesen | efried: common hardware is typically 2, some is 4. exotic hardware has many more. | |
| 20:47:29 | efried | Okay, so if you care about strict affinity, you always ask for all your memory in one numbered request group. If you don't - if you want to allow spread - you ask for blocks in separate numbered request groups. | |
| 20:47:33 | cfriesen | efried: first we need to figure out how we want to handle host NUMA affinity for instances with no numa_topology. | |
| 20:48:32 | cfriesen | efried: because if we continue to allow it to consume VCPUs and 4KB pages from the whole compute node, then we can't track those per numa node | |
| 20:48:48 | efried | Bear with me, then | |
| 20:49:54 | efried | cfriesen: So let's say you want 4096 4K pages and 4 VCPUs. | |
| 20:50:18 | efried | cfriesen: If you want strict affinity, you say resources1=VCPU:4,MEMORY_4K_PAGE:4096 | |
| 20:50:33 | efried | cfriesen: But if you don't care about strict affinity... | |
| 20:50:49 | efried | let's say you know your cloud doesn't have any hosts that support more than 4 NUMA nodes. | |
| 20:52:03 | efried | Your flavor that doesn't need strict affinity could say resources1=MEMORY_4K_PAGE:1024&resources2=MEMORY_4K_PAGE:1024&resources3=MEMORY_4K_PAGE:1024&resources4=MEMORY_4K_PAGE:1024&resources5=VCPU:1&resources6=VCPU:1&resources7=VCPU:1&resources8=VCPU:1 | |
| 20:52:25 | efried | Now, the results you get back can still include permutations where all of those resources come from the same RP. | |
| 20:52:35 | efried | And if you're using a weigher, you can choose those first. | |
| 20:52:49 | efried | But it'll also allow for permutations where the resources come from different RPs. | |
| 20:52:53 | sean-k-mooney | efried: there is nothing in the api preventing you form creating a vm with 4 virtual numa nodes on a host with on 2 phyical numanodes | |
| 20:52:53 | cfriesen | efried: the problem arises when we actually start up the qemu process. The memory is actually consumed on whatever numa node happens to request it...and now nova/placement no longer knows how much memory is available on each host numa node. | |
| 20:53:25 | efried | cfriesen: That would be a libvirt fix. | |
| 20:53:29 | efried | sean-k-mooney: Cool, that's useful. | |
| 20:53:53 | sean-k-mooney | efried: the libvirt dirver does not allow that but the only guarentee that the api provides in this case is that if i request a vm with 1 numa node it will not span 2+ host numa nodes | |
| 20:54:00 | cfriesen | efried: it's not a libvirt problem, it's how linux works. If you haven't overridden the numa affinity, then by default memory allocations occur on the numa node that you're on. | |
| 20:54:28 | sean-k-mooney | cfriesen: not quite | |
| 20:54:30 | efried | cfriesen: Then I don't understand how we ever wind up with memory coming from disparate NUMA nodes. | |
| 20:54:55 | sean-k-mooney | the linux kernel prefers to but numactl makes the desision and it may allocated form a remote numa node in some cases | |
| 20:55:30 | sean-k-mooney | efried: a vm without hugepages can have its memory provided by any numa node | |
| 20:55:47 | cfriesen | efried: if there is no instance numa topology, then nova doesn't specify any affinity. this means the qemu threads are free to float across the whole compute node. when they do a memory allocation they will by default be allocated memory from the numa node they're currently running on. | |
| 20:55:47 | efried | But it's all provided by the same NUMA node? | |
| 20:55:52 | cfriesen | efried: no | |
| 20:55:56 | sean-k-mooney | efried: infact today we do not have a facility to enforce that it comes form the same numa node in nova | |
| 20:56:25 | cfriesen | we explicitly let the host decide....we did for a while restrict it to a single numa node, then removed that for increased density | |
| 20:56:32 | cfriesen | (we being nova) | |
| 20:56:44 | efried | And just so I'm clear, you want that kind of VM to be able to run on the same node as the strictly-affinitized one? | |
| 20:56:52 | efried | s/node/host/ | |
| 20:57:00 | sean-k-mooney | cfriesen: did that ever land upstream. i dont think we ever did a release with that behavior for 4k pages | |
| 20:57:28 | sean-k-mooney | efried: yes i dont see why not | |
| 20:57:42 | efried | Well, y'all seem to be splainin why not. | |
| 20:57:46 | sean-k-mooney | efried: the only way to request numa affined memory in openstack today is via hugepages | |
| 20:57:51 | cfriesen | efried: same compute node, yes. there's a spec under review right now to support shared and dedicated vcpus on the same compute node, and we already support 4KB and 2MB page backing on the same compute node | |
| 20:58:25 | cfriesen | it's just that right now the resource tracking is kind of messed up for 4KB pages | |
| 20:59:01 | sean-k-mooney | cfriesen: yes 4k pages are not tracked in the numa topology blob so we cant numa afine them | |
| 20:59:38 | sean-k-mooney | cfriesen: you can request 4k pages specifically but the is special case code the skips the numa suff for them | |
| 20:59:58 | cfriesen | sean-k-mooney: aren't they part of mempages? | |
| 21:00:09 | openstackgerrit | Merged openstack/nova master: Always pass 'NUMACell.siblings' to _pack_instance_onto_cores' https://review.openstack.org/537364 | |
| 21:00:28 | sean-k-mooney | mempages? | |
| 21:00:38 | cfriesen | NUMACell.mempages | |
| 21:01:04 | sean-k-mooney | you can set hw:mem_page_size=4k but i dont think they are stored in NUMACell.mempages | |
| 21:01:13 | sean-k-mooney | i guess i can check the db one seck | |
| 21:04:22 | sean-k-mooney | that is what the numatopology blob looks like http://paste.openstack.org/show/706682/ | |
| 21:05:05 | sean-k-mooney | cfriesen: so yes "nova_object.data": {"used": 0, "total": 2043576, "reserved": 0, "size_kb": 4} they are there | |
| 21:06:22 | sean-k-mooney | cfriesen: they are actully tracked per numa node but we cant tie teh per numa node values back to the host memory_mb value simply today | |
| 21:06:55 | cfriesen | I think we can, for ones that are strictly pinned. you just subtract the same value from both | |
| 21:07:13 | cfriesen | but for floating ones we don't know the actual per-host-numa-node consumption | |
| 21:07:19 | sean-k-mooney | i mean technically host memory_mb is the sum of all the cells 4k pages but im not sure if they will always agreee | |
| 21:08:49 | sean-k-mooney | cfriesen: the reall issue is the host reserved memory option. that is host wide and we jsut subtract it form the memory_mb value in the code but not sure how to translate that to per numa reserved values | |
| 21:09:10 | mriedem | efried: melwitt: i reckon an upgrade release note will be in order for this endpoint not found thing with the legacy notification payload image_ref_url being an image id rather than a url | |
| 21:09:19 | sean-k-mooney | cfriesen: that said we really need to deprecate it and replace it with a per numa version at some point | |
| 21:09:24 | cfriesen | sean-k-mooney: agreed | |
| 21:09:53 | efried | mriedem: Yeah, sounds like a plan. Though I expect it to be a while before folks quit using api_servers IRL. | |
| 21:10:03 | sean-k-mooney | anyway its 9 so im going to go home and have dinner o/ | |
| 21:10:08 | cfriesen | later | |
| 21:10:11 | mriedem | efried: same | |
| 21:10:11 | efried | ...which (I think) makes the issue moot. | |
| 21:10:26 | efried | This bug came out of a devstack? | |
| 21:10:38 | efried | Cause that's the one place I know for sure we got rid of api_servers. | |
| 21:10:44 | mriedem | yes it did | |
| 21:10:55 | mriedem | "This on devstack with commit id: 5d2add74534719c5670b29152964a60e8f23b42b" | |
| 21:11:22 | melwitt | yeah, release note is always helpful. I do wonder if the field is nullable and if sending nothing would be better than breaking the contract, but meh, not sure | |
| 21:11:26 | efried | But this is good - flushing out these bugs/corner cases in devstack before they hit the proverbial fan in production. | |
| 21:11:59 | mriedem | melwitt: nullable is also breaking the contract a bit | |
| 21:12:13 | mriedem | fwiw, this is already dumb if you're using bfv, | |
| 21:12:21 | mriedem | because instance.image_ref is '' for bfv | |
| 21:12:32 | mriedem | so we're sending an image ref url with no image id in it | |
| 21:12:42 | melwitt | heh, okay | |
| 21:12:46 | cfriesen | sean-k-mooney: prior to 1231c469d (circa 2014) we did pin instances without a numa_topology to a single host numa node | |
| 21:13:32 | melwitt | efried: yeah, we need to check when this broke so we know how far back to backport. and also if it's in ocata then people are probably hitting this in production | |
| 21:14:21 | efried | melwitt: https://review.openstack.org/#/c/490057/ | |
| 21:14:23 | melwitt | it's maybe not "as noticeable" since it's likely only this sync power states periodic that results in notifications maybe | |
| 21:14:54 | melwitt | efried: okay, cool. | |
| 21:15:21 | melwitt | so only need to backport to queens | |
| 21:15:27 | efried | melwitt: And here's the change that removed api_servers from devstack: https://review.openstack.org/#/c/490031/ | |
| 21:16:12 | efried | melwitt: Well, I'll be a little surprised if that's the only place we're using a tokenless auth context to look up a glance endpoint. | |
| 21:16:20 | efried | given the scope of 490057 | |
| 21:19:43 | melwitt | efried: yeah ... probably. but since it's so recent, I think you're right this is being caught a lot sooner than most people upgrade to queens. I was worried it was going to go back to ocata or something crazy. I know you linked the patch earlier but I didn't notice it was pretty recent | |
| 21:22:14 | melwitt | mriedem: what I was thinking with the nullable thing, would be in the worst case if someone had automation parsing image_ref_url and we make it not a url, but empty string isn't one either. not sure if they're equally bad. just thinking out loud | |
| 21:22:40 | mriedem | if someone blindly takes the image_ref_url and makes a GET curl request with it, it's going to blow up for all volume-backed instances | |
| 21:23:12 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Handle EndpointNotFound when building image_ref_url in notifications https://review.openstack.org/554703 | |
| 21:23:14 | mriedem | i think in the long ago, you could also create servers by passing an image URL to the imageRef parameter | |
| 21:23:15 | melwitt | yeah. I was thinking before that as far as the format of it. probably overthinking it | |
| 21:23:16 | mriedem | and nova would parse it | |