| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-20 | |||
| 20:38:49 | dansmith | edleafe: so, just trying to figure out what to do there.. maybe we can have a hangout with jaypipes (et al) tomorrow after I have stewed on it a bit and make sure we're happy with what we've got | |
| 20:39:05 | cfriesen | efried: but other than that, I think we had been talking about tracking PCPU entirely separately from VCPU (where PCPU is the number of "dedicated" pcpus we have, and VCPU is the number of VCPUs we can support) | |
| 20:39:06 | edleafe | dansmith: we could add an 'and' case by leaving off the 'in:' operator | |
| 20:39:06 | efried | I guess if pressed, I would have said we were implementing in: for parity with GET /rps, but wouldn't actually be using it. | |
| 20:39:07 | dansmith | efried: well, I thought of that, but I'm not sure that's right either | |
| 20:39:16 | cfriesen | efried: PCPUs are always consumed on a per-numa-node basis | |
| 20:39:38 | cfriesen | efried: VCPUs can be consumed either per-numa-node or per-compute node, which is a problem for resource tracking. | |
| 20:40:26 | efried | cfriesen: Just so. Which is why I was of the opinion that we should only be presenting that inventory from within the NUMA node RP. | |
| 20:40:31 | dansmith | efried: if you have two groups of computes and both are legit for a tenant, then you don't want all | |
| 20:40:58 | efried | dansmith: Yeah, I wasn't really up on the use cases for GET /a_c?member_of | |
| 20:41:00 | dansmith | efried: almost want member_of=one:$agg1,$agg2&member_of=$agg3 | |
| 20:41:02 | cfriesen | efried: but that doesn't line up with the current logic, where you can consume a VCPU (and 4KB pages) from the entire compute node | |
| 20:41:06 | sean-k-mooney | mriedem: fyi http://paste.openstack.org/show/706642/ any guess why the "attach" by recreating the port and hard rebooting did not work? the port look identical to me bar the fact the status is down on the new port | |
| 20:41:33 | efried | dansmith: But I knew we would want in:[any] for parity. | |
| 20:41:47 | dansmith | efried: edleafe: I'll try to come up with a few different hard examples to talk about and we can decide whether we care (or care right now) about them | |
| 20:42:23 | edleafe | dansmith: ok. If it's needed, it shouldn't be too big of a change | |
| 20:42:23 | efried | Yeah, sounds good. I mean, the bp is approved and completed, but heck, let's throw another one after it. | |
| 20:42:27 | cfriesen | efried: and for various reasons it's really hard to track 4KB pages, since the host can consume them from either compute node at will. (Unless you run host stuff in a separate cgroup or container to provide hard limits.) | |
| 20:42:38 | dansmith | edleafe: okay | |
| 20:42:46 | efried | cfriesen: s/either compute node/either NUMA node/ ? | |
| 20:42:46 | mriedem | sean-k-mooney: not really | |
| 20:43:00 | cfriesen | efried: whoops,yes | |
| 20:43:55 | cfriesen | efried: we try to account for memory fairly tightly on our compute nodes, and had to make 4KB pages use "preferred" rather than "strict" numa mempolicy because we were hitting the oom killer | |
| 20:44:11 | efried | cfriesen: You could have your memory expressed in terms of resource class MEMORY_4K_PAGE. | |
| 20:44:20 | efried | cfriesen: But I guess that's not the issue you're stuck on. | |
| 20:44:51 | efried | cfriesen: You're concerned about being allowed to get those pages from separate NUMA nodes in cases where a) NUMA affinity is not strictly required; and b) you can't get 'em all from one. | |
| 20:45:39 | efried | cfriesen: So what you can actually do here is express your resource requests as separate numbered groups. | |
| 20:45:57 | cfriesen | efried: sort of. currently if you have an instance numa_topology nova will strictly constrain your instance to a single host numa node. | |
| 20:46:10 | efried | cfriesen: It *sort of* requires knowing how many NUMA nodes are possible on a host. What's the max for that, anyway? 2? 4? | |
| 20:46:12 | cfriesen | efried: and if you don't have an instance numa_topology then nova will let you float over the whole compute node | |
| 20:46:35 | efried | cfriesen: I'm talking about in the idyllic future where we're doing all this with NRP and granular resource requests. | |
| 20:46:42 | cfriesen | efried: common hardware is typically 2, some is 4. exotic hardware has many more. | |
| 20:47:29 | efried | Okay, so if you care about strict affinity, you always ask for all your memory in one numbered request group. If you don't - if you want to allow spread - you ask for blocks in separate numbered request groups. | |
| 20:47:33 | cfriesen | efried: first we need to figure out how we want to handle host NUMA affinity for instances with no numa_topology. | |
| 20:48:32 | cfriesen | efried: because if we continue to allow it to consume VCPUs and 4KB pages from the whole compute node, then we can't track those per numa node | |
| 20:48:48 | efried | Bear with me, then | |
| 20:49:54 | efried | cfriesen: So let's say you want 4096 4K pages and 4 VCPUs. | |
| 20:50:18 | efried | cfriesen: If you want strict affinity, you say resources1=VCPU:4,MEMORY_4K_PAGE:4096 | |
| 20:50:33 | efried | cfriesen: But if you don't care about strict affinity... | |
| 20:50:49 | efried | let's say you know your cloud doesn't have any hosts that support more than 4 NUMA nodes. | |
| 20:52:03 | efried | Your flavor that doesn't need strict affinity could say resources1=MEMORY_4K_PAGE:1024&resources2=MEMORY_4K_PAGE:1024&resources3=MEMORY_4K_PAGE:1024&resources4=MEMORY_4K_PAGE:1024&resources5=VCPU:1&resources6=VCPU:1&resources7=VCPU:1&resources8=VCPU:1 | |
| 20:52:25 | efried | Now, the results you get back can still include permutations where all of those resources come from the same RP. | |
| 20:52:35 | efried | And if you're using a weigher, you can choose those first. | |
| 20:52:49 | efried | But it'll also allow for permutations where the resources come from different RPs. | |
| 20:52:53 | cfriesen | efried: the problem arises when we actually start up the qemu process. The memory is actually consumed on whatever numa node happens to request it...and now nova/placement no longer knows how much memory is available on each host numa node. | |
| 20:52:53 | sean-k-mooney | efried: there is nothing in the api preventing you form creating a vm with 4 virtual numa nodes on a host with on 2 phyical numanodes | |
| 20:53:25 | efried | cfriesen: That would be a libvirt fix. | |
| 20:53:29 | efried | sean-k-mooney: Cool, that's useful. | |
| 20:53:53 | sean-k-mooney | efried: the libvirt dirver does not allow that but the only guarentee that the api provides in this case is that if i request a vm with 1 numa node it will not span 2+ host numa nodes | |
| 20:54:00 | cfriesen | efried: it's not a libvirt problem, it's how linux works. If you haven't overridden the numa affinity, then by default memory allocations occur on the numa node that you're on. | |
| 20:54:28 | sean-k-mooney | cfriesen: not quite | |
| 20:54:30 | efried | cfriesen: Then I don't understand how we ever wind up with memory coming from disparate NUMA nodes. | |
| 20:54:55 | sean-k-mooney | the linux kernel prefers to but numactl makes the desision and it may allocated form a remote numa node in some cases | |
| 20:55:30 | sean-k-mooney | efried: a vm without hugepages can have its memory provided by any numa node | |
| 20:55:47 | efried | But it's all provided by the same NUMA node? | |
| 20:55:47 | cfriesen | efried: if there is no instance numa topology, then nova doesn't specify any affinity. this means the qemu threads are free to float across the whole compute node. when they do a memory allocation they will by default be allocated memory from the numa node they're currently running on. | |
| 20:55:52 | cfriesen | efried: no | |
| 20:55:56 | sean-k-mooney | efried: infact today we do not have a facility to enforce that it comes form the same numa node in nova | |
| 20:56:25 | cfriesen | we explicitly let the host decide....we did for a while restrict it to a single numa node, then removed that for increased density | |
| 20:56:32 | cfriesen | (we being nova) | |
| 20:56:44 | efried | And just so I'm clear, you want that kind of VM to be able to run on the same node as the strictly-affinitized one? | |
| 20:56:52 | efried | s/node/host/ | |
| 20:57:00 | sean-k-mooney | cfriesen: did that ever land upstream. i dont think we ever did a release with that behavior for 4k pages | |
| 20:57:28 | sean-k-mooney | efried: yes i dont see why not | |
| 20:57:42 | efried | Well, y'all seem to be splainin why not. | |
| 20:57:46 | sean-k-mooney | efried: the only way to request numa affined memory in openstack today is via hugepages | |
| 20:57:51 | cfriesen | efried: same compute node, yes. there's a spec under review right now to support shared and dedicated vcpus on the same compute node, and we already support 4KB and 2MB page backing on the same compute node | |
| 20:58:25 | cfriesen | it's just that right now the resource tracking is kind of messed up for 4KB pages | |
| 20:59:01 | sean-k-mooney | cfriesen: yes 4k pages are not tracked in the numa topology blob so we cant numa afine them | |
| 20:59:38 | sean-k-mooney | cfriesen: you can request 4k pages specifically but the is special case code the skips the numa suff for them | |
| 20:59:58 | cfriesen | sean-k-mooney: aren't they part of mempages? | |
| 21:00:09 | openstackgerrit | Merged openstack/nova master: Always pass 'NUMACell.siblings' to _pack_instance_onto_cores' https://review.openstack.org/537364 | |
| 21:00:28 | sean-k-mooney | mempages? | |
| 21:00:38 | cfriesen | NUMACell.mempages | |
| 21:01:04 | sean-k-mooney | you can set hw:mem_page_size=4k but i dont think they are stored in NUMACell.mempages | |
| 21:01:13 | sean-k-mooney | i guess i can check the db one seck | |
| 21:04:22 | sean-k-mooney | that is what the numatopology blob looks like http://paste.openstack.org/show/706682/ | |
| 21:05:05 | sean-k-mooney | cfriesen: so yes "nova_object.data": {"used": 0, "total": 2043576, "reserved": 0, "size_kb": 4} they are there | |
| 21:06:22 | sean-k-mooney | cfriesen: they are actully tracked per numa node but we cant tie teh per numa node values back to the host memory_mb value simply today | |
| 21:06:55 | cfriesen | I think we can, for ones that are strictly pinned. you just subtract the same value from both | |
| 21:07:13 | cfriesen | but for floating ones we don't know the actual per-host-numa-node consumption | |
| 21:07:19 | sean-k-mooney | i mean technically host memory_mb is the sum of all the cells 4k pages but im not sure if they will always agreee | |
| 21:08:49 | sean-k-mooney | cfriesen: the reall issue is the host reserved memory option. that is host wide and we jsut subtract it form the memory_mb value in the code but not sure how to translate that to per numa reserved values | |
| 21:09:10 | mriedem | efried: melwitt: i reckon an upgrade release note will be in order for this endpoint not found thing with the legacy notification payload image_ref_url being an image id rather than a url | |
| 21:09:19 | sean-k-mooney | cfriesen: that said we really need to deprecate it and replace it with a per numa version at some point | |
| 21:09:24 | cfriesen | sean-k-mooney: agreed | |
| 21:09:53 | efried | mriedem: Yeah, sounds like a plan. Though I expect it to be a while before folks quit using api_servers IRL. | |
| 21:10:03 | sean-k-mooney | anyway its 9 so im going to go home and have dinner o/ | |
| 21:10:08 | cfriesen | later | |
| 21:10:11 | efried | ...which (I think) makes the issue moot. | |
| 21:10:11 | mriedem | efried: same | |
| 21:10:26 | efried | This bug came out of a devstack? | |
| 21:10:38 | efried | Cause that's the one place I know for sure we got rid of api_servers. | |
| 21:10:44 | mriedem | yes it did | |
| 21:10:55 | mriedem | "This on devstack with commit id: 5d2add74534719c5670b29152964a60e8f23b42b" | |
| 21:11:22 | melwitt | yeah, release note is always helpful. I do wonder if the field is nullable and if sending nothing would be better than breaking the contract, but meh, not sure | |
| 21:11:26 | efried | But this is good - flushing out these bugs/corner cases in devstack before they hit the proverbial fan in production. | |
| 21:11:59 | mriedem | melwitt: nullable is also breaking the contract a bit | |
| 21:12:13 | mriedem | fwiw, this is already dumb if you're using bfv, | |
| 21:12:21 | mriedem | because instance.image_ref is '' for bfv | |