| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-20 | |||
| 20:24:35 | sean-k-mooney | mriedem: yep the vm no has no nics | |
| 20:25:08 | sean-k-mooney | i wonder if the revers works. can i create a neutron port and then set its owner to a vm and trigger an attache | |
| 20:27:04 | cfriesen | So in the discussion around https://review.openstack.org/#/c/552924/ something interesting has come up...currently vcpus and 4KB pages can be consumed on a compute-node bases (where it floats across the whole compute node) or on a numa-node basis (where it's constrained to a single numa node) depending on whether the instance has a numa topology or not. (https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver. | |
| 20:27:06 | cfriesen | py#L4435) This makes it hard to reason about resource tracking. | |
| 20:30:54 | mriedem | sean-k-mooney: no the reverse doesn't work, | |
| 20:31:05 | mriedem | not might get a network-vif-plugged event but it won't know what to do with it and just log a warning | |
| 20:31:10 | mriedem | *nova | |
| 20:31:33 | sean-k-mooney | mriedem: ya just tried it. | |
| 20:31:40 | mriedem | the fact you can delete a port while it's attached is pretty scary | |
| 20:31:43 | sean-k-mooney | i create another port and did openstack port set --device 55a08a3b-fd27-495f-85da-39eedd5fde9c --device-owner "compute:nova" --host "ubuntu" myport | |
| 20:32:01 | sean-k-mooney | it ended up binding the port on the host correctly | |
| 20:32:18 | sean-k-mooney | im going to do a hard reboot but i think it will then update the xml correctly | |
| 20:32:31 | mriedem | in the compute api we check instance states when detaching a port, | |
| 20:32:34 | edleafe | dansmith: sorry, was away. Yes, the idea for member_of was "member of at least one of these". The typical use case would be only 1 agg. | |
| 20:32:39 | mriedem | but the networking api wouldn't have that | |
| 20:32:47 | mriedem | so you could detach ports by deleting them while an instance is migrating | |
| 20:33:14 | mriedem | which seems, not good | |
| 20:34:02 | cfriesen | does it mess up our accounting? | |
| 20:34:59 | cfriesen | why is "while it's migrating" different than "while it's running"? | |
| 20:35:34 | sean-k-mooney | mriedem: strange the hard reboot did not cause the interface to show up either. i must be missing something that the attach/detach whould have done | |
| 20:36:24 | dansmith | edleafe: yeah, so that's kinda what I was thinking at first, but then I tried to apply it to two things at once, so: | |
| 20:36:42 | dansmith | edleafe: imagine I have a tenant-restricted aggregate and the user boots with a requested AZ | |
| 20:37:08 | dansmith | edleafe: if I just concatenate the tenant and az list, then I won't restrict them properly | |
| 20:37:18 | efried | cfriesen: Are you suggesting that somewhere in the stack there will be code responsible for translating the allocation_request for [PCPU:2 from NUMA_RP_1] to [PCPU:2 from NUMA_RP_1 + VCPU:2 from compute node RP] ?? | |
| 20:37:37 | mriedem | cfriesen: it was an example of a bad time to detach a port | |
| 20:37:49 | cfriesen | efried: I'm suggesting that our logic right now is not rigorous. :) | |
| 20:37:53 | dansmith | edleafe: I can restrict the tenant aggs to only the one that matches the az, but they might not be the same size (and thus are different) | |
| 20:38:00 | sean-k-mooney | right we could have already caluated the new xml and miss the port is being removed | |
| 20:38:02 | mriedem | we do refresh the nw info cache when we get network-vif-deleted events | |
| 20:38:18 | dansmith | edleafe: I think placement _could_ know whether one is a subset of the other, but I can't really | |
| 20:38:43 | efried | dansmith, edleafe: Do we need member_of=all:... ? | |
| 20:38:46 | edleafe | dansmith: yeah, that really isn't how placement aggs should work | |
| 20:38:49 | dansmith | edleafe: so, just trying to figure out what to do there.. maybe we can have a hangout with jaypipes (et al) tomorrow after I have stewed on it a bit and make sure we're happy with what we've got | |
| 20:39:05 | cfriesen | efried: but other than that, I think we had been talking about tracking PCPU entirely separately from VCPU (where PCPU is the number of "dedicated" pcpus we have, and VCPU is the number of VCPUs we can support) | |
| 20:39:06 | edleafe | dansmith: we could add an 'and' case by leaving off the 'in:' operator | |
| 20:39:06 | efried | I guess if pressed, I would have said we were implementing in: for parity with GET /rps, but wouldn't actually be using it. | |
| 20:39:07 | dansmith | efried: well, I thought of that, but I'm not sure that's right either | |
| 20:39:16 | cfriesen | efried: PCPUs are always consumed on a per-numa-node basis | |
| 20:39:38 | cfriesen | efried: VCPUs can be consumed either per-numa-node or per-compute node, which is a problem for resource tracking. | |
| 20:40:26 | efried | cfriesen: Just so. Which is why I was of the opinion that we should only be presenting that inventory from within the NUMA node RP. | |
| 20:40:31 | dansmith | efried: if you have two groups of computes and both are legit for a tenant, then you don't want all | |
| 20:40:58 | efried | dansmith: Yeah, I wasn't really up on the use cases for GET /a_c?member_of | |
| 20:41:00 | dansmith | efried: almost want member_of=one:$agg1,$agg2&member_of=$agg3 | |
| 20:41:02 | cfriesen | efried: but that doesn't line up with the current logic, where you can consume a VCPU (and 4KB pages) from the entire compute node | |
| 20:41:06 | sean-k-mooney | mriedem: fyi http://paste.openstack.org/show/706642/ any guess why the "attach" by recreating the port and hard rebooting did not work? the port look identical to me bar the fact the status is down on the new port | |
| 20:41:33 | efried | dansmith: But I knew we would want in:[any] for parity. | |
| 20:41:47 | dansmith | efried: edleafe: I'll try to come up with a few different hard examples to talk about and we can decide whether we care (or care right now) about them | |
| 20:42:23 | edleafe | dansmith: ok. If it's needed, it shouldn't be too big of a change | |
| 20:42:23 | efried | Yeah, sounds good. I mean, the bp is approved and completed, but heck, let's throw another one after it. | |
| 20:42:27 | cfriesen | efried: and for various reasons it's really hard to track 4KB pages, since the host can consume them from either compute node at will. (Unless you run host stuff in a separate cgroup or container to provide hard limits.) | |
| 20:42:38 | dansmith | edleafe: okay | |
| 20:42:46 | efried | cfriesen: s/either compute node/either NUMA node/ ? | |
| 20:42:46 | mriedem | sean-k-mooney: not really | |
| 20:43:00 | cfriesen | efried: whoops,yes | |
| 20:43:55 | cfriesen | efried: we try to account for memory fairly tightly on our compute nodes, and had to make 4KB pages use "preferred" rather than "strict" numa mempolicy because we were hitting the oom killer | |
| 20:44:11 | efried | cfriesen: You could have your memory expressed in terms of resource class MEMORY_4K_PAGE. | |
| 20:44:20 | efried | cfriesen: But I guess that's not the issue you're stuck on. | |
| 20:44:51 | efried | cfriesen: You're concerned about being allowed to get those pages from separate NUMA nodes in cases where a) NUMA affinity is not strictly required; and b) you can't get 'em all from one. | |
| 20:45:39 | efried | cfriesen: So what you can actually do here is express your resource requests as separate numbered groups. | |
| 20:45:57 | cfriesen | efried: sort of. currently if you have an instance numa_topology nova will strictly constrain your instance to a single host numa node. | |
| 20:46:10 | efried | cfriesen: It *sort of* requires knowing how many NUMA nodes are possible on a host. What's the max for that, anyway? 2? 4? | |
| 20:46:12 | cfriesen | efried: and if you don't have an instance numa_topology then nova will let you float over the whole compute node | |
| 20:46:35 | efried | cfriesen: I'm talking about in the idyllic future where we're doing all this with NRP and granular resource requests. | |
| 20:46:42 | cfriesen | efried: common hardware is typically 2, some is 4. exotic hardware has many more. | |
| 20:47:29 | efried | Okay, so if you care about strict affinity, you always ask for all your memory in one numbered request group. If you don't - if you want to allow spread - you ask for blocks in separate numbered request groups. | |
| 20:47:33 | cfriesen | efried: first we need to figure out how we want to handle host NUMA affinity for instances with no numa_topology. | |
| 20:48:32 | cfriesen | efried: because if we continue to allow it to consume VCPUs and 4KB pages from the whole compute node, then we can't track those per numa node | |
| 20:48:48 | efried | Bear with me, then | |
| 20:49:54 | efried | cfriesen: So let's say you want 4096 4K pages and 4 VCPUs. | |
| 20:50:18 | efried | cfriesen: If you want strict affinity, you say resources1=VCPU:4,MEMORY_4K_PAGE:4096 | |
| 20:50:33 | efried | cfriesen: But if you don't care about strict affinity... | |
| 20:50:49 | efried | let's say you know your cloud doesn't have any hosts that support more than 4 NUMA nodes. | |
| 20:52:03 | efried | Your flavor that doesn't need strict affinity could say resources1=MEMORY_4K_PAGE:1024&resources2=MEMORY_4K_PAGE:1024&resources3=MEMORY_4K_PAGE:1024&resources4=MEMORY_4K_PAGE:1024&resources5=VCPU:1&resources6=VCPU:1&resources7=VCPU:1&resources8=VCPU:1 | |
| 20:52:25 | efried | Now, the results you get back can still include permutations where all of those resources come from the same RP. | |
| 20:52:35 | efried | And if you're using a weigher, you can choose those first. | |
| 20:52:49 | efried | But it'll also allow for permutations where the resources come from different RPs. | |
| 20:52:53 | cfriesen | efried: the problem arises when we actually start up the qemu process. The memory is actually consumed on whatever numa node happens to request it...and now nova/placement no longer knows how much memory is available on each host numa node. | |
| 20:52:53 | sean-k-mooney | efried: there is nothing in the api preventing you form creating a vm with 4 virtual numa nodes on a host with on 2 phyical numanodes | |
| 20:53:25 | efried | cfriesen: That would be a libvirt fix. | |
| 20:53:29 | efried | sean-k-mooney: Cool, that's useful. | |
| 20:53:53 | sean-k-mooney | efried: the libvirt dirver does not allow that but the only guarentee that the api provides in this case is that if i request a vm with 1 numa node it will not span 2+ host numa nodes | |
| 20:54:00 | cfriesen | efried: it's not a libvirt problem, it's how linux works. If you haven't overridden the numa affinity, then by default memory allocations occur on the numa node that you're on. | |
| 20:54:28 | sean-k-mooney | cfriesen: not quite | |
| 20:54:30 | efried | cfriesen: Then I don't understand how we ever wind up with memory coming from disparate NUMA nodes. | |
| 20:54:55 | sean-k-mooney | the linux kernel prefers to but numactl makes the desision and it may allocated form a remote numa node in some cases | |
| 20:55:30 | sean-k-mooney | efried: a vm without hugepages can have its memory provided by any numa node | |
| 20:55:47 | efried | But it's all provided by the same NUMA node? | |
| 20:55:47 | cfriesen | efried: if there is no instance numa topology, then nova doesn't specify any affinity. this means the qemu threads are free to float across the whole compute node. when they do a memory allocation they will by default be allocated memory from the numa node they're currently running on. | |
| 20:55:52 | cfriesen | efried: no | |
| 20:55:56 | sean-k-mooney | efried: infact today we do not have a facility to enforce that it comes form the same numa node in nova | |
| 20:56:25 | cfriesen | we explicitly let the host decide....we did for a while restrict it to a single numa node, then removed that for increased density | |
| 20:56:32 | cfriesen | (we being nova) | |
| 20:56:44 | efried | And just so I'm clear, you want that kind of VM to be able to run on the same node as the strictly-affinitized one? | |
| 20:56:52 | efried | s/node/host/ | |
| 20:57:00 | sean-k-mooney | cfriesen: did that ever land upstream. i dont think we ever did a release with that behavior for 4k pages | |
| 20:57:28 | sean-k-mooney | efried: yes i dont see why not | |
| 20:57:42 | efried | Well, y'all seem to be splainin why not. | |
| 20:57:46 | sean-k-mooney | efried: the only way to request numa affined memory in openstack today is via hugepages | |
| 20:57:51 | cfriesen | efried: same compute node, yes. there's a spec under review right now to support shared and dedicated vcpus on the same compute node, and we already support 4KB and 2MB page backing on the same compute node | |
| 20:58:25 | cfriesen | it's just that right now the resource tracking is kind of messed up for 4KB pages | |