Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-03-20
20:16:34 dansmith jaypipes: oh you said you were getting to work, so I figured you were around
20:16:54 dansmith mah bad, tomorrow is fine
20:17:57 melwitt getting to work cleaning dog poodoo
20:18:07 mriedem edmondsw: someone on your dev or qa team could actually test it first
20:18:11 mriedem to flush out any obvious problems
20:18:21 edmondsw mriedem oh definitely
20:18:29 edmondsw was just talking from a code perspective
20:18:58 mriedem sean-k-mooney: good question; i didn't trace the requests in the tempest job to see if the port was actually detached before it was deleted, or what the order was there,
20:19:15 mriedem sean-k-mooney: should be pretty easy to find out - create a server from a pre-existing port and try to delete the port
20:19:24 mriedem i don't have a devstack handy
20:19:32 mriedem i know you can't delete an attached volume
20:19:48 sean-k-mooney mriedem: ill give it a try i think i have an env running
20:23:26 openstackgerrit Eric Berglund proposed openstack/nova master: DNM: EXPERIMENTAL: Set proc_units_factor to 0.1 in VMBuilder init https://review.openstack.org/554688
20:23:49 sean-k-mooney mriedem: so ya i can create the port, boot with it then delete it and the interface gets detatched from the running vm
20:23:59 sean-k-mooney mriedem: im just going to check the xml to confirm
20:24:35 sean-k-mooney mriedem: yep the vm no has no nics
20:25:08 sean-k-mooney i wonder if the revers works. can i create a neutron port and then set its owner to a vm and trigger an attache
20:27:04 cfriesen So in the discussion around https://review.openstack.org/#/c/552924/ something interesting has come up...currently vcpus and 4KB pages can be consumed on a compute-node bases (where it floats across the whole compute node) or on a numa-node basis (where it's constrained to a single numa node) depending on whether the instance has a numa topology or not. (https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.
20:27:06 cfriesen py#L4435) This makes it hard to reason about resource tracking.
20:30:54 mriedem sean-k-mooney: no the reverse doesn't work,
20:31:05 mriedem not might get a network-vif-plugged event but it won't know what to do with it and just log a warning
20:31:10 mriedem *nova
20:31:33 sean-k-mooney mriedem: ya just tried it.
20:31:40 mriedem the fact you can delete a port while it's attached is pretty scary
20:31:43 sean-k-mooney i create another port and did openstack port set --device 55a08a3b-fd27-495f-85da-39eedd5fde9c --device-owner "compute:nova" --host "ubuntu" myport
20:32:01 sean-k-mooney it ended up binding the port on the host correctly
20:32:18 sean-k-mooney im going to do a hard reboot but i think it will then update the xml correctly
20:32:31 mriedem in the compute api we check instance states when detaching a port,
20:32:34 edleafe dansmith: sorry, was away. Yes, the idea for member_of was "member of at least one of these". The typical use case would be only 1 agg.
20:32:39 mriedem but the networking api wouldn't have that
20:32:47 mriedem so you could detach ports by deleting them while an instance is migrating
20:33:14 mriedem which seems, not good
20:34:02 cfriesen does it mess up our accounting?
20:34:59 cfriesen why is "while it's migrating" different than "while it's running"?
20:35:34 sean-k-mooney mriedem: strange the hard reboot did not cause the interface to show up either. i must be missing something that the attach/detach whould have done
20:36:24 dansmith edleafe: yeah, so that's kinda what I was thinking at first, but then I tried to apply it to two things at once, so:
20:36:42 dansmith edleafe: imagine I have a tenant-restricted aggregate and the user boots with a requested AZ
20:37:08 dansmith edleafe: if I just concatenate the tenant and az list, then I won't restrict them properly
20:37:18 efried cfriesen: Are you suggesting that somewhere in the stack there will be code responsible for translating the allocation_request for [PCPU:2 from NUMA_RP_1] to [PCPU:2 from NUMA_RP_1 + VCPU:2 from compute node RP] ??
20:37:37 mriedem cfriesen: it was an example of a bad time to detach a port
20:37:49 cfriesen efried: I'm suggesting that our logic right now is not rigorous. :)
20:37:53 dansmith edleafe: I can restrict the tenant aggs to only the one that matches the az, but they might not be the same size (and thus are different)
20:38:00 sean-k-mooney right we could have already caluated the new xml and miss the port is being removed
20:38:02 mriedem we do refresh the nw info cache when we get network-vif-deleted events
20:38:18 dansmith edleafe: I think placement _could_ know whether one is a subset of the other, but I can't really
20:38:43 efried dansmith, edleafe: Do we need member_of=all:... ?
20:38:46 edleafe dansmith: yeah, that really isn't how placement aggs should work
20:38:49 dansmith edleafe: so, just trying to figure out what to do there.. maybe we can have a hangout with jaypipes (et al) tomorrow after I have stewed on it a bit and make sure we're happy with what we've got
20:39:05 cfriesen efried: but other than that, I think we had been talking about tracking PCPU entirely separately from VCPU (where PCPU is the number of "dedicated" pcpus we have, and VCPU is the number of VCPUs we can support)
20:39:06 efried I guess if pressed, I would have said we were implementing in: for parity with GET /rps, but wouldn't actually be using it.
20:39:06 edleafe dansmith: we could add an 'and' case by leaving off the 'in:' operator
20:39:07 dansmith efried: well, I thought of that, but I'm not sure that's right either
20:39:16 cfriesen efried: PCPUs are always consumed on a per-numa-node basis
20:39:38 cfriesen efried: VCPUs can be consumed either per-numa-node or per-compute node, which is a problem for resource tracking.
20:40:26 efried cfriesen: Just so. Which is why I was of the opinion that we should only be presenting that inventory from within the NUMA node RP.
20:40:31 dansmith efried: if you have two groups of computes and both are legit for a tenant, then you don't want all
20:40:58 efried dansmith: Yeah, I wasn't really up on the use cases for GET /a_c?member_of
20:41:00 dansmith efried: almost want member_of=one:$agg1,$agg2&member_of=$agg3
20:41:02 cfriesen efried: but that doesn't line up with the current logic, where you can consume a VCPU (and 4KB pages) from the entire compute node
20:41:06 sean-k-mooney mriedem: fyi http://paste.openstack.org/show/706642/ any guess why the "attach" by recreating the port and hard rebooting did not work? the port look identical to me bar the fact the status is down on the new port
20:41:33 efried dansmith: But I knew we would want in:[any] for parity.
20:41:47 dansmith efried: edleafe: I'll try to come up with a few different hard examples to talk about and we can decide whether we care (or care right now) about them
20:42:23 efried Yeah, sounds good. I mean, the bp is approved and completed, but heck, let's throw another one after it.
20:42:23 edleafe dansmith: ok. If it's needed, it shouldn't be too big of a change
20:42:27 cfriesen efried: and for various reasons it's really hard to track 4KB pages, since the host can consume them from either compute node at will. (Unless you run host stuff in a separate cgroup or container to provide hard limits.)
20:42:38 dansmith edleafe: okay
20:42:46 mriedem sean-k-mooney: not really
20:42:46 efried cfriesen: s/either compute node/either NUMA node/ ?
20:43:00 cfriesen efried: whoops,yes
20:43:55 cfriesen efried: we try to account for memory fairly tightly on our compute nodes, and had to make 4KB pages use "preferred" rather than "strict" numa mempolicy because we were hitting the oom killer
20:44:11 efried cfriesen: You could have your memory expressed in terms of resource class MEMORY_4K_PAGE.
20:44:20 efried cfriesen: But I guess that's not the issue you're stuck on.
20:44:51 efried cfriesen: You're concerned about being allowed to get those pages from separate NUMA nodes in cases where a) NUMA affinity is not strictly required; and b) you can't get 'em all from one.
20:45:39 efried cfriesen: So what you can actually do here is express your resource requests as separate numbered groups.
20:45:57 cfriesen efried: sort of. currently if you have an instance numa_topology nova will strictly constrain your instance to a single host numa node.
20:46:10 efried cfriesen: It *sort of* requires knowing how many NUMA nodes are possible on a host. What's the max for that, anyway? 2? 4?
20:46:12 cfriesen efried: and if you don't have an instance numa_topology then nova will let you float over the whole compute node
20:46:35 efried cfriesen: I'm talking about in the idyllic future where we're doing all this with NRP and granular resource requests.
20:46:42 cfriesen efried: common hardware is typically 2, some is 4. exotic hardware has many more.
20:47:29 efried Okay, so if you care about strict affinity, you always ask for all your memory in one numbered request group. If you don't - if you want to allow spread - you ask for blocks in separate numbered request groups.
20:47:33 cfriesen efried: first we need to figure out how we want to handle host NUMA affinity for instances with no numa_topology.
20:48:32 cfriesen efried: because if we continue to allow it to consume VCPUs and 4KB pages from the whole compute node, then we can't track those per numa node
20:48:48 efried Bear with me, then
20:49:54 efried cfriesen: So let's say you want 4096 4K pages and 4 VCPUs.
20:50:18 efried cfriesen: If you want strict affinity, you say resources1=VCPU:4,MEMORY_4K_PAGE:4096
20:50:33 efried cfriesen: But if you don't care about strict affinity...
20:50:49 efried let's say you know your cloud doesn't have any hosts that support more than 4 NUMA nodes.
20:52:03 efried Your flavor that doesn't need strict affinity could say resources1=MEMORY_4K_PAGE:1024&resources2=MEMORY_4K_PAGE:1024&resources3=MEMORY_4K_PAGE:1024&resources4=MEMORY_4K_PAGE:1024&resources5=VCPU:1&resources6=VCPU:1&resources7=VCPU:1&resources8=VCPU:1
20:52:25 efried Now, the results you get back can still include permutations where all of those resources come from the same RP.
20:52:35 efried And if you're using a weigher, you can choose those first.
20:52:49 efried But it'll also allow for permutations where the resources come from different RPs.
20:52:53 sean-k-mooney efried: there is nothing in the api preventing you form creating a vm with 4 virtual numa nodes on a host with on 2 phyical numanodes
20:52:53 cfriesen efried: the problem arises when we actually start up the qemu process. The memory is actually consumed on whatever numa node happens to request it...and now nova/placement no longer knows how much memory is available on each host numa node.
20:53:25 efried cfriesen: That would be a libvirt fix.
20:53:29 efried sean-k-mooney: Cool, that's useful.
20:53:53 sean-k-mooney efried: the libvirt dirver does not allow that but the only guarentee that the api provides in this case is that if i request a vm with 1 numa node it will not span 2+ host numa nodes
20:54:00 cfriesen efried: it's not a libvirt problem, it's how linux works. If you haven't overridden the numa affinity, then by default memory allocations occur on the numa node that you're on.
20:54:28 sean-k-mooney cfriesen: not quite
20:54:30 efried cfriesen: Then I don't understand how we ever wind up with memory coming from disparate NUMA nodes.
20:54:55 sean-k-mooney the linux kernel prefers to but numactl makes the desision and it may allocated form a remote numa node in some cases

Earlier   Later