Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-03-27
20:44:09 mriedem good for them
20:44:35 cfriesen efried: if we don't restrict it, we have no idea how many 4KB pages are left on each host numa node.
20:44:54 sean-k-mooney[m] cfriesen: that breaks existing behavior where it cant fit in one numa node
20:45:03 cfriesen sean-k-mooney[m]: yes. and I think we have no option.
20:45:25 efried cfriesen: except splitting into individual pages, one per request group.
20:45:34 efried or... inventing something new.
20:46:15 cfriesen efried: I'm not sure we can do that with qemu.
20:46:56 efried cfriesen: I'm not talking about qemu splitting. I'm talking about the request being split. Then placement will give you back summed-up allocations per RP.
20:47:31 efried NUMA_0: PAGES=64, NUMA_1: PAGES=1024 or whatever
20:47:44 efried cfriesen: because btw, jaypipes and dansmith came down hard on the idea of placement tracking separate request groups in any way.
20:47:46 cfriesen efried: when you start up qemu and it's allowed to float across the whole host, we do not know how much it will consume from each host numa node
20:48:00 sean-k-mooney[m] jaypipes: so how would you feel about somthing like Resouce:VCPU:use_same_provider:true and then just define that all Rsource provider groups will not overlap with others
20:48:04 efried cfriesen: Oh, you have to let qemu have its head completely? Bogus.
20:48:43 cfriesen efried: at least the way we do it now, yes. for hugepages we map a file and tell it to use that, but for 4KB pages we just say "you're allowed to use up to X memory"
20:49:01 sean-k-mooney[m] efried: well no the said seperate request groups guarenteeed different RPs
20:49:03 efried cfriesen: Then wouldn't the hugepages be inventory on the compute RP?
20:49:15 sean-k-mooney[m] efried: that means the allocation candiates for those request groups will have to be reported sepreately
20:49:41 efried sean-k-mooney[m]: Which I don't agree with. Which was already discussed and decided in the original spec in Q.
20:49:41 cfriesen efried: hugepages are fine, they imply that we're limited to a numa node. instances with "shared" cpus and 4KB pages are allowed to float across the whole compute node currently
20:49:49 sean-k-mooney[m] cfriesen: for 4k we can map a file too if we want
20:50:12 cfriesen sean-k-mooney[m]: can we map multiple files for an instance with a single virtual numa node?
20:50:53 sean-k-mooney[m] cfriesen: we just dont but we can numa afinites 4k pages
20:51:26 cfriesen sean-k-mooney[m]: yes, but can we allocate 3GB of 4K pages from host numa node 0 and 1GB from host numa node 1 for an instance with a single virtual numa node?
20:52:26 sean-k-mooney[m] cfriesen: i think so. why would you want too
20:52:34 sean-k-mooney[m] to allow the memory to come form multiple host numa nodes
20:53:13 cfriesen sean-k-mooney[m]: If I have only 3GB memory free on one numa node and 1GB on the other, and I want to keep the current behaviour of letting the instance float across the whole compute node.
20:54:09 jaypipes sean-k-mooney[m]: again, I think that granular request groups should mean that the resources in each request group are provided by different resource providers.
20:54:26 sean-k-mooney[m] i would have to check. we added the abiltiy to use file desciptor memory by setting the souce elemet of this https://libvirt.org/formatdomain.html#elementsMemoryBacking
20:54:29 jaypipes sean-k-mooney[m]: I do not care about the use case of "general spreadability".
20:54:32 efried jaypipes: but only if they're the same resource class
20:54:38 jaypipes efried: yes.
20:54:54 sean-k-mooney[m] i know that backing file can be affinites but i dont know if we can create two backing files attach to the same guest numa node
20:55:14 efried jaypipes: That's gonna be tough to implement, just for starters.
20:55:17 jaypipes efried: though I don't see a reason why you would separate request groups where one request group does *not* contain a resource class...
20:55:33 efried jaypipes: So that they aren't forced to land on the *same* RP.
20:55:34 sean-k-mooney[m] cfriesen: i think it would work at the qemu level but i need to dig deeper
20:55:50 jaypipes efried: say wha?
20:56:02 cfriesen sean-k-mooney[m]: unless we can do that, we can't let a single-numa-node guest use memory from multiple host numa nodes.
20:56:05 sean-k-mooney[m] jaypipes: ya i actully prefer that each group is a different RP. that is what i had originally assumed
20:56:32 jaypipes efried: I mean, I don't see a use case for doing, for example, this: resources1=VCPU:1,required2=HW_NIC_OFFLOAD_GENEVE
20:56:33 efried jaypipes: It wouldn't make sense for me to say DISK_GB:1024,VF:1. Maybe I misunderstood your statement.
20:56:40 jaypipes efried: that doesn't make sense to me.
20:57:42 sean-k-mooney[m] cfriesen: well thats the thing they are not really singel numa guests
20:57:45 efried jaypipes: resources1=VF:1,BW:200&required1=PHYSNET_A&resources2=VF:1,BW:300&required2=REALLY_FAST
20:58:06 sean-k-mooney[m] they are guest that did no specify a numa topology so there is no reason nova could not make them multi numa guests
20:58:12 jaypipes efried: ok, and?
20:58:29 jaypipes efried: that makes total sense to me.
20:58:29 cfriesen sean-k-mooney[m]: ooh, fun. that won't surprise anyone. :)
20:58:31 efried Contrived example. I don't care which physnet that second VF is on. I just want it to be fast. Why can't it land on the same RP as the first one?
20:58:56 cfriesen sean-k-mooney[m]: don't forget we'd have to preserve the numa topology over live migration
20:59:15 sean-k-mooney[m] cfriesen: if we made them multi numa guest when there resouces were split across host numa nodes it would fix the accounting issue and improve the performance of the guest
20:59:28 jaypipes efried: because it's just crazy to reason about for the user, frankly.
20:59:29 sean-k-mooney[m] cfriesen: your relying on an implentation deatil that i vift-driver specific
20:59:45 cfriesen sean-k-mooney[m]: it could also make the guest slower if it's OS isn't numa-aware.
21:00:10 jaypipes efried: if a user requests two groups of a resource, the user expects those groups to be separate.
21:00:25 jaypipes efried: and "separate" means "provided by different providers" in my and dansmith's book.
21:00:33 sean-k-mooney[m] cfriesen: the guest os
21:00:46 cfriesen sean-k-mooney[m]: yep
21:00:48 efried jaypipes: disagree completely. IMO it's way easier for the user to reason about the groups individually, without regard for their interrelations.
21:01:06 sean-k-mooney[m] cfriesen: how would it make it slow when before it would have been spread on the phyical host but it never would have knonw
21:01:43 jaypipes efried: I'm afraid we'll just have to agree to disagree.
21:01:48 efried jaypipes: Fact is, either way we decide this will leave a hole, cases we can't express.
21:01:55 cfriesen sean-k-mooney[m]: code running in the guest on two separate cpus that wants to share a lot of memory between the two CPUs. now you've got cross-NUMA latencies
21:01:56 jaypipes efried: true.
21:02:43 efried jaypipes: The way it's currently written is simpler, more flexible, and easier to implement; and IMO easier to reason about. But that may just be because it's what I've had in my head since early Q.
21:03:18 cfriesen sean-k-mooney[m]: you're correct that it's no worse than it is now
21:03:18 sean-k-mooney[m] cfriesen: yes and today that can happen too. the guest sees 1 numa node but on the host ist ram can be allocate form multiple because it floats
21:03:28 jaypipes My brain is, frankly, pretty fried.
21:03:32 efried I've thought through all the things you can't express with it, and it boils down to just one thing: you can't express "separate these request groups".
21:03:41 jaypipes I really need to eat and refresh my brain juices.
21:03:54 efried ...but adding that feature would be relatively straightforward with the addition of some new syntax.
21:04:16 efried Yeah, for my part I really ought to go review some more specs that aren't already approved.
21:06:19 sean-k-mooney[m] cfriesen: :) so i was suggesting was if you did not spcify a numa topology then we leave it up to the virt-driver but it then has to reflect the topolgy to the guest. it would be no wose then today but actully fix the 4k pages issue and technically i dont think we are breaking any guartees as i dont hink we define that if numa_nodes is not set its 1 numa node
21:07:41 sean-k-mooney[m] cfriesen: anyway that an issue for another day
21:11:22 openstackgerrit Hongbin Lu proposed openstack/nova-specs master: Choose default network on ambiguity https://review.openstack.org/520247
21:11:50 openstackgerrit Ed Leafe proposed openstack/nova-specs master: Add Generation to Consumers https://review.openstack.org/556971
21:12:09 edleafe cdent: efried: jaypipes: ^^ updated with your suggestions
21:14:38 efried edleafe: Check yer whitespace
21:15:46 edleafe efried: ugh, do I have to? <whine>
21:15:55 efried edleafe: But hold, there's another typo
21:16:02 efried edleafe: And another thing that needs to be said.
21:16:54 edleafe well, I wasn't gonna push a rev for whitespace until everyone had a crack at it
21:20:23 efried edleafe: Never mind that last thing. I was gonna ask if the generation in PUT /allocations/{c} was going to quit being ignored. But that's a RP generation, and not really relevant here.
21:21:04 efried edleafe: Consider me cracked.
21:22:24 sean-k-mooney[m] you know that feeling when you find the souce of a bug you hit and you wish you had not...
21:23:04 sean-k-mooney[m] here we convert form the disk bus the use asked for to a prefix https://github.com/openstack/nova/blob/ef0ce4d692d28a7f5a0079e24acdbfe7d2767e8b/nova/virt/libvirt/blockinfo.py#L124-L143
21:23:15 cdent cracked
21:23:23 sean-k-mooney[m] here we convert form the prefix to the disk bus https://github.com/openstack/nova/blob/ef0ce4d692d28a7f5a0079e24acdbfe7d2767e8b/nova/virt/libvirt/blockinfo.py#L297-L304
21:23:54 sean-k-mooney[m] we map sata scis and usb to sd then hard code sd = scsi
21:24:40 edleafe efried: I always have
21:24:40 sean-k-mooney[m] that mean on kvm you can request sata or usb disk which is why i had to go to IDE to fix my centos image
21:25:10 sean-k-mooney[m] *cant request
21:26:30 openstackgerrit Hongbin Lu proposed openstack/nova-specs master: Choose default network on ambiguity https://review.openstack.org/520247
21:27:20 sean-k-mooney[m] setting the disk_bus for kvm host has apreantely been broken for 5 years since https://github.com/openstack/nova/commit/7be531fe9462f2b07d4a1abf6687f649d1dfbb89 was merged ...
21:28:25 openstackgerrit Ed Leafe proposed openstack/nova-specs master: Add Generation to Consumers https://review.openstack.org/556971
21:29:18 openstackgerrit Merged openstack/nova-specs master: Amend the migration paging spec for uuid in server migrations response https://review.openstack.org/532904
21:31:01 edleafe cdent: efried: jaypipes: ^^ fixed those ghastly errors
21:31:18 efried edleafe: +1
21:35:29 mriedem jaypipes: i'll come back to the mirror aggregates in placement spec, but not today, burned out
21:37:54 openstackgerrit Chris Dent proposed openstack/nova master: WIP: Ensure that os-traits sync is attempted only at start of process https://review.openstack.org/553857

Earlier   Later