| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-22 | |||
| 20:21:50 | jaypipes | cfriesen: none of that will impact the *amount* of PCPU or VCPU resources that are requested. | |
| 20:22:15 | cfriesen | jaypipes: sure it does, because it affects how many of each I need to ask for from the RP that represents a single numa node | |
| 20:23:30 | sean-k-mooney | cfriesen: but jay is suggesting we dont ask placement for x pcpus in numa node 1 and y on node 2 we just ask for x+y pcpus | |
| 20:24:01 | jaypipes | sean-k-mooney: we *could* ask for 1 PCPU on one provider and 2 PCPU on another provider using granular request groups if we wanted. | |
| 20:24:15 | cfriesen | jaypipes: suppose I have vcpus 0-3 on virtual numa node 0, and vcpus 4-7 on virtual numa node 1. If I specify that I want vpus 0-3 to be shared, that's fundamentally a different request than if I say I want 0,4 to be shared | |
| 20:24:19 | openstackgerrit | Julia Kreger proposed openstack/nova master: WIP: Add microversion to ironic client wrapper call https://review.openstack.org/554762 | |
| 20:24:43 | sean-k-mooney | jaypipes: yes we could. and as a step 2 we might want to for the numa reasons but what your proposing today would not block us doing that in the future | |
| 20:24:58 | jaypipes | cfriesen: could we do granular request groups for that? i.e. resources1:VCPU=1, resources2:VCPU:1 | |
| 20:25:23 | jaypipes | cfriesen: would say "find my a provider tree that has providers that can serve 1 VCPU each | |
| 20:25:51 | cfriesen | is that how we're representing virtual numa nodes? | |
| 20:26:01 | sean-k-mooney | jaypipes: that kind of what i was thinking of whith the numa stuff on line 203 of https://etherpad.openstack.org/p/cpu-resource-accounting | |
| 20:26:10 | jaypipes | k | |
| 20:26:38 | jaypipes | sean-k-mooney: ack | |
| 20:26:41 | cfriesen | because we're going to need to be able to ask for "X PCPU, Y VCPU, Z memory pages of size A" all on one host NUMA node | |
| 20:27:12 | sean-k-mooney | cfriesen: that should be fine with granular requrest | |
| 20:27:22 | sean-k-mooney | i think | |
| 20:27:23 | jaypipes | cfriesen: we haven't considered memory pages yet, but yeah, that's what granular request groups are for. | |
| 20:27:55 | jaypipes | btw, I proposed standardizing memory pages as resource classes a year ago: https://review.openstack.org/#/c/442718/ | |
| 20:28:00 | sean-k-mooney | jaypipes: the mem_page_size can just be a trait | |
| 20:28:15 | jaypipes | please no. | |
| 20:28:18 | openstackgerrit | melanie witt proposed openstack/nova master: Add functional regression test for bug 1746509 https://review.openstack.org/555092 | |
| 20:28:19 | openstack | bug 1746509 in OpenStack Compute (nova) "TypeError: Can't upgrade a READER transaction to a WRITER mid-transaction" [Medium,In progress] https://launchpad.net/bugs/1746509 - Assigned to melanie witt (melwitt) | |
| 20:28:19 | openstackgerrit | melanie witt proposed openstack/nova master: Move _make_instance_list call outside of DB transaction context https://review.openstack.org/555093 | |
| 20:28:26 | jaypipes | sean-k-mooney: are they consumable things? | |
| 20:28:35 | sean-k-mooney | jaypipes: the mempages are | |
| 20:28:45 | jaypipes | right, so they should be resource classes. | |
| 20:28:47 | cfriesen | okay. so I think the end-user would find it convenient to specify *which* guest CPUs are shared/dedicated. If we specify only "how many" then they have to discover it at boot time and have sufficiently flexible software to handle setting up dynamic affinity based on what they discovered. | |
| 20:29:00 | jaypipes | sean-k-mooney: but, meh, another day for that discussion on memory pages. | |
| 20:29:07 | sean-k-mooney | so we have a rp with an inventory of mempages with a 2MB trait as a child of the numa node | |
| 20:29:52 | jaypipes | cfriesen: no disagreement from me, but again.... not related to placement/resource accounting. | |
| 20:30:01 | sean-k-mooney | jaypipes: oh i was agreeing the mempage should be a resouce class. i just dont know if we want mempage2MB and mempage1G | |
| 20:30:24 | jaypipes | sean-k-mooney: let's discuss the mempages stuff another day, eh? :) | |
| 20:30:34 | sean-k-mooney | jaypipes: sure | |
| 20:31:17 | sean-k-mooney | can we put this in https://etherpad.openstack.org/p/cpu-resource-accounting then copy it into the spec | |
| 20:31:17 | cfriesen | jaypipes: so if we allow the end user to specify something like "hw:shared_vcpus=0,1,4,5", it seems to me that nova should internally map that to the desired granular request rather than needing to specify it explicitly in the flavor extra spec. | |
| 20:32:20 | cfriesen | jaypipes: given this, it is implied that we want 4 VCPU resources, with the remainder being PCPU | |
| 20:33:34 | cfriesen | jaypipes: alternately, if we explicitly specify the VCPU and PCPU count, and make them discover the mapping at boot time, then we wouldn't need the "hw:shared_vcpus" extra spec | |
| 20:33:34 | jaypipes | cfriesen: lemme make sure I understand you... | |
| 20:34:58 | jaypipes | cfriesen: so you are saying that, just for VCPU and PCPU resource classes, that if nova sees the magic hw:shared_vcpus extra spec, that nova should figure out how many VCPU and how many PCPU it should ask for from placement instead of requiring the admin to put a "resources:VCPU=X" and "resources:PCPU=X" extra spec in the flavor? | |
| 20:35:33 | sean-k-mooney | cfriesen: so flavor.vcpus=8, hw:shared_vcpus=0,1,4,5 resouce[vcpu]=4 resouces[pcpu]=4 meaning all odd guest cpus are shared and all even cores(those not in that list) are dedicate cores. | |
| 20:36:14 | cfriesen | jaypipes: I think that would be the most convenient option for the end user | |
| 20:37:00 | sean-k-mooney | cfriesen: the hw:shared_vcpus=0,1,4,5 dose not change the placement request however right its just for the virt diriver/numa topology filter | |
| 20:37:19 | jaypipes | cfriesen: I prefer to have them discover the mapping at boot time and just specify VCPU and PCPU counts. | |
| 20:37:46 | cfriesen | jaypipes: that would be more generic, yes. | |
| 20:38:26 | jaypipes | cfriesen: that gives the virt driver the freedom to map the guest CPUs whatever way it needs, and once the virt driver makes that mapping decision, it could just write it to the instance metadata for the guest to read on boot. | |
| 20:38:58 | cfriesen | jaypipes: in that case I'd suggest that we make the lower-numbered vCPUs in a virtual numa node be "shared", and the higher-numbered ones "dedicated" | |
| 20:39:15 | cfriesen | but yeah, up to the virt driver | |
| 20:39:59 | rybridges | mlavalle: {"versions": [{"min_version": "1.0", "max_version": "1.4", "id": "v1.0"} | |
| 20:40:02 | jaypipes | cfriesen, sean-k-mooney: ok, I'm going to update my spec and attempt as best as possible to recap the above decisions. | |
| 20:40:10 | cfriesen | we'd have to persist the mapping somewhere to preserve it over live migration. and it'd be nice to keep it over cold migration/evacuate too | |
| 20:40:24 | jaypipes | cfriesen: instance metadata... | |
| 20:40:45 | jaypipes | cfriesen: just like how we save device metadata/tags today, right? | |
| 20:41:00 | cfriesen | should work, I think | |
| 20:42:06 | sean-k-mooney | cfriesen: you say per numa node but you realsie libvirt does not map guest cores to virtual numa nodes right | |
| 20:42:53 | sean-k-mooney | actully you kind of can. | |
| 20:42:55 | sean-k-mooney | <numa> | |
| 20:42:57 | sean-k-mooney | <cell id='0' cpus='0-3' memory='512000' unit='KiB'/> | |
| 20:43:04 | sean-k-mooney | </numa> | |
| 20:43:38 | sean-k-mooney | cpus in the cell id is the guest logical cpu | |
| 20:45:48 | sean-k-mooney | jaypipes: the gotcha with the metadata preseting the mapping is that on live migration we would want to make sure the mapping did not change as the running workload would not likely call the metadata api again | |
| 20:46:16 | cfriesen | sean-k-mooney: yeah, that's doable | |
| 20:46:41 | sean-k-mooney | that said NFV + live migration does not mix no matter how much telcos want it to | |
| 20:47:00 | cfriesen | sean-k-mooney: it just means that NUMATopologyFilter needs to check for that case | |
| 20:47:19 | jaypipes | sean-k-mooney: we'd still want to update the instance metadata to reflect the pinning on the destination host, though, so when the workload rebooted, it was able to configure itself. | |
| 20:47:26 | sean-k-mooney | cfriesen: meaning the numa toplogy filter would have to query the metatdata service | |
| 20:47:55 | cfriesen | jaypipes: no, we'd need to keep the "which vcpus are dedicated" mapping the same over the live migration | |
| 20:47:59 | sean-k-mooney | cfriesen: maybe put that in its own filter. | |
| 20:48:22 | cfriesen | jaypipes: we don't need to store the virtual-to-physical mapping in the metadata | |
| 20:49:08 | sean-k-mooney | cfriesen: ok i should have left 2 hours ago. letse leave the edgecase to live migration of snowflake vnf to another spec? | |
| 20:49:24 | cfriesen | sean-k-mooney: lol...me too. got an issue to debug | |
| 20:49:55 | sean-k-mooney | cfriesen: it has no effect on placement just the numatoplogy+everything_else_super filter | |
| 20:50:26 | sean-k-mooney | jaypipes: are you ok writing this up in the spec? | |
| 20:50:31 | cfriesen | sean-k-mooney: correct | |
| 20:52:02 | sean-k-mooney | ok im off tomorow but if you ping me ill proably check irc at some point | |
| 20:57:01 | cfriesen | if we report the "which vcpus are dedicated" mapping via metadata, what happens if the virtual routing is such that the guest has no access to the metadata server? will the config drive get suitably updated? | |
| 20:57:31 | sean-k-mooney | config drive is read only so no | |
| 20:57:58 | sean-k-mooney | you have to proxy to metadata via dhcp server if its an isolated neutron network | |
| 20:58:18 | sean-k-mooney | config dirive might get update on cold migrate or hard reboot | |
| 20:58:48 | cfriesen | sean-k-mooney: readonly is fine, the guest isn't going to be updating it | |
| 20:59:16 | cfriesen | and the mapping of which guest cpus are shared/dedicated shouldn't change dynamically once booted | |
| 20:59:29 | sean-k-mooney | yes but qemu can unplug it and update it on livemigrate either | |
| 20:59:42 | cfriesen | we need to keep it constant over live migrate | |
| 20:59:49 | cfriesen | so no problem | |
| 21:01:08 | mlavalle | rybridges: the microversion required on the Neutron side is 1.1. So it seems right. At the time of creating your subnet, do you you see any tracebacks in the Neutron server log? If so, please share it in a paste | |
| 21:01:30 | sean-k-mooney | sure. it just means more complicated code so when we calluate the pinning for the dest we have to maintian the shared/dedicated logical core mappings which is technical debt but ok | |
| 21:02:53 | cfriesen | sean-k-mooney: not really technical debt. logically the guest shouldn't need to be constantly checking which guest cpus are shared/dedicated. should only need to happen at boot time, or ideally on creation/rebuild | |
| 21:02:53 | dansmith | efried: excellent use of proper ITU phonetics | |
| 21:04:06 | sean-k-mooney | cfriesen: yes but it does mean that this is yes another thing the fit_instance_to_host function needs to enforce | |
| 21:04:38 | cfriesen | sean-k-mooney: yes, but it's required as soon as you allow mixed shared/dedicated in one instance. | |
| 21:05:40 | sean-k-mooney | cfriesen: yes which we dont allow today so once its added that fucntion need to be extended to allow passing in a set of mappings and the validate that the would still be correct for the new host. | |
| 21:08:41 | sean-k-mooney | cfriesen: again its doable but https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L4364-L4510 is not exactly the most plesant code to debug currently. anyway this time im really leaving | |
| 21:08:44 | sean-k-mooney | o/ | |
| 21:17:50 | cfriesen | jaypipes: so is your overall goal to get rid of the current hw:numa_mem.X=Y, hw:numa_cpu.X=Y, hw:mem_page_size=2048, etc. and specify it all explicitly as placement resources? | |
| 21:18:23 | cfriesen | jaypipes: because the alternative would be to keep them as-is and have nova calculate the allocation | |
| 21:18:43 | dansmith | cfriesen: fwiw, I have't been following all of this today, | |
| 21:19:00 | dansmith | but I'm highly allergic to plans for us trying to automate that fitting of things onto numa nodes | |
| 21:19:13 | dansmith | I don't like that it's manual, but it's a headache I don't really want to own | |
| 21:19:34 | dansmith | so convincing me is going to take some damn fine, simple code and a lot of tests | |
| 21:20:05 | cfriesen | dansmith: so are you suggesting we keep what we have now or require explicit resource specification in the flavor? | |
| 21:20:55 | dansmith | I would expect we'd take the thing we have now and turn it into a granular resource request based on the total, and the pieces you specified per node | |