| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-22 | |||
| 17:06:05 | jaypipes | stephenfin: CPU_SHARED is the consumable resource that emulator threads consume. | |
| 17:06:08 | cfriesen | stephenfin: fundamentally right now, emulator threads run on the same host CPUs as the guest CPUs do by default. | |
| 17:06:15 | sahid | jaypipes: yes basically i thought that was you idea since the beginning :) | |
| 17:06:39 | stephenfin | jaypipes: It could be though, if we added the 'emulator_pin_set' list to nova.conf that sahid suggests | |
| 17:06:49 | dansmith | sahid: I left you minor comments in that spec | |
| 17:07:10 | sahid | thanks dansmith i will address them | |
| 17:07:13 | cfriesen | stephenfin: I don't like the idea of having a separate dedicated pool just for emulator threads when the "shared" pool is perfectly fine | |
| 17:07:29 | jaypipes | stephenfin: you are conflating two different things. one is allocation of resources. the other is assignment of emulator threads to a particular set of host CPUs. | |
| 17:08:20 | cfriesen | I think accounting for the emulator threads as a VCPU (specifically if all the cpus in a guest are "dedicated") is a reasonable approximation | |
| 17:08:38 | cfriesen | If any of the cpus in a guest are "shared", then we can run the emulator threads with them. | |
| 17:09:00 | jaypipes | omg, chris you're killing me. | |
| 17:09:08 | stephenfin | jaypipes: Am I? We're saying for the resources for CPU_DEDICATED will come from CONF.cpu_dedicated_set. I'm saying the resources for CPU_EMULATOR_THREADS could come from CONF.cpu_emulator_thread_set | |
| 17:09:32 | sean-k-mooney | jaypipes: can i sugges we create an etherpad and start listing exampels and then clearly state what each means in terms of host,guest and placement allocatiosn | |
| 17:09:40 | cfriesen | jaypipes: ah, forget it. just account for the emulator threads as a VCPU. | |
| 17:09:41 | stephenfin | Then, as a guest, I want N CPU_EMULATOR_THREADS | |
| 17:10:08 | stephenfin | sean-k-mooney: Not a bad idea | |
| 17:10:12 | jaypipes | stephenfin: but according to cfriesen, you can't request that. | |
| 17:10:23 | jaypipes | stephenfin: you can't say "I want 3 emulator threads". | |
| 17:10:48 | bauzas | jaypipes: stephenfin: I'm done with my meeting | |
| 17:10:54 | bauzas | can we jump into a hangout ? | |
| 17:10:58 | jaypipes | bauzas: uhm,.... | |
| 17:11:00 | cfriesen | stephenfin: yeah, there's just some random number of emulator threads that are all pinned the same. | |
| 17:11:05 | jaypipes | bauzas: you've missed quite a bit. | |
| 17:11:24 | bauzas | jaypipes: yeah, that's what I just see... | |
| 17:12:09 | stephenfin | Riight. OK, forget that | |
| 17:12:10 | sean-k-mooney | maybe this would help https://libvirt.org/formatdomain.html#elementsCPUTuning | |
| 17:12:24 | bauzas | jaypipes: stephenfin: any possible tl;dr ? | |
| 17:12:24 | sean-k-mooney | and this https://libvirt.org/formatdomain.html#elementsIOThreadsAllocation | |
| 17:12:25 | jaypipes | sean-k-mooney: that's the problem... | |
| 17:12:44 | stephenfin | You can tell libvirt what CPUs the emulator threads will be placed on. Currently we say to pin only to one dedicated CPU | |
| 17:12:52 | jaypipes | sean-k-mooney: I am not interested in the pinning questions. I am only concerned with how to do resource accounting. | |
| 17:13:02 | stephenfin | But in this new world, we'd be affining to the entire pool of shared CPUs | |
| 17:13:13 | stephenfin | That first bit was what I was mixing up | |
| 17:13:20 | sean-k-mooney | jaypipes: yes i know. | |
| 17:13:31 | cfriesen | jaypipes: if every guest CPU is "shared" then the emulator threads just float across the same set of host cpus. It's when you have "dedicated" guest CPUs that the question of how to account for the emulator thread work arises. | |
| 17:13:48 | jaypipes | cfriesen: ack, understood. | |
| 17:13:49 | openstackgerrit | Nguyen Hai proposed openstack/nova-specs master: Enhance nova-specs webpage and clean up repo https://review.openstack.org/551802 | |
| 17:13:53 | sean-k-mooney | so looking at the docs you can only have 1 emulator thread. you can have multiple io thread and multiple vcpu trheads | |
| 17:14:13 | cfriesen | sean-k-mooney: don't we spawn another emulator thread when live migrating? | |
| 17:14:17 | stephenfin | bauzas: We're enraging jaypipes to the point that I fear for his pugs' lives | |
| 17:14:23 | jaypipes | cfriesen: but it's not like that is something dynamically calculated, right? I mean, a guest either has dedicated CPUs or it doesn.t' | |
| 17:14:26 | jaypipes | https://tenor.com/view/angry-panda-mascot-mad-gif-3456638 | |
| 17:14:29 | sean-k-mooney | cfriesen: i dont know but in that case to we care | |
| 17:14:38 | cfriesen | jaypipes: correct. | |
| 17:14:41 | bauzas | stephenfin: hold my beer | |
| 17:14:45 | sean-k-mooney | cfriesen: its live migrating all sla are out the window at that point | |
| 17:14:53 | stephenfin | bauzas: Also, we're trying to figure out how we map things like emulator threads policy and the ISOLATE CPU threads policy features to placement | |
| 17:15:00 | bauzas | k | |
| 17:15:11 | jaypipes | cfriesen: so it would be a request for, say resources:CPU_DEDICATED=4 (the vCPU threads) and resources:CPU_SHARED=1 (the emulator thread) | |
| 17:15:50 | openstackgerrit | Nguyen Hai proposed openstack/nova-specs master: Enhance nova-specs webpage and clean up repo https://review.openstack.org/551802 | |
| 17:15:51 | cfriesen | jaypipes: if we assume that the emulator thread is doing a non-trivial amount of work then that would be accurate. | |
| 17:15:59 | jaypipes | cfriesen: and no request for resources:VCPU, since there's no "VCPU" resources being consumed by the guest. | |
| 17:16:16 | cfriesen | jaypipes: if we assumed the emulator thread is doing a trivial amount of work then we could avoid allocating resources for the emulator thread entirely | |
| 17:16:33 | jaypipes | cfriesen: it's not about whether the emulator thread is doing a lot of work or not. it's about whether we want to *account* for it -- which obviously would be up to the operator, yes? | |
| 17:17:01 | jaypipes | cfriesen: if the operator didn't care, they wouldn't put resources:CPU_SHARED=1 in the flavor extra spec | |
| 17:17:18 | cfriesen | jaypipes: okay, I get you. | |
| 17:17:46 | cfriesen | keeping resource allocation separate from the nitty-gritty of actually affining things, I think what you say is true | |
| 17:17:56 | jaypipes | ++ | |
| 17:17:57 | stephenfin | I wonder what operators _would_ care about this? It's not exactly a thing you'd use in a bog-standard cloud | |
| 17:18:12 | sean-k-mooney | jaypipes: i dodnt think we should assume resources:CPU_SHARED=1 is the emulator thread though | |
| 17:18:23 | stephenfin | Another idea: we could add sahid's emulator_thread_pin_set option and forget the whole thing accounting thing | |
| 17:18:52 | jaypipes | sean-k-mooney: we're not assuming that. | |
| 17:18:53 | stephenfin | Same way we don't account for things that currently run on the host outside of 'vcpu_pin_set' | |
| 17:19:19 | jaypipes | sean-k-mooney: it could be anything associated with the guest that is consuming some amount of shared CPU resources. | |
| 17:19:36 | stephenfin | That's a variant of the "use the shared pool and ignore the impact idea", but this would ensure our emulator threads wouldn't impact the guests running on said shared pool | |
| 17:19:44 | sean-k-mooney | ok can i sugges we document this here https://etherpad.openstack.org/p/cpu-resource-accounting | |
| 17:19:48 | stephenfin | e.g. during a live migration | |
| 17:19:52 | sean-k-mooney | then we can pull it into the spec | |
| 17:20:02 | jaypipes | sure | |
| 17:20:12 | cfriesen | jaypipes: the thing here though is that "emulator_thread_policy=isolate" is what we currently ask for. I'm not enthusiastic about saying that they *also* need to set CPU_SHARED=1 or similar | |
| 17:20:16 | jaypipes | I need to take a quick walk to screw my head back on, though. | |
| 17:20:34 | cfriesen | I need coffee | |
| 17:20:54 | jaypipes | cfriesen: they wouldn't *need* to. if they didn't, the resource accounting would just continue to be messed up and inaccurate. | |
| 17:20:57 | bauzas | I need to catch up the convo... | |
| 17:21:08 | jaypipes | anyway, I will now take a walk. | |
| 17:21:40 | openstackgerrit | sahid proposed openstack/nova-specs master: libvirt: add support for virtio-net rx/tx queue sizes https://review.openstack.org/539605 | |
| 17:21:42 | sean-k-mooney | bauzas: the reason i suggested the etherpad is i think we all need to catch up and get it donw on "paper" | |
| 17:22:24 | bauzas | sean-k-mooney: etherpads are good, but I tend to think hangouts are way better | |
| 17:22:31 | bauzas | for reaching to a conclusion | |
| 17:22:35 | sahid | hum.. I just noticed your comments johnthetubaguy, let me update the spec | |
| 17:22:46 | stephenfin | bauzas: I think we need to get our ideas in order first though | |
| 17:22:58 | sean-k-mooney | bauzas: we can do both | |
| 17:23:00 | stephenfin | Also, it's almost home time for me and I have dinner plans this evening :) | |
| 17:23:17 | bauzas | your fault | |
| 17:23:37 | bauzas | I just spent the whole day between meetings and troubleshooting | |
| 17:23:47 | bauzas | I just feel I need to pass my nerves on something | |
| 17:24:29 | bauzas | ah, and kids-sitting, thanks to our glorifous country that doesn't like work | |
| 17:30:00 | sahid | mikal: sorry to annoy you each time I have CI problem, but our CI is really slow, no? | |
| 17:30:42 | sahid | wrong channel sorry | |
| 17:37:15 | rybridges | Hello. Got a question for you guys. I am trying to setup routed provider networks with Neutron. I created a network with a segment and a subnet associated with that segment. Then the host aggregate for the segment gets created automatically in nova when I do those things. However, when I do openstack aggregate show on the aggregate, I don't see any hosts associated with the aggregate that gets | |
| 17:37:17 | rybridges | created. Is this normal? Or a bug? Or some problem on our end? | |
| 17:37:19 | rybridges | I am running the openvswitch agent on my hypervisors. Dont have any other pre-existing aggregates | |
| 17:43:00 | sean-k-mooney | stephenfin: by the way we should keep https://etherpad.openstack.org/p/cpu-resource-accounting and turn it into docs for this stuff | |
| 17:49:24 | openstackgerrit | Merged openstack/osc-placement master: Resolve nits from I552688b9ee32b719a576a7a9ed5e4d5aa31d7b3f https://review.openstack.org/537971 | |
| 17:51:01 | melwitt | rybridges: I'm not familiar with the details of the feature but at a high level, I would think you (the admin) would have to add hosts to the host aggregates. that is, how would neutron know which hosts to put in which aggregates for you? mlavalle, can you confirm how one is expected to configure on the nova side to use routed provider networks? ^ | |
| 17:53:14 | rybridges | Thanks so much for the response melwitt. I was thinking nova might know because of the bridge_mappings in the openvswitch conf. Since all of the bridge mappings on my hypervisor point to the same physical network, and I created the neutron network with that physical network, I was thinking nova / neutron might be able to pick that up. | |
| 17:53:42 | rybridges | We can add the hosts manuall to the aggregate. I just wanted to make sure that is normal and we aren't expecting them to be added for us | |
| 17:56:19 | melwitt | yeah, understood. hopefully mlavalle can confirm when he's around. there are some docs and summit videos about the feature (https://etherpad.openstack.org/p/nova-ptg-rocky L240) but from skimming them I don't see mention of what to do on the nova side | |
| 17:56:46 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: Allow to specify granular CPU feature flags https://review.openstack.org/534384 | |
| 18:02:15 | openstackgerrit | Merged openstack/nova stable/pike: Revert "Refine waiting for vif plug events during _hard_reboot" https://review.openstack.org/553818 | |