| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-08-07 | |||
| 14:37:00 | efried | But I think we need to have a realistic fallback plan so that, if such a nirvana does not come into being, operators will still be able to ask for a GPU by product ID. | |
| 14:37:24 | sean-k-mooney | from an openstack point of view that what the standard traits in os-traits are for | |
| 14:37:59 | jaypipes | efried: well, if vendors want to enable their technology in OpenStack, they should work with us. Instead, nova needs to jump through a bunch of hacky hoops to work with vendors. ala https://review.openstack.org/#/c/579897/. which is complete bullshit, IMHO. | |
| 14:38:22 | efried | yes, I noticed you getting excited about that yesterday :) | |
| 14:39:16 | sean-k-mooney | jaypipes: ya.. well that is working around a limitation that nvidia put in there driver to prevent you using there gpus in vm unless you baught there datacenter gpus | |
| 14:39:37 | jaypipes | sean-k-mooney: I'm fully aware of that, yes. | |
| 14:40:14 | jaypipes | sean-k-mooney: please see my comment on whether we are OpenStack or WorkaroundForClosedStack | |
| 14:40:42 | jaypipes | sean-k-mooney: I think the answer has already been answered on that, actually. we have been VendorStack for about 5 years now. | |
| 14:40:45 | sean-k-mooney | oh i see it i have it open now. i agree that we should not need to do this but its a usecase they have | |
| 14:41:49 | sean-k-mooney | i think there are usecase beyond the gpu case for wanting to hide the hypervios signture | |
| 14:42:01 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Use placement 1.28 in scheduler report client https://review.openstack.org/583667 | |
| 14:42:01 | jaypipes | name one. | |
| 14:42:11 | sean-k-mooney | linux kernel dev | |
| 14:42:57 | sean-k-mooney | anyway i know that is a reach but for ovs-dpdk i dev i have used trick in the past to allow me to use openstack as a dev env | |
| 14:43:19 | sean-k-mooney | such as setting the nic model to e1000 to i could test the non viutalised nic binding workflow | |
| 14:44:40 | jaypipes | sean-k-mooney: you're performing use case gymnastics here. | |
| 14:45:37 | sean-k-mooney | yes and no. in the early days of dpdk support i had to do thing like set the cpu-mode to host-passthough becasue i did not have the cpu feature reqiure to even compile it otherwise | |
| 14:46:35 | dansmith | jaypipes: it's a common case for kernel development to use virtualization to simulate environments for that dev.. like fake numa nodes, pci devices, etc | |
| 14:46:39 | sean-k-mooney | as i said i was reaching a bit but if i want to do driver dev and i want to do that in a vm on openstack then i might need to hide the hypervisor signiture. that said i also recognise that openstack is not a virtualisation stack | |
| 14:46:49 | dansmith | jaypipes: that's not to say that I think IaaS really needs to worry about such things | |
| 14:47:05 | dansmith | jaypipes: given that developers can use bare hypervisors on their laptops to do that with much better control | |
| 14:47:47 | sean-k-mooney | dansmith: yep that said for ci of that code openstack is a much more compelling option | |
| 14:48:05 | dansmith | yup, CI is another reasonable use case imho | |
| 14:49:04 | sean-k-mooney | the alternitive without hypervior hiding is ironic but that is arguable more complex. the difference is the complexity is on the enduer not the technical debt nova has to maintain | |
| 14:49:38 | dansmith | sean-k-mooney: note that I'm defending the dev case for needing finer grained control, but *not* defending the hypervisor-hiding case | |
| 14:49:51 | dansmith | the latter is mostly total closed-source proprietary BS | |
| 14:50:20 | jaypipes | sean-k-mooney, dansmith: in that case, the use case would be to be able to *set* the vendor ID to a particular value, not hardcode it to "1234567890ab" just for the HyperV emulator just to avoid licensing issues with Nvidia: https://review.openstack.org/#/c/579897/4/nova/virt/libvirt/config.py@2241 | |
| 14:50:50 | lyarwood | s10: hey, thanks for the report, so what's the actual knock of impact of update_available_resource taking longer here? Scheduling accuracy? | |
| 14:51:26 | sean-k-mooney | jaypipes: well i had originally asked shoudl we just remove the hypervier section entirly if hiding was asked for to fully hide the fact we were in a vm but that has performance impacts | |
| 14:52:34 | sean-k-mooney | but yes making this more generic to allow setting a spefic vendor id would be just as vaild | |
| 14:53:06 | s10 | lyarwood: It looks like, when I try to live migrate instances from this host, all operations should wait for this update_available_resource every minute. So I have a small live migration window: 30 seconds every minute, only after update_available_resource ends and before it starts. | |
| 14:54:34 | sean-k-mooney | s10: why? | |
| 14:55:28 | jaypipes | s10: why are you running update_available_resource every minute? | |
| 14:55:45 | s10 | sean-k-mooney: I don't know, that's what I see in logs. No live migration could be performed during update_available_resources. If change libvirt/driver.py to like before first commits, migrations goes without pause. | |
| 14:56:02 | cdent | jaypipes: that's the periodic job default | |
| 14:56:15 | sean-k-mooney | s10: well that makes more sense | |
| 14:56:23 | jaypipes | quoi? wtf is it that often? :( | |
| 14:56:35 | sean-k-mooney | we are running the periodic job on a green thread | |
| 14:56:43 | cdent | jaypipes: dunno, but that's what it is | |
| 14:56:50 | jaypipes | sean-k-mooney: there is a semaphore lock around it. | |
| 14:57:26 | sean-k-mooney | so if update_available_resources is taking 30sec on a node with 150 instance then the compute agent would be tied up for 30secs every minute executing it | |
| 14:58:24 | s10 | jaypipes: because update_resources_interval run with default periodic interval with default value update_resources_interval=0, and periodic_task_interval=60 by default. | |
| 14:58:30 | sean-k-mooney | jaypipes: the point im trying to make is the periodic jobs are time sharing with the rest of the compute agent | |
| 14:59:18 | stephenfin | efried: Does POWER expose NUMA to the OS? | |
| 14:59:35 | sean-k-mooney | stephenfin: as in powerpc ? | |
| 14:59:41 | stephenfin | sean-k-mooney: up | |
| 14:59:42 | stephenfin | *yup | |
| 14:59:45 | dansmith | sean-k-mooney: but most of that is waiting for IO so it's not keeping the process busy | |
| 14:59:47 | mdbooth | lyarwood: I have another meeting now, so I've chucked a brain dump in the functional failure review. | |
| 14:59:56 | efried | stephenfin: My understanding is that POWER handles NUMA under the covers, and does it well enough that the deployer doesn't need control. | |
| 15:00:07 | sean-k-mooney | stephenfin: numa is exposed to linux via the bios so it should would the same on powerpc | |
| 15:00:27 | sean-k-mooney | the quest is does the hypervior expose it or not | |
| 15:00:30 | stephenfin | efried: Is that the hardware doing the work or the hypervisor? | |
| 15:00:33 | stephenfin | sean-k-mooney: yeah ^ | |
| 15:00:46 | stephenfin | (the work of abstracting NUMA'ness) | |
| 15:01:25 | sean-k-mooney | dansmith: good point it shoudl yeild exection | |
| 15:01:40 | jaypipes | cdent, dansmith: is there any reason now that placement claims are doing most of the resource consumption work (in an atomic manner) that we can't set the update_available_resource default to something sensible like 15 minutes? | |
| 15:02:42 | sean-k-mooney | jaypipes: just trying to think is there any late claim still dont in the compute node. i think not | |
| 15:02:50 | cdent | there's not | |
| 15:02:52 | dansmith | jaypipes: I would have to look, because I think we have other things hung off that process | |
| 15:03:27 | cdent | jaypipes: do we do anything with host states and Filters where that info needs to be up to date? | |
| 15:03:38 | cdent | non-placement-related filtering | |
| 15:03:59 | sean-k-mooney | stephenfin: im pretty sure that hyperv hides the numa info from nova | |
| 15:04:24 | jroll | jaypipes: one catch would be picking up new/changed ironic nodes, but maybe we can just doc that caveat | |
| 15:04:34 | sean-k-mooney | stephenfin: i know they have made numa affinity of instance memory a hypervir config option | |
| 15:04:46 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP libvirt: Reduce calls to qemu-img during update_available_resource https://review.openstack.org/589513 | |
| 15:05:00 | jroll | "changed" e.g. ironic node goes to / comes out of maintenance mode | |
| 15:05:05 | lyarwood | s10: ^ we can remove half of the qemu-img calls with that | |
| 15:07:29 | stephenfin | sean-k-mooney: Fair enough. I was trying to write a high-level overview of NUMA but needed to figure out what platforms would be affected. I'll just leave that piece out :) | |
| 15:08:22 | sean-k-mooney | stephenfin: i would recommend putting it in the context of which hypervior rather then which plathform e.g. achitecuter if you do | |
| 15:09:50 | sean-k-mooney | i would expect power-kvm via libvirt to work the same as kvm on x86 but i would expect powervm direct driver to work very different | |
| 15:11:18 | stephenfin | sean-k-mooney: Good call. I'll do that | |
| 15:11:29 | cdent | jaypipes: we could consider have a different periodic job for the placement calls? There's a bazillion periodic jobs in the compute manager already, aren't there? What's one more :() | |
| 15:12:25 | sean-k-mooney | cdent: they mainly all share the same config option today however. we dont have a interval time option for each | |
| 15:12:56 | sean-k-mooney | that said i dont see an issue with adding a new config option and default it to the current periodic job interval | |
| 15:13:30 | sean-k-mooney | or if you want the behavior change to be implicit on upgrade then default to 15 mins or whatever makes sense | |
| 15:13:51 | cdent | sean-k-mooney: there are several *_internal conf settings in nova/conf/compute.py and they all fall back to the default for any periodic in an oslo_service based service | |
| 15:14:06 | cdent | s/_internal/_interval/ | |
| 15:15:58 | efried | stephenfin, sean-k-mooney: Sorry, had to step away for a sec. From your point of view, the "hardware" and the "hypervisor" are the same thing. | |
| 15:16:35 | sean-k-mooney | efried: meaning we cant see the hardware so we should only care about what the hypervior reporst | |
| 15:17:12 | efried | sean-k-mooney: That's probably a sane way to think about it. I don't think the hypervisor reports NUMA cells at all. You see procs and memory. | |
| 15:17:27 | efried | I'm not an expert here, for sure. | |
| 15:17:54 | sean-k-mooney | efried: that is how the host is reported form hyperv also as far as i know | |
| 15:18:07 | sean-k-mooney | just one big pool for ram and cpus | |
| 15:18:39 | sean-k-mooney | they have a hypervior config option (not in openstack) to turn on numa affinity at the host level | |
| 15:19:11 | efried | yeah, see, I think NUMA affinity Just Happens (tm) on Power. | |
| 15:19:12 | jaypipes | cdent: perhaps, yes | |
| 15:19:38 | sean-k-mooney | efried: i dont think vsphere exposes it either. its just libvirt that exposes it as far as i can tell | |
| 15:19:55 | efried | frickin libvirt <rolls eyes> | |
| 15:20:43 | sean-k-mooney | hehe well libvirt exposes it because libvirt is not a hypervior its an hypervior abstraction layer and isnce qemu does not do this by default it fell to nova to fill in the gapps | |
| 15:21:04 | efried | :) | |
| 15:22:08 | jaypipes | sean-k-mooney: WRONG! libvirt is an XML file management system. | |
| 15:22:40 | sean-k-mooney | jaypipes: lol | |
| 15:22:58 | sean-k-mooney | jaypipes: it does a little bit more then that but more or less | |
| 15:24:24 | sean-k-mooney | you know it could be argured that a new service could be insrted between nova and libvirt that did all the nfv stuff and the libvirt driver could be made way simpeler | |
| 15:25:24 | cdent | I sure hope those unicorns are free range | |
| 15:35:10 | s10 | lyarwood: with https://review.openstack.org/589513 update_available_resource() lasts 10 seconds for 100 instances instead of 20 seconds without. | |
| 15:39:13 | lyarwood | s10: kk, that's a single qemu-img info call per disk to avoid the other issues fixed by the original changes | |
| 15:39:52 | dansmith | s10: this sounds like a silly question, but what does it matter how long that takes? | |
| 15:39:57 | dansmith | it's not blocking other work right? | |