| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-08-07 | |||
| 14:59:42 | stephenfin | *yup | |
| 14:59:45 | dansmith | sean-k-mooney: but most of that is waiting for IO so it's not keeping the process busy | |
| 14:59:47 | mdbooth | lyarwood: I have another meeting now, so I've chucked a brain dump in the functional failure review. | |
| 14:59:56 | efried | stephenfin: My understanding is that POWER handles NUMA under the covers, and does it well enough that the deployer doesn't need control. | |
| 15:00:07 | sean-k-mooney | stephenfin: numa is exposed to linux via the bios so it should would the same on powerpc | |
| 15:00:27 | sean-k-mooney | the quest is does the hypervior expose it or not | |
| 15:00:30 | stephenfin | efried: Is that the hardware doing the work or the hypervisor? | |
| 15:00:33 | stephenfin | sean-k-mooney: yeah ^ | |
| 15:00:46 | stephenfin | (the work of abstracting NUMA'ness) | |
| 15:01:25 | sean-k-mooney | dansmith: good point it shoudl yeild exection | |
| 15:01:40 | jaypipes | cdent, dansmith: is there any reason now that placement claims are doing most of the resource consumption work (in an atomic manner) that we can't set the update_available_resource default to something sensible like 15 minutes? | |
| 15:02:42 | sean-k-mooney | jaypipes: just trying to think is there any late claim still dont in the compute node. i think not | |
| 15:02:50 | cdent | there's not | |
| 15:02:52 | dansmith | jaypipes: I would have to look, because I think we have other things hung off that process | |
| 15:03:27 | cdent | jaypipes: do we do anything with host states and Filters where that info needs to be up to date? | |
| 15:03:38 | cdent | non-placement-related filtering | |
| 15:03:59 | sean-k-mooney | stephenfin: im pretty sure that hyperv hides the numa info from nova | |
| 15:04:24 | jroll | jaypipes: one catch would be picking up new/changed ironic nodes, but maybe we can just doc that caveat | |
| 15:04:34 | sean-k-mooney | stephenfin: i know they have made numa affinity of instance memory a hypervir config option | |
| 15:04:46 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP libvirt: Reduce calls to qemu-img during update_available_resource https://review.openstack.org/589513 | |
| 15:05:00 | jroll | "changed" e.g. ironic node goes to / comes out of maintenance mode | |
| 15:05:05 | lyarwood | s10: ^ we can remove half of the qemu-img calls with that | |
| 15:07:29 | stephenfin | sean-k-mooney: Fair enough. I was trying to write a high-level overview of NUMA but needed to figure out what platforms would be affected. I'll just leave that piece out :) | |
| 15:08:22 | sean-k-mooney | stephenfin: i would recommend putting it in the context of which hypervior rather then which plathform e.g. achitecuter if you do | |
| 15:09:50 | sean-k-mooney | i would expect power-kvm via libvirt to work the same as kvm on x86 but i would expect powervm direct driver to work very different | |
| 15:11:18 | stephenfin | sean-k-mooney: Good call. I'll do that | |
| 15:11:29 | cdent | jaypipes: we could consider have a different periodic job for the placement calls? There's a bazillion periodic jobs in the compute manager already, aren't there? What's one more :() | |
| 15:12:25 | sean-k-mooney | cdent: they mainly all share the same config option today however. we dont have a interval time option for each | |
| 15:12:56 | sean-k-mooney | that said i dont see an issue with adding a new config option and default it to the current periodic job interval | |
| 15:13:30 | sean-k-mooney | or if you want the behavior change to be implicit on upgrade then default to 15 mins or whatever makes sense | |
| 15:13:51 | cdent | sean-k-mooney: there are several *_internal conf settings in nova/conf/compute.py and they all fall back to the default for any periodic in an oslo_service based service | |
| 15:14:06 | cdent | s/_internal/_interval/ | |
| 15:15:58 | efried | stephenfin, sean-k-mooney: Sorry, had to step away for a sec. From your point of view, the "hardware" and the "hypervisor" are the same thing. | |
| 15:16:35 | sean-k-mooney | efried: meaning we cant see the hardware so we should only care about what the hypervior reporst | |
| 15:17:12 | efried | sean-k-mooney: That's probably a sane way to think about it. I don't think the hypervisor reports NUMA cells at all. You see procs and memory. | |
| 15:17:27 | efried | I'm not an expert here, for sure. | |
| 15:17:54 | sean-k-mooney | efried: that is how the host is reported form hyperv also as far as i know | |
| 15:18:07 | sean-k-mooney | just one big pool for ram and cpus | |
| 15:18:39 | sean-k-mooney | they have a hypervior config option (not in openstack) to turn on numa affinity at the host level | |
| 15:19:11 | efried | yeah, see, I think NUMA affinity Just Happens (tm) on Power. | |
| 15:19:12 | jaypipes | cdent: perhaps, yes | |
| 15:19:38 | sean-k-mooney | efried: i dont think vsphere exposes it either. its just libvirt that exposes it as far as i can tell | |
| 15:19:55 | efried | frickin libvirt <rolls eyes> | |
| 15:20:43 | sean-k-mooney | hehe well libvirt exposes it because libvirt is not a hypervior its an hypervior abstraction layer and isnce qemu does not do this by default it fell to nova to fill in the gapps | |
| 15:21:04 | efried | :) | |
| 15:22:08 | jaypipes | sean-k-mooney: WRONG! libvirt is an XML file management system. | |
| 15:22:40 | sean-k-mooney | jaypipes: lol | |
| 15:22:58 | sean-k-mooney | jaypipes: it does a little bit more then that but more or less | |
| 15:24:24 | sean-k-mooney | you know it could be argured that a new service could be insrted between nova and libvirt that did all the nfv stuff and the libvirt driver could be made way simpeler | |
| 15:25:24 | cdent | I sure hope those unicorns are free range | |
| 15:35:10 | s10 | lyarwood: with https://review.openstack.org/589513 update_available_resource() lasts 10 seconds for 100 instances instead of 20 seconds without. | |
| 15:39:13 | lyarwood | s10: kk, that's a single qemu-img info call per disk to avoid the other issues fixed by the original changes | |
| 15:39:52 | dansmith | s10: this sounds like a silly question, but what does it matter how long that takes? | |
| 15:39:57 | dansmith | it's not blocking other work right? | |
| 15:40:36 | efried | stephenfin: I think https://review.openstack.org/#/c/588422/ is ready for your +A now | |
| 15:40:46 | stephenfin | ack | |
| 15:40:53 | lyarwood | dansmith: I did ask above and it's causing issues with LM apparently | |
| 15:41:09 | dansmith | lyarwood: oh sorry I missed that.. maybe because it's holding the RT semaphore? | |
| 15:41:42 | lyarwood | dansmith: yes I think so | |
| 15:41:53 | dansmith | okay makes sense I guess | |
| 15:42:19 | dansmith | lyarwood: the new call checks the allocated value, which doesn't change over the lifecycle of the image right? | |
| 15:42:54 | sean-k-mooney | dansmith: correct it should not | |
| 15:43:26 | lyarwood | dansmith: allocated can, virtual shouldn't | |
| 15:43:26 | sean-k-mooney | dansmith: unless the call is checking the actul used space on the host and we are not preallocting ? | |
| 15:43:36 | lyarwood | yeah pretty much | |
| 15:43:53 | dansmith | lyarwood: but which are we looking at now? | |
| 15:44:09 | lyarwood | dansmith: now it's just a single qemu-img info call that grabs both | |
| 15:44:15 | sean-k-mooney | lyarwood: the resouce track should be tracking the virtual amont right not the currently allcoated amount? | |
| 15:44:33 | dansmith | lyarwood: ah, both so one of them is dynamic and we can't really cache it yeah? | |
| 15:44:34 | lyarwood | sean-k-mooney: the allocated amount is used to work out over commit etc | |
| 15:44:58 | sean-k-mooney | lyarwood: that seams wrong the maxium it could use should be used for that | |
| 15:45:30 | dansmith | sean-k-mooney: IIRC, this is not for reporting to placement but for some of the legacy values (right lyarwood ?) | |
| 15:45:32 | openstack | Launchpad bug 1784826 in OpenStack Compute (nova) "Guest remain in origin host after evacuate and unset force-down nova-compute" [Undecided,In progress] - Assigned to huanhongda (hongda) | |
| 15:45:32 | jaypipes | dansmith: what are your thoughts on https://bugs.launchpad.net/nova/+bug/1784826? I can't tell what the expected behaviour there should be... | |
| 15:45:40 | dansmith | because auditing on the compute node doesn't change what placement is counting | |
| 15:46:04 | lyarwood | dansmith: yeah correct, this doesn't make it to placement AFAIK | |
| 15:47:04 | dansmith | lyarwood: so we really only need to be collecting this info for old RT, which is ignored if you don't have DiskFilter enabled, and only useful for people using the deprecated CachingScheduler yeah? | |
| 15:47:28 | sean-k-mooney | lyarwood: what advantage is there to using allocated though? since placement would already be using its allocation ratio to filter the hosts before the compute ever need to check its over_subsciption ratio setting | |
| 15:47:54 | lyarwood | dansmith: in master but this series was backported all the way back to ocata | |
| 15:48:16 | lyarwood | dansmith: wasn't it still used back then? | |
| 15:48:29 | dansmith | lyarwood: yeah I know, but diskfilter hasn't been needed since claims in the scheduler | |
| 15:48:36 | dansmith | *maybe* ocata, but not pike or later IIRC | |
| 15:50:40 | lyarwood | sean-k-mooney: right, I think the only reason this came up before is that it's visable from the CLI | |
| 15:50:46 | dansmith | lyarwood: anyway, just wondering if maybe we should add a workaround config tweak to disable this extra inspection so you can turn it off if you're not using DiskFilter and/or are on a new enough version | |
| 15:51:04 | lyarwood | dansmith: yeah that sounds like the way to go tbh | |
| 15:51:19 | dansmith | lyarwood: ah, well, that makes it even more pain than gain if it was just "make the numbers line up" :) | |
| 15:52:04 | dansmith | lyarwood: probably best to get some sign-offs from the PTL(s) of the affected releases, but that's kinda where I'm thinking | |
| 15:52:14 | sean-k-mooney | lyarwood: so if its visable via the cli i think thats even more reason to use the virtual not allocated disk size to have it corralte with placement | |
| 15:52:53 | sean-k-mooney | lyarwood: that said that would be a behavior change i guess. | |
| 15:53:22 | sean-k-mooney | lyarwood: i assume this is visable via the hyperviors api? | |
| 15:54:34 | openstack | Launchpad bug 1764489 in OpenStack Compute (nova) queens "Preallocated disks are deducted twice from disk_available_least when using preallocated_images = space" [Medium,Fix committed] - Assigned to Lee Yarwood (lyarwood) | |
| 15:54:34 | lyarwood | sean-k-mooney: yeah via a hypervisor-show - https://bugs.launchpad.net/nova/+bug/1764489 | |
| 15:56:41 | sean-k-mooney | right so its messing up disk_available_least not local_gb_used... | |
| 16:01:44 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Use placement 1.28 in scheduler report client https://review.openstack.org/583667 | |
| 16:11:02 | mdbooth | lyarwood: I'm partially through the functional failure. Got past the shelve/unshelve failure, error is now in _test_attach_volume_error | |
| 16:11:51 | mdbooth | lyarwood: However, I'm going to leave shortly, it doesn't pass yet, and there are still print statements all over it | |
| 16:12:06 | melwitt | . | |
| 16:12:19 | mdbooth | lyarwood: So I'm not inclined to push it this evening unless that would be especially helpful to you | |
| 16:15:04 | lyarwood | mdbooth: yeah np leave it for now | |
| 16:50:48 | openstackgerrit | Sergii Golovatiuk proposed openstack/nova master: Fix URI for IPv6 https://review.openstack.org/589548 | |