Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-08-07
15:47:54 lyarwood dansmith: in master but this series was backported all the way back to ocata
15:48:16 lyarwood dansmith: wasn't it still used back then?
15:48:29 dansmith lyarwood: yeah I know, but diskfilter hasn't been needed since claims in the scheduler
15:48:36 dansmith *maybe* ocata, but not pike or later IIRC
15:50:40 lyarwood sean-k-mooney: right, I think the only reason this came up before is that it's visable from the CLI
15:50:46 dansmith lyarwood: anyway, just wondering if maybe we should add a workaround config tweak to disable this extra inspection so you can turn it off if you're not using DiskFilter and/or are on a new enough version
15:51:04 lyarwood dansmith: yeah that sounds like the way to go tbh
15:51:19 dansmith lyarwood: ah, well, that makes it even more pain than gain if it was just "make the numbers line up" :)
15:52:04 dansmith lyarwood: probably best to get some sign-offs from the PTL(s) of the affected releases, but that's kinda where I'm thinking
15:52:14 sean-k-mooney lyarwood: so if its visable via the cli i think thats even more reason to use the virtual not allocated disk size to have it corralte with placement
15:52:53 sean-k-mooney lyarwood: that said that would be a behavior change i guess.
15:53:22 sean-k-mooney lyarwood: i assume this is visable via the hyperviors api?
15:54:34 lyarwood sean-k-mooney: yeah via a hypervisor-show - https://bugs.launchpad.net/nova/+bug/1764489
15:54:34 openstack Launchpad bug 1764489 in OpenStack Compute (nova) queens "Preallocated disks are deducted twice from disk_available_least when using preallocated_images = space" [Medium,Fix committed] - Assigned to Lee Yarwood (lyarwood)
15:56:41 sean-k-mooney right so its messing up disk_available_least not local_gb_used...
16:01:44 openstackgerrit Balazs Gibizer proposed openstack/nova master: Use placement 1.28 in scheduler report client https://review.openstack.org/583667
16:11:02 mdbooth lyarwood: I'm partially through the functional failure. Got past the shelve/unshelve failure, error is now in _test_attach_volume_error
16:11:51 mdbooth lyarwood: However, I'm going to leave shortly, it doesn't pass yet, and there are still print statements all over it
16:12:06 melwitt .
16:12:19 mdbooth lyarwood: So I'm not inclined to push it this evening unless that would be especially helpful to you
16:15:04 lyarwood mdbooth: yeah np leave it for now
16:50:48 openstackgerrit Sergii Golovatiuk proposed openstack/nova master: Fix URI for IPv6 https://review.openstack.org/589548
16:57:56 melwitt dansmith: this review looks in your wheelhouse https://review.openstack.org/589425
16:58:37 dansmith yeah, gibi and I have been discussing it
16:58:51 dansmith I'll hit it in a few
16:59:02 melwitt k
17:10:07 efried stephenfin, sean-k-mooney: I found out a little more about NUMA affinity on Power. It is indeed done dynamically; you can't ask for it to be done a certain way. However, the NUMA cell and affinity information *is* exposed to the guest, so if the guest has the savvy and ability to use such information for clever scheduling of threads/processes, it can do so.
17:11:39 efried stephenfin, sean-k-mooney: Also of note, the system may swizzle affinities dynamically for various reasons (including, apparently, at the behest of a generic "improve my affinity" background job you can run) so the guest needs to be able to hang with that.
17:26:22 openstackgerrit Dan Smith proposed openstack/nova master: Fix resize revert to use non-legacy alloc handling https://review.openstack.org/589425
17:38:07 sean-k-mooney efried: stephenfin so am i right in saying i can request a partcalar virtual numa topology for the guest but powervm will dynmically map that as it sees fit and adjust as needed
17:39:18 sean-k-mooney brb
18:25:28 efried sean-k-mooney: You can *not* request a particular topology for a guest. PowerVM will set it up dynamically and adjust as needed.
18:29:51 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP libvirt: Reduce calls to qemu-img during update_available_resource https://review.openstack.org/589513
18:29:52 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP libvirt: Add workaround to stop use of qemu-img by the RT https://review.openstack.org/589567
18:29:58 lyarwood dansmith: ^ was that what you had in mind for the qemu-img workaround? If it is I'll clean it up and ping the ML to see if others agree.
18:30:49 dansmith lyarwood: kinda, but we weren't just returning zero before right? what were we doing?
18:31:00 dansmith I was thinking just make the workaround trigger "do the old thing"
18:31:49 lyarwood dansmith: well before we were calling os.path.getsize
18:32:13 dansmith ah right, so .. I would just make the workaround revert to that I think
18:45:09 sean-k-mooney efried: does that imply that from within the guest i can see that the topoloyg is chaning?
18:45:27 efried sean-k-mooney: Yeah. Is that crazy?
18:45:32 sean-k-mooney efried: yes
18:45:53 efried sean-k-mooney: It's possible that the topo only changes if you run that job I mentioned.
18:46:01 sean-k-mooney windows and linux are really not going to do the right thing
18:46:42 sean-k-mooney im pretry sure that neither will check the numa topology on an ongoing basis
18:47:26 sean-k-mooney so the linux schduler is going to detect the numa topology when it start up and then contiue to use the same mappings for the liftime of the vm
18:51:06 sean-k-mooney if the hyperviors hide the phyical topology from the guest but dynamically rempas the ram across the host numa nodes then atleast the info present to the guest is consitent and it can try to make sane decisions but if the virtual tolopogy the guest sees is dynamic things like numactl are going to not work correctly in the guest
18:53:27 efried sean-k-mooney: Yeah, I don't know to this level of detail. I imagine it'd be in response to other LPARs going away or whatever, and thus freeing up the system to shuffle my stuff to make it more efficient. Or maybe I hot plug a device and now it's better to move my procs/mem to the cell that's affined to that device. But I don't know how that actually appears to the guest.
18:55:25 sean-k-mooney efried: well the guest view is what i was asking about when i said can you request a virtual numa toplogy. as long as the geust view does not change the hypervior can do what it like underneath without worriying it will break the guest
18:56:19 sean-k-mooney the guest can still make bad desisions if the hyperviour dicides to split a virutal numa node across host phyical numa nodes but it wont break anything
18:57:09 sean-k-mooney if the gust view is changing things like hugepages within the guest would break
18:57:20 efried sean-k-mooney: When you say "request a virtual numa topology" do you mean "ask what the topology is" or do you mean "request that the virtual topology look like XYZ" ?
18:57:40 efried s/look like/get set up as/
18:58:50 sean-k-mooney i mean "power vm please create a vm that to the guest looks like it has 2 numa node, do whatever you like on the host"
18:59:25 sean-k-mooney that is effectivly what hw:numa_nodes=2 means
18:59:26 efried sean-k-mooney: Right, okay, so I'm saying you definitely can't do that in PowerVM. I'm not sure whether you can do it in PowerKVM.
18:59:42 sean-k-mooney ok cool
19:00:25 sean-k-mooney you can do that with hyperv i think but i dont think i guarntees they are mapped to 2 phyical host numa nodes
19:01:14 sean-k-mooney or rather if they are mapped/affinitised to host numa nodes is governed by a hypervior config option as far as i know
19:14:52 sean-k-mooney efried: its burried in the docs but ya it looks like you can do numa affinity with powerkvm https://www.ibm.com/support/knowledgecenter/SSZJY4_3.1.0/liabp/liabphotplugcpunuma.htm
19:15:11 sean-k-mooney you can also do cpu pinning
19:15:17 efried orly?
19:15:58 sean-k-mooney yep and discribe the virual cpu topoloyg in terms of socket,cores and thread
19:16:05 efried sean-k-mooney: This appears to be talking about just virtual topology, yah?
19:16:23 sean-k-mooney yes unlsee you can use the numa tune elements
19:18:04 sean-k-mooney efried: so ya this all apears to be virtual thought the docs are so hard to navigate one could almost belive ibm did not want you to use those features :P
19:18:25 efried sean-k-mooney: Oh, don't worry, that's not specific to NUMA.
19:20:36 sean-k-mooney haha well i partcalarly like how the numa stuff is not linked too for the "manage processor and memory" section https://www.ibm.com/support/knowledgecenter/en/SSZJY4_3.1.0/liabp/liabpmanageprocessors.htm
19:22:20 efried sean-k-mooney: I think I can now officially say you've spent more time on the IBM "knowledge" center in the past twenty minutes than I have in the past five years.
19:23:59 sean-k-mooney haha well from what i can tell if you used the libvirt kvm driver an pointed it at a power-kvm host i think it would "just work" with all of the nuam and epa stuff we do for normal kvm
19:24:24 efried sean-k-mooney: I have found out that there is some kind of notification the OS can get when affinities change.
19:24:49 sean-k-mooney proably some ahci interupt or something
19:25:29 sean-k-mooney i know there are intrupts that can be sent for memory and cpu hotplug which argubly could change numa topology
19:26:31 sean-k-mooney so it not out of the question that linux or widnows could handel a dynmaic topology if you were to notify it. not sure i would like a numa node to go away but adding one might be ok
19:27:14 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP libvirt: Add workaround to stop use of qemu-img by the RT https://review.openstack.org/589567
19:27:36 efried sean-k-mooney: I think it's more likely to be like, "You thought procs 0 and 2 were next to each other, and 1 and 3 were next to each other. Well, now it's 0-1 and 2-3. Deal."
19:28:23 sean-k-mooney haha ya maybe. if you ever test it let me know
19:28:36 efried sean-k-mooney: I feel confident in saying that will never happen.
19:28:58 efried ...for some value of "never".
19:29:11 sean-k-mooney efried: you obvioulsy are not spending enough time talking to telcos
19:30:11 efried If it's anything like talking to the AT&T help desk, I assure you I don't want to spend any more.
19:46:49 openstackgerrit Matt Riedemann proposed openstack/nova stable/pike: Make ResourceTracker.stats node-specific https://review.openstack.org/588037
19:49:03 mriedem was there a zuul reset or something such that we need to recheck things?
19:51:06 dansmith 513 backlog,
19:51:10 dansmith so I imagine it's catching up
19:51:35 dansmith backlog has been up and down all day so I figured something was going on
19:53:31 sean-k-mooney nothing on the openstack-infra twitter for the last 4 days. then normally post somthing if they need to do a reset
19:54:27 sean-k-mooney well it auto posts when then set a status message on the infra channel but in either case the gate is proably just busy/backed up
19:56:20 mriedem ok
20:00:14 openstackgerrit Matt Riedemann proposed openstack/nova stable/ocata: Make ResourceTracker.stats node-specific https://review.openstack.org/588077
20:20:58 openstackgerrit Matt Riedemann proposed openstack/nova master: doc: mark the max microversion for rocky https://review.openstack.org/589598
20:31:30 maciejjozefczyk mriedem: efried please check my response: https://review.openstack.org/#/c/520024
20:32:11 efried Thanks maciejjozefczyk
20:32:47 maciejjozefczyk efried: u2 for pinging me, I forgot about this topic
20:37:34 mriedem melwitt: comments in the reno prelude https://review.openstack.org/#/c/589303/
20:37:46 melwitt ack
20:43:17 mriedem jroll: you might want to check the rebalance question in here https://review.openstack.org/#/c/520024/
20:49:53 jroll "want" is a weird word to use :P
20:49:59 jroll looking
20:53:03 mriedem i'm just asking for a witness that i asked for an ironic person to look before we break the rebalance stuff

Earlier   Later