Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-08-07
16:57:56 melwitt dansmith: this review looks in your wheelhouse https://review.openstack.org/589425
16:58:37 dansmith yeah, gibi and I have been discussing it
16:58:51 dansmith I'll hit it in a few
16:59:02 melwitt k
17:10:07 efried stephenfin, sean-k-mooney: I found out a little more about NUMA affinity on Power. It is indeed done dynamically; you can't ask for it to be done a certain way. However, the NUMA cell and affinity information *is* exposed to the guest, so if the guest has the savvy and ability to use such information for clever scheduling of threads/processes, it can do so.
17:11:39 efried stephenfin, sean-k-mooney: Also of note, the system may swizzle affinities dynamically for various reasons (including, apparently, at the behest of a generic "improve my affinity" background job you can run) so the guest needs to be able to hang with that.
17:26:22 openstackgerrit Dan Smith proposed openstack/nova master: Fix resize revert to use non-legacy alloc handling https://review.openstack.org/589425
17:38:07 sean-k-mooney efried: stephenfin so am i right in saying i can request a partcalar virtual numa topology for the guest but powervm will dynmically map that as it sees fit and adjust as needed
17:39:18 sean-k-mooney brb
18:25:28 efried sean-k-mooney: You can *not* request a particular topology for a guest. PowerVM will set it up dynamically and adjust as needed.
18:29:51 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP libvirt: Reduce calls to qemu-img during update_available_resource https://review.openstack.org/589513
18:29:52 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP libvirt: Add workaround to stop use of qemu-img by the RT https://review.openstack.org/589567
18:29:58 lyarwood dansmith: ^ was that what you had in mind for the qemu-img workaround? If it is I'll clean it up and ping the ML to see if others agree.
18:30:49 dansmith lyarwood: kinda, but we weren't just returning zero before right? what were we doing?
18:31:00 dansmith I was thinking just make the workaround trigger "do the old thing"
18:31:49 lyarwood dansmith: well before we were calling os.path.getsize
18:32:13 dansmith ah right, so .. I would just make the workaround revert to that I think
18:45:09 sean-k-mooney efried: does that imply that from within the guest i can see that the topoloyg is chaning?
18:45:27 efried sean-k-mooney: Yeah. Is that crazy?
18:45:32 sean-k-mooney efried: yes
18:45:53 efried sean-k-mooney: It's possible that the topo only changes if you run that job I mentioned.
18:46:01 sean-k-mooney windows and linux are really not going to do the right thing
18:46:42 sean-k-mooney im pretry sure that neither will check the numa topology on an ongoing basis
18:47:26 sean-k-mooney so the linux schduler is going to detect the numa topology when it start up and then contiue to use the same mappings for the liftime of the vm
18:51:06 sean-k-mooney if the hyperviors hide the phyical topology from the guest but dynamically rempas the ram across the host numa nodes then atleast the info present to the guest is consitent and it can try to make sane decisions but if the virtual tolopogy the guest sees is dynamic things like numactl are going to not work correctly in the guest
18:53:27 efried sean-k-mooney: Yeah, I don't know to this level of detail. I imagine it'd be in response to other LPARs going away or whatever, and thus freeing up the system to shuffle my stuff to make it more efficient. Or maybe I hot plug a device and now it's better to move my procs/mem to the cell that's affined to that device. But I don't know how that actually appears to the guest.
18:55:25 sean-k-mooney efried: well the guest view is what i was asking about when i said can you request a virtual numa toplogy. as long as the geust view does not change the hypervior can do what it like underneath without worriying it will break the guest
18:56:19 sean-k-mooney the guest can still make bad desisions if the hyperviour dicides to split a virutal numa node across host phyical numa nodes but it wont break anything
18:57:09 sean-k-mooney if the gust view is changing things like hugepages within the guest would break
18:57:20 efried sean-k-mooney: When you say "request a virtual numa topology" do you mean "ask what the topology is" or do you mean "request that the virtual topology look like XYZ" ?
18:57:40 efried s/look like/get set up as/
18:58:50 sean-k-mooney i mean "power vm please create a vm that to the guest looks like it has 2 numa node, do whatever you like on the host"
18:59:25 sean-k-mooney that is effectivly what hw:numa_nodes=2 means
18:59:26 efried sean-k-mooney: Right, okay, so I'm saying you definitely can't do that in PowerVM. I'm not sure whether you can do it in PowerKVM.
18:59:42 sean-k-mooney ok cool
19:00:25 sean-k-mooney you can do that with hyperv i think but i dont think i guarntees they are mapped to 2 phyical host numa nodes
19:01:14 sean-k-mooney or rather if they are mapped/affinitised to host numa nodes is governed by a hypervior config option as far as i know
19:14:52 sean-k-mooney efried: its burried in the docs but ya it looks like you can do numa affinity with powerkvm https://www.ibm.com/support/knowledgecenter/SSZJY4_3.1.0/liabp/liabphotplugcpunuma.htm
19:15:11 sean-k-mooney you can also do cpu pinning
19:15:17 efried orly?
19:15:58 sean-k-mooney yep and discribe the virual cpu topoloyg in terms of socket,cores and thread
19:16:05 efried sean-k-mooney: This appears to be talking about just virtual topology, yah?
19:16:23 sean-k-mooney yes unlsee you can use the numa tune elements
19:18:04 sean-k-mooney efried: so ya this all apears to be virtual thought the docs are so hard to navigate one could almost belive ibm did not want you to use those features :P
19:18:25 efried sean-k-mooney: Oh, don't worry, that's not specific to NUMA.
19:20:36 sean-k-mooney haha well i partcalarly like how the numa stuff is not linked too for the "manage processor and memory" section https://www.ibm.com/support/knowledgecenter/en/SSZJY4_3.1.0/liabp/liabpmanageprocessors.htm
19:22:20 efried sean-k-mooney: I think I can now officially say you've spent more time on the IBM "knowledge" center in the past twenty minutes than I have in the past five years.
19:23:59 sean-k-mooney haha well from what i can tell if you used the libvirt kvm driver an pointed it at a power-kvm host i think it would "just work" with all of the nuam and epa stuff we do for normal kvm
19:24:24 efried sean-k-mooney: I have found out that there is some kind of notification the OS can get when affinities change.
19:24:49 sean-k-mooney proably some ahci interupt or something
19:25:29 sean-k-mooney i know there are intrupts that can be sent for memory and cpu hotplug which argubly could change numa topology
19:26:31 sean-k-mooney so it not out of the question that linux or widnows could handel a dynmaic topology if you were to notify it. not sure i would like a numa node to go away but adding one might be ok
19:27:14 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP libvirt: Add workaround to stop use of qemu-img by the RT https://review.openstack.org/589567
19:27:36 efried sean-k-mooney: I think it's more likely to be like, "You thought procs 0 and 2 were next to each other, and 1 and 3 were next to each other. Well, now it's 0-1 and 2-3. Deal."
19:28:23 sean-k-mooney haha ya maybe. if you ever test it let me know
19:28:36 efried sean-k-mooney: I feel confident in saying that will never happen.
19:28:58 efried ...for some value of "never".
19:29:11 sean-k-mooney efried: you obvioulsy are not spending enough time talking to telcos
19:30:11 efried If it's anything like talking to the AT&T help desk, I assure you I don't want to spend any more.
19:46:49 openstackgerrit Matt Riedemann proposed openstack/nova stable/pike: Make ResourceTracker.stats node-specific https://review.openstack.org/588037
19:49:03 mriedem was there a zuul reset or something such that we need to recheck things?
19:51:06 dansmith 513 backlog,
19:51:10 dansmith so I imagine it's catching up
19:51:35 dansmith backlog has been up and down all day so I figured something was going on
19:53:31 sean-k-mooney nothing on the openstack-infra twitter for the last 4 days. then normally post somthing if they need to do a reset
19:54:27 sean-k-mooney well it auto posts when then set a status message on the infra channel but in either case the gate is proably just busy/backed up
19:56:20 mriedem ok
20:00:14 openstackgerrit Matt Riedemann proposed openstack/nova stable/ocata: Make ResourceTracker.stats node-specific https://review.openstack.org/588077
20:20:58 openstackgerrit Matt Riedemann proposed openstack/nova master: doc: mark the max microversion for rocky https://review.openstack.org/589598
20:31:30 maciejjozefczyk mriedem: efried please check my response: https://review.openstack.org/#/c/520024
20:32:11 efried Thanks maciejjozefczyk
20:32:47 maciejjozefczyk efried: u2 for pinging me, I forgot about this topic
20:37:34 mriedem melwitt: comments in the reno prelude https://review.openstack.org/#/c/589303/
20:37:46 melwitt ack
20:43:17 mriedem jroll: you might want to check the rebalance question in here https://review.openstack.org/#/c/520024/
20:49:53 jroll "want" is a weird word to use :P
20:49:59 jroll looking
20:53:03 mriedem i'm just asking for a witness that i asked for an ironic person to look before we break the rebalance stuff
20:53:08 jroll heh
20:54:04 jroll so I'm thinking it's fine to remove that, but... I don't know everything that _update does and I'm not sure things like _update_usages_from_instances() and such will do the right thing, if we haven't updated which compute hosts we moved active ironic instances to
20:55:07 jroll I feel like I'm nervous to remove it
20:56:40 sean-k-mooney mriedem: is there an ironic job we can trigger maybe via the experimental pipelien to test that patch
20:57:32 jroll sean-k-mooney: there isn't one that actively kills compute hosts and such to trigger rebalances
20:58:40 sean-k-mooney jroll: it looks like the only ironic job that did run on this skipp 22 out of the 23 test ... http://logs.openstack.org/24/520024/9/check/ironic-tempest-dsvm-ipa-wholedisk-bios-agent_ipmitool-tinyipa/1fe9001/testr_results.html.gz
21:07:34 jroll sean-k-mooney: yes, we run a very small set of tests on nova changes because our tests take a long time
21:08:01 jroll sean-k-mooney: that doesn't change the fact that there aren't any full-stack tests that exercise this code
21:08:37 jroll because it's really hard to do - would involve multiple nova-compute instances and orchestrating bringing them up and down and such
21:08:58 efried just mock everything
21:09:59 jroll anyway, I'm gonna post my comments on the change and then not computer anymore today. see y'all later
21:11:57 sean-k-mooney jroll: im not saying there would be test for this just taking note of the fact that the ironic testing is much lighter then i would have expected
21:12:44 sean-k-mooney jroll: that is not a slight againt ironic i just would have expect more tests even if they only ran in the gate pipeline and not check
21:12:50 jroll sean-k-mooney: figure out how to get nested virt in the gate and we can run them fast enough to test more :)
21:13:31 jroll that one instance boot takes like 20 minutes or something iirc
21:14:00 sean-k-mooney whell i honestly dont know why we dont use nested virt in the gate other then the fact that rackspace provides xen based instance and most of our test would like kvm
21:14:13 jroll there's a long history of why that I don't have time to get into
21:14:46 sean-k-mooney jroll: i have been trying to get nested virt in the gate since 2013 im well awaree of the history but no worries
21:18:22 melwitt only sometimes?
21:18:25 sean-k-mooney i just spent 6 hours debuging network connectivit issue with ovs-dpdk because it has buggy numa detection logic
21:19:30 sean-k-mooney melwitt: normally it works fine but when it doesnt its a pain in the ass to figure out why
21:31:10 mriedem dansmith: do you have any idea why we have a Service.availability_zone field on the versioned object?

Earlier   Later