| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-08-07 | |||
| 18:47:26 | sean-k-mooney | so the linux schduler is going to detect the numa topology when it start up and then contiue to use the same mappings for the liftime of the vm | |
| 18:51:06 | sean-k-mooney | if the hyperviors hide the phyical topology from the guest but dynamically rempas the ram across the host numa nodes then atleast the info present to the guest is consitent and it can try to make sane decisions but if the virtual tolopogy the guest sees is dynamic things like numactl are going to not work correctly in the guest | |
| 18:53:27 | efried | sean-k-mooney: Yeah, I don't know to this level of detail. I imagine it'd be in response to other LPARs going away or whatever, and thus freeing up the system to shuffle my stuff to make it more efficient. Or maybe I hot plug a device and now it's better to move my procs/mem to the cell that's affined to that device. But I don't know how that actually appears to the guest. | |
| 18:55:25 | sean-k-mooney | efried: well the guest view is what i was asking about when i said can you request a virtual numa toplogy. as long as the geust view does not change the hypervior can do what it like underneath without worriying it will break the guest | |
| 18:56:19 | sean-k-mooney | the guest can still make bad desisions if the hyperviour dicides to split a virutal numa node across host phyical numa nodes but it wont break anything | |
| 18:57:09 | sean-k-mooney | if the gust view is changing things like hugepages within the guest would break | |
| 18:57:20 | efried | sean-k-mooney: When you say "request a virtual numa topology" do you mean "ask what the topology is" or do you mean "request that the virtual topology look like XYZ" ? | |
| 18:57:40 | efried | s/look like/get set up as/ | |
| 18:58:50 | sean-k-mooney | i mean "power vm please create a vm that to the guest looks like it has 2 numa node, do whatever you like on the host" | |
| 18:59:25 | sean-k-mooney | that is effectivly what hw:numa_nodes=2 means | |
| 18:59:26 | efried | sean-k-mooney: Right, okay, so I'm saying you definitely can't do that in PowerVM. I'm not sure whether you can do it in PowerKVM. | |
| 18:59:42 | sean-k-mooney | ok cool | |
| 19:00:25 | sean-k-mooney | you can do that with hyperv i think but i dont think i guarntees they are mapped to 2 phyical host numa nodes | |
| 19:01:14 | sean-k-mooney | or rather if they are mapped/affinitised to host numa nodes is governed by a hypervior config option as far as i know | |
| 19:14:52 | sean-k-mooney | efried: its burried in the docs but ya it looks like you can do numa affinity with powerkvm https://www.ibm.com/support/knowledgecenter/SSZJY4_3.1.0/liabp/liabphotplugcpunuma.htm | |
| 19:15:11 | sean-k-mooney | you can also do cpu pinning | |
| 19:15:17 | efried | orly? | |
| 19:15:58 | sean-k-mooney | yep and discribe the virual cpu topoloyg in terms of socket,cores and thread | |
| 19:16:05 | efried | sean-k-mooney: This appears to be talking about just virtual topology, yah? | |
| 19:16:23 | sean-k-mooney | yes unlsee you can use the numa tune elements | |
| 19:18:04 | sean-k-mooney | efried: so ya this all apears to be virtual thought the docs are so hard to navigate one could almost belive ibm did not want you to use those features :P | |
| 19:18:25 | efried | sean-k-mooney: Oh, don't worry, that's not specific to NUMA. | |
| 19:20:36 | sean-k-mooney | haha well i partcalarly like how the numa stuff is not linked too for the "manage processor and memory" section https://www.ibm.com/support/knowledgecenter/en/SSZJY4_3.1.0/liabp/liabpmanageprocessors.htm | |
| 19:22:20 | efried | sean-k-mooney: I think I can now officially say you've spent more time on the IBM "knowledge" center in the past twenty minutes than I have in the past five years. | |
| 19:23:59 | sean-k-mooney | haha well from what i can tell if you used the libvirt kvm driver an pointed it at a power-kvm host i think it would "just work" with all of the nuam and epa stuff we do for normal kvm | |
| 19:24:24 | efried | sean-k-mooney: I have found out that there is some kind of notification the OS can get when affinities change. | |
| 19:24:49 | sean-k-mooney | proably some ahci interupt or something | |
| 19:25:29 | sean-k-mooney | i know there are intrupts that can be sent for memory and cpu hotplug which argubly could change numa topology | |
| 19:26:31 | sean-k-mooney | so it not out of the question that linux or widnows could handel a dynmaic topology if you were to notify it. not sure i would like a numa node to go away but adding one might be ok | |
| 19:27:14 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP libvirt: Add workaround to stop use of qemu-img by the RT https://review.openstack.org/589567 | |
| 19:27:36 | efried | sean-k-mooney: I think it's more likely to be like, "You thought procs 0 and 2 were next to each other, and 1 and 3 were next to each other. Well, now it's 0-1 and 2-3. Deal." | |
| 19:28:23 | sean-k-mooney | haha ya maybe. if you ever test it let me know | |
| 19:28:36 | efried | sean-k-mooney: I feel confident in saying that will never happen. | |
| 19:28:58 | efried | ...for some value of "never". | |
| 19:29:11 | sean-k-mooney | efried: you obvioulsy are not spending enough time talking to telcos | |
| 19:30:11 | efried | If it's anything like talking to the AT&T help desk, I assure you I don't want to spend any more. | |
| 19:46:49 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Make ResourceTracker.stats node-specific https://review.openstack.org/588037 | |
| 19:49:03 | mriedem | was there a zuul reset or something such that we need to recheck things? | |
| 19:51:06 | dansmith | 513 backlog, | |
| 19:51:10 | dansmith | so I imagine it's catching up | |
| 19:51:35 | dansmith | backlog has been up and down all day so I figured something was going on | |
| 19:53:31 | sean-k-mooney | nothing on the openstack-infra twitter for the last 4 days. then normally post somthing if they need to do a reset | |
| 19:54:27 | sean-k-mooney | well it auto posts when then set a status message on the infra channel but in either case the gate is proably just busy/backed up | |
| 19:56:20 | mriedem | ok | |
| 20:00:14 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/ocata: Make ResourceTracker.stats node-specific https://review.openstack.org/588077 | |
| 20:20:58 | openstackgerrit | Matt Riedemann proposed openstack/nova master: doc: mark the max microversion for rocky https://review.openstack.org/589598 | |
| 20:31:30 | maciejjozefczyk | mriedem: efried please check my response: https://review.openstack.org/#/c/520024 | |
| 20:32:11 | efried | Thanks maciejjozefczyk | |
| 20:32:47 | maciejjozefczyk | efried: u2 for pinging me, I forgot about this topic | |
| 20:37:34 | mriedem | melwitt: comments in the reno prelude https://review.openstack.org/#/c/589303/ | |
| 20:37:46 | melwitt | ack | |
| 20:43:17 | mriedem | jroll: you might want to check the rebalance question in here https://review.openstack.org/#/c/520024/ | |
| 20:49:53 | jroll | "want" is a weird word to use :P | |
| 20:49:59 | jroll | looking | |
| 20:53:03 | mriedem | i'm just asking for a witness that i asked for an ironic person to look before we break the rebalance stuff | |
| 20:53:08 | jroll | heh | |
| 20:54:04 | jroll | so I'm thinking it's fine to remove that, but... I don't know everything that _update does and I'm not sure things like _update_usages_from_instances() and such will do the right thing, if we haven't updated which compute hosts we moved active ironic instances to | |
| 20:55:07 | jroll | I feel like I'm nervous to remove it | |
| 20:56:40 | sean-k-mooney | mriedem: is there an ironic job we can trigger maybe via the experimental pipelien to test that patch | |
| 20:57:32 | jroll | sean-k-mooney: there isn't one that actively kills compute hosts and such to trigger rebalances | |
| 20:58:40 | sean-k-mooney | jroll: it looks like the only ironic job that did run on this skipp 22 out of the 23 test ... http://logs.openstack.org/24/520024/9/check/ironic-tempest-dsvm-ipa-wholedisk-bios-agent_ipmitool-tinyipa/1fe9001/testr_results.html.gz | |
| 21:07:34 | jroll | sean-k-mooney: yes, we run a very small set of tests on nova changes because our tests take a long time | |
| 21:08:01 | jroll | sean-k-mooney: that doesn't change the fact that there aren't any full-stack tests that exercise this code | |
| 21:08:37 | jroll | because it's really hard to do - would involve multiple nova-compute instances and orchestrating bringing them up and down and such | |
| 21:08:58 | efried | just mock everything | |
| 21:09:59 | jroll | anyway, I'm gonna post my comments on the change and then not computer anymore today. see y'all later | |
| 21:11:57 | sean-k-mooney | jroll: im not saying there would be test for this just taking note of the fact that the ironic testing is much lighter then i would have expected | |
| 21:12:44 | sean-k-mooney | jroll: that is not a slight againt ironic i just would have expect more tests even if they only ran in the gate pipeline and not check | |
| 21:12:50 | jroll | sean-k-mooney: figure out how to get nested virt in the gate and we can run them fast enough to test more :) | |
| 21:13:31 | jroll | that one instance boot takes like 20 minutes or something iirc | |
| 21:14:00 | sean-k-mooney | whell i honestly dont know why we dont use nested virt in the gate other then the fact that rackspace provides xen based instance and most of our test would like kvm | |
| 21:14:13 | jroll | there's a long history of why that I don't have time to get into | |
| 21:14:46 | sean-k-mooney | jroll: i have been trying to get nested virt in the gate since 2013 im well awaree of the history but no worries | |
| 21:18:22 | melwitt | only sometimes? | |
| 21:18:25 | sean-k-mooney | i just spent 6 hours debuging network connectivit issue with ovs-dpdk because it has buggy numa detection logic | |
| 21:19:30 | sean-k-mooney | melwitt: normally it works fine but when it doesnt its a pain in the ass to figure out why | |
| 21:31:10 | mriedem | dansmith: do you have any idea why we have a Service.availability_zone field on the versioned object? | |
| 21:31:31 | mriedem | i see that we try to lazy-load it for notifications | |
| 21:31:32 | mriedem | 2018-08-07 17:24:53,166 DEBUG [nova.notifications.objects.base] Defaulting the value of the field 'availability_zone' to None in ServiceStatusPayload due to 'Object action obj_load_attr failed because: attribute availability_zone not lazy-loadable' | |
| 21:31:32 | mriedem | 2018-08-07 17:24:53,165 DEBUG [nova.objects.service] Lazy-loading 'availability_zone' on Service id 2 | |
| 21:34:13 | mriedem | i guess for https://github.com/openstack/nova/commit/a2c6838ff5ff095940a76ebd4d578e24575c30d8#diff-b9be5fa188b7efd457da79e9c543344bR110 | |
| 21:39:28 | sean-k-mooney | jaypipes: are you around? | |
| 21:43:49 | sean-k-mooney | i just realised that we missed | |
| 21:45:06 | sean-k-mooney | * i just realised that https://review.openstack.org/#/c/587378/3/vif_plug_ovs/ovs.py misses passing the ovsdb_connection on on of the vhost-user code paths. should i just submit a patch for the missing fucntion or revert and submit an updated versions | |
| 21:46:14 | sean-k-mooney | im thinking just add a patch on top the get the missing function call but just said i would ask | |
| 22:01:30 | sean-k-mooney | mriedem: by the way im assuming you did not have time to test livemigrating between different neutron backends as part of the multiple port binding blueprint? | |
| 22:02:02 | mriedem | sean-k-mooney: mlavalle did it between ovs and linuxbridge using neutron directly, not via nova | |
| 22:02:11 | mriedem | i don't have a mixed vif type env setup no | |
| 22:02:26 | sean-k-mooney | mriedem: now that i have figured out why ovs-dpdk was not working from me ill try an test it out later this week. ill also try it via linux bridge if i get a chance | |
| 22:02:38 | sean-k-mooney | cool no worries | |
| 22:04:00 | sean-k-mooney | i normaly have ovs, ovs-dpdk and linux bride deployed concurrnetly or at least i did before i move. ill test the matirx of all 3 setups and let you know how it goes | |
| 22:30:40 | mriedem | Kevin_Zheng: comments all over https://review.openstack.org/#/q/topic:bug/1781880+(status:open+OR+status:merged) so it should be clear to update now | |
| 22:30:48 | mriedem | and abandon the functional test since it will never work | |
| 22:39:00 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Update really old comments about vmware hosts managing multiple nodes https://review.openstack.org/589666 | |
| 22:47:44 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add microversion info in the os-server-groups API samples https://review.openstack.org/589006 | |
| 22:50:34 | melwitt | mriedem: I'm not sure what to make of this comment after trying to make the changes to make things fail if a cell raises an exception https://review.openstack.org/#/c/540258/10/nova/scheduler/utils.py@843 | |
| 22:51:16 | melwitt | it's not clear to me if a cell raising an exception will be part of our "down cell" handling down the line or if we should expect to potentially fail a server create with a 500 if a cell raises an exception when we query it | |
| 22:51:55 | melwitt | pre-down-cell handling | |
| 22:53:11 | melwitt | initially I was thinking yes, it makes sense to fail the create if a cell raises an exception, but that would propagate up to the user as a 500 because there's something wrong with a cell, and then I became unsure if that falls under the "down cell" stuff or if it's just something we should do now | |
| 22:58:09 | efried | mriedem: I assume that +1 is so we wait until stein to land it | |