Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-08-07
18:59:25 sean-k-mooney that is effectivly what hw:numa_nodes=2 means
18:59:26 efried sean-k-mooney: Right, okay, so I'm saying you definitely can't do that in PowerVM. I'm not sure whether you can do it in PowerKVM.
18:59:42 sean-k-mooney ok cool
19:00:25 sean-k-mooney you can do that with hyperv i think but i dont think i guarntees they are mapped to 2 phyical host numa nodes
19:01:14 sean-k-mooney or rather if they are mapped/affinitised to host numa nodes is governed by a hypervior config option as far as i know
19:14:52 sean-k-mooney efried: its burried in the docs but ya it looks like you can do numa affinity with powerkvm https://www.ibm.com/support/knowledgecenter/SSZJY4_3.1.0/liabp/liabphotplugcpunuma.htm
19:15:11 sean-k-mooney you can also do cpu pinning
19:15:17 efried orly?
19:15:58 sean-k-mooney yep and discribe the virual cpu topoloyg in terms of socket,cores and thread
19:16:05 efried sean-k-mooney: This appears to be talking about just virtual topology, yah?
19:16:23 sean-k-mooney yes unlsee you can use the numa tune elements
19:18:04 sean-k-mooney efried: so ya this all apears to be virtual thought the docs are so hard to navigate one could almost belive ibm did not want you to use those features :P
19:18:25 efried sean-k-mooney: Oh, don't worry, that's not specific to NUMA.
19:20:36 sean-k-mooney haha well i partcalarly like how the numa stuff is not linked too for the "manage processor and memory" section https://www.ibm.com/support/knowledgecenter/en/SSZJY4_3.1.0/liabp/liabpmanageprocessors.htm
19:22:20 efried sean-k-mooney: I think I can now officially say you've spent more time on the IBM "knowledge" center in the past twenty minutes than I have in the past five years.
19:23:59 sean-k-mooney haha well from what i can tell if you used the libvirt kvm driver an pointed it at a power-kvm host i think it would "just work" with all of the nuam and epa stuff we do for normal kvm
19:24:24 efried sean-k-mooney: I have found out that there is some kind of notification the OS can get when affinities change.
19:24:49 sean-k-mooney proably some ahci interupt or something
19:25:29 sean-k-mooney i know there are intrupts that can be sent for memory and cpu hotplug which argubly could change numa topology
19:26:31 sean-k-mooney so it not out of the question that linux or widnows could handel a dynmaic topology if you were to notify it. not sure i would like a numa node to go away but adding one might be ok
19:27:14 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP libvirt: Add workaround to stop use of qemu-img by the RT https://review.openstack.org/589567
19:27:36 efried sean-k-mooney: I think it's more likely to be like, "You thought procs 0 and 2 were next to each other, and 1 and 3 were next to each other. Well, now it's 0-1 and 2-3. Deal."
19:28:23 sean-k-mooney haha ya maybe. if you ever test it let me know
19:28:36 efried sean-k-mooney: I feel confident in saying that will never happen.
19:28:58 efried ...for some value of "never".
19:29:11 sean-k-mooney efried: you obvioulsy are not spending enough time talking to telcos
19:30:11 efried If it's anything like talking to the AT&T help desk, I assure you I don't want to spend any more.
19:46:49 openstackgerrit Matt Riedemann proposed openstack/nova stable/pike: Make ResourceTracker.stats node-specific https://review.openstack.org/588037
19:49:03 mriedem was there a zuul reset or something such that we need to recheck things?
19:51:06 dansmith 513 backlog,
19:51:10 dansmith so I imagine it's catching up
19:51:35 dansmith backlog has been up and down all day so I figured something was going on
19:53:31 sean-k-mooney nothing on the openstack-infra twitter for the last 4 days. then normally post somthing if they need to do a reset
19:54:27 sean-k-mooney well it auto posts when then set a status message on the infra channel but in either case the gate is proably just busy/backed up
19:56:20 mriedem ok
20:00:14 openstackgerrit Matt Riedemann proposed openstack/nova stable/ocata: Make ResourceTracker.stats node-specific https://review.openstack.org/588077
20:20:58 openstackgerrit Matt Riedemann proposed openstack/nova master: doc: mark the max microversion for rocky https://review.openstack.org/589598
20:31:30 maciejjozefczyk mriedem: efried please check my response: https://review.openstack.org/#/c/520024
20:32:11 efried Thanks maciejjozefczyk
20:32:47 maciejjozefczyk efried: u2 for pinging me, I forgot about this topic
20:37:34 mriedem melwitt: comments in the reno prelude https://review.openstack.org/#/c/589303/
20:37:46 melwitt ack
20:43:17 mriedem jroll: you might want to check the rebalance question in here https://review.openstack.org/#/c/520024/
20:49:53 jroll "want" is a weird word to use :P
20:49:59 jroll looking
20:53:03 mriedem i'm just asking for a witness that i asked for an ironic person to look before we break the rebalance stuff
20:53:08 jroll heh
20:54:04 jroll so I'm thinking it's fine to remove that, but... I don't know everything that _update does and I'm not sure things like _update_usages_from_instances() and such will do the right thing, if we haven't updated which compute hosts we moved active ironic instances to
20:55:07 jroll I feel like I'm nervous to remove it
20:56:40 sean-k-mooney mriedem: is there an ironic job we can trigger maybe via the experimental pipelien to test that patch
20:57:32 jroll sean-k-mooney: there isn't one that actively kills compute hosts and such to trigger rebalances
20:58:40 sean-k-mooney jroll: it looks like the only ironic job that did run on this skipp 22 out of the 23 test ... http://logs.openstack.org/24/520024/9/check/ironic-tempest-dsvm-ipa-wholedisk-bios-agent_ipmitool-tinyipa/1fe9001/testr_results.html.gz
21:07:34 jroll sean-k-mooney: yes, we run a very small set of tests on nova changes because our tests take a long time
21:08:01 jroll sean-k-mooney: that doesn't change the fact that there aren't any full-stack tests that exercise this code
21:08:37 jroll because it's really hard to do - would involve multiple nova-compute instances and orchestrating bringing them up and down and such
21:08:58 efried just mock everything
21:09:59 jroll anyway, I'm gonna post my comments on the change and then not computer anymore today. see y'all later
21:11:57 sean-k-mooney jroll: im not saying there would be test for this just taking note of the fact that the ironic testing is much lighter then i would have expected
21:12:44 sean-k-mooney jroll: that is not a slight againt ironic i just would have expect more tests even if they only ran in the gate pipeline and not check
21:12:50 jroll sean-k-mooney: figure out how to get nested virt in the gate and we can run them fast enough to test more :)
21:13:31 jroll that one instance boot takes like 20 minutes or something iirc
21:14:00 sean-k-mooney whell i honestly dont know why we dont use nested virt in the gate other then the fact that rackspace provides xen based instance and most of our test would like kvm
21:14:13 jroll there's a long history of why that I don't have time to get into
21:14:46 sean-k-mooney jroll: i have been trying to get nested virt in the gate since 2013 im well awaree of the history but no worries
21:18:22 melwitt only sometimes?
21:18:25 sean-k-mooney i just spent 6 hours debuging network connectivit issue with ovs-dpdk because it has buggy numa detection logic
21:19:30 sean-k-mooney melwitt: normally it works fine but when it doesnt its a pain in the ass to figure out why
21:31:10 mriedem dansmith: do you have any idea why we have a Service.availability_zone field on the versioned object?
21:31:31 mriedem i see that we try to lazy-load it for notifications
21:31:32 mriedem 2018-08-07 17:24:53,166 DEBUG [nova.notifications.objects.base] Defaulting the value of the field 'availability_zone' to None in ServiceStatusPayload due to 'Object action obj_load_attr failed because: attribute availability_zone not lazy-loadable'
21:31:32 mriedem 2018-08-07 17:24:53,165 DEBUG [nova.objects.service] Lazy-loading 'availability_zone' on Service id 2
21:34:13 mriedem i guess for https://github.com/openstack/nova/commit/a2c6838ff5ff095940a76ebd4d578e24575c30d8#diff-b9be5fa188b7efd457da79e9c543344bR110
21:39:28 sean-k-mooney jaypipes: are you around?
21:43:49 sean-k-mooney i just realised that we missed
21:45:06 sean-k-mooney * i just realised that https://review.openstack.org/#/c/587378/3/vif_plug_ovs/ovs.py misses passing the ovsdb_connection on on of the vhost-user code paths. should i just submit a patch for the missing fucntion or revert and submit an updated versions
21:46:14 sean-k-mooney im thinking just add a patch on top the get the missing function call but just said i would ask
22:01:30 sean-k-mooney mriedem: by the way im assuming you did not have time to test livemigrating between different neutron backends as part of the multiple port binding blueprint?
22:02:02 mriedem sean-k-mooney: mlavalle did it between ovs and linuxbridge using neutron directly, not via nova
22:02:11 mriedem i don't have a mixed vif type env setup no
22:02:26 sean-k-mooney mriedem: now that i have figured out why ovs-dpdk was not working from me ill try an test it out later this week. ill also try it via linux bridge if i get a chance
22:02:38 sean-k-mooney cool no worries
22:04:00 sean-k-mooney i normaly have ovs, ovs-dpdk and linux bride deployed concurrnetly or at least i did before i move. ill test the matirx of all 3 setups and let you know how it goes
22:30:40 mriedem Kevin_Zheng: comments all over https://review.openstack.org/#/q/topic:bug/1781880+(status:open+OR+status:merged) so it should be clear to update now
22:30:48 mriedem and abandon the functional test since it will never work
22:39:00 openstackgerrit Matt Riedemann proposed openstack/nova master: Update really old comments about vmware hosts managing multiple nodes https://review.openstack.org/589666
22:47:44 openstackgerrit Matt Riedemann proposed openstack/nova master: Add microversion info in the os-server-groups API samples https://review.openstack.org/589006
22:50:34 melwitt mriedem: I'm not sure what to make of this comment after trying to make the changes to make things fail if a cell raises an exception https://review.openstack.org/#/c/540258/10/nova/scheduler/utils.py@843
22:51:16 melwitt it's not clear to me if a cell raising an exception will be part of our "down cell" handling down the line or if we should expect to potentially fail a server create with a 500 if a cell raises an exception when we query it
22:51:55 melwitt pre-down-cell handling
22:53:11 melwitt initially I was thinking yes, it makes sense to fail the create if a cell raises an exception, but that would propagate up to the user as a 500 because there's something wrong with a cell, and then I became unsure if that falls under the "down cell" stuff or if it's just something we should do now
22:58:09 efried mriedem: I assume that +1 is so we wait until stein to land it
22:58:32 efried (the "Update resources once" patch)
22:58:59 mriedem efried: yeah
22:59:33 efried mriedem: I'm going to -2 it just in case, but I'm with ya.
22:59:45 mriedem efried: don't think you need to -2
22:59:50 openstackgerrit Merged openstack/os-traits master: Update reno for stable/rocky https://review.openstack.org/586103
23:00:07 mriedem efried: i -W'ed it
23:00:11 efried mriedem: You mean because no other cores are going to "accidentally" merge it? :)
23:00:19 mriedem melwitt: how does a cell raise an exception?
23:00:26 mriedem if the called function does?

Earlier   Later