Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-01-09
11:52:15 ygk_12345 sean-k-mooney: it seems tot fix itself after sometime
11:52:22 ygk_12345 sean-k-mooney: bit strange
11:52:23 sean-k-mooney virsh is just a client that publishes request to the libvirt daemon to execute commands via dbus
11:52:24 jangutter ygk_12345: Ubuntu 16.04 doesn't always use systemd: you might have to look at libvirt's logs directly. Check your system logs to see if everything's healthy or something isn't restarting all the time.
11:52:33 gibi nehaalhat: if you have both the placement change an the nova change locally then you have to instruct the nova test to use your local placement change for the functional test
11:53:06 sean-k-mooney jangutter: well the process could still be running but not respoinding on the libvirt contol socket also
11:53:18 gibi nehaalhat: when you will upload that to gerrit you can use a Depends-On tag in the commit message of the nova change that point to the placement change and the CI will act accordinlgy
11:54:06 ygk_12345 jangutter: i see some unable to connect to hyoervisor meaages from the nova in the syslog
11:54:08 sean-k-mooney ygk_12345: your floating ip issue coudl be several things the first thing i would check is if the vm actully has a privte ip and can ping its router
11:54:32 sean-k-mooney if that works then you shoudl check the fip namespace and see if the dnat rules for the vm are present
11:54:45 nehaalhat gibi: placement change yet to implemenetd by community members: https://review.openstack.org/#/c/603352/4/specs/stein/approved/negative-aggregate-membership.rst
11:54:50 jangutter sean-k-mooney: yeah, it could be many fun things. Hopefully there's logs.
11:55:02 ygk_12345 sean-k-mooney: jangutter usually rebooting the compute node fixes this issue. but that is not a viable solution always
11:55:25 jangutter ygk_12345: the nova message is not the root cause, it's a consequence of something else bad happening.
11:55:32 ygk_12345 jangutter: yes true
11:55:38 gibi nehaalhat: I'm affraid that without the placement feature exists you cannot make the final functional test in nova
11:55:48 ygk_12345 jangutter: unable to pinpoint it in the libvirtd
11:57:02 nehaalhat gibi: ya that will not be final FT but for my patch I want to add FT and for that need to mock placement method
11:57:05 sean-k-mooney ygk_12345: a less invasive way to correct it when it is broken is to restart libvirtd service and then restart the nova-compute agent
11:57:09 gibi nehaalhat: to run the nova functional test with local placement version, you can change tox.ini here https://github.com/openstack/nova/blob/master/tox.ini#L87
11:57:22 gibi nehaalhat: temporarly you can try to mock sure
11:57:37 sean-k-mooney still if you can find the actully issue with libvirtd that is obviously better
11:57:42 ygk_12345 sean-k-mooney: but even the restart of libvirts is not fixing the problem
11:57:42 gibi nehaalhat: I think the normal mock.patch() call should work in the functional tests
11:57:57 jangutter ygk_12345: you also should restart nova-compute
11:58:14 jangutter ygk_12345: after libvirtd is restarted
11:58:50 nehaalhat gibi: yes I used normal mock.patch() but still the call goes to method, like: @mock.patch('placement.util.normalize_member_of_qs_param')
11:58:59 sean-k-mooney ygk_12345: nova-compute could still have open file handels to the old libvirt socket so restarting nova-compute forces it to reopen them
11:59:26 ygk_12345 sean-k-mooney: ok
11:59:33 gibi nehaalhat: give me a couple of minutes, I will try to mock it mysef.
11:59:47 nehaalhat gibi: ok thank you
12:02:56 jangutter Weird question: I submitted a review, and openstack-infra updated the link to the review with a broken link on a blueprint. Did I do something stupid?
12:03:34 sean-k-mooney which revierw
12:04:56 jangutter It added: Gerrit topic: https://review.openstack.org/#q,topic:bp/vrouter-os-vif-conversion,n,z where it should have added https://review.openstack.org/#/q/topic:bp/vrouter-hw-offloads,n,z
12:05:34 jangutter triggered by: https://review.openstack.org/571325
12:06:51 sean-k-mooney i assume you used git review to submit it
12:06:57 sean-k-mooney what is your local branch name
12:07:11 sean-k-mooney that is where it pulls the topic from if you dont pass -t
12:07:35 alex_xu sean-k-mooney: jaypipes, yonglihe why cpu_thread_policy' is libvirt specific? re:https://review.openstack.org/#/c/612256/20/specs/stein/approved/show-server-numa-topology.rst@104
12:07:56 sean-k-mooney alex_xu: no other virt dirive supports it
12:07:58 jaypipes alex_xu: it isn't.
12:08:14 jaypipes alex_xu: the one I mentioned was emulator_threads_policy.
12:08:20 alex_xu no other virt driver support it, it doesn't mean libvirt specific, right?
12:08:33 sean-k-mooney jaypipes: cpu_thread_policy is only supported by libvirt also
12:08:34 alex_xu sean-k-mooney: not all the virt driver implement numa affinity also
12:08:47 jangutter sean-k-mooney: local branch name is: review/jan_gutter/bp/vrouter-os-vif-conversion
12:08:47 jaypipes alex_xu: "emulator" specifically refers to a libvirt/QEMU implementation detail :)
12:08:47 sean-k-mooney alex_xu: yes but at least 2 implement it
12:08:53 sean-k-mooney hyperv and libvirt
12:08:57 alex_xu jangutter: yea, I agree with you that
12:09:26 alex_xu jangutter: sorry, ping the wrong name...
12:09:30 sean-k-mooney jangutter: yep that is why it create that topic upstream :)
12:09:41 alex_xu hah
12:10:03 jangutter sean-k-mooney: aaaah.... sorry, I pasted the wrong link in.... let me repaste the correct link...
12:10:31 jangutter It added: Gerrit topic: https://review.openstack.org/#q,topic:bp/vrouter-os-vif-conversion,n,z where it should have added https://review.openstack.org/#/q/topic:bp/vrouter-os-vif-conversion,n,z
12:11:21 sean-k-mooney jangutter: hum weird
12:11:50 jangutter sean-k-mooney: it picked the correct topic, but it generated a URL where "#q," should be "#/q/"
12:12:14 jangutter sean-k-mooney: maybe the link was valid on an older version of gerrit?
12:12:34 alex_xu jaypipes: we also should put it into the same level of 'nodes' field, right? sean-k-mooney mention that also
12:12:43 alex_xu we can't set cpu thread policy for each node
12:13:09 jangutter sean-k-mooney: sorry, just realised, I can probably ask this on #openstack-infra (facepalm).... it's been a looooong Monday.
12:13:46 sean-k-mooney jangutter: hehe dont worry about it but they will likely have a better answer
12:13:49 jangutter alex_xu: heh, no worries, I was confused for a bit, then I realised something autocompleted.
12:14:00 ygk_12345 sean-k-mooney: how to enable logging in libvirtd ?
12:14:11 jaypipes alex_xu: is that a fundamental problem with the underlying modeling? in other words, is this: https://github.com/openstack/nova/blob/master/nova/objects/instance_numa_topology.py#L56 incorrectly placed on the NUMA cell instead of the InstanceNUMATopology object? :)
12:14:19 alex_xu jangutter: yea, I type tab too early :)
12:15:14 alex_xu jaypipes: interesting...we put that attribute into cell obj
12:15:31 alex_xu jaypipes: but we only have one extra spec to set cpu policy for the whole guest
12:15:47 sean-k-mooney jaypipes well there is no fundemental resone we could not do this per cell but we only support setting for the entire instnace
12:16:29 jaypipes sean-k-mooney: overkill? :)
12:16:36 sean-k-mooney totally
12:17:01 sean-k-mooney i mean at that point we might as well let people chose the cpus
12:17:34 sean-k-mooney i mean that is what they really wanted all along...
12:18:49 sean-k-mooney jaypipes: my guess would be we store it in the instance cell to not need to pass it as a paramater to fuctions later when calulating the pinning or some other reason liek that
12:18:56 alex_xu actually those policy can easy to get from the flavor extra spec
12:19:05 sean-k-mooney alex_xu: yes
12:19:55 sean-k-mooney alex_xu: wasing the main point of this spec that it would be reading the numa/cpu related extra specs and image metadata and presenting the compined view to the admin
12:20:08 sean-k-mooney *wasn't
12:20:41 alex_xu at least I don't want to the api user to parse the extra spec, and copy the parse rule which inside nova
12:20:57 alex_xu and yes, we have image meta for those policy also
12:21:00 jaypipes alex_xu, sean-k-mooney, yonglihe: easy to get sidetracked here. my bottom line recommendation: a) don't put emulator_threads_policy in the HTTP response AND b) put the cpu_threads_policy as an attribute at the same level in the HTTP response as "nodes" (like sean-k-mooney and alex_xu suggest)
12:21:31 sean-k-mooney jaypipes: sure that sounds resonable
12:22:38 alex_xu jaypipes: cool
12:27:16 sean-k-mooney jaypipes: alex_xu what is the process for specless blueprint by the way
12:27:23 sean-k-mooney jaypipes: alex_xu specifcally for https://blueprints.launchpad.net/nova/+spec/libvirt-disable-pmu-for-realtime-guests
12:28:07 alex_xu sean-k-mooney: get an approve in nova meeting? I guess
12:28:15 jangutter sean-k-mooney: Last time I proposed one, it went up as an agenda item on the Nova meeting.
12:28:38 sean-k-mooney ya i was going to add it for this weeks
12:29:29 sean-k-mooney if it gets punted to next cycle it wont kill me but it was a simple addtion i was going to work on after m2 as a 1 patch feature
12:41:31 openstackgerrit Merged openstack/nova-specs master: Fix format flavor-extra-spec-image-property-validation https://review.openstack.org/629311
12:41:34 gibi nehaalhat: I was able to mock the placement function you need http://paste.openstack.org/show/740847/
12:44:22 nehaalhat gibi: ok
12:46:12 kashyap gibi: Hi there; when you get a moment: I've answered your question on the review. What Martin posted on the review is the successful test evidence.
12:47:07 openstackgerrit Chris Dent proposed openstack/nova master: Remove link to placement configuration from nova config docs https://review.openstack.org/628185
12:47:07 openstackgerrit Chris Dent proposed openstack/nova master: Remove placement from nova install docs https://review.openstack.org/628184
12:47:08 openstackgerrit Chris Dent proposed openstack/nova master: Remove placement from contributor doc https://review.openstack.org/628186
12:47:56 kashyap gibi: The test results posted by Martin's matches my own (which I also posted in the review).
12:48:27 kashyap Thanks for reviewing!
12:48:42 gibi kashyap: ohh, then I have to read a bit more carfeully :)
12:49:23 kashyap Heh, no problem at all. It is low-level QMP log; so not everyone can make sense of it. I just added context as to what it means in the bug comment and the review

Earlier   Later