Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-01-09
11:47:01 ygk_12345 i cant do a virsh list command
11:47:14 nehaalhat gibi: working on this feature: https://review.openstack.org/#/c/609960/3/specs/stein/approved/placement-req-filter-forbidden-aggregates.rst
11:47:20 sean-k-mooney ygk_12345: that usually means the libvirtd daemon has exited
11:47:33 ygk_12345 sean-k-mooney: what does that mean ?
11:47:56 sean-k-mooney the libvirtd service has crashed/stopped on the compute node
11:48:12 jangutter ygk_12345 what distro and which version are you running?
11:48:33 ygk_12345 jangutter: Ubuntu 16.04.3 LTS \n \l
11:48:40 sean-k-mooney ygk_12345: try systemctl status libvirtd
11:49:03 ygk_12345 sean-k-mooney: I see this
11:49:05 ygk_12345 sean-k-mooney: 2019-01-09 11:48:44.365+0000: 16119: error : virPidFileAcquirePath:422 : Failed to acquire pid file '/var/run/libvirtd.pid': Resource temporarily unavailable
11:49:19 ygk_12345 sean-k-mooney: i did libvirtd -l
11:49:41 ygk_12345 sean-k-mooney: Failed to open file '/proc/28288/stat': No such file or directory
11:49:41 sean-k-mooney ygk_12345: that will try to start a new libvirtd
11:49:50 sean-k-mooney that is not the same thing as runing status
11:50:08 ygk_12345 sean-k-mooney: the libvirtd is running finr though
11:50:16 ygk_12345 sean-k-mooney: failed to connect to monitor socket: No such process
11:50:36 sean-k-mooney you said you cant run virsh command so its obviouly in an error state
11:50:41 nehaalhat gibi: and this spec is dependent on https://review.openstack.org/#/c/603352/4/specs/stein/approved/negative-aggregate-membership.rst
11:50:52 ygk_12345 sean-k-mooney: virsh list command is hanging
11:50:56 ygk_12345 sean-k-mooney: no output
11:51:04 gibi nehaalhat: OK. Do you need to mock something from placement because the placement change that supports your feature is not merged yet?
11:51:11 sean-k-mooney yes because the libvirt daemon is not responding to requests
11:51:17 nehaalhat gibi: yes
11:51:31 f10d4 hi
11:51:46 ygk_12345 sean-k-mooney: now after sometime the vrish command is working but the flaoting IP of the instances is not pingable
11:52:15 ygk_12345 sean-k-mooney: it seems tot fix itself after sometime
11:52:22 ygk_12345 sean-k-mooney: bit strange
11:52:23 sean-k-mooney virsh is just a client that publishes request to the libvirt daemon to execute commands via dbus
11:52:24 jangutter ygk_12345: Ubuntu 16.04 doesn't always use systemd: you might have to look at libvirt's logs directly. Check your system logs to see if everything's healthy or something isn't restarting all the time.
11:52:33 gibi nehaalhat: if you have both the placement change an the nova change locally then you have to instruct the nova test to use your local placement change for the functional test
11:53:06 sean-k-mooney jangutter: well the process could still be running but not respoinding on the libvirt contol socket also
11:53:18 gibi nehaalhat: when you will upload that to gerrit you can use a Depends-On tag in the commit message of the nova change that point to the placement change and the CI will act accordinlgy
11:54:06 ygk_12345 jangutter: i see some unable to connect to hyoervisor meaages from the nova in the syslog
11:54:08 sean-k-mooney ygk_12345: your floating ip issue coudl be several things the first thing i would check is if the vm actully has a privte ip and can ping its router
11:54:32 sean-k-mooney if that works then you shoudl check the fip namespace and see if the dnat rules for the vm are present
11:54:45 nehaalhat gibi: placement change yet to implemenetd by community members: https://review.openstack.org/#/c/603352/4/specs/stein/approved/negative-aggregate-membership.rst
11:54:50 jangutter sean-k-mooney: yeah, it could be many fun things. Hopefully there's logs.
11:55:02 ygk_12345 sean-k-mooney: jangutter usually rebooting the compute node fixes this issue. but that is not a viable solution always
11:55:25 jangutter ygk_12345: the nova message is not the root cause, it's a consequence of something else bad happening.
11:55:32 ygk_12345 jangutter: yes true
11:55:38 gibi nehaalhat: I'm affraid that without the placement feature exists you cannot make the final functional test in nova
11:55:48 ygk_12345 jangutter: unable to pinpoint it in the libvirtd
11:57:02 nehaalhat gibi: ya that will not be final FT but for my patch I want to add FT and for that need to mock placement method
11:57:05 sean-k-mooney ygk_12345: a less invasive way to correct it when it is broken is to restart libvirtd service and then restart the nova-compute agent
11:57:09 gibi nehaalhat: to run the nova functional test with local placement version, you can change tox.ini here https://github.com/openstack/nova/blob/master/tox.ini#L87
11:57:22 gibi nehaalhat: temporarly you can try to mock sure
11:57:37 sean-k-mooney still if you can find the actully issue with libvirtd that is obviously better
11:57:42 ygk_12345 sean-k-mooney: but even the restart of libvirts is not fixing the problem
11:57:42 gibi nehaalhat: I think the normal mock.patch() call should work in the functional tests
11:57:57 jangutter ygk_12345: you also should restart nova-compute
11:58:14 jangutter ygk_12345: after libvirtd is restarted
11:58:50 nehaalhat gibi: yes I used normal mock.patch() but still the call goes to method, like: @mock.patch('placement.util.normalize_member_of_qs_param')
11:58:59 sean-k-mooney ygk_12345: nova-compute could still have open file handels to the old libvirt socket so restarting nova-compute forces it to reopen them
11:59:26 ygk_12345 sean-k-mooney: ok
11:59:33 gibi nehaalhat: give me a couple of minutes, I will try to mock it mysef.
11:59:47 nehaalhat gibi: ok thank you
12:02:56 jangutter Weird question: I submitted a review, and openstack-infra updated the link to the review with a broken link on a blueprint. Did I do something stupid?
12:03:34 sean-k-mooney which revierw
12:04:56 jangutter It added: Gerrit topic: https://review.openstack.org/#q,topic:bp/vrouter-os-vif-conversion,n,z where it should have added https://review.openstack.org/#/q/topic:bp/vrouter-hw-offloads,n,z
12:05:34 jangutter triggered by: https://review.openstack.org/571325
12:06:51 sean-k-mooney i assume you used git review to submit it
12:06:57 sean-k-mooney what is your local branch name
12:07:11 sean-k-mooney that is where it pulls the topic from if you dont pass -t
12:07:35 alex_xu sean-k-mooney: jaypipes, yonglihe why cpu_thread_policy' is libvirt specific? re:https://review.openstack.org/#/c/612256/20/specs/stein/approved/show-server-numa-topology.rst@104
12:07:56 sean-k-mooney alex_xu: no other virt dirive supports it
12:07:58 jaypipes alex_xu: it isn't.
12:08:14 jaypipes alex_xu: the one I mentioned was emulator_threads_policy.
12:08:20 alex_xu no other virt driver support it, it doesn't mean libvirt specific, right?
12:08:33 sean-k-mooney jaypipes: cpu_thread_policy is only supported by libvirt also
12:08:34 alex_xu sean-k-mooney: not all the virt driver implement numa affinity also
12:08:47 jangutter sean-k-mooney: local branch name is: review/jan_gutter/bp/vrouter-os-vif-conversion
12:08:47 jaypipes alex_xu: "emulator" specifically refers to a libvirt/QEMU implementation detail :)
12:08:47 sean-k-mooney alex_xu: yes but at least 2 implement it
12:08:53 sean-k-mooney hyperv and libvirt
12:08:57 alex_xu jangutter: yea, I agree with you that
12:09:26 alex_xu jangutter: sorry, ping the wrong name...
12:09:30 sean-k-mooney jangutter: yep that is why it create that topic upstream :)
12:09:41 alex_xu hah
12:10:03 jangutter sean-k-mooney: aaaah.... sorry, I pasted the wrong link in.... let me repaste the correct link...
12:10:31 jangutter It added: Gerrit topic: https://review.openstack.org/#q,topic:bp/vrouter-os-vif-conversion,n,z where it should have added https://review.openstack.org/#/q/topic:bp/vrouter-os-vif-conversion,n,z
12:11:21 sean-k-mooney jangutter: hum weird
12:11:50 jangutter sean-k-mooney: it picked the correct topic, but it generated a URL where "#q," should be "#/q/"
12:12:14 jangutter sean-k-mooney: maybe the link was valid on an older version of gerrit?
12:12:34 alex_xu jaypipes: we also should put it into the same level of 'nodes' field, right? sean-k-mooney mention that also
12:12:43 alex_xu we can't set cpu thread policy for each node
12:13:09 jangutter sean-k-mooney: sorry, just realised, I can probably ask this on #openstack-infra (facepalm).... it's been a looooong Monday.
12:13:46 sean-k-mooney jangutter: hehe dont worry about it but they will likely have a better answer
12:13:49 jangutter alex_xu: heh, no worries, I was confused for a bit, then I realised something autocompleted.
12:14:00 ygk_12345 sean-k-mooney: how to enable logging in libvirtd ?
12:14:11 jaypipes alex_xu: is that a fundamental problem with the underlying modeling? in other words, is this: https://github.com/openstack/nova/blob/master/nova/objects/instance_numa_topology.py#L56 incorrectly placed on the NUMA cell instead of the InstanceNUMATopology object? :)
12:14:19 alex_xu jangutter: yea, I type tab too early :)
12:15:14 alex_xu jaypipes: interesting...we put that attribute into cell obj
12:15:31 alex_xu jaypipes: but we only have one extra spec to set cpu policy for the whole guest
12:15:47 sean-k-mooney jaypipes well there is no fundemental resone we could not do this per cell but we only support setting for the entire instnace
12:16:29 jaypipes sean-k-mooney: overkill? :)
12:16:36 sean-k-mooney totally
12:17:01 sean-k-mooney i mean at that point we might as well let people chose the cpus
12:17:34 sean-k-mooney i mean that is what they really wanted all along...
12:18:49 sean-k-mooney jaypipes: my guess would be we store it in the instance cell to not need to pass it as a paramater to fuctions later when calulating the pinning or some other reason liek that
12:18:56 alex_xu actually those policy can easy to get from the flavor extra spec

Earlier   Later