Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-01-09
09:59:54 sean-k-mooney maciejjozefczyk: was it related to the socket timeout or unrelated.
10:00:03 kashyap As I noted, my bar for error messages is: "give me something actionable"
10:00:19 kashyap It is really frustrating to just see "error: please figure it out"
10:00:42 sean-k-mooney you mean like "no valid host"
10:00:46 openstackgerrit Kashyap Chamarthy proposed openstack/nova master: libvirt: Support native TLS for migration and disks over NBD https://review.openstack.org/625216
10:00:47 kashyap sean-k-mooney: Heh
10:01:06 sean-k-mooney its my favorite nova error message
10:01:17 kashyap Yeah, it's the most awful message.
10:01:25 kashyap One day, if I feel angry enough, I might clean them all up
10:03:00 sean-k-mooney well its availd error message but the issue is 90% of the time it because you hit the retry filter limit and you hit the retry filter because or a different error which was lost in the logs
10:03:22 kashyap Yeah, I recall that it's "best we could do"
10:35:43 maciejjozefczyk sean-k-mooney: nope, different topic
10:35:48 maciejjozefczyk already fixed in new patch
10:36:22 sean-k-mooney cool i had only tried the followon patch as i was testing on master so did not need the db change
10:38:50 maciejjozefczyk for me I needed to update about 7k vifs only for one db
11:08:54 nehaalhat gibi: Hi, in case of functional test cases, how nova sync with placement service? Do you any idea about this?
11:11:32 gibi nehaalhat: in functional test nova imports placement code via https://github.com/openstack/nova/blob/e3eb5f916580a9bab8f67b0fd685c6b3b23a97b7/nova/tests/functional/fixtures.py#L41
11:12:42 nehaalhat like aggregates created in nova, how they are synced in placement service?
11:22:28 nehaalhat gibi: and do you have idea how to mock a method from placement in nova functional test cases where placementfixture is used like: https://github.com/openstack/nova/blob/e3eb5f916580a9bab8f67b0fd685c6b3b23a97b7/nova/tests/functional/test_aggregates.py#L385
11:31:47 gibi nehaalhat: do you have a patch with a functional test which doesn't work without mocking something in placement?
11:32:08 gibi nehaalhat: in general mocking in functional test is discuraged
11:32:26 gibi nehaalhat: but I can look at your patch to understand more
11:34:21 nehaalhat gibi: No, i dont have a patch on community gerrit right now
11:35:02 gibi nehaalhat: let me try to dig out where nova sync the aggregates to placement...
11:35:18 nehaalhat ok
11:36:01 gibi nehaalhat: there is a nova-manage CLI command for aggregates sync https://github.com/openstack/nova/blob/da17e0ed8a9b9ddc79b6d2004e6fb941c6163fbf/nova/cmd/manage.py#L2158
11:37:02 gibi nehaalhat: and this is the patch that introduced the automatic syncing https://review.openstack.org/#/c/553597/
11:37:11 nehaalhat gibi: actually I want to add a FT under: https://github.com/openstack/nova/blob/e3eb5f916580a9bab8f67b0fd685c6b3b23a97b7/nova/tests/functional/test_aggregates.py#L385 where placement fixture is used, and I want to mock this method in my FT
11:38:01 gibi nehaalhat: and this is the point where nova adds a host to a placement aggregate https://review.openstack.org/#/c/553597/15/nova/compute/api.py@5147
11:38:54 gibi nehaalhat: so I think if your functional test adds a host to an aggregate then nova will automatically add that host to the placement aggregate as weel
11:38:57 gibi well
11:39:30 nehaalhat gibi: ok thanks
11:40:18 gibi nehaalhat: but if you push a wip patch to gerrit then I can help making it work
11:41:11 nehaalhat gibi: I want to add a FT under: https://github.com/openstack/nova/blob/e3eb5f916580a9bab8f67b0fd685c6b3b23a97b7/nova/tests/functional/test_aggregates.py#L385 where placement fixture is used
11:41:25 nehaalhat and I want to mock this method in my FT : https://github.com/openstack/placement/blob/master/placement/util.py#L236
11:42:05 nehaalhat actually I cant push on community gerrit without internal approval
11:42:07 gibi nehaalhat: why do you need to mock that?
11:42:12 openstackgerrit Jan Gutter proposed openstack/nova master: Add support for vrouter HW datapath offloads https://review.openstack.org/572082
11:43:01 gibi nehaalhat: what is the use case you want to tests?
11:46:09 ygk_12345 hi all
11:46:46 ygk_12345 I am seeing this error on the compute node
11:46:53 ygk_12345 HypervisorUnavailable: Connection to the hypervisor is broken on host: compute-7
11:47:01 ygk_12345 i cant do a virsh list command
11:47:14 nehaalhat gibi: working on this feature: https://review.openstack.org/#/c/609960/3/specs/stein/approved/placement-req-filter-forbidden-aggregates.rst
11:47:20 sean-k-mooney ygk_12345: that usually means the libvirtd daemon has exited
11:47:33 ygk_12345 sean-k-mooney: what does that mean ?
11:47:56 sean-k-mooney the libvirtd service has crashed/stopped on the compute node
11:48:12 jangutter ygk_12345 what distro and which version are you running?
11:48:33 ygk_12345 jangutter: Ubuntu 16.04.3 LTS \n \l
11:48:40 sean-k-mooney ygk_12345: try systemctl status libvirtd
11:49:03 ygk_12345 sean-k-mooney: I see this
11:49:05 ygk_12345 sean-k-mooney: 2019-01-09 11:48:44.365+0000: 16119: error : virPidFileAcquirePath:422 : Failed to acquire pid file '/var/run/libvirtd.pid': Resource temporarily unavailable
11:49:19 ygk_12345 sean-k-mooney: i did libvirtd -l
11:49:41 ygk_12345 sean-k-mooney: Failed to open file '/proc/28288/stat': No such file or directory
11:49:41 sean-k-mooney ygk_12345: that will try to start a new libvirtd
11:49:50 sean-k-mooney that is not the same thing as runing status
11:50:08 ygk_12345 sean-k-mooney: the libvirtd is running finr though
11:50:16 ygk_12345 sean-k-mooney: failed to connect to monitor socket: No such process
11:50:36 sean-k-mooney you said you cant run virsh command so its obviouly in an error state
11:50:41 nehaalhat gibi: and this spec is dependent on https://review.openstack.org/#/c/603352/4/specs/stein/approved/negative-aggregate-membership.rst
11:50:52 ygk_12345 sean-k-mooney: virsh list command is hanging
11:50:56 ygk_12345 sean-k-mooney: no output
11:51:04 gibi nehaalhat: OK. Do you need to mock something from placement because the placement change that supports your feature is not merged yet?
11:51:11 sean-k-mooney yes because the libvirt daemon is not responding to requests
11:51:17 nehaalhat gibi: yes
11:51:31 f10d4 hi
11:51:46 ygk_12345 sean-k-mooney: now after sometime the vrish command is working but the flaoting IP of the instances is not pingable
11:52:15 ygk_12345 sean-k-mooney: it seems tot fix itself after sometime
11:52:22 ygk_12345 sean-k-mooney: bit strange
11:52:23 sean-k-mooney virsh is just a client that publishes request to the libvirt daemon to execute commands via dbus
11:52:24 jangutter ygk_12345: Ubuntu 16.04 doesn't always use systemd: you might have to look at libvirt's logs directly. Check your system logs to see if everything's healthy or something isn't restarting all the time.
11:52:33 gibi nehaalhat: if you have both the placement change an the nova change locally then you have to instruct the nova test to use your local placement change for the functional test
11:53:06 sean-k-mooney jangutter: well the process could still be running but not respoinding on the libvirt contol socket also
11:53:18 gibi nehaalhat: when you will upload that to gerrit you can use a Depends-On tag in the commit message of the nova change that point to the placement change and the CI will act accordinlgy
11:54:06 ygk_12345 jangutter: i see some unable to connect to hyoervisor meaages from the nova in the syslog
11:54:08 sean-k-mooney ygk_12345: your floating ip issue coudl be several things the first thing i would check is if the vm actully has a privte ip and can ping its router
11:54:32 sean-k-mooney if that works then you shoudl check the fip namespace and see if the dnat rules for the vm are present
11:54:45 nehaalhat gibi: placement change yet to implemenetd by community members: https://review.openstack.org/#/c/603352/4/specs/stein/approved/negative-aggregate-membership.rst
11:54:50 jangutter sean-k-mooney: yeah, it could be many fun things. Hopefully there's logs.
11:55:02 ygk_12345 sean-k-mooney: jangutter usually rebooting the compute node fixes this issue. but that is not a viable solution always
11:55:25 jangutter ygk_12345: the nova message is not the root cause, it's a consequence of something else bad happening.
11:55:32 ygk_12345 jangutter: yes true
11:55:38 gibi nehaalhat: I'm affraid that without the placement feature exists you cannot make the final functional test in nova
11:55:48 ygk_12345 jangutter: unable to pinpoint it in the libvirtd
11:57:02 nehaalhat gibi: ya that will not be final FT but for my patch I want to add FT and for that need to mock placement method
11:57:05 sean-k-mooney ygk_12345: a less invasive way to correct it when it is broken is to restart libvirtd service and then restart the nova-compute agent
11:57:09 gibi nehaalhat: to run the nova functional test with local placement version, you can change tox.ini here https://github.com/openstack/nova/blob/master/tox.ini#L87
11:57:22 gibi nehaalhat: temporarly you can try to mock sure
11:57:37 sean-k-mooney still if you can find the actully issue with libvirtd that is obviously better
11:57:42 ygk_12345 sean-k-mooney: but even the restart of libvirts is not fixing the problem
11:57:42 gibi nehaalhat: I think the normal mock.patch() call should work in the functional tests
11:57:57 jangutter ygk_12345: you also should restart nova-compute
11:58:14 jangutter ygk_12345: after libvirtd is restarted
11:58:50 nehaalhat gibi: yes I used normal mock.patch() but still the call goes to method, like: @mock.patch('placement.util.normalize_member_of_qs_param')
11:58:59 sean-k-mooney ygk_12345: nova-compute could still have open file handels to the old libvirt socket so restarting nova-compute forces it to reopen them
11:59:26 ygk_12345 sean-k-mooney: ok
11:59:33 gibi nehaalhat: give me a couple of minutes, I will try to mock it mysef.
11:59:47 nehaalhat gibi: ok thank you
12:02:56 jangutter Weird question: I submitted a review, and openstack-infra updated the link to the review with a broken link on a blueprint. Did I do something stupid?
12:03:34 sean-k-mooney which revierw

Earlier   Later