Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-01-25
21:51:26 melwitt I feel like I've seen that before
21:52:56 dansmith yeah, libvirtd doesn't look like it was restarted across that connection failure boundary
21:52:58 mriedem interestingly, it's the first guest to be created it looks like http://logs.openstack.org/04/536904/3/check/legacy-tempest-dsvm-cells/13aacfc/logs/libvirt/qemu/instance-0000000a.txt.gz
21:53:17 dansmith this would be the first time I've ever seen that symptom not associated with a crash
21:53:34 dansmith so I still feel a little doubtful, but it's certainly not as glaring as normal
21:53:41 mriedem makes me think of this https://review.openstack.org/#/c/526748/
21:53:50 mriedem where guestfs on the first bring up runs in a tpool
21:53:56 mriedem to get the guestfs capabilities
21:54:14 dansmith so I wonder if it's something like qemu crashes and the libvirt thread we're talking to exits or something
21:54:23 melwitt yeah, that's kinda how I found that patch of yours mriedem
21:54:48 dansmith the journal should have dmesg in it to capture any segv
21:55:07 dansmith but we don't have qemu logs otherwise
21:55:10 melwitt I wonder if the journal collection on the jobs has been busted this whole time
21:55:32 melwitt or is it not always empty?
21:55:51 dansmith it wasn't busted a month or so ago
21:55:52 melwitt I haven't tried to look at it before
21:55:56 melwitt oh okay
21:56:24 dansmith I expect it would be in syslog too, but it's clearer in the journal
21:57:25 melwitt I didn't see anything that looked like crashing in syslog http://logs.openstack.org/04/536904/3/check/legacy-tempest-dsvm-cells/13aacfc/logs/syslog.txt.gz
21:57:38 dansmith 2018-01-25 19:06:13.215+0000: 3732: error : virKeepAliveTimerInternal:143 : internal error: connection closed due to keepalive timeout
21:57:45 dansmith is right before you get it in nova
21:59:13 melwitt where do you see that? libvirtd.log?
21:59:27 dansmith I think this is all in a separate thread in nova so it shouldn't be that nova is hung up and unable to do the keepalive bit, unless it's just system starvation of some sort
21:59:30 dansmith yeah, libvirt
21:59:50 melwitt damn
22:00:06 dansmith and nova is doing stuff milliseconds before that, so it's not hung I don't think
22:00:51 dansmith oh yeah in the nova log:
22:00:52 dansmith 2018-01-25 19:06:13.224 16016 INFO nova.virt.libvirt.driver [req-ea2c4c13-3bbe-4b35-a2b3-84976c11fc4b - -] Connection event '0' reason 'Connection to libvirt lost'
22:01:08 dansmith that's the real thread detecting the connection close
22:01:20 dansmith oooh, but
22:01:49 dansmith 2018-01-25 19:06:09.554 16016 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 9ebc51b2583b4f4da811e951e41068c9 __call__ /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:407
22:01:50 dansmith 2018-01-25 19:06:13.218 16016 DEBUG nova.virt.libvirt.firewall [req-733b7704-8aa3-45a5-8798-67eab246baf8 tempest-InstanceActionsV221TestJSON-176157016 tempest-InstanceActionsV221TestJSON-176157016] Cannot find UUID for filter 'nova-instance-instance-0000000d-fa163e269665': 'Cannot recv data: Connection reset by peer' _get_filter_uuid /opt/stack/new/nova/nova/virt/libvirt/firewall.py:254
22:02:00 dansmith that is five seconds apart with nothing in between
22:02:48 openstackgerrit Jim Rollenhagen proposed openstack/nova master: Add functional tests for virt driver get_traits() method https://review.openstack.org/538047
22:02:58 melwitt hm
22:03:04 dansmith meh, I dunno, looking around that I don't think that's anything related to being stuck though
22:03:14 dansmith because that's the exact instant libvirt logs its message
22:06:48 openstackgerrit Jim Rollenhagen proposed openstack/nova master: Add functional tests for virt driver get_traits() method https://review.openstack.org/538047
22:08:40 jackie-truong dansmit, mriedem: re: git commands
22:08:52 jroll mriedem: ^ see my comment there, fairly certain some of this traits stack is somewhat broken. probably fine to land as drivers aren't using it yet, but ya know.
22:09:12 jackie-truong i want to add a /new/ file (test_trusted_certs.py) to a patch in the middle of my patch stack
22:09:47 jackie-truong checking out the top of the patch stack, then adding and committing the file results in the file being added to the top patch
22:09:52 mriedem efried: https://gist.github.com/jimrollenhagen/384077e2ed606ee84051cb4a7f66c51c
22:10:08 mriedem jackie-truong: git rebase -i master
22:10:10 dansmith jackie-truong: right, so git rebase -i master, edit the middle patch, git add path/to/file, git commit --amend -a
22:10:14 mriedem jackie-truong: edit the commit you want to add the file to,
22:10:20 mriedem create said file,
22:10:22 mriedem git add <file>
22:10:25 mriedem git rebase --continue
22:11:06 jackie-truong ah, i rarely actually edit commits when rebasing from master. got it, thanks again
22:11:09 efried mriedem VMX ain't a standard trait, or a valid CUSTOM_ one.
22:11:22 efried jroll ^
22:11:24 mriedem efried: it's in os-traits
22:11:29 jroll ^
22:12:24 mriedem oh wait
22:12:25 mriedem i see
22:12:28 mriedem jroll: he's right
22:12:29 efried HW_CPU_X86_VMX ?
22:12:39 mriedem yeah, the traits are generated from their directory structure within os-traits
22:12:44 jroll oh gdi
22:12:54 mriedem https://docs.openstack.org/os-traits/latest/user/index.html
22:15:54 jroll mriedem: efried: that did it, now I feel dumb, thanks :)
22:16:08 mriedem don't feel dumb
22:16:16 efried For not understanding the bizarre architecture of os-traits? Yeah, what mriedem said.
22:16:30 mriedem ironicclient change is 85 in the gate
22:16:44 efried I always just GET /traits from the API to find my testable traits.
22:17:29 efried Because you'll also get tripped up by this one at some point in the future: Your compute's os-traits may not be the same as the placement server's os-traits.
22:18:35 efried which is why https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L1005-L1009
22:22:13 openstackgerrit Jackie Truong proposed openstack/nova master: Reduce complexity of _from_db_object https://review.openstack.org/537728
22:22:14 openstackgerrit Jackie Truong proposed openstack/nova master: Add trusted_certs object https://review.openstack.org/489408
22:22:14 openstackgerrit Jackie Truong proposed openstack/nova master: Add trusted_certs to instance_extra https://review.openstack.org/537897
22:22:15 openstackgerrit Jackie Truong proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204
22:22:15 openstackgerrit Jackie Truong proposed openstack/nova master: Implement certificate_utils https://review.openstack.org/479949
22:25:17 dansmith woot
22:25:50 dansmith I shall gaze upon thy patches upon my return from a brief respite
22:25:51 mriedem melwitt: this makes me think about the eventlet/python thing in the ML https://review.openstack.org/#/c/537811/1
22:26:10 jackie-truong *\o/*
22:26:29 melwitt are those pom poms?
22:28:05 melwitt mriedem: interesting
22:29:55 melwitt seems like it could be related. nothing makes too much sense so far, only cells v1 job is getting constant libvirt keepalive timeouts, starting about 10 days ago, and only on stable
22:30:10 melwitt pike and ocata
22:30:36 mriedem melwitt: unrelated, but related to lyarwood's change to hard reboot and blow things away all the time https://bugs.launchpad.net/nova/+bug/1744361
22:30:37 openstack Launchpad bug 1744361 in neutron "test_server_security_groups failed to reboot with "Domain not found: no domain with matching uuid" because of missing vif event from linuxbridge agent" [Undecided,Confirmed]
22:31:14 mriedem i thought there were some assertions in the review that we didn't need to wait for vif plugged events, or something along those lines
22:31:52 mriedem https://review.openstack.org/#/c/400384/
22:31:52 melwitt ah, suck
22:32:26 melwitt well, I had commented that we do need to wait for them since we are re-plugging vifs. but I didn't realize we'd be in a vm_state that would ignore them
22:32:44 jroll efried: fun, thanks
22:33:02 melwitt that is, I didn't expect the events to be ignored when they're being waited for
22:36:06 mriedem melwitt: it's not saying it's ignored b/c of the instance state
22:36:20 mriedem it's saying the compute got the event but no callback was registered waiting for it
22:36:28 mriedem Jan 19 10:33:35.986648 ubuntu-xenial-rax-iad-0002010825 nova-compute[28907]: DEBUG nova.compute.manager [None req-4bcf94cf-2048-4e99-94e6-646519900469 service nova] [instance: dd2ec3b0-f909-4104-ae2a-9b878d936ed4] No waiting events found dispatching network-vif-plugged-567525a4-4b16-4004-91d5-545086d01227 {{(pid=28907) pop_instance_event /opt/stack/new/nova/nova/compute/manager.py:363}}
22:37:49 melwitt mriedem: okay, I was going by the last comment in the bug. okay, so maybe there's a bug in the _create_domain_and_network vifs_already_plugged logic somewhere
22:38:29 melwitt or maybe vifs_already_plugged doesn't mean what I thought it means
22:40:45 mriedem i wonder if we don't register the callback before the vif plug event has already happened
22:41:10 mriedem no that doesn't make sense
22:41:15 mriedem we register the callback before calling plug_vifs
22:41:26 melwitt "Timeout waiting for vif plugging callback for instance with vm_state active and task_state reboot_started_hard.: Timeout: 300 seconds" so that means it was waiting
22:45:02 melwitt so "Received unexpected event" happens when there weren't events in self.instance_events in compute manager. so how do they get there ...
22:45:07 dansmith mriedem: yeah the whole design of that thing is so you can't trigger before you're registered

Earlier   Later