Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-03-29
16:30:23 clarkb mdbooth: yes, which is why I'm trying to find the anwer of what they will support and work with us on
16:30:24 mdbooth clarkb: Well, I hate to sound like a salesperson, but RH does maintain its virt stack...
16:30:28 clarkb and that answer doesn't seem to exist anywhere
16:30:50 clarkb mdbooth: and if you can get openstack to actually work on centos/fedora that would be awesome, but the current jobs that exist don't work
16:30:58 kashyap clarkb: About reproducing the problem, I'd say:
16:31:29 clarkb mdbooth: the fedora 25 job is currently experimental on devstack-gate and devstack and tempest I think if people are interested
16:31:30 kashyap libvirt logs the precise version that was used
16:33:29 mdbooth sdague ianw: I spotted a small bug in that libvirtd core dump patch. Care to take a look at my follow-on?
16:34:33 mdbooth I've tested the follow-on on Fedora 25, and barring package name typos I'm pretty confident it works on Ubuntu too
16:35:51 mdbooth clarkb: What's the scope of the problem running on CentOS, btw?
16:36:00 clarkb http://logs.openstack.org/78/448078/1/check/gate-tempest-dsvm-neutron-full-fedora-25-nv/28c5592/console.html is an example job failing on fedora 25
16:36:03 mdbooth Just an abundance of Ubuntu assumptions?
16:36:39 clarkb mdbooth: no, actually looks like centos7 job may currently be working. Typically the issues it has are more related to old software vs new software assumptions
16:36:48 clarkb not a debuntu vs not assumption
16:38:30 mdbooth clarkb: Ok, that's probably not so bad then.
16:39:55 jamespage mriedem: cpaezler has put test binaries up for that problem - is that something we can test with to confirm fixes look OK?
16:40:09 clarkb mdbooth: systemd has helped the two sides converge a bit more on managing the systems themselves (though still with completely separate ideas on unit contents and what should remain a sys v init script etc)
16:40:25 openstackgerrit Rodolfo Alonso Hernandez proposed openstack/nova master: Modify PciPassthroughFilter to accept lists https://review.openstack.org/444946
16:40:25 openstackgerrit Rodolfo Alonso Hernandez proposed openstack/nova master: Read Neutron port 'binding_profile' during boot https://review.openstack.org/449257
16:40:25 openstackgerrit Rodolfo Alonso Hernandez proposed openstack/nova master: Read NIC features in libvirt https://review.openstack.org/444415
16:40:59 mdbooth clarkb: yeah, that's pretty nice actually. See that patch I posted above to enable running libvirtd under electric fence.
16:41:14 clarkb mdbooth: but a recent example of problem on centos7 was old version of apache having poor reload performance (not sure if due to unit or apache itself or what, but then apache package got updated on centos and problem went away)
16:41:20 mdbooth It's good to be able to set environment variables via systemd, because it's common
16:41:32 mriedem jamespage: i don't think we have a way to hook in test repos/packages for our CI jobs
16:41:41 mriedem jamespage: and not sure we have a recreate outside of CI
16:41:43 mriedem would have to ask kashyap
16:42:22 clarkb mriedem: jamespage: it should be pretty simple to have devstack wget && dpkg -i at the very least, but maybe also add a ppa and then let normal install of libvirt pull from there
16:42:48 mriedem mdbooth: lyarwood: there is a new lock added to the refactor here https://review.openstack.org/#/c/439520/
16:43:09 openstack Launchpad bug 1643911 in OpenStack Compute (nova) "libvirt randomly crashes on xenial nodes with "*** Error in `/usr/sbin/libvirtd': malloc(): memory corruption:"" [Medium,Confirmed]
16:43:09 kashyap jamespage: If you're referring to the fixes from 1673483, just wrote this comment - https://bugs.launchpad.net/nova/+bug/1643911/comments/19
16:43:13 mriedem mdbooth: lyarwood: i'm really not happy about adding new things to an already large and not easy refactor change
16:43:19 mriedem mdbooth: lyarwood: stuff like that should be separated
16:44:13 mdbooth mriedem: Actually this version of the refactor is pretty trivial: it's just code motion
16:44:19 clarkb I think dmsimard was trying to track down filesystem problems on centos7 too. I should swing back around and find out what happened wit hthat
16:44:27 mdbooth The initial version did more than this
16:44:36 mdbooth The lock doesn't need to be there, sure
16:44:59 mriedem mdbooth: slipping things into a refactor is not cool, period
16:45:01 openstack bug 1673483 in libvirt (Ubuntu) "libvirt: test_attach_volume_shelved_or_offload_server times out waiting for device detach (which fails)" [Medium,Triaged] https://launchpad.net/bugs/1673483
16:45:01 jamespage kashyap: yes I was referring to bug 1673483
16:45:22 mriedem mdbooth: as i've already noted in the review, even with just code motion, there are issues,
16:45:25 mriedem which makes stuff like this hard to review
16:45:27 kashyap jamespage: They're both are not same -- we haven't definitively identified so
16:45:29 clarkb kashyap: if we can get an answer to what an appropriate version of libvirt to repreoduce with is (maybe that 1.3.3 maintenance release) I would be happy to help get that in place on devstack test nodes
16:45:31 mriedem because it's hard to miss differences
16:46:03 openstack bug 1673483 in libvirt (Ubuntu) "libvirt: test_attach_volume_shelved_or_offload_server times out waiting for device detach (which fails)" [Medium,Triaged] https://launchpad.net/bugs/1673483
16:46:03 jamespage not sure I said they where - just wanted to see if we could get some validation of the fixes for bug 1673483
16:46:23 kashyap jamespage: Because, the Nova stack traces for both bugs are different
16:46:28 clarkb jamespage: fwiw we also have http://status.openstack.org/elastic-recheck/index.html#1643911 http://status.openstack.org/elastic-recheck/index.html#1638982 http://status.openstack.org/elastic-recheck/index.html#1646779 which are our top failures according to elastic-recheck
16:46:32 mdbooth mriedem: Re the **kwargs thing, I agree btw. However, it is consistent with attach, and that's the primary goal here.
16:46:45 kashyap clarkb: The buggy version should surely be easily identified - let me look
16:47:14 clarkb jamespage: all three are marked new against libvirt in launchpad
16:47:47 mriedem mdbooth: being consistent with a bad idea is not reasonable justification to continue doing it
16:47:48 mriedem imo
16:49:25 clarkb kashyap: I can find the version we are using (that is buggy) easily. I am trying to find the version that would be an acceptable reproduction platform in order to push back to upstream (and maybe we find its a non issue there and we bisect and try to find root cause)
16:50:08 kashyap clarkb: Acceptable reproduction platform would be: the same test case be run on (a) upstream current stable; or (b) Git master
16:50:42 clarkb kashyap: yes what is upstream current stable?
16:50:48 clarkb that is the question :)
16:50:50 kashyap 1 se
16:50:51 kashyap 1 sec
16:51:05 kashyap v3.1.0 (2017-03-03)
16:51:14 clarkb the downloads page says you can't base "stable" on versions anymore and I can't find anything that says what is stable
16:51:17 kashyap That's the latest released, though
16:51:26 clarkb right that isn't necessarily latest stable according to the docs
16:51:35 clarkb since versions are time based now
16:51:52 kashyap clarkb: Yes, you can get a sense of WTF the release numbering means: http://libvirt.org/downloads.html#numbering
16:52:02 kashyap "Prior to 2.0.0, the major/minor numbers were incremented fairly arbitrarily, and maintenance releases appended a fourth digit."
16:52:13 clarkb kashyap: right and what that says is nothing in a version implies stability
16:52:19 clarkb so where do I find the list of what is stable?
16:52:59 clarkb maybe I need more caffeine but nothing seems to actually say. Which amkes me sympathize with ubuntu
16:53:02 kashyap clarkb: In context of upstream distributions (non-RHEL), I'd interpret "stable" as whatever is currently released. I know that's a "loose" definition
16:53:44 kashyap Since the "maintainance" release in context of libvirt are only for bug / CVE fixes.
16:54:37 clarkb oh hey 3.1.0 is what I already have on tumbleweed cool
16:55:22 kashyap Cool
16:55:54 kashyap clarkb: It seems to happen during an instance termination (with Nova's "ImagesTestJSON"), and libvirtd dying subsequently.
16:56:31 openstack Launchpad bug 1673483 in libvirt (Ubuntu) "libvirt: test_attach_volume_shelved_or_offload_server times out waiting for device detach (which fails)" [Medium,Triaged]
16:56:31 mriedem clarkb: sorry if you guys already talked about this, but in https://bugs.launchpad.net/nova/+bug/1673483 there are comments from ubuntu devs that backports have been made to some packages and are available for testing,
16:56:34 openstackgerrit Dan Smith proposed openstack/nova master: Teach HostAPI about cells https://review.openstack.org/442162
16:56:35 openstackgerrit Dan Smith proposed openstack/nova master: Make scheduler target cells to get compute node instance info https://review.openstack.org/439891
16:56:36 openstackgerrit Dan Smith proposed openstack/nova master: Sort CellMappingList.get_all() for safety https://review.openstack.org/443174
16:56:37 openstackgerrit Dan Smith proposed openstack/nova master: Clean up ClientRouter debt https://review.openstack.org/444487
16:56:37 openstackgerrit Dan Smith proposed openstack/nova master: Add get_by_instance_uuids() to InstanceMappingList https://review.openstack.org/443292
16:56:38 openstackgerrit Dan Smith proposed openstack/nova master: Make server_groups determine deleted-ness from InstanceMappingList https://review.openstack.org/443293
16:56:39 openstackgerrit Dan Smith proposed openstack/nova master: Remove Mitaka-era service version check https://review.openstack.org/442861
16:56:41 openstackgerrit Dan Smith proposed openstack/nova master: Add workaround to disable group policy check upcall https://review.openstack.org/442736
16:56:42 mriedem clarkb: it's not clear to me if we have a way to hook testing packages in our CI?
16:56:45 clarkb mriedem: yes but that bug isn't a problem
16:56:54 clarkb mriedem: we have 3 other libvirt bugs that are actually major issues
16:57:09 clarkb mriedem: and I mentioned it would be simple to modify devstack to use a ppa or just wget and dpkg -i
16:57:45 mriedem ok, and we don't have fixes identified to backport for 1.3.3 in xenial LTS for testing yet
16:58:03 jamespage clarkb: can I get context please on those three bugs
16:58:14 jamespage refs would be ok for now
16:58:28 mriedem jamespage: https://bugs.launchpad.net/nova/+bug/1643911/
16:58:28 openstack bug 1643911 in OpenStack Compute (nova) "libvirt randomly crashes on xenial nodes with "*** Error in `/usr/sbin/libvirtd': malloc(): memory corruption:"" [Medium,Confirmed] https://launchpad.net/bugs/1643911
16:58:28 jamespage guessing one is bug 1643911
16:58:45 clarkb jamespage: its the three elastic recheck links I gave you above
16:58:49 clarkb jamespage: they have links to launchpad
16:58:54 jamespage clarkb: ta
16:59:18 mriedem jamespage: some info in http://lists.openstack.org/pipermail/openstack-dev/2017-March/114648.html too
16:59:31 mdbooth dansmith: https://review.openstack.org/#/c/443292/ https://review.openstack.org/#/c/445139/ :)
17:01:32 openstackgerrit Anusha Unnam proposed openstack/nova master: Add description to policies in servers_migrations.py https://review.openstack.org/451487

Earlier   Later