Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-03-29
16:15:27 mdbooth clarkb: To be clear... what kashyap said
16:15:57 clarkb https://libvirt.org/downloads.html#maintenance ?
16:15:57 openstack Launchpad bug 1673483 in libvirt (Ubuntu) "libvirt: test_attach_volume_shelved_or_offload_server times out waiting for device detach (which fails)" [Medium,Triaged]
16:15:57 kashyap clarkb: BTW, even for this bug -- https://bugs.launchpad.net/nova/+bug/1673483
16:15:59 mriedem clarkb: https://wiki.openstack.org/wiki/LibvirtDistroSupportMatrix
16:16:08 mriedem mdbooth: what do you mean we're running on unmaintained infra?
16:16:14 mriedem we're running on 16.04 LTS right?
16:16:15 kashyap clarkb: I had to spend a good half a day talking to upstream libvirt, who identified what exact commits Ubuntu should backport
16:16:25 mdbooth mriedem: Sure. Lets file a bug there.
16:16:41 kashyap So they don't even bother to look at what's breaking, and simply apply the available bug fixes
16:16:41 clarkb kashyap: that bug doesn't seem to be one of the ones we are tracking though?
16:17:09 mdbooth mriedem: I just think it's unreasonable to expect upstream libvirt to spin their wheels on this.
16:17:10 kashyap clarkb: No, it's not. Was just giving an example, that upstream spends more time than is fair (in some cases) with downstream distrubtions
16:17:18 clarkb kashyap: gotcha
16:17:20 openstack Launchpad bug 1673483 in libvirt (Ubuntu) "libvirt: test_attach_volume_shelved_or_offload_server times out waiting for device detach (which fails)" [Medium,Triaged]
16:17:20 mriedem if libvirt devs have identified fixes that are needed in https://bugs.launchpad.net/nova/+bug/1673483 then we can reach out to jamespage and/or coreycb for help i'd think
16:18:03 gabor_antal Sorry for the slightly off topic, I have a question: Some time before I read an article about the effect of the VM workloads which are on the same physical host, but I cannot find it. Can someone please help me with it?
16:19:09 clarkb kashyap: mdbooth is http://wiki.libvirt.org/page/Maintenance_Releases#1.3.3_series what would be considered the most current supported release?
16:19:32 kashyap clarkb: I guess there's a more automated version -- was there one? mriedem?
16:19:50 kashyap (Version of a wiki page I mean)
16:20:28 mriedem i only looked at https://wiki.openstack.org/wiki/LibvirtDistroSupportMatrix
16:20:29 clarkb https://wiki.openstack.org/wiki/LibvirtDistroSupportMatrix shows what linux distros are supporting downstream of libvirt, I'm looking for what upstream libvirt considers supported
16:20:29 kashyap mriedem: Disregard me, please.
16:20:32 mriedem i'm not sure i understand the question
16:20:37 mdbooth clarkb: Well latest stable is 2.2
16:20:55 clarkb mdbooth: ok so stable and maintenance are different /me keeps reading
16:20:59 mdbooth clarkb: I'm guessing that in practise backporting to 1.3 is best effort
16:21:34 vladikr vks1, hmm, I think this is a more of a question for the neutron team. We are only configuring vlan tags for SRIOV devices. otherwise, I think it's neutron it setup up the environment
16:22:18 vks1 vladikr: ok thanks
16:23:16 kashyap clarkb: Yes, you're right. (On stable / maint releases)
16:23:59 clarkb https://libvirt.org/downloads.html#numbering says since 2.0 you can't use the version numbers to know what is stable
16:24:04 kashyap clarkb: Tomorrow morning I'll check with upstream if we can get to the bottom of the issue with whatever debugging info we have as of now
16:24:06 clarkb so I'm trying to figure out how I can determine that
16:24:36 clarkb there are maintenance releases which I think are also stable but there may be newer stables that are not maintenance
16:26:58 kashyap clarkb: More about those maintainance releases:
16:26:59 kashyap "These maintenance branches should only contain bug fixes, and no new features, backported from the master branch, and are supported as long as at least one downstream distribution expresses interest in a given branch. These maintenance branches are considered during CVE analysis. In contrast to the primary releases which are made once a month, there is no formal schedule for the maintenance
16:27:05 kashyap releases, which are made whenever there is a need to make available key bugfixes to downstream consumers."
16:28:32 jamespage mriedem: looking
16:28:38 clarkb right so two things I am trying to figure out here 1) if I was ubunut (I am not) how would I properly select the version of libvirt to include in the next version of a distro release? 2) if I'm openstack infra and want to attempt to reproduce a problem on a supported stable version of libvirt where do I find such a version of libvirt?
16:28:41 clarkb kashyap: ^
16:28:52 clarkb kashyap: and unfortunately I am not really get a clear answer for either thing
16:28:52 openstackgerrit Anusha Unnam proposed openstack/nova master: Add description to policies in services.py https://review.openstack.org/449782
16:29:42 clarkb is using the newest maintenace release the answer to both questions?
16:29:43 kashyap clarkb: Can't tell you without really digging a bit closer.
16:30:01 mdbooth clarkb: Anyway, if we want to keep running an old release, we need downstream support for that. My experience of upstream libvirt is that when I've given them something concrete they've been extremely helpful. However, we can't expect them to go on a huge fishing trip for an old downstream release.
16:30:15 kashyap Yeah, precisely.
16:30:23 clarkb mdbooth: yes, which is why I'm trying to find the anwer of what they will support and work with us on
16:30:24 mdbooth clarkb: Well, I hate to sound like a salesperson, but RH does maintain its virt stack...
16:30:28 clarkb and that answer doesn't seem to exist anywhere
16:30:50 clarkb mdbooth: and if you can get openstack to actually work on centos/fedora that would be awesome, but the current jobs that exist don't work
16:30:58 kashyap clarkb: About reproducing the problem, I'd say:
16:31:29 clarkb mdbooth: the fedora 25 job is currently experimental on devstack-gate and devstack and tempest I think if people are interested
16:31:30 kashyap libvirt logs the precise version that was used
16:33:29 mdbooth sdague ianw: I spotted a small bug in that libvirtd core dump patch. Care to take a look at my follow-on?
16:34:33 mdbooth I've tested the follow-on on Fedora 25, and barring package name typos I'm pretty confident it works on Ubuntu too
16:35:51 mdbooth clarkb: What's the scope of the problem running on CentOS, btw?
16:36:00 clarkb http://logs.openstack.org/78/448078/1/check/gate-tempest-dsvm-neutron-full-fedora-25-nv/28c5592/console.html is an example job failing on fedora 25
16:36:03 mdbooth Just an abundance of Ubuntu assumptions?
16:36:39 clarkb mdbooth: no, actually looks like centos7 job may currently be working. Typically the issues it has are more related to old software vs new software assumptions
16:36:48 clarkb not a debuntu vs not assumption
16:38:30 mdbooth clarkb: Ok, that's probably not so bad then.
16:39:55 jamespage mriedem: cpaezler has put test binaries up for that problem - is that something we can test with to confirm fixes look OK?
16:40:09 clarkb mdbooth: systemd has helped the two sides converge a bit more on managing the systems themselves (though still with completely separate ideas on unit contents and what should remain a sys v init script etc)
16:40:25 openstackgerrit Rodolfo Alonso Hernandez proposed openstack/nova master: Modify PciPassthroughFilter to accept lists https://review.openstack.org/444946
16:40:25 openstackgerrit Rodolfo Alonso Hernandez proposed openstack/nova master: Read Neutron port 'binding_profile' during boot https://review.openstack.org/449257
16:40:25 openstackgerrit Rodolfo Alonso Hernandez proposed openstack/nova master: Read NIC features in libvirt https://review.openstack.org/444415
16:40:59 mdbooth clarkb: yeah, that's pretty nice actually. See that patch I posted above to enable running libvirtd under electric fence.
16:41:14 clarkb mdbooth: but a recent example of problem on centos7 was old version of apache having poor reload performance (not sure if due to unit or apache itself or what, but then apache package got updated on centos and problem went away)
16:41:20 mdbooth It's good to be able to set environment variables via systemd, because it's common
16:41:32 mriedem jamespage: i don't think we have a way to hook in test repos/packages for our CI jobs
16:41:41 mriedem jamespage: and not sure we have a recreate outside of CI
16:41:43 mriedem would have to ask kashyap
16:42:22 clarkb mriedem: jamespage: it should be pretty simple to have devstack wget && dpkg -i at the very least, but maybe also add a ppa and then let normal install of libvirt pull from there
16:42:48 mriedem mdbooth: lyarwood: there is a new lock added to the refactor here https://review.openstack.org/#/c/439520/
16:43:09 openstack Launchpad bug 1643911 in OpenStack Compute (nova) "libvirt randomly crashes on xenial nodes with "*** Error in `/usr/sbin/libvirtd': malloc(): memory corruption:"" [Medium,Confirmed]
16:43:09 kashyap jamespage: If you're referring to the fixes from 1673483, just wrote this comment - https://bugs.launchpad.net/nova/+bug/1643911/comments/19
16:43:13 mriedem mdbooth: lyarwood: i'm really not happy about adding new things to an already large and not easy refactor change
16:43:19 mriedem mdbooth: lyarwood: stuff like that should be separated
16:44:13 mdbooth mriedem: Actually this version of the refactor is pretty trivial: it's just code motion
16:44:19 clarkb I think dmsimard was trying to track down filesystem problems on centos7 too. I should swing back around and find out what happened wit hthat
16:44:27 mdbooth The initial version did more than this
16:44:36 mdbooth The lock doesn't need to be there, sure
16:44:59 mriedem mdbooth: slipping things into a refactor is not cool, period
16:45:01 openstack bug 1673483 in libvirt (Ubuntu) "libvirt: test_attach_volume_shelved_or_offload_server times out waiting for device detach (which fails)" [Medium,Triaged] https://launchpad.net/bugs/1673483
16:45:01 jamespage kashyap: yes I was referring to bug 1673483
16:45:22 mriedem mdbooth: as i've already noted in the review, even with just code motion, there are issues,
16:45:25 mriedem which makes stuff like this hard to review
16:45:27 kashyap jamespage: They're both are not same -- we haven't definitively identified so
16:45:29 clarkb kashyap: if we can get an answer to what an appropriate version of libvirt to repreoduce with is (maybe that 1.3.3 maintenance release) I would be happy to help get that in place on devstack test nodes
16:45:31 mriedem because it's hard to miss differences
16:46:03 openstack bug 1673483 in libvirt (Ubuntu) "libvirt: test_attach_volume_shelved_or_offload_server times out waiting for device detach (which fails)" [Medium,Triaged] https://launchpad.net/bugs/1673483
16:46:03 jamespage not sure I said they where - just wanted to see if we could get some validation of the fixes for bug 1673483
16:46:23 kashyap jamespage: Because, the Nova stack traces for both bugs are different
16:46:28 clarkb jamespage: fwiw we also have http://status.openstack.org/elastic-recheck/index.html#1643911 http://status.openstack.org/elastic-recheck/index.html#1638982 http://status.openstack.org/elastic-recheck/index.html#1646779 which are our top failures according to elastic-recheck
16:46:32 mdbooth mriedem: Re the **kwargs thing, I agree btw. However, it is consistent with attach, and that's the primary goal here.
16:46:45 kashyap clarkb: The buggy version should surely be easily identified - let me look
16:47:14 clarkb jamespage: all three are marked new against libvirt in launchpad
16:47:47 mriedem mdbooth: being consistent with a bad idea is not reasonable justification to continue doing it
16:47:48 mriedem imo
16:49:25 clarkb kashyap: I can find the version we are using (that is buggy) easily. I am trying to find the version that would be an acceptable reproduction platform in order to push back to upstream (and maybe we find its a non issue there and we bisect and try to find root cause)

Earlier   Later