| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-03-29 | |||
| 16:08:59 | clarkb | kashyap: mdbooth: hello, right now three of the four top bugs tracked by elastic-recheck originate in libvirt http://status.openstack.org/elastic-recheck/index.html#1643911 http://status.openstack.org/elastic-recheck/index.html#1638982 http://status.openstack.org/elastic-recheck/index.html#1646779 I know kashyap has poked at these in the past, but would it be possible to try and figure out these | |
| 16:09:01 | clarkb | issues? ianw started working on getting devstack to collect core dumps (though those aren't thought to be of help here) and was going to try running libvirt under electric fence. Maybe we can get a small group of people to coordinate around this and work together? | |
| 16:09:02 | vks1 | artom: I have 2 VMs launched on 2 different compute. When I see the tap ports on OVS , on one compute I see tag:1 and on other compute its tag:3 | |
| 16:09:24 | vks1 | artom: both the VMs are of same n/w | |
| 16:09:33 | kashyap | clarkb: Hi there, I've seen yours and Ian's responses in the Inbox, but I still haven't had a chance to act on them | |
| 16:09:44 | kashyap | clarkb: Yeah, core dumps won't help in this case | |
| 16:09:58 | artom | vks1, err, you set the tags yourself at instance create? | |
| 16:10:00 | kashyap | clarkb: At least that's what the libvirt dev involved (Michal Privoznik) said | |
| 16:10:06 | vks1 | artom: no | |
| 16:10:19 | artom | vks1, err, which tags are you talking about then? | |
| 16:10:29 | mdbooth | clarkb: Couple of things, I posted an update to ianw 's patch which enables efence and also works on Fedora: https://review.openstack.org/#/c/451430/ | |
| 16:10:33 | clarkb | kashyap: ianw figured it was something devstack should do anyways so wouldn't hurt to have it, then we can actually rule it out as not being helpful | |
| 16:10:48 | vks1 | Interface "tap319bae97-0b" | |
| 16:10:48 | vks1 | tag: 1 | |
| 16:10:48 | vks1 | artom: Port "tap319bae97-0b" | |
| 16:10:49 | mdbooth | clarkb: Secondly, I think we have to treat this as a downstream Ubuntu bug | |
| 16:10:52 | mriedem | mdbooth: i want to get that refactor merged this week | |
| 16:10:56 | kashyap | clarkb: Oh that's certainly true -- having a coredump ready, it's super easy with systemd :-) | |
| 16:11:06 | artom | vks1, what is that the output of? | |
| 16:11:09 | kashyap | clarkb: I mean, having them available in general | |
| 16:11:15 | mdbooth | clarkb: The Ubuntu maintainers need to look at this | |
| 16:11:16 | clarkb | mdbooth: it certainly affects ubuntu (I don't have evidence of it affecting fedora or centos), but that doesn't necessarily mean its their bug | |
| 16:11:17 | vks1 | artom: ovs-vsctl show | |
| 16:11:20 | artom | vks1, I have a feeling you're talking about vlan ids... | |
| 16:11:27 | vks1 | artom: yeah | |
| 16:11:35 | clarkb | mdbooth: the last time we had a similar situation it was very clearly libvirts bug once someone from libvirt actually looked at it (instead of blaming ubuntu) | |
| 16:11:49 | mdbooth | clarkb: As I said in some LP bug somewhere, upstream libvirt aren't going to be that interested in this, because it's not the latest stable release | |
| 16:11:54 | artom | mriedem, you volunteered me for the wrong thing dude ;) | |
| 16:12:12 | mdbooth | clarkb: A libvirt maintainer has already looked, and doesn't think it's broken in the latest stable release | |
| 16:12:15 | mriedem | artom: redirect to vladikr? | |
| 16:12:15 | artom | vks1, so... different ports have different vlans? | |
| 16:12:28 | mriedem | artom: delegation is the sign of true leadership :) | |
| 16:12:28 | vks1 | artom: yeah | |
| 16:12:39 | artom | mriedem, *proper* delegation ;) | |
| 16:12:52 | artom | vks1, so is that bad? What's your expectation? | |
| 16:12:57 | openstack | Launchpad bug 1673483 in libvirt (Ubuntu) "libvirt: test_attach_volume_shelved_or_offload_server times out waiting for device detach (which fails)" [Medium,Triaged] | |
| 16:12:57 | kashyap | mdbooth: To answer your question from earlier in the day -- it's not actually the same bug as this one (https://bugs.launchpad.net/nova/+bug/1673483) | |
| 16:12:59 | vladikr | :) | |
| 16:13:05 | kashyap | mdbooth: Where danpb identified the missing patches | |
| 16:13:19 | vks1 | artom: thanks vladikr is here | |
| 16:13:21 | clarkb | mdbooth: ok, this is frustrating because have had had similar issues in the past with the same exact attitude from upstream and the end result was libvirt had a terrible bug in it that affected every version since the code went in | |
| 16:13:33 | clarkb | and it took months to get fixed because no one would look at it for more than 5 minutes | |
| 16:13:42 | vks1 | artom: vlan tagging should be same | |
| 16:13:45 | clarkb | I'm not saying thats the case here, but that attitude hasn't been productive in the past | |
| 16:13:47 | vks1 | vladikr: hi | |
| 16:14:19 | kashyap | mdbooth: Because, the Nova stack traces are different from both bugs, that's why we didn't close them as duplicates | |
| 16:14:25 | mdbooth | clarkb: Well, we're running our CI on unmaintained, unsupported infrastructure | |
| 16:14:25 | openstackgerrit | Sujitha proposed openstack/nova master: Adding tags field to InstancePayload https://review.openstack.org/407228 | |
| 16:14:47 | vks1 | vladikr: I have 2 VMs launched on 2 different compute. When I see the tap ports on OVS , on one compute I see tag:1 and on other compute its tag:3 . both the VMs are of same n/w. | |
| 16:14:49 | mdbooth | clarkb: Support for that is always going to be frustrating | |
| 16:14:51 | vladikr | vks1, Hi, let me just read up. | |
| 16:14:59 | kashyap | clarkb: Speaking of attitudes, I'd argue (not the first one) that Virt maintainance on Ubuntu is a joke | |
| 16:15:18 | clarkb | where do I find the list of supported versions of libvirt? | |
| 16:15:27 | mdbooth | clarkb: To be clear... what kashyap said | |
| 16:15:57 | clarkb | https://libvirt.org/downloads.html#maintenance ? | |
| 16:15:57 | openstack | Launchpad bug 1673483 in libvirt (Ubuntu) "libvirt: test_attach_volume_shelved_or_offload_server times out waiting for device detach (which fails)" [Medium,Triaged] | |
| 16:15:57 | kashyap | clarkb: BTW, even for this bug -- https://bugs.launchpad.net/nova/+bug/1673483 | |
| 16:15:59 | mriedem | clarkb: https://wiki.openstack.org/wiki/LibvirtDistroSupportMatrix | |
| 16:16:08 | mriedem | mdbooth: what do you mean we're running on unmaintained infra? | |
| 16:16:14 | mriedem | we're running on 16.04 LTS right? | |
| 16:16:15 | kashyap | clarkb: I had to spend a good half a day talking to upstream libvirt, who identified what exact commits Ubuntu should backport | |
| 16:16:25 | mdbooth | mriedem: Sure. Lets file a bug there. | |
| 16:16:41 | kashyap | So they don't even bother to look at what's breaking, and simply apply the available bug fixes | |
| 16:16:41 | clarkb | kashyap: that bug doesn't seem to be one of the ones we are tracking though? | |
| 16:17:09 | mdbooth | mriedem: I just think it's unreasonable to expect upstream libvirt to spin their wheels on this. | |
| 16:17:10 | kashyap | clarkb: No, it's not. Was just giving an example, that upstream spends more time than is fair (in some cases) with downstream distrubtions | |
| 16:17:18 | clarkb | kashyap: gotcha | |
| 16:17:20 | openstack | Launchpad bug 1673483 in libvirt (Ubuntu) "libvirt: test_attach_volume_shelved_or_offload_server times out waiting for device detach (which fails)" [Medium,Triaged] | |
| 16:17:20 | mriedem | if libvirt devs have identified fixes that are needed in https://bugs.launchpad.net/nova/+bug/1673483 then we can reach out to jamespage and/or coreycb for help i'd think | |
| 16:18:03 | gabor_antal | Sorry for the slightly off topic, I have a question: Some time before I read an article about the effect of the VM workloads which are on the same physical host, but I cannot find it. Can someone please help me with it? | |
| 16:19:09 | clarkb | kashyap: mdbooth is http://wiki.libvirt.org/page/Maintenance_Releases#1.3.3_series what would be considered the most current supported release? | |
| 16:19:32 | kashyap | clarkb: I guess there's a more automated version -- was there one? mriedem? | |
| 16:19:50 | kashyap | (Version of a wiki page I mean) | |
| 16:20:28 | mriedem | i only looked at https://wiki.openstack.org/wiki/LibvirtDistroSupportMatrix | |
| 16:20:29 | clarkb | https://wiki.openstack.org/wiki/LibvirtDistroSupportMatrix shows what linux distros are supporting downstream of libvirt, I'm looking for what upstream libvirt considers supported | |
| 16:20:29 | kashyap | mriedem: Disregard me, please. | |
| 16:20:32 | mriedem | i'm not sure i understand the question | |
| 16:20:37 | mdbooth | clarkb: Well latest stable is 2.2 | |
| 16:20:55 | clarkb | mdbooth: ok so stable and maintenance are different /me keeps reading | |
| 16:20:59 | mdbooth | clarkb: I'm guessing that in practise backporting to 1.3 is best effort | |
| 16:21:34 | vladikr | vks1, hmm, I think this is a more of a question for the neutron team. We are only configuring vlan tags for SRIOV devices. otherwise, I think it's neutron it setup up the environment | |
| 16:22:18 | vks1 | vladikr: ok thanks | |
| 16:23:16 | kashyap | clarkb: Yes, you're right. (On stable / maint releases) | |
| 16:23:59 | clarkb | https://libvirt.org/downloads.html#numbering says since 2.0 you can't use the version numbers to know what is stable | |
| 16:24:04 | kashyap | clarkb: Tomorrow morning I'll check with upstream if we can get to the bottom of the issue with whatever debugging info we have as of now | |
| 16:24:06 | clarkb | so I'm trying to figure out how I can determine that | |
| 16:24:36 | clarkb | there are maintenance releases which I think are also stable but there may be newer stables that are not maintenance | |
| 16:26:58 | kashyap | clarkb: More about those maintainance releases: | |
| 16:26:59 | kashyap | "These maintenance branches should only contain bug fixes, and no new features, backported from the master branch, and are supported as long as at least one downstream distribution expresses interest in a given branch. These maintenance branches are considered during CVE analysis. In contrast to the primary releases which are made once a month, there is no formal schedule for the maintenance | |
| 16:27:05 | kashyap | releases, which are made whenever there is a need to make available key bugfixes to downstream consumers." | |
| 16:28:32 | jamespage | mriedem: looking | |
| 16:28:38 | clarkb | right so two things I am trying to figure out here 1) if I was ubunut (I am not) how would I properly select the version of libvirt to include in the next version of a distro release? 2) if I'm openstack infra and want to attempt to reproduce a problem on a supported stable version of libvirt where do I find such a version of libvirt? | |
| 16:28:41 | clarkb | kashyap: ^ | |
| 16:28:52 | clarkb | kashyap: and unfortunately I am not really get a clear answer for either thing | |
| 16:28:52 | openstackgerrit | Anusha Unnam proposed openstack/nova master: Add description to policies in services.py https://review.openstack.org/449782 | |
| 16:29:42 | clarkb | is using the newest maintenace release the answer to both questions? | |
| 16:29:43 | kashyap | clarkb: Can't tell you without really digging a bit closer. | |
| 16:30:01 | mdbooth | clarkb: Anyway, if we want to keep running an old release, we need downstream support for that. My experience of upstream libvirt is that when I've given them something concrete they've been extremely helpful. However, we can't expect them to go on a huge fishing trip for an old downstream release. | |
| 16:30:15 | kashyap | Yeah, precisely. | |
| 16:30:23 | clarkb | mdbooth: yes, which is why I'm trying to find the anwer of what they will support and work with us on | |