| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-03-21 | |||
| 16:01:02 | jroll | bauzas: yep. we don't want a flavor to appear that suddenly can claim a node with an instance already on it | |
| 16:01:08 | jroll | though, thinking back to this morning | |
| 16:01:39 | jroll | if we make a change that makes nova claim all resources for ironic nodes with an instance... we don't need this extra migration | |
| 16:03:11 | bauzas | jroll: graaah, just entering a team meeting, need to defer my thinkins for that :( | |
| 16:03:17 | bauzas | thinkings* | |
| 16:03:22 | jroll | bauzas: it's fine, no worries | |
| 16:05:40 | jamespage | mriedem: hello | |
| 16:05:48 | jamespage | mriedem: would you have time to chat about https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1665698 | |
| 16:05:48 | openstack | Launchpad bug 1665698 in libvirt (Ubuntu Yakkety) "/etc/qemu-ifup not allowed by apparmor" [Medium,Triaged] | |
| 16:06:36 | mriedem | jamespage: ok, but i have no context on that. | |
| 16:06:48 | jamespage | mriedem: let me give you some :-) | |
| 16:06:53 | jamespage | mriedem: ok so... | |
| 16:06:56 | jamespage | mriedem: some history | |
| 16:07:04 | mriedem | this is the empty script path element thing? | |
| 16:07:11 | mriedem | i saw there was a revert proposed for that | |
| 16:07:18 | mriedem | sgordon: ^ | |
| 16:07:25 | jamespage | mriedem: that's the one | |
| 16:07:32 | jamespage | mriedem: the fix for bug 1649527 | |
| 16:07:32 | openstack | bug 1649527 in OpenStack Compute (nova) newton "nova creates an invalid ethernet/bridge interface definition in virsh xml" [Medium,Fix committed] https://launchpad.net/bugs/1649527 - Assigned to Roman Podoliaka (rpodolyaka) | |
| 16:08:17 | jamespage | appears to have broken baseline compatibility for libvirt so for certain vif types, xenial libvirt + newton stable is broken. | |
| 16:08:23 | bauzas | jroll: one other thinking while attending my meeting, if we rebuild an instance having a modified nested flavor, we could just put it to a Pike node then, right? | |
| 16:08:54 | openstackgerrit | Matt Riedemann proposed openstack/nova-specs master: add spec for instance flavor API https://review.openstack.org/265282 | |
| 16:08:57 | bauzas | jroll: to make it clear, I'm not trying to punt your spec, just trying to think about all the possible move operations | |
| 16:09:44 | jroll | bauzas: yeah, I know what you're doing, I'm not upset about it | |
| 16:09:57 | bauzas | :) | |
| 16:10:18 | bauzas | I'm just an overthinking person :( | |
| 16:10:18 | jroll | bauzas: I would need to read rebuild code to answer that question, but I assume we use instance.flavor for that, which means the scheduling decision would be different. interesting. | |
| 16:10:35 | jroll | bauzas: "pike node" means a node with a resource class set, yes? | |
| 16:10:46 | mriedem | jamespage: ok it's confusing because of comments like this in the original bug report https://bugs.launchpad.net/nova/+bug/1649527/comments/19 | |
| 16:10:46 | openstack | Launchpad bug 1649527 in OpenStack Compute (nova) newton "nova creates an invalid ethernet/bridge interface definition in virsh xml" [Medium,Fix committed] - Assigned to Roman Podoliaka (rpodolyaka) | |
| 16:11:00 | bauzas | jroll: a node that supports that custom RC | |
| 16:11:08 | mriedem | jamespage: also though libvirt 2.5.0 isn't base xenial | |
| 16:11:13 | mriedem | base xenial libvirt is 1.3.1 right? | |
| 16:11:14 | jroll | bauzas: right, yeah, it would be scheduled that way | |
| 16:11:19 | jamespage | mriedem: nope - its 1.3.1 | |
| 16:11:28 | jamespage | mriedem: that was a somewhat unhelpful comment | |
| 16:11:37 | jroll | bauzas: I'm thinking if we can do the "allocate all resources" thing outside of instance.flavor, we don't need to do this instance.flavor migration | |
| 16:11:55 | jamespage | mriedem: I found the bug history quite confusing as well | |
| 16:12:24 | bauzas | jroll: yeah I tend to agree with you although there could be some corner cases | |
| 16:13:12 | mriedem | jamespage: yeah they later clarified they were running in a container with debian so different versions :/ | |
| 16:13:36 | bauzas | jroll: I'm just afraid of us possibly needing to do like we made for Ocata and just having the scheduler calling placement about custom RCs only if all the nodes are pike | |
| 16:13:37 | mriedem | jamespage: so i guess this didn't affect rhel 7.3 or centos because they are running with a newer libvirt which doesn't cause issues that 1.3.1 does | |
| 16:14:06 | mriedem | jamespage: this isn't just newton right? i mean it would also be broken in ocata wouldn't it? | |
| 16:14:15 | mriedem | i see the backport was merged to newton though https://review.openstack.org/#/c/425637/ | |
| 16:14:28 | mriedem | and we didn't raise the minimum required libvirt in nova in ocata | |
| 16:15:28 | jroll | bauzas: yeah, I see now, because we upgrade scheduler before compute, right? | |
| 16:15:40 | bauzas | jroll: yup | |
| 16:16:05 | jroll | bauzas: which means operators must wait until after the pike deploy to change their flavors. which isn't ideal but I think it's fine? | |
| 16:16:26 | bauzas | jroll: hum no | |
| 16:16:38 | bauzas | jroll: operators can modify their flavors in Ocata | |
| 16:16:52 | bauzas | given it's extra specs, it's just meaningless until Pike, that's it | |
| 16:17:07 | jroll | bauzas: pike scheduler + ocata compute means scheduling on new style flavors would be broken | |
| 16:17:22 | jroll | because we won't be reading resource class | |
| 16:18:03 | bauzas | jroll: yeah the worst case scenario would be all but computes upgraded to Pike, and n-cpus still on Ocata | |
| 16:18:14 | bauzas | jroll: then, if flavors are upgraded, kaboom | |
| 16:18:20 | bauzas | NoValidHosts | |
| 16:18:24 | mriedem | jamespage: so if i'm understanding this correctly, between https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1665698/comments/50 and https://review.openstack.org/#/c/425637/1/releasenotes/notes/libvirt-script-with-empty-path-2b49caa68b05278d.yaml | |
| 16:18:24 | openstack | Launchpad bug 1665698 in libvirt (Ubuntu Yakkety) "/etc/qemu-ifup not allowed by apparmor" [Medium,Triaged] | |
| 16:18:44 | mriedem | jamespage: we want script path to be '' if libvirt < 1.3.3, and None if libvirt >= 1.3.3 yes? | |
| 16:18:51 | jroll | bauzas: right, which I guess is why I need reviews on this, too :) https://etherpad.openstack.org/p/nova-ironic-flavor-transition-plan | |
| 16:19:04 | bauzas | oh, never heard of that yet ^ | |
| 16:19:13 | jroll | bauzas: read more scheduler meeting :P | |
| 16:19:29 | jroll | bauzas: but yeah, I guess my spec kind of depends on this plan being solid | |
| 16:19:49 | bauzas | jroll: heh, not my fault, but my kid's one | |
| 16:19:58 | jroll | bauzas: we log for a reason! | |
| 16:20:01 | jroll | I'm mostly kidding | |
| 16:20:24 | jamespage | mriedem: correct | |
| 16:20:27 | bauzas | argh, no good fake reason to provide, I'm doomed | |
| 16:21:00 | mriedem | jamespage: ok, i'll tinker with seeing how easy a fix that is | |
| 16:21:15 | jamespage | mriedem: thankyou much appreciated | |
| 16:22:51 | jroll | bauzas: line 39-42 there is me walking through rebuild in my head, I think it's okay | |
| 16:27:39 | sgordon | mriedem, jamespage mmm yes | |
| 16:27:56 | sgordon | mriedem, jamespage revert of what specifically? | |
| 16:28:10 | sgordon | mriedem, jamespage libvirt has actually been fixed to handle the old xml more gracefully | |
| 16:28:22 | mriedem | sgordon: https://review.openstack.org/#/c/447169/ | |
| 16:28:46 | mriedem | sgordon: yes but according to https://bugs.launchpad.net/nova/+bug/1665698/comments/50 that's in libvirt 3.1 | |
| 16:28:46 | openstack | Launchpad bug 1665698 in libvirt (Ubuntu Yakkety) "/etc/qemu-ifup not allowed by apparmor" [Medium,Triaged] | |
| 16:28:46 | jamespage | sgordon: yeah we're all good on forward version compat I think | |
| 16:28:54 | mriedem | which is lightyears from where we'll be testing against in the gate | |
| 16:29:09 | jamespage | its backwards compat with 1.3.1 in Ubuntu Xenial | |
| 16:29:10 | sgordon | mriedem, yes so the distributor should backport it | |
| 16:29:29 | jamespage | backport what exactly? | |
| 16:29:35 | mriedem | the '' or None fix i guess | |
| 16:29:41 | mriedem | but what about someone not using a distro package? | |
| 16:29:48 | jamespage | nova has established min version compats | |
| 16:29:48 | sgordon | jamespage, the libvirt fix | |
| 16:29:49 | mriedem | or using not red hat or ubuntu distro package? | |
| 16:29:56 | mriedem | like suse | |
| 16:30:04 | mriedem | gentoo, etc | |
| 16:30:20 | sgordon | jamespage, yes but you obvs bumped the distro libvirt version at some point or you would not be hitting the issue | |
| 16:30:30 | mriedem | anyway, let me see how hard the conditional version check is in the code before we talk about making packagers handle this :) | |
| 16:30:34 | sgordon | jamespage, which is also why it only appeared on RHEL/CentOS 7.3 | |
| 16:30:59 | jamespage | sgordon: the libvirt version in xenial has been 1.3.1 since 16.04 release last year | |
| 16:31:03 | mriedem | ubuntu has libvirt 1.3.1 in stock 16.04 | |
| 16:31:07 | mriedem | that's what we test with in the gate | |
| 16:31:17 | mriedem | we just don't test with the same vif types that this hits | |
| 16:31:45 | sgordon | jamespage, the problem was only introduced in libvirt 1.3.2 | |
| 16:31:54 | sgordon | jamespage, so at some point it's been brought in via a backport | |
| 16:33:13 | mriedem | sgordon: why does the release note, that i think you wrote, say 1.3.3? https://review.openstack.org/#/c/411936/6/releasenotes/notes/libvirt-script-with-empty-path-2b49caa68b05278d.yaml | |
| 16:33:34 | jamespage | behaviour changed in 1.3.3 | |