| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-01-31 | |||
| 12:07:44 | lyarwood | sfinucan: yeah we do make exceptions for simple fixes, I'm just going by the bug subject and Importance posted in the channel | |
| 12:10:00 | openstack | Launchpad bug 1643911 in OpenStack Compute (nova) "libvirt randomly crashes on xenial nodes with "*** Error in `/usr/sbin/libvirtd': malloc(): memory corruption:"" [Medium,Confirmed] | |
| 12:10:00 | kashyap | Hmm, this has become a bit of blocker it seems -- https://bugs.launchpad.net/bugs/1643911 | |
| 12:10:57 | kashyap | And the upstream libvirt folks go: "the stack trace will be totally useless for malloc memory corruption" | |
| 12:12:19 | openstack | Launchpad bug 1643911 in OpenStack Compute (nova) "libvirt randomly crashes on xenial nodes with "*** Error in `/usr/sbin/libvirtd': malloc(): memory corruption:"" [Medium,Confirmed] | |
| 12:12:19 | kashyap | Because: https://bugs.launchpad.net/nova/+bug/1643911/comments/1 | |
| 12:12:37 | kashyap | If you run libvirtd under valgrind in the Gate, then it might hide the problem | |
| 12:13:02 | kashyap | At least libvirt upstream admits: "I mean, we [libvirt] are clearly overwriting the memory somewhere; but stack trace is not helpful in those cases" | |
| 12:32:24 | openstackgerrit | Ghanshyam Mann proposed openstack/nova: Add release note and docs for filter/sort whitelist https://review.openstack.org/421760 | |
| 12:37:54 | openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/nova: Optimize time during PCI alias parsing process https://review.openstack.org/427145 | |
| 13:07:26 | openstackgerrit | Emilien Macchi proposed openstack/nova: DNM - tripleo CI test https://review.openstack.org/426949 | |
| 13:13:19 | cdent | johnthetubaguy, andymccr: Did you guys work out what was going on a while back with the placement db or are things still weird? | |
| 13:13:48 | cdent | andymccr: I'm pretty sure the original 409 that is happening is because the generation on the resource provider is run, but it corrects itself so the next writes out to work | |
| 13:13:53 | cdent | s/out/ought/ | |
| 13:14:06 | cdent | (they do for me when I've been testing things in devstack) | |
| 13:14:24 | cdent | so your theory that it is a database problem is likely | |
| 13:17:19 | johnthetubaguy | cdent: I think the DB wasn't configured correctly | |
| 13:17:34 | johnthetubaguy | cdent: but I could be wrong, not seen andymccr latest results | |
| 13:18:02 | johnthetubaguy | cdent: I forgot we didn't merge the stuff to support the separate placement_database | |
| 13:18:59 | andymccr | johnthetubaguy: cdent - thanks for the info :) I'm testing some more things with database changes/perms etc - will update when i (hopefully) have something working! | |
| 13:19:03 | cdent | yeah, it totally stalled out. I have a working change that I keep up to date for it, but people stated unspecific reservations about it, without specifics there was no way to move on... | |
| 13:19:54 | johnthetubaguy | cdent: jaypipes: I still had worries about the allocations in that ironic patch, I added comments/suggestions | |
| 13:20:04 | cdent | johnthetubaguy: that's next | |
| 13:20:26 | johnthetubaguy | cool, thanks | |
| 13:20:28 | cdent | I skimmed your comments earlier today and my immediate thought was "moar tests" | |
| 13:21:12 | markus_z | mriedem: Would you please remove/re-evaluate your -1 on https://review.openstack.org/#/c/406897/ ? | |
| 13:22:40 | openstackgerrit | Andy McCrae proposed openstack/nova: Allow placement endpoint interface to be set https://review.openstack.org/426163 | |
| 13:23:06 | openstackgerrit | Sylvain Bauza proposed openstack/nova: Add a status check for enabled filters https://review.openstack.org/427200 | |
| 13:24:47 | openstack | Launchpad bug 1660317 in ironic (Ubuntu) "NotImplementedError for detach_interface in nova-compute during instance deletion" [Undecided,New] | |
| 13:24:47 | sfinucan | Kevin_Zheng: Could you take a look at https://bugs.launchpad.net/nova/+bug/1660317 ? fwict, the error message that they're receiving should be mitigated by a patch of yours | |
| 13:25:01 | sfinucan | Namely, this patch https://github.com/openstack/nova/commit/4ad414f3b1216393301ef268a64e61ca1a3d5be9 | |
| 13:25:15 | sfinucan | ...though I could be reading that incorrectly :) | |
| 13:26:02 | mriedem | sfinucan: have you seen https://review.openstack.org/#/c/403925/ ? | |
| 13:26:18 | mriedem | markus_z: removed the -1 | |
| 13:26:21 | mriedem | until i can dig deeper | |
| 13:26:44 | markus_z | mriedem: Thanks | |
| 13:27:26 | sfinucan | mriedem: I had not. I can take a look now | |
| 13:28:18 | Kevin_Zheng | sfinucan: sorry, I'm on vacation and didn't got my computer with me:( | |
| 13:28:30 | mriedem | sfinucan: just wondering if it's related to what you're asking about | |
| 13:28:34 | bauzas | mriedem: FYI https://review.openstack.org/427200 | |
| 13:28:52 | sfinucan | Kevin_Zheng: No problem at all - it's not that important. Enjoy your vacation :) | |
| 13:29:16 | sfinucan | mriedem: I'm looking but I don't think so | |
| 13:29:28 | Kevin_Zheng | Thanks | |
| 13:32:29 | openstackgerrit | OpenStack Proposal Bot proposed openstack/os-vif: Updated from global requirements https://review.openstack.org/395347 | |
| 13:37:20 | mriedem | toabctl: +2 on https://review.openstack.org/#/c/402318/ with some comments - the libvirt driver ignores the requested device name that tempest is sending in anyway | |
| 13:37:46 | mriedem | so if the libvirt driver was always defaulting to vda i'm not sure how things worked with the xenproject ci | |
| 13:38:35 | mriedem | doesn't look like the libvirt+xen ci overrides that in tempest.conf either http://logs.openstack.xenproject.org/18/402318/1/check/dsvm-tempest-xen/6b9829a/logs/tempest_conf.txt.gz | |
| 13:39:44 | mriedem | anthonyper: ^ | |
| 13:39:47 | toabctl | mriedem, is libvirt+xen tested? | |
| 13:39:54 | mriedem | yes | |
| 13:39:57 | mriedem | that's the xenproject ci | |
| 13:41:00 | mriedem | 2016-11-25 08:10:13.591 13563 WARNING nova.virt.libvirt.driver [req-98364dbb-0b80-49f3-9d59-ac476fdd3855 tempest-AttachVolumeShelveTestJSON-134397458 tempest-AttachVolumeShelveTestJSON-134397458] [instance: b0cf5c0f-ae34-4ca1-b441-08a3f4a85b71] Ignoring supplied device name: /dev/vdb | |
| 13:41:00 | mriedem | e.g. | |
| 13:45:48 | mriedem | toabctl: dropped to a +1 until someone that knows about the xenproject ci can explain why this wasn't a failure before | |
| 13:46:56 | mriedem | maybe it's just because that ci wasn't running the test in question, test_volume_boot_pattern | |
| 13:47:37 | mriedem | anthonyper: is the xenproject ci whitelist/blacklist available somewhere? i'm wondering if i could enable test_volume_boot_pattern against https://review.openstack.org/#/c/402318/ | |
| 13:47:43 | BobBall | DEVSTACK_GATE_TEMPEST_REGEX='(?!.*\[.*\bslow\b.*\]|.*TestVolumeBootPattern\.test_(volume_boot_pattern|create <-- confirmed | |
| 13:48:08 | mriedem | where is that defined though? | |
| 13:48:36 | BobBall | Defined here http://xenbits.xen.org/gitweb/?p=openstack/ci-loop-config.git;a=blob;f=jenkins/jobs/jobs.yaml;h=a1a29f4b0fdaa4730c842736ee7e2ae4e49d06da;hb=HEAD | |
| 13:48:44 | BobBall | which is not easily testable | |
| 13:48:45 | toabctl | mriedem, yes. the test is not executed. | |
| 13:48:58 | toabctl | but then I should reenable the test with my changeset | |
| 13:49:17 | openstackgerrit | Andy McCrae proposed openstack/nova: Allow placement endpoint interface to be set https://review.openstack.org/426163 | |
| 13:49:33 | mriedem | BobBall: ah so i can't use a depends-on like the xenserver ci | |
| 13:49:43 | toabctl | oh. it's on the external CI. so I can not simply change that | |
| 13:49:53 | mriedem | right | |
| 13:50:08 | mriedem | BobBall: we could move that blacklist into the nova tree... | |
| 13:50:09 | toabctl | so I did test that manually on our internal CI... | |
| 13:50:11 | mriedem | then it's branch-specific | |
| 13:50:13 | BobBall | Sorry mriedem, it's not... | |
| 13:50:28 | mriedem | BobBall: like this https://github.com/openstack/nova/blob/master/devstack/tempest-dsvm-lxc-rc | |
| 13:51:04 | mriedem | then that job definition just needs to use the DEVSTACK_GATE_TEMPEST_REGEX variable | |
| 13:51:32 | BobBall | That's nice, but curious because clearly it's not *just* nova implicated with some of the tempest tests - for example there have historically been some tempest tests that only work with libvirt | |
| 13:52:09 | BobBall | i.e. the tests make assumptions about the hypervisor e.g. that it can support qcow2 images made in the test themselves | |
| 13:53:08 | mriedem | right so the tempest defaults are based on the nova defaults, which are libvirt | |
| 13:53:19 | mriedem | if you're not using those defaults, you have to configure stufff | |
| 13:53:34 | mriedem | that doesn't mean we couldn't control the xenproject ci blacklist from within the nova tree | |
| 13:53:42 | BobBall | Indeed - but the case I was thinking about wasn't configurable | |
| 13:55:17 | mriedem | bauzas: rechecked your scheduler stuff, the grenade patch merged before cells v1 broke, but the cells v1 thing is fixed now so all of the rechecks are running | |
| 13:55:36 | bauzas | mriedem: cool, thanks for following that | |
| 13:57:30 | mriedem | oomichi: sdague: johnthetubaguy: want to take a crack at this? i've got a +2 on it https://review.openstack.org/#/c/424759/ | |
| 13:57:43 | mriedem | and the tempest change is approved https://review.openstack.org/#/c/426991/ | |
| 13:58:58 | johnthetubaguy | mriedem: talking a look now, would be good to get that in | |
| 13:59:16 | johnthetubaguy | mriedem: there was a spec to match it too, but thats more a documentation thing at this point I guess? | |
| 13:59:35 | tdurakov | PaulMurray paul-carlton2 eliqiao shaohe_feng pkoniszewski yuntongjin mdbooth kashyap kmartin claudiub tobasco tdurakov abhishekk mrhillsman raj_singh wznoinsk siva_krishnan hi, live-migration subteam meeting in a minute on #openstack-meeting-3 | |
| 14:10:55 | mriedem | johnthetubaguy: the spec was merged yesterday | |
| 14:11:00 | mriedem | johnthetubaguy: the spec was a normal spec | |
| 14:11:13 | mriedem | since it's an api change and we require a spec for api changes | |
| 14:11:23 | johnthetubaguy | mriedem: yep, found that, all good | |
| 14:14:02 | edleafe | cdent: about to get sucked into 4 hours of meetings, but for "moar tests", you could start adding on to https://review.openstack.org/#/c/404472/28/nova/tests/functional/api/openstack/placement/gabbits/ironic-inventory.yaml | |
| 14:14:31 | edleafe | It's just the barest of tests now, but I started it with "moar" in mind | |
| 14:14:32 | cdent | edleafe: reviewing that stuff now, will look | |
| 14:19:47 | cdent | edleafe: I think we need functional tests on the resource tracker side, not the gabbi side? | |
| 14:24:03 | sean-k-mooney | mriedem: regarding os-vif no i dont think there is anything that is outstanding that needs to merge before creating stable/ocata | |
| 14:24:40 | edleafe | cdent: ah, ok. I just had time to briefly read the scrollback. | |
| 14:25:16 | cdent | edleafe: I'll see if I can come up with something. At least think on it. | |
| 14:39:36 | dansmith | hmm, the scheduler patches are still not through the gate? | |
| 14:45:10 | derekh | could somebody put a extra pair of eyes on https://review.openstack.org/#/c/417227/ , its holding back another patch I'm trying to fix | |
| 14:58:47 | johnthetubaguy | dansmith: I think its that nasty gate bug fix that hadn't merged, when I last checked | |
| 14:59:02 | dansmith | which one? is it approved? | |