Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-03-29
16:41:41 mriedem jamespage: and not sure we have a recreate outside of CI
16:41:43 mriedem would have to ask kashyap
16:42:22 clarkb mriedem: jamespage: it should be pretty simple to have devstack wget && dpkg -i at the very least, but maybe also add a ppa and then let normal install of libvirt pull from there
16:42:48 mriedem mdbooth: lyarwood: there is a new lock added to the refactor here https://review.openstack.org/#/c/439520/
16:43:09 openstack Launchpad bug 1643911 in OpenStack Compute (nova) "libvirt randomly crashes on xenial nodes with "*** Error in `/usr/sbin/libvirtd': malloc(): memory corruption:"" [Medium,Confirmed]
16:43:09 kashyap jamespage: If you're referring to the fixes from 1673483, just wrote this comment - https://bugs.launchpad.net/nova/+bug/1643911/comments/19
16:43:13 mriedem mdbooth: lyarwood: i'm really not happy about adding new things to an already large and not easy refactor change
16:43:19 mriedem mdbooth: lyarwood: stuff like that should be separated
16:44:13 mdbooth mriedem: Actually this version of the refactor is pretty trivial: it's just code motion
16:44:19 clarkb I think dmsimard was trying to track down filesystem problems on centos7 too. I should swing back around and find out what happened wit hthat
16:44:27 mdbooth The initial version did more than this
16:44:36 mdbooth The lock doesn't need to be there, sure
16:44:59 mriedem mdbooth: slipping things into a refactor is not cool, period
16:45:01 openstack bug 1673483 in libvirt (Ubuntu) "libvirt: test_attach_volume_shelved_or_offload_server times out waiting for device detach (which fails)" [Medium,Triaged] https://launchpad.net/bugs/1673483
16:45:01 jamespage kashyap: yes I was referring to bug 1673483
16:45:22 mriedem mdbooth: as i've already noted in the review, even with just code motion, there are issues,
16:45:25 mriedem which makes stuff like this hard to review
16:45:27 kashyap jamespage: They're both are not same -- we haven't definitively identified so
16:45:29 clarkb kashyap: if we can get an answer to what an appropriate version of libvirt to repreoduce with is (maybe that 1.3.3 maintenance release) I would be happy to help get that in place on devstack test nodes
16:45:31 mriedem because it's hard to miss differences
16:46:03 openstack bug 1673483 in libvirt (Ubuntu) "libvirt: test_attach_volume_shelved_or_offload_server times out waiting for device detach (which fails)" [Medium,Triaged] https://launchpad.net/bugs/1673483
16:46:03 jamespage not sure I said they where - just wanted to see if we could get some validation of the fixes for bug 1673483
16:46:23 kashyap jamespage: Because, the Nova stack traces for both bugs are different
16:46:28 clarkb jamespage: fwiw we also have http://status.openstack.org/elastic-recheck/index.html#1643911 http://status.openstack.org/elastic-recheck/index.html#1638982 http://status.openstack.org/elastic-recheck/index.html#1646779 which are our top failures according to elastic-recheck
16:46:32 mdbooth mriedem: Re the **kwargs thing, I agree btw. However, it is consistent with attach, and that's the primary goal here.
16:46:45 kashyap clarkb: The buggy version should surely be easily identified - let me look
16:47:14 clarkb jamespage: all three are marked new against libvirt in launchpad
16:47:47 mriedem mdbooth: being consistent with a bad idea is not reasonable justification to continue doing it
16:47:48 mriedem imo
16:49:25 clarkb kashyap: I can find the version we are using (that is buggy) easily. I am trying to find the version that would be an acceptable reproduction platform in order to push back to upstream (and maybe we find its a non issue there and we bisect and try to find root cause)
16:50:08 kashyap clarkb: Acceptable reproduction platform would be: the same test case be run on (a) upstream current stable; or (b) Git master
16:50:42 clarkb kashyap: yes what is upstream current stable?
16:50:48 clarkb that is the question :)
16:50:50 kashyap 1 se
16:50:51 kashyap 1 sec
16:51:05 kashyap v3.1.0 (2017-03-03)
16:51:14 clarkb the downloads page says you can't base "stable" on versions anymore and I can't find anything that says what is stable
16:51:17 kashyap That's the latest released, though
16:51:26 clarkb right that isn't necessarily latest stable according to the docs
16:51:35 clarkb since versions are time based now
16:51:52 kashyap clarkb: Yes, you can get a sense of WTF the release numbering means: http://libvirt.org/downloads.html#numbering
16:52:02 kashyap "Prior to 2.0.0, the major/minor numbers were incremented fairly arbitrarily, and maintenance releases appended a fourth digit."
16:52:13 clarkb kashyap: right and what that says is nothing in a version implies stability
16:52:19 clarkb so where do I find the list of what is stable?
16:52:59 clarkb maybe I need more caffeine but nothing seems to actually say. Which amkes me sympathize with ubuntu
16:53:02 kashyap clarkb: In context of upstream distributions (non-RHEL), I'd interpret "stable" as whatever is currently released. I know that's a "loose" definition
16:53:44 kashyap Since the "maintainance" release in context of libvirt are only for bug / CVE fixes.
16:54:37 clarkb oh hey 3.1.0 is what I already have on tumbleweed cool
16:55:22 kashyap Cool
16:55:54 kashyap clarkb: It seems to happen during an instance termination (with Nova's "ImagesTestJSON"), and libvirtd dying subsequently.
16:56:31 openstack Launchpad bug 1673483 in libvirt (Ubuntu) "libvirt: test_attach_volume_shelved_or_offload_server times out waiting for device detach (which fails)" [Medium,Triaged]
16:56:31 mriedem clarkb: sorry if you guys already talked about this, but in https://bugs.launchpad.net/nova/+bug/1673483 there are comments from ubuntu devs that backports have been made to some packages and are available for testing,
16:56:34 openstackgerrit Dan Smith proposed openstack/nova master: Teach HostAPI about cells https://review.openstack.org/442162
16:56:35 openstackgerrit Dan Smith proposed openstack/nova master: Make scheduler target cells to get compute node instance info https://review.openstack.org/439891
16:56:36 openstackgerrit Dan Smith proposed openstack/nova master: Sort CellMappingList.get_all() for safety https://review.openstack.org/443174
16:56:37 openstackgerrit Dan Smith proposed openstack/nova master: Clean up ClientRouter debt https://review.openstack.org/444487
16:56:37 openstackgerrit Dan Smith proposed openstack/nova master: Add get_by_instance_uuids() to InstanceMappingList https://review.openstack.org/443292
16:56:38 openstackgerrit Dan Smith proposed openstack/nova master: Make server_groups determine deleted-ness from InstanceMappingList https://review.openstack.org/443293
16:56:39 openstackgerrit Dan Smith proposed openstack/nova master: Remove Mitaka-era service version check https://review.openstack.org/442861
16:56:41 openstackgerrit Dan Smith proposed openstack/nova master: Add workaround to disable group policy check upcall https://review.openstack.org/442736
16:56:42 mriedem clarkb: it's not clear to me if we have a way to hook testing packages in our CI?
16:56:45 clarkb mriedem: yes but that bug isn't a problem
16:56:54 clarkb mriedem: we have 3 other libvirt bugs that are actually major issues
16:57:09 clarkb mriedem: and I mentioned it would be simple to modify devstack to use a ppa or just wget and dpkg -i
16:57:45 mriedem ok, and we don't have fixes identified to backport for 1.3.3 in xenial LTS for testing yet
16:58:03 jamespage clarkb: can I get context please on those three bugs
16:58:14 jamespage refs would be ok for now
16:58:28 mriedem jamespage: https://bugs.launchpad.net/nova/+bug/1643911/
16:58:28 openstack bug 1643911 in OpenStack Compute (nova) "libvirt randomly crashes on xenial nodes with "*** Error in `/usr/sbin/libvirtd': malloc(): memory corruption:"" [Medium,Confirmed] https://launchpad.net/bugs/1643911
16:58:28 jamespage guessing one is bug 1643911
16:58:45 clarkb jamespage: its the three elastic recheck links I gave you above
16:58:49 clarkb jamespage: they have links to launchpad
16:58:54 jamespage clarkb: ta
16:59:18 mriedem jamespage: some info in http://lists.openstack.org/pipermail/openstack-dev/2017-March/114648.html too
16:59:31 mdbooth dansmith: https://review.openstack.org/#/c/443292/ https://review.openstack.org/#/c/445139/ :)
17:01:32 openstackgerrit Anusha Unnam proposed openstack/nova master: Add description to policies in servers_migrations.py https://review.openstack.org/451487
17:02:39 mdbooth Absolutely nothing! To be expected, I guess as it's pretty much boilerplate, but it's funny they're character identical. My test is a bit longer.
17:03:05 mriedem mdbooth: it's not the size of the patch, but the motion on the ocean
17:03:14 mriedem that matters that is
17:06:15 mdbooth mriedem: I've just been back through that refactor, btw.
17:06:41 mdbooth I had also glossed over the poll thing, having previously seen it when johnthetubaguy spotted it :)
17:37:33 mriedem sfinucan: you probably have to take this over https://review.openstack.org/#/c/396289/
17:41:53 openstackgerrit Gergely Csatari proposed openstack/nova master: api-ref: adminPass is optional in Create Server https://review.openstack.org/451497
18:01:38 mriedem dumb question: what config option defines if the scheduler is packing or spreading?
18:02:39 mriedem ram_weight_multiplier ?
18:03:08 mriedem dansmith: melwitt: didn't we just talk about something with the default options being weird?
18:03:16 melwitt the subset_size right
18:03:39 melwitt host_subset_size
18:03:57 melwitt if that's 1, it will pack
18:04:07 mriedem or host_subset_size ?
18:04:08 mriedem yeah
18:04:20 mriedem ok and 1 is the default
18:04:29 melwitt right
18:07:29 mriedem were we really thinking about creating a new claims resource in placement for claims in the scheduler? or just re-use allocations?
18:07:44 mriedem creating a new claims resource and ticket system really sounds like the old style quota system to me, with the reservations table
18:07:49 mriedem i would think we want to avoid redoing that
18:07:50 mriedem https://review.openstack.org/#/c/437424/
18:07:57 mriedem jaypipes: ^ would you agree?
18:17:16 edleafe mriedem: the sticking point is how to keep track of groups of allocations as a unit
18:17:58 edleafe I don't believe that that's necessary

Earlier   Later