Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-03-29
16:45:31 mriedem because it's hard to miss differences
16:46:03 openstack bug 1673483 in libvirt (Ubuntu) "libvirt: test_attach_volume_shelved_or_offload_server times out waiting for device detach (which fails)" [Medium,Triaged] https://launchpad.net/bugs/1673483
16:46:03 jamespage not sure I said they where - just wanted to see if we could get some validation of the fixes for bug 1673483
16:46:23 kashyap jamespage: Because, the Nova stack traces for both bugs are different
16:46:28 clarkb jamespage: fwiw we also have http://status.openstack.org/elastic-recheck/index.html#1643911 http://status.openstack.org/elastic-recheck/index.html#1638982 http://status.openstack.org/elastic-recheck/index.html#1646779 which are our top failures according to elastic-recheck
16:46:32 mdbooth mriedem: Re the **kwargs thing, I agree btw. However, it is consistent with attach, and that's the primary goal here.
16:46:45 kashyap clarkb: The buggy version should surely be easily identified - let me look
16:47:14 clarkb jamespage: all three are marked new against libvirt in launchpad
16:47:47 mriedem mdbooth: being consistent with a bad idea is not reasonable justification to continue doing it
16:47:48 mriedem imo
16:49:25 clarkb kashyap: I can find the version we are using (that is buggy) easily. I am trying to find the version that would be an acceptable reproduction platform in order to push back to upstream (and maybe we find its a non issue there and we bisect and try to find root cause)
16:50:08 kashyap clarkb: Acceptable reproduction platform would be: the same test case be run on (a) upstream current stable; or (b) Git master
16:50:42 clarkb kashyap: yes what is upstream current stable?
16:50:48 clarkb that is the question :)
16:50:50 kashyap 1 se
16:50:51 kashyap 1 sec
16:51:05 kashyap v3.1.0 (2017-03-03)
16:51:14 clarkb the downloads page says you can't base "stable" on versions anymore and I can't find anything that says what is stable
16:51:17 kashyap That's the latest released, though
16:51:26 clarkb right that isn't necessarily latest stable according to the docs
16:51:35 clarkb since versions are time based now
16:51:52 kashyap clarkb: Yes, you can get a sense of WTF the release numbering means: http://libvirt.org/downloads.html#numbering
16:52:02 kashyap "Prior to 2.0.0, the major/minor numbers were incremented fairly arbitrarily, and maintenance releases appended a fourth digit."
16:52:13 clarkb kashyap: right and what that says is nothing in a version implies stability
16:52:19 clarkb so where do I find the list of what is stable?
16:52:59 clarkb maybe I need more caffeine but nothing seems to actually say. Which amkes me sympathize with ubuntu
16:53:02 kashyap clarkb: In context of upstream distributions (non-RHEL), I'd interpret "stable" as whatever is currently released. I know that's a "loose" definition
16:53:44 kashyap Since the "maintainance" release in context of libvirt are only for bug / CVE fixes.
16:54:37 clarkb oh hey 3.1.0 is what I already have on tumbleweed cool
16:55:22 kashyap Cool
16:55:54 kashyap clarkb: It seems to happen during an instance termination (with Nova's "ImagesTestJSON"), and libvirtd dying subsequently.
16:56:31 openstack Launchpad bug 1673483 in libvirt (Ubuntu) "libvirt: test_attach_volume_shelved_or_offload_server times out waiting for device detach (which fails)" [Medium,Triaged]
16:56:31 mriedem clarkb: sorry if you guys already talked about this, but in https://bugs.launchpad.net/nova/+bug/1673483 there are comments from ubuntu devs that backports have been made to some packages and are available for testing,
16:56:34 openstackgerrit Dan Smith proposed openstack/nova master: Teach HostAPI about cells https://review.openstack.org/442162
16:56:35 openstackgerrit Dan Smith proposed openstack/nova master: Make scheduler target cells to get compute node instance info https://review.openstack.org/439891
16:56:36 openstackgerrit Dan Smith proposed openstack/nova master: Sort CellMappingList.get_all() for safety https://review.openstack.org/443174
16:56:37 openstackgerrit Dan Smith proposed openstack/nova master: Clean up ClientRouter debt https://review.openstack.org/444487
16:56:37 openstackgerrit Dan Smith proposed openstack/nova master: Add get_by_instance_uuids() to InstanceMappingList https://review.openstack.org/443292
16:56:38 openstackgerrit Dan Smith proposed openstack/nova master: Make server_groups determine deleted-ness from InstanceMappingList https://review.openstack.org/443293
16:56:39 openstackgerrit Dan Smith proposed openstack/nova master: Remove Mitaka-era service version check https://review.openstack.org/442861
16:56:41 openstackgerrit Dan Smith proposed openstack/nova master: Add workaround to disable group policy check upcall https://review.openstack.org/442736
16:56:42 mriedem clarkb: it's not clear to me if we have a way to hook testing packages in our CI?
16:56:45 clarkb mriedem: yes but that bug isn't a problem
16:56:54 clarkb mriedem: we have 3 other libvirt bugs that are actually major issues
16:57:09 clarkb mriedem: and I mentioned it would be simple to modify devstack to use a ppa or just wget and dpkg -i
16:57:45 mriedem ok, and we don't have fixes identified to backport for 1.3.3 in xenial LTS for testing yet
16:58:03 jamespage clarkb: can I get context please on those three bugs
16:58:14 jamespage refs would be ok for now
16:58:28 mriedem jamespage: https://bugs.launchpad.net/nova/+bug/1643911/
16:58:28 openstack bug 1643911 in OpenStack Compute (nova) "libvirt randomly crashes on xenial nodes with "*** Error in `/usr/sbin/libvirtd': malloc(): memory corruption:"" [Medium,Confirmed] https://launchpad.net/bugs/1643911
16:58:28 jamespage guessing one is bug 1643911
16:58:45 clarkb jamespage: its the three elastic recheck links I gave you above
16:58:49 clarkb jamespage: they have links to launchpad
16:58:54 jamespage clarkb: ta
16:59:18 mriedem jamespage: some info in http://lists.openstack.org/pipermail/openstack-dev/2017-March/114648.html too
16:59:31 mdbooth dansmith: https://review.openstack.org/#/c/443292/ https://review.openstack.org/#/c/445139/ :)
17:01:32 openstackgerrit Anusha Unnam proposed openstack/nova master: Add description to policies in servers_migrations.py https://review.openstack.org/451487
17:02:39 mdbooth Absolutely nothing! To be expected, I guess as it's pretty much boilerplate, but it's funny they're character identical. My test is a bit longer.
17:03:05 mriedem mdbooth: it's not the size of the patch, but the motion on the ocean
17:03:14 mriedem that matters that is
17:06:15 mdbooth mriedem: I've just been back through that refactor, btw.
17:06:41 mdbooth I had also glossed over the poll thing, having previously seen it when johnthetubaguy spotted it :)
17:37:33 mriedem sfinucan: you probably have to take this over https://review.openstack.org/#/c/396289/
17:41:53 openstackgerrit Gergely Csatari proposed openstack/nova master: api-ref: adminPass is optional in Create Server https://review.openstack.org/451497
18:01:38 mriedem dumb question: what config option defines if the scheduler is packing or spreading?
18:02:39 mriedem ram_weight_multiplier ?
18:03:08 mriedem dansmith: melwitt: didn't we just talk about something with the default options being weird?
18:03:16 melwitt the subset_size right
18:03:39 melwitt host_subset_size
18:03:57 melwitt if that's 1, it will pack
18:04:07 mriedem or host_subset_size ?
18:04:08 mriedem yeah
18:04:20 mriedem ok and 1 is the default
18:04:29 melwitt right
18:07:29 mriedem were we really thinking about creating a new claims resource in placement for claims in the scheduler? or just re-use allocations?
18:07:44 mriedem creating a new claims resource and ticket system really sounds like the old style quota system to me, with the reservations table
18:07:49 mriedem i would think we want to avoid redoing that
18:07:50 mriedem https://review.openstack.org/#/c/437424/
18:07:57 mriedem jaypipes: ^ would you agree?
18:17:16 edleafe mriedem: the sticking point is how to keep track of groups of allocations as a unit
18:17:58 edleafe I don't believe that that's necessary
18:18:25 edleafe Just allocating when a host is selected, and de-allocating if the build fails
18:20:28 mriedem hmm yeah i didn't think about the grouping
18:20:53 mriedem couldn't you use the consumer_id field to group them?
18:21:12 mriedem essentially your 'claim ticket' is a uuid generated during the build process in conductor,
18:21:19 mriedem and that's used for the claims
18:21:40 mriedem if you successfully build the instance on the compute host, then update the consumer_id with the instance.uuid in the allocatoins
18:21:42 mriedem *allocatoins
18:21:43 edleafe Consumer_id is one way, yeah
18:21:57 mriedem i love fixing my misspell with the same misspell
18:22:12 edleafe at least you're consistent :)
18:22:12 mriedem but we'd still need to keep track of the 'claim ticket' uuid somehow
18:22:19 mriedem maybe req spec?
18:22:42 mriedem i'd say build request, but that's deleted once we pick a cell
18:23:15 mriedem once we pick a cell, we create the instance in that cell, and that instance has a uuid, so why not just use the instance uuid?
18:23:19 edleafe yeah, there has to be something that is persistent from scheduling through build
18:23:41 edleafe But I don't think a separate table and new resource URLs are the way to go
18:24:12 mriedem so conductor asks scheduler for the list of hosts (or a single host), we get the cell that the host is in, and create the instance in that cell and delete the build request,
18:24:22 mriedem so that's a chicken/egg,
18:24:23 mriedem however,

Earlier   Later