| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-03-16 | |||
| 17:01:58 | jaypipes | mriedem: https://review.openstack.org/#/c/416669/ looks sensible and has had several cdent reviews around API correctness. | |
| 17:02:04 | openstackgerrit | John Garbutt proposed openstack/nova-specs master: Add spec to use cinder's new attachment API https://review.openstack.org/373203 | |
| 17:02:20 | sean-k-mooney | jaypipes: i proably should have been more stringent in my own review but the main think i was reviewing for was if the os-vif delegation looked correct. | |
| 17:02:35 | jaypipes | sean-k-mooney: it's cool duder :) | |
| 17:02:37 | mdbooth | mriedem lyarwood: I've been looking at that detach gate failure, discussing with danpb downstream. | |
| 17:03:04 | mdbooth | I currently suspect it's a libvirt issue. | |
| 17:03:40 | mdbooth | If you look at the libvirt logs, we see the device_del, then the job disappears into a black hole, then Nova reports a disconnect. | |
| 17:03:47 | sean-k-mooney | jaypipes: that said alot of that code that deligate to os-vif in nova is not unittested correctly so this is something i guess we should improve going forward | |
| 17:04:10 | jaypipes | sean-k-mooney: ack | |
| 17:04:26 | mriedem | jaypipes: ok, after meetings and other things, like this big ass bowl of chili in front of me | |
| 17:05:25 | mdbooth | lyarwood mriedem: All of the above occurs within the context of, from Nova's pov, self._domain.detachDeviceFlags(device_xml, flags=flags) | |
| 17:05:37 | mdbooth | So at the moment I don't see how it can be a Nova issue | |
| 17:06:14 | sean-k-mooney | mhenkel: the unitest in https://review.openstack.org/#/c/334048/ may be of use to you as a reference. | |
| 17:07:55 | mdbooth | mriedem: How long do those log files live, btw? If I link to them in bz will they still be there in a month or 2? | |
| 17:09:20 | mriedem | mdbooth: i'm not sure about that, it used to be 6 months, | |
| 17:09:35 | mriedem | fungi: jeblair: ^ how long does infra keep ci job logs around? | |
| 17:09:53 | fungi | mriedem: now it's down to about 45 days | |
| 17:10:08 | fungi | mdbooth: ^ | |
| 17:10:15 | jaypipes | mriedem: :) | |
| 17:10:16 | mdbooth | fungi: Thanks | |
| 17:10:36 | sean-k-mooney | jaypipes: actully while you are around if you get a chance to look at https://review.openstack.org/#/c/441590/ let me know if you are ok with this direction. i belive that with this change the host_info class has all the info we need to pass to neutron for the port binding negciation. | |
| 17:10:52 | fungi | mriedem: mdbooth: that's keeping us barely afloat with the present job log volume http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=717&rra_id=all | |
| 17:10:57 | jeblair | (it's directly tied to log volume; we've had to reduce it due to a significant increase in the amount of data jobs are storing in logs) | |
| 17:11:05 | sean-k-mooney | jaypipes: that said i dont want to merge this until i have some prototype code to consume it | |
| 17:11:09 | mriedem | mdbooth: fwiw i didn't notice this type of error when we were using trusty | |
| 17:11:12 | mriedem | which was libvirt 1.2.2 | |
| 17:11:55 | mdbooth | fungi: NP. I originally assumed it would be days, tbh. | |
| 17:12:01 | mdbooth | 45 days is excellent. | |
| 17:12:30 | sean-k-mooney | sfinucan: https://review.openstack.org/#/c/441590/ is proably of interest to you too. again im going to keep the workflow -1 until i have an end to end poc working to ensure it does capture all required info | |
| 17:14:12 | fungi | mdbooth: we wish it could be more, but we generate _lots_ of logs these days | |
| 17:19:52 | openstackgerrit | Christopher Brown proposed openstack/nova master: Add lan9118 as valid nic for hw_vif_model property for qemu https://review.openstack.org/393489 | |
| 17:22:03 | jaypipes | sean-k-mooney: reviewed. | |
| 17:25:19 | mriedem | fungi: totally fine | |
| 17:25:24 | mriedem | was just wondering the timeline | |
| 17:25:47 | openstackgerrit | Octave Orgeron proposed openstack/nova master: Enables MySQL Cluster Support for Nova https://review.openstack.org/446643 | |
| 17:27:14 | sean-k-mooney | jaypipes: thanks that also needs units so ill adress your comments when i add them | |
| 17:27:35 | openstackgerrit | Octave Orgeron proposed openstack/nova master: Enables MySQL Cluster Support for Nova https://review.openstack.org/446643 | |
| 17:33:25 | mdbooth | lyarwood mriedem: libvirtd crashed | |
| 17:34:23 | mriedem | mdbooth: gah | |
| 17:34:33 | mriedem | remember how i said i'd always chalked this failure up to libvirt crashing :) | |
| 17:34:55 | mdbooth | mriedem: So, I think we're going to request some changes in the CI environment | |
| 17:35:10 | mdbooth | e.g. It would be nice if libvirt didn't restart | |
| 17:35:20 | mdbooth | Also, if we collected core dumps | |
| 17:35:28 | mdbooth | And there are some additional logs which would be useful | |
| 17:35:34 | openstackgerrit | Dan Smith proposed openstack/nova master: Teach HostAPI about cells https://review.openstack.org/442162 | |
| 17:35:35 | openstackgerrit | Dan Smith proposed openstack/nova master: Make scheduler target cells to get compute node instance info https://review.openstack.org/439891 | |
| 17:35:39 | mriedem | i.e. http://status.openstack.org/elastic-recheck/#1643911 http://status.openstack.org/elastic-recheck/#1646779 http://status.openstack.org/elastic-recheck/#1638982 | |
| 17:35:39 | mdbooth | mriedem: What's the best way to request ^^^ ? | |
| 17:36:02 | mriedem | mdbooth: i think the libvirtd logging stuff is handled in devstack | |
| 17:37:56 | mriedem | mdbooth: https://github.com/openstack-dev/devstack/blob/master/lib/nova_plugins/functions-libvirt#L104 | |
| 17:38:15 | mdbooth | http://logs.openstack.org/75/446175/1/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/479c7bf/logs/syslog.txt.gz#_Mar_16_01_01_32 | |
| 17:38:16 | mdbooth | btw | |
| 17:38:47 | openstackgerrit | John Garbutt proposed openstack/nova master: compute: Only destroy BDMs after successful detach call https://review.openstack.org/440693 | |
| 17:38:51 | cfriesen | is anyone aware of a way to have libvirt/qemu live migration maintain the "sparseness" of a qcow2 disk file? After live migration I'm seeing the physical size match the virtual size even though most of that space is not actually used. | |
| 17:42:23 | mdbooth | cfriesen: Is this an image-backed qcow2? | |
| 17:42:30 | mdbooth | Because if so, it should already do that. | |
| 17:44:25 | mriedem | mdbooth: ok so we can just duplicate this bug i guess, | |
| 17:44:29 | mriedem | and i wasted everyone's time | |
| 17:44:35 | mriedem | but thanks for digging into it, | |
| 17:44:44 | mriedem | if we can get more details on the actual crashes, that'd be great, | |
| 17:44:45 | openstackgerrit | sean mooney proposed openstack/nova master: remove flake8-import-order for test requirements https://review.openstack.org/445622 | |
| 17:44:54 | mdbooth | mriedem: We should raise a bug against Ubuntu's libvirt package | |
| 17:44:57 | mriedem | becaues it's a pretty high probability gate failure since we moved to xenial | |
| 17:45:08 | mriedem | i wouldn't really know what to say in the bug, | |
| 17:45:13 | mriedem | except "it crashes :P " | |
| 17:45:24 | mriedem | what's that palms up idk guy? | |
| 17:45:24 | mdbooth | Well, we've a dump from one of those crashes | |
| 17:45:29 | mdbooth | Sorry, not the dump | |
| 17:45:42 | mdbooth | The 'panic' or whatever, summary | |
| 17:45:46 | mriedem | ¯\_(ツ)_/¯ | |
| 17:45:47 | mriedem | that guy | |
| 17:45:51 | mdbooth | Hehe | |
| 17:46:02 | johnthetubaguy | lyarwood: I have a tiny nit on this one: https://review.openstack.org/#/c/437597/ | |
| 17:46:13 | lyarwood | looking | |
| 17:46:37 | johnthetubaguy | lyarwood: I am wondering if we don't actually need the get_by_attachment_id method? | |
| 17:48:14 | mdbooth | lyarwood: In https://review.openstack.org/#/c/440693/9/nova/compute/manager.py you've still got the notification out of order | |
| 17:48:36 | mdbooth | lyarwood: I thought we said we wouldn't do that? | |
| 17:49:17 | lyarwood | mdbooth: didn't your review say that it was fine to raise the notification after the detach? | |
| 17:49:32 | johnthetubaguy | yeah, I saw the note added in the commit message | |
| 17:49:33 | mdbooth | lyarwood: It did, but then we discussed it with gibi iirc? | |
| 17:49:35 | johnthetubaguy | it sounds sensible | |
| 17:49:50 | mdbooth | I also said I don't really understand the contract :) | |
| 17:50:04 | johnthetubaguy | right now its half way through the detach right, it should be either before we call cinder, or after I think | |
| 17:50:11 | mdbooth | As it stands today, you'll get a notification even if the detach fails | |
| 17:50:25 | mdbooth | With this patch, you *won't* get a notification if the detach fails | |
| 17:50:36 | mdbooth | That makes sense to me, but it is an unrelated change of behaviour | |
| 17:50:46 | johnthetubaguy | well, depends where it fails | |
| 17:50:54 | johnthetubaguy | really, you should get an error notification if it fails | |
| 17:50:57 | mdbooth | johnthetubaguy: In volume_api.detach() | |
| 17:51:19 | mdbooth | johnthetubaguy: The point was also raised that we could change that entirely | |
| 17:51:29 | mdbooth | But I thought for safety we should just leave it as is | |
| 17:51:33 | johnthetubaguy | the real error happen in _driver_detach_volume | |
| 17:51:38 | mdbooth | And change it deliberately at another time if that's what we want | |
| 17:51:43 | johnthetubaguy | so I think the tweak seems good | |
| 17:51:54 | johnthetubaguy | its tempting to do it after the BDM clean up I supose | |
| 17:52:18 | lyarwood | I'm working on this now so it's not an issue to kick it out | |
| 17:52:19 | johnthetubaguy | oh, wait, we can't we need to send that info as part of the instance | |
| 17:52:56 | johnthetubaguy | so here is a reason to go lyarwood's way | |
| 17:53:03 | johnthetubaguy | in the new API there is only one cinder API call | |
| 17:55:58 | johnthetubaguy | lyarwood: I was thinking the fake BDMs should probably have attachment_id = None, so it matches the current BDMs generated by the code: https://review.openstack.org/#/c/437665/11/nova/tests/unit/objects/test_block_device.py@36 | |