| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-03-16 | |||
| 17:27:35 | openstackgerrit | Octave Orgeron proposed openstack/nova master: Enables MySQL Cluster Support for Nova https://review.openstack.org/446643 | |
| 17:33:25 | mdbooth | lyarwood mriedem: libvirtd crashed | |
| 17:34:23 | mriedem | mdbooth: gah | |
| 17:34:33 | mriedem | remember how i said i'd always chalked this failure up to libvirt crashing :) | |
| 17:34:55 | mdbooth | mriedem: So, I think we're going to request some changes in the CI environment | |
| 17:35:10 | mdbooth | e.g. It would be nice if libvirt didn't restart | |
| 17:35:20 | mdbooth | Also, if we collected core dumps | |
| 17:35:28 | mdbooth | And there are some additional logs which would be useful | |
| 17:35:34 | openstackgerrit | Dan Smith proposed openstack/nova master: Teach HostAPI about cells https://review.openstack.org/442162 | |
| 17:35:35 | openstackgerrit | Dan Smith proposed openstack/nova master: Make scheduler target cells to get compute node instance info https://review.openstack.org/439891 | |
| 17:35:39 | mdbooth | mriedem: What's the best way to request ^^^ ? | |
| 17:35:39 | mriedem | i.e. http://status.openstack.org/elastic-recheck/#1643911 http://status.openstack.org/elastic-recheck/#1646779 http://status.openstack.org/elastic-recheck/#1638982 | |
| 17:36:02 | mriedem | mdbooth: i think the libvirtd logging stuff is handled in devstack | |
| 17:37:56 | mriedem | mdbooth: https://github.com/openstack-dev/devstack/blob/master/lib/nova_plugins/functions-libvirt#L104 | |
| 17:38:15 | mdbooth | http://logs.openstack.org/75/446175/1/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/479c7bf/logs/syslog.txt.gz#_Mar_16_01_01_32 | |
| 17:38:16 | mdbooth | btw | |
| 17:38:47 | openstackgerrit | John Garbutt proposed openstack/nova master: compute: Only destroy BDMs after successful detach call https://review.openstack.org/440693 | |
| 17:38:51 | cfriesen | is anyone aware of a way to have libvirt/qemu live migration maintain the "sparseness" of a qcow2 disk file? After live migration I'm seeing the physical size match the virtual size even though most of that space is not actually used. | |
| 17:42:23 | mdbooth | cfriesen: Is this an image-backed qcow2? | |
| 17:42:30 | mdbooth | Because if so, it should already do that. | |
| 17:44:25 | mriedem | mdbooth: ok so we can just duplicate this bug i guess, | |
| 17:44:29 | mriedem | and i wasted everyone's time | |
| 17:44:35 | mriedem | but thanks for digging into it, | |
| 17:44:44 | mriedem | if we can get more details on the actual crashes, that'd be great, | |
| 17:44:45 | openstackgerrit | sean mooney proposed openstack/nova master: remove flake8-import-order for test requirements https://review.openstack.org/445622 | |
| 17:44:54 | mdbooth | mriedem: We should raise a bug against Ubuntu's libvirt package | |
| 17:44:57 | mriedem | becaues it's a pretty high probability gate failure since we moved to xenial | |
| 17:45:08 | mriedem | i wouldn't really know what to say in the bug, | |
| 17:45:13 | mriedem | except "it crashes :P " | |
| 17:45:24 | mdbooth | Well, we've a dump from one of those crashes | |
| 17:45:24 | mriedem | what's that palms up idk guy? | |
| 17:45:29 | mdbooth | Sorry, not the dump | |
| 17:45:42 | mdbooth | The 'panic' or whatever, summary | |
| 17:45:46 | mriedem | ¯\_(ツ)_/¯ | |
| 17:45:47 | mriedem | that guy | |
| 17:45:51 | mdbooth | Hehe | |
| 17:46:02 | johnthetubaguy | lyarwood: I have a tiny nit on this one: https://review.openstack.org/#/c/437597/ | |
| 17:46:13 | lyarwood | looking | |
| 17:46:37 | johnthetubaguy | lyarwood: I am wondering if we don't actually need the get_by_attachment_id method? | |
| 17:48:14 | mdbooth | lyarwood: In https://review.openstack.org/#/c/440693/9/nova/compute/manager.py you've still got the notification out of order | |
| 17:48:36 | mdbooth | lyarwood: I thought we said we wouldn't do that? | |
| 17:49:17 | lyarwood | mdbooth: didn't your review say that it was fine to raise the notification after the detach? | |
| 17:49:32 | johnthetubaguy | yeah, I saw the note added in the commit message | |
| 17:49:33 | mdbooth | lyarwood: It did, but then we discussed it with gibi iirc? | |
| 17:49:35 | johnthetubaguy | it sounds sensible | |
| 17:49:50 | mdbooth | I also said I don't really understand the contract :) | |
| 17:50:04 | johnthetubaguy | right now its half way through the detach right, it should be either before we call cinder, or after I think | |
| 17:50:11 | mdbooth | As it stands today, you'll get a notification even if the detach fails | |
| 17:50:25 | mdbooth | With this patch, you *won't* get a notification if the detach fails | |
| 17:50:36 | mdbooth | That makes sense to me, but it is an unrelated change of behaviour | |
| 17:50:46 | johnthetubaguy | well, depends where it fails | |
| 17:50:54 | johnthetubaguy | really, you should get an error notification if it fails | |
| 17:50:57 | mdbooth | johnthetubaguy: In volume_api.detach() | |
| 17:51:19 | mdbooth | johnthetubaguy: The point was also raised that we could change that entirely | |
| 17:51:29 | mdbooth | But I thought for safety we should just leave it as is | |
| 17:51:33 | johnthetubaguy | the real error happen in _driver_detach_volume | |
| 17:51:38 | mdbooth | And change it deliberately at another time if that's what we want | |
| 17:51:43 | johnthetubaguy | so I think the tweak seems good | |
| 17:51:54 | johnthetubaguy | its tempting to do it after the BDM clean up I supose | |
| 17:52:18 | lyarwood | I'm working on this now so it's not an issue to kick it out | |
| 17:52:19 | johnthetubaguy | oh, wait, we can't we need to send that info as part of the instance | |
| 17:52:56 | johnthetubaguy | so here is a reason to go lyarwood's way | |
| 17:53:03 | johnthetubaguy | in the new API there is only one cinder API call | |
| 17:55:58 | johnthetubaguy | lyarwood: I was thinking the fake BDMs should probably have attachment_id = None, so it matches the current BDMs generated by the code: https://review.openstack.org/#/c/437665/11/nova/tests/unit/objects/test_block_device.py@36 | |
| 17:56:45 | lyarwood | johnthetubaguy: true, I was thinking ahead to testing detach but you're right that doesn't make sense yet. | |
| 17:57:37 | johnthetubaguy | lyarwood: I would keep the uuids for the object compat one though, that seems legit | |
| 17:57:51 | lyarwood | johnthetubaguy: kk | |
| 17:58:56 | lyarwood | johnthetubaguy: and re get_by_attachment_id , I just assumed we would want that around tbh, if you can't think of a case where we would want that over get_by_instance* then I'll drop it | |
| 18:00:20 | johnthetubaguy | lyarwood: I think we will always fetch using volume-uuid and instance-uuid, because of API flow. Its easy to add when we need it though, best to leave it out for now I think. | |
| 18:00:32 | johnthetubaguy | lyarwood: I will probably regret that tomorrow, but it seems the best way to go | |
| 18:00:45 | lyarwood | johnthetubaguy: kk, should I leave the index on attachment_id? | |
| 18:00:59 | mriedem | probably don't need the index if we don't query by attachment_id right? | |
| 18:01:01 | johnthetubaguy | lyarwood: good question, I guess we can leave that off too | |
| 18:01:07 | johnthetubaguy | yeah, what mriedem said | |
| 18:01:14 | lyarwood | right, just checking before I pull it | |
| 18:01:17 | lyarwood | thanks | |
| 18:01:30 | johnthetubaguy | index will slow down write performance, so worth dropping tht | |
| 18:08:04 | mriedem | gibi: have you ever seen this? http://logs.openstack.org/08/445308/3/check/gate-tempest-dsvm-py35-ubuntu-xenial/7bf0d72/logs/screen-n-api.txt.gz#_2017-03-16_05_31_09_399 | |
| 18:08:13 | mriedem | ValueError: Circular reference detected | |
| 18:08:17 | mriedem | during send_notification | |
| 18:08:40 | openstackgerrit | Ken'ichi Ohmichi proposed openstack/nova master: Clarify os-stop API description https://review.openstack.org/446264 | |
| 18:13:00 | mriedem | oomichi: left a question in ^ | |
| 18:13:01 | mriedem | for os-start | |
| 18:15:11 | oomichi | mriedem: oh, nice catch. I am a fun for removing it also | |
| 18:17:40 | mdbooth | mriedem: I've also seen Circular reference detected recently | |
| 18:17:52 | mdbooth | After an error | |
| 18:17:59 | openstackgerrit | Ken'ichi Ohmichi proposed openstack/nova master: Clarify os-stop API description https://review.openstack.org/446264 | |
| 18:18:14 | openstackgerrit | Ildiko Vancsa proposed openstack/nova master: Remove check_detach https://review.openstack.org/446671 | |
| 18:18:33 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/446672 | |
| 18:18:34 | oomichi | mriedem: ^^^ thanks, done | |
| 18:18:42 | mdbooth | mriedem: http://logs.openstack.org/42/445142/3/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/07e282a/logs/screen-n-api.txt.gz?level=ERROR#_2017-03-15_10_18_17_459 | |
| 18:18:57 | mdbooth | Is that similar? | |
| 18:19:09 | mriedem | mdbooth: yes | |
| 18:19:19 | mriedem | mdbooth: https://bugs.launchpad.net/nova/+bug/1673375/comments/3 | |
| 18:19:19 | openstack | Launchpad bug 1673375 in oslo.messaging ""ValueError: Circular reference detected" in send_notification" [Undecided,New] | |
| 18:19:32 | mriedem | the notification is coming from the wrap_exception decorator here https://github.com/openstack/nova/blob/2380659e358770a3f36253b93a112b9779a23958/nova/compute/api.py#L4602 | |
| 18:19:38 | mriedem | but i don't know what the circular reference is | |
| 18:20:55 | mdbooth | I expect if we knew that and decoded the big chunk of text above we'd discover it | |
| 18:21:53 | mriedem | something is messed up in that change of yours | |
| 18:22:01 | mriedem | TypeError: get_by_instance_mapping_list() got an unexpected keyword argument 'expected_addrs' | |