| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-03-16 | |||
| 17:44:44 | mriedem | if we can get more details on the actual crashes, that'd be great, | |
| 17:44:45 | openstackgerrit | sean mooney proposed openstack/nova master: remove flake8-import-order for test requirements https://review.openstack.org/445622 | |
| 17:44:54 | mdbooth | mriedem: We should raise a bug against Ubuntu's libvirt package | |
| 17:44:57 | mriedem | becaues it's a pretty high probability gate failure since we moved to xenial | |
| 17:45:08 | mriedem | i wouldn't really know what to say in the bug, | |
| 17:45:13 | mriedem | except "it crashes :P " | |
| 17:45:24 | mriedem | what's that palms up idk guy? | |
| 17:45:24 | mdbooth | Well, we've a dump from one of those crashes | |
| 17:45:29 | mdbooth | Sorry, not the dump | |
| 17:45:42 | mdbooth | The 'panic' or whatever, summary | |
| 17:45:46 | mriedem | ¯\_(ツ)_/¯ | |
| 17:45:47 | mriedem | that guy | |
| 17:45:51 | mdbooth | Hehe | |
| 17:46:02 | johnthetubaguy | lyarwood: I have a tiny nit on this one: https://review.openstack.org/#/c/437597/ | |
| 17:46:13 | lyarwood | looking | |
| 17:46:37 | johnthetubaguy | lyarwood: I am wondering if we don't actually need the get_by_attachment_id method? | |
| 17:48:14 | mdbooth | lyarwood: In https://review.openstack.org/#/c/440693/9/nova/compute/manager.py you've still got the notification out of order | |
| 17:48:36 | mdbooth | lyarwood: I thought we said we wouldn't do that? | |
| 17:49:17 | lyarwood | mdbooth: didn't your review say that it was fine to raise the notification after the detach? | |
| 17:49:32 | johnthetubaguy | yeah, I saw the note added in the commit message | |
| 17:49:33 | mdbooth | lyarwood: It did, but then we discussed it with gibi iirc? | |
| 17:49:35 | johnthetubaguy | it sounds sensible | |
| 17:49:50 | mdbooth | I also said I don't really understand the contract :) | |
| 17:50:04 | johnthetubaguy | right now its half way through the detach right, it should be either before we call cinder, or after I think | |
| 17:50:11 | mdbooth | As it stands today, you'll get a notification even if the detach fails | |
| 17:50:25 | mdbooth | With this patch, you *won't* get a notification if the detach fails | |
| 17:50:36 | mdbooth | That makes sense to me, but it is an unrelated change of behaviour | |
| 17:50:46 | johnthetubaguy | well, depends where it fails | |
| 17:50:54 | johnthetubaguy | really, you should get an error notification if it fails | |
| 17:50:57 | mdbooth | johnthetubaguy: In volume_api.detach() | |
| 17:51:19 | mdbooth | johnthetubaguy: The point was also raised that we could change that entirely | |
| 17:51:29 | mdbooth | But I thought for safety we should just leave it as is | |
| 17:51:33 | johnthetubaguy | the real error happen in _driver_detach_volume | |
| 17:51:38 | mdbooth | And change it deliberately at another time if that's what we want | |
| 17:51:43 | johnthetubaguy | so I think the tweak seems good | |
| 17:51:54 | johnthetubaguy | its tempting to do it after the BDM clean up I supose | |
| 17:52:18 | lyarwood | I'm working on this now so it's not an issue to kick it out | |
| 17:52:19 | johnthetubaguy | oh, wait, we can't we need to send that info as part of the instance | |
| 17:52:56 | johnthetubaguy | so here is a reason to go lyarwood's way | |
| 17:53:03 | johnthetubaguy | in the new API there is only one cinder API call | |
| 17:55:58 | johnthetubaguy | lyarwood: I was thinking the fake BDMs should probably have attachment_id = None, so it matches the current BDMs generated by the code: https://review.openstack.org/#/c/437665/11/nova/tests/unit/objects/test_block_device.py@36 | |
| 17:56:45 | lyarwood | johnthetubaguy: true, I was thinking ahead to testing detach but you're right that doesn't make sense yet. | |
| 17:57:37 | johnthetubaguy | lyarwood: I would keep the uuids for the object compat one though, that seems legit | |
| 17:57:51 | lyarwood | johnthetubaguy: kk | |
| 17:58:56 | lyarwood | johnthetubaguy: and re get_by_attachment_id , I just assumed we would want that around tbh, if you can't think of a case where we would want that over get_by_instance* then I'll drop it | |
| 18:00:20 | johnthetubaguy | lyarwood: I think we will always fetch using volume-uuid and instance-uuid, because of API flow. Its easy to add when we need it though, best to leave it out for now I think. | |
| 18:00:32 | johnthetubaguy | lyarwood: I will probably regret that tomorrow, but it seems the best way to go | |
| 18:00:45 | lyarwood | johnthetubaguy: kk, should I leave the index on attachment_id? | |
| 18:00:59 | mriedem | probably don't need the index if we don't query by attachment_id right? | |
| 18:01:01 | johnthetubaguy | lyarwood: good question, I guess we can leave that off too | |
| 18:01:07 | johnthetubaguy | yeah, what mriedem said | |
| 18:01:14 | lyarwood | right, just checking before I pull it | |
| 18:01:17 | lyarwood | thanks | |
| 18:01:30 | johnthetubaguy | index will slow down write performance, so worth dropping tht | |
| 18:08:04 | mriedem | gibi: have you ever seen this? http://logs.openstack.org/08/445308/3/check/gate-tempest-dsvm-py35-ubuntu-xenial/7bf0d72/logs/screen-n-api.txt.gz#_2017-03-16_05_31_09_399 | |
| 18:08:13 | mriedem | ValueError: Circular reference detected | |
| 18:08:17 | mriedem | during send_notification | |
| 18:08:40 | openstackgerrit | Ken'ichi Ohmichi proposed openstack/nova master: Clarify os-stop API description https://review.openstack.org/446264 | |
| 18:13:00 | mriedem | oomichi: left a question in ^ | |
| 18:13:01 | mriedem | for os-start | |
| 18:15:11 | oomichi | mriedem: oh, nice catch. I am a fun for removing it also | |
| 18:17:40 | mdbooth | mriedem: I've also seen Circular reference detected recently | |
| 18:17:52 | mdbooth | After an error | |
| 18:17:59 | openstackgerrit | Ken'ichi Ohmichi proposed openstack/nova master: Clarify os-stop API description https://review.openstack.org/446264 | |
| 18:18:14 | openstackgerrit | Ildiko Vancsa proposed openstack/nova master: Remove check_detach https://review.openstack.org/446671 | |
| 18:18:33 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/446672 | |
| 18:18:34 | oomichi | mriedem: ^^^ thanks, done | |
| 18:18:42 | mdbooth | mriedem: http://logs.openstack.org/42/445142/3/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/07e282a/logs/screen-n-api.txt.gz?level=ERROR#_2017-03-15_10_18_17_459 | |
| 18:18:57 | mdbooth | Is that similar? | |
| 18:19:09 | mriedem | mdbooth: yes | |
| 18:19:19 | openstack | Launchpad bug 1673375 in oslo.messaging ""ValueError: Circular reference detected" in send_notification" [Undecided,New] | |
| 18:19:19 | mriedem | mdbooth: https://bugs.launchpad.net/nova/+bug/1673375/comments/3 | |
| 18:19:32 | mriedem | the notification is coming from the wrap_exception decorator here https://github.com/openstack/nova/blob/2380659e358770a3f36253b93a112b9779a23958/nova/compute/api.py#L4602 | |
| 18:19:38 | mriedem | but i don't know what the circular reference is | |
| 18:20:55 | mdbooth | I expect if we knew that and decoded the big chunk of text above we'd discover it | |
| 18:21:53 | mriedem | something is messed up in that change of yours | |
| 18:22:01 | mriedem | TypeError: get_by_instance_mapping_list() got an unexpected keyword argument 'expected_addrs' | |
| 18:22:07 | openstackgerrit | OpenStack Proposal Bot proposed openstack/python-novaclient master: Updated from global requirements https://review.openstack.org/446674 | |
| 18:22:18 | mdbooth | mriedem: Yeah, I've fixed it since | |
| 18:22:22 | mriedem | ah ok | |
| 18:22:31 | mdbooth | It was just a typo | |
| 18:22:49 | mriedem | johnthetubaguy: https://review.openstack.org/#/c/446672/ | |
| 18:22:50 | mriedem | yup | |
| 18:22:52 | mriedem | addrs/attrs | |
| 18:23:01 | mriedem | johnthetubaguy: ^ is the cinderclient min version bump | |
| 18:23:20 | mdbooth | mriedem: Now *you* saw that straight away. I stared at it so long I became blind to it. | |
| 18:23:45 | mriedem | pair programming | |
| 18:23:54 | mriedem | the ibm execs were right! | |
| 18:24:52 | mriedem | we were told that spotify did everything correctly, and so we needed to do everything they did | |
| 18:24:53 | mriedem | like, | |
| 18:25:05 | mriedem | pair programming, squads, kanban, agile, | |
| 18:25:11 | mriedem | work around a single table in a room | |
| 18:25:14 | mriedem | eat together | |
| 18:25:17 | mriedem | sleep together | |
| 18:25:18 | mriedem | etc etc | |
| 18:25:22 | mriedem | out pops innovation | |
| 18:25:39 | mriedem | yada yada yada, 18 months later, i left ibm | |
| 18:25:41 | mriedem | ") | |
| 18:25:43 | mriedem | :) | |
| 18:26:08 | mdbooth | mriedem: So, looking at that circular reference thing, there are a few python objects on those dicts | |