| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-02-02 | |||
| 14:33:16 | openstackgerrit | Ed Leafe proposed openstack/nova master: Fix invalid UUIDs in test_compute.py https://review.openstack.org/531479 | |
| 14:33:53 | lyarwood | mriedem: re the connection_info.get('volume_id') lookup, I'm pretty sure that actually works, I can't find where but I recall one of the bdm layers using connection_info to store connection_info['data'] | |
| 14:34:47 | openstackgerrit | Ed Leafe proposed openstack/nova master: Escalate UUID validation warning to error in test https://review.openstack.org/540386 | |
| 14:35:36 | mriedem | connection_info['data'] != connection_info['volume_id'] | |
| 14:35:55 | mriedem | lyarwood: i'm assuming you meant that line of code to be the same as the others that looked for connection_info['data']['volume_id'] | |
| 14:36:02 | mriedem | it was probably a typo, that wasn't tested so it wasn't noticed | |
| 14:40:50 | lyarwood | mriedem: right, I'm pretty confused how the tempest tests I've been working on have passed without this | |
| 14:41:03 | mriedem | lyarwood: volume-backed live migratoin isn't enabled in tempest by default | |
| 14:41:08 | mriedem | so the tests likely aren't getting run | |
| 14:41:15 | lyarwood | mriedem: it is in my DNM change | |
| 14:41:18 | lyarwood | mriedem: and locally | |
| 14:41:24 | lyarwood | mriedem: where these tests have been passing | |
| 14:41:47 | lyarwood | http://logs.openstack.org/50/536350/5/check/legacy-tempest-dsvm-multinode-live-migration/539c51d/job-output.txt.gz for example | |
| 14:44:26 | mriedem | idk, i don't see the lvm volume driver setting connection_info['volume_id'] | |
| 14:45:33 | lyarwood | mriedem: right, anyway, ignore that, I'll look into how these actually passed now, it doesn't change the fix and lack of unit tests | |
| 14:46:01 | lyarwood | mriedem: happy to take over the updating the fix if you don't have time today btw | |
| 14:46:10 | lyarwood | mriedem: this is my mess, I should really be the one to clean it up | |
| 14:47:04 | mriedem | i've got it, just updating test_pre_live_migration_vol_backed_works_correctly_mocked so it doesn't fake out the bdm stuff that the compute manager is doing, | |
| 14:47:13 | mriedem | i.e. actually going to make that test go through driver.get_block_device_info | |
| 14:47:42 | mriedem | bauwser: question in https://review.openstack.org/#/c/540168/ | |
| 14:47:49 | openstackgerrit | Chris Dent proposed openstack/nova master: Reset the _RC_CACHE between tests https://review.openstack.org/540404 | |
| 14:48:34 | leakypipes | lyarwood: on Fridays, I always see your IRC nick and think of pirates. "Yarrr" wood is here. | |
| 14:50:51 | lyarwood | leakypipes: haha, yeah you're not the first to suggest that, had a guy in consulting who would scream that across the office while I was on the phone back in my support days | |
| 14:51:24 | leakypipes | lyarwood: you should wear a pirate hat on Fridays. just sayin'. | |
| 14:51:33 | leakypipes | :) yay! \o/ | |
| 14:51:58 | Spazmotic | I've started putting commits into my monthly planner since I have such a backlog and don't want to merge conflict the poop out of xenapi.vmops hehe | |
| 14:53:21 | bauwser | mriedem: I'm not a specialist on VIF plugging and unplugging, so I could be wrong | |
| 14:53:56 | bauwser | mriedem: but from what I can see, we destroy and cleanup the instance (hence the VIF unplugs) when we hard reboot | |
| 14:54:25 | bauwser | what happens behind the VIF unplug call is unbound territory to me | |
| 14:54:45 | bauwser | I'm just saying that the change melwitt provides makes it consistent with all the other server actions | |
| 14:57:28 | bauwser | 2 mins for making me coffee before an internal meeting, I can do | |
| 14:58:06 | openstackgerrit | Chris Dent proposed openstack/nova master: Optional separate database for placement API https://review.openstack.org/362766 | |
| 15:03:40 | openstackgerrit | rahul bardia proposed openstack/python-novaclient master: nova limits ERROR (Exception): Field names must be unique https://review.openstack.org/527922 | |
| 15:04:12 | melwitt | mriedem: when I read your original comment and when I rewrote it I was thinking 'locally' meant plugging vifs through os-vif only and not making any neutron api calls. plugging locally on the libvirt domain | |
| 15:09:07 | mriedem | we don't make any neutron api calls when plugging/unplugging vifs | |
| 15:09:22 | mriedem | i'm looking back on https://github.com/openstack/nova/commit/e4e1c6a2f060a91899549b9643132abde27d033e | |
| 15:09:44 | mriedem | "this is the explanation I've come up with, albeit not a great one, and indicates a gap in some of the nova/neutron integration story." | |
| 15:09:45 | mriedem | heh | |
| 15:09:48 | mriedem | thanks past me | |
| 15:11:08 | melwitt | mriedem: I know, I mean plugging a vif with no contact with neutron at all. on instance boot we create a port. honestly, I don't really know but from the other code comments, we've run into this before it looks like, and it must work at least *some* of the time otherwise reboot would be 100% broken right now. we really need a neutron expert to explain when/how events are/aren't emitted | |
| 15:11:14 | bauwser | mriedem: the fact that we don't call Neutron when we unplug is understandable, but don't we emit something about the port itself ? | |
| 15:11:41 | bauwser | because if we don't, why should we wait for Neutron ? it's a non-sense | |
| 15:11:45 | mriedem | no we don't update the port | |
| 15:11:50 | mriedem | we plug/unplug, | |
| 15:11:58 | bauwser | so, definitely no need to wait for Neutron, right? | |
| 15:12:01 | mriedem | then neutron agent picks up that change, tells the server, and the server sends an event to nova when it's done | |
| 15:12:01 | fried_rice | cdent: If this isn't already on your radar... https://review.openstack.org/#/c/539712/ | |
| 15:12:10 | bauwser | ah-ha | |
| 15:12:15 | melwitt | the neutron agent doesn't always pick up the change, but we don't know why not | |
| 15:12:21 | melwitt | I assume some kind of race or something | |
| 15:12:26 | mriedem | then we likely refresh out nw info cache on the instance to say if the port is gone or not or up/down | |
| 15:12:35 | cdent | fried_rice: yeah, it's in my queue, but may not be seen until monday | |
| 15:12:36 | mriedem | *our | |
| 15:12:40 | fried_rice | ight | |
| 15:12:57 | fried_rice | Hey, what's the lib I need to install to get around this pcre thing in py35? | |
| 15:13:09 | bauwser | mriedem: I see | |
| 15:13:17 | bauwser | mriedem: thanks for the explanations | |
| 15:13:22 | bauwser | it was black magic to me | |
| 15:13:39 | melwitt | either way, I feel like this isn't in our control. if we can't depend on the neutron agent to detect a unplug/plug, then I'm not sure what else we can do other than not depend on the events | |
| 15:13:49 | mriedem | melwitt: i guess what you're talking about might be this part of my comment: | |
| 15:13:50 | mriedem | and the status change in the port might go undetected by the neutron + # L2 agent (or neutron server) so neutron may not know that the VIF was + # unplugged in the first place and never send an event. | |
| 15:14:02 | melwitt | yeah | |
| 15:14:24 | mriedem | ok i'll come back to your fix after i get those luks patch fixed up | |
| 15:14:25 | melwitt | I was thinking that part came out of a convo with a neutron person long ago | |
| 15:14:42 | bauwser | question I have out of my mind | |
| 15:14:51 | bauwser | how the agent notices that unplug/plug ? | |
| 15:15:03 | bauwser | don't we somehow notify it ? | |
| 15:15:08 | melwitt | I don't know how it would | |
| 15:15:18 | bauwser | that's fun | |
| 15:15:18 | mriedem | we don't notify the agent | |
| 15:15:30 | mriedem | i think the agent is polling for changes, or gets a notification from the backend | |
| 15:15:31 | melwitt | but it must because hard reboot does work most of the time i.e. we do get plug events | |
| 15:15:32 | mriedem | like ovs | |
| 15:15:38 | bauwser | so we rely on something that's not using Nova as the single source of truth | |
| 15:15:43 | mriedem | mlavalle: how does the neutron agent pick up changes for plugged/unplugged vifs? | |
| 15:15:45 | mriedem | polling? | |
| 15:15:56 | mriedem | sean-k-mooney: ^ | |
| 15:16:09 | Spazmotic | ... I feel like I need to look at some quark logs while rebooting a server over and over now.. | |
| 15:16:31 | melwitt | it's just *sometimes* doesn't send the event and if it doesn't, we timeout waiting | |
| 15:18:14 | bauwser | yeah, so again, what's the purpose of waiting Neutron in case of a reboot ? | |
| 15:18:30 | bauwser | we're grown-ups, we can do the unplug/plug ourselves | |
| 15:18:34 | bauwser | if Neutron catches it, cool | |
| 15:18:44 | bauwser | but that shouldn't be a blocking situation | |
| 15:18:45 | Spazmotic | what is neutron need to update on the port? Just the state field? | |
| 15:19:02 | Spazmotic | It's too late to english | |
| 15:20:20 | melwitt | bauwser: originally it was just because we wait during instance boot so I thought that's the normal flow is plug and wait for vif-plugged. but then this bug cropped up and then I found the other comment from mriedem in the file and realized there are situations where we should not wait | |
| 15:20:42 | bauwser | melwitt: for the boot case, I agree, it's important | |
| 15:21:02 | mriedem | this does likely mean that after a reboot, we say the instance is active but it's not really, since you can't ssh into it, | |
| 15:21:08 | bauwser | melwitt: for a reboot, the unplug is transient because we want to just cleanup the instance | |
| 15:21:10 | mriedem | which was the problem we had with initial create | |
| 15:21:38 | bauwser | arf, I need to do some parenting for 20-ish mins | |
| 15:21:55 | melwitt | that's still true, fwiw. there's going to be some time until ssh is possible. but I guess you mean not even getting connection refused | |
| 15:22:23 | melwitt | bauwser: I know, just saying why the wait was put there initially | |
| 15:22:42 | melwitt | I don't think it should be there now, based on how it behaves and based on the other comment in the file | |
| 15:22:58 | bauwser | melwitt: yup, I just said in the review that with your patch, the only case now where we wait is boot | |
| 15:23:07 | melwitt | yeah | |
| 15:23:24 | mriedem | what did we do before the "blow everything away on reboot"? did we wait, or we just didn't touch the vifs at all? | |
| 15:23:53 | melwitt | didn't touch the vifs | |
| 15:24:22 | melwitt | well, the code has always called plug_vifs for some reason, but before, we didn't *unplug* them | |
| 15:24:40 | melwitt | so we'd call plug_vifs on vifs that were already plugged | |