| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-06-21 | |||
| 17:10:16 | stephenfin | /worry about the stuff we missed 🙈 | |
| 17:11:08 | zcorneli | stephenfin: Both, of course | |
| 17:11:58 | zcorneli | stephenfin, dansmith: Thanks for all the help getting that merged in | |
| 17:40:08 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Tighten up ReportClient use of generation https://review.openstack.org/556669 | |
| 17:46:32 | zcorneli | dansmith: https://review.openstack.org/#/c/574792/7 | |
| 17:51:28 | dansmith | zcorneli: cool, we need devstack cores to look at that | |
| 17:51:30 | dansmith | mtreinish: ^ | |
| 17:58:28 | openstackgerrit | Eric Fried proposed openstack/nova master: Tighten up ReportClient use of generation https://review.openstack.org/556669 | |
| 18:11:35 | mtreinish | dansmith: ok, I put it on my list. I'll look at it after I get back from lunch | |
| 18:12:01 | dansmith | mtreinish: thanks, I left some comments on things for you to respond to, but it's fairly minor | |
| 19:23:55 | SpamapS | Running into confusing stuff with cells v2 | |
| 19:24:50 | SpamapS | we had a "rogue cell" recently and since deleting it we're not seeing nova-api listen to the cell rabbitmq's | |
| 19:24:59 | SpamapS | have restarted everything | |
| 19:25:46 | SpamapS | and nova-conductor isn't either | |
| 19:28:59 | SpamapS | the symptom is that when nova-compute sends back the 'vif-plugged' neutron is never informed and never finishes activating network ports | |
| 19:30:47 | dansmith | SpamapS: you have that backwards | |
| 19:31:01 | dansmith | SpamapS: vif-plugged comes _from_ neutron to nova-api, over the bus, and to nova-compute | |
| 19:31:49 | dansmith | if you deleted a cell, you should make sure that your instance and host mapping records all refer to legit cell records | |
| 19:32:35 | SpamapS | have verified all mappings are correct | |
| 19:32:48 | SpamapS | how does neutron know to send the vif-plugged event? | |
| 19:33:07 | SpamapS | we see os-vif successfully plugged logs | |
| 19:33:21 | SpamapS | but then nova-compute times out waiting for network-vif-plugged | |
| 19:33:41 | SpamapS | begging the question (we've been tracing through the code all last night and today) .. what actually triggers that? | |
| 19:33:49 | dansmith | it sends it in response to us doing plug on the compute side and asking neutron to wire it up | |
| 19:34:13 | dansmith | not sure if that request to neutron actually starts in os-vif now or not (it didn't use to) | |
| 19:35:17 | dansmith | do you see the event come into nova-api from neutron? | |
| 19:35:33 | openstackgerrit | Lenny Verkhovsky proposed openstack/nova master: WIP workaround for RHEL7.5 qemu 2.0 https://review.openstack.org/577280 | |
| 19:36:02 | SpamapS | dansmith: not since our unsuccessful deploy (we assume it's because of our rogue cell that the deploy created, but it could of course be something else) | |
| 19:36:32 | SpamapS | so that's where we've been trying to find what tells neutron to try that. | |
| 19:36:56 | dansmith | looks like it's in os-vif now, as I don't see (in paths that take os-vif) where we do it in nova anymore | |
| 19:37:49 | dansmith | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L5604-L5607 | |
| 19:37:58 | dansmith | that context won't return until the event is received | |
| 19:38:09 | SpamapS | correct | |
| 19:38:10 | dansmith | so we start the context, do something that will trigger the event, and on __exit__ we wait for the event | |
| 19:38:11 | SpamapS | we're timing out there | |
| 19:39:02 | SpamapS | dansmith: and that do something is..... | |
| 19:39:17 | dansmith | plug | |
| 19:42:03 | SpamapS | dansmith: right, I see that, but how does neutron find out that you plugged? | |
| 19:42:29 | dansmith | I'm looking through os-vif now | |
| 19:42:36 | SpamapS | I had thought it was something like, nova-compute calls os_vif.<plugin>.plug(things) and then sends some kind of message to the conductors to say "we plugged!" and the conductors call neutron. | |
| 19:42:46 | dansmith | no | |
| 19:43:40 | dansmith | I dunno, I don't really see os-vif calling neutron anywhere | |
| 19:43:52 | SpamapS | nor do I | |
| 19:44:37 | SpamapS | only nova.network.neutronv2 calls neutron AFAICT | |
| 19:45:13 | dansmith | well, that's our interface for it of course | |
| 19:46:49 | SpamapS | IIRC, network-vif-plugged is sent when a port goes from DOWN -> ACTIVE | |
| 19:47:01 | SpamapS | and the neutron agent does that transition (we're using linuxbridge) | |
| 19:47:15 | SpamapS | so then I'm looking for what tells the neutron agent to do *that* | |
| 19:50:10 | jroll | doesn't that happen when the 'host' is sent in the portbinding part of the port body? (whether during port create or update | |
| 19:52:03 | jroll | which I guess would be here: https://github.com/openstack/nova/blob/master/nova/network/neutronv2/api.py#L1072 | |
| 19:53:09 | SpamapS | right, I think that triggers binding | |
| 19:53:32 | dansmith | the key is, we can't do it until we've created the tap or whatever type of interface | |
| 19:53:49 | dansmith | which is why this interlock exists | |
| 19:54:38 | SpamapS | the tap happens | |
| 19:55:36 | SpamapS | so, IIRC, nova-compute doesn't call that, nova-conductor calls that | |
| 19:55:47 | dansmith | nova conductor calls what? | |
| 19:55:53 | dansmith | bind? | |
| 19:56:21 | SpamapS | well I mean, nova-conductor is the only one making calls to the neutron API | |
| 19:56:38 | dansmith | no, that's definitely not true | |
| 19:56:39 | jroll | _update_ports_for_instance is called by allocate_for_instance, which was in nova-compute last I checked | |
| 19:56:48 | dansmith | right | |
| 19:57:20 | dansmith | but, the thing that triggers this plug event has to happen within that context or we could miss it | |
| 19:57:21 | SpamapS | ah ok | |
| 19:57:56 | SpamapS | so, the port gets a binding_host_id, but does not get status set to ACTIVE | |
| 19:58:29 | jroll | sounds like something is failing in neutron, then | |
| 19:59:01 | dansmith | could be that, but neutron is supposed to indicate failure to us if it does | |
| 19:59:20 | jroll | or locking up in neutron, I guess :) | |
| 19:59:36 | dansmith | or neutron is sending it and it's not hitting nova-api, which is fairly common | |
| 19:59:57 | jroll | neutron sets the port to ACTIVE, no? | |
| 20:00:01 | SpamapS | network-vif-plugged will not be sent until ACTIVE | |
| 20:00:10 | SpamapS | agents make ports ACTIVE AFAICT | |
| 20:02:15 | dansmith | SpamapS: which version is this again? and specifically which version of os-vif? | |
| 20:02:32 | SpamapS | Pike | |
| 20:02:34 | dansmith | IIRC, ovs used to be the only one that supported events at all, | |
| 20:02:40 | dansmith | oh okay, that's new enough I think | |
| 20:02:48 | dansmith | are you sure these events were coming in before? | |
| 20:03:01 | SpamapS | os-vif==1.7.0 | |
| 20:03:01 | dansmith | and that you haven't just flipped vif-plugging-is-fatal recently or something? | |
| 20:03:10 | SpamapS | yes they were coming in | |
| 20:03:21 | dansmith | neutron will log when it sends the events, IIRC as they are batched, | |
| 20:03:29 | dansmith | so can you see it doing stuff to generate those? | |
| 20:04:08 | SpamapS | we did a deploy which mostly didn't change much, just some external things, and everything stopped working. We saw this once before when we had a badly configured cell that had the top level rmq's as the transport_url, and deleting that cell, and restarting everything fixed that.. but now.. stuck. | |
| 20:04:42 | SpamapS | Which is why I'm trying to figure out if nova-compute<->nova-conductor comms fail are causing the issue. | |
| 20:04:59 | SpamapS | neutron is not logging trying to send network-vif-plugged | |
| 20:05:09 | SpamapS | which is not a surprise because the ports are not going ACTIVE | |
| 20:05:16 | dansmith | a conductor or compute with a badly-configured transport_url in the config would prevent them from talking to other services in their cell | |
| 20:05:20 | SpamapS | I presume that is because whatever triggers the agent to make them active is not happening. | |
| 20:05:30 | dansmith | for events, nova-api looks up the cell of the instance and fires the event direct to the compute via its mq | |
| 20:06:44 | SpamapS | We've doubled checked all the config files, which didn't change anyway, they're all talking to the right rabbits. | |
| 20:06:46 | SpamapS | And the cells are fixed now | |
| 20:07:11 | SpamapS | just worried that something is stuck somewhere that is making a service still send events meant for cell-rabbit to top-level-rabbit. | |
| 20:07:36 | SpamapS | but neutron isn't really "cell aware" so I still suspect nova | |
| 20:07:53 | SpamapS | and can't seem to find the thing that nova does to tell neutron to activate a port | |
| 20:08:16 | dansmith | so, having looked through all the plug path stuff just now, | |
| 20:08:41 | dansmith | the only thing I can think of is if we create the tap, name it as we were told, and neutron agent on the compute notices that it got created, adds it to the bridge, and then fires the event | |
| 20:09:29 | dansmith | I thought we specifically called to neutron to tell it it was ready, but maybe we don't | |
| 20:09:35 | SpamapS | I think we do too | |
| 20:10:43 | dansmith | well, the plug path for linux bridge is pretty darn simple | |
| 20:11:22 | SpamapS | yeah, neutron even reports that it doesn't have to do anything when it gets triggered.. | |
| 20:11:23 | SpamapS | when things are working | |