| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-06-21 | |||
| 20:03:10 | SpamapS | yes they were coming in | |
| 20:03:21 | dansmith | neutron will log when it sends the events, IIRC as they are batched, | |
| 20:03:29 | dansmith | so can you see it doing stuff to generate those? | |
| 20:04:08 | SpamapS | we did a deploy which mostly didn't change much, just some external things, and everything stopped working. We saw this once before when we had a badly configured cell that had the top level rmq's as the transport_url, and deleting that cell, and restarting everything fixed that.. but now.. stuck. | |
| 20:04:42 | SpamapS | Which is why I'm trying to figure out if nova-compute<->nova-conductor comms fail are causing the issue. | |
| 20:04:59 | SpamapS | neutron is not logging trying to send network-vif-plugged | |
| 20:05:09 | SpamapS | which is not a surprise because the ports are not going ACTIVE | |
| 20:05:16 | dansmith | a conductor or compute with a badly-configured transport_url in the config would prevent them from talking to other services in their cell | |
| 20:05:20 | SpamapS | I presume that is because whatever triggers the agent to make them active is not happening. | |
| 20:05:30 | dansmith | for events, nova-api looks up the cell of the instance and fires the event direct to the compute via its mq | |
| 20:06:44 | SpamapS | We've doubled checked all the config files, which didn't change anyway, they're all talking to the right rabbits. | |
| 20:06:46 | SpamapS | And the cells are fixed now | |
| 20:07:11 | SpamapS | just worried that something is stuck somewhere that is making a service still send events meant for cell-rabbit to top-level-rabbit. | |
| 20:07:36 | SpamapS | but neutron isn't really "cell aware" so I still suspect nova | |
| 20:07:53 | SpamapS | and can't seem to find the thing that nova does to tell neutron to activate a port | |
| 20:08:16 | dansmith | so, having looked through all the plug path stuff just now, | |
| 20:08:41 | dansmith | the only thing I can think of is if we create the tap, name it as we were told, and neutron agent on the compute notices that it got created, adds it to the bridge, and then fires the event | |
| 20:09:29 | dansmith | I thought we specifically called to neutron to tell it it was ready, but maybe we don't | |
| 20:09:35 | SpamapS | I think we do too | |
| 20:10:43 | dansmith | well, the plug path for linux bridge is pretty darn simple | |
| 20:11:22 | SpamapS | yeah, neutron even reports that it doesn't have to do anything when it gets triggered.. | |
| 20:11:23 | SpamapS | when things are working | |
| 20:12:02 | dansmith | there is a neutron agent running on that box at the time, right/ | |
| 20:12:18 | dansmith | and not in a container in such a way that it can't see the interfaces nova would create? | |
| 20:13:12 | SpamapS | yes the agent is running | |
| 20:13:22 | SpamapS | and they share a netns (deployed with kolla-ansible) | |
| 20:13:31 | SpamapS | they actually don't even have a netns | |
| 20:13:34 | SpamapS | they're on the host's networking | |
| 20:15:59 | dansmith | from looking through the agent (for the first time ever) it surely looks like it's just polling for new interfaces to pop up | |
| 20:18:49 | SpamapS | dansmith: I can verify that brctl show from inside the agent shows the taps | |
| 20:20:36 | openstackgerrit | Brianna Poulos proposed openstack/python-novaclient master: Fix trusted-image-certificate-id help text https://review.openstack.org/576555 | |
| 20:23:25 | dansmith | yeah, so the agent runs a loop, | |
| 20:23:46 | dansmith | each time through it looks for new interfaces, calls process_network_devices(), which goes through each new one, ends up at plug_interface(), etc, etc | |
| 20:24:16 | dansmith | so I think we create a properly-named tap, it notices, does the plug into the bridge and the wiring and then emits the notification | |
| 20:24:39 | dansmith | which explains why I can't find any calls to neutron in the critical path while we're waiting | |
| 20:26:05 | SpamapS | dansmith: makes sense I 'spose | |
| 20:27:00 | dansmith | looks like with debug on, | |
| 20:27:12 | dansmith | the agent should log "Agent loop found changes!" and then the device info it found | |
| 20:27:23 | dansmith | do you see that happen once nova has created the taps? | |
| 20:28:24 | dansmith | this: https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/drivers/agent/_common_agent.py#L453-L453 | |
| 20:30:38 | SpamapS | dansmith: right, we never get Agent loop found changes | |
| 20:30:45 | SpamapS | stracing now to see why | |
| 20:33:49 | dansmith | got confirmation from someone who would know that "plug" communication between neutron and nova just involves creating the interface | |
| 20:36:30 | SpamapS | k | |
| 20:36:55 | SpamapS | well I can see the taps from the linuxbridge container | |
| 20:37:07 | SpamapS | but I never see the agent actually polling brctl or ip or anything | |
| 20:37:39 | dansmith | well, it may use iproute directly | |
| 20:37:49 | dansmith | which would strace out as a socket(AF_NETLINK,...) early on | |
| 20:37:59 | SpamapS | "or ip" was my vague reference to iproute | |
| 20:38:07 | SpamapS | but .. oh that? | |
| 20:38:29 | dansmith | er, pyiproute I mean | |
| 20:39:01 | dansmith | `ip` is a CLI that speaks over a special type of socket called a "netlink" socket, which is a communication channel to the network subsystem inside the kernel | |
| 20:39:05 | dansmith | it's how it does all its magic | |
| 20:39:07 | SpamapS | yeah makes sense | |
| 20:39:11 | dansmith | anything else can speak that as well | |
| 20:40:28 | dansmith | if it did, it could also get async notification of networking events, so it would make sense | |
| 20:40:59 | SpamapS | no looks like it execs 'ip' | |
| 20:41:06 | SpamapS | or at least, sometimes it does | |
| 20:41:08 | SpamapS | still reading | |
| 20:41:30 | SpamapS | and digs around in /sys | |
| 20:42:45 | dansmith | https://github.com/openstack/os-vif/blob/master/os_vif/internal/command/ip/impl_pyroute2.py | |
| 20:43:12 | dansmith | 4 | |
| 20:43:12 | dansmith | % grep import.*pyroute2 -r neutron/ | wc -l | |
| 20:43:21 | dansmith | so, neutron uses it some, in master at least | |
| 20:43:50 | dansmith | in privileged/agent/linux/ip_lib.py | |
| 20:44:00 | dansmith | anyway | |
| 20:44:11 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Address nits from consumer generation https://review.openstack.org/577227 | |
| 20:44:16 | SpamapS | not in pike | |
| 20:44:38 | SpamapS | neutron doesn't use os_vif directly even, AFAICT | |
| 20:45:00 | dansmith | I know it doesn't use os-vif | |
| 20:45:09 | dansmith | because os-vif is the integration library between nova and neutron | |
| 20:45:10 | dansmith | so naturally, | |
| 20:45:14 | dansmith | it's only used by nova | |
| 20:47:47 | SpamapS | in all the ways I see the agent code checking for interfaces, I can see the interface | |
| 20:47:52 | SpamapS | but the agent never "notices" | |
| 20:49:58 | openstackgerrit | Chris Dent proposed openstack/nova master: WIP: [placement] Add /reshaper handler for POST https://review.openstack.org/576927 | |
| 21:03:06 | SpamapS | dansmith: one last question. Do you know how the neutron agent gets the tap device name it should be looking for? | |
| 21:07:25 | dansmith | SpamapS: nova meeting right now | |
| 21:07:36 | dansmith | SpamapS: it's tap${uuid} | |
| 21:07:42 | dansmith | where uuid is the port uuid truncated | |
| 21:07:43 | dansmith | or something | |
| 21:07:58 | dansmith | you should probably talk to neutron people at this point, since it seems clearly on that side of the fence | |
| 21:13:16 | dansmith | if I was PTL all our meetings would be over in 12 minutes | |
| 21:13:17 | dansmith | just sayin' | |
| 21:13:46 | tssurya | dansmith: :P | |
| 21:15:37 | openstackgerrit | Takashi NATSUME proposed openstack/nova-specs master: Create specs directory for Stein https://review.openstack.org/573602 | |
| 22:30:25 | SpamapS | dansmith: fyi, found the problem I think. All of the agents were stuck in the loop waiting for /run/ebtables.lock | |
| 22:30:36 | SpamapS | (even on restart, the lock persisted) | |
| 22:39:30 | openstackgerrit | Merged openstack/nova master: Add check if neutron "binding-extended" extension is available https://review.openstack.org/523548 | |
| 22:44:10 | openstackgerrit | Zack Cornelius proposed openstack/nova master: Fix nits from change Ia7cf4414feb335b3c2e863b4c8b4ff559b275c34 https://review.openstack.org/577314 | |
| 23:15:31 | SpamapS | now to figure out why my linuxbridge agents are using /run as their lock_path instead of /var/lib/neutron/tmp | |
| 23:58:45 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in test_compute_api.py (4) https://review.openstack.org/568462 | |
| 23:59:25 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in virt/test_block_device.py https://review.openstack.org/566153 | |
| 23:59:40 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in nova/tests/unit/virt/xenapi/stubs.py https://review.openstack.org/568412 | |
| #openstack-nova - 2018-06-22 | |||
| 01:54:14 | openstackgerrit | Tetsuro Nakamura proposed openstack/nova master: Add microversion for nested allocation candidate https://review.openstack.org/565487 | |
| 01:54:15 | openstackgerrit | Tetsuro Nakamura proposed openstack/nova master: Fix nits in placement-return-all-resources series https://review.openstack.org/573604 | |
| 02:02:52 | openstackgerrit | Merged openstack/nova master: Clarify placement DB schema migration https://review.openstack.org/576265 | |
| 02:05:08 | openstackgerrit | Tetsuro Nakamura proposed openstack/nova master: Add microversion for nested allocation candidate https://review.openstack.org/565487 | |
| 02:05:09 | openstackgerrit | Tetsuro Nakamura proposed openstack/nova master: Fix nits in placement-return-all-resources series https://review.openstack.org/573604 | |
| 02:52:34 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: api-ref: Fix parameters about trusted certificate IDs https://review.openstack.org/576046 | |