Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-06-21
20:07:36 SpamapS but neutron isn't really "cell aware" so I still suspect nova
20:07:53 SpamapS and can't seem to find the thing that nova does to tell neutron to activate a port
20:08:16 dansmith so, having looked through all the plug path stuff just now,
20:08:41 dansmith the only thing I can think of is if we create the tap, name it as we were told, and neutron agent on the compute notices that it got created, adds it to the bridge, and then fires the event
20:09:29 dansmith I thought we specifically called to neutron to tell it it was ready, but maybe we don't
20:09:35 SpamapS I think we do too
20:10:43 dansmith well, the plug path for linux bridge is pretty darn simple
20:11:22 SpamapS yeah, neutron even reports that it doesn't have to do anything when it gets triggered..
20:11:23 SpamapS when things are working
20:12:02 dansmith there is a neutron agent running on that box at the time, right/
20:12:18 dansmith and not in a container in such a way that it can't see the interfaces nova would create?
20:13:12 SpamapS yes the agent is running
20:13:22 SpamapS and they share a netns (deployed with kolla-ansible)
20:13:31 SpamapS they actually don't even have a netns
20:13:34 SpamapS they're on the host's networking
20:15:59 dansmith from looking through the agent (for the first time ever) it surely looks like it's just polling for new interfaces to pop up
20:18:49 SpamapS dansmith: I can verify that brctl show from inside the agent shows the taps
20:20:36 openstackgerrit Brianna Poulos proposed openstack/python-novaclient master: Fix trusted-image-certificate-id help text https://review.openstack.org/576555
20:23:25 dansmith yeah, so the agent runs a loop,
20:23:46 dansmith each time through it looks for new interfaces, calls process_network_devices(), which goes through each new one, ends up at plug_interface(), etc, etc
20:24:16 dansmith so I think we create a properly-named tap, it notices, does the plug into the bridge and the wiring and then emits the notification
20:24:39 dansmith which explains why I can't find any calls to neutron in the critical path while we're waiting
20:26:05 SpamapS dansmith: makes sense I 'spose
20:27:00 dansmith looks like with debug on,
20:27:12 dansmith the agent should log "Agent loop found changes!" and then the device info it found
20:27:23 dansmith do you see that happen once nova has created the taps?
20:28:24 dansmith this: https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/drivers/agent/_common_agent.py#L453-L453
20:30:38 SpamapS dansmith: right, we never get Agent loop found changes
20:30:45 SpamapS stracing now to see why
20:33:49 dansmith got confirmation from someone who would know that "plug" communication between neutron and nova just involves creating the interface
20:36:30 SpamapS k
20:36:55 SpamapS well I can see the taps from the linuxbridge container
20:37:07 SpamapS but I never see the agent actually polling brctl or ip or anything
20:37:39 dansmith well, it may use iproute directly
20:37:49 dansmith which would strace out as a socket(AF_NETLINK,...) early on
20:37:59 SpamapS "or ip" was my vague reference to iproute
20:38:07 SpamapS but .. oh that?
20:38:29 dansmith er, pyiproute I mean
20:39:01 dansmith `ip` is a CLI that speaks over a special type of socket called a "netlink" socket, which is a communication channel to the network subsystem inside the kernel
20:39:05 dansmith it's how it does all its magic
20:39:07 SpamapS yeah makes sense
20:39:11 dansmith anything else can speak that as well
20:40:28 dansmith if it did, it could also get async notification of networking events, so it would make sense
20:40:59 SpamapS no looks like it execs 'ip'
20:41:06 SpamapS or at least, sometimes it does
20:41:08 SpamapS still reading
20:41:30 SpamapS and digs around in /sys
20:42:45 dansmith https://github.com/openstack/os-vif/blob/master/os_vif/internal/command/ip/impl_pyroute2.py
20:43:12 dansmith 4
20:43:12 dansmith % grep import.*pyroute2 -r neutron/ | wc -l
20:43:21 dansmith so, neutron uses it some, in master at least
20:43:50 dansmith in privileged/agent/linux/ip_lib.py
20:44:00 dansmith anyway
20:44:11 openstackgerrit Matt Riedemann proposed openstack/nova master: Address nits from consumer generation https://review.openstack.org/577227
20:44:16 SpamapS not in pike
20:44:38 SpamapS neutron doesn't use os_vif directly even, AFAICT
20:45:00 dansmith I know it doesn't use os-vif
20:45:09 dansmith because os-vif is the integration library between nova and neutron
20:45:10 dansmith so naturally,
20:45:14 dansmith it's only used by nova
20:47:47 SpamapS in all the ways I see the agent code checking for interfaces, I can see the interface
20:47:52 SpamapS but the agent never "notices"
20:49:58 openstackgerrit Chris Dent proposed openstack/nova master: WIP: [placement] Add /reshaper handler for POST https://review.openstack.org/576927
21:03:06 SpamapS dansmith: one last question. Do you know how the neutron agent gets the tap device name it should be looking for?
21:07:25 dansmith SpamapS: nova meeting right now
21:07:36 dansmith SpamapS: it's tap${uuid}
21:07:42 dansmith where uuid is the port uuid truncated
21:07:43 dansmith or something
21:07:58 dansmith you should probably talk to neutron people at this point, since it seems clearly on that side of the fence
21:13:16 dansmith if I was PTL all our meetings would be over in 12 minutes
21:13:17 dansmith just sayin'
21:13:46 tssurya dansmith: :P
21:15:37 openstackgerrit Takashi NATSUME proposed openstack/nova-specs master: Create specs directory for Stein https://review.openstack.org/573602
22:30:25 SpamapS dansmith: fyi, found the problem I think. All of the agents were stuck in the loop waiting for /run/ebtables.lock
22:30:36 SpamapS (even on restart, the lock persisted)
22:39:30 openstackgerrit Merged openstack/nova master: Add check if neutron "binding-extended" extension is available https://review.openstack.org/523548
22:44:10 openstackgerrit Zack Cornelius proposed openstack/nova master: Fix nits from change Ia7cf4414feb335b3c2e863b4c8b4ff559b275c34 https://review.openstack.org/577314
23:15:31 SpamapS now to figure out why my linuxbridge agents are using /run as their lock_path instead of /var/lib/neutron/tmp
23:58:45 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in test_compute_api.py (4) https://review.openstack.org/568462
23:59:25 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in virt/test_block_device.py https://review.openstack.org/566153
23:59:40 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in nova/tests/unit/virt/xenapi/stubs.py https://review.openstack.org/568412
#openstack-nova - 2018-06-22
01:54:14 openstackgerrit Tetsuro Nakamura proposed openstack/nova master: Add microversion for nested allocation candidate https://review.openstack.org/565487
01:54:15 openstackgerrit Tetsuro Nakamura proposed openstack/nova master: Fix nits in placement-return-all-resources series https://review.openstack.org/573604
02:02:52 openstackgerrit Merged openstack/nova master: Clarify placement DB schema migration https://review.openstack.org/576265
02:05:08 openstackgerrit Tetsuro Nakamura proposed openstack/nova master: Add microversion for nested allocation candidate https://review.openstack.org/565487
02:05:09 openstackgerrit Tetsuro Nakamura proposed openstack/nova master: Fix nits in placement-return-all-resources series https://review.openstack.org/573604
02:52:34 openstackgerrit Takashi NATSUME proposed openstack/nova master: api-ref: Fix parameters about trusted certificate IDs https://review.openstack.org/576046
02:52:52 openstackgerrit Takashi NATSUME proposed openstack/nova master: Use nova.test.TestingException https://review.openstack.org/575012
02:53:32 openstackgerrit Takashi NATSUME proposed openstack/nova master: Update admin/flavors document https://review.openstack.org/573063
02:54:03 openstackgerrit Takashi NATSUME proposed openstack/nova master: api-ref: Example verification for servers.inc https://review.openstack.org/529520
03:08:48 openstackgerrit Merged openstack/nova master: Remove network info stubbing in functional test https://review.openstack.org/570018
03:08:54 openstackgerrit Merged openstack/nova master: Cleanup nits in placement database changes https://review.openstack.org/577107
03:10:28 openstackgerrit Chen proposed openstack/python-novaclient master: Fix help text in server-group-create https://review.openstack.org/577341
03:37:34 CBR09 Hi everyone
03:37:38 CBR09 Is there anybody here
03:37:57 CBR09 I want to ask something about upgrading nova
03:39:37 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Merge server create schema for BDM v2 extension https://review.openstack.org/577185
04:05:31 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in test_conductor.py (1) https://review.openstack.org/564658
04:05:48 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in test_conductor.py (2) https://review.openstack.org/564659
04:12:22 openstackgerrit jiang wei proposed openstack/nova master: Add action initiator attribute to the update payload https://review.openstack.org/536243

Earlier   Later