Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-12-04
14:35:04 sean-k-mooney mnaser: that is likely not related to the multiple port binding changes
14:35:28 mnaser sean-k-mooney: well in a r=>r live migration situation, i dont see it
14:35:40 mnaser and after restarting neutron-openvswitch-agent, the vm becomes pingable afterwards, its almost like something isnt properly getting wired up
14:36:12 sean-k-mooney mnaser: did you leave the vm for a period of time or just try to ping it imediatly
14:36:22 mnaser sean-k-mooney: it was on for a while, it was def dead
14:37:09 sean-k-mooney are you using conntrack security group driver or ip tables?
14:37:56 sean-k-mooney basically im wonder if ovs_hybrid_plug in the port bindings was true or false
14:38:05 sean-k-mooney that changes how the port is wired up
14:39:47 mnaser sean-k-mooney: iptables
14:39:56 alex_xu sean-k-mooney: mriedem, I give a try on the resize bug https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bug/1805767, found we miss the pci request also
14:41:09 sean-k-mooney alex_xu: yes you cannot resize to or from a flavor with a pci device alais in the extra specs
14:41:32 sean-k-mooney this is know to be broken
14:42:02 sean-k-mooney actully resize is cold migrate
14:42:08 sean-k-mooney maybe that should work
14:42:08 alex_xu sean-k-mooney: ah, cool, hope the patch fixes that
14:42:41 alex_xu sean-k-mooney: or you say there is other bug for pci in resize more than just parse the extra specs?
14:42:43 sean-k-mooney alex_xu: we know we have prolems in this areay im not sure if that one is new thinking about it
14:42:53 alex_xu ok
14:43:25 sean-k-mooney ill take a look at the patach in any case. i can see if i can test it locally
14:43:37 sean-k-mooney but it will be a few days
14:44:12 alex_xu sean-k-mooney: thanks, I tested the numa one. But still looking for a hardware for pci, not easy to test
14:44:56 mnaser https://review.openstack.org/#/q/topic:bp/neutron-new-port-binding-api
14:45:59 sean-k-mooney mnaser: so iptables means hybrid plug is true which means that os-vif plugged a veth pari into ovs and a per port liunx bridge and libvirt add the vm tap to the linux bridge on migration
14:46:27 mnaser im trying to find here the specific patch that implemented the logic which checks if the dest supports it or not, i suspect *maybe* for some reason it's thinking the other side supports it
14:46:28 sean-k-mooney so neutron should of had time to wire up the ovs port before the vm migrated
14:46:46 sean-k-mooney we do it in the conductor one sec
14:47:45 sean-k-mooney https://github.com/openstack/nova/blob/stable/rocky/nova/conductor/tasks/live_migrate.py#L245-L252
14:48:51 sean-k-mooney this is the support_extended_port_binding fucntion https://github.com/openstack/nova/blob/stable/rocky/nova/conductor/tasks/live_migrate.py#L39
14:49:14 sean-k-mooney it just checks for service version >35 on the two nova compute services
14:50:39 sean-k-mooney so unless you have backport something in to you queens version that bumps the service version it should be pretty bulletproof
14:51:34 sean-k-mooney all of the new code uses the presence of the vifs object in the migration_data as a sentinel to enable the new workflow
14:53:08 mriedem alex_xu: ok, i see you based that on https://review.openstack.org/#/c/582417/ ? i've had a hard time sorting out why that fails
14:53:23 mriedem oh nvm the one below that - smart :)
14:59:18 mnaser sean-k-mooney: is it a valid possible theory that somehow rocky n-cpu doesn't try to plug the port because it assumes the conductor might have already done it
15:00:31 mnaser aka maybe we need to look at the logic in the compute side, ill try to dig in a bit more
15:00:58 MrRon gmann: efried: thanks a lot. That helped me a lot. Have a nice day :)
15:02:39 sean-k-mooney mnaser: no the the plug is initated in both queens and rocky by a direct rpc call form the conductor
15:03:35 mnaser sean-k-mooney: ok, i might be misunderstanding, so i might have some code to read
15:03:39 sean-k-mooney the difference is when is when the port binding are updated in neutron and do we have 1 or 2 bindings
15:03:48 mnaser oh so we always have a binding
15:03:55 sean-k-mooney yes
15:03:55 mnaser its just either 2 or 1
15:03:58 mnaser okay i see
15:04:15 mnaser okay okay, so that's why it would make sense that this wouldn't cause an issue
15:04:21 sean-k-mooney in queens we and one per port and then updated the host id as part of migration
15:04:45 mnaser hmm
15:04:57 sean-k-mooney in rocky we have a source and dest binding per port and atomicly set the dest as active and source as not
15:05:11 sean-k-mooney then if the migration successd we delete the source binding
15:05:51 mnaser because in this cloud i can do rocky=>rocky migrations with no problem
15:06:02 mnaser but queens=>rocky, they stop pinging until i restart n-ovs-agent
15:09:08 mnaser https://github.com/openstack/nova/blob/stable/rocky/nova/virt/libvirt/driver.py#L7073-L7094
15:09:35 mnaser that still is supposed to execute even in pre-new-port-binding in neutron right?
15:09:47 mnaser sorry for all the questions, im just trying to gather some sense to figure out where the issue is
15:09:52 sean-k-mooney no
15:10:11 sean-k-mooney vifs will not be in migrate_data in queens
15:10:21 mnaser hmmmmm
15:10:37 mnaser ok that's confusing because if you remember we had that bug with host_mtu
15:10:37 sean-k-mooney that is how we disable it on queens to rocky
15:10:55 mnaser oh sorry that was in rocky only
15:13:18 mnaser so im assuming that because 'vifs' was added in rocky, that field will be dropped by anything running queens
15:13:27 mnaser so that portion should be skipped
15:13:28 mnaser sigh
15:17:37 sean-k-mooney mnaser: yes but also the vifs field is only added in one place in the conductor and the conductor checks the versions before adding it
15:18:18 mdbooth o/
15:18:20 openstackgerrit garyk proposed openstack/nova master: Ensure that block device rollback is done when attachment fails https://review.openstack.org/622207
15:18:21 mnaser sean-k-mooney: yeah and im seeing similar workflows here https://github.com/openstack/nova/blob/stable/rocky/nova/virt/libvirt/driver.py#L7587-L7615 vs https://github.com/openstack/nova/blob/stable/queens/nova/virt/libvirt/driver.py#L7564-L7582
15:18:31 mnaser so re-running with debug enabled and seeing which path it picks
15:18:33 lyarwood mdbooth: https://review.openstack.org/#/c/618478/ - can you take a swing at that when you have time
15:23:38 sean-k-mooney mnaser: if it take the one it should not let me know and we can open a bug as we intended and tried really hard not to enable unless both supproted it
15:39:55 cfriesen curious what people think about Sylvain's suggestion to add the ability to query server groups by server UUID instead of including the server group in the server details.
15:40:10 cfriesen (in the context of https://review.openstack.org/#/c/612255)
15:50:29 bauzas cfriesen: mmm ?
15:50:56 bauzas hah
15:52:14 openstackgerrit Eric Fried proposed openstack/nova master: Reduce calls to placement from _ensure https://review.openstack.org/615677
15:52:15 openstackgerrit Eric Fried proposed openstack/nova master: Commonize _update code path https://review.openstack.org/615705
15:52:15 openstackgerrit Eric Fried proposed openstack/nova master: Consolidate inventory refresh https://review.openstack.org/615695
15:52:16 openstackgerrit Eric Fried proposed openstack/nova master: Use a static resource tracker in compute manager https://review.openstack.org/620711
15:52:16 openstackgerrit Eric Fried proposed openstack/nova master: Rip out the SchedulerClient https://review.openstack.org/617049
15:52:16 openstackgerrit Eric Fried proposed openstack/nova master: Rip the report client out of SchedulerClient https://review.openstack.org/617042
15:53:50 openstackgerrit Eric Fried proposed openstack/nova master: Turn off rp association refresh in nova-next https://review.openstack.org/616033
15:54:51 cfriesen bauzas: I think the suggestion could work, but it would probably mean that the CLI client would issue another HTTP request to get that info for each instance (unless we added a specific flag to tell it to show the groups, which seems not-friendly)
15:55:20 bauzas cfriesen: my biggest concern is about the inflation for the instance API
15:55:48 cfriesen bauzas: isn't that what pagination is for?
15:56:40 bauzas of course, but how many users use server groups ?
15:57:05 cfriesen among our users, anti-affinity is pretty common
16:00:27 cfriesen is there a way to do anti-affinity in placement? I know it was discussed at one point, not sure if anything came of it
16:00:50 cdent cfriesen: not got off the ground yet
16:01:02 cdent nested took all brains
16:05:57 openstackgerrit Jack Ding proposed openstack/nova master: [WIP] Preserve UEFI NVRAM variable store https://review.openstack.org/621646
16:36:42 openstackgerrit Merged openstack/python-novaclient master: Change openstack-dev to openstack-discuss https://review.openstack.org/621869
16:49:14 openstackgerrit Balazs Gibizer proposed openstack/nova master: Remove port allocation during detach https://review.openstack.org/622421
16:56:07 mriedem cfriesen: i think i also suggested that in denver
16:56:26 mriedem ala https://developer.openstack.org/api-ref/compute/?expanded=#list-port-interfaces
16:56:33 mriedem https://developer.openstack.org/api-ref/compute/?expanded=#list-volume-attachments-for-an-instance
16:58:15 mriedem although i guess that wasn't noted in the etherpad, so maybe i thought it elsewhere
16:58:18 cfriesen mriedem: you mean querying server groups by instance?
16:58:34 mriedem yes, rather than return server group information in the servers resource api,
16:58:40 mriedem you filter server groups by member
16:59:03 sean-k-mooney jaypipes: just spent the last few minute talking to the kuryr folks they are now aware of the ML post and will follow up there.
16:59:37 bauzas cfriesen: I agree with mriedem

Earlier   Later