Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-12-04
14:08:07 alex_xu ok, failed to guess that
14:08:11 jaypipes :)
14:10:31 mnaser rocky introduced a new way of binding ports, right?
14:10:57 mnaser i have some questions around it, possibly around failures live migrating from a queens compute to rocky compute
14:20:14 openstackgerrit Guo Jingyu proposed openstack/nova master: Add rfb.VNC support for novncproxy https://review.openstack.org/622336
14:32:46 sean-k-mooney mnaser: for live migration yes
14:33:39 sean-k-mooney mnaser: we disabled the use of multiple port bindings entirely unles neutron and both the source and dest nodes supprot it
14:34:00 sean-k-mooney mnaser: so live migrating form queens to rocky will disable all the new code
14:34:05 mnaser sean-k-mooney: i have a case of a cloud where live migrating from a queens host to rocky host results in the vm being unpingable
14:34:12 mnaser unless you restart the neutron-openvswitch-agent
14:35:04 sean-k-mooney mnaser: that is likely not related to the multiple port binding changes
14:35:28 mnaser sean-k-mooney: well in a r=>r live migration situation, i dont see it
14:35:40 mnaser and after restarting neutron-openvswitch-agent, the vm becomes pingable afterwards, its almost like something isnt properly getting wired up
14:36:12 sean-k-mooney mnaser: did you leave the vm for a period of time or just try to ping it imediatly
14:36:22 mnaser sean-k-mooney: it was on for a while, it was def dead
14:37:09 sean-k-mooney are you using conntrack security group driver or ip tables?
14:37:56 sean-k-mooney basically im wonder if ovs_hybrid_plug in the port bindings was true or false
14:38:05 sean-k-mooney that changes how the port is wired up
14:39:47 mnaser sean-k-mooney: iptables
14:39:56 alex_xu sean-k-mooney: mriedem, I give a try on the resize bug https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bug/1805767, found we miss the pci request also
14:41:09 sean-k-mooney alex_xu: yes you cannot resize to or from a flavor with a pci device alais in the extra specs
14:41:32 sean-k-mooney this is know to be broken
14:42:02 sean-k-mooney actully resize is cold migrate
14:42:08 sean-k-mooney maybe that should work
14:42:08 alex_xu sean-k-mooney: ah, cool, hope the patch fixes that
14:42:41 alex_xu sean-k-mooney: or you say there is other bug for pci in resize more than just parse the extra specs?
14:42:43 sean-k-mooney alex_xu: we know we have prolems in this areay im not sure if that one is new thinking about it
14:42:53 alex_xu ok
14:43:25 sean-k-mooney ill take a look at the patach in any case. i can see if i can test it locally
14:43:37 sean-k-mooney but it will be a few days
14:44:12 alex_xu sean-k-mooney: thanks, I tested the numa one. But still looking for a hardware for pci, not easy to test
14:44:56 mnaser https://review.openstack.org/#/q/topic:bp/neutron-new-port-binding-api
14:45:59 sean-k-mooney mnaser: so iptables means hybrid plug is true which means that os-vif plugged a veth pari into ovs and a per port liunx bridge and libvirt add the vm tap to the linux bridge on migration
14:46:27 mnaser im trying to find here the specific patch that implemented the logic which checks if the dest supports it or not, i suspect *maybe* for some reason it's thinking the other side supports it
14:46:28 sean-k-mooney so neutron should of had time to wire up the ovs port before the vm migrated
14:46:46 sean-k-mooney we do it in the conductor one sec
14:47:45 sean-k-mooney https://github.com/openstack/nova/blob/stable/rocky/nova/conductor/tasks/live_migrate.py#L245-L252
14:48:51 sean-k-mooney this is the support_extended_port_binding fucntion https://github.com/openstack/nova/blob/stable/rocky/nova/conductor/tasks/live_migrate.py#L39
14:49:14 sean-k-mooney it just checks for service version >35 on the two nova compute services
14:50:39 sean-k-mooney so unless you have backport something in to you queens version that bumps the service version it should be pretty bulletproof
14:51:34 sean-k-mooney all of the new code uses the presence of the vifs object in the migration_data as a sentinel to enable the new workflow
14:53:08 mriedem alex_xu: ok, i see you based that on https://review.openstack.org/#/c/582417/ ? i've had a hard time sorting out why that fails
14:53:23 mriedem oh nvm the one below that - smart :)
14:59:18 mnaser sean-k-mooney: is it a valid possible theory that somehow rocky n-cpu doesn't try to plug the port because it assumes the conductor might have already done it
15:00:31 mnaser aka maybe we need to look at the logic in the compute side, ill try to dig in a bit more
15:00:58 MrRon gmann: efried: thanks a lot. That helped me a lot. Have a nice day :)
15:02:39 sean-k-mooney mnaser: no the the plug is initated in both queens and rocky by a direct rpc call form the conductor
15:03:35 mnaser sean-k-mooney: ok, i might be misunderstanding, so i might have some code to read
15:03:39 sean-k-mooney the difference is when is when the port binding are updated in neutron and do we have 1 or 2 bindings
15:03:48 mnaser oh so we always have a binding
15:03:55 sean-k-mooney yes
15:03:55 mnaser its just either 2 or 1
15:03:58 mnaser okay i see
15:04:15 mnaser okay okay, so that's why it would make sense that this wouldn't cause an issue
15:04:21 sean-k-mooney in queens we and one per port and then updated the host id as part of migration
15:04:45 mnaser hmm
15:04:57 sean-k-mooney in rocky we have a source and dest binding per port and atomicly set the dest as active and source as not
15:05:11 sean-k-mooney then if the migration successd we delete the source binding
15:05:51 mnaser because in this cloud i can do rocky=>rocky migrations with no problem
15:06:02 mnaser but queens=>rocky, they stop pinging until i restart n-ovs-agent
15:09:08 mnaser https://github.com/openstack/nova/blob/stable/rocky/nova/virt/libvirt/driver.py#L7073-L7094
15:09:35 mnaser that still is supposed to execute even in pre-new-port-binding in neutron right?
15:09:47 mnaser sorry for all the questions, im just trying to gather some sense to figure out where the issue is
15:09:52 sean-k-mooney no
15:10:11 sean-k-mooney vifs will not be in migrate_data in queens
15:10:21 mnaser hmmmmm
15:10:37 mnaser ok that's confusing because if you remember we had that bug with host_mtu
15:10:37 sean-k-mooney that is how we disable it on queens to rocky
15:10:55 mnaser oh sorry that was in rocky only
15:13:18 mnaser so im assuming that because 'vifs' was added in rocky, that field will be dropped by anything running queens
15:13:27 mnaser so that portion should be skipped
15:13:28 mnaser sigh
15:17:37 sean-k-mooney mnaser: yes but also the vifs field is only added in one place in the conductor and the conductor checks the versions before adding it
15:18:18 mdbooth o/
15:18:20 openstackgerrit garyk proposed openstack/nova master: Ensure that block device rollback is done when attachment fails https://review.openstack.org/622207
15:18:21 mnaser sean-k-mooney: yeah and im seeing similar workflows here https://github.com/openstack/nova/blob/stable/rocky/nova/virt/libvirt/driver.py#L7587-L7615 vs https://github.com/openstack/nova/blob/stable/queens/nova/virt/libvirt/driver.py#L7564-L7582
15:18:31 mnaser so re-running with debug enabled and seeing which path it picks
15:18:33 lyarwood mdbooth: https://review.openstack.org/#/c/618478/ - can you take a swing at that when you have time
15:23:38 sean-k-mooney mnaser: if it take the one it should not let me know and we can open a bug as we intended and tried really hard not to enable unless both supproted it
15:39:55 cfriesen curious what people think about Sylvain's suggestion to add the ability to query server groups by server UUID instead of including the server group in the server details.
15:40:10 cfriesen (in the context of https://review.openstack.org/#/c/612255)
15:50:29 bauzas cfriesen: mmm ?
15:50:56 bauzas hah
15:52:14 openstackgerrit Eric Fried proposed openstack/nova master: Reduce calls to placement from _ensure https://review.openstack.org/615677
15:52:15 openstackgerrit Eric Fried proposed openstack/nova master: Commonize _update code path https://review.openstack.org/615705
15:52:15 openstackgerrit Eric Fried proposed openstack/nova master: Consolidate inventory refresh https://review.openstack.org/615695
15:52:16 openstackgerrit Eric Fried proposed openstack/nova master: Use a static resource tracker in compute manager https://review.openstack.org/620711
15:52:16 openstackgerrit Eric Fried proposed openstack/nova master: Rip out the SchedulerClient https://review.openstack.org/617049
15:52:16 openstackgerrit Eric Fried proposed openstack/nova master: Rip the report client out of SchedulerClient https://review.openstack.org/617042
15:53:50 openstackgerrit Eric Fried proposed openstack/nova master: Turn off rp association refresh in nova-next https://review.openstack.org/616033
15:54:51 cfriesen bauzas: I think the suggestion could work, but it would probably mean that the CLI client would issue another HTTP request to get that info for each instance (unless we added a specific flag to tell it to show the groups, which seems not-friendly)
15:55:20 bauzas cfriesen: my biggest concern is about the inflation for the instance API
15:55:48 cfriesen bauzas: isn't that what pagination is for?
15:56:40 bauzas of course, but how many users use server groups ?
15:57:05 cfriesen among our users, anti-affinity is pretty common
16:00:27 cfriesen is there a way to do anti-affinity in placement? I know it was discussed at one point, not sure if anything came of it
16:00:50 cdent cfriesen: not got off the ground yet
16:01:02 cdent nested took all brains
16:05:57 openstackgerrit Jack Ding proposed openstack/nova master: [WIP] Preserve UEFI NVRAM variable store https://review.openstack.org/621646
16:36:42 openstackgerrit Merged openstack/python-novaclient master: Change openstack-dev to openstack-discuss https://review.openstack.org/621869

Earlier   Later