| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-12-04 | |||
| 13:53:04 | cdent | \o/ | |
| 13:53:38 | jaypipes | cdent: :) | |
| 13:53:51 | jaypipes | Clipper4 or FoxPro, baby! | |
| 13:54:07 | cdent | totes foxpro | |
| 13:54:23 | jaypipes | also, throw in a nice dose of CSV files for good measure. | |
| 13:54:47 | jaypipes | and replace configdrive with ftp | |
| 13:56:38 | cdent | let's do TSV files to be extra sure | |
| 13:56:46 | cdent | alex_xu++ | |
| 13:56:48 | gmann | efried: MrRon https://review.openstack.org/#/c/448482/ that time, there were only scheduler test case in that list which could be covered in functional tests. | |
| 13:56:55 | jaypipes | alex_xu: LOL! :) | |
| 13:57:14 | jangutter | jaypipes, cdent: Ashton-Tate's dBASE IV, accept no substitute. | |
| 14:00:02 | jaypipes | jangutter: I used to use that. No joke... | |
| 14:00:29 | efried | jaypipes: Are you waiting for belmoreira/tssurya to spin up ironic nodes with the series at https://review.openstack.org/#/c/615677/ before reviewing? | |
| 14:00:43 | jaypipes | jangutter, alex_xu, cdent: pop quiz... what was Nova's original data store? | |
| 14:00:59 | jangutter | jaypipes: I think 90% of Apartheid-era South Africa relied on one pirated copy of dBASE III. | |
| 14:01:21 | jaypipes | jangutter, alex_xu, cdent: believe it or not, Redis was used as Nova's original data store. | |
| 14:01:33 | jangutter | jaypipes: :-O | |
| 14:01:38 | jaypipes | yup. | |
| 14:01:42 | edleafe | My first contributions to nova was removing redis | |
| 14:01:53 | jaypipes | edleafe: yup, I remember it well. | |
| 14:01:53 | alex_xu | ....what... | |
| 14:02:01 | jaypipes | alex_xu: yep, hard to believe, eh?: :) | |
| 14:02:10 | edleafe | I had an equally negative reaction when I learned that | |
| 14:02:25 | jangutter | jaypipes: it was a simpler time, when all Nova needed was a key-value store? | |
| 14:02:26 | edleafe | Hey, nova was thrown together over a weekend | |
| 14:02:26 | jaypipes | edleafe: and we've been data-battling ever since... | |
| 14:02:45 | edleafe | it worked well enough for a POC | |
| 14:03:02 | jaypipes | jangutter: erm, no :) it was a perfect example of how to use Redis as an RDBMS and implement JOINs in Python. | |
| 14:03:20 | alex_xu | jaypipes: yea...I will talk this to everyone when lunch | |
| 14:04:25 | jaypipes | alex_xu: here's another trivia piece for you... there is an entire copy of openwrt Linux distribution in the Nova source repository. | |
| 14:06:53 | alex_xu | jaypipes: yes...begin to brainstorming what is it for... | |
| 14:07:22 | jaypipes | alex_xu: vish added it as a way to do functional SSH testing way back when... | |
| 14:08:07 | alex_xu | ok, failed to guess that | |
| 14:08:11 | jaypipes | :) | |
| 14:10:31 | mnaser | rocky introduced a new way of binding ports, right? | |
| 14:10:57 | mnaser | i have some questions around it, possibly around failures live migrating from a queens compute to rocky compute | |
| 14:20:14 | openstackgerrit | Guo Jingyu proposed openstack/nova master: Add rfb.VNC support for novncproxy https://review.openstack.org/622336 | |
| 14:32:46 | sean-k-mooney | mnaser: for live migration yes | |
| 14:33:39 | sean-k-mooney | mnaser: we disabled the use of multiple port bindings entirely unles neutron and both the source and dest nodes supprot it | |
| 14:34:00 | sean-k-mooney | mnaser: so live migrating form queens to rocky will disable all the new code | |
| 14:34:05 | mnaser | sean-k-mooney: i have a case of a cloud where live migrating from a queens host to rocky host results in the vm being unpingable | |
| 14:34:12 | mnaser | unless you restart the neutron-openvswitch-agent | |
| 14:35:04 | sean-k-mooney | mnaser: that is likely not related to the multiple port binding changes | |
| 14:35:28 | mnaser | sean-k-mooney: well in a r=>r live migration situation, i dont see it | |
| 14:35:40 | mnaser | and after restarting neutron-openvswitch-agent, the vm becomes pingable afterwards, its almost like something isnt properly getting wired up | |
| 14:36:12 | sean-k-mooney | mnaser: did you leave the vm for a period of time or just try to ping it imediatly | |
| 14:36:22 | mnaser | sean-k-mooney: it was on for a while, it was def dead | |
| 14:37:09 | sean-k-mooney | are you using conntrack security group driver or ip tables? | |
| 14:37:56 | sean-k-mooney | basically im wonder if ovs_hybrid_plug in the port bindings was true or false | |
| 14:38:05 | sean-k-mooney | that changes how the port is wired up | |
| 14:39:47 | mnaser | sean-k-mooney: iptables | |
| 14:39:56 | alex_xu | sean-k-mooney: mriedem, I give a try on the resize bug https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bug/1805767, found we miss the pci request also | |
| 14:41:09 | sean-k-mooney | alex_xu: yes you cannot resize to or from a flavor with a pci device alais in the extra specs | |
| 14:41:32 | sean-k-mooney | this is know to be broken | |
| 14:42:02 | sean-k-mooney | actully resize is cold migrate | |
| 14:42:08 | sean-k-mooney | maybe that should work | |
| 14:42:08 | alex_xu | sean-k-mooney: ah, cool, hope the patch fixes that | |
| 14:42:41 | alex_xu | sean-k-mooney: or you say there is other bug for pci in resize more than just parse the extra specs? | |
| 14:42:43 | sean-k-mooney | alex_xu: we know we have prolems in this areay im not sure if that one is new thinking about it | |
| 14:42:53 | alex_xu | ok | |
| 14:43:25 | sean-k-mooney | ill take a look at the patach in any case. i can see if i can test it locally | |
| 14:43:37 | sean-k-mooney | but it will be a few days | |
| 14:44:12 | alex_xu | sean-k-mooney: thanks, I tested the numa one. But still looking for a hardware for pci, not easy to test | |
| 14:44:56 | mnaser | https://review.openstack.org/#/q/topic:bp/neutron-new-port-binding-api | |
| 14:45:59 | sean-k-mooney | mnaser: so iptables means hybrid plug is true which means that os-vif plugged a veth pari into ovs and a per port liunx bridge and libvirt add the vm tap to the linux bridge on migration | |
| 14:46:27 | mnaser | im trying to find here the specific patch that implemented the logic which checks if the dest supports it or not, i suspect *maybe* for some reason it's thinking the other side supports it | |
| 14:46:28 | sean-k-mooney | so neutron should of had time to wire up the ovs port before the vm migrated | |
| 14:46:46 | sean-k-mooney | we do it in the conductor one sec | |
| 14:47:45 | sean-k-mooney | https://github.com/openstack/nova/blob/stable/rocky/nova/conductor/tasks/live_migrate.py#L245-L252 | |
| 14:48:51 | sean-k-mooney | this is the support_extended_port_binding fucntion https://github.com/openstack/nova/blob/stable/rocky/nova/conductor/tasks/live_migrate.py#L39 | |
| 14:49:14 | sean-k-mooney | it just checks for service version >35 on the two nova compute services | |
| 14:50:39 | sean-k-mooney | so unless you have backport something in to you queens version that bumps the service version it should be pretty bulletproof | |
| 14:51:34 | sean-k-mooney | all of the new code uses the presence of the vifs object in the migration_data as a sentinel to enable the new workflow | |
| 14:53:08 | mriedem | alex_xu: ok, i see you based that on https://review.openstack.org/#/c/582417/ ? i've had a hard time sorting out why that fails | |
| 14:53:23 | mriedem | oh nvm the one below that - smart :) | |
| 14:59:18 | mnaser | sean-k-mooney: is it a valid possible theory that somehow rocky n-cpu doesn't try to plug the port because it assumes the conductor might have already done it | |
| 15:00:31 | mnaser | aka maybe we need to look at the logic in the compute side, ill try to dig in a bit more | |
| 15:00:58 | MrRon | gmann: efried: thanks a lot. That helped me a lot. Have a nice day :) | |
| 15:02:39 | sean-k-mooney | mnaser: no the the plug is initated in both queens and rocky by a direct rpc call form the conductor | |
| 15:03:35 | mnaser | sean-k-mooney: ok, i might be misunderstanding, so i might have some code to read | |
| 15:03:39 | sean-k-mooney | the difference is when is when the port binding are updated in neutron and do we have 1 or 2 bindings | |
| 15:03:48 | mnaser | oh so we always have a binding | |
| 15:03:55 | sean-k-mooney | yes | |
| 15:03:55 | mnaser | its just either 2 or 1 | |
| 15:03:58 | mnaser | okay i see | |
| 15:04:15 | mnaser | okay okay, so that's why it would make sense that this wouldn't cause an issue | |
| 15:04:21 | sean-k-mooney | in queens we and one per port and then updated the host id as part of migration | |
| 15:04:45 | mnaser | hmm | |
| 15:04:57 | sean-k-mooney | in rocky we have a source and dest binding per port and atomicly set the dest as active and source as not | |
| 15:05:11 | sean-k-mooney | then if the migration successd we delete the source binding | |
| 15:05:51 | mnaser | because in this cloud i can do rocky=>rocky migrations with no problem | |
| 15:06:02 | mnaser | but queens=>rocky, they stop pinging until i restart n-ovs-agent | |
| 15:09:08 | mnaser | https://github.com/openstack/nova/blob/stable/rocky/nova/virt/libvirt/driver.py#L7073-L7094 | |
| 15:09:35 | mnaser | that still is supposed to execute even in pre-new-port-binding in neutron right? | |
| 15:09:47 | mnaser | sorry for all the questions, im just trying to gather some sense to figure out where the issue is | |
| 15:09:52 | sean-k-mooney | no | |
| 15:10:11 | sean-k-mooney | vifs will not be in migrate_data in queens | |
| 15:10:21 | mnaser | hmmmmm | |
| 15:10:37 | mnaser | ok that's confusing because if you remember we had that bug with host_mtu | |
| 15:10:37 | sean-k-mooney | that is how we disable it on queens to rocky | |