| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-07-17 | |||
| 13:08:23 | jkulik | Seems to be just that one instance, that's not deleted, though. Thank you. | |
| 13:09:05 | mnaser | i think the archive code didn't cover cell0 before and recently added support to go over all cells | |
| 13:09:06 | mnaser | if i'm remembering right | |
| 13:09:36 | sean-k-mooney | mnaser: so by default nova manage used to only look at cell1 | |
| 13:09:59 | sean-k-mooney | you could create a second config can use that to run the commands on cell0 | |
| 13:10:10 | mnaser | yeah, i get this feeling i remember there was an improvement that let you run it against all cells more recently | |
| 13:10:10 | sean-k-mooney | but i dont know if we have now made it do that by default or not | |
| 13:10:40 | sean-k-mooney | it came up as a downstream bug i think and melwitt? might have imporved it | |
| 13:11:01 | mnaser | sean-k-mooney: good memory https://review.opendev.org/#/c/507486/ | |
| 13:11:45 | sean-k-mooney | 11 months so proably train? | |
| 13:11:58 | mnaser | yes, the earliest tag this commit is part of is train | |
| 13:12:03 | mnaser | or 20.0.0 | |
| 13:12:10 | mnaser | so yeah, train onwards | |
| 13:12:19 | lyarwood | melwitt: https://review.opendev.org/#/c/741561/ - would you be able to take a look at this today? | |
| 13:13:00 | sean-k-mooney | im not sure if this is backportable or not but jkulik are you pre train | |
| 13:13:52 | jkulik | I'm fine for now, thank you. (we're on queens but going to upgrade soon. it can wait. the instance in question just wasn't marked deleted and thus still came up in "server list" and customers wondered ...) | |
| 13:14:19 | sean-k-mooney | if it was in cell0 then it would have been in error | |
| 13:14:36 | jkulik | It was actually in state BUILD | |
| 13:14:52 | sean-k-mooney | really? that should not happen | |
| 13:15:16 | sean-k-mooney | that likely means you had an rpc issue at some point that lost a callback to set it to error | |
| 13:15:25 | jkulik | but since that happened somewhere last year, I also don't have logs anymore. this being the only instance in that state, I think it can be ignored. | |
| 13:15:53 | sean-k-mooney | ya but if you do see this crop up more often let us know | |
| 13:16:13 | sean-k-mooney | queens is sufficently old that if you hit an issue its proably already fixed but still good to know | |
| 13:16:35 | mnaser | yeah, i've ran into a few of the isssues that jkulik ran into and cleaned them up in later release | |
| 13:16:46 | mnaser | mostly around making things happen in a single transaction to avoid BUILD inside cell0 | |
| 13:17:43 | sean-k-mooney | maybe that is something we shoudl add to nova audit | |
| 13:18:06 | mnaser | btw - debug from yesterday's issue exposes this https://www.irccloud.com/pastebin/FdpUKxPC/ | |
| 13:18:08 | sean-k-mooney | set instance in build in cell0 to error after x time | |
| 13:18:48 | mnaser | so it sounds like the actual commit to addport happened 34 seconds after | |
| 13:19:54 | sean-k-mooney | ya so if your using iptables os-vif need to create a linux bridge then a veth pair then add one end of the veth pair to the linux bridge and the other end to ovs | |
| 13:20:17 | mnaser | so maybe that part took a long time so i need to do some more log grepping | |
| 13:20:19 | sean-k-mooney | so that can take a few second although i would not normally expect it to be that long | |
| 13:20:54 | mnaser | system has 1157 total devices | |
| 13:21:00 | sean-k-mooney | going form vsctl to native seams to have cut it form 3 mins to ~30 seconds so thats an improvement at least | |
| 13:21:11 | mnaser | from qvo/qvb/qbr | |
| 13:21:34 | mnaser | yeah -- i still have instances that take 900s to provision so it might still be bad too | |
| 13:21:40 | sean-k-mooney | mnaser: if you change form iptables to conntrack then it elimiandes all the hybrid plug stuff so os-vif just add a port to ovs and is done | |
| 13:22:35 | mnaser | sean-k-mooney: yeah i have another one here that took 3 minutes too | |
| 13:22:45 | mnaser | so i suspect native vs ovsdb wasn't the big help here | |
| 13:22:47 | sean-k-mooney | likely the large number of ports is cause some of the operation to be slower | |
| 13:22:55 | sean-k-mooney | ah ok | |
| 13:23:34 | mnaser | sean-k-mooney: but firewall driver is defined in neutron? and this is happening inside nova? | |
| 13:23:53 | mnaser | does nova become somehow aware from what firewall driver is being used? | |
| 13:24:23 | sean-k-mooney | yes | |
| 13:24:50 | sean-k-mooney | the vif binding_details has a hybrid_plug atribute | |
| 13:25:01 | mnaser | ah gotcha | |
| 13:25:07 | sean-k-mooney | its true for iptables and false for contrac | |
| 13:25:21 | sean-k-mooney | os-vif just does what neutron tells it to do | |
| 13:25:33 | mnaser | and so when donig the PUT for the binding | |
| 13:25:39 | mnaser | we get the info back telling us how to do it | |
| 13:25:44 | sean-k-mooney | yep | |
| 13:25:55 | mnaser | ok i see. i wonder if i can restart straight into openvswitch firewall driver | |
| 13:26:05 | sean-k-mooney | we use that respocne both to generate the libvirt xml and pass it to os-vif to tell it how to add the port | |
| 13:26:16 | sean-k-mooney | not quite | |
| 13:26:28 | mnaser | i've looked at the code and it seems like it does have some code which 'remanages' it fwiw | |
| 13:26:28 | sean-k-mooney | you can but the ports on exiting vms wont be rebound | |
| 13:26:45 | sean-k-mooney | so you need to live migrate teh vms or shelve and unshelve them | |
| 13:27:00 | sean-k-mooney | i guess any move operation but hardreboot wont fix it | |
| 13:27:18 | mnaser | sean-k-mooney: have you seen this code? https://github.com/openstack/neutron/blob/master/neutron/agent/linux/openvswitch_firewall/iptables.py | |
| 13:27:39 | mnaser | i guess i don't really care that much if the existing vms stay hybrid plugged | |
| 13:28:11 | sean-k-mooney | so wehn you change it i think that code is nolonger loaded or run | |
| 13:28:12 | mnaser | as long as the new ones don't use hybrid plugging which likely will result in faster os_vif plug time | |
| 13:28:48 | sean-k-mooney | so i thnk you will lose the ablity to manage security group rules on the exsting vms | |
| 13:29:53 | sean-k-mooney | i cant remeber if it was stien we added multiple port bining but when we started using that in nova we got the ability to live migrate between hosts with different firewall driver | |
| 13:30:09 | sean-k-mooney | which will allow you to change in a rolling upgade style | |
| 13:31:44 | mnaser | sean-k-mooney: i think looking at the code, switching can work, it removes the iptables rules and then just starts applying the rules on the existing qvo iface | |
| 13:31:50 | mnaser | https://github.com/openstack/neutron/blob/master/neutron/agent/linux/openvswitch_firewall/firewall.py#L614-L636 | |
| 13:32:11 | sean-k-mooney | ah ok ya i just notice it did not change the hybridg plug state | |
| 13:32:23 | sean-k-mooney | if it actully removes the iptable rules then it will work fine | |
| 13:32:45 | sean-k-mooney | the existging vm will keep using hybrid_plug untill there ports are rebound | |
| 13:32:46 | mnaser | we've switched them in the past and it wasn't an issue | |
| 13:32:58 | mnaser | yep, but the firwall rules will apply on the qvo instead of tap | |
| 13:33:12 | sean-k-mooney | cool | |
| 13:33:16 | mnaser | and new plugs will plumb tap in and eliminate qbr/qvo/qvb | |
| 13:33:28 | sean-k-mooney | yep | |
| 13:33:33 | mnaser | which will significantly reduce the number of ports on the system | |
| 13:33:47 | sean-k-mooney | it will devide it by 3 | |
| 13:33:55 | stephenfin | lyarwood: replied on https://review.opendev.org/#/c/699291/ | |
| 13:34:01 | mnaser | also i think tap interfaces won't have a random ipv6 addr show up | |
| 13:34:15 | sean-k-mooney | that is also likely | |
| 13:34:26 | sean-k-mooney | well maybe not | |
| 13:34:37 | sean-k-mooney | let me check | |
| 13:34:49 | mnaser | dmesg shows ADDRCONF events for qvb and qvo | |
| 13:34:52 | mnaser | but none for tap | |
| 13:35:01 | sean-k-mooney | they still get link local adresses | |
| 13:35:24 | sean-k-mooney | that siad it is some thing that we could disable in os-vif | |
| 13:35:46 | stephenfin | yo, bauzas. Want some d̶r̶u̶g̶s̶ patches? https://review.opendev.org/#/c/729595/ https://review.opendev.org/#/c/729596/ | |
| 13:35:49 | sean-k-mooney | actully no we cant | |
| 13:36:42 | sean-k-mooney | when os-vif creates the port on ovs its before qemu has created the tap | |
| 13:36:55 | bauzas | stephenfin: https://media.tenor.com/images/a2b7c73a67cf6c1a775466e6ad87d8b7/tenor.gif | |
| 13:37:19 | stephenfin | :P | |
| 13:37:19 | bauzas | (you need to know about Weeds, dude) | |
| 13:37:40 | stephenfin | I know GIFs | |
| 13:38:19 | bauzas | could I provide a French spec ? :p | |
| 13:38:59 | stephenfin | Sure! Just hope you're happy with artom being the only one reviewing it | |
| 13:41:07 | mnaser | sean-k-mooney: check this out -- https://github.com/openstack/neutron/blob/23e3213a07eb0b0fcdd2a1da36a847dde9beba57/neutron/tests/fullstack/test_firewall.py | |
| 13:41:17 | mnaser | neutron tests switching to openvswitch with agent restart :) | |
| 13:42:59 | stephenfin | lyarwood: Also, in case you didn't know already, I removed the auto-branch naming feature from git-review. If you want topics for e.g. https://review.opendev.org/#/c/741561/ you need to create the branch yourself | |
| 13:43:50 | sean-k-mooney | mnaser: cool | |
| 13:44:07 | sean-k-mooney | mnaser: that makes the upgrade path much smother | |