| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-07-17 | |||
| 09:55:11 | gibi | sean-k-mooney, stephenfin: fyi I pushed a small spec for SRIOV attach https://review.opendev.org/#/c/741601/ | |
| 10:22:21 | openstackgerrit | Wenping Song proposed openstack/nova master: Cyborg shelve/unshelve support https://review.opendev.org/729563 | |
| 10:26:52 | brinzhang0 | gibi: hi, I update the nova runway, added the cyborg-shelve-and-unshelve patches to the queue | |
| 10:46:51 | stephenfin | gibi: I lost the race for some o.vo versions. Could I get your +2 on this again? https://review.opendev.org/#/c/739210/ | |
| 10:51:29 | stephenfin | gibi: -1 on spec for nits but otherwise perfect | |
| 10:53:05 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Re-enable live snapshot for paused instances https://review.opendev.org/740334 | |
| 10:53:06 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Use better variable names for '_create_guest' https://review.opendev.org/740945 | |
| 10:53:06 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Remove workaround for really old QEMU https://review.opendev.org/740335 | |
| 10:53:07 | openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: Attempt to restore some sanity to snapshot https://review.opendev.org/740336 | |
| 10:53:51 | stephenfin | lyarwood: The first two of those are pretty simple but would benefit from your eyes, IMO. The third is a FUP for your comments ^ | |
| 10:56:38 | kashyap | stephenfin: Good catch on re-enabling the 'paused' thing | |
| 10:56:40 | kashyap | I recall the old change | |
| 10:57:25 | kashyap | Yep, it makes sense, given how far we've come w/ newer (and largely improved) versions | |
| 11:37:06 | sean-k-mooney | gibi: ill read your sepc shortly. did you see my comments on the code change. | |
| 11:37:47 | sean-k-mooney | your code change wont fix PF detach either so i might just add a small patch on top to do that | |
| 12:04:10 | openstackgerrit | Stephen Finucane proposed openstack/nova master: api: Add type hints for 'nova.compute.api' https://review.opendev.org/741623 | |
| 12:04:11 | openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: api: Add 'nova.compute.API.migrate' https://review.opendev.org/741624 | |
| 12:05:47 | openstackgerrit | Elod Illes proposed openstack/nova stable/queens: Reject boot request for unsupported images https://review.opendev.org/708609 | |
| 12:21:03 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Handle VIR_ERR_DEVICE_MISSING when detaching devices https://review.opendev.org/741561 | |
| 13:01:09 | jkulik | How often and how do you clean up your instances table in the cell0 db? | |
| 13:01:52 | sean-k-mooney | i think you do it with nova-manage | |
| 13:02:13 | sean-k-mooney | how often is up to you | |
| 13:02:29 | sean-k-mooney | i think downstream we default to every 30days | |
| 13:03:52 | jkulik | hm ... I thought so, but I can't seem to find the docs or the code for that | |
| 13:05:10 | sean-k-mooney | i think you do "nova-manage db archive_deleted_rows" then "nova-manage db purge" | |
| 13:05:28 | sean-k-mooney | but you need to us a config file with the cell0 credentials | |
| 13:06:37 | gibi | stephenfin: thanks. | |
| 13:06:50 | jkulik | deleted_at is NULL for the instance I'm looking at. it's over 200 days old | |
| 13:06:51 | gibi | sean-k-mooney: I haven't read the comments on the code yet | |
| 13:07:26 | gibi | sean-k-mooney: I have some review to make before I can circle back to the sriov code patches | |
| 13:07:37 | sean-k-mooney | gibi: there is no rush | |
| 13:08:02 | sean-k-mooney | it can wait till next week | |
| 13:08:20 | gibi | ack | |
| 13:08:23 | jkulik | Seems to be just that one instance, that's not deleted, though. Thank you. | |
| 13:09:05 | mnaser | i think the archive code didn't cover cell0 before and recently added support to go over all cells | |
| 13:09:06 | mnaser | if i'm remembering right | |
| 13:09:36 | sean-k-mooney | mnaser: so by default nova manage used to only look at cell1 | |
| 13:09:59 | sean-k-mooney | you could create a second config can use that to run the commands on cell0 | |
| 13:10:10 | sean-k-mooney | but i dont know if we have now made it do that by default or not | |
| 13:10:10 | mnaser | yeah, i get this feeling i remember there was an improvement that let you run it against all cells more recently | |
| 13:10:40 | sean-k-mooney | it came up as a downstream bug i think and melwitt? might have imporved it | |
| 13:11:01 | mnaser | sean-k-mooney: good memory https://review.opendev.org/#/c/507486/ | |
| 13:11:45 | sean-k-mooney | 11 months so proably train? | |
| 13:11:58 | mnaser | yes, the earliest tag this commit is part of is train | |
| 13:12:03 | mnaser | or 20.0.0 | |
| 13:12:10 | mnaser | so yeah, train onwards | |
| 13:12:19 | lyarwood | melwitt: https://review.opendev.org/#/c/741561/ - would you be able to take a look at this today? | |
| 13:13:00 | sean-k-mooney | im not sure if this is backportable or not but jkulik are you pre train | |
| 13:13:52 | jkulik | I'm fine for now, thank you. (we're on queens but going to upgrade soon. it can wait. the instance in question just wasn't marked deleted and thus still came up in "server list" and customers wondered ...) | |
| 13:14:19 | sean-k-mooney | if it was in cell0 then it would have been in error | |
| 13:14:36 | jkulik | It was actually in state BUILD | |
| 13:14:52 | sean-k-mooney | really? that should not happen | |
| 13:15:16 | sean-k-mooney | that likely means you had an rpc issue at some point that lost a callback to set it to error | |
| 13:15:25 | jkulik | but since that happened somewhere last year, I also don't have logs anymore. this being the only instance in that state, I think it can be ignored. | |
| 13:15:53 | sean-k-mooney | ya but if you do see this crop up more often let us know | |
| 13:16:13 | sean-k-mooney | queens is sufficently old that if you hit an issue its proably already fixed but still good to know | |
| 13:16:35 | mnaser | yeah, i've ran into a few of the isssues that jkulik ran into and cleaned them up in later release | |
| 13:16:46 | mnaser | mostly around making things happen in a single transaction to avoid BUILD inside cell0 | |
| 13:17:43 | sean-k-mooney | maybe that is something we shoudl add to nova audit | |
| 13:18:06 | mnaser | btw - debug from yesterday's issue exposes this https://www.irccloud.com/pastebin/FdpUKxPC/ | |
| 13:18:08 | sean-k-mooney | set instance in build in cell0 to error after x time | |
| 13:18:48 | mnaser | so it sounds like the actual commit to addport happened 34 seconds after | |
| 13:19:54 | sean-k-mooney | ya so if your using iptables os-vif need to create a linux bridge then a veth pair then add one end of the veth pair to the linux bridge and the other end to ovs | |
| 13:20:17 | mnaser | so maybe that part took a long time so i need to do some more log grepping | |
| 13:20:19 | sean-k-mooney | so that can take a few second although i would not normally expect it to be that long | |
| 13:20:54 | mnaser | system has 1157 total devices | |
| 13:21:00 | sean-k-mooney | going form vsctl to native seams to have cut it form 3 mins to ~30 seconds so thats an improvement at least | |
| 13:21:11 | mnaser | from qvo/qvb/qbr | |
| 13:21:34 | mnaser | yeah -- i still have instances that take 900s to provision so it might still be bad too | |
| 13:21:40 | sean-k-mooney | mnaser: if you change form iptables to conntrack then it elimiandes all the hybrid plug stuff so os-vif just add a port to ovs and is done | |
| 13:22:35 | mnaser | sean-k-mooney: yeah i have another one here that took 3 minutes too | |
| 13:22:45 | mnaser | so i suspect native vs ovsdb wasn't the big help here | |
| 13:22:47 | sean-k-mooney | likely the large number of ports is cause some of the operation to be slower | |
| 13:22:55 | sean-k-mooney | ah ok | |
| 13:23:34 | mnaser | sean-k-mooney: but firewall driver is defined in neutron? and this is happening inside nova? | |
| 13:23:53 | mnaser | does nova become somehow aware from what firewall driver is being used? | |
| 13:24:23 | sean-k-mooney | yes | |
| 13:24:50 | sean-k-mooney | the vif binding_details has a hybrid_plug atribute | |
| 13:25:01 | mnaser | ah gotcha | |
| 13:25:07 | sean-k-mooney | its true for iptables and false for contrac | |
| 13:25:21 | sean-k-mooney | os-vif just does what neutron tells it to do | |
| 13:25:33 | mnaser | and so when donig the PUT for the binding | |
| 13:25:39 | mnaser | we get the info back telling us how to do it | |
| 13:25:44 | sean-k-mooney | yep | |
| 13:25:55 | mnaser | ok i see. i wonder if i can restart straight into openvswitch firewall driver | |
| 13:26:05 | sean-k-mooney | we use that respocne both to generate the libvirt xml and pass it to os-vif to tell it how to add the port | |
| 13:26:16 | sean-k-mooney | not quite | |
| 13:26:28 | sean-k-mooney | you can but the ports on exiting vms wont be rebound | |
| 13:26:28 | mnaser | i've looked at the code and it seems like it does have some code which 'remanages' it fwiw | |
| 13:26:45 | sean-k-mooney | so you need to live migrate teh vms or shelve and unshelve them | |
| 13:27:00 | sean-k-mooney | i guess any move operation but hardreboot wont fix it | |
| 13:27:18 | mnaser | sean-k-mooney: have you seen this code? https://github.com/openstack/neutron/blob/master/neutron/agent/linux/openvswitch_firewall/iptables.py | |
| 13:27:39 | mnaser | i guess i don't really care that much if the existing vms stay hybrid plugged | |
| 13:28:11 | sean-k-mooney | so wehn you change it i think that code is nolonger loaded or run | |
| 13:28:12 | mnaser | as long as the new ones don't use hybrid plugging which likely will result in faster os_vif plug time | |
| 13:28:48 | sean-k-mooney | so i thnk you will lose the ablity to manage security group rules on the exsting vms | |
| 13:29:53 | sean-k-mooney | i cant remeber if it was stien we added multiple port bining but when we started using that in nova we got the ability to live migrate between hosts with different firewall driver | |
| 13:30:09 | sean-k-mooney | which will allow you to change in a rolling upgade style | |
| 13:31:44 | mnaser | sean-k-mooney: i think looking at the code, switching can work, it removes the iptables rules and then just starts applying the rules on the existing qvo iface | |
| 13:31:50 | mnaser | https://github.com/openstack/neutron/blob/master/neutron/agent/linux/openvswitch_firewall/firewall.py#L614-L636 | |